Submit Job
dataproc.submit_job
ActionGoogle Cloudconnection required
Calls Dataproc POST /regions/{r}/jobs:submit.
Submit a PySpark, Spark SQL or Hive job to a Dataproc cluster.
Connection
- Required
- Yes — the node will not save without one
- Auth type
- OAuth 2.0
- Providers
- Google Cloud Dataproc
- Secret shape
- {"client_id", "client_secret", "refresh_token"}
- Config it needs
- project_id, region
- Resolved by
- config.connection_id, else node.connection_id
Configuration
7 fields, 1 required. A dotted path is a nested object and name[] an array of objects — the editor renders the same tree.
| Field | Type | Required | Default | Constraints | Example |
|---|---|---|---|---|---|
project_id GCP project id (overrides connection). | string | No | — | — | — |
region GCP region (overrides connection). | string | No | — | — | — |
cluster_name Target cluster. | string | Yes | — | — | — |
job_type Job kind — Dataproc accepts only the listed values. | string | No | pyspark | one of "pyspark", "spark_sql", "hive" | — |
main_python_file_uri gs:// URI of the PySpark driver (pyspark). | string | No | — | — | — |
args Driver arguments as JSON array (pyspark). | array | No | — | array of string | — |
query SQL / HiveQL text (spark_sql, hive). | string | No | — | — | — |
- At least one of main_python_file_uri, query must be set.
Example configuration
Every value below is the contract’s own example or default — nothing invented.
{
"job_type": "pyspark"
}Output
One shape.
result
| Field | Type | Example |
|---|---|---|
job_id | string | — |
state | string | — |
cluster_name | string | — |
done | boolean | — |
{
"cluster_name": "",
"done": false,
"job_id": "",
"state": ""
}Related Google Cloud Dataproc nodes
See Submit Job run in a workflow.
In a technical demo we add the Google Cloud Dataproc connection, drop the node on the canvas and let Workflow Health pre-flight it before the first real run.