dataintermediate20 min setupby Mantle

Nightly CSV feed to SQL

Download a CSV export every night, parse it, normalise the columns with the Process engine and upsert into a table.

Use this template See it run with us

React Flow

Drag to pan, pinch to zoom. This is the real graph the template installs.

How it works

What it does

Cron 0 2 * * * → download the CSV → Parse CSV (headers, rows, row_count).
Normalise columns runs Process-engine cards (trim/lowercase email, title-case name, literal source) with a required-email rule that aborts on bad rows.
Upsert customers writes the shaped record keyed on email.

Setup

Set the CSV url, pick your MySQL connection on Upsert customers, and set table plus the column mapping in row.
Open Normalise columns to edit the operation cards in the Process editor.
To process every row of a large file, insert a For Each over steps.parse_csv.rows before Normalise columns (see the "Fan out list to API" template).

Nodes 5

01Nightly at 02:00trigger.scheduleTrigger
02Download CSVhttp.requestHTTP
03Parse CSVfiles.parse_csvData
04Normalise columnsdata.processData
05Upsert customersdb.upsertDestination

View template JSON

{
  "format": "mantle-workflow/v1",
  "id": "data-csv-sync-to-sql",
  "name": "Nightly CSV feed to SQL",
  "graph": {
    "nodes": [
      {
        "id": "schedule",
        "type": "trigger.schedule",
        "label": "Nightly at 02:00",
        "config": {
          "cron_expr": "0 2 * * *",
          "timezone": "UTC"
        },
        "connection_id": null,
        "worker_type": "generic-worker",
        "position": {
          "x": 60,
          "y": 220
        }
      },
      {
        "id": "fetch_csv",
        "type": "http.request",
        "label": "Download CSV",
        "config": {
          "method": "GET",
          "url": "https://example.com/exports/customers.csv",
          "timeout_seconds": 60
        },
        "connection_id": null,
        "worker_type": "generic-worker",
        "position": {
          "x": 380,
          "y": 220
        }
      },
      {
        "id": "parse_csv",
        "type": "files.parse_csv",
        "label": "Parse CSV",
        "config": {
          "file_content": "{{ steps.fetch_csv.body }}",
          "has_header": true
        },
        "connection_id": null,
        "worker_type": "generic-worker",
        "position": {
          "x": 700,
          "y": 220
        }
      },
      {
        "id": "process_rows",
        "type": "data.process",
        "label": "Normalise columns",
        "config": {
          "operations": [
            {
              "id": "op_email",
              "kind": "map",
              "sources": [
                "email"
              ],
              "ops": [
                {
                  "name": "Trim"
                },
                {
                  "name": "Lowercase"
                }
              ],
              "out": "email"
            },
            {
              "id": "op_name",
              "kind": "map",
              "sources": [
                "full_name"
              ],
              "ops": [
                {
                  "name": "Trim"
                },
                {
                  "name": "Title Case"
                }
              ],
              "out": "fullName"
            },
            {
              "id": "op_src",
              "kind": "literal",
              "value": "nightly-csv",
              "out": "source"
            }
          ],
          "rules": [
            {
              "id": "r_trim",
              "type": "trim",
              "field": "email",
              "severity": "autofix",
              "on": true
            },
            {
              "id": "r_email",
              "type": "validate_email",
              "field": "email",
              "severity": "error",
              "on": true
            }
          ]
        },
        "connection_id": null,
        "worker_type": "generic-worker",
        "position": {
          "x": 1020,
          "y": 220
        }
      },
      {
        "id": "upsert",
        "type": "db.upsert",
        "label": "Upsert customers",
        "config": {
          "table": "customers",
          "keys": [
            "email"
          ],
          "row": {
            "email": "steps.process_rows.email",
            "full_name": "steps.process_rows.fullName",
            "source": "steps.process_rows.source"
          }
        },
        "connection_id": null,
        "worker_type": "generic-worker",
        "position": {
          "x": 1340,
          "y": 220
        }
      }
    ],
    "edges": [
      {
        "from": "schedule",
        "to": "fetch_csv"
      },
      {
        "from": "fetch_csv",
        "to": "parse_csv"
      },
      {
        "from": "parse_csv",
        "to": "process_rows"
      },
      {
        "from": "process_rows",
        "to": "upsert"
      }
    ]
  }
}

Use it, then make it yours.

Templates install as drafts. Change anything, and Workflow Health re-checks the graph on every save.

Start free Book a demo