Data integration

Datasets and pipelines

Versioned datasets and no-code pipelines.

Datasets

A dataset is the unit of stored data in Converge. Rows are stored in a database and files in object storage.

  • Branches let you change data without affecting the main version.
  • Transactions group changes. A transaction is opened, receives rows or files, and is then committed or aborted. Transaction types are APPEND, UPDATE, DELETE and SNAPSHOT.
  • Schemas are versioned per branch.
MethodPathDoes
GET, POST/api/v2/datasetsList or create datasets
POST/api/v2/datasets/{rid}/branches/{branch}/transactionsOpen a transaction
POST…/transactions/{tx}/commitCommit it
POST…/transactions/{tx}/abortAbort it
GET/api/v2/datasets/{rid}/branches/{branch}/readTableRead rows

File uploads are size-limited by the deployment.

Dataset SQL

SQL Query runs a single read-only SELECT or WITH statement over datasets. Queries run with a read-only database role and a timeout.

Pipelines

A pipeline is a no-code graph that reads datasets, transforms them and writes the results to new datasets.

  • Nodes are sources, transforms and sinks.
  • Transforms include SQL, filter, map, join, aggregate and LLM steps.
  • Modes: snapshot rebuilds the output; incremental processes only new rows past a watermark.
  • Schedules: a pipeline can be activated on a cron expression or interval, and paused.
  • Checks: health checks run on the output datasets after each build.

Pipelines can be validated and tested before they run, and every run keeps its jobs and logs. Builds show in the Builds application, and Data Lineage shows how datasets depend on each other.

Converge