Skip to content

Pipelines and Nodes

A pipeline is a workflow made of connected nodes.

Node

A node performs one step in a pipeline. Nodes are the building blocks users place on the Workspace canvas to read, prepare, route, write, store, or send data.

SchemAlign uses three primary node categories:

  • Source nodes read from an API, database, file, service, or other upstream system.
  • Compose nodes prepare, shape, validate, branch, combine, or transform data. SQL-centered transformation logic belongs inside compose patterns.
  • Destination nodes write, store, send, or publish the result.

Examples:

  • HTTP GET
  • HTTP POST
  • SQL Transform
  • Oracle Select
  • SFTP Write
  • Output Artifact
  • Gate

Terminology

SchemAlign uses node as the product term for these pipeline building blocks. Some integration tools, older notes, or technical conversations may use the word connector for the same idea. In SchemAlign documentation and user-facing screens, treat connectors as nodes and use node when naming, building, configuring, or supporting pipeline steps.

Edge

An edge connects one node to another.

HTTP GET -> SQL Transform -> Output Artifact

Edges are important because they define data flow and dependency order. SchemAlign uses directed graph edges to decide which nodes are ready to run.

Canvas position should not control execution

The visual position of a node on the canvas should not determine execution order. Only directed graph edges should control dependency and flow.

Run

A run is one execution of a pipeline.

During a run, SchemAlign tracks:

  • pending nodes
  • running nodes
  • succeeded nodes
  • failed nodes
  • skipped or blocked nodes
  • artifacts
  • published values
  • error messages

Asynchronous graph execution

SchemAlign executes pipeline work asynchronously at two levels:

  1. Across pipeline runs — multiple pipeline runs can be queued and processed at the same time, subject to environment capacity and operational limits.
  2. Within a pipeline run — nodes execute according to graph dependencies. A node becomes ready when its required upstream nodes have completed successfully, and independent branches can continue without waiting on unrelated branches.
HTTP A -> Compose A -> Destination A
HTTP B -> Compose B -> Destination B
HTTP C -> Compose C -> Destination C

In this example, branches A, B, and C can progress independently once their own upstream dependencies are satisfied. A slow branch should not block unrelated branches from running, completing, or reporting status.