HTTP Get¶

Description¶

HTTP Get is a source node that calls an HTTP or HTTPS endpoint and makes the response available to downstream nodes.

Use HTTP Get when a pipeline needs to retrieve data from a REST API, exported report URL, generated file endpoint, or another web-accessible resource before loading, transforming, or delivering the data.

Field reference¶

Field	Description
URL	Required. The HTTP or HTTPS endpoint to call. The value must start with `http://` or `https://` and include a host. Do not embed usernames or passwords in the URL; select an Account instead. This field supports expressions.
Timeout seconds	Optional. Number of seconds to wait for the HTTP request before timing out. Defaults to `60` when left blank.
Account	Optional. Account UUID used for authentication. Leave blank for anonymous requests. HTTP Get supports Basic Auth-style accounts and API token accounts.
Headers	Optional. HTTP headers to send with the request, provided as a JSON object. This field supports expressions.
Query params	Optional. Query parameters to append to the request, provided as a JSON object. This field supports expressions.
Output filename	Optional. Filename to use for the stream packet and optional artifact copy. When blank, SchemAlign derives a filename from the response or URL. This field supports expressions.
Format	Optional. How SchemAlign should classify the response for metadata and downstream consumers. Supported values are `auto`, `csv`, `json`, `xml`, and `txt`. Defaults to `auto`.
Also publish artifact copy	Optional. When enabled, SchemAlign keeps a durable artifact copy in addition to the normal stream handoff. Defaults to off.
Keep artifact history	Optional. When artifact publishing is enabled, controls whether prior artifact versions are retained. Defaults to on.
Max bytes	Optional. Maximum number of bytes the node may download. The value must be a positive integer and is capped by the server-level HTTP Get maximum.
Allow private network URLs	Optional. Allows private, loopback, local, link-local, reserved, or multicast targets. Defaults to off. Keep this disabled unless the pipeline intentionally calls an internal endpoint.

When to use HTTP Get¶

Use this node when the pipeline should:

fetch data from an external API endpoint
download a report, CSV, JSON, XML, text file, or similar response
pass the downloaded response directly to a downstream node
optionally keep a durable artifact copy of the response for storage or run history

HTTP Get is a source node. It normally appears at the beginning of a pipeline branch, or after an upstream node that provides single-row placeholder values for the request.

Common pipeline patterns¶

Pattern	Use case
`HTTP Get -> SQL Loader -> SQL Transform`	Retrieve a remote file or API response, load it into SQL-accessible working data, then transform it.
`SQL Transform -> HTTP Get -> SQL Loader`	Use a single-row SQL result to fill request placeholders, then retrieve the matching remote resource.
`HTTP Get -> File Put`	Download a remote response and write it to a configured file destination.
`HTTP Get -> SQL Transform -> File Put`	Retrieve, shape, and deliver a transformed output file.

Expressions and placeholders¶

HTTP Get supports expressions in these fields:

Field	Expression result type	Example
URL	String	`https://api.example.com/report?period={{report_period}}`
Headers	Object	`{ "X-Report-Period": "{{report_period}}" }`
Query params	Object	`{ "period": "{{report_period}}" }`
Output filename	String	`orders_{{report_period}}.csv`

HTTP Get also supports single-row upstream placeholders using {column_name} syntax when it is connected to exactly one upstream producer. For example, if an upstream SQL Transform returns one row with a report_period column, the HTTP Get URL can include:

https://api.example.com/report?period={report_period}

The upstream placeholder model is intentionally narrow:

the HTTP Get node must have exactly one upstream producer for placeholder binding
CSV, JSON, NDJSON, or text payloads must resolve to a single scalar record
unresolved placeholders remain unchanged
generic artifact-backed URL binding is no longer supported

For new work, prefer named runtime values such as {{report_period}} when the value is reusable or should be understandable from the field itself.

Authentication¶

HTTP Get can run anonymously or use an Account.

Account type	Behavior
Basic Auth / HTTP Basic	Sends the configured username and password using HTTP Basic authentication.
API token account	Uses the token or access token as a Bearer token, unless an `Authorization` header is already supplied.

Do not include usernames or passwords directly in the URL. The node rejects URLs that contain embedded credentials.

Response handling¶

HTTP Get streams the response into the pipeline so connected downstream nodes can consume it directly. This stream handoff is the normal output path.

When Also publish artifact copy is enabled, the node also writes a durable artifact copy. This is useful when the response should be retained for storage, auditing, or later review.

The node records output metadata including:

resolved URL
HTTP status code
output mode: stream or stream+artifact
inferred or configured format
filename
MIME type
max byte limit used
whether private network URLs were allowed
stream availability
optional artifact metadata

Format inference¶

When Format is set to auto, SchemAlign infers the response format from the response content type or URL path.

Signal	Inferred format
Content type contains `json`	`json`
Content type contains `xml`	`xml`
Content type contains `csv`	`csv`
Content type is `text/plain`	`txt`
URL path ends in `.json`	`json`
URL path ends in `.xml`	`xml`
URL path ends in `.csv` or `.tsv`	`csv`
URL path ends in `.txt`	`txt`
No matching signal	`csv`

Validation and common errors¶

Message or condition	What it means	How to fix it
`http_get requires config.url`	The URL field is empty.	Enter a valid `http://` or `https://` URL.
`http_get url must start with http:// or https://`	The URL uses an unsupported scheme.	Use an HTTP or HTTPS endpoint.
`http_get url must include a host`	The URL does not include a valid hostname.	Enter a complete URL, such as `https://api.example.com/report`.
`http_get url must not include username/password`	Credentials were embedded in the URL.	Move credentials into an Account and select that Account in the node.
`http_get url cannot target localhost/.local unless allow_private_network=true`	The node is trying to call a local/private endpoint while private network access is disabled.	Use a public endpoint or enable Allow private network URLs only when internal access is intentional.
`http_get config.max_bytes must be a positive integer`	The max byte limit is invalid.	Enter a positive integer or leave the field blank.
`http_get response exceeded max_bytes limit`	The downloaded response is larger than the configured or server-capped limit.	Increase the node limit if appropriate, reduce the response size, or use a more targeted endpoint.
`Unsupported account.type for http_get`	The selected Account type is not supported by this node.	Use a Basic Auth-style account, API token account, or run the request anonymously.
`http_get no longer supports artifact-backed URL binding`	Legacy artifact URL binding fields are present.	Replace artifact URL binding with named runtime values, field expressions, or a connected upstream placeholder source.

Security notes¶

Keep Allow private network URLs disabled unless the pipeline intentionally calls an internal service.
Store credentials in Accounts rather than URLs, headers pasted into many nodes, or pipeline notes.
Use the smallest practical Max bytes limit for endpoints with unpredictable response sizes.
Treat headers, query parameters, filenames, and expression results as runtime configuration that may appear in run metadata depending on logging and redaction rules.