Ethos Get¶
Description¶
Ethos Get is a source node that retrieves records from an Ethos Integration API endpoint, handles Ethos authentication, pages through the response, flattens the returned records, and loads the results into derived tenant tables.
Use Ethos Get when a pipeline needs to pull structured records from Ethos and make the result available to downstream SQL-centered compose nodes.
Field reference¶
| Field | Description |
|---|---|
| API account | Required. API account containing the Ethos API key. The node uses the configured API key to request an Ethos bearer token before calling the endpoint. |
| Base URL | Optional. Ethos Integration API host. Defaults to https://integrate.elluciancloud.com. Enter only the scheme and host; do not include the endpoint path, credentials, query string, or fragment. |
| Endpoint path | Required. Path to the Ethos resource, such as /api/example-resource/. Enter the path only, not a full URL. If the path does not start with /, SchemAlign adds it at runtime. This field supports expressions. |
| Version / Accept header | Optional. Value sent as the HTTP Accept header. Defaults to application/json. Use the media type required by the selected Ethos resource version. |
| HTTP timeout seconds | Optional. Number of seconds to wait for each Ethos HTTP request before timing out. Defaults to 60. |
| Page size / limit | Optional. Number of records requested per Ethos page. Defaults to 1000. Runtime clamps the value between 1 and 10000. This is not a total row cap. |
| Max records / preview cap | Optional. Hard cap for preview or test runs. 0 means fetch all matching records. When set, Ethos Get stops after this many records have been fetched. |
| Starting offset | Optional. Offset where paging should begin. Defaults to 0. |
| Page workers | Optional. Number of parallel page fetch workers. Defaults to 8. Runtime clamps the value between 1 and 32. |
| Parameters JSON | Optional. JSON object containing additional Ethos query parameters. Object and list values are serialized to compact JSON and URL-encoded by SchemAlign. Do not include limit or offset; use the paging fields instead. This field supports expressions. |
| If target exists | Optional. Current behavior is replace only. Ethos Get loads through an atomic replace pattern for the derived parent and child tables. |
| Create table if missing | Optional. Controls whether derived tables may be created when they do not already exist. Defaults to enabled. |
| Allow global account | Optional. Allows a global API account to be selected when the project permits global accounts. Defaults to disabled. |
| Allow private network URLs | Optional. Allows localhost, .local, private, loopback, link-local, reserved, or multicast hosts. Defaults to disabled. Keep this off unless the pipeline intentionally calls an internal endpoint. |
Allow insecure http:// |
Optional. Allows the Base URL to use http:// instead of https://. Defaults to disabled. |
| Response root | Optional. JSON property that contains the record array when the response body wraps records inside a named property. Leave blank when the response body itself is the record array. |
| Parent id field | Optional. Field used as the parent row identifier when creating parent and child tables. Defaults to id. |
| Include child index | Optional. Adds a child index column to derived child tables. Defaults to disabled. |
| Debug mode | Optional. Emits additional runtime diagnostics for troubleshooting. Defaults to disabled. |
| Max retries | Optional. Number of retry attempts for failed Ethos page requests. Defaults to 3. Runtime clamps the value between 1 and 20. |
| Skip failed pages | Optional. Allows the run to continue when a page fails after retries. Defaults to enabled. If the first page fails before any records are fetched, the node fails loudly instead of creating empty tables. |
| Bisect failed pages | Optional. When enabled, the node can split failed page ranges to isolate problematic records. Defaults to enabled. |
| Consumer queue timeout | Optional. Number of seconds the consumer waits while coordinating page worker output. Defaults to 1.0 and is runtime-clamped between 0.2 and 10.0. |
| Target schema | Read-only. Ethos Get writes only to the tenant data schema selected by the executor for the current pipeline run. The node does not allow a manual schema override. |
| Derived tables | Read-only. SchemAlign derives table names from the endpoint path slug. The main table uses <slug>_parent; nested record collections use <slug>_child_*. |
When to use Ethos Get¶
Use this node when the pipeline should:
- call an Ethos Integration API endpoint using an approved API account
- retrieve one or more pages of structured records
- flatten the response into deterministic parent and child tables
- publish table handles for downstream SQL Transform nodes
- optionally run with a preview cap before a full data pull
Ethos Get is a source node. It normally appears at the beginning of a pipeline branch and is commonly followed by SQL Transform.
Common pipeline patterns¶
| Pattern | Use case |
|---|---|
Ethos Get -> SQL Transform |
Retrieve records, load derived tables, then shape or filter the result with SQL. |
Ethos Get -> SQL Transform -> File Put |
Retrieve records, transform them, then deliver a file output. |
Ethos Get -> SQL Transform -> HTTP Response |
Retrieve and transform records as part of a request-driven workflow. |
Expressions and parameters¶
Ethos Get supports expressions in these fields:
| Field | Expression result type | Example |
|---|---|---|
| Endpoint path | String | /api/{{resource_name}}/ |
| Parameters JSON | JSON object | { "criteria": { "status": "{{record_status}}" } } |
Use Parameters JSON for all request filters and additional query parameters. SchemAlign manages limit and offset from the dedicated paging fields, so those names are rejected inside Parameters JSON.
When Parameters JSON contains an object or list value, SchemAlign serializes that value to compact JSON and URL-encodes it before sending the request.
Authentication¶
Ethos Get requires an API account. The selected account must be an API-type account with an API key configured.
At runtime, SchemAlign:
- resolves the selected API account
- requests an Ethos bearer token from the Base URL
/authendpoint - calls the configured Endpoint path with
Authorization: Bearer <token> - refreshes the token when the API returns an authorization failure during paging
Do not place API keys, bearer tokens, usernames, or passwords in the Base URL, Endpoint path, or Parameters JSON fields.
Output and table loading¶
Ethos Get writes the raw response records to a run artifact and loads flattened tables into the tenant data schema for the current run.
The node derives table names from the endpoint path:
| Output | Description |
|---|---|
<slug>_parent |
Main table containing parent record fields. |
<slug>_child_* |
Child tables created from nested record collections. |
| Tables manifest | JSON manifest describing stored tables, columns, row counts, parent/child roles, and tenant guardrails. |
| Stored table handles | Runtime table references published for downstream SQL Transform nodes. |
The load behavior uses stream copy plus atomic table replacement. If no records are returned, the node records a no_data status and skips table creation instead of creating empty placeholder tables.
Validation and common errors¶
| Message or condition | What it means | How to fix it |
|---|---|---|
ethos_get requires account_id |
No API account was selected or resolved. | Select an API account with an Ethos API key. |
ethos_get requires endpoint_path |
Endpoint path is blank. | Enter the Ethos resource path. |
endpoint_path must be a path |
A full URL was entered in Endpoint path. | Put only the path in Endpoint path and keep the scheme/host in Base URL. |
base_url must start with https:// or http:// |
Base URL uses an unsupported scheme. | Use an HTTP or HTTPS base URL. |
base_url uses http:// but allow_insecure_http is not enabled |
Base URL uses insecure HTTP while the safety override is off. | Prefer HTTPS, or enable Allow insecure http:// only when intentional. |
base_url is missing a host |
Base URL does not contain a hostname. | Enter a complete base URL such as https://integrate.elluciancloud.com. |
base_url must not include credentials |
Credentials were embedded in the Base URL. | Store credentials in an API account instead. |
base_url must not include a path |
Base URL includes endpoint path content. | Move the path into Endpoint path. |
base_url points to a local/private host |
The request targets a private or local host while private network access is disabled. | Use a public host or enable Allow private network URLs only when internal access is intentional. |
Parameters JSON may not contain limit or offset |
Paging fields were included in Parameters JSON. | Remove limit and offset from Parameters JSON and use the paging fields. |
query_params_json must be valid JSON |
Parameters JSON could not be parsed. | Enter a valid JSON object. |
query_params_json must be a JSON object |
Parameters JSON is not an object. | Use an object such as { "criteria": { "status": "active" } }. |
ethos_get currently supports only if_exists=replace |
A target behavior other than replace was supplied. | Use the default replace behavior. |
Target table missing and create_table_if_missing=false |
A derived table does not exist and table creation is disabled. | Enable Create table if missing or create the expected table first. |
Ethos Get failed before fetching any records |
The node could not fetch records before table loading began. | Review the failed page log path and endpoint configuration, then rerun. |
Security notes¶
- Use API accounts for credentials; do not paste secrets into request fields.
- Keep Allow private network URLs and Allow insecure http:// disabled unless they are intentionally required.
- Use Max records / preview cap when testing a new endpoint or filter.
- Treat Parameters JSON and expression results as runtime configuration that may appear in run metadata depending on logging and redaction rules.
- Ethos Get writes only to the tenant data schema selected by the executor for the run.