Skippr Cloud

Appearance

ELT ​

Managed ELT is a pipeline registry in Skippr Cloud. You declare the pipeline, start a run, and inspect status. Preview records the run; managed skipprd execution is coming soon.

If you want to run extract and load on your machine or VPC, use the self-hosted skipprd engine (skipprd discover, skipprd schema, skipprd sync). Data Engineer workflows use sde. CLI commands for this service are on CLI and SDKs.

Status: Preview. You can create a pipeline, start a run, and inspect status. A started run stays RUNNING until the engine reports a result. See the API action reference for the full command list.

Before you start ​

  • Sign in with skippr-cloud login. See Cloud User Directory.
  • Ensure your policy allows the elt actions you need.

Concepts ​

  • A pipeline is a named source-to-destination registry entry, including intended loads into the Datalake.
  • A run is one recorded execution with lifecycle status. Preview records the run; the engine has not started yet.
  • Compacted files live in objects. Table metadata lives in tables.

Create a pipeline and start a run ​

Creating the pipeline is CLI, Terraform, CDKTF TypeScript, or CDKTF Python. Starting a run is a data-plane call. For provider credentials and the shared cloud provider block, see Terraform and CDKTF.

bash
skippr-cloud elt create-pipeline --name orders-bronze --source postgres://orders --sink iceberg://lake/orders
hcl
resource "cloud_pipeline" "orders_bronze" {
  name   = "orders-bronze"
  source = "postgres://orders"
  sink   = "iceberg://lake/orders"
}
ts
import { CloudProvider, CloudPipeline } from "@skippr/provider-cloud";

const cloud = new CloudProvider(this, "cloud", { region: "eu-central-1" });

new CloudPipeline(this, "orders-bronze", {
  name: "orders-bronze",
  source: "postgres://orders",
  sink: "iceberg://lake/orders",
  provider: cloud,
});
python
from skippr_cdktf import CloudProvider, CloudPipeline

cloud = CloudProvider(self, "cloud", region="eu-central-1")

CloudPipeline(
    self,
    "orders-bronze",
    name="orders-bronze",
    source="postgres://orders",
    sink="iceberg://lake/orders",
    provider=cloud,
)

Then start a run:

bash
skippr-cloud elt start-run --pipeline-name orders-bronze

Use DescribeRun with the returned runId. Preview runs stay RUNNING until the engine reports. StopRun requests a halt. UpdatePipeline changes source/sink in place.

Access ​

  • CLI: skippr-cloud elt <operation>
  • Hostname: elt.{region}.cloud.skippr.io
  • Auth: Bearer JWT or SigV4 service elt