███████╗███╗ ███╗ █████╗ ████████╗██╗██╗ ██╗ ██╔════╝████╗ ████║██╔══██╗╚══██╔══╝██║╚██╗██╔╝ █████╗ ██╔████╔██║███████║ ██║ ██║ ╚███╔╝ ██╔══╝ ██║╚██╔╝██║██╔══██║ ██║ ██║ ██╔██╗ ███████╗██║ ╚═╝ ██║██║ ██║ ██║ ██║██╔╝ ██╗ ╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝ ╚═╝╚═╝ ╚═╝
DECLARATIVE PYTHON DATA PIPELINES.
Rust + Apache Arrow under the hood. Move data between databases, files, and streams with one decorator. Cron schedules, DAG dependencies, watermarks, schema evolution, restart-safe state, at-least-once delivery — built in. No extra scheduler service to deploy.
User Guide
Install, connections, pipelines, modes, scheduling, streaming, stream processing, CLI. Each chapter is a copy-paste-runnable example.
Specs & Benchmarks
Why ematix-flow exists, what's shipped, TPC-H numbers (1.75× DuckDB, 2.77× Polars, 13.4× PySpark geomean), and how it stacks up against the field.
Pipelines view ships with
flow web — one card per pipeline, last 10
executions as a clickable strip, with a "Next: <UTC>"
forecast for batch and an amber LIVE STREAMING
pill for streaming. Click any square to drill into that run's task DAG.
from ematix_flow import ematix, ManagedTable, Annotated, BigInt, Text, TimestampTZ, pk
# Where rows are READ FROM (operational app database).
@ematix.connection
class app_db:
kind = "postgres"
url = "${APP_DB_URL}"
# Where rows are WRITTEN TO (analytics warehouse).
@ematix.connection
class warehouse:
kind = "postgres"
url = "${WAREHOUSE_URL}"
# Target table — typed Python; framework migrates on first run.
class Events(ManagedTable):
__schema__ = "analytics"
__tablename__ = "events"
event_id: Annotated[BigInt, pk()]
name: Text | None
received_at: TimestampTZ
# Pipeline: SELECT runs against app_db; rows land in warehouse.analytics.events.
@ematix.pipeline(
source_connection="app_db",
target=Events,
target_connection="warehouse",
schedule="*/5 * * * *",
mode="append",
)
def ingest_events(conn): # conn is the source (app_db)
return "SELECT event_id, name, received_at FROM public.events" ematix-flow. Surfaces are stabilizing; minor APIs may still shift before beta.