Flowfile

Visual ETL that compiles to Polars.
_{Build pipelines on a canvas or in Python — two views of the same graph.
Runs on your laptop, in Docker for a team, or entirely in your browser.}

▶  Try it in your browser  →
_{No install. No signup. Polars in the browser via Pyodide.}

Docs · Releases · Discussions · Architecture deep-dive

---

Build pipelines on a visual canvas with a live preview at every node, or write them in Python with a Polars-like API. Code and visual are two views of the same graph: drag nodes or write Polars-style code, your choice. Beyond the canvas: a Delta-backed catalog with time-travel and virtual tables, a SQL editor with embedded viz, a scheduler, flow parameters, sandboxed Python kernels, and group sharing for teams.

• Just want to see it? demo.flowfile.org.
• Transforming files on your laptop? pip install Flowfile (Python 3.10–3.13).
• Running it for a team? Docker Compose, which adds accounts, groups and a shared catalog. See the Quick Start.

_{Building a Superstore pipeline on the canvas — filters, a join, a pivot, and aggregations, with a live data preview updating at every step.}

 

---

What's in Flowfile

Canvas and code

A visual canvas with 45 node types — joins, fuzzy matching, filters, pivots, aggregations, text-to-rows, window functions. Read from local files, databases (PostgreSQL, MySQL, SQL Server, SQLite, DuckDB), cloud storage (S3, ADLS, GCS), Kafka, Google Analytics, or REST APIs. Write the result wherever you want. Beyond the nodes, the formula editor brings 95 transformation functions, and a Polars code node gives you full Polars for anything the palette doesn't cover — all running in-process, no external engine.

A Python API with Polars-like syntax. Code and visual are two ways to build the same object graph — write a pipeline, call open_graph_in_editor(), and see it visually without re-building anything. Since the syntax mirrors Polars, porting an existing Polars script over is mostly mechanical.

_{A pipeline written with the Python API, opened as an editable flow in the visual editor.}

 

An AI assistant. Tell it what you want and it builds the flow with you, on the canvas. Bring your own key — Anthropic, OpenAI, Google, Groq, OpenRouter — or point it at Ollama or a local model.

_{Describe what you want, get a runnable flow.}

 

Code generation. Prototype visually, ship a plain script: flows export as Python, and save as human-readable YAML so version control works. A flow of standard transforms on local files exports as pure Polars, usually with import polars as pl as its only import.

What the export needs, exactly

Fuzzy matching, graph solving, and formulas that don't translate to a native Polars expression pull in the helper packages Flowfile itself is built on (pl-fuzzy-frame-match, polars-grouper, polars-expr-transformer). They're normal pip installs and don't drag Flowfile along.

Database and REST nodes export as flowfile calls, so their stored connections and secrets resolve at run time. The platform nodes (catalog, cloud storage, Kafka, ML) export against the FlowFrame API rather than raw Polars. And a few nodes (Google Analytics, SQL query, API response) have no code generation yet.

_{The same flow as code: toggle between pure Polars and FlowFrame output.}

 

Other features

Everything a flow produces can land in the data catalog: a catalog > schema > table hierarchy with Delta Lake underneath, so tables get version history and time travel. Flows write into it through a Catalog Writer node — or register their output as a virtual table, with nothing materialised. For those, Flowfile stores the Polars query plan rather than the data (as long as the producing graph is lazy-safe), so a consumer's filters push down straight through the flow boundary, and upstream Delta versions are tracked per read to catch stale data.

There's a SQL editor on top (Polars SQLContext under the hood): query any registered table, chart the result in the embedded Graphic Walker, and if an ad-hoc query turns out to be useful, save it as a flow in one click.

_{SQL queries run against catalog tables, with results feeding into Graphic Walker for visual exploration.}

 

You can put flows on a schedule: on an interval, when a catalog table updates, or once a whole set of tables has refreshed. Run history, logs and cancellation live in the UI, and the scheduler runs embedded, standalone or in Docker. Any node setting takes ${variable} parameters — file paths, SQL queries, formulas — with defaults managed in the Designer and overridden at run time with --param.

With the Docker deployment it all becomes multi-user: accounts, an admin role, user groups. Share a connection, flow or catalog namespace with a group at "use" or "manage" level (secrets can be shared too, read-only); everything else stays private to its owner. The desktop app is single-user.

Python kernels. Run user code in isolated Docker containers with their own package environments, keeping the host process safe. Jupyter-style notebook editor with cell execution, autocompletions, and rich display output (matplotlib, plotly, PIL, HTML).

Custom nodes and a community registry. Build your own nodes in the visual Node Designer — a typed settings form, live preview, and a single-file .py output — then share them through the community registry. Publishing is a pull request opened straight from the app; installing is one click from Catalog → Community Nodes, with sha256-pinned downloads and a capability consent dialog.

An embeddable editor. The browser editor also ships as a standalone Vue component, flowfile-editor, so you can drop a Polars-powered visual ETL canvas into any web app with zero backend: npm install flowfile-editor.

Templates and clipboard import. Get started with built-in flow templates, or paste tabular data from Excel / Google Sheets directly onto the canvas to create a pre-filled input node.

A single flow can filter, join, pivot and aggregate, then branch into as many outputs as you need:

_{Joins, pivots and aggregations feeding a product leaderboard, a monthly trend and a city matrix.}

---

Quick Start

In the browser — demo.flowfile.org runs a 23-node subset on Pyodide; good for a first look and small files.

On your laptop (Python 3.10–3.13):

pip install Flowfile
flowfile run ui

Or start from code:

import flowfile as ff
from flowfile import col, open_graph_in_editor

df = ff.from_dict({
    "id": [1, 2, 3, 4, 5],
    "category": ["A", "B", "A", "C", "B"],
    "value": [100, 200, 150, 300, 250]
})

result = (
    df.filter(col("value") > 150)
      .with_columns((col("value") * 2).alias("double_value"))
      .group_by("category")
      .agg(col("value").sum().alias("total"))
)

open_graph_in_editor(result.flow_graph)

For a team — the Docker stack runs core, worker, and the web UI with user accounts and a shared catalog:

git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile
docker compose up -d   # UI at http://localhost:8080

The compose file in this repo builds from source. To run the published Docker Hub images on a server, behind HTTPS, use flowfile-hosting.

Desktop app — installers for Windows, macOS, and Linux on the Releases page.

From source — for contributors (Python 3.10–3.13, Node.js 20+)

git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile
poetry install

# Backend (two separate terminals)
poetry run flowfile_worker  # :63579
poetry run flowfile_core    # :63578

# Frontend
cd flowfile_frontend
npm install && npm run dev:web  # :8080

Note: Windows installers aren't code-signed yet — SmartScreen will warn; click "More info" → "Run anyway". On macOS, if the app shows as damaged after download: find /Applications/Flowfile.app -exec xattr -c {} \;

---

Where Flowfile fits

Instead of	The difference
Alteryx / KNIME	The same canvas idea, but MIT-licensed, Polars underneath, and every pipeline exports to Python you can take with you.
dbt	dbt transforms data that's already in a warehouse. Flowfile works on files, databases and streams directly, no warehouse needed, and adds a visual layer.
Airflow / Dagster	Orchestrators run pipelines; Flowfile is where the pipeline gets built. It has a small scheduler of its own, and exported scripts run fine under any orchestrator.
Plain Polars	You keep Polars. Flowfile adds a canvas, a preview at every node, a catalog and a scheduler, and gets out of the way again when you export.

What it doesn't do

Flowfile runs on one machine. Polars in-process, no cluster — core and worker can run as separate containers, but they still need a shared filesystem.

It's a transformation tool, not an ingestion platform: files (CSV, Parquet, Excel, JSON, IPC, NDJSON, Avro), five databases (Postgres, MySQL, SQL Server, SQLite, DuckDB), S3/ADLS/GCS, Kafka, Google Analytics and REST APIs. There's no Oracle, Snowflake or BigQuery driver, and no CDC. If your data lives in one of those, land it somewhere Flowfile can reach first.

Smaller caveats: Kafka is read-only (there's a consumer, no producer), Iceberg reads but doesn't write, the Python Script node needs Docker running locally, and accounts/sharing only exist in the Docker deployment — the desktop app is single-user.

---

Architecture

Three services, plus an embedded scheduler and a sandboxed kernel runtime for the Python Script nodes:

• Designer (Tauri + Vue) — visual interface
• Core (FastAPI) — ETL engine running Polars (:63578)
• Worker (FastAPI) — computation and caching (:63579)

Each flow is a directed acyclic graph: nodes are data operations, edges are data flow.

Deeper dive: Architecting a Visual ETL Tool with Polars.

---

Project status

Actively developed, pre-1.0. Releases go out from this repo to PyPI, the desktop installers, Docker Hub, and npm. The test suite is over 7,000 Python tests plus 150+ frontend and WASM test files, spread over seven CI test workflows (unit, Playwright e2e, kernel, Kafka, Docker auth); backend coverage lands on Codecov.

What's next is tracked in Issues and discussed in Discussions; every release has a feedback thread on Releases.

License

MIT

---

Acknowledgments

Built on Polars, Vue.js, FastAPI, VueFlow, Delta Lake, Graphic Walker, and Tauri.