Open Source Data Engineering Tool Landscape
Key Insights
- Curated reference of open source tools for modern data platforms.
Open Source Data Engineering Tools 2026
Curated reference of production-grade open source tools for each data platform layer.
Orchestration
- Apache Airflow — Mature DAG-based scheduler. Largest ecosystem of operators and integrations. Best for teams that need battle-tested stability.
- Dagster — Asset-centric orchestrator with software-defined assets, explicit lineage, and first-class testability. Growing rapidly in 2026.
- Prefect — Python-native orchestration with automatic retries, caching, and cloud UI. Strong DX for smaller teams.
Data Integration (ELT)
- Airbyte — 600+ connectors, protocol-level schema handling, incremental syncs. Deploy self-hosted or Cloud.
- Meltano — Singer-based integration platform with CI/CD for pipelines. Git-native pipeline management.
- Apache NiFi — Visual data flow designer with real-time routing and transformation. Best for complex topologies.
- Debezium — CDC platform for MySQL, PostgreSQL, MongoDB, etc. Streams changes to Kafka.
Transformation
- dbt — SQL transformation framework with testing, documentation, and package management. Industry standard.
- SQLMesh — SQL transformation with automatic diff-based reconciliation, virtual data environments, and backfill optimization.
Data Quality
- Great Expectations — Python expectation framework with automatic profiling, data docs, and suite management.
- Soda — YAML-defined quality checks with built-in anomaly detection and Slack/API integrations.
- dbt Tests — Built-in uniqueness, not-null, accepted-values, foreign-key tests plus custom generic tests.
Catalog & Governance
- OpenMetadata — Unified metadata platform with data discovery, lineage, glossary, and data quality integration.
- DataHub — LinkedIn's metadata platform. Strong lineage and search. Kubernetes-native.
- Amundsen — Lyft's data discovery platform. Lighter weight, simpler deployment.
Stream Processing
- Apache Kafka — Distributed event store and stream processing. De facto standard for data streaming.
- Apache Flink — True stream processing with exactly-once semantics, event-time processing, and state management.
- RisingWave — Streaming SQL database. Materialized views on streams with PostgreSQL-compatible interface.
Storage & Query
- Apache Iceberg — Open table format with ACID, time travel, partition evolution. The 2026 standard for lakehouse tables.
- Delta Lake — Linux Foundation table format with ACID, schema enforcement, and unified batch/streaming.
- Trino — Distributed SQL query engine for federated queries across data sources. Extremely fast.
- DuckDB — Embedded OLAP database. Ideal for local analytics and embedded use cases.
- MinIO — S3-compatible object storage for on-premise and edge deployments.
ML Platform
- MLflow — Experiment tracking, model registry, deployment. De facto standard for ML lifecycle.
- Feast — Feature store for ML. Consistent feature computation across training and serving.
- BentoML — Model serving framework with Python-native APIs and Kubernetes deployment.
- Kubeflow — MLOps platform on Kubernetes for end-to-end ML workflows.
Observability
- Prometheus + Grafana — Metrics collection and dashboarding. Standard for infrastructure and pipeline monitoring.
- OpenTelemetry — Vendor-neutral observability framework for traces, metrics, and logs.
- Grafana Loki — Log aggregation system. Lightweight, cost-effective, Grafana-native.
Article Metadata
Further Reading
Databricks Blog
Lakehouse, Spark, Delta Lake, Unity Catalog — engineering blog
Apache Kafka
Kafka documentation, KIPs, and ecosystem updates
Apache Flink
Flink documentation and release notes
Apache Iceberg
Iceberg table format — specs, REST catalog, performance
dbt Blog
dbt Labs engineering blog — analytics engineering, Semantic Layer
Dagster Blog
Dagster orchestration — software-defined assets, IO managers
Feynman Concept Cards
Master each concept: read the ELI5, explore analogies, work examples, and teach it back.
Dagster Orchestrator is a advanced techniques concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain Dagster Orchestrator as if teaching a colleague who is new to advanced techniques. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Dagster Orchestrator in a real-world advanced techniques scenario. Walk through your design decisions.
Show solution
A diagram for Dagster Orchestrator should include: 1. The core components of dagster 2. How they interact 3. Expected outcomes or outputs
Workflow Orchestration is a advanced techniques concept in the Data Engineering pillar.
Analogy
Example
Find Gaps
Explain Workflow Orchestration as if teaching a colleague who is new to advanced techniques. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Workflow Orchestration in a real-world advanced techniques scenario. Walk through your design decisions.
Show solution
A diagram for Workflow Orchestration should include: 1. The core components of orchestration 2. How they interact 3. Expected outcomes or outputs