Data Engineering Knowledge Base
Wiki-style knowledge structure: foundation → core → advanced → specialized
Foundation (6)
- Batch Processing
- Data Lake
- Data Warehouse
- Extract-Load-Transform
- Extract-Transform-Load
- MapReduce Programming Model MapReduce, introduced by Dean and Ghemawat (2004), is a programming model for processing large datasets in parallel acro…
Core (44)
- AI Conformity Assessment & Auditing
- Batch & Stream Merging
- Change Data Capture Patterns
- DORA — ICT Risk Management
- Data Catalog
- Data Contract Testing
- Data Contracts
- Data Cost Intelligence
- Data Discovery & Cataloging
- Data Governance
- Data Lineage
- Data Mesh
- Data Mesh Governance
- Data Observability
- Data Pipeline Architecture
- Data Platform KPIs
- Data Quality
- Data Quality Frameworks Data quality frameworks, pioneered by Wang and Strong (1996), provide a systematic taxonomy of data quality dimensions i…
- Data Quality Monitoring
- Data Quality SLAs
- Data Security & Access Control
- Data Versioning
- Data Warehouse Design Patterns
- DataOps
- Debiasing Pipeline
- Distributed Systems for Data
- ELT Pipeline Architecture
- EU AI Act — High-Risk Classification
- EU Data Act — Data Interoperability
- Fairness Metrics in ML
- Feature Store
- GDPR Anonymization & Pseudonymization
- Lakehouse Architecture
- Lakehouse Architecture The Lakehouse architecture, formalized by Armbrust et al. (2021), combines the flexibility of data lakes (cheap object s…
- Metric Store
- Model Cards & Documentation Standards
- NIS2 Directive — Cybersecurity Resilience
- Pipeline Cost Optimization
- Query Optimization
- Real-Time Analytics Architecture
- Reverse ETL
- Schema Migration Strategies
- Schema Registry
- Streaming-First Lakehouse An emerging architecture that keeps a single copy of data in a streaming log while exposing it as Iceberg tables for his…
Advanced (16)
- Apache Arrow / Parquet
- Apache Flink
- Apache Iceberg
- Apache Kafka
- Change Data Capture
- Dagster Orchestrator
- Differential Privacy
- Federated Learning
- ML Pipeline Engineering
- Real-Time Analytics
- Record Linkage & Entity Matching Record linkage, formalized by Fellegi and Sunter (1969), is the methodology for identifying records that refer to the sa…
- Stream Processing
- Stream Processing & The Dataflow Model The Dataflow Model, introduced by Akidau et al. (2015), provides a unified framework for processing both batch and strea…
- Synthetic Data Generation for GDPR Compliance
- Workflow Orchestration
- dbt (data build tool)
Specialized (4)
- Acid Transaction Management A concept related to acid-transaction-management
- Enterprise Architecture A concept related to enterprise-architecture
- Infrastructure A concept related to infrastructure
- Ux Design Patterns A concept related to ux-design-patterns