Data Engineering & Infrastructure
Data pipelines, orchestration, quality engineering, streaming, storage, and analytics infrastructure.
Not sure where to start? Take the 2-minute diagnostic to be placed into a learning mode tuned to your level.
Take the quiz →📖 Key Terms (4)
🧠 Concepts (15)
📚 Learning Paths (6)
PySpark Fundamentals: Distributed Data Processing at Scale
Master PySpark DataFrame API, Spark SQL, and distributed processing patterns for building production
Partitioning Strategies: Optimizing Data Layout for Performance
Learn how partitioning strategies affect query performance, storage efficiency, and cost in modern d
Building Resilient Data Pipelines with Spark and dbt
Combine Spark's processing power with dbt's transformation layer to build production-grade, tested,
Data Contracts & Data Lineage: Building Trust in Modern Data Pipelines
Data contracts formalize the agreement between data producers and consumers, while data lineage trac
Data Mesh: Architecture, Governance, and Real-World Adoption
Data mesh applies domain-driven design to data architecture, organizing ownership by business domain
Change Data Capture: Real-Time Sync Patterns and Tools
Change Data Capture (CDC) captures row-level changes in databases and streams them to downstream sys
All Articles
Batch-Dynamic Coreness and Applications with Logarithmic Work via Balanced Pressures
We give a parallel algorithm that maintains a one-sided $(2+\varepsilon)$-approximation of all core numbers under batches of edge insertions and deletions. For fixed $\varepsilon>0$, the expected amor
Mine Locally, Verify Globally: Exact Recovery, Compilation, and Distortion of Mined Temporal Rules
Specification mining extracts temporal rules that describe relationships between events in recorded system traces. Applying these rules in distributed monitoring systems is more difficult because obse
What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes
Despite the proliferation of survival analysis methods, current benchmarks often operate under narrow conditions: they heavily rely on the concordance index (C-index) for evaluation and model selectio
DispersedBFT: Bottleneck-Free Primary-Backup Consensus using Dispersed Replication
The advent of blockchains is fueling the development of novel resilient data management systems (RDSs). Typically, these RDSs are managed using a primary-backup design using a consensus protocol such
MLIR-MRacle: Memory-Model-Aware Metamorphic Fuzzing of Concurrent MLIR
Multi-level compilers built on MLIR lower concurrent programs through many stages towards targets with potentially drastically different memory models, where an optimisation which appears reasonable,
SotSort: a stable quicksort with a tunable buffer
SotSort is a stable comparison sort whose single code path runs with anything from a fixed 16 KB buffer to a full n-element buffer. Its core is a stable quicksort that partitions blocks through the bu
Stealing Exact Training Data from Other Clients during Federated Learning
Malicious adversaries can employ gradient inversion attacks to reverse-engineer private data using the model updates of a round in Federated Learning. While there have been developments in gradient in
Metadata-Driven AI: An Architectural Pattern for Sustainable Data Analysis with LLMs
Large Language Models (LLMs) let users analyse data in natural language, either by giving the data to the model to read directly or by having the model generate code to process it. Both routes send th
GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits
As large language models (LLMs) and LLM-based agents are increasingly deployed, monitoring their behavior becomes important for detecting unsafe or unintended actions. Activation probes offer an effic
Playing the SMR Game with Existential Honesty
State machine replication protocols are usually proven secure in the honest-Byzantine setting, which requires an honest supermajority of validators. Deployed systems offer no such guarantee: validator
Towards Online Adaptive LLM-based Semantic Filtering
Large language models (LLMs) enable semantic filtering over unstructured documents using natural language predicates, but invoking an oracle LLM on every document incurs prohibitive monetary cost and
Harvesting Expert-Parallel Bubbles for Fine-Tuning
GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside se
Mitochondria-Localized Three-Color Fluorescent Sensor for Concurrent Tracking of Mitochondrial H2O2, Hypochlorite and Subcellular Viscosity during Oxidative Stress and Drug-Related Hepatic Damage.
We report a naphthalimide derived mitochondrial targeting fluorescent probe MT for real time simultaneous visualization of mitochondrial hydrogen peroxide (H2O2), hypochlorite (ClO-), and subcellular microviscosity, three key biomarkers closely associated with oxidative stress mediated drug induced...
Energy and carbon reallocation underlie the growth-storage trade-off in Haematococcus lacustris.
Resource allocation between growth and storage represents a fundamental biological trade-off, yet how unicellular photosynthetic organisms reorganize cellular resources during this transition remains incompletely understood. Here, we used the naturally occurring transition from growth-oriented motil...
When Divergence Falls: Generative AI and the Information Environment
Generative artificial intelligence (AI) changes the architecture of information production by routing information producers' analysis through the same foundation models and public information sources, potentially reducing information diversity. In this study, we examine this possibility using financ...
Beyond Banks versus Markets: Financial Structure and Sustainable Debt in Europe
We examine how the financial structure underlying the EU’s Savings and Investments Union is associated with sustainable corporate debt issuance. Using transaction-level data on bonds and syndicated loans over 2019–2023 and Poisson pseudo-maximum likelihood estimation, we distinguish use-of-proceeds...
Delta Lake vs Apache Iceberg vs Apache Hudi: Lakehouse Format Shootout
Head-to-head comparison of the three major lakehouse storage formats: table mutation semantics, time travel, schema evolution, compaction, and ecosystem integration (Spark, Flink, Trino, DuckDB). Benchmark results included.
The development pipeline is a production system
An argument for operating CI/CD pipelines with production discipline: observability, SLOs, capacity planning, and incident response for the build-and-release path. Treats the development pipeline as a first-class system whose downtime directly blocks engineering throughput.
Rune 1.1: adds Python, an Emacs editor, a symbol index and is now free
Release notes for Rune 1.1, a developer tool that now embeds Python support, ships an Emacs-compatible editor mode, adds a symbol index for fast code navigation, and moves to a free pricing model. Useful signal for tooling teams evaluating in-process editing and analysis environments.
CosmosEscape: Taking over Every Database in Azure Cosmos DB
A security research disclosure (CosmosEscape) demonstrating sandbox escape in Azure Cosmos DB: chaining flaws in the JavaScript user-defined function / stored procedure execution environment to move from one database tenant to others.
Show HN: A local merge queue for parallel Claude Code agents
A Show HN for a local merge queue that coordinates parallel Claude Code agents. Multiple AI agents editing the same repository need serialized integration to avoid conflicting writes; a local merge queue stages, merges, and replays agent outputs so parallelism does not corrupt the working tree.
Choose DuckDB rather than SQLite
A comparison argument for choosing DuckDB over SQLite. Where SQLite is a row-oriented OLTP engine optimized for point lookups and transactional writes, DuckDB is a columnar, vectorized OLAP engine designed for analytical queries over large datasets.
SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers
A production-focused guide to running SQLite for low-latency application servers. Covers WAL (write-ahead logging) mode for concurrent readers and a single writer, busy_timeout and checkpoint tuning to keep WAL files bounded, and custom VFS layers that intercept I/O for durability, encryption, or...
Debezium and CDC: Capturing Database Changes at Scale
Change Data Capture with Debezium: connector configuration, schema evolution handling, initial snapshots, and integration with Kafka and Flink. Patterns for reliable replication and exactly-once semantics.
Fellegi & Sunter (1969) - A Theory for Record Linkage
Fellegi and Sunter provide the mathematical foundation for record linkage, establishing a probabilistic framework for determining whether two records refer to the same entity based on agreement patterns of common identifying fields.
Wang & Strong (1996) - Beyond Accuracy: What Data Quality Means to Data Consumers
Wang and Strong develop the first comprehensive data quality framework, identifying 15 dimensions in four categories (intrinsic, contextual, representational, accessibility), changing how organizations think about data quality beyond accuracy.
Dean & Ghemawat (2004) - MapReduce: Simplified Data Processing on Large Clusters
MapReduce abstracts distributed computation into map and reduce phases, enabling petabyte-scale data processing across commodity clusters while hiding parallelization, fault tolerance, and load balancing complexities.
Chang et al (2006) - Bigtable: A Distributed Storage System for Structured Data
Bigtable is a distributed storage system for petabyte-scale structured data across thousands of commodity servers, providing a sparse, distributed, persistent multi-dimensional sorted map that influenced the NoSQL revolution.
Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost
The Dataflow Model provides a unified framework for batch and stream processing by separating What, Where, When, and How — enabling exactly-once processing of unbounded, out-of-order data.
Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms
The Lakehouse architecture combines data lake flexibility with warehouse reliability through a metadata/transaction layer providing ACID transactions, schema enforcement, and performance on cloud object storage.
Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing
Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, persistent log processing that became the industry standard for building real-time data pipelines.
Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.
Apache Iceberg - Open Table Format Specification
Apache Iceberg defines an open table format for petabyte-scale analytic datasets with ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation on cloud object storage.
Stonebraker & Hellerstein (2005) - What Goes Around Comes Around
A historical analysis of data management systems tracing the cycles from hierarchical to relational to NoSQL to NewSQL, arguing that specialization (one size does not fit all) is the enduring lesson.
Apache Iceberg Deep Dive: Table Formats for the Lakehouse Era
How Apache Iceberg enables ACID transactions on data lakes: partitioning, hidden partitioning, time travel, snapshot isolation, and Iceberg REST catalog. Migration strategies from Hive-style tables and Parquet-only storage.
Real-Time Streaming with Apache Kafka: From Pub/Sub to Event-Driven Architecture
Production patterns for Apache Kafka: topic design strategies, consumer group rebalancing, exactly-once semantics, Kafka Connect for source/sink integration, and ksqlDB for stream processing. Real case studies from financial services.
Data Contracts: Schema as API for the Analytics Team
Implementing data contracts with dbt-expectations, Soda, and Great Expectations. Covers schema evolution, contract versioning, producer/consumer ownership patterns, and breaking change detection in production pipelines.
Airflow vs Prefect vs Dagster: Choosing the Right Orchestrator in 2026
Comprehensive comparison of the three leading Python orchestrators: execution model, DAG vs asset paradigm, scaling characteristics, monitoring, and community ecosystem. Decision framework for greenfield and migration scenarios.
Data Quality at Scale: Great Expectations Beyond Unit Tests for Data
Advanced patterns for Great Expectations in production: custom expectations, data docs auto-generation, checkpoint orchestration, and integration with Dagster and Airflow for automated quality gating.
2027 Data Engineering Predictions: AI-Augmented Pipelines, Real-Time Universal Catalogs, and the Death of Batch
Predictions for data engineering in 2027: AI-assisted pipeline generation, universal catalogs with Unity Catalog and Iceberg REST, real-time streaming replacing nightly batches, and the convergence of data and ML platforms.
Data Platform as a Product: UX Patterns for Internal Developer Platforms
Treating the data platform as an internal product: developer experience design, self-service data ingestion, catalog/search UX, pipeline debugging tools, and SLA dashboards. Patterns from leading platform teams.
The Rise of the Analytics Engineer: dbt, SQLMesh, and the Modern Data Stack
The analytics engineering discipline: how dbt and SQLMesh transformed the data workflow, the shift from ETL to ELT, analytics engineering best practices, and the evolving role between data engineering and data science.
Cost Optimization in Data Pipelines: Engineering for Efficiency at Petabyte Scale
Strategies for reducing data pipeline costs: intelligent partitioning, incremental processing, compute auto-scaling, storage tiering (Iceberg maintenance), query optimization, and workload scheduling on spot/preemptible instances.
Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production
Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production Schema registry architectures enable versioned, contract-enforced data serialization across distributed pipelines.
Dagster's Asset-Centric Orchestration Model
Deep dive into Dagster's asset-based orchestration model, software-defined assets, and the shift from DAG-centric to asset-centric pipeline design. Covers partitioning, backfills, and the Dagster UI.
Data Products: Designing APIs for the Internal Data Platform
Data product design patterns: API contracts, SLAs, versioning, discovery, and access control. Implementation with dbt (data products as models), Dagster (software-defined assets), and DataHub for cataloging.
Data Mesh in Practice: Implementing Domain Ownership Without Chaos
Practical guide to data mesh adoption: domain ownership patterns, data product definitions, federated governance, and the compute platform. Case studies of mesh implementations and common failure modes.
Terraform for Data Infrastructure: Infrastructure as Code for the Data Platform
Infrastructure as Code patterns for data platforms: Terraform modules for Kafka clusters, Iceberg catalogs, dbt Cloud projects, and Dagster deployments. State management, CI/CD for infrastructure, and multi-environment strategies.
Kubernetes for Data Engineering: Running Data Pipelines on K8s
Running data workloads on Kubernetes: Airflow Executor types (Celery vs Kubernetes), Dagster on K8s, Spark on Kubernetes with the Spark Operator, and stateful workloads (Kafka, Flink) on K8s. Resource management and cost optimization.
Feature Stores at Scale: Feast vs Tecton in Production Deployments
Deep comparison of Feast (open source) and Tecton (managed): feature definitions, online/offline serving, point-in-time correctness, stream feature computation, and cost models. Production deployment patterns.
ML Pipeline Orchestration: From Notebook to Production with Feast and MLflow
Production ML pipeline patterns: Feast feature serving for training/inference consistency, MLflow model registry and deployment, and Dagster for ML pipeline orchestration. Feature engineering at scale with dbt and Spark.
Data Observability: Monitoring, Lineage, and Incident Response for Pipelines
Implementing data observability with open source tools: OpenLineage for lineage, Great Expectations for quality monitoring, and custom health checks. Incident response runbooks and SLAs for data products.
Building a Data Platform on a Budget: The Open Source Stack in 2026
Complete open source data stack: Dagster + dbt + Iceberg + Trino + DuckDB + Superset. Cost analysis against Snowflake and Databricks. Deployment patterns with Docker Compose, Terraform, and Kubernetes.
SQLMesh: The SQL-First Data Transformation Framework Challenging dbt
SQLMesh's approach to data transformation: physical vs logical plans, virtual data environments, automatic column-level lineage, and backward-incompatible change detection. Comparison with dbt's materialization model.
dbt Mesh: Decentralizing Data Transformation at Enterprise Scale
dbt Mesh architecture: dbt projects as domains, cross-project refs, blue/green deployments for models, and governance through the dbt Cloud Discovery API. Migration guide from monolithic dbt projects.
Data Pipeline Patterns for High-Throughput Genomics: Orchestrating Bioinformatics Workflows with Dagster
Applying modern DataOps orchestration to genomics: Dagster assets for sequencing pipeline stages, Great Expectations for quality gates on base-call accuracy, and dbt for cohort-level analytical transformations.
Reproducible ML Pipelines in Computational Biology: MLOps for CRISPR Target Discovery
Applying DataOps and MLOps principles to computational biology workflows: version-controlled training data, reproducible feature engineering, automated model validation, and pipeline CI/CD for CRISPR target prediction.
EY Canada published a cybersecurity report and most citations were hallucinated -- 🛡️ AML 2026-06-01
Trending (HackerNews, 2026-06-01) EY Canada published a cybersecurity report and most citations were hallucinated (discussion) (320 pts) White House's Aliens.gov Site Brags That ICE Arrested More