⚙️

Data Engineering & Infrastructure

Data pipelines, orchestration, quality engineering, streaming, storage, and analytics infrastructure.

Not sure where to start? Take the 2-minute diagnostic to be placed into a learning mode tuned to your level.

Take the quiz →
58 articles · Beginner friendly · 4 key terms

All Articles

Research

Batch-Dynamic Coreness and Applications with Logarithmic Work via Balanced Pressures

We give a parallel algorithm that maintains a one-sided $(2+\varepsilon)$-approximation of all core numbers under batches of edge insertions and deletions. For fixed $\varepsilon>0$, the expected amor

Nov 18, 2026
Research

Mine Locally, Verify Globally: Exact Recovery, Compilation, and Distortion of Mined Temporal Rules

Specification mining extracts temporal rules that describe relationships between events in recorded system traces. Applying these rules in distributed monitoring systems is more difficult because obse

Nov 18, 2026
Research

What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes

Despite the proliferation of survival analysis methods, current benchmarks often operate under narrow conditions: they heavily rely on the concordance index (C-index) for evaluation and model selectio

Nov 18, 2026
Research

DispersedBFT: Bottleneck-Free Primary-Backup Consensus using Dispersed Replication

The advent of blockchains is fueling the development of novel resilient data management systems (RDSs). Typically, these RDSs are managed using a primary-backup design using a consensus protocol such

Nov 18, 2026
Research

MLIR-MRacle: Memory-Model-Aware Metamorphic Fuzzing of Concurrent MLIR

Multi-level compilers built on MLIR lower concurrent programs through many stages towards targets with potentially drastically different memory models, where an optimisation which appears reasonable,

Nov 18, 2026
Research

SotSort: a stable quicksort with a tunable buffer

SotSort is a stable comparison sort whose single code path runs with anything from a fixed 16 KB buffer to a full n-element buffer. Its core is a stable quicksort that partitions blocks through the bu

Nov 18, 2026
Research

Stealing Exact Training Data from Other Clients during Federated Learning

Malicious adversaries can employ gradient inversion attacks to reverse-engineer private data using the model updates of a round in Federated Learning. While there have been developments in gradient in

Nov 18, 2026
Research

Metadata-Driven AI: An Architectural Pattern for Sustainable Data Analysis with LLMs

Large Language Models (LLMs) let users analyse data in natural language, either by giving the data to the model to read directly or by having the model generate code to process it. Both routes send th

Nov 18, 2026
Research

GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits

As large language models (LLMs) and LLM-based agents are increasingly deployed, monitoring their behavior becomes important for detecting unsafe or unintended actions. Activation probes offer an effic

Nov 18, 2026
Research

Playing the SMR Game with Existential Honesty

State machine replication protocols are usually proven secure in the honest-Byzantine setting, which requires an honest supermajority of validators. Deployed systems offer no such guarantee: validator

Nov 18, 2026
Research

Towards Online Adaptive LLM-based Semantic Filtering

Large language models (LLMs) enable semantic filtering over unstructured documents using natural language predicates, but invoking an oracle LLM on every document incurs prohibitive monetary cost and

Nov 18, 2026
Research

Harvesting Expert-Parallel Bubbles for Fine-Tuning

GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside se

Nov 18, 2026
Research

Mitochondria-Localized Three-Color Fluorescent Sensor for Concurrent Tracking of Mitochondrial H2O2, Hypochlorite and Subcellular Viscosity during Oxidative Stress and Drug-Related Hepatic Damage.

We report a naphthalimide derived mitochondrial targeting fluorescent probe MT for real time simultaneous visualization of mitochondrial hydrogen peroxide (H2O2), hypochlorite (ClO-), and subcellular microviscosity, three key biomarkers closely associated with oxidative stress mediated drug induced...

Nov 17, 2026
Research

Energy and carbon reallocation underlie the growth-storage trade-off in Haematococcus lacustris.

Resource allocation between growth and storage represents a fundamental biological trade-off, yet how unicellular photosynthetic organisms reorganize cellular resources during this transition remains incompletely understood. Here, we used the naturally occurring transition from growth-oriented motil...

Nov 16, 2026
Research

When Divergence Falls: Generative AI and the Information Environment

Generative artificial intelligence (AI) changes the architecture of information production by routing information producers' analysis through the same foundation models and public information sources, potentially reducing information diversity. In this study, we examine this possibility using financ...

Nov 9, 2026
Research

Beyond Banks versus Markets: Financial Structure and Sustainable Debt in Europe

We examine how the financial structure underlying the EU’s Savings and Investments Union is associated with sustainable corporate debt issuance. Using transaction-level data on bonds and syndicated loans over 2019–2023 and Poisson pseudo-maximum likelihood estimation, we distinguish use-of-proceeds...

Oct 25, 2026
Research Advanced

Delta Lake vs Apache Iceberg vs Apache Hudi: Lakehouse Format Shootout

Head-to-head comparison of the three major lakehouse storage formats: table mutation semantics, time travel, schema evolution, compaction, and ecosystem integration (Spark, Flink, Trino, DuckDB). Benchmark results included.

Jun 9, 2026
Research

The development pipeline is a production system

An argument for operating CI/CD pipelines with production discipline: observability, SLOs, capacity planning, and incident response for the build-and-release path. Treats the development pipeline as a first-class system whose downtime directly blocks engineering throughput.

Aug 1, 2026
Research

Rune 1.1: adds Python, an Emacs editor, a symbol index and is now free

Release notes for Rune 1.1, a developer tool that now embeds Python support, ships an Emacs-compatible editor mode, adds a symbol index for fast code navigation, and moves to a free pricing model. Useful signal for tooling teams evaluating in-process editing and analysis environments.

Jul 30, 2026
Research

CosmosEscape: Taking over Every Database in Azure Cosmos DB

A security research disclosure (CosmosEscape) demonstrating sandbox escape in Azure Cosmos DB: chaining flaws in the JavaScript user-defined function / stored procedure execution environment to move from one database tenant to others.

Jul 30, 2026
Research

Show HN: A local merge queue for parallel Claude Code agents

A Show HN for a local merge queue that coordinates parallel Claude Code agents. Multiple AI agents editing the same repository need serialized integration to avoid conflicting writes; a local merge queue stages, merges, and replays agent outputs so parallelism does not corrupt the working tree.

Jul 30, 2026
Research

Choose DuckDB rather than SQLite

A comparison argument for choosing DuckDB over SQLite. Where SQLite is a row-oriented OLTP engine optimized for point lookups and transactional writes, DuckDB is a columnar, vectorized OLAP engine designed for analytical queries over large datasets.

Jul 29, 2026
Research

SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

A production-focused guide to running SQLite for low-latency application servers. Covers WAL (write-ahead logging) mode for concurrent readers and a single writer, busy_timeout and checkpoint tuning to keep WAL files bounded, and custom VFS layers that intercept I/O for durability, encryption, or...

Jul 29, 2026
Research Advanced

Debezium and CDC: Capturing Database Changes at Scale

Change Data Capture with Debezium: connector configuration, schema evolution handling, initial snapshots, and integration with Kafka and Flink. Patterns for reliable replication and exactly-once semantics.

Jun 9, 2026
Research Advanced

Fellegi & Sunter (1969) - A Theory for Record Linkage

Fellegi and Sunter provide the mathematical foundation for record linkage, establishing a probabilistic framework for determining whether two records refer to the same entity based on agreement patterns of common identifying fields.

Jan 1, 1969
Research Intermediate

Wang & Strong (1996) - Beyond Accuracy: What Data Quality Means to Data Consumers

Wang and Strong develop the first comprehensive data quality framework, identifying 15 dimensions in four categories (intrinsic, contextual, representational, accessibility), changing how organizations think about data quality beyond accuracy.

Jan 1, 1996
Research Intermediate

Dean & Ghemawat (2004) - MapReduce: Simplified Data Processing on Large Clusters

MapReduce abstracts distributed computation into map and reduce phases, enabling petabyte-scale data processing across commodity clusters while hiding parallelization, fault tolerance, and load balancing complexities.

Jan 1, 2004
Research Advanced

Chang et al (2006) - Bigtable: A Distributed Storage System for Structured Data

Bigtable is a distributed storage system for petabyte-scale structured data across thousands of commodity servers, providing a sparse, distributed, persistent multi-dimensional sorted map that influenced the NoSQL revolution.

Jan 1, 2006
Research Advanced

Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost

The Dataflow Model provides a unified framework for batch and stream processing by separating What, Where, When, and How — enabling exactly-once processing of unbounded, out-of-order data.

Jan 1, 2015
Research Intermediate

Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms

The Lakehouse architecture combines data lake flexibility with warehouse reliability through a metadata/transaction layer providing ACID transactions, schema enforcement, and performance on cloud object storage.

Jan 1, 2021
Research Advanced

Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing

Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, persistent log processing that became the industry standard for building real-time data pipelines.

Jan 1, 2011
Research Advanced

Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.

Jan 1, 2012
Research Intermediate

Apache Iceberg - Open Table Format Specification

Apache Iceberg defines an open table format for petabyte-scale analytic datasets with ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation on cloud object storage.

Jan 1, 2018
Research Intermediate

Stonebraker & Hellerstein (2005) - What Goes Around Comes Around

A historical analysis of data management systems tracing the cycles from hierarchical to relational to NoSQL to NewSQL, arguing that specialization (one size does not fit all) is the enduring lesson.

Jan 1, 2018
Research Advanced

Apache Iceberg Deep Dive: Table Formats for the Lakehouse Era

How Apache Iceberg enables ACID transactions on data lakes: partitioning, hidden partitioning, time travel, snapshot isolation, and Iceberg REST catalog. Migration strategies from Hive-style tables and Parquet-only storage.

Jun 9, 2026
Research Advanced

Real-Time Streaming with Apache Kafka: From Pub/Sub to Event-Driven Architecture

Production patterns for Apache Kafka: topic design strategies, consumer group rebalancing, exactly-once semantics, Kafka Connect for source/sink integration, and ksqlDB for stream processing. Real case studies from financial services.

Jun 9, 2026
Research Advanced

Data Contracts: Schema as API for the Analytics Team

Implementing data contracts with dbt-expectations, Soda, and Great Expectations. Covers schema evolution, contract versioning, producer/consumer ownership patterns, and breaking change detection in production pipelines.

Jun 9, 2026
Research Advanced

Airflow vs Prefect vs Dagster: Choosing the Right Orchestrator in 2026

Comprehensive comparison of the three leading Python orchestrators: execution model, DAG vs asset paradigm, scaling characteristics, monitoring, and community ecosystem. Decision framework for greenfield and migration scenarios.

Jun 9, 2026
Research Advanced

Data Quality at Scale: Great Expectations Beyond Unit Tests for Data

Advanced patterns for Great Expectations in production: custom expectations, data docs auto-generation, checkpoint orchestration, and integration with Dagster and Airflow for automated quality gating.

Jun 9, 2026
Research Advanced

2027 Data Engineering Predictions: AI-Augmented Pipelines, Real-Time Universal Catalogs, and the Death of Batch

Predictions for data engineering in 2027: AI-assisted pipeline generation, universal catalogs with Unity Catalog and Iceberg REST, real-time streaming replacing nightly batches, and the convergence of data and ML platforms.

Jun 19, 2026
Research Advanced

Data Platform as a Product: UX Patterns for Internal Developer Platforms

Treating the data platform as an internal product: developer experience design, self-service data ingestion, catalog/search UX, pipeline debugging tools, and SLA dashboards. Patterns from leading platform teams.

Jun 18, 2026
Research Advanced

The Rise of the Analytics Engineer: dbt, SQLMesh, and the Modern Data Stack

The analytics engineering discipline: how dbt and SQLMesh transformed the data workflow, the shift from ETL to ELT, analytics engineering best practices, and the evolving role between data engineering and data science.

Jun 17, 2026
Research Advanced

Cost Optimization in Data Pipelines: Engineering for Efficiency at Petabyte Scale

Strategies for reducing data pipeline costs: intelligent partitioning, incremental processing, compute auto-scaling, storage tiering (Iceberg maintenance), query optimization, and workload scheduling on spot/preemptible instances.

Jun 16, 2026
Research Advanced

Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production

Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production Schema registry architectures enable versioned, contract-enforced data serialization across distributed pipelines.

Jun 15, 2026
Research Advanced

Dagster's Asset-Centric Orchestration Model

Deep dive into Dagster's asset-based orchestration model, software-defined assets, and the shift from DAG-centric to asset-centric pipeline design. Covers partitioning, backfills, and the Dagster UI.

Jun 9, 2026
Research Advanced

Data Products: Designing APIs for the Internal Data Platform

Data product design patterns: API contracts, SLAs, versioning, discovery, and access control. Implementation with dbt (data products as models), Dagster (software-defined assets), and DataHub for cataloging.

Jun 14, 2026
Research Advanced

Data Mesh in Practice: Implementing Domain Ownership Without Chaos

Practical guide to data mesh adoption: domain ownership patterns, data product definitions, federated governance, and the compute platform. Case studies of mesh implementations and common failure modes.

Jun 13, 2026
Research Advanced

Terraform for Data Infrastructure: Infrastructure as Code for the Data Platform

Infrastructure as Code patterns for data platforms: Terraform modules for Kafka clusters, Iceberg catalogs, dbt Cloud projects, and Dagster deployments. State management, CI/CD for infrastructure, and multi-environment strategies.

Jun 12, 2026
Research Advanced

Kubernetes for Data Engineering: Running Data Pipelines on K8s

Running data workloads on Kubernetes: Airflow Executor types (Celery vs Kubernetes), Dagster on K8s, Spark on Kubernetes with the Spark Operator, and stateful workloads (Kafka, Flink) on K8s. Resource management and cost optimization.

Jun 11, 2026
Research Advanced

Feature Stores at Scale: Feast vs Tecton in Production Deployments

Deep comparison of Feast (open source) and Tecton (managed): feature definitions, online/offline serving, point-in-time correctness, stream feature computation, and cost models. Production deployment patterns.

Jun 10, 2026
Research Advanced

ML Pipeline Orchestration: From Notebook to Production with Feast and MLflow

Production ML pipeline patterns: Feast feature serving for training/inference consistency, MLflow model registry and deployment, and Dagster for ML pipeline orchestration. Feature engineering at scale with dbt and Spark.

Jun 9, 2026
Research Advanced

Data Observability: Monitoring, Lineage, and Incident Response for Pipelines

Implementing data observability with open source tools: OpenLineage for lineage, Great Expectations for quality monitoring, and custom health checks. Incident response runbooks and SLAs for data products.

Jun 9, 2026
Research Advanced

Building a Data Platform on a Budget: The Open Source Stack in 2026

Complete open source data stack: Dagster + dbt + Iceberg + Trino + DuckDB + Superset. Cost analysis against Snowflake and Databricks. Deployment patterns with Docker Compose, Terraform, and Kubernetes.

Jun 9, 2026
Research Advanced

SQLMesh: The SQL-First Data Transformation Framework Challenging dbt

SQLMesh's approach to data transformation: physical vs logical plans, virtual data environments, automatic column-level lineage, and backward-incompatible change detection. Comparison with dbt's materialization model.

Jun 9, 2026
Research Advanced

dbt Mesh: Decentralizing Data Transformation at Enterprise Scale

dbt Mesh architecture: dbt projects as domains, cross-project refs, blue/green deployments for models, and governance through the dbt Cloud Discovery API. Migration guide from monolithic dbt projects.

Jun 9, 2026
Research Advanced

Data Pipeline Patterns for High-Throughput Genomics: Orchestrating Bioinformatics Workflows with Dagster

Applying modern DataOps orchestration to genomics: Dagster assets for sequencing pipeline stages, Great Expectations for quality gates on base-call accuracy, and dbt for cohort-level analytical transformations.

Jun 8, 2026
Research Advanced

Reproducible ML Pipelines in Computational Biology: MLOps for CRISPR Target Discovery

Applying DataOps and MLOps principles to computational biology workflows: version-controlled training data, reproducible feature engineering, automated model validation, and pipeline CI/CD for CRISPR target prediction.

Jun 8, 2026
Research Advanced

EY Canada published a cybersecurity report and most citations were hallucinated -- 🛡️ AML 2026-06-01

Trending (HackerNews, 2026-06-01) EY Canada published a cybersecurity report and most citations were hallucinated (discussion) (320 pts) White House's Aliens.gov Site Brags That ICE Arrested More

Jun 1, 2026