Data Engineering Research

Data Engineering

2027 Data Engineering Predictions: AI-Augmented Pipelines, Real-Time Universal Catalogs, and the Death of Batch

Data Engineering

Data Platform as a Product: UX Patterns for Internal Developer Platforms

Data Engineering

The Rise of the Analytics Engineer: dbt, SQLMesh, and the Modern Data Stack

Data Engineering

Cost Optimization in Data Pipelines: Engineering for Efficiency at Petabyte Scale

Data Engineering

Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production

Data Engineering

Data Products: Designing APIs for the Internal Data Platform

Data Engineering

Data Mesh in Practice: Implementing Domain Ownership Without Chaos

Data Engineering

Terraform for Data Infrastructure: Infrastructure as Code for the Data Platform

Data Engineering

Kubernetes for Data Engineering: Running Data Pipelines on K8s

Data Engineering

Feature Stores at Scale: Feast vs Tecton in Production Deployments

Data Engineering

ML Pipeline Orchestration: From Notebook to Production with Feast and MLflow

Data Engineering

Data Observability: Monitoring, Lineage, and Incident Response for Pipelines

Data Engineering

Building a Data Platform on a Budget: The Open Source Stack in 2026

Data Engineering

SQLMesh: The SQL-First Data Transformation Framework Challenging dbt

Data Engineering

dbt Mesh: Decentralizing Data Transformation at Enterprise Scale

Data Engineering

Delta Lake vs Apache Iceberg vs Apache Hudi: Lakehouse Format Shootout

Data Engineering

Debezium and CDC: Capturing Database Changes at Scale

Data Engineering

Apache Iceberg Deep Dive: Table Formats for the Lakehouse Era

Data Engineering

Real-Time Streaming with Apache Kafka: From Pub/Sub to Event-Driven Architecture

Data Engineering

Data Contracts: Schema as API for the Analytics Team

Data Engineering

Airflow vs Prefect vs Dagster: Choosing the Right Orchestrator in 2026

Data Engineering

Data Quality at Scale: Great Expectations Beyond Unit Tests for Data

Data Engineering

Dagster's Asset-Centric Orchestration Model

Data Engineering

Data Pipeline Patterns for High-Throughput Genomics: Orchestrating Bioinformatics Workflows with Dagster

Data Engineering

Reproducible ML Pipelines in Computational Biology: MLOps for CRISPR Target Discovery

Data Engineering

EY Canada published a cybersecurity report and most citations were hallucinated -- 🛡️ AML 2026-06-01

Data Engineering

Fellegi & Sunter (1969) - A Theory for Record Linkage

Data Engineering

Wang & Strong (1996) - Beyond Accuracy: What Data Quality Means to Data Consumers

Data Engineering

Dean & Ghemawat (2004) - MapReduce: Simplified Data Processing on Large Clusters

Data Engineering

Chang et al (2006) - Bigtable: A Distributed Storage System for Structured Data

Data Engineering

Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost

Data Engineering

Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms

Data Engineering

Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing

Data Engineering

Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Data Engineering

Apache Iceberg - Open Table Format Specification

Data Engineering

Stonebraker & Hellerstein (2005) - What Goes Around Comes Around

Data Engineering

SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

Data Engineering

Rune 1.1: adds Python, an Emacs editor, a symbol index and is now free

Data Engineering

Choose DuckDB rather than SQLite

Data Engineering

CosmosEscape: Taking over Every Database in Azure Cosmos DB

Data Engineering

The development pipeline is a production system

Data Engineering

Show HN: A local merge queue for parallel Claude Code agents

Data Engineering

Nearly-Optimal Time-Sample Trade-off in Consensus Dynamics

Data Engineering

Tessera: Fast Join-Aggregates over Skewed Sparse Relations in Database Engines

Data Engineering

Reinforcement learning for control of partial differential equations

Data Engineering

Deep Learning Based Behavioral Drift Modeling for Continuous Biometric Authentication

Data Engineering

Finding Competitive Equilibria in Online Marketplaces with Platform Fees, Taxes, and Subsidies

Data Engineering

RecHash: Segmented Hash-Entangled Chains for Stateful Sequential Authentication

Data Engineering

Factorized Representations of Concurrent Temporal Variables in Pretrained Language Models

Data Engineering

Zero-freeness for Random Regular Graphs up to the Replica Symmetry Breaking Threshold

Data Engineering

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

Data Engineering

Exact Competitive Ratios of Index Rules for Scheduling with Obligatory Testing

Data Engineering

On Zeros and Algorithms for Disordered Systems II: The Antiferromagnetic Ising Model up to Kesten-Stigum

Data Engineering

Can We Rely on Language Models to Orchestrate Services across the Edge-Cloud Continuum?

Data Engineering

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Data Engineering

When Imputation Changes the Nearest Neighbor: Exact Certification of Similarity Retrieval for Incomplete Sensor Time Series

Data Engineering

Power Adjusting Games Under Block-Dependent Pool Payout Schemes

Data Engineering

AI in the EGI e-infrastructure -- Past experiences and future strategies

Data Engineering

SPECTRUM: A Strategic Framework and Technical Blueprint for European Exascale Research Data and Compute Infrastructure

Data Engineering

True 3D Holography: A Communication Service of Tomorrow and Its Requirements for a New Converged Cloud and Network Architecture on the Path to 6G

Data Engineering

NECA: Neighbor Event Curve Augmentation for Multi-Horizon ICU Deterioration Prediction

Data Engineering

Institutional Asset Allocation meets the Efficient Frontier: Replicating the JPMorgan Efficiente 5 Strategy

Data Engineering

Beyond Banks versus Markets: Financial Structure and Sustainable Debt in Europe

Data Engineering

Monitoring Mitochondrial Function With High-Resolution Respirometry During Ex-Situ Heart Perfusion.

Data Engineering

Crosstalk between mitophagy and metabolism in cancer: Mechanisms and therapeutic opportunities.

Data Engineering

Aperiodic EEG features of cognition: a systematic review.