⚙️

Data Engineering & Infrastructure

Data pipelines, orchestration, quality engineering, streaming, storage, and analytics infrastructure.

Not sure where to start? Take the 2-minute diagnostic to be placed into a learning mode tuned to your level.

Take the quiz →
66 articles · Beginner friendly · 4 key terms

All Articles

Research

AgentSSL: Can MLE Agents Leverage Unlabeled Data?

Applying machine learning systems to new domains introduces bottlenecks in both labeled data acquisition and manual pipeline engineering. While recent MLE agents show promise in automating pipeline de

Nov 12, 2026
Research

Separations Between Submodular-Like Graph Widths

Many algorithmic frameworks exist for evaluating join queries. They range from simple ones, such as dynamic programming over a single tree decomposition, high-degree-low-degree splitting from graph al

Nov 12, 2026
Research

Consistent Video-to-Video Translation via Explicit Correspondences

Interactive video-to-video applications require real-time generation while maintaining long-range temporal consistency. However, recent methods achieve speed by restricting attention to a fixed-length

Nov 12, 2026
Research

Patterns for Designing CI/CD Workflows

CI/CD pipelines are difficult to build and maintain, and information about them is scattered across empirical studies, practitioner books, and catalogues of 'smells'. We collected candidate practices

Nov 12, 2026
Research

Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral

Nov 12, 2026
Research

PACER: GPU Excursion Management for ML Workloads

Variability is an inherent characteristic of modern GPU systems, arising from a combination of process variation, dynamic environmental conditions, and power management decisions. Prior work has chara

Nov 12, 2026
Research

Symmetry in Time: A Representation Theory of Linear Recurrent Memory

State space models (SSMs) are an emerging alternative to Transformers. SSMs offer linear-time inference and parallel training, but most implementations make the parallel scan efficient by making the r

Nov 12, 2026
Research

STeP-C: Real-Time Neural Compression for Distributed Fiber-Optic Sensing at Scale

Distributed fiber-optic sensing (DFOS) technologies such as Distributed Acoustic Sensing (DAS) generate continuous, high-resolution spatio-temporal data at rates exceeding hundreds of megabytes per se

Nov 12, 2026
Research

Broadcast, Don't Replicate: Eliminating Weight HBM in Datacenter LLM Serving

Large language model (LLM) inference at datacenter scale is fundamentally bottlenecked by memory capacity and bandwidth limits. Because model weights are read-only and identical across instances, conv

Nov 12, 2026
Research

Reasoning Supervision and Capability Retention in LoRA Domain Adaptation

Fine-tuning foundation models on financial data has a wide range of applications, including identifying event types from news, monitoring risk and compliance language, and answering questions over fil

Nov 12, 2026
Research

Evaluation of Music Version Retrieval Methods for Setlist Identification in Live Concerts

Automatic setlist identification aims to identify the songs performed in live concert recordings by matching noisy and variable-quality audio against a database of reference tracks. This task presents

Nov 12, 2026
Research

[Experiment, Analysis, and Benchmark] How well do LLMs comprehend and design relational database schemata?

Large Language Models (LLMs) are increasingly used for database tasks, yet most work, such as Text2SQL, assumes that the relational schema already exists. We ask whether LLMs can design a schema from

Nov 12, 2026
Research

An Integrated Architecture for Data-Efficient Real-Time Neural Imitation of Nonlinear Time-Optimal Model Predictive Control

Nonlinear time-optimal model predictive control (TOMPC) enables highly performant control of constrained systems, but its computational cost prohibits real-time use on fast dynamical systems. Neural n

Nov 12, 2026
Research

Channel-Chart-Driven Antenna Activation for Energy-Efficient Cell-Free Massive MIMO

Cell-free massive MIMO (CF-mMIMO) can provide uniformly high system performance across the coverage area through distributed cooperation. However, keeping all antennas active leads to high circuit and

Nov 12, 2026
Research

Conductive Composite-Based Wearable Sensors for Sports Science: Intelligent Data Analysis and Emerging Applications.

Wearable sensors enable continuous acquisition of physiological, biochemical, and biomechanical signals during exercise, providing data to support sports performance assessment, training regulation, fatigue monitoring, and rehabilitation interventions. However, signal reliability and multidimensiona...

Nov 11, 2026
Research

Pre-Strain-Activated Strain-Sensing Ink for Facile Printing of Highly Sensitive and Stable Electronic Textiles.

Printable strain-sensing electronic textiles (e-textiles) are promising for wearable health monitoring and human-machine interfaces, yet achieving simultaneous high sensitivity, mechanical stability, and comfort remains challenging due to unstable conductive networks and weak interfacial integration...

Nov 10, 2026
Research

Enantioselective Surface-Enhanced Raman Scattering Enabled by a Biomimetic Metal-Organic Framework with Catechol Oxidase-Like Activity.

Distinguishing chiral compounds remains a major challenge in pharmaceutical development, environmental monitoring, and food safety, largely because conventional optical detection methods suffer from weak signals and insufficient sensitivity. Surface-enhanced Raman scattering (SERS) has emerged as a...

Nov 10, 2026
Research

Payments Canada announces launch of Canada's Real-Time Rail (RTR) payment system

HackerNews discussion (586 points) about Payments Canada announces launch of Canada's Real-Time Rail (RTR) payment system.

Nov 9, 2026
Research

Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

HackerNews discussion (181 points) about Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation.

Nov 9, 2026
Research

When Divergence Falls: Generative AI and Information Environment

Generative artificial intelligence (AI) changes the architecture of information production by routing information producers' analysis through the same foundation models and public information sources, potentially reducing information diversity. In this study, we examine this possibility using financ...

Nov 9, 2026
Research

A comprehensive review of emerging flexible tactile sensors: mechanisms, nanomaterials, microstructures and applications.

Flexible tactile sensors represent pivotal physical interfaces for human-machine interaction and embodied intelligence. However, the inherent trade-offs between sensitivity and sensing range, multimodal crosstalk, and long-term biosafety remain critical bottlenecks hindering practical implementation...

Nov 9, 2026
Research

GitHub's new Git storage architecture

HackerNews discussion (242 points) about GitHub's new Git storage architecture.

Nov 6, 2026
Research

Association of menopausal symptom burden with performance and self-monitoring across cognitive domains.

Perimenopause is a dynamic neuroendocrine transition with increased risk of cognitive, psychological and physical symptoms. How population variability in symptom burden relates to different aspects of cognitive functioning remains unclear.

Nov 6, 2026
Research

Beyond Banks versus Markets: Financial Structure and Sustainable Debt in Europe

We examine how the financial structure underlying the EU’s Savings and Investments Union is associated with sustainable corporate debt issuance. Using transaction-level data on bonds and syndicated loans over 2019–2023 and Poisson pseudo-maximum likelihood estimation, we distinguish use-of-proceeds...

Oct 25, 2026
Research Advanced

Delta Lake vs Apache Iceberg vs Apache Hudi: Lakehouse Format Shootout

Head-to-head comparison of the three major lakehouse storage formats: table mutation semantics, time travel, schema evolution, compaction, and ecosystem integration (Spark, Flink, Trino, DuckDB). Benchmark results included.

Jun 9, 2026
Research

The development pipeline is a production system

An argument for operating CI/CD pipelines with production discipline: observability, SLOs, capacity planning, and incident response for the build-and-release path. Treats the development pipeline as a first-class system whose downtime directly blocks engineering throughput.

Aug 1, 2026
Research

Rune 1.1: adds Python, an Emacs editor, a symbol index and is now free

Release notes for Rune 1.1, a developer tool that now embeds Python support, ships an Emacs-compatible editor mode, adds a symbol index for fast code navigation, and moves to a free pricing model. Useful signal for tooling teams evaluating in-process editing and analysis environments.

Jul 30, 2026
Research

CosmosEscape: Taking over Every Database in Azure Cosmos DB

A security research disclosure (CosmosEscape) demonstrating sandbox escape in Azure Cosmos DB: chaining flaws in the JavaScript user-defined function / stored procedure execution environment to move from one database tenant to others.

Jul 30, 2026
Research

Show HN: A local merge queue for parallel Claude Code agents

A Show HN for a local merge queue that coordinates parallel Claude Code agents. Multiple AI agents editing the same repository need serialized integration to avoid conflicting writes; a local merge queue stages, merges, and replays agent outputs so parallelism does not corrupt the working tree.

Jul 30, 2026
Research

Choose DuckDB rather than SQLite

A comparison argument for choosing DuckDB over SQLite. Where SQLite is a row-oriented OLTP engine optimized for point lookups and transactional writes, DuckDB is a columnar, vectorized OLAP engine designed for analytical queries over large datasets.

Jul 29, 2026
Research

SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

A production-focused guide to running SQLite for low-latency application servers. Covers WAL (write-ahead logging) mode for concurrent readers and a single writer, busy_timeout and checkpoint tuning to keep WAL files bounded, and custom VFS layers that intercept I/O for durability, encryption, or...

Jul 29, 2026
Research Advanced

Debezium and CDC: Capturing Database Changes at Scale

Change Data Capture with Debezium: connector configuration, schema evolution handling, initial snapshots, and integration with Kafka and Flink. Patterns for reliable replication and exactly-once semantics.

Jun 9, 2026
Research Advanced

Fellegi & Sunter (1969) - A Theory for Record Linkage

Fellegi and Sunter provide the mathematical foundation for record linkage, establishing a probabilistic framework for determining whether two records refer to the same entity based on agreement patterns of common identifying fields.

Jan 1, 1969
Research Intermediate

Wang & Strong (1996) - Beyond Accuracy: What Data Quality Means to Data Consumers

Wang and Strong develop the first comprehensive data quality framework, identifying 15 dimensions in four categories (intrinsic, contextual, representational, accessibility), changing how organizations think about data quality beyond accuracy.

Jan 1, 1996
Research Intermediate

Dean & Ghemawat (2004) - MapReduce: Simplified Data Processing on Large Clusters

MapReduce abstracts distributed computation into map and reduce phases, enabling petabyte-scale data processing across commodity clusters while hiding parallelization, fault tolerance, and load balancing complexities.

Jan 1, 2004
Research Advanced

Chang et al (2006) - Bigtable: A Distributed Storage System for Structured Data

Bigtable is a distributed storage system for petabyte-scale structured data across thousands of commodity servers, providing a sparse, distributed, persistent multi-dimensional sorted map that influenced the NoSQL revolution.

Jan 1, 2006
Research Advanced

Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost

The Dataflow Model provides a unified framework for batch and stream processing by separating What, Where, When, and How — enabling exactly-once processing of unbounded, out-of-order data.

Jan 1, 2015
Research Intermediate

Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms

The Lakehouse architecture combines data lake flexibility with warehouse reliability through a metadata/transaction layer providing ACID transactions, schema enforcement, and performance on cloud object storage.

Jan 1, 2021
Research Advanced

Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing

Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, persistent log processing that became the industry standard for building real-time data pipelines.

Jan 1, 2011
Research Advanced

Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.

Jan 1, 2012
Research Intermediate

Apache Iceberg - Open Table Format Specification

Apache Iceberg defines an open table format for petabyte-scale analytic datasets with ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation on cloud object storage.

Jan 1, 2018
Research Intermediate

Stonebraker & Hellerstein (2005) - What Goes Around Comes Around

A historical analysis of data management systems tracing the cycles from hierarchical to relational to NoSQL to NewSQL, arguing that specialization (one size does not fit all) is the enduring lesson.

Jan 1, 2018
Research Advanced

Apache Iceberg Deep Dive: Table Formats for the Lakehouse Era

How Apache Iceberg enables ACID transactions on data lakes: partitioning, hidden partitioning, time travel, snapshot isolation, and Iceberg REST catalog. Migration strategies from Hive-style tables and Parquet-only storage.

Jun 9, 2026
Research Advanced

Real-Time Streaming with Apache Kafka: From Pub/Sub to Event-Driven Architecture

Production patterns for Apache Kafka: topic design strategies, consumer group rebalancing, exactly-once semantics, Kafka Connect for source/sink integration, and ksqlDB for stream processing. Real case studies from financial services.

Jun 9, 2026
Research Advanced

Data Contracts: Schema as API for the Analytics Team

Implementing data contracts with dbt-expectations, Soda, and Great Expectations. Covers schema evolution, contract versioning, producer/consumer ownership patterns, and breaking change detection in production pipelines.

Jun 9, 2026
Research Advanced

Airflow vs Prefect vs Dagster: Choosing the Right Orchestrator in 2026

Comprehensive comparison of the three leading Python orchestrators: execution model, DAG vs asset paradigm, scaling characteristics, monitoring, and community ecosystem. Decision framework for greenfield and migration scenarios.

Jun 9, 2026
Research Advanced

Data Quality at Scale: Great Expectations Beyond Unit Tests for Data

Advanced patterns for Great Expectations in production: custom expectations, data docs auto-generation, checkpoint orchestration, and integration with Dagster and Airflow for automated quality gating.

Jun 9, 2026
Research Advanced

2027 Data Engineering Predictions: AI-Augmented Pipelines, Real-Time Universal Catalogs, and the Death of Batch

Predictions for data engineering in 2027: AI-assisted pipeline generation, universal catalogs with Unity Catalog and Iceberg REST, real-time streaming replacing nightly batches, and the convergence of data and ML platforms.

Jun 19, 2026
Research Advanced

Data Platform as a Product: UX Patterns for Internal Developer Platforms

Treating the data platform as an internal product: developer experience design, self-service data ingestion, catalog/search UX, pipeline debugging tools, and SLA dashboards. Patterns from leading platform teams.

Jun 18, 2026
Research Advanced

The Rise of the Analytics Engineer: dbt, SQLMesh, and the Modern Data Stack

The analytics engineering discipline: how dbt and SQLMesh transformed the data workflow, the shift from ETL to ELT, analytics engineering best practices, and the evolving role between data engineering and data science.

Jun 17, 2026
Research Advanced

Cost Optimization in Data Pipelines: Engineering for Efficiency at Petabyte Scale

Strategies for reducing data pipeline costs: intelligent partitioning, incremental processing, compute auto-scaling, storage tiering (Iceberg maintenance), query optimization, and workload scheduling on spot/preemptible instances.

Jun 16, 2026
Research Advanced

Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production

Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production Schema registry architectures enable versioned, contract-enforced data serialization across distributed pipelines.

Jun 15, 2026
Research Advanced

Dagster's Asset-Centric Orchestration Model

Deep dive into Dagster's asset-based orchestration model, software-defined assets, and the shift from DAG-centric to asset-centric pipeline design. Covers partitioning, backfills, and the Dagster UI.

Jun 9, 2026
Research Advanced

Data Products: Designing APIs for the Internal Data Platform

Data product design patterns: API contracts, SLAs, versioning, discovery, and access control. Implementation with dbt (data products as models), Dagster (software-defined assets), and DataHub for cataloging.

Jun 14, 2026
Research Advanced

Data Mesh in Practice: Implementing Domain Ownership Without Chaos

Practical guide to data mesh adoption: domain ownership patterns, data product definitions, federated governance, and the compute platform. Case studies of mesh implementations and common failure modes.

Jun 13, 2026
Research Advanced

Terraform for Data Infrastructure: Infrastructure as Code for the Data Platform

Infrastructure as Code patterns for data platforms: Terraform modules for Kafka clusters, Iceberg catalogs, dbt Cloud projects, and Dagster deployments. State management, CI/CD for infrastructure, and multi-environment strategies.

Jun 12, 2026
Research Advanced

Kubernetes for Data Engineering: Running Data Pipelines on K8s

Running data workloads on Kubernetes: Airflow Executor types (Celery vs Kubernetes), Dagster on K8s, Spark on Kubernetes with the Spark Operator, and stateful workloads (Kafka, Flink) on K8s. Resource management and cost optimization.

Jun 11, 2026
Research Advanced

Feature Stores at Scale: Feast vs Tecton in Production Deployments

Deep comparison of Feast (open source) and Tecton (managed): feature definitions, online/offline serving, point-in-time correctness, stream feature computation, and cost models. Production deployment patterns.

Jun 10, 2026
Research Advanced

ML Pipeline Orchestration: From Notebook to Production with Feast and MLflow

Production ML pipeline patterns: Feast feature serving for training/inference consistency, MLflow model registry and deployment, and Dagster for ML pipeline orchestration. Feature engineering at scale with dbt and Spark.

Jun 9, 2026
Research Advanced

Data Observability: Monitoring, Lineage, and Incident Response for Pipelines

Implementing data observability with open source tools: OpenLineage for lineage, Great Expectations for quality monitoring, and custom health checks. Incident response runbooks and SLAs for data products.

Jun 9, 2026
Research Advanced

Building a Data Platform on a Budget: The Open Source Stack in 2026

Complete open source data stack: Dagster + dbt + Iceberg + Trino + DuckDB + Superset. Cost analysis against Snowflake and Databricks. Deployment patterns with Docker Compose, Terraform, and Kubernetes.

Jun 9, 2026
Research Advanced

SQLMesh: The SQL-First Data Transformation Framework Challenging dbt

SQLMesh's approach to data transformation: physical vs logical plans, virtual data environments, automatic column-level lineage, and backward-incompatible change detection. Comparison with dbt's materialization model.

Jun 9, 2026
Research Advanced

dbt Mesh: Decentralizing Data Transformation at Enterprise Scale

dbt Mesh architecture: dbt projects as domains, cross-project refs, blue/green deployments for models, and governance through the dbt Cloud Discovery API. Migration guide from monolithic dbt projects.

Jun 9, 2026
Research Advanced

Data Pipeline Patterns for High-Throughput Genomics: Orchestrating Bioinformatics Workflows with Dagster

Applying modern DataOps orchestration to genomics: Dagster assets for sequencing pipeline stages, Great Expectations for quality gates on base-call accuracy, and dbt for cohort-level analytical transformations.

Jun 8, 2026
Research Advanced

Reproducible ML Pipelines in Computational Biology: MLOps for CRISPR Target Discovery

Applying DataOps and MLOps principles to computational biology workflows: version-controlled training data, reproducible feature engineering, automated model validation, and pipeline CI/CD for CRISPR target prediction.

Jun 8, 2026
Research Advanced

EY Canada published a cybersecurity report and most citations were hallucinated -- 🛡️ AML 2026-06-01

Trending (HackerNews, 2026-06-01) EY Canada published a cybersecurity report and most citations were hallucinated (discussion) (320 pts) White House's Aliens.gov Site Brags That ICE Arrested More

Jun 1, 2026