Data Engineering & Infrastructure
Data pipelines, orchestration, quality engineering, streaming, storage, and analytics infrastructure.
Not sure where to start? Take the 2-minute diagnostic to be placed into a learning mode tuned to your level.
Take the quiz →📖 Key Terms (4)
🧠 Concepts (15)
📚 Learning Paths (6)
PySpark Fundamentals: Distributed Data Processing at Scale
Master PySpark DataFrame API, Spark SQL, and distributed processing patterns for building production
Partitioning Strategies: Optimizing Data Layout for Performance
Learn how partitioning strategies affect query performance, storage efficiency, and cost in modern d
Building Resilient Data Pipelines with Spark and dbt
Combine Spark's processing power with dbt's transformation layer to build production-grade, tested,
Data Contracts & Data Lineage: Building Trust in Modern Data Pipelines
Data contracts formalize the agreement between data producers and consumers, while data lineage trac
Data Mesh: Architecture, Governance, and Real-World Adoption
Data mesh applies domain-driven design to data architecture, organizing ownership by business domain
Change Data Capture: Real-Time Sync Patterns and Tools
Change Data Capture (CDC) captures row-level changes in databases and streams them to downstream sys
All Articles
AgentSSL: Can MLE Agents Leverage Unlabeled Data?
Applying machine learning systems to new domains introduces bottlenecks in both labeled data acquisition and manual pipeline engineering. While recent MLE agents show promise in automating pipeline de
Separations Between Submodular-Like Graph Widths
Many algorithmic frameworks exist for evaluating join queries. They range from simple ones, such as dynamic programming over a single tree decomposition, high-degree-low-degree splitting from graph al
Consistent Video-to-Video Translation via Explicit Correspondences
Interactive video-to-video applications require real-time generation while maintaining long-range temporal consistency. However, recent methods achieve speed by restricting attention to a fixed-length
Patterns for Designing CI/CD Workflows
CI/CD pipelines are difficult to build and maintain, and information about them is scattered across empirical studies, practitioner books, and catalogues of 'smells'. We collected candidate practices
Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline
Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral
PACER: GPU Excursion Management for ML Workloads
Variability is an inherent characteristic of modern GPU systems, arising from a combination of process variation, dynamic environmental conditions, and power management decisions. Prior work has chara
Symmetry in Time: A Representation Theory of Linear Recurrent Memory
State space models (SSMs) are an emerging alternative to Transformers. SSMs offer linear-time inference and parallel training, but most implementations make the parallel scan efficient by making the r
STeP-C: Real-Time Neural Compression for Distributed Fiber-Optic Sensing at Scale
Distributed fiber-optic sensing (DFOS) technologies such as Distributed Acoustic Sensing (DAS) generate continuous, high-resolution spatio-temporal data at rates exceeding hundreds of megabytes per se
Broadcast, Don't Replicate: Eliminating Weight HBM in Datacenter LLM Serving
Large language model (LLM) inference at datacenter scale is fundamentally bottlenecked by memory capacity and bandwidth limits. Because model weights are read-only and identical across instances, conv
Reasoning Supervision and Capability Retention in LoRA Domain Adaptation
Fine-tuning foundation models on financial data has a wide range of applications, including identifying event types from news, monitoring risk and compliance language, and answering questions over fil
Evaluation of Music Version Retrieval Methods for Setlist Identification in Live Concerts
Automatic setlist identification aims to identify the songs performed in live concert recordings by matching noisy and variable-quality audio against a database of reference tracks. This task presents
[Experiment, Analysis, and Benchmark] How well do LLMs comprehend and design relational database schemata?
Large Language Models (LLMs) are increasingly used for database tasks, yet most work, such as Text2SQL, assumes that the relational schema already exists. We ask whether LLMs can design a schema from
An Integrated Architecture for Data-Efficient Real-Time Neural Imitation of Nonlinear Time-Optimal Model Predictive Control
Nonlinear time-optimal model predictive control (TOMPC) enables highly performant control of constrained systems, but its computational cost prohibits real-time use on fast dynamical systems. Neural n
Channel-Chart-Driven Antenna Activation for Energy-Efficient Cell-Free Massive MIMO
Cell-free massive MIMO (CF-mMIMO) can provide uniformly high system performance across the coverage area through distributed cooperation. However, keeping all antennas active leads to high circuit and
Conductive Composite-Based Wearable Sensors for Sports Science: Intelligent Data Analysis and Emerging Applications.
Wearable sensors enable continuous acquisition of physiological, biochemical, and biomechanical signals during exercise, providing data to support sports performance assessment, training regulation, fatigue monitoring, and rehabilitation interventions. However, signal reliability and multidimensiona...
Pre-Strain-Activated Strain-Sensing Ink for Facile Printing of Highly Sensitive and Stable Electronic Textiles.
Printable strain-sensing electronic textiles (e-textiles) are promising for wearable health monitoring and human-machine interfaces, yet achieving simultaneous high sensitivity, mechanical stability, and comfort remains challenging due to unstable conductive networks and weak interfacial integration...
Enantioselective Surface-Enhanced Raman Scattering Enabled by a Biomimetic Metal-Organic Framework with Catechol Oxidase-Like Activity.
Distinguishing chiral compounds remains a major challenge in pharmaceutical development, environmental monitoring, and food safety, largely because conventional optical detection methods suffer from weak signals and insufficient sensitivity. Surface-enhanced Raman scattering (SERS) has emerged as a...
Payments Canada announces launch of Canada's Real-Time Rail (RTR) payment system
HackerNews discussion (586 points) about Payments Canada announces launch of Canada's Real-Time Rail (RTR) payment system.
Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation
HackerNews discussion (181 points) about Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation.
When Divergence Falls: Generative AI and Information Environment
Generative artificial intelligence (AI) changes the architecture of information production by routing information producers' analysis through the same foundation models and public information sources, potentially reducing information diversity. In this study, we examine this possibility using financ...
A comprehensive review of emerging flexible tactile sensors: mechanisms, nanomaterials, microstructures and applications.
Flexible tactile sensors represent pivotal physical interfaces for human-machine interaction and embodied intelligence. However, the inherent trade-offs between sensitivity and sensing range, multimodal crosstalk, and long-term biosafety remain critical bottlenecks hindering practical implementation...
GitHub's new Git storage architecture
HackerNews discussion (242 points) about GitHub's new Git storage architecture.
Association of menopausal symptom burden with performance and self-monitoring across cognitive domains.
Perimenopause is a dynamic neuroendocrine transition with increased risk of cognitive, psychological and physical symptoms. How population variability in symptom burden relates to different aspects of cognitive functioning remains unclear.
Beyond Banks versus Markets: Financial Structure and Sustainable Debt in Europe
We examine how the financial structure underlying the EU’s Savings and Investments Union is associated with sustainable corporate debt issuance. Using transaction-level data on bonds and syndicated loans over 2019–2023 and Poisson pseudo-maximum likelihood estimation, we distinguish use-of-proceeds...
Delta Lake vs Apache Iceberg vs Apache Hudi: Lakehouse Format Shootout
Head-to-head comparison of the three major lakehouse storage formats: table mutation semantics, time travel, schema evolution, compaction, and ecosystem integration (Spark, Flink, Trino, DuckDB). Benchmark results included.
The development pipeline is a production system
An argument for operating CI/CD pipelines with production discipline: observability, SLOs, capacity planning, and incident response for the build-and-release path. Treats the development pipeline as a first-class system whose downtime directly blocks engineering throughput.
Rune 1.1: adds Python, an Emacs editor, a symbol index and is now free
Release notes for Rune 1.1, a developer tool that now embeds Python support, ships an Emacs-compatible editor mode, adds a symbol index for fast code navigation, and moves to a free pricing model. Useful signal for tooling teams evaluating in-process editing and analysis environments.
CosmosEscape: Taking over Every Database in Azure Cosmos DB
A security research disclosure (CosmosEscape) demonstrating sandbox escape in Azure Cosmos DB: chaining flaws in the JavaScript user-defined function / stored procedure execution environment to move from one database tenant to others.
Show HN: A local merge queue for parallel Claude Code agents
A Show HN for a local merge queue that coordinates parallel Claude Code agents. Multiple AI agents editing the same repository need serialized integration to avoid conflicting writes; a local merge queue stages, merges, and replays agent outputs so parallelism does not corrupt the working tree.
Choose DuckDB rather than SQLite
A comparison argument for choosing DuckDB over SQLite. Where SQLite is a row-oriented OLTP engine optimized for point lookups and transactional writes, DuckDB is a columnar, vectorized OLAP engine designed for analytical queries over large datasets.
SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers
A production-focused guide to running SQLite for low-latency application servers. Covers WAL (write-ahead logging) mode for concurrent readers and a single writer, busy_timeout and checkpoint tuning to keep WAL files bounded, and custom VFS layers that intercept I/O for durability, encryption, or...
Debezium and CDC: Capturing Database Changes at Scale
Change Data Capture with Debezium: connector configuration, schema evolution handling, initial snapshots, and integration with Kafka and Flink. Patterns for reliable replication and exactly-once semantics.
Fellegi & Sunter (1969) - A Theory for Record Linkage
Fellegi and Sunter provide the mathematical foundation for record linkage, establishing a probabilistic framework for determining whether two records refer to the same entity based on agreement patterns of common identifying fields.
Wang & Strong (1996) - Beyond Accuracy: What Data Quality Means to Data Consumers
Wang and Strong develop the first comprehensive data quality framework, identifying 15 dimensions in four categories (intrinsic, contextual, representational, accessibility), changing how organizations think about data quality beyond accuracy.
Dean & Ghemawat (2004) - MapReduce: Simplified Data Processing on Large Clusters
MapReduce abstracts distributed computation into map and reduce phases, enabling petabyte-scale data processing across commodity clusters while hiding parallelization, fault tolerance, and load balancing complexities.
Chang et al (2006) - Bigtable: A Distributed Storage System for Structured Data
Bigtable is a distributed storage system for petabyte-scale structured data across thousands of commodity servers, providing a sparse, distributed, persistent multi-dimensional sorted map that influenced the NoSQL revolution.
Akidau et al (2015) - The Dataflow Model: Balancing Correctness, Latency, and Cost
The Dataflow Model provides a unified framework for batch and stream processing by separating What, Where, When, and How — enabling exactly-once processing of unbounded, out-of-order data.
Armbrust et al (2021) - Lakehouse: A New Generation of Open Platforms
The Lakehouse architecture combines data lake flexibility with warehouse reliability through a metadata/transaction layer providing ACID transactions, schema enforcement, and performance on cloud object storage.
Kreps, Narkhede & Rao (2011) - Kafka: A Distributed Messaging System for Log Processing
Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, persistent log processing that became the industry standard for building real-time data pipelines.
Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.
Apache Iceberg - Open Table Format Specification
Apache Iceberg defines an open table format for petabyte-scale analytic datasets with ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation on cloud object storage.
Stonebraker & Hellerstein (2005) - What Goes Around Comes Around
A historical analysis of data management systems tracing the cycles from hierarchical to relational to NoSQL to NewSQL, arguing that specialization (one size does not fit all) is the enduring lesson.
Apache Iceberg Deep Dive: Table Formats for the Lakehouse Era
How Apache Iceberg enables ACID transactions on data lakes: partitioning, hidden partitioning, time travel, snapshot isolation, and Iceberg REST catalog. Migration strategies from Hive-style tables and Parquet-only storage.
Real-Time Streaming with Apache Kafka: From Pub/Sub to Event-Driven Architecture
Production patterns for Apache Kafka: topic design strategies, consumer group rebalancing, exactly-once semantics, Kafka Connect for source/sink integration, and ksqlDB for stream processing. Real case studies from financial services.
Data Contracts: Schema as API for the Analytics Team
Implementing data contracts with dbt-expectations, Soda, and Great Expectations. Covers schema evolution, contract versioning, producer/consumer ownership patterns, and breaking change detection in production pipelines.
Airflow vs Prefect vs Dagster: Choosing the Right Orchestrator in 2026
Comprehensive comparison of the three leading Python orchestrators: execution model, DAG vs asset paradigm, scaling characteristics, monitoring, and community ecosystem. Decision framework for greenfield and migration scenarios.
Data Quality at Scale: Great Expectations Beyond Unit Tests for Data
Advanced patterns for Great Expectations in production: custom expectations, data docs auto-generation, checkpoint orchestration, and integration with Dagster and Airflow for automated quality gating.
2027 Data Engineering Predictions: AI-Augmented Pipelines, Real-Time Universal Catalogs, and the Death of Batch
Predictions for data engineering in 2027: AI-assisted pipeline generation, universal catalogs with Unity Catalog and Iceberg REST, real-time streaming replacing nightly batches, and the convergence of data and ML platforms.
Data Platform as a Product: UX Patterns for Internal Developer Platforms
Treating the data platform as an internal product: developer experience design, self-service data ingestion, catalog/search UX, pipeline debugging tools, and SLA dashboards. Patterns from leading platform teams.
The Rise of the Analytics Engineer: dbt, SQLMesh, and the Modern Data Stack
The analytics engineering discipline: how dbt and SQLMesh transformed the data workflow, the shift from ETL to ELT, analytics engineering best practices, and the evolving role between data engineering and data science.
Cost Optimization in Data Pipelines: Engineering for Efficiency at Petabyte Scale
Strategies for reducing data pipeline costs: intelligent partitioning, incremental processing, compute auto-scaling, storage tiering (Iceberg maintenance), query optimization, and workload scheduling on spot/preemptible instances.
Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production
Schema Registry Patterns: Avro, Protobuf, and JSON Schema in Production Schema registry architectures enable versioned, contract-enforced data serialization across distributed pipelines.
Dagster's Asset-Centric Orchestration Model
Deep dive into Dagster's asset-based orchestration model, software-defined assets, and the shift from DAG-centric to asset-centric pipeline design. Covers partitioning, backfills, and the Dagster UI.
Data Products: Designing APIs for the Internal Data Platform
Data product design patterns: API contracts, SLAs, versioning, discovery, and access control. Implementation with dbt (data products as models), Dagster (software-defined assets), and DataHub for cataloging.
Data Mesh in Practice: Implementing Domain Ownership Without Chaos
Practical guide to data mesh adoption: domain ownership patterns, data product definitions, federated governance, and the compute platform. Case studies of mesh implementations and common failure modes.
Terraform for Data Infrastructure: Infrastructure as Code for the Data Platform
Infrastructure as Code patterns for data platforms: Terraform modules for Kafka clusters, Iceberg catalogs, dbt Cloud projects, and Dagster deployments. State management, CI/CD for infrastructure, and multi-environment strategies.
Kubernetes for Data Engineering: Running Data Pipelines on K8s
Running data workloads on Kubernetes: Airflow Executor types (Celery vs Kubernetes), Dagster on K8s, Spark on Kubernetes with the Spark Operator, and stateful workloads (Kafka, Flink) on K8s. Resource management and cost optimization.
Feature Stores at Scale: Feast vs Tecton in Production Deployments
Deep comparison of Feast (open source) and Tecton (managed): feature definitions, online/offline serving, point-in-time correctness, stream feature computation, and cost models. Production deployment patterns.
ML Pipeline Orchestration: From Notebook to Production with Feast and MLflow
Production ML pipeline patterns: Feast feature serving for training/inference consistency, MLflow model registry and deployment, and Dagster for ML pipeline orchestration. Feature engineering at scale with dbt and Spark.
Data Observability: Monitoring, Lineage, and Incident Response for Pipelines
Implementing data observability with open source tools: OpenLineage for lineage, Great Expectations for quality monitoring, and custom health checks. Incident response runbooks and SLAs for data products.
Building a Data Platform on a Budget: The Open Source Stack in 2026
Complete open source data stack: Dagster + dbt + Iceberg + Trino + DuckDB + Superset. Cost analysis against Snowflake and Databricks. Deployment patterns with Docker Compose, Terraform, and Kubernetes.
SQLMesh: The SQL-First Data Transformation Framework Challenging dbt
SQLMesh's approach to data transformation: physical vs logical plans, virtual data environments, automatic column-level lineage, and backward-incompatible change detection. Comparison with dbt's materialization model.
dbt Mesh: Decentralizing Data Transformation at Enterprise Scale
dbt Mesh architecture: dbt projects as domains, cross-project refs, blue/green deployments for models, and governance through the dbt Cloud Discovery API. Migration guide from monolithic dbt projects.
Data Pipeline Patterns for High-Throughput Genomics: Orchestrating Bioinformatics Workflows with Dagster
Applying modern DataOps orchestration to genomics: Dagster assets for sequencing pipeline stages, Great Expectations for quality gates on base-call accuracy, and dbt for cohort-level analytical transformations.
Reproducible ML Pipelines in Computational Biology: MLOps for CRISPR Target Discovery
Applying DataOps and MLOps principles to computational biology workflows: version-controlled training data, reproducible feature engineering, automated model validation, and pipeline CI/CD for CRISPR target prediction.
EY Canada published a cybersecurity report and most citations were hallucinated -- 🛡️ AML 2026-06-01
Trending (HackerNews, 2026-06-01) EY Canada published a cybersecurity report and most citations were hallucinated (discussion) (320 pts) White House's Aliens.gov Site Brags That ICE Arrested More