Click any tag below to further narrow down your results
Links
This article outlines the evolution of a data pipeline from using JSON to AVRO for Change Data Capture (CDC). As the business expanded, the limitations of JSON became apparent, leading to the adoption of AVRO, which improved performance, reduced storage costs, and streamlined schema evolution.
This article outlines four orchestration levels for data workflows, from simple cron jobs to full orchestration systems like Prefect. It emphasizes that teams should choose tools based on their project's maturity and specific needs, rather than following trends or trying to adopt complex solutions prematurely.
This article introduces the features of Apache Spark 4.1, highlighting advancements like Spark Declarative Pipelines for easier data transformation, Real-Time Mode for low-latency streaming, and improved PySpark performance with Arrow-native UDFs. It also covers enhancements in SQL capabilities and Spark Connect for better stability and scalability.
This article discusses the evolution of data engineering as it adapts to the growing role of AI agents in 2026. It emphasizes the need for reliability, context, and safety within data platforms, highlighting the shift from human-centric workflows to autonomous systems that require new architectural approaches.
The article argues for a shift from Developer Experience (DX) to Agent Experience (AX) in data platform engineering. As AI agents take on more coding and system management tasks, platforms must be designed for machine readability and autonomy, prioritizing structured data and programmatic interfaces over human-centric designs.
This article explores the integration of Flink, Airflow, and StarRocks for real-time data processing. It details different methods for data ingestion, including routine loads and Kafka connectors, while sharing lessons learned from implementation. The author concludes with a preference for the Flink Connector due to its flexibility and existing infrastructure.
This article lists notable data engineering projects from late December 2025. It features a variety of pipelines and platforms, highlighting their purposes and technologies used, like Airflow, Kafka, and machine learning tools. Users can explore, vote, and share their own projects within the community.
Google Cloud has introduced the Data Engineering Agent in BigQuery, designed to automate complex data engineering tasks. It allows users to create and modify pipelines using natural language, integrates with Dataplex for enhanced data governance, and simplifies data preparation and troubleshooting.
This article discusses the evolving role of data engineers in the age of AI, emphasizing the need to adapt data preparation strategies. It highlights the shift from traditional data workflows to flexible, context-aware systems that prioritize data curation over mere collection.
This article explains how to use vector embeddings to quantify the similarity between SQL queries. It covers techniques for generating embeddings, storing queries, and analyzing their relationships through clustering and distance measurements. The approach enhances understanding of user behavior and query efficiency in data lakes.
This article features insights from four experienced data engineers discussing common questions from the r/dataengineering subreddit. They cover topics like job interview preparation, data quality challenges, and the choice between data warehouses and lakehouses. Each expert provides practical advice based on their experiences in the field.
This article explains how to use dbt's defer feature to improve CI/CD pipeline efficiency by only rebuilding modified models instead of the entire project. It covers the setup process, benefits, and potential pitfalls of implementing defer in dbt workflows.
Kostas Pardalis discusses Fenic, an open-source DataFrame engine inspired by PySpark, aimed at enhancing data engineering for AI applications. He highlights how Fenic incorporates semantic operators to improve data transformation and management, addressing the limitations of traditional data infrastructure in the AI era.
This article discusses the upcoming incremental view maintenance (IVM) feature in StarRocks, focusing on its architecture and advantages over previous methods. It explains how IVM allows for efficient updates to materialized views by only processing changed data, thus improving performance and reducing computational overhead.
This article outlines how a team at Astronomer transformed their data pipeline creation process by adopting a standardized, modular approach. They implemented a declarative framework using Airflow Task Groups, allowing them to automate repetitive tasks, improve efficiency, and focus on core business logic rather than boilerplate code.
This article explains how DLT-META, a metadata-driven framework, helps automate and standardize Spark Declarative Pipelines. It addresses common data engineering challenges like scaling, maintenance, and logic consistency, allowing teams to onboard new data sources quickly and efficiently.
This article explains how Apache Hudi manages schema evolution in data lakehouses, allowing for seamless changes in data structures without disrupting pipelines. It covers practical implementation using PySpark and highlights the benefits of agility, backward compatibility, and pipeline reliability.
This article explains how to use built-in PySpark functions to efficiently manipulate map data types in data pipelines. It covers functions like `transform_keys`, `map_filter`, and `map_contains_key`, highlighting their utility in cleaning and transforming semi-structured data.
This article details how Flipkart developed a real-time advertising system capable of processing one million events per second while maintaining precise budget control. It discusses the architecture, including event-time semantics, deduplication, and a hybrid stream-batch approach, to ensure financial accuracy and prevent overspending.
Cloud Composer now supports Apache Airflow 3.1, offering new features like Human-in-the-Loop workflows and proactive Deadline Alerts. These enhancements improve oversight, reliability, and support for international teams. The update promotes open orchestration, allowing greater flexibility and community-driven innovation.
This article outlines the evolving role of data engineering as we approach 2026, focusing on the integration of agentic AI systems. It emphasizes the need for data engineers to create context-rich data products, manage active metadata, and design systems that support AI workflows.
This article explores the evolving role of data engineers over the past 50 years, highlighting their often unnoticed contributions to data infrastructure. It discusses the challenges they face, such as managing dependencies and schema changes, while emphasizing that the core problems remain unchanged despite new tools and technologies.
The article highlights a looming crisis in data engineering talent, emphasizing that the industry is failing to cultivate junior engineers needed for future demand. It critiques current hiring practices that prioritize experienced candidates while neglecting the development of entry-level roles, leading to burnout among existing engineers. Additionally, it explores the role of AI in enhancing productivity but warns against relying solely on it to address talent shortages.
The article outlines five critical trends reshaping the role of data engineers as AI evolves into a more integrated and operational force by 2026. Emphasizing the need for unified data and AI infrastructures, it highlights the shift from data quantity to quality, the importance of real-time processing, and the necessity of handling multimodal data. Data engineers are urged to adapt their skills to build context-aware systems that can support the demands of AI agents.
Data engineering teams are facing soaring infrastructure costs that challenge the initial promises of cloud scalability. With fragmented systems and a lack of financial awareness, organizations struggle to manage expenses effectively, but embracing a platform team model and improved cost visibility can lead to significant savings and optimized operations.
Andrei Kaparthy's insights on AI's role in work resonate with many, prompting a reflection on how to integrate these ideas into data engineering practices. The article emphasizes the importance of mastering fundamentals to effectively evaluate AI-generated work and encourages active participation in the evolving landscape of technology.
The article introduces Apache Spark 4.0, highlighting its new features, performance improvements, and enhancements aimed at simplifying data processing tasks. It emphasizes the importance of this release for developers and data engineers seeking to leverage Spark's capabilities for big data analytics and machine learning applications.
The article discusses the advancements in data engineering over the past year and highlights the current trends shaping the field. It emphasizes the importance of evolving technologies and methodologies that enhance data management and analytics. Insights into best practices and challenges faced by data engineers are also provided.
The article discusses the medallion architecture, highlighting its importance in data engineering for organizing data into layers. It revisits the principles of this architecture, emphasizing its role in enhancing data accessibility and quality for analytics and machine learning tasks. The piece also explores practical implementations and benefits of adopting this architectural approach in modern data workflows.
Real-time analytics solutions enable querying vast datasets, such as weather records, with rapid response times. The article outlines how to effectively model data in ClickHouse for optimized real-time analytics, covering techniques from ingestion to advanced strategies like materialized views and denormalization, while emphasizing the importance of efficient data flow and trade-offs between data freshness and accuracy.
The article discusses the future of data engineering in 2025, focusing on the integration of AI technologies to enhance data processing and management. It highlights the evolving roles of data engineers and the importance of automation and machine learning in improving efficiency and accuracy in data workflows.
The article discusses the evolving landscape of data engineering tools, particularly focusing on SQLMesh, dbt, and Fivetran. It highlights the integration and future developments of these platforms in the context of data transformation and analytics workflows. The piece aims to provide insights into what users can expect next in the realm of modern data stack solutions.
Open lakehouses are reshaping the data engineering landscape, presenting both opportunities and challenges for Databricks as competitors like DuckDB and Apache Ray emerge. These tools offer simplified and cost-effective alternatives for data processing and analytics, leading to potential integration complexities and the need for Databricks to adapt or risk losing its competitive edge. The future success of Databricks may hinge on its ability to manage this evolving ecosystem.
Medallion Architecture organizes data into three distinct layers—Bronze, Silver, and Gold—enhancing data quality and usability as it progresses through the system. Originating from Databricks' Lakehouse vision, this design pattern emphasizes the importance of structured and unstructured data integration for effective decision-making.
Apache Airflow 3.1 is set to release soon, featuring significant updates such as Human-in-the-Loop integration for workflows requiring human approval, a new React plugin system for customization, and various quality of life improvements in the UI. The release also includes internationalization support, making it more accessible for global teams. Users are excited about the potential of these enhancements to improve data orchestration processes.
Professor Paul Groth from the University of Amsterdam discusses his research on knowledge graphs and data engineering, addressing the evolution of data provenance and lineage, challenges in data integration, and the transformative impact of large language models (LLMs) on the field. He emphasizes the importance of human-AI collaboration and shares insights from his work at the intelligent data engineering lab, shedding light on the interplay between industry and academia in advancing data practices.
The article discusses the capabilities and benefits of Databricks SQL Scripting, highlighting its features that enable data engineers to write complex SQL queries and automate workflows efficiently. It emphasizes the integration of SQL with data processing and visualization tools, allowing for enhanced data analytics and insights.
OpenMetadata is an open-source platform that simplifies metadata management, enabling organizations to effectively manage their data assets through a centralized repository. It addresses challenges such as fragmented data sources and enhances data discoverability, governance, and collaboration by providing features like lineage tracking, data quality monitoring, and a user-friendly interface. With extensive connector support and a schema-first approach, OpenMetadata is gaining popularity in the data engineering community.
The content appears to be corrupted or unformatted text without coherent information or context about Dagster or SLURM. It fails to convey a clear message or topic for analysis.
Meta has developed a "Global Feature Importance" approach to enhance feature selection in machine learning by aggregating feature importance scores from multiple models. This method allows for systematic exploration and selection of features, addressing challenges of isolated assessments and improving model performance significantly. The framework supports data engineers and ML engineers in making informed decisions about feature utilization across various contexts, resulting in better predictive outcomes.
The provided content appears to be corrupted or unreadable text, lacking coherent information or context. There doesn't seem to be any meaningful data or insights regarding data engineering or related topics.
Building Kafka on top of S3 presents several challenges, including data consistency, latency issues, and the need for efficient data retrieval. The article explores these obstacles in depth and discusses potential solutions and architectural considerations necessary for successful integration. Understanding these challenges is crucial for engineers looking to leverage Kafka with S3 effectively.
The article discusses the rise of single-node architectures as a rebellion against traditional multi-node systems in data engineering. It highlights the advantages of simplicity, cost-effectiveness, and ease of management that single-node setups provide, particularly for smaller projects and startups. The piece also explores the implications for scalability and performance in various use cases.
The article critiques the current state of data engineering, arguing that the field has become cluttered with unnecessary jargon and complexity that detracts from its core purpose. It calls for a more straightforward approach that emphasizes practicality over buzzwords.
The article provides an overview of dbt (data build tool), explaining its role in data transformation and analytics workflows. It highlights how dbt enables data teams to manage and version control their data transformations, fostering collaboration and improving data quality. Additionally, it discusses the benefits of using dbt in modern data architecture and analytics practices.
The article introduces PyIceberg, a tool designed to help data engineers manage and query large datasets efficiently. It emphasizes the importance of handling data in motion and how PyIceberg integrates with modern data infrastructure to streamline processes. Key features and use cases are highlighted to showcase its effectiveness in data engineering workflows.
Rapid consolidation in the data engineering market is leading to the unification of tools into larger data platforms. The article provides a timeline of significant acquisitions from 2022 to the present, highlighting trends in open-source versus closed-source strategies in the industry. It discusses the challenges of monetizing open-source products while advocating for their importance in fostering trust and innovation.
Data engineering best practices are being challenged by modern demands for speed, agility, and purpose-driven architecture. Experts advocate for a shift from traditional centralized models to more flexible, intent-driven approaches that prioritize real business outcomes and guided autonomy. The need for a balance between standardization and freedom is crucial to avoid chaos and technical debt in data platforms.
Shifting left in data engineering involves moving data quality checks and business logic closer to the data source, enhancing data quality, performance, and maintainability. This approach, which has evolved from concepts in software testing and security, allows organizations to catch errors earlier and optimize costs by leveraging a declarative data stack. As data architectures mature, adopting shifting left practices can lead to significant improvements in data governance and collaboration among domain experts.
Maintaining high data quality is challenging due to unclear ownership, bugs, and messy source data. By embedding continuous testing within Airflow's data workflows, teams can proactively address quality issues, ensuring data integrity and building trust with consumers while fostering shared responsibility across data engineering and business domains.
Netflix has developed a Real-Time Distributed Graph (RDG) to address the complexities arising from their evolving business model, which includes streaming, ads, and gaming. The first part of this series details the architecture and ingestion pipeline that processes vast amounts of data to facilitate quick querying and insights.
Effective documentation in dbt is essential for enhancing team collaboration, reducing onboarding time, and improving data quality. Best practices include documenting at the column and model levels, integrating documentation into the development workflow, and tailoring content for various audiences. By prioritizing clear and comprehensive documentation, teams can transform their data projects into transparent and understandable systems.
The author critiques the Medallion Architecture promoted by Databricks, arguing that it is merely marketing jargon that confuses data modeling concepts. They believe it misleads new data engineers and pushes unnecessary complexity, advocating instead for traditional data modeling practices that have proven effective over decades.
The article discusses the growing importance of vector databases and engines in the data landscape, particularly for AI applications. It highlights the differences between specialized vector solutions like Pinecone and Weaviate versus traditional databases with vector capabilities, while addressing their integration into existing data engineering frameworks. Key considerations for choosing between vector engines and databases are also examined, as well as the evolving technology landscape driven by AI demands.
Understanding Kafka and Flink is essential for Python data engineers as these tools are integral for handling real-time data processing and streaming. Proficiency in these technologies enhances a data engineer's capability to build robust data pipelines and manage data workflows effectively. Learning these frameworks can significantly improve job prospects and performance in data-centric roles.
A local data platform can be built using Terraform and Docker to replicate cloud data architecture without incurring costs. This setup allows for hands-on experimentation and learning of data engineering concepts, utilizing popular open-source tools like Airflow, Minio, and DuckDB. The project emphasizes the use of infrastructure as code principles while providing a realistic environment for developing data pipelines.
Fiverr rebuilt its data warehouse using dbt Cloud and Prefect to create dynamic data pipelines that execute only necessary components based on upstream changes. By implementing a custom orchestration layer, they achieved faster data delivery, reduced compute costs, and improved overall efficiency in managing data transformations. The solution emphasizes real-time readiness checks and targeted execution to optimize resource usage.
The article explores the evolving nature of data and AI engineering, arguing for a shift from defined processes to empirical approaches that embrace adaptability and variability. It draws parallels between the martial arts philosophies of Bruce Lee and Chuck Norris to illustrate the need for data teams to be innovative and responsive in their work. By discussing the definitions and professional standards in engineering, the piece advocates for recognizing data and AI engineering as legitimate engineering disciplines.
Tuning Spark Shuffle Partitions is essential for optimizing performance in data processing, particularly in managing DataFrame partitions effectively. By understanding how to adjust the number of partitions and leveraging features like Adaptive Query Execution, users can significantly enhance the efficiency of their Spark jobs. Experimentation with partition settings can reveal notable differences in runtime, emphasizing the importance of performance tuning in Spark applications.
The podcast episode features an interview with Pete Hunt of Dagster, discussing the evolution of data engineering and the role of AI abstractions in shaping its future. Hunt emphasizes the importance of improving workflows and the integration of AI tools to enhance data management and processing efficiency.
Chakravarthy Kotaru discusses the importance of scaling data operations through standardized platform offerings, sharing his experience in managing diverse database technologies and transitioning from DevOps to a platform engineering approach. He highlights the challenges of migrating legacy systems, integrating AI and ML for automation, and the need for organizational buy-in to ensure the success of data platforms.
The article provides an honest review of Polars Cloud, focusing on its performance and usability for data engineering tasks. It highlights the advantages and disadvantages of the platform, comparing it with other solutions in the market. The review aims to give potential users insight into whether Polars Cloud is a suitable choice for their data processing needs.
The article explores the mindset and skills essential for effective data engineering, emphasizing the importance of thinking critically about data systems and architecture. It discusses the necessity for engineers to not only understand data pipelines but also to approach problems with a holistic view, considering scalability, performance, and data quality. Techniques and methodologies are suggested to cultivate this engineering mindset for better outcomes in data projects.
The article outlines five key concepts in data engineering that are essential for professionals in the field. It emphasizes the importance of understanding data architecture, pipeline construction, data governance, scalable systems, and the use of cloud technologies. These concepts are crucial for building efficient and effective data solutions.
The linked content appears to be corrupted and does not contain coherent information about the Data Engineering Podcast or its episodes. As a result, it is not possible to provide a summary or extract relevant details about the podcast.
The article focuses on the principles and practices of security data engineering and ETL (Extract, Transform, Load) processes, emphasizing the importance of data protection and compliance in the handling of sensitive information. It discusses various strategies for implementing secure ETL workflows while ensuring data integrity and accessibility. Best practices and tools are also highlighted to aid professionals in improving their data engineering processes.
The article provides insights into implementing Identity and Access Management (IAM) within data engineering processes. It discusses the importance of security in data management and offers practical guidelines for data engineers to effectively integrate IAM into their workflows.
The article delves into the complexities of StarRocks' implementation of Iceberg's Merge-on-Read (MoR) functionality, specifically focusing on how it efficiently manages deletes with positional and equality delete files. It explores the intricacies of query planning, the role of queue structures in processing, and the handling of schema evolution, all while shedding light on the technical challenges encountered during the exploration of the system's codebase.
Many data engineers experience heightened stress due to inadequate tools and practices, which lead to constant monitoring of systems and unexpected issues. Emphasizing the need for local testing, visibility, and proper troubleshooting, the article advocates for a more structured approach to data engineering that allows professionals to maintain work-life balance without sacrificing system reliability.
The article discusses a common data engineering exam question focused on optimizing SQL queries with range predicates. It emphasizes adopting a first principles mindset, thinking mathematically about SQL, and using set operations for improved performance. The author provides a step-by-step solution for rewriting a SQL condition to illustrate the benefits of this approach.
Tulika Bhatt, a senior software engineer at Netflix, discusses her experiences with large-scale data processing and the challenges of managing impression data for personalization. She emphasizes the need for a balance between off-the-shelf solutions and custom-built systems while highlighting the complexities of ensuring data quality and observability in high-speed environments. The conversation also touches on the future of data engineering technologies and the impact of generative AI on data management practices.
The article discusses the reasons why data engineers may feel stuck in their careers, particularly at the senior level. It emphasizes the importance of continuous learning, adaptability, and exploring new technologies to overcome stagnation and enhance career growth. Strategies for professional development and expanding skill sets are also highlighted.
+ career-development
data-engineering ✓
+ continuous-learning
+ professional-growth
+ technology-trends
MLOps integrates machine learning with DevOps practices to streamline the model development lifecycle, focusing on automation, reproducibility, and performance monitoring. This blog details a practical project to build a House Price Predictor using Azure DevOps for CI/CD, covering setup, data processing, feature engineering, model training, and deployment to Azure Kubernetes Service.
The article provides a comprehensive overview of various architectures that can be implemented using Databricks, highlighting their benefits and use cases for data engineering and analytics. It serves as a resource for organizations looking to optimize their data workflows and leverage the capabilities of the Databricks platform effectively.
Data modeling is considered "dead" by the author due to the shift in focus towards modern data architectures like Data Lakes and Lake Houses, which prioritize flexibility over traditional modeling techniques. The article criticizes the lack of clarity and guidance in contemporary data modeling practices, contrasting it with the structured approaches of the past, particularly those advocated by Kimball. The author expresses a longing for a definitive framework or authority to restore the importance of data modeling in the industry.
The article outlines six key performance indicators (KPIs) that leaders should monitor throughout the data engineering lifecycle to improve efficiency and decision-making. These KPIs cover various aspects of data quality, productivity, and operational performance, providing a framework for evaluating the effectiveness of data engineering processes. By tracking these metrics, organizations can better align their data initiatives with business goals and enhance overall data strategy.
Data engineering is evolving rapidly due to the integration of artificial intelligence, necessitating professionals to acquire new skills. Key areas of focus include data architecture, machine learning, and data governance, which are essential for harnessing AI's potential in data-driven decision-making. Continuous learning and adaptation are crucial for engineers to stay relevant in this AI-centric landscape.
data-engineering ✓
+ artificial-intelligence
+ machine-learning
+ data-governance
+ skills-development
Data engineers play a crucial role in achieving GDPR compliance by implementing systems that manage personal data responsibly. This guide outlines key concepts such as encryption, hashing, and anonymization, as well as best practices for designing data architectures that ensure privacy and security. It also covers practical considerations for incident response and interview preparation related to GDPR.
Deletion Vectors in Delta Lake provide a soft-delete mechanism that enhances performance by allowing updates and deletes without rewriting entire Parquet files. While they improve write efficiency and maintain ACID semantics, they require regular maintenance to manage read overhead and ensure optimal query performance.