Click any tag below to further narrow down your results
Links
This article lists notable data engineering projects from late December 2025. It features a variety of pipelines and platforms, highlighting their purposes and technologies used, like Airflow, Kafka, and machine learning tools. Users can explore, vote, and share their own projects within the community.
Google Cloud has introduced the Data Engineering Agent in BigQuery, designed to automate complex data engineering tasks. It allows users to create and modify pipelines using natural language, integrates with Dataplex for enhanced data governance, and simplifies data preparation and troubleshooting.
This article explains how to use dbt's defer feature to improve CI/CD pipeline efficiency by only rebuilding modified models instead of the entire project. It covers the setup process, benefits, and potential pitfalls of implementing defer in dbt workflows.
This article explains how DLT-META, a metadata-driven framework, helps automate and standardize Spark Declarative Pipelines. It addresses common data engineering challenges like scaling, maintenance, and logic consistency, allowing teams to onboard new data sources quickly and efficiently.