1 link tagged with all of: big-data + data-engineering + cloud-computing + business-intelligence + data-modeling
Links
This article explores the evolving role of data engineers over the past 50 years, highlighting their often unnoticed contributions to data infrastructure. It discusses the challenges they face, such as managing dependencies and schema changes, while emphasizing that the core problems remain unchanged despite new tools and technologies.
- The core problems of data engineering—dependencies, integration issues, schema changes—haven't changed in 50 years despite new tools like dbt, Iceberg, and cloud warehouses.
- Data engineers are invisible when things work but instantly blamed when something breaks.
- Real-time data requests are often unjustified since almost no one can explain how a 10-minute delay would actually change a decision.
- Industry growth reflects better marketing and rebranded terminology more than genuine advances in solving data management problems.
data-engineering
business-intelligence
big-data
data-modeling
cloud-computing