← Back to Mind Maps
Data Engineering Roadmap
A comprehensive macro-level view of the entire Data Engineering landscape, from basic scripting to modern data architectures.
1. Programming & Scripting
The core languages to write your data pipelines.
SQL (Essential)Python (Standard)Scala / Java (Big Data)Bash / Shell Scripting
2. Databases & Storage
Where the data lives at rest.
Relational (PostgreSQL, MySQL)NoSQL (MongoDB, Cassandra)Object Storage (S3, GCS, ADLS)File Formats (Parquet, ORC, Avro)
3. Data Warehousing & Modeling
Structuring data for analytical workloads.
SnowflakeBigQuery / RedshiftKimball (Star Schema)Data VaultMedallion Architecture (Bronze/Silver/Gold)
4. Data Processing
Transforming and moving data (ETL/ELT).
Batch ProcessingStream ProcessingApache Spark (PySpark)Apache Flink / Kafkadbt (Data Build Tool)Pandas / Polars
5. Orchestration
Scheduling and monitoring your pipelines.
Apache AirflowDagsterPrefectCron (Basics)Cloud Native (Step Functions, Composer)
6. DataOps & Infrastructure
Deploying and managing data systems reliably.
Git (Version Control)Docker & KubernetesCI/CD (GitHub Actions)Terraform (IaC)Data Quality (Great Expectations)
7. Advanced Concepts
Scaling and modern data architectures.
Data Mesh & Data FabricLakehouse ArchitectureStreaming AnalyticsData Governance & CatalogingFinOps (Cost Management)