Course · Data platforms
Delta Lake
Delta Lake adds ACID transactions, schema enforcement and time travel to files in a data lake, which is the foundation of the lakehouse pattern.
- Lessons
- 6
- Interview questions
- 3
- Projects & case studies
- 5
- Reading time
- ~1 h
About this course
Delta Lake is an open table format that stores data as Parquet files plus a transaction log. That log turns a folder of files into a table with atomic commits, consistent reads and a history you can query.
Learn what the transaction log provides first, then schema enforcement and evolution, then maintenance such as compaction.
Your progress
Saved in this browser onlyPractise
- InterviewDelta Lake interview questionsThe full list with difficulty, type and a box to tick off each one.
- Cheat sheetDatabricks Cheat SheetA quick Databricks reference: Unity Catalog names and grants, Delta table operations, medallion layers, job design and the compute choices that keep costs down.
- InterviewAll interview questionsEvery question across all topics in one filterable list.
Course structure
Lessons
Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.
Start here
The complete overview of the course in one read.
Beginner
Core concepts you will use every day.
Intermediate
Patterns used in production pipelines.
- Delta Lake: Transactions, Schema Evolution and Time TravelSee how Delta Lake's transaction log gives ACID guarantees on data lake files, how schema enforcement and evolution work, and how time travel and VACUUM interact.
- Delta Lake vs Traditional Data Lake TablesWhat changes when a folder of Parquet files becomes a Delta table: atomic writes, updates and deletes, schema enforcement, time travel and faster metadata.
- Parquet vs Avro vs ORC for Data EngineeringCompare Parquet, ORC and Avro by layout, compression, schema evolution and use case, with a measured size comparison and how column pruning works in Spark.
Projects and case studies
Apply what you learned and prepare material to discuss in interviews.
Projects
- AdvancedChange Data Capture PipelineReplicate an operational PostgreSQL table into a lakehouse table within minutes, including updates and deletes, so analysts query current data without touching the production database.
- AdvancedFraud Detection Data PipelineBuild the data side of a fraud-detection system: compute per-card behavioural features from a transaction stream, flag suspicious transactions with transparent rules, and maintain a feature table that a model could use.
- AdvancedKafka → Spark → Delta Lake Streaming PipelineAn application emits user events to Kafka. Build a streaming pipeline that lands them in Delta Lake within a minute, deduplicates replays, and produces per-minute aggregates that tolerate late events.
- IntermediateLarge-Scale Batch Processing PipelineProcess a large public dataset (several gigabytes or more) with PySpark into partitioned, query-ready tables, and document how you found and fixed the main performance bottleneck.
System design case studies
Resources
Cheat sheets
Related courses
- Apache SparkUnderstand how Spark turns your code into jobs, stages and tasks, and why partitions, shuffles and data skew drive performance.
- PySparkPySpark is the Python API for Apache Spark. Learn DataFrames, joins, window functions and how partitions and shuffles decide performance.
- Data modelingData modeling and warehousing: star schemas and grain, fact and dimension design, SCDs, Data Vault and other methods, dbt, semantic layers and incremental models.