AfAirflow

Course structure

• 1 lessonIntermediatePatterns used in production pipelines.Starts with: Airflow DAGs, Scheduling, Retries and Task Dependencies

Lessons

Work through the lessons in order. Completed lessons show a tick; lessons you have opened are outlined.

Projects and case studies

Apply what you learned and prepare material to discuss in interviews.

Projects

• IntermediateS3 → PySpark → Snowflake Data PipelineRaw event files arrive in object storage every day. Clean and aggregate them with PySpark and load curated tables into a cloud warehouse for analysts, with each day reprocessable on demand.PySpark · Amazon S3 (or any object storage) · Snowflake (or another cloud warehouse)

System design case studies

• IntermediateDesign a Scalable Batch Data PipelineDesign a pipeline that ingests daily order and customer extracts from several operational systems and produces reliable, analysis-ready tables for reporting by 07:00 each morning.

Resources

Cheat sheets

• Cheat sheetAirflow Cheat SheetA quick Airflow reference: TaskFlow DAGs, schedules and data intervals, retries, templating, sensors, trigger rules and the Airflow 3 CLI commands you use most.

Related courses

• PyPythonPython glues pipelines together: ingestion, validation, orchestration and PySpark jobs. Focus on functions, generators, error handling and testable code.5 lessons4 questions
• KfKafkaKafka is a distributed log used for streaming data. Learn topics, partitions, consumer groups and delivery semantics before building streaming pipelines.3 lessons3 questions
• SpApache SparkUnderstand how Spark turns your code into jobs, stages and tasks, and why partitions, shuffles and data skew drive performance.4 lessons9 questions

Plan your learning

• RoadmapData Engineer roadmapSee where this course fits in the full journey from beginner to job ready.
• PlansStudy plans30, 60 and 90-day plans that sequence courses, practice and projects.