Spark Performance Tuning in Production
Read a query plan, find the shuffle or skew that is actually costing you, and fix it without throwing more hardware at the problem.
This course is in production. Leave your details and I will let you know the moment it is live.
Checkout is coming soon. For now, reach out on LinkedIn.
Most Spark jobs are slow for a handful of repeatable reasons. This course is the diagnostic loop I use on real workloads: read the plan, find the shuffle or skew that is actually costing you, change one thing, and confirm it in the UI.
You will not memorise config flags. You will learn to read what Spark is telling you, so the next slow job is a five-minute diagnosis instead of a week of guessing.
Who this is for
Data engineers who write Spark that works but occasionally runs slower than it should, and who want to stop reaching for a bigger cluster as the first move.
What you need
A working knowledge of PySpark or Spark SQL and access to any Spark environment (Databricks, EMR, or a local build) to follow along.
Curriculum
How Spark actually runs your job
- Jobs, stages, and the shuffle boundary Free preview 14m
- Reading a physical plan without guessing 18m
- The one number in the Spark UI that matters most 12m
Skew and shuffle pathologies
- Spotting skew after joins and aggregations 16m
- Salting, and when it makes things worse 15m
- Partition sizing and the small-files trap 13m
Joins that pay for themselves
- Broadcast join thresholds you can actually reason about 17m
- Sort-merge vs broadcast, decided by the plan 14m
The expensive habits
- Why a Python UDF is the most expensive three letters 19m
- Cache misuse and how to prove it 13m