You can be productive in Spark for months before you understand what it is doing
I showed one of my mentees a Spark query plan for the first time.
He asked: what’s an Exchange?
Forty minutes later we were still on that one question.
Not because he was slow. Because the real answer isn’t “Exchange means shuffle.” That’s the definition.
The answer is:
- Why data has to move across nodes at all
- What it costs when it does
- Why the way you write a join determines whether it happens
- And why Spark can’t always avoid it, even when it looks like it should
He had written Spark code without needing to know any of that. The code ran. The jobs finished. Nobody asked why they were slow.
That’s the gap nobody talks about: you can be productive in Spark for months before you understand what Spark is actually doing.
.explain() is the most underused tool in data engineering. Not because it’s hidden, but because nobody teaches you what to look for inside it.
Tools first, understanding later. Usually much later, and only when something breaks.
When did you first understand what a shuffle actually costs you?