Data platform engineer, teaching what actually works
Your data platform bill is climbing and nobody can explain why. Learn to fix it.
I tune production Spark, cut cloud data platform cost, and lead migrations off CDH and Teradata. Now I teach the same diagnostic loops in video courses and a guided roadmap - so the next slow job or climbing bill is a diagnosis, not a guess.
Courses
All courses →Spark Performance Tuning in Production
Read a query plan, find the shuffle or skew that is actually costing you, and fix it without throwing more hardware at the problem.
Cloud Data Platform Cost Optimization
Tear down the bill in the right order - idle compute first, then query efficiency, then storage hygiene - so the number actually moves.
Migrating off CDH and Teradata
Lead a migration onto Snowflake or Databricks with validation first, so the business can see the numbers provably match before the cutover.
Free guide
Not sure where to start? Follow the roadmap.
The Data Engineer Roadmap - the full path from your first SQL query to owning a platform you can defend on cost, reliability, and scale. Free and public, with the deep phases pointing at a course.
Recent writing
All writing →Your Iceberg metadata directory is growing faster than your data
Every commit writes a snapshot and nothing cleans them up by default. Streaming ingest can leave tens of thousands of files in one directory.
Read →You can be productive in Spark for months before you understand what it is doing
I showed a mentee a query plan for the first time. He asked what an Exchange was. Forty minutes later we were still on that one question.
Read →Your Delta table is Iceberg-compatible. Your readers just don't know it yet
Iceberg compatibility on a Delta table is not instant. The metadata sync lags, and downstream readers can silently consume a stale snapshot.
Read →Or work with me directly
How I work →Apache Spark performance tuning
Jobs that run slower than they should, without anyone knowing why.
- Skew and shuffle pathologies after joins and aggregations
- Broadcast join thresholds and partition sizing
- Killing expensive Python UDFs and cache misuse
- Reading query plans to find the number that actually matters
Cloud platform cost optimization
A bill that keeps climbing and nobody can fully explain.
- Idle compute and always-on clusters running around the clock
- Ephemeral job clusters, auto-termination, and spot instances with fallback
- Vectorized engines and autoscaling where they pay for themselves, not everywhere
- Delta table maintenance and small-file cleanup
Platform migrations
Moving off CDH or Teradata without breaking trust.
- Migrations onto modern cloud data platforms
- Validation-first approach so the numbers provably match
- The boring patterns that survive the next platform change
- De-risking the cutover before it reaches production
Platform architecture and reliability
A modern platform one promotion away from being a legacy monolith.
- Data contracts, SLAs, and pipeline trust
- Delta and Iceberg maintenance, retention, and metadata hygiene
- Catching silent degradation before a missed deadline finds it
- Boring, durable design over clever, fragile design
Got a pipeline that costs too much or runs too slow?
That is the work. Tell me the symptom - a platform bill nobody can explain, a Spark job that keeps creeping, a migration you are not sure will land. If it is not something I can help with, I will tell you straight.