UDF is the most expensive three letters in your PySpark job

UDF in PySpark is the most expensive three letters in your job config.

Every row crosses the JVM-Python boundary twice. Spark serializes data out to Python, your function runs, then it serializes back. On a billion-row table, that overhead doesn’t add minutes, it adds hours.

Replace them with native pyspark.sql.functions expressions. Same logic, zero infrastructure changes.

The fix order when you see udf() in production:

• Check if pyspark.sql.functions already has what you need. It usually does
• If you need custom logic, use @pandas_udf instead. It is vectorized, with no row-by-row serialization overhead
• Only keep scalar UDFs when there is genuinely no other path

The issue isn’t that people write bad UDFs. It’s that Python UDFs look identical to native functions in the code, so nobody questions them until the job becomes a problem.

When did you last run .explain(True) on a job with UDFs, instead of trusting that the runtime is just how Spark works?