Tuning spark.sql.shuffle.partitions Dynamically.

Static 200 partitions are rarely optimal; use AQE to tune this dynamically.

Too many partitions cause task scheduling overhead. Too few cause OOM or spill to disk. With AQE enabled, Spark starts with a high number and merges small partitions at runtime.