How does latency p99 optimization differ from average latency optimization in ChatGPT systems?

p99 optimization focuses on worst-case latency experiences, while average latency optimization focuses on overall system efficiency.

In ChatGPT systems, optimizing average latency can still leave some users with extremely slow responses. p99 latency measures the worst 1% of requests, which is critical for user experience consistency.

Techniques like request shedding, adaptive batching, priority queues, and timeout-based rerouting are used to reduce tail latency. These strategies often sacrifice some average efficiency to improve worst-case performance.

Tail latency is especially important in interactive systems where slow responses significantly degrade user experience.