black-box models

Taming Black-Box LLM Inference Scheduling

A scheduling approach for black-box LLM inference that uses predicted output lengths to reduce queueing friction at scale.