timebill-inference

Guarantee LLM inference completes within time budgets via response length prediction, execution time estimation, and adaptive KV cache eviction. Three-stage pipeline predicts response length, estimates end-to-end time with 1.22% accuracy, adjusts cache eviction ratio—enabling time-critical deployment in robotics and autonomous systems.

Independent GitHub snapshot · Automated classification · Original documentation governs