Together AI adds canary rollouts that caught a 137% p95 latency regression at 10% of traffic
Together AI Blog·medium signal
Together's dedicated endpoints now shift traffic in staged steps, 5/25/50/100% by default. Each step waits on a gate over router_latency, router_error_rate or inflight_requests, with either relative-regression or absolute-threshold rules, and it is driven by `tg beta endpoints rollout --canary` in together 2.34.0+. In the demo, a Qwen 7B to 9B swap paused at 10% traffic after the latency regression and served 6,800 requests with zero failures. A rollout ends COMPLETED or CANCELED, never in a half-shifted failed state.