Together AI moves model A/B testing out of app code and into the inference endpoint
Together's new traffic-tests feature splits an endpoint's live traffic across one control and up to 20 variant deployments by percentage, re-sampling requests at the infrastructure layer rather than requiring a branch in your application. Variants must carry zero weight in the base split so the experiment owns their allocation; an optional sampling_key gives sticky per-user assignment, and etag versioning guards concurrent config overwrites. Platform metrics (latency, errors, throughput) come free, while product metrics like ratings or task completion are joined by deployment ID from response metadata — Together's argument being that shadow traffic proves operational soundness but can never tell you which model users prefer.
↳ Follow the thread