1️⃣ Simultaneous calls to the endpoint must reach models with the same configuration

Correct strategy: All at once traffic shifting

Why:

“All at once” replaces the old model with the new model immediately. After the switch, all concurrent requests hit only the new model configuration, guaranteeing configuration consistency across simultaneous calls.

2️⃣ The new model must receive only a fraction of the requests for validation before receiving all the traffic

Correct strategy: Canary traffic shifting

Why:

Canary deployments route a small percentage of traffic (for example, 5% or 10%) to the new model first. This allows validation of correctness and performance before shifting 100% of traffic.

3️⃣ Traffic to the new model must increase gradually to ensure that pipelines that rely on the endpoint do not fail because of changes in latency

Correct strategy: Linear traffic shifting guardrail

Why:

Linear traffic shifting gradually increases traffic in equal increments over time and includes guardrails (such as CloudWatch alarms) to automatically roll back if latency or errors exceed thresholds.