AiGatewayDynamicRouting

Source: src/Cloudflare/AiGateway/DynamicRouting.ts

A dynamic routing configuration (“route”) on a Cloudflare AI Gateway.

Dynamic routing models request handling as a graph of elements — start, conditional, percentage split, rate limit, model, and end nodes — so a single gateway endpoint can A/B test models, enforce per-user budgets, and fall back between providers without app changes.

Cloudflare versions route configurations: changing elements creates a new version and deploys it; the reconciler also re-deploys when the live deployed version drifts from the desired graph. Renames are applied in place; only moving the route to a different gateway forces a replacement.

Creating a Route

const gateway = yield* Cloudflare.AiGateway("Gateway");

const route = yield* Cloudflare.AiGatewayDynamicRouting("Llama", {
  gatewayId: gateway.gatewayId,
  elements: [
    { id: "start", type: "start", outputs: { next: { elementId: "model" } } },
    {
      id: "model",
      type: "model",
      properties: {
        provider: "workers-ai",
        model: "@cf/meta/llama-3.1-8b-instruct",
        retries: 1,
        timeout: 30000,
      },
      outputs: {
        success: { elementId: "end" },
        fallback: { elementId: "end" },
      },
    },
    { id: "end", type: "end", outputs: {} },
  ],
});

Updating a Route

const route = yield* Cloudflare.AiGatewayDynamicRouting("Llama", {
  gatewayId: gateway.gatewayId,
  elements: [
    { id: "start", type: "start", outputs: { next: { elementId: "model" } } },
    {
      id: "model",
      type: "model",
      properties: {
        provider: "workers-ai",
        model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
        retries: 2,
        timeout: 60000,
      },
      outputs: {
        success: { elementId: "end" },
        fallback: { elementId: "end" },
      },
    },
    { id: "end", type: "end", outputs: {} },
  ],
});