Skip to content

2.0.0-beta.47 - AI Gateway Models

v2.0.0-beta.47 makes a Cloudflare AI Gateway a first-class Effect AI provider, and fixes streaming return values on the built-in Worker/Durable Object method bridge.

AI Gateway as a LanguageModel provider

Section titled “AI Gateway as a LanguageModel provider”

Every Effect AI provider — OpenAI, Anthropic, Workers AI — ships a LanguageModel layer; once it’s in scope, generateText, streamText, tool calls, and structured output are all provider-agnostic. beta.47 makes Cloudflare.AiGateway one of those providers.

Bind the gateway in the Init phase, then call .model({...}) with a Workers AI model id. It returns a LanguageModel layer directly — the binding handles auth and the gateway URL, so you pass neither an API key nor a Layer.unwrap:

const aiGateway = yield* Cloudflare.AiGateway.bind(Gateway);


const languageModel = aiGateway.model({
  client: aiGateway,
  model: "@cf/meta/llama-3.1-8b-instruct",
  parameters: { temperature: 0.7, maxTokens: 1024 },
});

You then provide that layer to your handler, and provide Cloudflare.AiGatewayBindingLive once at the bottom of the Init chain so the binding resolves at runtime:

export default Cloudflare.Worker(
  "Worker",
  { main: import.meta.filename },
  Effect.gen(function* () {
    const aiGateway = yield* Cloudflare.AiGateway.bind(Gateway);
    const languageModel = aiGateway.model({
      client: aiGateway,
      model: "@cf/meta/llama-3.1-8b-instruct",
    });


    return {
      fetch: Effect.gen(function* () {
        const response = yield* LanguageModel.generateText({
          prompt: "Say hello.",
        }).pipe(Effect.orDie);
        return yield* HttpServerResponse.json({ text: response.text });
      }).pipe(Effect.provide(languageModel)),
    };
  }).pipe(Effect.provide(Cloudflare.AiGatewayBindingLive)),
);

Under the hood makeLanguageModel translates Effect AI requests into Workers AI ai.run(...) calls — SSE parsing for streaming, schema-typed usage reconstruction, tool calls, and structured output all routed through the gateway, so you also get its caching, rate limiting, retries, and unified request log for free.

The Effect AI guide covers the provider pattern (reading keys with Config.redacted, chat persistence), and the AI Gateway tutorial walks the gateway end to end including streaming. (#389)

Implicit Worker and Durable Object RPC can return a Stream

Section titled “Implicit Worker and Durable Object RPC can return a Stream”

An Implicit Worker or Durable Object RPC method can now return a Stream directly — no Effect<Stream> wrapper. Value methods stay Effects; the client stub reads each naturally:

A Durable Object exposing both kinds of method — get returns a value, tick returns a Stream:

src/counter.ts
export class Counter extends Cloudflare.DurableObjectNamespace<Counter>()(
  "Counter",
  Effect.gen(function* () {
    return Effect.gen(function* () {
      const state = yield* Cloudflare.DurableObjectState;
      return {
        get: () => state.storage.get<number>("n"), // Effect → value
        tick: (n: number) =>
          Effect.succeed(Stream.iterate(0, (i) => i + 1).pipe(Stream.take(n))),
          Stream.iterate(0, (i) => i + 1).pipe(Stream.take(n)), // Stream
      };
    });
  }),
) {}

A Worker binds the namespace and pipes the tick Stream straight onto the HTTP response:

src/worker.ts
export default class Worker extends Cloudflare.Worker<Worker>()(
  "Worker",
  { main: import.meta.filename },
  Effect.gen(function* () {
    const counters = yield* Counter;
    return {
      fetch: Effect.gen(function* () {
        const stream = counters
          .getByName("default")
          .tick(5)
          .pipe(Stream.map((i) => `${i}\n`), Stream.encodeText);
        return HttpServerResponse.stream(stream);
      }),
    };
  }),
) {}

Previously a returned Stream was fed into the Effect runtime and crashed with Fiber.runLoop: Not a valid effect; the fix lifts it into the success channel in both WorkerBridge and DurableObjectBridge.

(#478)

Also in this release

Section titled “Also in this release”
  • login no longer evaluates your stack — bun alchemy login used to eval the stack (which could fail on a missing Config or build the state store before auth was configured, bricking the flow). Providers/state are now read statically and state-store init is deferred until actually used (#492).
  • props.env works in local Workers — Config values declared via env weren’t being wired into the local Worker provider under alchemy dev. Thanks John Royal (#473).

Where to go next

Section titled “Where to go next”