Model APIs and Runtime

The request lifecycle around model calls: messages, streaming, tool loops, batching, caching, rate limits, and provider boundaries.

The Model API Is a Runtime Boundary

The model API is where your deterministic application hands a context package to a probabilistic service and receives content, tool requests, usage, and stop metadata.

For Claude, the core interface is the Messages API. Other providers expose similar concepts with different request fields. Your architecture should preserve the common lifecycle while isolating provider-specific features.

Messages and Stateless Calls

Most model endpoints are stateless: the provider does not infer your application's conversation unless you send the required prior messages or a summarized state. The application decides what history to retain and resend.

request = {
  "model": selected_model,
  "system": stable_instructions,
  "messages": assembled_history,
  "tools": eligible_tools,
  "max_tokens": output_budget
}

Streaming

Streaming improves perceived latency by delivering events or content increments before the full response is complete. The client must handle partial content, disconnects, tool-use events, final usage, and a response that ends before the UI has rendered a complete artifact.

Do not execute a partially streamed tool request. Wait until the tool call is complete and validated.

The Tool-Use Loop and Stop Reasons

The runtime should inspect why generation stopped: completed response, tool request, output limit, refusal, or another provider-specific condition. A tool request begins another controlled iteration:

1. Receive complete tool request.
2. Validate and authorize.
3. Execute or reject.
4. Append a tool result.
5. Call the model again.
6. Stop on completion, budget, or escalation condition.

Batching, Caching, and Rate Limits

Batch APIs are appropriate for asynchronous workloads such as offline evaluation, document enrichment, or large classification jobs. Prompt caching can reduce repeated processing of stable prefixes. Rate limits require queueing, backoff, concurrency control, and request prioritization.

Direct API vs. Managed Cloud Runtime

Claude can be accessed through Anthropic's platform and through managed cloud services such as Amazon Bedrock. Managed platforms can integrate with cloud IAM, networking, billing, and regional controls, while API surfaces and feature timing may differ. Keep these deployment choices out of domain logic.