AI providers
What it is
When a product uses AI (chat, summaries, transcription, voice), the backend calls a provider's API (for example Anthropic, OpenAI or Google) with an API key. Keys belong to a project or workspace at the provider, which has its own spend limit, model access and data settings.
Why we do it carefully
Our AI incidents were all configuration, not code: a spend cap reached and mistaken for a bug, a new key that could not use the model we needed, and only one of three keys ever tested.
Before you start
- Which provider and models, agreed with the client, and who owns the account per
OWNERSHIP.md. - A data processing agreement with the provider, and the client's decision on data retention.
- The consent design for AI features. See AI processing.
Steps
- One provider project or workspace per environment:
acme-dev,acme-staging,acme-prod. Each has its own key. - Set a monthly spend limit and an alert at 80% in each project. Write the limit in
OWNERSHIP.md; when it is hit, calls fail in a way that looks like an outage. - Check model access per key. Providers retire models and new keys sometimes cannot use older ones. Call the exact model id from each environment once.
- Data settings: turn off use of our data for training where the provider offers it, and choose the shortest retention allowed.
- Store keys in SSM:
/acme/<env>/api/AI_PROVIDER_API_KEY. - In code: a timeout on every call, retries with backoff for rate limits, and a clear error when the provider refuses (quota, model not found), logged with the provider's message.
Check it worked
From each environment, call a small health route that makes one cheap request with the production model id and reports the model, latency and success. Run it for every key you store, not only the main one.
If it goes wrong
| You see | It means |
|---|---|
| Sudden errors on every AI call in one environment | Spend limit reached; check the provider console before the code |
| "Model not found" with a new key | The project lacks access to that model, or the model was retired |
| Rate-limit errors in prod only | Prod traffic needs a higher tier; request it before launch |
Rules
Users are told, in the app, what is sent to which AI provider, and agree before the first call. A privacy policy alone was not enough for App Review.
Done when
- A project and key per environment, in SSM.
- Spend limit and 80% alert per project.
- Model access checked for every key.
- Training opt-out and retention set; DPA signed.
- Health check passes in every environment.