One API, six LLM providers, 1,200+ automated checks
API, LLM response and desktop testing of a gateway that routes requests to several model providers.
1,200+ automated API checks- 1,200+ automated API checks
- 6 model providers covered
- 300 regression prompts on every release
- PlatformsAPI, web console, desktop app
- Engagement7 months · 3 QA engineers
A gateway that routes requests to several LLM providers through one API was tested against only one or two of them. Streaming errors and wrong budget counting were found by customers, and nobody could tell if a release made model answers worse.
We tested the inference API against each provider's format, including streaming, errors, rate limits and budgets. We built regression prompt sets to evaluate LLM responses on every release, ran prompt-injection checks, and covered the desktop app on macOS, Windows and Linux.
Every release is now checked against all providers with 1,200+ automated API checks and 300 regression prompts. Differences between providers are caught before customers see them.
- Weeks 1–3
API contract review, provider matrix
- Weeks 4–12
Automated API checks, streaming and limits
- Weeks 13–20
LLM evaluation sets, prompt-injection testing
- Weeks 21–28
Desktop app, bots, load testing
Tools
- Postman
- Newman
-
Playwright - Python
- pytest
- k6
- Datadog
- Rollbar
- Grafana
- Prometheus
- OpenTelemetry
-
Langfuse
Testing types
Want results like these?
Book a short call about your product. We can start with a free QA audit or a trial.