Services
Semantic caching, prompt compression and model routing, with an eval score on every change so quality never quietly drops. Most of the savings show up within weeks.
Week-long cost audits start at $3,500 (same price as a Production Audit because the work is structurally similar). Full rollouts with caching, routing, prompt compression, and dashboards scale to $15K. On engagements over $10K I'll offer a savings-indexed fee: if I don't hit a pre-agreed % reduction, you only pay the floor.
Instrument the app, pull 7 days of traces, produce a 1-page report: top spend hotspots, expected savings per intervention, estimated engineering cost.
Semantic caching and prompt compression — almost always the biggest unlock. Ship behind a feature flag.
Classifier routes easy queries to Haiku / 4o-mini, hard queries to Sonnet / 4o. Eval-gated rollout.
Dashboards, runbook, monitoring. Deliver the final savings report.
LLM cost optimization is a precise discipline, not a heuristic. My standard three-layer stack: (1) pgvector-backed semantic caching with ~31% hit rate and sub-50ms latency, (2) prompt compression that prunes average prompt length by ~38% using lazy context loading, (3) intent-aware model routing that redirects ~62% of queries to cheaper models (Haiku, 4o-mini) while reserving expensive models for complex reasoning. Teams that implement systematic caching + multi-model routing cut average monthly API spend ~65% without dropping CSAT — but only if it's eval-gated. Without evals, you ship silent quality drops.
Real questions from scoping calls. If yours isn't here, ask it on a free 30-minute call.
Usually three things: the most expensive model answers questions a cheaper one could handle, the same questions are paid for again and again with no cache, and prompts carry far more context than they need. OpenAI cost optimization and Anthropic bills follow the same pattern. A one-week audit shows exactly where your LLM costs go, endpoint by endpoint.
Most teams I audit are overpaying by 2–5×. Typical outcomes: 30% from semantic caching alone, another 15–25% from prompt compression and model routing combined. 40–70% total reduction is normal on support and RAG workloads; less on pure generation workloads.
One week. You grant read access to tracing (LangSmith, Helicone, or raw API logs), I instrument what's missing, pull 7 days of data, and deliver a report with prioritized interventions and expected savings per intervention.
Not necessarily. Semantic caching and prompt compression work without any model change. Model routing is optional — I only recommend it if the eval score holds up on the cheaper model for the easy-query bucket.
Works the same. The optimization layer (caching, routing, compression) sits above the provider, so savings carry whether you're on OpenAI direct, Anthropic direct, Bedrock, or Azure.
Three: (1) caching requires you to accept some semantic fuzziness on repeat queries, (2) model routing requires a strong eval harness or quality will silently drop, (3) savings decay if your team keeps adding new prompts without re-running the audit. I deliver a runbook so your team can maintain it.