OpenAI & Groq compatible

Stop paying your LLM twice for the same question.

Semantic Cache Gateway sits in front of your model calls and returns a cached answer when someone asks something close enough in meaning — not just exact text. One URL swap, up to 70% lower inference cost.

Request an API key Try the live demo

client.py

# only the base_url changes. BYOK keeps your key on your side.
from openai import OpenAI

client = OpenAI(
  base_url="https://semantic-cache-6qvc.onrender.com/v1",
  api_key="YOUR_CACHE_GATEWAY_KEY",
  default_headers={"X-Upstream-Key": "YOUR_OPENAI_OR_GROQ_KEY"},
)

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{"role": "user", "content": "What's the capital of France?"}],
)

Three steps, no new SDK

You keep your existing OpenAI or Groq client code. We change what happens to the request before it reaches the model.

Point your client at us

Swap the base URL to ours and add your upstream key in a header. Everything else — model names, message format, streaming — stays exactly the same.

We cache by meaning

Each prompt is embedded and compared against recent queries. Close enough in meaning, not just matching text, counts as a hit.

You pay less

Cache hits skip the model call entirely. Misses pass straight through to your provider using your own key, unmodified.

Why you bring your own key

We route requests. We never become the thing standing between you and your model bill.

You stay billed directly

Cache misses hit OpenAI or Groq on your account. No markup, no reselling tokens, no surprise invoice from us.

Nothing to migrate off

Remove us tomorrow by changing one line back. Your account, quota, and rate limits were never touched.

Your terms with the provider

Data retention, fine-tuning opt-outs, enterprise agreements — whatever you negotiated with OpenAI still applies.

No cold-start trust problem

We're a routing layer, not a model reseller. There's less to take on faith before you send your first request.

Try it before you sign up

Use the public demo key demo_key with your own Groq key. Send the same request twice — the second is served from cache, at zero cost.

curl https://semantic-cache-6qvc.onrender.com/v1/chat/completions \
  -H "Authorization: Bearer demo_key" \
  -H "X-Upstream-Key: YOUR_GROQ_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "What is 2+2?"}]
  }'

First call: MISS (real LLM) Second call: HIT (cache) Cost on hit: $0.00 Latency on hit: <50ms

What's actually running under the URL

A gateway, an embedding model, and a vector store, wired together so you don't have to.

Semantic matching

Queries are compared by meaning using sentence embeddings, so paraphrased questions still hit the cache.

Multi-provider

Works with OpenAI and Groq today, using the same request format your client already sends.

Persistent storage

Per-tenant usage

Every API key tracks its own request count, hit rate, and spend independently of everyone else's traffic.

Stats endpoint

Pull hit rate, request counts, and estimated savings as JSON via GET /stats.

OpenAI-compatible

Same request and response shape as the OpenAI SDK, so existing tooling and libraries keep working.

Pricing that scales with requests, not seats

Every plan includes the full cache engine. You're paying for request volume and retention window, nothing else.

Free

For testing the gateway against a real project.

1,000 requests / month
Semantic cache, all providers
7-day cache retention
Community support

Get free access

Starter

$19/mo

For a side project that's starting to see real traffic.

50,000 requests / month
30-day cache retention
Usage stats endpoint
Email support

Start on Starter