TokenPak + LiteLLM Adapter¶

LiteLLM is a tested SDK adapter; TokenPak records usage and estimated cost.

Why Use LiteLLM + TokenPak?¶

Usage records: TokenPak records usage and estimated cost for requests routed through it.
Default behavior: The default proxy preserves conversation turns, so a forwarded request can report zero tokens saved. Explicit context tools can reduce eligible content.

Setup¶

1. Install TokenPak and LiteLLM¶

pip install litellm tokenpak

# TokenPak runs as a standalone proxy service
# See: https://github.com/tokenpak/tokenpak

2. Start TokenPak Proxy¶

# TokenPak default: http://localhost:8766/v1
tokenpak serve --port 8766

Verify proxy is running:

curl http://localhost:8766/health
# { "status": "ok" }

Usage Patterns¶

Pattern 1: Direct `litellm.completion()`¶

Route a single completion request through TokenPak:

import litellm

# Set your Anthropic API key
import os
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."

response = litellm.completion(
    model="openai/claude-sonnet-4-6",  # "openai/" prefix = OpenAI-compatible endpoint
    api_base="http://localhost:8766/v1",  # Point to TokenPak proxy
    messages=[
        {"role": "user", "content": "What is quantum computing?"}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)

Key points: - Use openai/<model-name> format: LiteLLM routes to the OpenAI-compatible endpoint - api_base points to TokenPak proxy (default: localhost:8766/v1) - TokenPak records requests and handles token accounting

Pattern 2: LiteLLM Proxy Mode with TokenPak Backend¶

Run LiteLLM's own proxy to manage multiple clients, all routing through TokenPak:

# litellm_config.yaml
model_list:
  - model_name: "claude-sonnet"
    litellm_params:
      model: "openai/claude-sonnet-4-6"
      api_base: "http://localhost:8766/v1"
      api_key: "sk-ant-..."  # Or set via env var: ANTHROPIC_API_KEY

  - model_name: "claude-opus"
    litellm_params:
      model: "openai/claude-opus-4-8"
      api_base: "http://localhost:8766/v1"
      api_key: "sk-ant-..."

router_settings:
  fallback_ratio: 0.1  # Fallback after 10% failure rate

Start LiteLLM proxy:

litellm --config litellm_config.yaml --port 8000

Use from your application:

import litellm

response = litellm.completion(
    model="claude-sonnet",  # Routes to LiteLLM proxy
    api_base="http://localhost:8000",
    messages=[{"role": "user", "content": "Hello!"}]
)

Flow:

Your App → LiteLLM Proxy (8000) → TokenPak Proxy (8766) → Anthropic API

Pattern 3: LiteLLM Fallback with TokenPak¶

Use LiteLLM's own fallback settings with TokenPak recording the requests it receives:

# litellm_config.yaml
model_list:
  - model_name: "smart-router"
    litellm_params:
      model: "openai/claude-opus-4-8"
      api_base: "http://localhost:8766/v1"  # Primary: TokenPak → Claude
      api_key: "sk-ant-..."

  - model_name: "smart-router"
    litellm_params:
      model: "openai/gpt-4-turbo"
      api_base: "http://localhost:8766/v1"  # Fallback: TokenPak → GPT-4
      api_key: "sk-openai-..."

router_settings:
  fallback_ratio: 0.2  # Fallback after 20% failure rate
  allowed_fails: 1  # Allow 1 request to fail before fallback kicks in

How it works: 1. LiteLLM handles routing and fallback according to its own configuration 2. TokenPak records the requests it receives on each path

Model Name Convention¶

TokenPak uses the openai/<model-name> pattern for all providers. This tells LiteLLM to treat the endpoint as OpenAI-compatible.

Provider	Model Name
Anthropic	`openai/claude-opus-4-8`, `openai/claude-sonnet-4-6`
OpenAI	`openai/gpt-4-turbo`, `openai/gpt-4o`
Google	`openai/gemini-2.0-flash`

See TokenPak Model Support for the full list.

Verify Routing¶

Check that requests are flowing through TokenPak correctly:

# Per-session stats (counters + cache attribution by origin)
curl http://localhost:8766/stats | jq .

# Cache-hit metrics
curl http://localhost:8766/cache-stats | jq .

# The cache-stats endpoint reports keys such as:
{
  "total_requests": 45,
  "cache_hits": 12,
  "hit_rate": 0.27
}

The exact fields depend on your TokenPak version; run the commands above to see the current shape. If your request doesn't appear in stats, check: 1. TokenPak is running (curl http://localhost:8766/health) 2. api_base in LiteLLM config matches TokenPak port (default 8766) 3. Firewall/network allows localhost:8766 connection

Limitations & Known Gaps¶

LiteLLM Specific¶

Limitation	Impact	Workaround
Streaming	LiteLLM streaming via TokenPak works, but caching doesn't apply to streamed responses	Cache only unstreamed requests
Custom headers	LiteLLM passes headers through; TokenPak ignores non-standard headers	Use `api_key` + `api_base` only
Async routing	LiteLLM's async API works with TokenPak, but fallback logic runs serially	Acceptable for most use cases

TokenPak Specific¶

Limitation	Impact	Workaround
Cache key	Caching uses model + prompt hash (ignores temperature, top_p)	Ideal for deterministic requests; less useful for creative tasks
Rate limits	TokenPak respects upstream rate limits; LiteLLM's retry logic stacks on top	Configure reasonable `max_retries` in LiteLLM config
Request size	Max request size is 10 MB	Unlikely to hit in practice

Troubleshooting¶

"Connection refused" on `api_base`¶

Error:

litellm.exceptions.APIConnectionError: Failed to connect to http://localhost:8766/v1

Fix: 1. Verify TokenPak is running: curl http://localhost:8766/health 2. Check port is correct (default 8766) 3. If remote: use actual IP instead of localhost (e.g., http://192.168.1.100:8766/v1)

"Invalid API key" errors¶

Error:

litellm.exceptions.AuthenticationError: Invalid API key

Fix: 1. Verify ANTHROPIC_API_KEY is set correctly 2. TokenPak passes your API key upstream — check it matches your provider (Anthropic, OpenAI, etc.) 3. Test directly against TokenPak's health endpoint: curl http://localhost:8766/health

Requests not cached¶

Symptom: cache-stats shows cache_hits: 0 even after repeated identical requests

Cause: Likely mismatch in request parameters (temperature, top_p, etc.)

Fix: 1. Cache key = model + prompt hash. Other parameters are ignored. 2. Check your requests are truly identical:

# These will cache:
msg = [{"role": "user", "content": "What is 2+2?"}]
r1 = litellm.completion(model="openai/claude-sonnet-4-6", messages=msg, api_base=proxy)
r2 = litellm.completion(model="openai/claude-sonnet-4-6", messages=msg, api_base=proxy)

# These won't cache (different parameters):
r1 = litellm.completion(model="openai/claude-sonnet-4-6", messages=msg, temperature=0.7, api_base=proxy)
r2 = litellm.completion(model="openai/claude-sonnet-4-6", messages=msg, temperature=0.5, api_base=proxy)

Next Steps¶

OpenAI SDK Adapter — Use TokenPak with openai library
LangChain Adapter — Integrate with LangChain
TokenPak Configuration — Advanced proxy settings
Model Support — Full provider & model reference

TokenPak + LiteLLM Adapter¶

Why Use LiteLLM + TokenPak?¶

Setup¶

1. Install TokenPak and LiteLLM¶

2. Start TokenPak Proxy¶

Usage Patterns¶

Pattern 1: Direct litellm.completion()¶

Pattern 2: LiteLLM Proxy Mode with TokenPak Backend¶

Pattern 3: LiteLLM Fallback with TokenPak¶

Model Name Convention¶

Verify Routing¶

Limitations & Known Gaps¶

LiteLLM Specific¶

TokenPak Specific¶

Troubleshooting¶

"Connection refused" on api_base¶

"Invalid API key" errors¶

Requests not cached¶

Next Steps¶

Pattern 1: Direct `litellm.completion()`¶

"Connection refused" on `api_base`¶