All figures on this site were measured on NVIDIA H100 with vLLM 0.23, against a baseline with the engine's prefix and KV caching enabled. Latencies are medians of five runs; output identity is verified byte-for-byte against the unmodified engine.
The exact-reuse path returns in 0.8 ms (0.62 ms at scale) without an engine call. Verified acceleration on similar requests measured at 13.8× on Mistral-7B, 9.2× on Qwen2.5-7B and 3.6–4.9× on Llama-3.1-8B, with zero code changes between model families; the multiple depends on how far into the answer the similarity holds. Non-repeating traffic carries approximately 1% overhead.
Every reused answer is confirmed by the serving engine before it is returned. Across three model families, 100% of verified answers were byte-identical to the unmodified engine's output. A 200-request mixed-traffic soak completed with zero failures, and identity held in full under 16-way concurrent load.
The memory holds under pressure. After 574,000 tokens of unrelated traffic, AIe's exact-reuse latency was unchanged at 0.62 ms, while the engine's prefix cache had evicted the entry (60 ms) and disk-offloaded KV recovered partially (21.3 ms). Following a full process restart, 89.5% of the memory answered warm.
The memory answers in about a millisecond with more than a million entries stored — on commodity hardware, with capacity bounded by storage rather than GPU memory. That is the difference in kind: a serving engine's cache holds what fits beside the model in GPU memory and evicts in minutes under load; the fabric is designed to hold months of traffic, and grows with use. On traffic composition: a 50,000-request public corpus of raw consumer chat showed a 15.1% exact-duplicate rate before any similar-request counting; per-deployment rates are measured by the profiler on the customer's own logs.
| Validated | In progress |
|---|---|
| Open-weight dense models (Mistral-7B, Qwen2.5-7B, Llama-3.1-8B) · vLLM 0.23 · NVIDIA H100 · single-node deployment · exact and similar-request reuse | Frontier-scale and mixture-of-experts models · multi-node shared memory · professional-domain reuse profiling · partial reuse within dissimilar requests |
Validation status is updated as results complete, each with its date and method.
The evaluation pack includes the reproducible harness, the methodology, and shadow-mode deployment on your own traffic — so the numbers you present internally are measured on your workload.