Multi-Model LLM Inference on EKS, Across Three Generations of Silicon

The three iterations at a glance

The requirement was the same each time: self-host large models for Nexus on spot capacity, cheaply, with no lock-in. What changed each iteration was the silicon. Here are the three side by side before the deep dive.

Generation 1: Llama 3 70B on AWS Inferentia2

I started on Inferentia2 because inf2 spot capacity costs a fraction of GPU instances and had far better spot availability, which mattered for a spot-first platform. I served Llama 3 70B Instruct with a RayServe + vLLM backend (PagedAttention and dynamic batching) orchestrated by the KubeRay operator with RayServe autoscaling, on a pool of 6–8× inf2.48xlarge, sized for 2,000+ users at roughly 30% concurrency.

Benchmarked with LLMPerf at 160 concurrent requests, it held ~7 tokens/sec per user and 700–1,300 tokens/sec aggregate.

• The hard problem: a spot interruption forced a pod restart, and downloading plus Neuron-compiling the model could take up to an hour. I fixed it by pre-staging compiled models in S3 over AWS's internal network, collapsing cold-start to minutes.
• The honest limitation: I targeted 10+ tok/s per user and maxed around 8 (5–6 during and just after disruptions). Reaching the target meant scaling to 12–18 nodes, which was feasible but cost-constrained, so I iterated on the hardware instead.

Generation 2: Llama 4 Maverick (MoE) on Trainium2

Llama 4 Maverick is a 17B-active mixture-of-experts model with ~400B total parameters, so it needs large HBM. Trainium2 fit perfectly: NeuronX Distributed Inference exploits all 16 Trn2 accelerators with tensor and expert parallelism, and a single trn2.48xlarge (1.5 TiB) holds Maverick (~800 GiB) in native BF16 with no quantization.

I served it with vLLM + optimum-neuron on trn2.48xlarge spot, persisting the compilation cache in S3 to avoid the ~20-minute compile on every cold start.

• At 160 concurrent: ~16 tok/s per user and 2,100–3,700 tok/s aggregate, more than 2× Generation 1.
• Pushed to 200 concurrent, aggregate held steady and per-user settled around 12 tok/s.
• I also trialed Llama 4 Scout via Kubernetes LeaderWorkerSets, but it couldn't match Maverick's numbers, so I didn't ship it.

Generation 3: GPT-OSS 20B on NVIDIA L40S, behind an Envoy AI Gateway

To add a fast, cheap general-purpose model I brought up GPT-OSS 20B (a 3.6B-active MoE) on NVIDIA L40S via AWS g6e. It isn't Neuron-optimized, so I ran the NVIDIA device plugin with vLLM and let Karpenter pick g6e.12xlarge/24xlarge by spot price and availability, pre-staged in S3 like everything else. Each node holds a full copy with tensor_parallel_size=4, scaled across 6–8 replicas.

At 200 concurrent on 8 nodes it reached 3,600–5,200 tok/s aggregate and 18–26 tok/s per user, the highest raw throughput of the three.

• The Envoy AI Gateway became the front door: multi-model routing (GPT-OSS for ~80% of traffic, Llama 4 Maverick for the rest) behind OpenAI-compatible APIs, so applications needed zero changes.
• It also gave me automatic failover on spot disruption, rate-limiting via the Kubernetes Gateway API and Envoy AI Gateway CRDs, and a single place for cost control and observability.

# One vLLM process per g6e node, four L40S GPUs each; 6-8 replicas behind the gateway.
vllm serve openai/gpt-oss-20b \
  --tensor-parallel-size 4 \
  --served-model-name gpt-oss-20b

The numbers, measured not quoted

Throughput is the whole game, so I measured it under real concurrency with LLMPerf at every step rather than trusting a spec sheet. Aggregate output more than quadrupled from Generation 1 to Generation 3 on the same 6-8 node budget. Toggle between per-user and aggregate:

The final architecture: many models, one gateway

No single model served every job, so the final design runs two. The Envoy AI Gateway sits in front of both as one OpenAI-compatible endpoint, routing roughly 80% of traffic to the fast GPT-OSS cluster and 20% to Llama 4 Maverick, with rate-limiting and automatic failover if a spot-backed cluster wobbles. Because it speaks the OpenAI API, no application had to change.

# Route by model name to the right backend; both are OpenAI-compatible.
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: llm-routing
spec:
  schema: { name: OpenAI }
  rules:
    - matches: [{ headers: [{ name: x-ai-eg-model, value: gpt-oss-20b }] }]
      backendRefs: [{ name: gpt-oss-g6e }]         # ~80% of traffic
    - matches: [{ headers: [{ name: x-ai-eg-model, value: llama-4-maverick }] }]
      backendRefs: [{ name: llama4-trn2 }]         # ~20% of traffic

The stack behind it

I was an early tester of the AI-on-EKS initiative, and the platform leans on a lot of it. These are the pieces that turn spot-priced accelerators into a real serving platform:

What it powers, and the through-line

This platform serves a deliberately diverse set of AI workloads: executive insights and AI summaries, customer-facing chat, agentic incident triage, investigation and remediation, CopilotKit-driven dynamic dashboards, and autonomous pentest agents. Langfuse gives me TPS, latency, cost, and failure visibility across all of it.

The through-line across all three generations: spot-first economics, no vendor lock-in, and a refusal to trust a benchmark I hadn't run myself.