Polymorphic inference
Polymorphic Inference: Tuned for your model, your traffic, your chips
Your models and traffic workloads are dynamic. So is your inference deployment. Polymorphic inference applies our research to optimize your deployment to your unique traffic patterns overnight.
Inference moved from static to polymorphic
Polymorphic inference adapts automatically to your inference workloads, even as your models, traffic, or hardware deployment targets change. Hand-optimizing inference workloads takes top researchers days or weeks, but polymorphic inference can get it done overnight.
- Static inference
Built for one model, one chip, and one kind of traffic.
Configurations tuned for one setupKernels for a fixed traffic shapeDeployments tuned by handWeeks of expert time per deploymentOnly the deployments researchers have time for get tuned - Polymorphic inference
What we are building toward, starting with Together Compiler.
Agents tune the deployment overnightFaster than hand-optimized deploymentsImmediately adapts to new hardwareYour deployment adapts when your traffic changesRuns in production today
Together Compiler, live today
Together Compiler is the first part of polymorphic inference in production. A swarm of agents proposes kernel changes, and a persistent engine tests each one for speed and for correctness against the stock engine. Only verified changes reach production. That is what enables continual compilation: when the model, the hardware, or the workload changes, agents can run the optimization again instead of waiting for a researcher.
Optimize the whole forward pass
Whole forward passVendor kernels includedGPU agnosticAgents optimize the kernel development and research process. Agents benchmark the engine forward pass and rebuild the engine for your specific deployment.
Run parallel experiments
Parallel agentsSeconds per trialSpeed and correctness testedSwarms of agents work together to find diverse optimization strategies.
Hot-reload a persistent engine
Start onceReload in secondsTwo measurement tiersOptimizations are benchmarked on a Tiered Persistent Engine with hot-reloading. Together Compiler starts one engine and then hot-reloads kernels and PyTorch code as agents optimize the forward pass. The Persistent Engine allows rapid iteration and testing.
Build a shared knowledge base
Champion RegistryReasoning loggedReused across modelsWinning kernels, and the reasoning behind them, go into a shared registry. The next model or chip starts from what earlier runs learned, not from scratch.
Accelerating across models and chips
Together Compiler improves throughput and latency across top open models, each optimized in less than a day, compared to the best open engines.
NVIDIA Blackwell GPUs
AMD and Trainium
vs megakernels- Single-user speed
Throughput vs interactivity
- Together Compiler
- Best open engine
Tok/s per GPU ↑DeepSeek V4.1 FlashSGLang · B200400K300K200K100K00200400600800GLM 5.3SGLang · B20030K20K10K00100200300400500Kimi K3vLLM · B30020K15K10K5K00100200300MiniMax M3.1 FlashvLLM · B200100K80K60K40K20K00200400600800P90 TPS per user →4 models · NVIDIA Blackwell
Up to 2.6x tokens/sec
Further right means faster per user (p90), higher means more tokens per GPU. At the same speed per user, the compiled engine serves up to 2.6x the tokens per GPU, after a 24-hour compile using just two nodes per model.
Read the paperThroughput vs interactivity
- Together Compiler
- Best available benchmark
Tok/s per accelerator ↑GLM 5.3vLLM · MI355X15K10K5K00100200300400MiniMax M3vLLM Neuron · Trainium215K10K5K0020406080100P90 TPS per user →AMD and AWS
Up to 3.7x tokens/sec
At the same p90 speed per user, GLM on AMD MI355X serves up to 3.7x the tokens per GPU. MiniMax M3 on AWS Trainium2 serves up to 2.7x the tokens per chip.
Read the paperThroughput (output tok/s)
- Together Compiler
- Optimized baseline
Output tok/s ↑Llama-3.2-1B · B200 · batch 1
Up to 2.5x tokens/sec
Faster than an expert-written megakernel and machine learning compilers at every output length tested on H100 and B200. The gap widens at long outputs, where one fixed schedule stops fitting.
Read the paperSingle-user speed (p90 TPS per user)
- Together Compiler
- InferenceX best
P90 TPS per user ↑4 models · B200, GB300 and MI355X
Up to 1.8x faster per user
Together Compiler beats the best InferenceX result on all six settings, by up to 1.8x on MiniMax M3.1 Flash.
Read the paper
Built on research the industry runs on
Our researchers built FlashAttention, ThunderKittens, and ThunderAgent. Together Compiler is how that research gets applied to the models we serve.
Try polymorphic inference today
Our inference platform is designed for polymorphic inference, starting with supported serverless models.
Together Compiler is on by default for our most-used serverless models. Nothing to turn on.
Polymorphic inference
Reserved token capacity with SLA guarantees, optimized by Together Compiler.
Polymorphic inference
Ask our researchers and FDEs to optimize your deployment with Together Compiler.
Polymorphic inference
Run your own engine and model, optimized for performance by Together Compiler.
Polymorphic inference
What our customers are seeing
“Running Together Compiler cut our median time to first speech by nearly 50%, and it happened overnight. Getting Phonic-1 optimized that quickly let us ship sooner. We’ve been a customer of Together AI for over two years and are excited to keep building with them as they find new ways to push performance.”
FAQ
Can't find the answer you need? Talk to an expert.
What is polymorphic inference?
Inference in many shapes. New models arrive every few weeks on different architectures, while traffic and hardware keep changing. Static inference tunes one configuration for one setup. Polymorphic inference tunes each deployment for the model, hardware, and workload it actually runs, with AI agents doing work that used to take researchers days or weeks. Together Compiler is the first part of it in production.
What is Together Compiler?
Agents search for faster GPU kernels across a model's whole forward pass, and a persistent engine tests each candidate quickly. It is how Together enables continual compilation: when the model, hardware, or workload changes, the optimization can run again.
Do I need to change anything?
No. Compiled kernels are on by default for supported serverless models, at the same price. There is no flag to set.
How much faster is it?
On NVIDIA Blackwell GPUs, Together Compiler reaches 1.3 to 3x higher throughput than the best open engines in our benchmarks, after under 24 hours of compilation. Gains hold on AMD MI355X and AWS Trainium2 too.
Which models and hardware are supported?
On serverless today: GLM 5.3, GLM 5.3 Flash, and DeepSeek V4.1 Flash, with more to follow. We can also run it against your model. On Dedicated Model Inference or Dedicated Container Inference, Together Compiler can be applied to any endpoint on any GPU in our fleet.
Can I get it on a dedicated deployment?
Yes. Dedicated Model Inference and Dedicated Container Inference customers can ask our research team and FDEs to optimize their deployment with Together Compiler. Self-serve optimization is coming.