resources

LLM Inference

Determinism

• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference - https://arxiv.org/abs/2506.09501
• vLLM Batch Invariance - https://docs.vllm.ai/en/stable/features/batch_invariance/
• Defeating Nondeterminism in LLM Inference (Thinking Machines) - https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Libraries

vLLM

https://github.com/vllm-project/vllm

https://blog.infocruncher.com/2025/02/27/llm-serving-with-vllm-on-vertexai/

https://www.aleksagordic.com/blog/vllm

SGLang

https://github.com/sgl-project/sglang

TensorRT-LLM by NVIDIA

https://github.com/NVIDIA/TensorRT-LLM

Text Generation Inference by HuggingFace

https://github.com/huggingface/text-generation-inference

DeepSpeed by Microsoft

https://github.com/deepspeedai/DeepSpeed

llama.cpp

https://github.com/ggml-org/llama.cpp