Guided decoding with xgrammar

[windreamer]

Motivation

LMDeploy’s TurboMind backend is the fastest inference stack in the ecosystem, yet it still lacks Guided Decoding – a feature that is already available in the PyTorch backend and heavily requested by the community.
This PR closes the gap by bringing token-level, C++ native Guided Decoding to TurboMind while keeping the API 100 % compatible with the existing PyTorch backend.
The implementation is built on xGrammar (Apache-2.0), a high-performance C++ library that compiles JSON / Choice / Regex grammars into token FSMs and applies them with negligible overhead.

Modification

1. Build-system

   • Add xgrammar as a header-only dependency via CMake FetchContent (CUDA & Python bindings disabled).
   • Export xgrammar::tokenizer_info and xgrammar::grammar_compiler symbols under lmdeploy::xgrammar.

2. Core C++ changes

   • DynamicDecodeLayer pipeline extended with two new layers:
     • GuidedDecodeMaskLayer: in setup() compiles / reuses grammar → builds per-request token bitmask; in forward() launches a light CUDA kernel to mask disallowed logits to -INF.
     • GuidedDecodeUpdateLayer: in forward() calls matcher->AcceptToken(output_id) to advance the FSM.
   • Grammar compiler cache (LRU, keyed by schema hash) shared across all sessions to avoid re-compilation.

3. Python frontend

   • Re-use existing guided_decoding utilities from PyTorch backend; no new API surface.
   • turbo.TurboMindEngine now accepts the same response_format= / guided_json= / guided_choice= arguments.

Checklist

• Pre-commit hooks (clang-format, flake8, mypy) passed.
• Document updated

[shell-nlp]

good job!

[windreamer]

Also done replace outdated outlines with xgrammar in PyTorch Engine. And this enabled us to:

• No longer need to restrict numpy<2
• No buggy pyairports packages as dependencies

[lvhan028]

Does xgrammar support npu?

[windreamer]

No，but currently XGrammar in PyTorch engine uses cpu kernels instead of gpu ones. And it seems also works in arm64 (untested)

[grimoire]

I don't know much about guided decoding. But I think there are bugs in the pytorch implementation (from the main branch)

The matcher is maintained in instances of RegexLogitsProcessor or JSONLogitsProcessor. And _get_guided_logits_processor will only cache 32 instances. Different request with same guide would get the same processor and old processor would be removed if more than 32 guide request comes in.

[windreamer]

I don't know much about guided decoding. But I think there are bugs in the pytorch implementation (from the main branch)

The matcher is maintained in instances of RegexLogitsProcessor or JSONLogitsProcessor. And _get_guided_logits_processor will only cache 32 instances. Different request with same guide would get the same processor and old processor would be removed if more than 32 guide request comes in.

You are right! It is a bit tough...