Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.

The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there's little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn't occupy much memory, which we achieve via offloading.

Accuracy and time-to-first-token against device weight size

Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either

Based on Qwen3.5 MoE 35B and Qwen3.5 2B

Copied from readme.

We are excited to release Infinity-Parser2, our latest flagship document understanding model. We offer two distinct variants to address diverse deployment constraints: Infinity-Parser2-Pro, optimized for maximum accuracy in precision-critical tasks, achieves state-of-the-art results on olmOCR-Bench (87.6%) and ParseBench (74.3%), surpassing frontier models including DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU-2.5. Infinity-Parser2-Flash, engineered for low-latency inference, delivers a 3.68x speedup over our previous Infinity-Parser-7B model. With significant upgrades to both our data engine and multi-task reinforcement learning approach, the model consolidates robust multi-modal parsing capabilities into a unified architecture, unlocking brand-new zero-shot capabilities across a wide range of real-world business scenarios.

Not what we expected...

Built on Qwen, these models incorporate our latest advances in post-training techniques. MindLink demonstrates strong performance across various common benchmarks and is widely applicable in diverse AI scenarios.

72B 32B