Apertus 1.5 adds multimodal understanding, 4x context window to open-weights LLMs
Swiss AI's Apertus 1.5 extends its 8B and 70B open-weights models with native image/audio understanding, a 262k-token context window, optional chain-of-thought reasoning, and improved tool use, trained on 4T additional tokens for the 8B model and 2T for the 70B.
Summary
Apertus 1.5, released by Swiss AI on Hugging Face under Apache 2.0, upgrades the project's 8B and 70B parameter models with multimodal input capabilities through early-fusion tokenization. The context window expands from 65k to 262,144 tokens. A new optional thinking mode improves reasoning performance on benchmarks like AIME. The release maintains the project's fully open approach: weights, training data, and evaluation details are all published.
How Apertus 1.5 handles multimodality
- Early-fusion architecture: Images and audio are tokenized directly into the text sequence rather than using a separate vision encoder. Images use Emu3.5 tokenizer (512×512 → 1,024 tokens); audio uses WavTokenizer at 40 tokens/second.
- Vocabulary-only extension: The transformer architecture is unchanged; training speed matches text-only models.
- Four-stage continued pretraining: Unpaired modalities → paired alignment → cooldown with higher-quality math/code data → gradual long-context scaling from 8k to 262k tokens.
- Five-stage post-training: SFT (3.7M samples) → long-context SFT → RL with verifiable rewards (350k prompts) → DPO with judge-scored preferences → SDPO charter alignment.
- Training scale: 8B model reaches 19T total tokens; 70B model reaches 17T total tokens.
Why it matters
The release demonstrates that fully open model development—including published weights, data, and evaluation infrastructure—can remain competitive with partially open alternatives on multimodal and multilingual tasks. The EvalEval/EEE integration may influence how other open model projects report benchmark results.
Original reference: huggingface.co →