NVIDIA and AI4Bharat release 2.8 trillion token Indic language corpus

NVIDIA, in collaboration with IIT Madras's AI4Bharat lab, has introduced the Nemotron-pretraining-Indic-v1 corpus. The dataset contains 2.8 trillion tokens across 22 Indian languages, including both native-script and romanized text. Alongside the data, they are releasing an open GPU-accelerated pipeline based on NeMo Data Designer to allow developers to generate and refine their own Indic language datasets.

Technical details of the Indic pipeline

• The corpus includes 2,601.0 billion native-script tokens and 224.6 billion romanized tokens (using the Gemma-4 tokenizer).
• It covers 22 languages listed in India's Constitution, spanning 34 different language-script configurations.
• The generation process uses NeMo Data Designer and Gemma-4-31B-it for translation, quality assessment, and repair.
• A quality rubric inspired by Multidimensional Quality Metrics (MQM) is used to ensure outputs score at least 75/100.
• The workflow preserves technical content such as code, equations, tables, and URLs during translation.

Why it matters

This release provides a scalable method for addressing the data scarcity in Indian languages, enabling better multilingual LLM performance for over 1 billion people.