# Simon Willison Web & Open Source: ttok 1.0

**Published:** 2026-10-09T00:34:43+00:00  
**Source:** Simon Willison Web & Open Source  
**Category:** software-engineering  
**Canonical URL:** https://fosswire.org/news/simon-willison-web-open-source-ttok-10.html  

## Executive Summary
Release: ttok 1.0 I released ttok 0.4 , ran uv tool upgrade ttok , piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead! I figured switching the default was a reasonable excuse to finally ship a 1.0. OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same: All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one...

## Architectural & Systems Analysis
From an artificial intelligence architecture, model weights governance, and inference efficiency perspective:

- **Weights Accessibility & Sovereignty:** Evaluates whether weights are open for private self-hosting or locked behind centralized cloud APIs.
- **Quantization & Edge Performance:** Kernel optimizations (4-bit/8-bit GGUF, AWQ, EXL2) allow high tokens-per-second on consumer GPUs and Apple Silicon.
- **Reasoning & Architectural Scaling:** Scrutinizes mixture-of-experts (MoE), attention mechanisms, and fine-tuning datasets against open community benchmarks.

## Impact on the Open Ecosystem
Protects developers and enterprises from proprietary black-box entrapment, fostering auditable, sovereign AI infrastructure.