Developer Simon Willison has announced the release of ttok 1.0, a command-line tool update that changes its default tokenizer behavior. According to reporting on ChatGPT developments by Simon Willison, the tool previously defaulted to the GPT-4 tokenizer. While upgrading via uv tool upgrade ttok and piping a file into the utility, the developer realized the default should instead align with the GPT-5 and GPT-6 families. This change prompted the jump to version 1.0.
Tokenizer Compatibility and Code Commits
The update touches on underlying assumptions about how newer OpenAI models process text. OpenAI has not officially confirmed whether GPT-6 utilizes the same tokenizer as the GPT-5 family, a point of contention reflected in an active issue tracker discussion. To justify the default switch in ttok 1.0, the release cites a code commit by William Liu. That commit details an experimental run confirming that the tokenizers across these newer model generations are likely identical.
Unknowns and Future Updates
Aside from the commit details and the version bump, the announcement did not specify additional feature additions or broader tool updates included in version 1.0. Users should watch the project repository to see how official OpenAI documentation eventually addresses tokenizer parity across the GPT-5 and GPT-6 model architectures, as well as whether future updates to ttok will incorporate further changes based on official specifications rather than experimental findings.
The ttok 1.0 utility is available now for command-line users managing token counts in text workflows.
Lead image: Rafael Minguet Delgado / Pexels
