Token Analyzer

I kept guessing how many tokens my prompts were, so I made a one-page tool for it. Paste text (or a system + user prompt) and it shows the token count, the token IDs, how the text splits into tokens, and roughly what it would cost on each model.

Every provider tokenizes differently, and Gemini and Claude don't ship a local tokenizer, so you can't count their tokens exactly offline.

The 4 OpenAI models use tiktoken for exact counts. The 4 Gemini/Claude models use a ~4 characters per token estimate, and the app clearly says it's an estimate. Cost is tokens ÷ 1M × the price per 1M tokens.

• One MODEL_CONFIG dict holds each model's tokenizer, provider, price per 1M and an exact-or-estimate flag.
• get_tokenizer() picks get_encoding() or encoding_for_model() depending on the model.
• The per-token breakdown is hidden for estimated models, since it would be made up.
• An optional side-by-side with the gpt-4o-mini tokenizer.

It does its job. The prices are hardcoded, though, so they go stale whenever a provider changes its pricing.