tiktoken count doesn't match API usage.prompt_tokens

Max Petersen
Max PetersenJan 3, 2026

I'm using tiktoken to estimate costs before making API calls, but my count consistently differs from what the API reports.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o") messages = [{"role": "user", "content": "Hello, how are you?"}]

My count

tokens = len(enc.encode("Hello, how are you?")) print(f"My count: {tokens}") # 6

API reports: usage.prompt_tokens = 12

The API always reports more tokens than my tiktoken count. I know there's overhead for message formatting but the difference seems too large. Am I missing something?

2.6k views14 replies29 likesSolved

Log in to reply to this topic.