tiktoken count doesn't match API usage.prompt_tokens
Max PetersenJan 3, 2026
I'm using tiktoken to estimate costs before making API calls, but my count consistently differs from what the API reports.
import tiktokenenc = tiktoken.encoding_for_model("gpt-4o")
messages = [{"role": "user", "content": "Hello, how are you?"}]
My count
tokens = len(enc.encode("Hello, how are you?"))
print(f"My count: {tokens}") # 6API reports: usage.prompt_tokens = 12
The API always reports more tokens than my tiktoken count. I know there's overhead for message formatting but the difference seems too large. Am I missing something?
2.6k views14 replies29 likesSolved
Log in to reply to this topic.