His prompts were each using 2.9 million token. That’s a massive amount, he was basically purposefully using complex tasks that needed massive amounts of data parsing. We’re talking about like 1 percent of users that are using 2.9 million per prompt.
The method used to calculate doesn’t include batching. These companies aren’t running one request per gpu here. Even the authors of the method admit it over estimates by 4 to 20x.
The max context window of most of the big players is 1 million right now. Author almost certainly means per turn. But the whole context gets included with every prompt sent to the API. If you’re using like 200k context, which is not unheard of with a large codebase and without compacting often, 2-3 million input tokens per turn to the agent is actually not THAT much. But most of it would be cached tokens, which are cheaper and presumably more energy efficient than fresh input tokens.
E.g I just had a debugging session with 6 turns, shows me about 580k input tokens, but over 20 million cached input tokens, ~60k output tokens. But it also took like an hour with all the test runs and everything. 6 prompts to a chatbot takes like a minute or two in comparison.
God I hate these fake science calculation posts.
His prompts were each using 2.9 million token. That’s a massive amount, he was basically purposefully using complex tasks that needed massive amounts of data parsing. We’re talking about like 1 percent of users that are using 2.9 million per prompt.
The method used to calculate doesn’t include batching. These companies aren’t running one request per gpu here. Even the authors of the method admit it over estimates by 4 to 20x.
Hmmmm wait a moment
The max context window of most of the big players is 1 million right now. Author almost certainly means per turn. But the whole context gets included with every prompt sent to the API. If you’re using like 200k context, which is not unheard of with a large codebase and without compacting often, 2-3 million input tokens per turn to the agent is actually not THAT much. But most of it would be cached tokens, which are cheaper and presumably more energy efficient than fresh input tokens.
E.g I just had a debugging session with 6 turns, shows me about 580k input tokens, but over 20 million cached input tokens, ~60k output tokens. But it also took like an hour with all the test runs and everything. 6 prompts to a chatbot takes like a minute or two in comparison.