The max context window of most of the big players is 1 million right now. Author almost certainly means per turn. But the whole context gets included with every prompt sent to the API. If you’re using like 200k context, which is not unheard of with a large codebase and without compacting often, 2-3 million input tokens per turn to the agent is actually not THAT much. But most of it would be cached tokens, which are cheaper and presumably more energy efficient than fresh input tokens.
E.g I just had a debugging session with 6 turns, shows me about 580k input tokens, but over 20 million cached input tokens, ~60k output tokens. But it also took like an hour with all the test runs and everything. 6 prompts to a chatbot takes like a minute or two in comparison.
Hmmmm wait a moment
The max context window of most of the big players is 1 million right now. Author almost certainly means per turn. But the whole context gets included with every prompt sent to the API. If you’re using like 200k context, which is not unheard of with a large codebase and without compacting often, 2-3 million input tokens per turn to the agent is actually not THAT much. But most of it would be cached tokens, which are cheaper and presumably more energy efficient than fresh input tokens.
E.g I just had a debugging session with 6 turns, shows me about 580k input tokens, but over 20 million cached input tokens, ~60k output tokens. But it also took like an hour with all the test runs and everything. 6 prompts to a chatbot takes like a minute or two in comparison.