Tokenmaxxing Is Over: A Practical Guide to AI Efficiency

Chris Willis: Welcome to Domo's podcast, Governing Data for Agentic AI. I'm Chris Willis, chief design officer at Domo, and this is our third episode, and a special one, because today we are welcoming Joseph Clark, our senior software architect of AI, also at Domo. Joe, welcome. We're going to be talking about token maxing today, but before we do that, can you tell us a little bit about what you've been up to?

Joseph Clark: Yeah. That'd be great. I'm excited for the conversation. Like Chris said, I'm the senior software architect here at Domo, primarily over a lot of the AI infrastructure. So I was able to establish our AI service layer, architect that out, do a lot of the implementation, as well as now assisting the architecture of our new agent platform, AI library. So real excited about what we have and what we've built, and also our conversation today.

Chris Willis: Well, this is fantastic because top of mind recently has been this shift in the concept of token maxing. And for those who aren't familiar with that, and it feels like it's been everywhere. I was in San Francisco recently and I saw ads, like billboards that said, "Are you token maxing enough? Are you spending enough on AI?" But for those who haven't heard that before, it's this sort of new, kind of almost like a pop culture metric, like in the same way look maxing kind of popped up out of nowhere, but this idea that your productivity and your understanding of AI is correlated to how many tokens you're burning on these models. Mm-hmm. How many prompts you're putting in. And it feels like there's been a shift that's been happening where there's a bit of a reckoning when it comes to token maxing, and I've got some thoughts about that. But before we do it, have you heard about token maxing? Have you talked to customers about it?

Joseph Clark: I have heard about it. It came up at the engineering org. From an engineering perspective, the way that I think about token maxing or maybe how it came about is you have this new technology, and you've got a bunch of engineers that are probably set in their ways, used to doing things a certain way. And again, we didn't have maybe some of the same problems that the companies had that really had the leaderboards and whatnot. Yeah. But we were really encouraged to explore different use cases for AI. How can we use AI to be more productive? Now we've kind of settled internally. We use a lot of Claude code, but our engineering leadership has been very encouraging of us to go out and try different things. Go a little crazy, figure out what works, figure out what doesn't. And I think along the way, maybe some people got a little bit focused that, hey, if you're not spending tokens, you're not trying hard enough.

Chris Willis: Yeah. That's a great point. And I know obviously at Domo, we were intentionally not creating token leaderboards because we thought that's a very easy, convenient metric, but it's not necessarily the metric you want. However, I can see the reason why companies would do it, because no one's really exactly sure how this technology works or where it's going or where it's best applied and where the efficiencies are and where the innovation can happen. So it makes sense that you'd rather have people try stuff and maybe find something versus just kind of sitting it out. But I think to your point, it is really about trying to figure out outcomes. However, it seems very clear that the all-you-can-eat token buffet has come to an end. Like the $20 a month fees for signing up to a large language model. Mm-hmm. Those, I think, are kind of going away. You used to get a lot more than $20 worth. And in fact, there was an interesting story the other day, which was kind of the opposite of that, where this was a story in Axios where a client spent $500 million in one month on AI because they didn't set any limits on the Claude model that they were using. Uber said they burned through their entire budget for 2026 in just a quarter. Amazon took down their token leaderboard. So there's clearly a big shift moving away from that, and I think that's- Mm-hmm ... in many ways a very good thing, but also raises new kinds of questions. For example, if you are moving from a token maxing model where you're just trying stuff, see what sticks, to just a flat rate pricing or a token-based pricing, it still seems like there's a disconnect because it doesn't necessarily connect what you're spending to an outcome. You could run an agent and maybe forget it's running and get a big bill, but maybe that agent deleted your production database. Are you going to pay OpenAI $5,000 for having an agent delete a production database or send out a terrible email to a bunch of customers?

Joseph Clark: Yeah. The cost isn't necessarily just in tokens.

Chris Willis: That's right. There are other costs there. Yeah, so I was kind of curious, how do you sort of think about, based on how you've been thinking about AI, and you've worked very deeply on some very complex AI projects, is it possible to move into a realm where if we had the right transparency and auditability, that you could move to a outcome-based model? Is that something you think is possible?

Joseph Clark: I think we're definitely getting there, especially as working for Domo, where we have all these great tools for working with AI. We're definitely transitioning to that point where you're not going in and manually building these workflows and these agents that are doing things. Instead, you have maybe your interface with an AI agent, and you're asking the agent, "Hey, here's a problem that I want to solve. What's the best way to solve that?" And maybe that agent is going to go and connect to your data over in BigQuery or Snowflake. Maybe that agent then is going to build a data flow to do some transformations and set up some workflows based on certain events that are occurring in your data. And I think that's really our goal is, hey- Yeah ... tell me what your end goal is, and we'll figure out how to get there. And as an engineer, a lot of times that takes the shape instead of building data flow, I'm writing code, or the agent's writing some code for me, and I come back and I review and make sure it is what I want to do. Even if it solves the problem, it needs to solve it in a way that's efficient. But we're definitely getting there. We're definitely a lot closer than I thought maybe we would be three years ago. Mm-hmm. And that's really exciting to me.

Chris Willis: So that's pretty dramatic. And on that topic, I saw this floating around LinkedIn, which I think gets to your point, or at least where things are going. So this was kind of interesting. It's called the AI price sheet lies. And so, that focus on tokens, price per token, might play out very differently in the real world. So you can see right here, and I can't vouch for these figures, but it feels, based on my experience, right. I know you're definitely seeing a lot of companies move more to a hybrid model, where they might use certain frontier models to do certain things, but then they're going to offload other work to cheaper models. And I thought this was interesting in that if you look at just the price per tokens, you might be missing a big part of the story, which is: what does it actually take to run these tokens on particular kinds of workflows? And I think you're going to see a lot of new tools come out that help answer this question. But I think it's pretty interesting here that, Gemini Flash 3, which is $3.50 per million tokens, is actually the second most expensive, at least in this case, to run. So there doesn't seem to be any real correlation between the price per token and the actual sort of outcome cost. I don't know if any of that resonates with you.

Joseph Clark: Yeah. And again, I agree with you. I'm not an expert in these numbers, but I think it does make sense that a more intelligent model might be able to get to the point a little bit more quickly. It may have to think a little less than a smaller model to understand what it needs to do. It might have to, maybe the smaller model has to experiment and call more tools- Yeah ... and maybe gets some wrong and has to retry. So I could definitely see the larger model being able to get to that final answer or complete that task a little bit more efficiently.

Chris Willis: Yeah. I think what's interesting is, I guess what surprised me on this one, so we don't have Fable 5, which is the latest and greatest from Anthropic, but you would think that the bigger the model, it might be a little bit more efficient, it's planned, but we also don't know how those were broken down, like how many tokens are being used in the planning stage. So a lot of times when you're using these models, you put in a prompt, and then you get an answer, but then there's all that stuff in the middle that you may not be paying attention to, like the reasoning steps. And so it's kind of interesting that you might get an answer, and you're like, "Oh, yeah, that makes sense," or, "That's useful," but it doesn't necessarily give you any insight into how much it had to work. So Joe, this really sets us up well, I think, for the natural next part of our conversation, which is: what are some tactics that we've started to see about optimizing token and AI use?

Joseph Clark: Yeah. I'd love to get into that. And there are definitely a few things that you can look at. One of the first things that I look at when I talk to customers about some of the agents that they're running that maybe aren't as efficient as they'd like, or maybe even don't work as well as they want them to, is that they're actually trying to solve deterministic problems with a non-deterministic solution. And could you just- So this diagram--

Chris Willis: Yeah, go ahead. Before you go on, could you maybe explain how you define deterministic from non-deterministic? Like some examples of, how would you identify what that means or what you're looking for in solving a problem?

Joseph Clark: Yeah. So deterministic problems, at least from an engineering perspective, again, it's something where I can kind of fully define- Okay ... this task, I need to do A, then I need to do B, then I need to do C. So there's always kind of a fixed set of actions that I need to take. So if I want to get a weekly summary email about, I don't know, a specific customer that's a really high priority customer, if it's always the same type of data that maybe I can query that same data set every time, run it through a template to produce the body of an email and just send it to me. Maybe it doesn't need to be really fancy. It's just, hey, I've got this basic set of metrics that I want to know about this customer, so let's query that data set and populate the template and send me that email. And so that's something that-

Chris Willis: So if you can break something down into steps, and you're confident that those steps for the same situation will get you what you want, that's deterministic.

Joseph Clark: Right. You want that repeatability.

Chris Willis: And so have you seen places or instances where people confuse that and say, "Oh, this should be a deterministic," like basically something handled by code or rules- Mm-hmm ... and instead use an agent? And what happens?

Joseph Clark: Yeah. Sometimes it's a little easier to ask the agent, and we get caught up in this writing code, too. It's like, hey, I need to refactor. Maybe I just need to rename a class. It's a lot more efficient for me to just tell the IDE, "Hey, rename this class, rename all of the references to it." But if we get caught up in this flow of, no, I got to ask the agent, I got to ask Claude to do it for me, now it's going and having to search and do it really inefficiently. And maybe it's catching things that I don't want renamed. There are specific tools for doing that task that are really efficient and have been around for a long time. And the same thing can happen in our product in workflows. Instead of dragging, finding the three different steps that I want and kind of constructing that narrative that I want, sometimes it's easier to just drag and drop the one action, the one tile, and say, "Hey, this is what I want." So I think the ease of use is one of the things that gets people, even though it's maybe the wrong solution. And what happens is, hey, maybe instead of querying those same metrics, maybe it's querying different metrics because it's generating the query every run, and so you get a different set of data. And if that report gets emailed to you, it's just not consistent from week to week, and maybe you change the model, and it's just completely different. If you know what data set needs to be queried, if you know what metrics you want, then there's no reason to make the model generate that query over and over and over again and spend all those tokens doing so-

Chris Willis: Well, you're also- ... when you can just do it one time. Because you're also always running the risk of Introducing some sort of random element- Right ... in there, right? Yeah. So I could see where you're saying if you're creating a weekly summary, that seems like a perfect deterministic workflow. It's like, on this day, at this time, I want these metrics shown in this way. But you also might want to add some aspect to it, which is like, okay, now I want AI to write a summary as part of it. But you don't necessarily want AI to recreate what your- Right ... what your weekly report or sales summary is going to look like. Right. I was curious, one of the things that I hear a lot coming up is the idea of agent loops. In some ways, I hear it in a very positive manner like, oh yeah, it's all about looping. But to me, it feels like the people talking about that are also interested in token maxing. And I don't know if I'm jaded, but can you maybe explain a little bit about agent loops? Is that something we should embrace, or is that something you want to avoid, and why would that matter? If you're either building these things or you're- Mm-hmm ... working with people to help build these things, how do I think about loops?

Joseph Clark: Again, it depends on the use case. So- Yeah ... at the bottom, we have those deterministic tasks. No, you don't need an agent loop that can solve that. And even going up a little bit, on that second level where it's maybe we see some more traditional machine learning tasks where it's like, hey, I've got maybe a customer support ticket coming in. I'm not asking the model to solve this for me. I just want it to tell me, to classify it between, hey, is this a bug? Is there a major outage? Do I need to get somebody on the phone? Are they just asking a question and we just need to- Yeah ... have somebody respond to them, or we just need to extract some information from those invoices. And these tools have been around for a long time, but what generative AI has added is a way to... It made them more flexible, right? Mm-hmm. And they've lowered the bar to entry. You don't have to train, you don't have to have a data scientist come in and do quite as much preparation on your data or quite as much training your own model, depending on what you want. You can kind of just tell- Yeah ... the model, "Hey, classify this for me." And we have classification services where you say, "Here's the classes that I want you to tell me, hey, what is this thing?" And then you can use that for control flow in your workflow or just output that somewhere else.

Chris Willis: Yeah. I was going to say, I think that's a really important point. I just want to make sure- Yeah ... that we capture that, which is a few things. One, there is a convenience aspect to it, right? Mm-hmm. Because AI used in enterprises before tended to be supervised types of machine learning. Right? You had a specific task, you had a certain kind of machine learning or AI technology, and you typically had to train it. And that was expensive, time-consuming. There was a lot of planning required. There was a lot of effort that you had to put into finding the right kinds of labeled data. But the flexibility of large language models for things like classification of comments or other kinds of text or things, has dropped so dramatically. People can, we call it, like a zero-shot prompt where you can say- Right ... "Here's an example of what I'm looking for. Go do it." And it can get pretty far, and that's pretty enticing, right? And so-

Joseph Clark: Yeah. And token for token or at the specific answer level, it may not be the most cost efficient, but if you consider that, hey, I don't have to train my own model, I don't have to spend months figuring this out. I can just drag and drop a tile onto- Yeah ... onto the screen and wire up my data and it's done. I think there's a lot of value in that.

Chris Willis: So speaking of which, you hear a lot about context and- Yes ... context is a big broad word, but context, in the case of models, is about adding more tokens. Right? Mm-hmm. Is there something that you have learned or do you have some strategies around how you can minimize the tokens you need? So if you are creating a workflow, giving the proper context at the right time is great and can prove accuracy. I think we're kind of coming off of, along with that token max wave moving, you had tools like OpenClaw, which were really agent loops, just sucking in everything they could into large context windows. We call that context bloat, where it's like, I'm just going to... If your model can handle a million or three million tokens, which is unbelievable, just from a technical perspective, but let's say it means you can just feed stuff in there. Right. The challenge, of course, was it's expensive, and just filling up an entire token context window, it doesn't mean you're going to get the best answers. Mm-hmm. And I was kind of curious what your experience has been with that, or if you have any kind of learnings or strategies you can share.

Joseph Clark: Yeah. There's definitely a balance, and I don't think we have a reference to this paper, but a few years ago, there were several, or at least one paper on what happens to accuracy as the number of input tokens goes up. You see a dramatic decrease in accuracy just because the model has to wade through that much more information, and it's less likely to find the correct answer. So there's definitely a scenario where too many tokens, not only is it more expensive, you're having to pay for those tokens, but it does reduce accuracy as well. But of course, not enough context, and the agent doesn't have the information that it needs to solve the ask that it received. So context management definitely becoming a much bigger part of the AI conversation and strategy in organizations.

Chris Willis: But I think right along with that is model selection. There's a lot of talk about models are becoming more commoditized, but just because they're becoming more commoditized doesn't mean there aren't choices to be made. We were talking about sort of the expenses. You've got these- Mm-hmm ... leading edge frontier models, but then you have a lot of interesting open source models. And the conversations I've been having with a lot of companies is they're, due to a number of different factors, from cost to reliability, et cetera, they're looking at using a mix of models. But Joe, from your standpoint, like How do you even begin choosing a model? And then how do you evaluate whether or not you made the right choice?

Joseph Clark: Yeah. So it really depends on the use case, but we definitely see if you've got maybe a harder task, more complex task, or if you've got that full agent loop going on, where larger models can maybe choose better tool calls, can reason a little bit more efficiently, as we saw earlier from that study. So sometimes, the more complex the task, you may want to use that larger model. Simpler tasks, if you're just summarizing or doing, again, some of those simpler traditional ML problems like data extraction, you can kind of get away from spending a lot of money and getting a smaller, faster model, and whether that's GPT Mini or a Haiku, or even go into these open source models. But the key really there is that while we have these general guidelines of, hey, yeah, complex, big, simple, small. Really, it comes down to your specific use case, and you really need a way to evaluate that. So actually get some test cases in. You need to evaluate accuracy alongside time to execution and things like cost, and find the right balance for your use case. And for some use cases, you may be okay if it's a little bit less accurate. For other use cases, accuracy is paramount, and it's worth the extra cost to go with a bigger model. And another thing to keep in mind is also scale. If you're going to be spending trillions of tokens on this, really you've got a lot of data that you've got to churn through. Maybe that is where you do consider, hey, maybe I take this open weight model and fine tune it, or use maybe a fine tuning API from one of the big model providers.

Chris Willis: Yeah. That's definitely something I've been hearing more about is fine tuning, which I was a little surprised at because I think most people thought it was really about prompt engineering. But I think to your point, there's three things that companies are trying to figure out how to provide for their models. There's visibility into the models. What are they actually doing? Mm-hmm. There's predictability, right? Which is, are they doing the same thing in the same situation in a way that's reliable? Yes. And then there's the control, which is, can I get them to kind of align with the tasks and the goals and the outcomes that we're looking for? And it seems like on top of that, the fact that there are new models coming all the time, you might not have changed anything in a particular workflow or in an assistant, but model gets swapped out, and you might be getting dramatically different responses or- Right ... generated content, and that can be problematic. So building an infrastructure around, or just maybe starting with an awareness. And I think it seems like companies that are doing the fine tuning are not necessarily using these to create their own bespoke behemoth models. That's kind of what you saw at the start- Mm-hmm ... which is big companies were like, "You're going to love our model. It's going to do everything for everyone all at once." Saw a lot of those. As opposed to, no, there are certain parts of a particular automation or process that are kind of special enough, and because no model has ever been trained on it, you can use 95-plus percent of the model's understanding of reasoning and language, and then just tweak it in the right way so it understands your business a little bit better. That seems like a great place to start. Any ideas around evaluations of that? Is that infrastructure that is yet to be built, or do you see other gotchas along the way?

Joseph Clark: I do see some of that out in the market, a lot in the open source landscape. There's definitely some evaluation metrics that we build into our own products, things like ROUGE or BLEU score. And these are good, again, for more traditional ML tasks. And they have their place, but they also have, when we, again, get back into generated text, a ROUGE score that looks kind of word to word, "Hey, does this output match what I'm expecting?" While one LLM may use a synonym that means approximately the same thing, it may be just as correct as the thing that you are comparing it against. But because it's not the exact same string of letters, it's going to score lower. So that's been a challenge in this new generative AI world is how do we evaluate these things? And kind of going beyond ROUGE score, we also use embedding models internally. It's like, "Hey, conceptually, is this kind of the same thing as this other thing over here what I'm expecting?"

Chris Willis: Yeah. Great. That's awesome.

Joseph Clark: Yeah. And then yet another pattern that has really emerged and become more popular lately is using LLMs as judges themselves. Actually asking the model or another model, "Here is some generated output. I'm expecting an answer kind of like this. How closely does it match my expected output? Does it hit the important points, even if it's using different words?"

Chris Willis: Yeah. And that's what I've seen. It seems like one great way to start, let's say, if you're familiar with Domo and you're using our Domo data flows, is to identify sort of simple tasks that LLMs can do well. One in particular is our Domo text generation tile. Mm-hmm. Any thoughts on that and maybe how people could kind of dip their toes in the pool, so to speak?

Joseph Clark: Yeah. And use this for- And alongside that, I just wanted to call out, we did just announce at this year's Domopalooza two other tiles that go along with that, which are classification and sentiment analysis. Again, these are solving traditional ML problems, but they're much more flexible than traditional models ever were because you can just prompt it. You don't have to have a fixed set of topics that are trained into your model. You can tell it, "Hey, these are the topics that I want sentiment around." And so I think all three of those are some of the most exciting products that we've released. Honestly, in the entire time I've been working at Domo, I think those have, I don't know, the most excitement, at least for me. Lets us really apply the capabilities of LLMs at scale. And one of the things that we get by using these batch processing is cheaper token costs. So our providers give us a 50% discount, at least all of the ones that I've seen as I've been integrating that, and so our pricing follows that as well. If you can cut your inference cost in half by doing batch, that's a really great way to save on tokens. Man, you can do a lot with that, even though it's not necessarily a full agent experience where, again, because it's in batch, you can't just do that agentic tool-calling loop. But going back to that pyramid, there's a lot of tasks, a lot of use cases that don't need a full agent. If you're just doing basic classification, if you're doing basic content generation, text generation, you can really churn through a lot of data. And we love data at Domo. We have a lot of data, and so when we think about taking all of that data and generating requests at a time to the model providers, it's just not viable. And so these new tiles let us solve these problems in a way that's, I think, really unique to Domo.

Chris Willis: So Joe, there's a lot going on out there. How is Domo helping customers optimize their AI usage and kind of navigating ways to build smarter workflows?

Joseph Clark: Yeah, and thinking of workflows specifically, again, we've got the agent tile, so if you do need that full agentic experience, you can give it the knowledge that it needs. Again, we talked about context engineering. You don't have to worry about all the technical details of how that works. If you add a dataset into knowledge or a document collection, if you've got unstructured data in there, document collection into knowledge, we'll go ahead and query what we need, or the agent, I should say, will query what it needs from either your datasets or those documents so that it can complete the task that you give it. So really excited about that for the use cases where we do need that full agentic experience. But again, and we started this before AI got really popular, but we have the whole infrastructure around workflows that really let you solve those deterministic problems in deterministic ways.

Chris Willis: And I would add to one piece. Yeah. I don't want it to... Maybe it doesn't sound super sexy, but the fact that it's in a governed way is really critical, especially if you've got autonomous- Yeah ... agents kind of just doing stuff on your behalf.

Joseph Clark: Yeah. And that was always when we very first started thinking about AI, the very first thing was, hey, Domo's got all this data. We can't just be shipping whatever data off to AI. Whatever we send to the model really needs to be the specific information that the user has access to because we don't really trust that the model... You can't train private data into the model and expect the model to keep that secret for you. Yeah. And so we really get that user-specific, governed view of the data that the model sees when it's performing actions on behalf of that user.

Chris Willis: So Joe, we started out this conversation talking about token maxing and sort of the consequences of that, and how people are kind of waking up to maybe some startling bills from their large language models. Do you have any examples or stories about how our customers are learning how to kind of tame this beast, how they're improving token efficiency, and maybe even getting better outcomes cheaper? We've been talking a lot about how things can kind of go maybe in a more expensive route, but are there ways that people are using these agents really effectively?

Joseph Clark: Yeah, absolutely. And actually going back to those data flow tiles, and actually one of the inspirations that made us think, hey, we need a solution here, was a large customer of ours who was just building this massive pipeline where they had a ton of survey data, and they wanted to do sentiment analysis on that information so they could say, "Hey, I've got this specific question and survey response, and I want to look for sentiment around these five topics." And what they did is they actually, at the time, they had to go into our Jupyter workspace environment. They had to write some Python code to hit our API. And again, we've got millions of survey responses, and they're making millions of requests to our API, generating billions of tokens, the input and output tokens. And it was relatively expensive. However, again, as a result of that, we developed those new tiles and data flows, and we were able to save for similar use cases. Not only would we be able to save the inference cost, a 50% cheaper per token cost, but also how much easier, instead of having to go into Jupyter, write Python code. Now all they have to do is drag one tile in there, drag our dataset input into that tile, give it the list of topics that we want sentiment on, and then write that to an output. And then from that output, and this is really where we start affecting business outcomes, is we take that output, and we feed that to another dataset and start building dashboards and apps and making decisions based on that data so that we know, hey, we have really good positive sentiment around this topic, but negative around this topic. Hey, if we're trying to make a business decision on improving things that we need to fix, now we know where to focus.

Chris Willis: I love that because I think that's one of the aspects of the learning process, right? We talk about these large language models as having kind of a jagged intelligence, and you've kind of mentioned it. There are certain places where you could let the models kind of run wild, and it's going to create chaos. But there are other instances where you can get real benefits if you kind of tame them correctly or put them in the right environment with the right sort of guardrails. And in those cases, you might be building greater capability for less cost. So at Domo, this is what we love to do. We love helping all of you use AI better and more efficiently. So please don't hesitate to reach out to us. We have people like Joe who have done a lot of this for many years. They know the platform better than anyone else, and we can walk you through it step by step. Joe, thanks for joining us- Yeah ... on our third podcast, and I think that's a great place for us to wrap it up. Thanks everybody for listening.

Joseph Clark: Been a pleasure. Thank you.