Human-in-the-Loop AI: Keeping Humans at the Center of AI

Mark: What's up everybody? Good morning. Hope you're doing well no matter where you are coming from. I have to share—don't be mad at me, Abby—Abby is really the star of the show just in case anybody's wondering. However, I have to share something that is so cool. I got a note yesterday from one of our amazing customers, and that individual said, "I want to take a moment to personally thank you for the Tuesday and Thursday live streams. They've been incredibly valuable and consistently give us new ideas for how we can leverage Domo within our organization." So if that individual is listening, and anybody else, I have the opportunity twice a week to jump on the phone with just amazing, brilliant people like Abby Stole to be able to show the power of what Domo is. And so hopefully you're taking the opportunity to share this with your colleagues, with your friends, because if you put to practice the things that Abby and Will and so many others are sharing, your business will go in much better places. I promise. With that, Abby, how are you doing?

Abby: Well, how are you, Mark?

Mark: I'm so good. We are talking today about something interesting. And Abby, I'm guessing every single conversation you have at this point is focused or at least adjacent to AI. "How do I get started? What do I do?" Is that fair to say?

Abby: Oh, every single one, inside of work and outside of work, right? Everyone's talking about it.

Mark: Totally. So, it's interesting as part of this presentation, part of the abstract, Abby is saying, "The future of AI isn't autonomous." Whoa. Some people are putting up stop signs right now like, "Abby, come on. Come on. What do you mean?" The statement is the future of AI isn't autonomous, it's actually a partnership. And what Abby is going to dig into today is really how to use human-in-the-loop to be able to be more effective as an organization. Abby, did I screw that up too bad?

Abby: No, I think you nailed it. And I think right, I think everyone's got strong opinions on AI, and I'm not the person to come in and say it's this way or that way. I think there's a place for both, right? I think there's a place for autonomous. I think there's also a place to come in and say, "Let's get the human in the loop. Let's make sure that this is ethical. Let's make sure this is accurate." So, time and place for everything, but a great place to start is human-in-the-loop.

Mark: I love it. Okay, Abby, where do you want to take us today?

Abby: Yeah. I guess kind of the basis of this is I'm chatting with customers every day, right? And a lot of people are super far advanced in AI—they're using it and everything—but there's also a large base of organizations and individuals and employees who aren't there yet, who maybe are just starting, still using ChatGPT on the website to clean up their email, and they haven't really explored too much more. And I talk to a lot of those customers who come and say, "I don't know where to start. We're nervous. We're scared to go in and say let's let AI go ahead and run everything." And I feel like a great answer, a great starting point, is human-in-the-loop, right? What's something you guys are doing today where we can take this and say let's automate a lot of this, but it doesn't have to be left completely to the AI?

Mark: So Abby, give me some examples. Of course we're not going to share customer names and whatnot, but what are some of those things? What are some of those processes that people are asking for right now? Is it: "I have five million invoices coming in and I don't have enough bandwidth to be able to go through each one"? Give me some examples of what you're being asked for.

Abby: All of the above. Invoices is a huge one where they say, "Hey, let's bring in these PDFs, right? Once we bring in these invoice PDFs, we're going to go in, maybe cleanse the data, summarize that data, and after we summarize that, we have some sort of action that needs to be taken. Let's go ahead and, before we send AI to do that action, we're going to have a human go in and double check." One that I built for a customer just a couple weeks ago—it was kind of a silly example, but they wanted to respond to Google reviews. They only wanted to respond to negative Google reviews. And actually, with their permission, I went in and built or rebuilt it in our Domo environment. I can actually walk through that if that's okay. I'm going to go ahead and share my screen.

Mark: And Abby, as you're pulling that up, what we'd love people to take away from this is: No matter the challenge, no matter the problem, whatever it is, we want you to think about what are those things that you need help with that potentially AI could help you be more efficient with, right?

Abby: Yep. Exactly. So, they were going in, and they are known for having awesome customer service, but there were some negative reviews that were being left unresponded to on Google reviews and some other applications, and they didn't want that to happen anymore. So, we came in and built a workflow that basically just goes in, pulls all of that data via the APIs. We pull those reviews, we go ahead and put a little agent on it to say, "Classify as positive or negative." From there, if it's negative, we want it to go ahead and do this. So, I'm going to go ahead and run this really quick, and we'll pretend that John Doe has come in and left a negative review and said the service was slow.

So, we go in, we pull it through the APIs. And not focusing on workflows today, but if you don't know what workflows are, I encourage you to go reach out to your account rep or start somewhere. Look it up on Google. There's a ton of great resources on Domo workflows. But we pull in that data, we're utilizing the workflows to come in and say, "Is this positive or negative?" If it's positive, we're not going to respond; we're only going to worry about the negative ones right now.

And then we came in and said, "Let's go ahead and utilize AI to respond to this." And this is kind of a silly response, right? Like I said, "Please write a response for a review less than 1,100 characters." That's a small novel, right? We probably want to clean that up with a better prompt for the AI, but then the AI comes back and writes a response, and before that's pushed out, it's actually sending an email to an individual who can come in and say, "Let's go ahead and approve or deny." In this usecase, I'm going to say, "That's the perfect response. I'm going to go ahead and approve that." And now it comes and finishes that task for us. So, kind of a small, simple example, but really a way where it can be related to everyone. Are we emailing? Are we sending marketing campaigns? What can we be doing to say, "Let's make this a little bit easier with AI, but make sure that there's some accuracy checks or just cleansing in there to make sure we're all on the same page before something happens?"

Mark: Very cool. And again, this could be any process—whether it's Google reviews, or invoices, the list goes on and on and on. Think about what is that thing that you have some challenge with or you need help with, and let's figure out how do we build some kind of automation where a human can be in the loop and make sure that it's a really stellar customer experience.

Abby: Yep. Exactly. And there's another example... talking about customers who aren't using AI at all. I was working with a customer who had kept track of their data just in Excel sheets, and they had manually entered everything, which is totally fine—you've got to start somewhere when it comes to data. They wanted to be able to upload this, but the problem was in their date column they had gone in and entered things like "4/21/26" and then they had written out "April 7th, 2026." So nothing was consistent.

So, we went in, put AI on it to go in and clean it up. But before that was pushed out, we created kind of a user interface—and I can share that as well—where in that interface, we gave this specific customer the ability to come in and say, "Yes, that's right. Nope, this isn't right." It's just taking a second. Sorry, let me make sure I've got the right page. Here we go.

Mark: Yep, we're seeing it.

Abby: Perfect. So, they can come in and say, "Okay, great. Now that the AI has cleansed this, I'm going to come in..." and they actually did this one wrong, so I'm going to go in and save those changes. In that AI prompt, we came in and said, "Give us a confidence level of how confident you think you are." So now we can come in and filter and say, "Anything from a range of 0 to 95% confidence, we want to double-check." Turns out that's wrong; we can come in, save, and edit that. So you don't have to start with AI doing everything for you. There's a ton of ways to start small, start slow, and get things going.

Mark: That's very cool. So, Abby, for people who haven't started at all—and again, you're dealing with customers like this all the time—what do they do? Where do they go?

Abby: Yeah.

Mark: Call Abby.

Abby: Yeah, call me. Directly, right? Leave my phone number in the chat. I think, just start, right? I think the first thing to do is come up with a usecase. A lot of people come to us and they say, "Where do we start?" It's like, okay, well, we could start in a million different places. Let's start and just brainstorm. Let's throw a couple ideas on the wall and say, "What are you guys doing that takes too long? What tasks are maybe annoying to take care of? What things could we go in and automate?" So, that's a great place to start. We are, as the solutions consultants, always happy to have those conversations with you guys. It doesn't have to be you guys doing this alone.

Mark: I love it. Okay, Abby, close us out. Why is Domo so critical in this whole process in what you built—whether it's a restaurant company, an HVAC company, or whatever else? Why is Domo so important to this human-in-the-loop effort that you're running?

Abby: Mark, that is a great question. I could go on and on about this. I think one of the biggest things is repeatable processes. It's awesome if you guys are doing these things in your own cloud instance or your own ChatGPT on your own website, but how can we make this repeatable? And I think workflows is a great area where you can come in and it checks the data, right? It doesn't have to be you uploading a spreadsheet every time something updates. So, I think repeatable is a big one, but then also governance. I think that's something that is so overlooked with data and AI and everything that's going on—who needs to see what—and Domo handles and orchestrates that for you from the start.

Mark: I love it. You've heard us say it before, everyone, and you'll hear me say it a whole bunch more. We help you with that data foundation wherever it is that it lives for you. If it's in Snowflake, if it's in Google, if it's in Databricks, or wherever else, we want to help you make that data readily available so that you can go and do cool things. You can take actions, you can build workflows, you can build apps, you can build agents that are going to help you drive business outcomes. We're going to help you publish it and distribute it in the right way. And most importantly, it's all going to be governed. It's all going to be safe, secure, and auditable, so that IT is not like, "I have no idea what's happening with AI in my company." Abby, you're amazing. Thanks so much for walking us through this. Hope everybody got some good stuff out of this, and we will see you again very shortly. Have a good one.

Abby: Thanks, Mark.

‍