ROI Case File No.565: 'They Couldn't Tell Whether the Tools They Bought Were Working'
They Couldn't Tell Whether the Tools They Bought Were Working
Chapter 1: We Use a Lot of Things—but We Can't Tell If They Work
"I want to look into an AI agent. But honestly, I can't even tell whether the tools we use now are working."
Takeshi Tatemoto, CEO of ConstructAI, a provider of solutions for the construction industry, described the situation. "We use the free versions of ChatGPT and Gemini for search support. For construction we use ArchiX for perspective and CG rendering, and Office 365's Copilot in parts. We do use a lot. But I want to push AI use further, so if there's an AI agent with a track record among peers, I'd bring it in."
"Are the tools you use now returning as much as you put in?" Claude asked.
"...That's what we can't measure," Tatemoto answered. "We use them on a vague sense of 'handy.' I've never worked out how much we pay and how much effect comes back."
"What about the sales assistant's clerical work?" I asked, to confirm.
"That's a heavy load," Tatemoto answered. "Delivery-date checks and quote preparation eat time in routine. I want to lighten it with AI, but which tool, at what spend, returns how much? Without that estimate, we're about to add the next tool anyway."
"Add tools without measuring whether they work, and waste piles up," I replied. "Let's break this down with ROAS."
Chapter 2: ROAS Asks—Is the Effect Returning Against the Spend?
"This case calls for ROAS."
Claude wrote on the whiteboard: "Spend, Effect, Ratio."
"ROAS—Return On Advertising Spend, the ratio of effect to spend—judges a measure by how much effect returns per yen invested," I explained. "The key is to measure the vague sense of 'handy' by ratio. It originated as an advertising metric, but the same thinking applies to AI-tool spend. Against the amount put in, how much labor was saved or value created? Put the ratio out, and the tools to add and the tools to drop separate."
"First, let's measure the current cost," Gemini said, opening ROI Polygraph. The data Tatemoto provided was entered.
"Here is the monthly cost," Gemini read out. "Labor for the sales assistant's routine work—delivery-date checks, quote preparation, and the like: 160 hours per month on average, at ¥3,800 per hour, ¥608,000 per month. Waste from vague spend decisions because AI-tool effects go unmeasured: ¥340,000 per month. Unmet efficiency from limited AI use: ¥360,000 per month. Opportunity loss as clerical work ties up staff who can't move to high-value tasks: ¥380,000 per month. Duplication and inefficiency from a sprawl of tools: ¥260,000 per month. A total of ¥1,956,000 per month. Roughly ¥23.47 million per year."
Tatemoto stared at the figures. "I settled for 'use it because it's handy.' Add the waste of adding tools without measuring effect, and it comes to this much."
"Then let's design it with ROAS," I continued.
[Spend—Lay out what you're paying now]
"First, lay out the spend," Claude said. "ChatGPT, Gemini, ArchiX, Copilot—hear in detail how much you invest in each tool now and how they're used. Only once the spend is visible does the denominator of the ratio fix."
[Effect—Measure the labor saved and value created]
"Next, measure the effect," Gemini continued. "How much labor each tool saves and how much value it creates. Narrow to the sales assistant's clerical work and put the time freed by automation into numbers. Effect is the numerator of the ratio."
[Compare—Line up each tool's ROAS]
"After effect, line up the ratios," I continued. "Compute the ratio of effect to spend, tool by tool. The tools that work and the tools that cost more than they return line up side by side. Judge the AI-agent candidates by this ratio, too."
[Select—Invest in the one move with the high ratio]
"Finally, invest in the high-ratio move," Claude continued. "Choose an AI agent that works on automating the sales assistant's clerical tasks, and build the scenarios for delivery-date checks and quote preparation. Don't spread thin across everything; concentrate on the one point with high ROAS. Move the freed hands to high-level tasks."
[Estimating the payback]
"Let's run the numbers on ROI Proposal Generator," Gemini proposed.
- Initial cost: AI-agent adoption, building clerical-automation scenarios, reorganizing existing tools, sales-support setup, and operational training—¥4.7 million total
- Monthly cost: agent licensing and operation combined, ¥200,000
- Monthly savings: automation of clerical routine = ¥430,000 (assuming a 70% reduction); removing duplication by reorganizing tools = ¥260,000; efficiency from expanded AI use = ¥340,000; effect of reassigning staff to high-level tasks = ¥340,000; totaling ¥1,370,000 per month
- Net monthly savings: ¥1,370,000 − ¥200,000 = ¥1,170,000 per month
- Payback period: ¥4.7 million ÷ ¥1.17 million = about 4.0 months
"Four months to recoup," Gemini summarized. "What works is not adding tools on a vague sense, but judging by the ratio of spend to effect. Don't measure whether they work and wasteful tools pile up. Put the ratio out with ROAS and concentrate on the high point, and what you invest returns. The investment doesn't miss."
Tatemoto checked the figures. "I kept adding on 'looks handy.' Measure by ratio, and where to invest gets decided."
"ROAS is a tool for judging tools by the ratio of spend to effect," I replied.
Chapter 3: An Implementation Plan That Judges by Ratio
"Let me lay out the approach," I said, standing at the whiteboard.
"Month one—lay out the spend and usage of existing AI tools. Month two—measure each tool's effect and compute its ROAS. Months three and four—build the clerical-automation scenarios and adopt the AI agent. Month five—reorganize existing tools and run operational training. Month six—trial operation and effect verification (re-measuring ROAS). Month seven onward—expand investment in high-ratio areas and continuously review the tool lineup."
"Won't the new agent's effect become unclear again, too?" Tatemoto asked.
"It won't," Claude replied. "It becomes unclear because you leave it running and never measure. Build the mechanism to put out the spend-to-effect ratio first with ROAS, and the new agent gets judged by the same yardstick. If it works, increase the investment; if it costs more than it returns, drop it. The vague sense of 'handy' turns into an investment you can judge by numbers."
Tatemoto took notes. "Before adding tools, measure by ratio whether they work. Now I see the order."
Chapter 4: The Day 'Whether It Works' Could Be Measured
Ten months later, a report arrived from Tatemoto.
After the agent went in, the sales assistant's clerical work grew much lighter. "The routine of delivery-date checks and quote preparation was automated, and hands opened up. We could turn the freed time to proposals and customer contact," Tatemoto wrote.
The tool lineup was reorganized, too. Low-effect duplicates were dropped and the set narrowed to what works. "Some tools we'd added on a vague sense turned out, by ratio, to cost more than they returned. Once reorganized, the monthly waste dropped," the report read.
The biggest change showed in how tools were brought in. From adding because it looks handy, to investing by judging by ratio. "We'd brought in new AI because 'it seems to have a track record.' Once we measured by ROAS, we could estimate the effect before investing," Tatemoto wrote.
Reassignment to high-level work advanced as well. Hands freed from clerical work moved to high-value tasks. "The assistant, buried in odd jobs, could concentrate on the work they should really do," the report read.
As a side effect, the way investment was viewed changed. Not the feel of 'handy,' but judging by ratio, took root. "We stopped 'using it because it's handy.' We started thinking in terms of how much returns per yen," Tatemoto wrote.
At the end of Tatemoto's report was this: "I thought the AI struggle was that we couldn't find a good tool. But the real problem was that we weren't measuring whether the tools we bought were working. The moment we put out the spend-to-effect ratio with ROAS, where to invest came into view. Before adding tools, measuring by ratio came first."
The day a company that couldn't tell whether its tools were working became a company that could invest by judging by ratio, AI-agent use had shifted from vague adoption to a design that judges by the ratio of spend to effect.
"AI-use requests usually arrive as 'I want to bring in a good tool.' But before adding tools, there's a question to ask: are the tools you have now returning as much as you put in? What ROAS asks is the ratio of effect to spend. Put the ratio out, and the tools to add and the tools to drop separate. The day a company that couldn't tell whether its tools were working could measure by ratio, what changed was not the tool's performance but the very perspective that judges tools by the ratio of spend to effect."
Related Files
Tools Used
- ROI Polygraph — Visualizing clerical-routine labor, waste from unmeasured effect, and opportunity loss in reassignment to high-level work
- ROI Proposal Generator — Payback simulation for AI-agent use, starting from the ratio of spend to effect
