ROI Case File No.512: Five Minutes If You Ask a Senior, a Whole Day If You Search
Five minutes if you ask a senior, a whole day if you search
Chapter 1: The Files Exist, But No One Can Reach Them
"Past machining records are always somewhere on the internal server. We just can't reach them."
Wataru Karasuma, Engineering Department Manager at NebulaTech, said this as he opened the file server hierarchy. Excel sheets of machining conditions, PowerPoint proposals, scanned PDFs of handwritten process notes—format and storage location varied by who did the work. "When a junior engineer is stuck, asking a senior gets an answer in five minutes. Searching on their own takes a whole day and turns up nothing."
"What about your current RAG chat system?" Claude asked.
"We rolled it out six months ago," Karasuma replied. "One-shot answers come back. But it can't go deeper. Ask 'are there past cases similar to these machining conditions?' and it doesn't pull related files across the system. Summaries are shallow. Juniors end up walking to a senior anyway. The structure of draining seniors' time didn't change."
"And the load on the seniors?" I asked.
"Three veterans absorb most of the questions," Karasuma answered. "Their own work stops for an average of sixty hours a month. And of course there's a serious knowledge succession risk too. If any of the veterans retire, the knowledge leaves with them."
"What RAG needed wasn't search precision—it was question design," I responded. "Let's rebuild it with Persona_Analysis."
Chapter 2: Persona_Analysis Asks Whose Question Is Being Answered
"This case calls for Persona_Analysis."
Claude wrote "P·A" on the whiteboard.
"Persona_Analysis is a framework that designs detailed fictional personas of users—attributes, behaviors, challenges—and proceeds with service design from that person's viewpoint," I explained. "Its marketing use is well known, but it's fundamentally effective for internal system design too. Because internal tools won't be used unless 'who, in what situation, asks what' is defined, no matter how much you improve search precision. The reason RAG didn't work was the persona was vague."
"Let's measure current costs first," Gemini said, opening ROI Polygraph. Karasuma's data went in.
"Monthly knowledge-access costs are out," Gemini read. "Information search labor for twelve juniors averages 360 hours monthly at 3,400 yen/hour, or 1.224 million yen monthly. Question-handling labor for three veterans averages 180 hours monthly at 5,800 yen/hour, or 1.044 million yen monthly. Rework cost from information that can't be found averages 900,000 yen monthly. Opportunity loss from lower proposal quality due to inability to leverage past cases is 700,000 yen monthly. Expected-value knowledge succession risk is 600,000 yen monthly—veterans' retirement impact times probability. Total: 4.468 million yen monthly. Annualized: roughly 53.6 million yen."
Karasuma stared at the numbers. "We thought RAG had improved things. The costs hadn't actually disappeared."
"Now let's design with Persona_Analysis," I continued.
[Persona Layer 1 — Drawing a typical junior engineer]
"First, the representative user persona," Claude said. "'Second-year machining engineer, 25 years old, no prior career, mechanical engineering background, currently handling prototyping, wants to consult past cases roughly three times a week but hesitates to repeatedly interrupt seniors.' We define behavioral context and psychological barriers, not just attributes."
[Persona Layer 2 — Decomposing the typical question structure]
"Next, we break down the questions the persona asks," Gemini continued. "'Are there past cases of similar shapes machined in this material?' 'What caused that failure case?' 'Will quality hold under these conditions?' 'What would a senior check next?' Sorting the question structure into four types reveals this is a response-design problem, not a search-precision problem."
[Persona Layer 3 — Mapping to information sources]
"We map the classified questions to internal information sources," I continued. "Excel machining condition sheets correspond to 'reference successful conditions.' PowerPoint proposals correspond to 'reference customer-facing templates.' Handwritten PDFs correspond to 'failure case notes and tacit knowledge.' The structure determines which file groups to prioritize by question type."
[Persona Layer 4 — Designing deep-dive dialogue]
"Finally, deep-dive dialogue design," Claude continued. "Not one-shot Q&A. To the persona's question, the system returns follow-up questions in the style of 'this is what a senior would check next.' Like 'we have cases with similar shapes in this material, but the machining speeds vary—do you want to prioritize which condition?' The dialogue naturally organizes the viewpoints the junior should be considering."
[Estimating investment recovery]
"Let's run ROI Proposal Generator," Gemini proposed.
- Initial cost: Persona design, source tagging, dialogue engine development, integration with existing RAG, and field training: 6.4 million yen total
- Monthly cost: Dialogue infrastructure operation and ongoing data updates: 220,000 yen/month combined
- Monthly savings: Junior search labor reduction = 840,000 yen/month (70% reduction assumed), veteran response labor reduction = 720,000 yen/month, rework reduction = 620,000 yen/month, proposal quality improvement = 450,000 yen/month, knowledge succession risk reduction = 400,000 yen/month. Total: 3.03 million yen/month
- Net monthly savings: 3.03 million yen − 220,000 yen = 2.81 million yen/month
- Payback period: 6.4 million yen ÷ 2.81 million yen = approximately 2.3 months
"Just over two months to recover," Gemini summarized. "What matters is the structure: persona design layered on top of RAG, not replacing it. We preserve the existing investment while lifting response quality."
Karasuma checked the numbers. "We were discussing how to improve RAG accuracy. The idea of designing a persona never came up."
"Persona_Analysis is a tool for translating technical problems into structural problems," I responded.
Chapter 3: A Small-Start Rollout Plan
"Let's lay out the path," I said at the whiteboard.
"Weeks 1–2: Interview five junior engineers, refine the persona. Week 3: Classify typical questions into four types, map information sources. Weeks 4–5: Prototype the dialogue engine, design deep-dive question patterns. Weeks 6–7: Integrate with existing RAG, tag file groups. Week 8: Pilot within the engineering department, gather feedback. Weeks 9–10: Tune response quality, prepare company-wide rollout. Week 11 onward: Phased rollout to other departments, add personas."
"You're not deploying company-wide from the start?" Karasuma asked.
"A small start with one department is the rule," Claude responded. "Persona design is the kind of work where accuracy improves through co-creation with the field. Aim for a company-wide design from the start and you end up with a persona that resonates with no one. We build a success case in engineering, then add personas for other departments."
Karasuma took notes. "I didn't think a six-month-stalled debate would start moving from a decision to define one persona."
Chapter 4: The Day Answers Came Back Before Anyone Called a Senior
Eight months later, Karasuma's report arrived.
Junior engineers' information search time, three months after the Persona_Analysis version of the system went live, had fallen 72% versus prior. "Past cases that took a day to find now come up in ten minutes. And not just come up—related failure cases are presented alongside them," Karasuma wrote.
Questions to veterans also dropped substantially. First-pass questions now closed in the new system, leaving only judgment-required cases for the seniors. "The three veterans' average sixty hours of monthly response labor dropped to fifteen hours. Their own work time came back," the report noted.
The most surprising change showed up in junior engineers' growth rate. The deep-dive dialogue design instilled a habit of pushing their thinking further. "They used to just take the answer and stop. Now the system asks them 'what should you check next?' Their thinking muscles are getting stronger," the report said. The time for first-year staff to operate independently shortened from twelve months to eight.
The structure of knowledge succession changed too. Veterans' day-to-day comments and supplementary notes now accumulate in the system as responses to the persona's questions. "What veterans wrote used to be personal diaries. Once they started writing as responses to the persona's questions, it became an organizational asset," Karasuma wrote.
Building on engineering's success, persona design for sales and procurement began in month seven. "Sales' persona asks very different questions. Not aiming for company-wide rollout from the start was the right call," the report said.
A side effect was a drop in junior engineers' psychological barriers. The reluctance of "feeling bad for asking the senior again" disappeared, and learning cycles now turn through the system. "The psychological cost of asking questions has dropped. That might be the biggest change of all," Karasuma wrote.
At the end of the report, Karasuma wrote: "Six months of debating search precision yielded no answers. The moment we defined a single persona, the design moved. What looked like a technical problem was a problem of not defining the target."
The mornings when answers came back before anyone called a senior had become routine; on the seniors' desks, hours of work had quietly returned, he wrote.
"The debate about improving RAG accuracy never ends. As long as who, in what situation, asks what is undefined, technology alone won't deliver. Persona_Analysis asks about the resolution of the target. Define one user and the necessary response is determined; once the response is determined, the information sources to reference are determined. Design the questioner before you organize the information. The reason scattered files don't become organizational assets isn't how they're stored—it's how the questions are designed. On the day the answer that a junior couldn't reach in a whole day started coming back in ten minutes, what changed wasn't the search engine—it was the user persona itself."
Related Files
Tools Used
- ROI Polygraph — Visualizing junior search labor, veteran response load, and knowledge succession risk
- ROI Proposal Generator — Investment recovery simulation for persona-driven knowledge infrastructure
