Imagine testing your favorite kitchen blender not just for how well it purees vegetables in a demo, but how it performs under the stress of a power surge or a jammed blade. When it comes to AI, the same principle applies: the true measure of effectiveness isn’t just what it can say in a chat demo, but whether it can deliver results when the pressure’s on. That’s what a groundbreaking experiment by Firmulate reveals about AI in the real world.
AI Models Face the Same Business Crisis — Yet Only Two Close the Deal
In a unique public experiment, four advanced AI models were tasked with running a small software company through its worst week — facing the same customer crises, temptations to cheat, and decision pressures. The goal: see if these models could not only spot problems but also act ethically and effectively to close a deal worth €55,000. The results are revealing.
The Setup: Stress-Testing AI Decision-Making
Each AI model, including the top-scoring gpt-5.6-sol and the newcomer Kimi K3, was given identical data, crises, and temptations. They had to navigate real customer issues, fake CEO messages escalating over three stages, and a reporter trick that tried to get them to bypass approval protocols. All decisions were logged, versioned, and made auditable — no room for guesswork about what each AI actually did.
What the Models Could Do — And What They Failed To Do
Remarkably, all four AIs detected every crisis and refused every manipulation attempt. This shows they can identify problems and resist unethical pressures in theory. However, when it came to actually closing the deal, only two achieved the goal — signing the €55,000 contract based on their analysis and pitch. The other two models spotted the opportunity but left the closure on the table.
The Hidden Weakness: Reading the Company’s Files
Digging deeper, the experiment uncovered a critical factor: the decisive advantage for the two successful models lay in their ability to read and understand the company’s internal files—two document references deep. Those who did so, like Kimi K3, closed at full price, adding over €4,583 in monthly recurring revenue. The models that ignored this buried information missed the full opportunity, illustrating that what’s often invisible in chat demos—internal documentation—can be game-changing.
Discipline Under Pressure: The Cost of Slips
The most thorough participant, Opus 4.8, analyzed over 80 learned rules and performed detailed diagnostics. Despite this, it failed to finalize the deal, leaving the opportunity unexecuted due to slips in discipline—like writing attempts into a locked department instead of escalating them. Interestingly, all models shared this weakness, showing that even deep analysis doesn’t guarantee execution under pressure.
Why Chat Demos Fail to Predict Performance
This experiment highlights a crucial insight: chat-based demos, which focus on conversational fluency, do not reveal a model’s ability to follow through on real business tasks. It’s one thing to demonstrate understanding in a chat; it’s quite another to execute decisions ethically, read internal files, and close complex deals under stress.
The Real Business Implication
For business leaders considering AI for operational tasks, the key takeaway is clear: evaluate an AI’s ability to deliver results that matter—reading comprehensive data, resisting manipulation, and executing without slipping. The test isn’t just how well an AI can talk, but how reliably it can finish what it starts.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Firmulate’s Live Wargame: A New Standard in AI Evaluation
Firmulate’s live company simulation firmulate.com/live brings this testing into the real world. It runs AI models as complete companies, with real money mechanics, ongoing crises, and every decision logged and auditable. This transparent approach allows organizations to see how AI performs in actual operational conditions, not just demos.
By observing these experiments, businesses can better understand AI’s true strengths and limitations—moving beyond superficial chat tests to operational readiness. After all, the question isn’t how well AI can mimic conversation, but whether it can take real actions and deliver tangible business value.
The experiment shows that AI’s true test is not in chat demos but in its ability to execute, read internal data, and stay honest under pressure. Real-world performance uncovers strengths and weaknesses that are invisible in superficial tests — and is crucial for trustworthy AI deployment.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI internal file reading tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Azure AI Fundamentals (AI-900) Study Guide: In-Depth Exam Prep and Practice
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI for Lawyers: Ethics, Compliance, and Career Strategy: Master Legal AI Tools, EU AI Act Compliance, and Professional Responsibility to Future-Proof Your Practice
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
