Enterprise AI’s ROI reckoning drives a focus on workflows, not models

Conversations at the Forum, particularly during a keynote discussion around enterprise AI adoption led by Trevor Upton, CFA, Mosaic, KeyBanc Capital Markets and featuring Leo Meyerovich, Founder and CEO of Graphistry; Maya Mikhailov, Co-Founder and CEO of Savvi AI; Kierra Dotson, Associate Director of AI Strategy & Governance at Further; and Aaron Delp, Head of Technical Marketing & Analyst Relations at Mistral AI, made it clear that high AI bills are driving pressure within enterprises to optimize costs and demonstrate positive ROI. In addition to well-publicized usage limits,¹ enterprises are exploring ways to optimize their AI spend through increased use of open-source models and model routing.

The coming ROI reckoning

Dotson noted during the keynote that enterprises’ AI costs are “not sustainable” and that a “more pragmatic approach” is needed, with Mikhailov adding that enterprises are facing “economic pressure … to prove operational efficiency.” Looking forward, Delp warned of a coming “reckoning” and expects enterprises to rationalize their AI spend over the coming year. A core challenge for enterprises is that, while they are successfully deploying AI across coding and software development, there are structural reasons that may prohibit that success from being replicated across other use cases.

Why coding may be the exception

Coding has two characteristics that make it unique among enterprise use cases: 1) a massive horizontal dataset of what “good” looks like; and 2) a means of validating if the output is “good.”

On the training side, software development is horizontal in nature, with virtually all enterprises using similar tools, resulting in copious examples of documentation, from open-source repositories like GitHub to resources like Stack Overflow. The result is that AI has a very large dataset to train from, which makes models uniquely accurate for coding. AI also benefits from the existing workflows that humans use for coding and software development, which includes multiple, verifiable, go/no-go checkpoints AI leverages when validating if the coding output is good. Passing each step doesn’t mean everything is always correct, but it gives the looping AI a means of confirming if something is wrong, and it can try again.

For the rest of enterprise use cases, AI has generally not been trained on the specific workflow, and the output of non-coding workflows tends to be more subjective and less verifiable than “does it run?” Accordingly, anecdotes from the Forum suggest a lack of expectations for non-coding use cases to demonstrate the same adoption that we have seen in coding.

The K-shaped innovation economy and the importance of small wins

Enterprise AI has bifurcated between a minority of nimble, tech-forward teams leveraging AI to “100x” their capabilities and the rest of the market largely focused on “efficiency.” On the latter, one source of underwhelming ROI has resulted from enterprise expectations that AI could accelerate entire tasks, but what they found was that they could only increase the efficiency of parts of tasks, which just put more work on someone else’s plate. As Mikhailov explained, “they spun one cog faster. The machine didn’t improve.” This also explains the gap between the touted productivity improvements at the individual level and the lack of similar improvement at the organizational level. What is needed, per Mikhailov, is a reimagining of workflows “in a very boring, software-oriented process-transformation manner.” In her article, The Most Lucrative Parts of AI Aren’t Sexy,² Dotson agrees, stating that “redesigning workflows and processes end-to-end” is necessary in order to create business value from AI.

Contrasting with the broad horizontal adoption of coding, Mikhailov and Meyerovich posit that achieving positive ROI through these reimagined workflows will accumulate “small wins” across “bite-size” projects and build an enterprise’s “intelligence capital.”

Models are good enough

As enterprises gain clarity around the problems they are solving with AI, both enterprises and non-model vendors point to AI capabilities that are sufficient for what enterprises need.

During their summer conferences, Databricks’ and Snowflake’s executives commented that existing models were already sufficiently capable, with the former observing, “AI does not have an intelligence problem … the question is how do we actually enable this at work,” and the latter stating, “differentiation is not access to the AI models … [it] is the access to the right data.” Similarly, Mosaic Specialist Ava Jodlowski recently led a conversation on data bottlenecks in AI, another theme from the Forum, which highlighted that enterprises don’t have an AI problem, but rather their “AI problems” are due to problems with data access, data silos, workflow designs, and maintaining workflows.

Closed- vs. open-source and good enough

Anecdotes during the Forum highlighted an increasing openness by enterprises to use open-source models, including Chinese models where there is no explicit prohibition against them. While interest in open-source models is not new, we haven't seen meaningful adoption in enterprises for two reasons.

First, the frontier was moving so quickly and closed-source models (e.g., models from Anthropic and OpenAI) were far enough ahead on that frontier, that enterprises were reluctant to miss future benefits by moving to open source. Second, AI budgets were largely unconstrained, and users defaulted to frontier models for ease of use. The former may be changing; the latter definitely seems to be changing.

One aspect of open- vs. closed-source models is less that open source is catching up to closed-source frontier models (though that is also happening), and more that the capabilities of frontier models increasingly exceed business needs. In other words, open-source models are “good enough” for many enterprise workloads.

Kierra Dotson, Associate Director of AI Strategy & Governance at Further, made this distinction recently in her article, The Frontier Model Premium Is Getting Harder to Defend.

That is, much of the value enterprises are seeking can be delivered without operating at the leading edge of model capability.

We’ve previously seen expectations for open-source models to take share from closed-source models; the excitement around open-source models and fine-tuning in 2024 was based on the same premise of open-source being “good enough.” Highlighting that things are different this time, we are currently seeing growing interest in non-frontier models play out even within model families. Financial Times recently noted, “[s]pending on Fable 5, Anthropic’s largest and priciest model, has plateaued at only about 11 percent of overall outlay on the company’s tools … [breaking] a pattern of corporate users defaulting to the most powerful models.”³

Model routing: Using the right model for the workload

Consistent with Financial Times’ reporting, enterprises are increasingly interested in model routing, where models are chosen based on the specific use case. While in the past, an enterprise might use frontier models for all workloads, they now want to reserve use of the most powerful – and most expensive – model for planning and reasoning, with individual tasks routed to smaller and cheaper closed-source models in the same family or open-source models.

Industry leader experts at the Forum noted that model routing “is confusing” and the shift to increasing routing will entail “significant change management,” but we also heard optimism that model routing could become table stakes in as soon as six months. Importantly, frontier labs offer model routing across their portfolios, and other vendors also offer routing capabilities, so it is not as binary as using one model family or undertaking complex DIY. At the same time, model routing introduces additional complexity, especially in terms of enterprises ensuring quality thresholds are maintained as underlying models change.

Security as a wild card

While the majority of the keynote discussion focused on non-technical aspects as the larger drivers of AI, there was one very large exception – security. During the keynote, Meyerovich, whose paper⁴ at Black Hat predicted the OpenAI/Hugging Face incident,⁵ expressed the greatest concern that the OpenAI agents were able to spend months finding vulnerabilities and coordinating, which represents a new type of threat.

Similarly, Delp noted that the new offensive capabilities of AI could require years of addressing tech debt, including patching decades worth of software. We also hear broad concerns about the security of the new, vibe-coded apps that are rapidly being produced, which exacerbates the problem. While positive ROI for productivity-enhancing enterprise AI has been elusive, the security threats posed by AI are very real, and as we move into a world of increasingly constrained AI spend, it is unclear how much AI budget may move to security.

Why it matters

Enterprise AI is entering a more demanding phase. Conversations at the Forum highlighted that the experimentation phase of enterprise AI is coming to an end, and as the focus shifts to driving positive ROI from individual workflows, the critical capability is shifting from which model is best to which capabilities are needed. Governance, cost discipline, employee adoption, and process redesign are becoming as important as model performance itself. The companies best positioned to succeed may be those that treat AI as an operational capability rather than a standalone technology initiative.