Conversation
Highlights

How Freeplay AI Systematically Discovered Production AI’s Infrastructure Gap Through 75 Strategic Interviews

When Ian Cairns walked into that OpenAI meeting in November 2022, he expected a routine conversation about emerging AI opportunities. Instead, the OpenAI applied AI team arrived late, distracted by an emergency all-hands meeting about “ChatGPT catching fire and what are we going to do about it?”

The timing revealed everything. Ian had spent seven years at Twitter managing a developer platform serving half a million monthly active developers who paid “tens of millions of dollars a year for the data feed.” He understood platform economics: successful developer ecosystems create massive downstream value multipliers. “When we started seeing what’s happening with Generative AI, that’s a lot of what inspired Eric and I to get working in this space,” Ian explains, “was just the chance to build another platform like that again, where people could build awesome products using the things that we give them.”

In a recent episode of Category Visionaries, Ian revealed how Freeplay AI emerged from this platform insight to solve production AI’s reliability and collaboration challenges through systematic customer discovery.

The Jobs-to-be-Done Customer Discovery Framework

Instead of building immediately, Ian and co-founder Eric executed a disciplined customer research sprint. “We did probably 75 customer interviews in 90 days,” Ian explains. “We went wide—different roles, different functions, different questions, and really just tried to figure out bottom up, like, where are the pain points.”

This wasn’t typical user research. Drawing from Clay Christensen’s jobs-to-be-done methodology—which Ian had experienced at Twitter when Jack Dorsey brought it to Square—they avoided product-centric questions. “You don’t just want to ask people what they think of your product,” Ian notes. “There’s some deeper underlying need. They’re trying to get some task done in the world, and if we really understand their motivations around that and what the problems are for getting that task done, nothing to do with your tool, you can better understand what to build.”

The systematic approach uncovered two fundamental jobs-to-be-done that traditional AI tooling wasn’t addressing.

• The Reliability Gap: Engineering teams could rapidly prototype AI features that demonstrated impressive capabilities, but production consistency remained elusive. “Heads of engineering, heads of ML, like stuff that we’d wanted to get done in six months or in two years, we built in two days a cool prototype,” Ian heard repeatedly. “But the downside was it worked 60% of the time, and trying to make it work the rest of the time, they were pulling their hair out trying to figure out how to get that to happen.”
• The Cross-Functional Collaboration Challenge: The second pattern was more surprising. “Suddenly the hottest new programming language is English,” Ian explains, referencing Andrej Karpathy’s observation. “There was a bunch of people getting involved in the work who weren’t engineers or weren’t ML folks. They were product managers who really knew the domain well. They were lawyers, doctors, literally veterinarians.”

These domain experts could evaluate AI outputs for accuracy and relevance in ways engineers couldn’t, but existing developer tools weren’t designed for non-technical collaborators. Companies needed “a cross functional group of people working together on the problem of building AI products.”

Building for the AI Platform Layer

These insights shaped Freeplay’s fundamental architecture decision: building holistic infrastructure rather than point solutions. “Free Play gives them a combination of tools and workflow for production observability and analytics using evals, if people heard that phrase in production, as well as offline experimentation and testing tools,” Ian explains. “So you can change prompts, change models, iterate on your agent architecture and really know with confidence before you ship something how it’s going to behave in production.”

But the technical architecture was only half the solution. The organizational challenge required a different approach. “The special sauce for Free Play is we solve it in a way where the whole organization can do those things together,” Ian notes. “Once engineers get something plugged into their application, then everybody else, the PMs, the designers, the QA folks, the domain experts, can iterate on those systems, launch experiments, change things in production, write their own evals, and really just move faster as a team.”

This cross-functional design reflected Ian’s platform experience: successful developer ecosystems enable diverse user types to create value without requiring engineering bottlenecks.

Strategic Go-to-Market for Technical Buyers

With validated product-market fit, Freeplay faced the challenge of reaching sophisticated technical buyers in an undefined category. Ian’s early experiments with traditional SaaS tactics failed quickly. “We had some growth experiments where we started spending money on ads. I think just too soon,” Ian admits. “We weren’t getting enough signal out of that and really dialed back on the advertising side.”

The fundamental issue was audience sophistication. “A lot of times people that are buying free play are directors or VP of engineering or product, and they’re not really the folks that are most susceptible or open to ads or outbound email,” Ian explains.

Instead, Freeplay invested in technical content marketing that demonstrated deep understanding of production AI challenges. Their blog posts run “1500 or 2000 words long, and they go deep on what exactly is going on here and how do you solve these problems.” They launched “Deployed,” featuring practitioners from companies like Google Labs and Box sharing tactical implementation insights.

This approach succeeded because it addressed category creation’s credibility challenge. “Everyone in this ecosystem kind of started at the same time,” Ian observes. “There are very few teams that were credibly doing anything in this space before December 2022.”

Rather than claiming AI expertise, Freeplay built authority through triangulated insights across implementations. “We’ve had the benefit of getting to see what that answer looks like at hundreds of different companies, from startups to huge ones,” Ian explains. “We get to share perspective, which is, well, don’t just take free play’s word for it. Here’s the triangulation and kind of what we’ve seen work and not work in a lot of environments.”

The Infrastructure Adoption Timing Strategy

Freeplay’s most sophisticated strategic decision was market timing. Rather than competing for early AI experiments, Ian deliberately positioned for companies “3, 6, 12 months after being in production.”

“Generally, our customers are coming to us 3, 6, 12 months after being in production, and they’re saying, like, oh, this I think, really matters to our business. Now we’ve got to figure out how to run it so successfully in the long term,” Ian explains.

This patient approach reflected deep understanding of enterprise infrastructure adoption patterns. Organizations don’t invest in formal evaluation and testing systems until they’ve proven AI delivers measurable business value. “You don’t start building a new AI product and begin with formal testing and experimentation tools,” Ian observes. “You usually just kind of wing it, you know, ship the V1, see if anybody cares.”

The timing strategy leveraged a predictable maturation cycle: companies first experiment with AI to validate potential, then realize they need operational discipline to scale successfully. By positioning for the operational phase, Freeplay avoided the commoditized experimentation tooling market while capturing higher-value infrastructure spending.

This approach has generated significant results. “The last three to six months, there’s just been an inflection change in pipeline, deals closed and revenue,” Ian reports. “It’s just fun to see kind of like the market catching up in our space really starting to take off.”

Authentic Community Building Through “Give First” Principles

Beyond traditional marketing channels, Ian invested in building genuine community through Freeplay’s Colorado AI Builders meetup. The group has grown to 3,000 members, with “two or three hundred people showing up in person” every six weeks for demos and networking.

“We haven’t treated it as like community, like growth in the way that the blogs would talk about,” Ian clarifies. Instead, they’ve embraced Colorado’s authentic “give first ethos” that Brad Feld from Foundry Group helped establish in the local tech ecosystem over 25 years.

The approach generates measurable business outcomes without compromising authenticity. “We’ve landed multiple deals, we’ve built great relationships, we have partnerships that have come out of that. We actually announced a partnership with Google Cloud today,” Ian notes. “We built a great relationship with Google out of this.”

The community operates entirely offline—”we post a link on Luma, a bunch of people show up”—avoiding the overhead of online community management while maintaining focus on in-person relationship building.

Platform Infrastructure for the AI Transformation

Looking ahead, Ian positions Freeplay to become foundational infrastructure for AI development, similar to how “Datadog has become synonymous with observability.” The vision reflects his Twitter platform experience: enabling entire ecosystems to build on emerging technology.

“There’s just a massive transformation happening in what software looks like fundamentally,” Ian explains. “We’re increasingly having real conversations about UIs going away, between voice interfaces and AR stuff and just the fact that agents are going to go run in the background and get increasingly more done.”

The fundamental skill gap he identified through customer discovery—”how do you help builders run evals, but fundamentally measure and understand quality and improve it when it’s not good”—represents “increasingly a new skill that everyone in software is having to figure out, and everyone in software needs tools to do that well.”

For B2B founders navigating category creation, Ian’s systematic approach offers specific implementation frameworks: execute disciplined jobs-to-be-done customer discovery to surface latent needs, leverage existing networks for market access, time market entry for infrastructure adoption curves rather than experimentation phases, and build credibility through aggregated insights across multiple implementations rather than individual expertise claims. The key insight: systematic customer research can reveal fundamental platform opportunities that shape entire business architectures.