Conversation
Highlights

How TwelveLabs Built Video AI Infrastructure by Rejecting Every Shortcut

The transformer architecture had been public for roughly a year when Jae Lee and his co-founders started mapping out their thesis in South Korea’s Cyber Command. While serving together doing multimodal video understanding research, they watched the AI landscape fracture into clear camps: teams scaling language models, teams adapting transformers for images, and essentially nobody building purpose-built infrastructure for video.

The obvious play was to wrap existing models, ship fast, and ride the wave. They chose the opposite: build proprietary video foundation models from scratch, knowing it would take years while competitors collected headlines and funding announcements.

In a recent episode of Category Visionaries, Jae shared how TwelveLabs achieved 10x revenue growth by making GTM decisions that felt wrong in the moment—turning down large deals, maintaining hiring standards during crunch periods, and spending the first two and a half years building infrastructure while others “flew by” with demos. The result is a company processing petabytes of video content for media companies, federal agencies, and enterprises that need to index 2 million hours of footage in two days.

Why Video Required Different Infrastructure, Not Adapted Models

The technical bet was specific from the beginning. “Our belief was, yeah, video seems very different and it probably deserves like its own research and its own inference like infrastructure,” Jae explains. “So someone should be building it from the ground up.”

This wasn’t about feature differentiation. Video’s temporal dimension, spatial complexity, and data volume requirements meant that approaches optimized for text or static images would hit fundamental performance walls at production scale. The team’s military research had already surfaced these constraints.

Their thesis extended beyond the models themselves. “We need to build an incredible indexing infrastructure,” Jae emphasizes. “We need to build purpose built proprietary foundation models. And that can’t be super large because we can’t break our customer’s bank.”

This created a three-year problem: build the full stack—models, indexing, inference infrastructure—while competitors demonstrated immediate utility with wrapped solutions. The decision to ignore that pressure came from observing where AI research itself was moving. “We’ve kind of entered a strange era where a lot of meaningful research work, especially in AI, could probably happen in private setting, especially now that it requires a lot of compute data, the kind of resources that you would normally not be able to access in school settings.”

Inverting the Sales Team for Established Markets

TwelveLabs’ GTM architecture reflects a sophisticated read on how technical infrastructure sells into mature industries. Their targets—media, entertainment, sports, advertising, defense—present a specific challenge. As Jae describes: “These are traditional markets, like really big markets. And that’s been around for many, many years. Meaning there’s lots of players. There are also lots of norms and layers and stacks.”

The team structure inverts conventional wisdom. “Our head of go to market who’s also the co founder, you know and our revenue leader, VP of revenue, they can sell any technology so they’re a generalist,” Jae explains. “But you know the AES, the solutions architects, the four deployment engineers that we hire are highly specialized.”

These front-line specialists come directly from the target verticals. “They come from these four different key industries. So they’re used to selling into these industries. They notice that our potential customers are excited about and they know exactly how to plug twelvelabs into our customers workflow.”

This structure solves a specific problem in established markets: technical buyers can immediately distinguish genuine domain fluency from trained sales messaging. When an AE understands media asset management workflows or post-production pipelines without explanation, deal velocity accelerates because you’re not fighting credibility battles on every call.

The second structural decision was positioning as infrastructure layer, not complete solution. “For our go to market, partnership driven distribution of our product is very much important,” Jae notes. Integration with media-specific GSIs, asset management platforms, and cloud providers means TwelveLabs fits into existing workflows rather than forcing migration.

The AWS Bedrock integration demonstrated this approach at scale: over 30,000 enterprise agreements in seven weeks. That distribution velocity is impossible through direct sales alone, regardless of team size.

What Federal Actually Requires Beyond Process

TwelveLabs’ federal business illustrates why opportunistic market expansion fails in government. “If you’re kind of entering because, oh, federal market is big and you go in, you’re going to get your butt kicked,” Jae states directly. “So I think like you need to actually build your team in a way that’s like passionate to work on this project.”

The distinction matters because federal sales cycles involve sustained engagement through security reviews, deployment complexity, and procurement processes that can span quarters. That momentum only comes from genuine conviction, not quota pressure.

For TwelveLabs, alignment was structural. The founders’ military backgrounds, In-Q-Tel as an investor providing interface to agencies, and core company values around mission created authentic credibility. “This is hard because you kind of have to believe that your technology and product is dual use,” Jae explains. “So you have to initially set up your team in a way that, you know, the folks, whether you’re in GTM or engineering or research, are fully bought in.”

The military culture shaped internal operations. “You know, this kind of like we treat almost every day as like a battle. And then the guys that’s next to you are your foxhole buddies,” Jae describes. “So we have this mission critical and we need to like serve our nation and be patriotic.”

That cultural foundation enables a different conversation with government buyers—one based on shared mission rather than vendor-customer transaction. It’s not replicable through hiring or training if it’s not embedded in company DNA.

The Three Failure Modes of Wrong-Fit Customers

The discipline of saying no to revenue created immediate pain but protected long-term positioning. “Early on saying no to customers is hard,” Jae acknowledges. “Because as a founder you want to grow your business and you know that’s going to be good for the morale. But that’s only true when the customers are actually their ideal customers.”

Wrong customers create three specific failure modes. First, they pull product roadmap toward one-off features that don’t serve core ICP needs. Second, they consume disproportionate support resources because your product wasn’t built for their use case. Third, they generate reference cases that attract more wrong-fit prospects, compounding the problem.

For infrastructure companies, this matters more than for application layer products. Every customer shapes your market position and future pipeline composition. Early customers particularly set expectations about what problems you solve and for whom.

The same discipline applied to hiring under pressure. With the team regularly working until 11pm and deadlines slipping, the temptation to hire adequate candidates was constant. “Okay, we need more mles,” Jae describes the internal pressure. “We weren’t fully satisfied, but let’s bring someone on, right? Because otherwise we won’t be able to hit XYZ deadlines.”

Leadership held the line. “You can literally see your people staying up until like 11pm like every day,” Jae acknowledges the difficulty. “But I think that’s also part of the journey and making sure you, the co founders, the extended leadership are fully aligned that this is going to be painful, but lowering the bar will kill the company.”

Why First-Principles Development Compounds

The payoff from years of focused building came when customers arrived with production requirements that demos couldn’t handle. “Hey Jay, we have 2 million hours of content,” Jae recounts actual customer requests. “We need to be able to index it in the next two days and I need to be able to search, analyze and I need to build really amazing agents workflow that can help you generate highlights or reports of some footages.”

Wrapped solutions optimized for impressive demos fail at this scale. Purpose-built infrastructure handles it because the engineering investments went into indexing performance, inference efficiency, and workflow integration—the unglamorous foundation work that doesn’t generate viral demos.

“It was painful journey in the first like two and a half, three years because folks are flying by,” Jae reflects on watching competitors gain momentum. “And we’ve been grimly building what we think is right and what the world deserves. But I think that is like finally kind of paying off.”

The company is now achieving 10x revenue growth while expanding beyond initial verticals into healthcare, manufacturing, and education—validation that video understanding infrastructure has applications across industries, not just the original four target markets.

The Infrastructure Endgame

Jae’s vision extends the infrastructure layer thesis to its logical conclusion. “Our mission, again, is to become the universal video intelligence layer for all videos in the world,” he explains. Current enterprise customers with petabytes of existing content are the initial market, but the real opportunity is ubiquity.

“In the next, I think, three years, you know, I think the world will see 40 or 50 billion devices with cameras,” Jae projects. “And it’s not just cameras. It’s going to have a very capable chip right next to it.”

TwelveLabs aims to be the infrastructure layer that makes all that video queryable and analyzable. “I think 12 labs can become that company that bridge the digital and the physical world. And we’re really excited to serve as the visual intelligence layer.”

For B2B founders selling infrastructure into established markets, the TwelveLabs trajectory illustrates an alternative to fast-follower tactics: build fundamental technology that solves production problems, structure GTM for domain credibility over volume, and maintain discipline on ICP and hiring even when it hurts. The companies that sprint past you with demos often can’t deliver when customers need production-grade infrastructure.

Sometimes the longest build cycle creates the widest moat.