A staggering amount of money is pouring into data center AI chip companies at the moment. Is the hype justified?
Data center AI chip companies are raising eye-watering amounts of money. In the last week, we’ve seen Groq announce a $300 million Series C round of funding, and SambaNova raise a staggering $676 million Series D. SambaNova is now valued at somewhere above $5 billion.
They are not the only ones in this sector raising these huge amounts of money. Fellow data center AI chip companies Graphcore (raised $710 million, valued at $2.77 billion) and Cerebras (raised more than $475 million, valued at $2.4 billion) are hot on their heels as the sector continues to gain momentum.
It seems data center AI chip companies are the latest hot sector for investors. What is driving them to pour so much money into these companies? And are these levels of investment justified, or is the whole sector becoming a victim of its own hype?
“The hype around AI investments is not overdone when one considers that the opportunity in AI is a once-in-a-lifetime event that will create billionaires out of engineers and investors,” industry analyst Karl Freund of Cambrian AI told EE Times. “However, all startups will need three things to be viable competitors, in addition to funding and a great team: a 5-10X advantage over Nvidia, software to provide an easy migration path, and a roadmap of innovations to keep them ahead.”
One big driver for investment is the size of the market opportunity, which is universally believed to be huge.
“In the computer world, there are very few examples of growth like Nvidia has seen in their data center AI business,” Cerebras CEO Andrew Feldman told EE Times. “It went roughly from zero to billions in about four or five years. That’s unheard of market size growth, number one, and number two, they did it with a part designed for a different market.”
Feldman’s point is Nvidia’s success points to a huge opportunity for chips that are specially designed for AI acceleration, like Cerebras’ wafer-scale engine.
“AI has begun to touch us in our lives and sneak in, in many ways we haven’t thought of… it hasn’t even had the big impact yet,” Feldman said. “I think we have a very large and rapidly growing market. We have proof that it exists, it’s not pie in the sky, it’s growing faster than any other market in any other segment of compute, real results are being demonstrated. And usually when a new market emerges, historically new companies have a chance to rise up and be born.”
Feldman compared the current boom in data center AI companies to the rise of Ethernet companies in the mid-1990s, the rise of cell phone compute which gave us companies like Arm and Qualcomm, and the emergence of specialized graphics chip companies which produced, amongst others, Nvidia.
“What happens is, when a new workload emerges, new great companies are born, and that’s what investors believe they can take a shot at. And that’s why we we’ve raised so much money,” he said.
The incumbents in this market, Intel and Nvidia, are two of the biggest companies in the semiconductor industry, with resources that even the best-funded startups can only dream of. Does going into battle with these companies necessarily require a certain level of resources? And to what extent does the amount of resources available determine who will win in this market?
“I think if it’s determined by who has the most funding, then Nvidia has already won,” said Groq CEO Jonathan Ross, in an interview with EE Times. “Any startup [founder] relying on that might want to consider an alternative career choice.”
Ross’s pitch has Groq relying on its technological innovation and simplified architecture, having raised a relatively modest $367 million to date.
“It’s not just [us] versus the large companies, it’s also versus the startups… we work much more efficiently than some of those other startups out there,” he said. “We’ve often seen startups borrowing from the large company playbook of trying to brute force how they develop chips, and they’re often incremental, very similar to other architectures. And what we focused on is a completely new architecture and being able to do it with a much smaller, but very talented group of individuals.”
Money pouring into data center AI chip companies is a reflection of the excitement surrounding the industry today, said Ross. The same excitement is attracting talent away from the software industry back into the hardware world.
“The caution I would offer is that there’s a limited capacity to use those dollars efficiently,” he said. “And so there’s only going to be a couple of companies, no matter how well all of them are funded that can reach critical mass in terms of the talent needed to build some of the world’s fastest chips, [no matter] how much money you pour in.”
Analysts agree there is room for more than one AI chip winner in the data center sector.
“Nvidia’s GPU and software solution has been the early leader in AI processing, but now others are coming and seeing the potential [total available market] and the opportunity validated by Nvidia,” said Shane Rau, research vice president for computing semiconductors at IDC. “The problem to be solved is so vast that probably no one company, no one chip type, no one approach is ideal for any data processing problem that you might throw at it.”
IDC’s figures have the total AI chip market (including data center and edge/endpoint) at $88 billion by 2025. Rau said that while there are as many use cases for AI as there are people on the planet, IDC is tracking 17 different workloads for servers, though these 17 could be subdivided further into individual use cases, each of which could potentially be best served by a different hardware solution.
“There will not be one winner,” said Rau. “Eventually AI processing will be ubiquitous. If you look at any system – data center, PC or otherwise – eventually it will have to do some form of AI processing, so that does suggest that there’s room for many solutions. As with any market there’s usually a ‘Wild West’ mentality when it first gets started, and then there’s a move towards eventual consolidation.”
The size of the market opportunity has not been overstated, Linley Gwennap, principal analyst at the Linley Group, told EE Times.
“Certainly, the market that we’re talking about is a big market and is growing very rapidly,” he said. “Nvidia is making almost all of the money right now, but in five or ten years, the market will be more diverse, there’ll be room for other companies to participate. SambaNova or Groq or somebody else could grab a sizeable share of this market and could make billions and billions of dollars in revenue. That would be the justification for the investment [levels that we’re seeing today].”
Gwennap described the data center AI chip market as very different from smartphone chips or PC chips, in which one company or architecture has come to dominate.
“In those markets, there’s been a strong software environment where all the apps had to be compatible,” he said. “In the AI market, Nvidia has a strong position with CUDA, but AI models are being developed with applications like TensorFlow or PyTorch, that can easily be made to run on different [hardware] architectures. So if there is value in using a chip other than Nvidia, it’s fairly easy to plug in, say, a Groq card and start running your neural network.”
This makes the barrier to entry for data center AI chip companies lower than for PCs and smartphones. Gwennap sees a future landscape with diverse players, similar to the situation in the microcontroller market today.
Data center versus edge
While there is also plenty of money being pumped into edge and endpoint AI chip companies, it’s not in the same league as we’re seeing with data center AI chip companies and the investment they are attracting. Does it take significantly more money to develop data center AI chips than it does for edge AI chips?
“It’s certainly more expensive [to develop] data center chips,” Gwennap said. “They’re bigger, they’re more complicated, they have to deliver higher performance, the software is much more complex. That being said, the amount of money being invested in companies is probably more than they really need… this is money to fund their current chip, their next chip, and whatever software they need.”
In SambaNova’s case, the company has developed a whole system around its chip, which will incur some extra expense, though the vast majority of the expenses are still likely to be around development of the company’s unique chip and software stack, Gwennap said.
The staggering amounts being poured into these companies may be driven from the investment side, particularly with the big funds involved, rather than solely based on how much money companies need to get a product to market.
“When you have a billion dollars to invest, you can’t invest one million dollars in a company,” Gwennap said, adding that funds are keen to show their investors that they are participating in this very hot market.
Hot markets like this are not unprecedented. Gwennap compared the current level of hype surrounding data center AI chip companies to the GPU boom of the late 1990s, and network processors in the 2000s. Outside of the semiconductor industry, he pointed to electric vehicles, pharmaceuticals and biotech as sectors also attracting a lot of investment.
“As soon as you get a successful exit, everyone wants to jump in,” he said. “It does eventually lead to investments that are too big, or values that are too high for reality, but I’m not sure we’re there yet [with data center AI chip companies], even if we’re headed in that direction.”
We haven’t seen all that many successful exits in this sector so far. Startup Nervana was acquired by Intel back in 2016 for a sum believed to be around $400 million. Habana Labs was acquired for around $2 billion at the end of 2019, also by Intel. While these figures serve to validate Intel’s view of the market opportunity for data center AI chips, they also no doubt serve to validate valuations for Habana’s competitors.
“You can look at the deal sizes for some of the other AI companies that have been snapped up and balance that against the investments,” said Nina Turner, research manager and worldwide semiconductor applications forecaster at IDC. “The exits strategies, as well as: is the end-market really confined to just the data center, or do you have to prove out yourself in the low-hanging fruit arena and then grow into the emerging applications?”
Turner refers to the data center as ‘low-hanging fruit’ since that’s where the majority of revenues for AI chips are today. However, she says, as big as the data center market opportunity is, there may be additional revenues to be made in other applications. Success in the data center could pave the way for success in edge infrastructure, and even in endpoint AI applications further down the line.
“If you look at any startups that are targeting the [data center] AI market, I don’t think they’re only going to be concerned about the data center AI market in the long run,” she said.
Of course, to get there, a company would first have to succeed in the data center, and that battle is far from over.
This article was originally published on EE Times.
Sally Ward-Foxton covers AI technology and related issues for EETimes.com and all aspects of the European industry for EETimes Europe magazine. Sally has spent more than 15 years writing about the electronics industry from London, UK. She has written for Electronic Design, ECN, Electronic Specifier: Design, Components in Electronics, and many more. She holds a Masters’ degree in Electrical and Electronic Engineering from the University of Cambridge.