Don't want to rely on NVIDIA! Microsoft releases two self-developed AI chips that can train large models

On November 16th, Microsoft released its first self-developed artificial intelligence (AI) chip on Wednesday, which can be used to train large language models and break free from dependence on expensive NVIDIA chips. Microsoft has also built CPU based on Arm architecture for cloud infrastructure

On November 16th, Microsoft released its first self-developed artificial intelligence (AI) chip on Wednesday, which can be used to train large language models and break free from dependence on expensive NVIDIA chips. Microsoft has also built CPU based on Arm architecture for cloud infrastructure. These two self-developed chips are designed to provide power for Azure data centers and help the company and its corporate customers prepare for the arrival of the AI era.

Microsoft's Azure Maia AI chip and Arm architecture Azure Cobalt CPU will be launched in 2024. This year, the demand for Nvidia's H100GPU has surged, and these processors are widely used for training and running image generation tools and large language models. The demand for these GPUs is very high, even selling for over $40000 on eBay.

Rani Borkar, Director of Hardware Systems and Infrastructure at Microsoft Azure, explained in an interview that "Microsoft has a long history in chip development." Over 20 years ago, Microsoft collaborated with others to develop Xbox chips and even designed surface device chips. Boca said, "These efforts are all built on these experiences. In 2017, we started building the cloud hardware stack and embarked on this journey, putting us on the path of building new self-developed chips

The new AzureMaiaAI chip and AzureCobaltCPU are both built internally by Microsoft, and their entire cloud server stack has been thoroughly overhauled to optimize performance, power consumption, and cost. Boca said, "We are reconsidering cloud infrastructure in the era of artificial intelligence and truly optimizing every layer of that infrastructure

AzureCobaltCPU is named after the element "cobalt" (cobalt blue is an important blue pigment) and is a 128 core chip designed based on Arm's NeoverseCSS and customized for Microsoft. It aims to provide support for universal cloud services on Azure. Boca explained, "We put a lot of effort into not only making it high-performance, but also paying attention to power management. We made many conscious design choices for this, including the ability to control performance and power consumption on each kernel and virtual machine

Microsoft is currently testing the workloads of CobaltCPU, such as Teams and SQL servers, and plans to provide customers with virtual machines for various workloads next year. Although Boca did not directly compare it with the Graviton3 server provided by Amazon on AWS, there should be some significant performance improvements compared to Microsoft's current ARM based server for Azure. Boca said, "Our preliminary testing shows that our performance has improved by 40% compared to data centers currently using commercial Arm servers." Microsoft has not yet shared the complete system specifications or benchmarks.

Microsoft's Maia100AI accelerator is named after "Pleiades IV" (a bright blue star known in Greek mythology as Mia) and is designed to run cloud based artificial intelligence workloads such as large language model training and inference. It will be used to support some of the company's largest artificial intelligence workloads on Azure, including a $10 billion partnership with OpenAI. Microsoft will provide support for all workloads of OpenAI. This software giant has been collaborating with OpenAI in the design and testing of Maia.

OpenAI CEO Sam Altman said, "When Microsoft first shared their Maia chip design, we were very excited. We worked together to improve and test it using our models. Azure's end-to-end artificial intelligence architecture has now been optimized through Maia, paving the way for training more capable models and making them cheaper for our customers

Maia is manufactured using TSMC's 5-nanometer process and has 105 billion transistors, which is about 30% less than AMD's MI300XAIGPU (153 billion transistors). Boca said, "MAIA supports our first 8-bit data type, the MX data type, for collaborative design of hardware and software, which helps us support faster model training and inference time

Microsoft, along with companies such as AMD, Arm, Intel, Meta, NVIDIA, and Qualcomm, has joined an organization that is developing standards for the next generation of data formats for artificial intelligence models. Meanwhile, Microsoft is building on the collaboration and open work of Open Computing Projects (OCP) to adapt the entire system to the needs of artificial intelligence.

Boca revealed, "Maia is the first complete liquid cooled server processor manufactured by Microsoft, and our goal is to achieve higher server density with higher efficiency. As we are reimagining the entire stack, we have specifically considered each layer, so these systems are actually suitable for our current data center

This is crucial for Microsoft as it can start these artificial intelligence servers faster without having to make room for them again in data centers around the world. Microsoft has built a unique rack to accommodate the Maia server motherboard and is equipped with a liquid cooler called the 'Assistant'. Its working principle is like a radiator installed on a car or high-end gaming PC, which can cool the surface of the Maia chip.

In addition to sharing MX data types, Microsoft also shares its rack design with partners so that they can use them on systems with other chips internally. But the design of the Maia chip will not be shared more widely, and Microsoft will keep it internally.

Maia100 is currently undergoing testing on GPT3.5Turbo, and the model is supporting tools such as ChatGPT, BingAI, and GitHubCopilot. Microsoft is in the early stages of deployment, and like Cobalt, the company is not yet willing to release exact Maia specifications or performance benchmarks.

This makes it difficult for us to accurately interpret how Maia will compare with NVIDIA's popular H100GPU, recently released H200, and even AMD's latest MI300X. Boca also does not want to compare, but reiterates that cooperation with NVIDIA and AMD remains crucial for the future of Azure artificial intelligence clouds. Boca said, "Optimizing and integrating every layer of the stack at the scale of cloud computing operations to maximize performance, diversify the supply chain, and provide infrastructure choices for our customers is really important

Diversification of the supply chain is equally important for Microsoft, especially considering that Nvidia is currently the main supplier of artificial intelligence server chips, and companies have been competing to purchase these chips. It is estimated that in order to promote the commercialization of ChatGPT, OpenAI requires over 30000 Nvidia's old A100GPUs, and Microsoft's self-developed chips can help its customers reduce the cost of artificial intelligence. Microsoft has also developed these chips for its Azure cloud workload without intending to sell them to other companies such as NVIDIA, AMD, Intel, and Qualcomm.

Boca explained, "I think it's more like complementing each other than competing with them. Today, we use both Intel and AMD chips in the cloud computing field. Similarly, in the artificial intelligence field, we can also use both AMD and NVIDIA chips. These partners are very important to our infrastructure, and we really want to give our customers a choice

The naming of Maia100 and Cobalt100 indicates that Microsoft is already designing second-generation versions of these chips. Boca said, "This is a series that doesn't just end in one generation, but we won't share our roadmap." It's not yet clear how often Microsoft will release iterations of Maia and Cobalt, but considering the speed of artificial intelligence development, if the successor of Maia100 releases at a speed similar to the Nvidia H200 (about 20 months), there's no need to be surprised.

The key now is how quickly Microsoft will get Maia running, accelerate its AI ambitions, and how these chips will affect the pricing of AI cloud services. Microsoft is not yet ready to discuss the pricing of this new server, but we have seen the company quietly launch a Copilot for Microsoft 365, charging $30 per user per month.

Currently, the Microsoft 365 version of Copilot is limited to Microsoft's major customers, and enterprise users must promise to have at least 300 users to use this new artificial intelligence Office assistant. With Microsoft launching more Copilot features this week and rebranding BingChat, Maia may soon help fill the demand for artificial intelligence chips that support these new experiences. (Small)