Eight Thoughts on the Age of Large Models: Computing Power, Data, Applications, and the Future

Eight Thoughts on the Age of Large Models: Computing Power, Data, Applications, and the FutureAt the 2024 Bund Finance Summit, Xiangyang Shen, Chairman of the Board of Governors of the Hong Kong University of Science and Technology and Foreign Member of the National Academy of Engineering, shared his eight thoughts on the opportunities and challenges of the age of large models, revealing a new era filled with both promise and uncertainty.Thought One: Computing Power is the ThresholdShen pointed out that computing power is the bedrock of large model development, its importance undeniable

Eight Thoughts on the Age of Large Models: Computing Power, Data, Applications, and the Future

At the 2024 Bund Finance Summit, Xiangyang Shen, Chairman of the Board of Governors of the Hong Kong University of Science and Technology and Foreign Member of the National Academy of Engineering, shared his eight thoughts on the opportunities and challenges of the age of large models, revealing a new era filled with both promise and uncertainty.

Thought One: Computing Power is the Threshold

Shen pointed out that computing power is the bedrock of large model development, its importance undeniable. Since 2010, the demand for computing power has been increasing at a rate of 6-7 times per year. While the growth has stabilized somewhat in recent years, it still maintains a 4-fold increase annually. This growth is primarily attributed to the continuous expansion of large model scale, with the increase in the number of parameters directly driving the need for computing power. Shen humorously remarked that it's difficult to be considered a large model company today without 10,000 cards. He further explained the speed and difficulty of computing power growth using Moore's Law and Huang's Law, highlighting the reality of the computing power threshold.

Thought Two: Reflections on Data

Shen delved into the sources and future trends of data used for training large models. He noted that the dataset size has grown exponentially, from 2 terabytes (TB) for GPT-3 to 12 TB for GPT-4, and an estimated 200 TB for GPT-5. However, the internet's data is nearing its mining limits, necessitating the search for new data sources. Shen believes that the 40 years of internet accumulation may be precisely for the arrival of the AI era.

Thought Three: The Next Chapter of Large Models: Embodied Intelligence

Shen believes that the future direction of large models lies in multi-modality and embodied intelligence. He acknowledged that while the current Sora model has made significant strides in multi-modality, its physical properties are not entirely reliable, making it impossible to build a complete "world model." In the future, large models need to evolve further towards embodied intelligence, with robots as one of their primary forms, with self-driving cars being a prime example. He believes that the combination of generation and understanding will be the key pathway to AGI.

Thought Four: Large Models Sweeping Across Industries

Shen stressed that large models will spark a profound revolution across industries, bringing unprecedented opportunities. He analyzed the computing power resources required for different types of large models: general-purpose models require tens of thousands of cards, industry models need thousands, enterprise models require hundreds, and personal models will emerge in the future. Using examples from companies like Lenovo, Microsoft, and Apple, he illustrated the trend of Personal Intelligence. Shen also shared data on currently registered large models in China, with 70% being industry models, a proportion that is expected to increase further in the future.

Thought Five: AI Agent - From Vision to Reality

Shen sees AI agents as the truly super-applications of the AI era, emphasizing their importance. He noted that although ChatGPT is powerful, it's still a long way from being a true agent. The emergence of agents will significantly enhance human productivity, while current GPTs can only achieve breakthroughs in specific areas. He believes that a real AI agent will require the seamless integration of workflow, industry analysis, large model application framework, platform, knowledge and skills, tasks, dialogue, and other aspects.

Thought Six: Prioritizing AI Governance

Shen emphasized the importance of AI governance and noted the differing approaches to AI governance across countries. He discussed the impact of AI on individuals, companies, governments, and societal development, including concerns about AI potentially influencing the outcome of US elections. He believes that future AI development must be from a global perspective, building sovereign AI with sovereign clouds as its foundation.

Thought Seven: Rethinking Human-Machine Relationships

Shen prompted us to contemplate the human-machine relationship. Is the impact of GPT a shock to human-computer interaction or progress in machine intelligence? He referenced John Markoff's book "Machines of Loving Grace," pointing out that the development of computer science over the past 50 years has followed two main lines: AI and IA (Intelligent Augmentation). He believes that the primary progress in the past few decades has been the breakthrough in human-computer interaction, with AI remaining a tool for a considerable time. He linked the development of technologies like Windows, search, and recommendation to the essence of human-computer interaction and pointed out that the combination of ChatGPT and Microsoft could potentially become the greatest company of the AI era.

Thought Eight: The Essence of Intelligence, the Century-Long Battle Between Neural Networks and Symbolic Systems

Shen pointed out that despite the rapid advancement of GPT, our understanding of intelligence remains limited. He drew an analogy between deep learning and physics, noting that deep learning has many unexplained problems and lacks robustness. He discussed the emergent intelligence of GPT-3 but acknowledged that the mathematical principles behind it are not yet clearly explained. He emphasized the importance of convening computer scientists from home and abroad last year at the Hong Kong University of Science and Technology to explore the mathematical principles behind emergent intelligence.

Shen concluded by saying that the development of AI is still in its early stages but has already led to numerous applications, with limitless possibilities for the future. His sharing provides us with valuable insights, helping us to better understand the opportunities and challenges of the AI era.