https://venturebeat.com/2020/05/19/openai-microsoft-azure-supercomputer-ai-model-training/ Roughly a year ago, Microsoft announced it would invest $1 billion in OpenAI to jointly develop new technologies for Microsoft’s Azure cloud platform and to “further extend†large-scale AI capabilities that “deliver on the promise†of artificial general intelligence (AGI). In exchange, OpenAI agreed to license some of its intellectual property to Microsoft, which the company would then commercialize and sell to partners, and to train and run AI models on Azure as OpenAI worked to develop next-generation computing hardware.
The new Azure-hosted, OpenAI-co-designed machine contains over 285,000 processor cores, 10,000 graphics cards, and 400 gigabits per second of connectivity for each graphics card server. It was designed to train single massive AI models, which are models that learn from ingesting billions of pages of text from self-published books, instruction manuals, history lessons, human resources guidelines, and other publicly available sources. Examples include a natural language processing (NLP) model from Nvidia that contains 8.3 billion parameters, or configurable variables internal to the model whose values are used in making predictions; Microsoft’s Turing NLG (17 billion parameters), which achieves state-of-the-art results on a number of language benchmarks; Facebook’s recently open-sourced Blender chatbot framework (9.4 billion parameters); and OpenAI’s own GPT-2 model (1.5 billion parameters), which generates impressively humanlike text given short prompts.
From the article, in recent years the amount of compute power needed to research AI at the cutting edge has been doubling approximately every four months. On the other hand, efficiency gains have seen the amount of compute power needed to obtain any given result halving every sixteen months. Therefore I would expect this kind of endeavour to continue to climb further and further out of reach of mere hobbyists like us for the foreseeable future, even taking into account Moore's law which observes that available compute power doubles every couple of years (or something like that).
However, there's no guarantee that this is the way to artificial general intelligence and certainly no guarantee that it's the only way. The best bets in my view are on some combination of symbolic and stochastic methods and the former are still well and truly within our reach.
