The rise of Generative AI has been nothing short of meteoric, transforming industries and capturing the public's imagination. At the heart of this revolution lies a technological marvel known as the Transformer, and one of its key architects is Illia Polosukhin.
Polosukhin is among the "Transformer 8," a group often hailed as the founding fathers of generative AI. Their groundbreaking paper, "Attention is All You Need," published in 2017 while they were at Google, introduced the Transformer architecture. This innovation has become the bedrock of many AI systems we use today, including large language models (LLMs) like ChatGPT.
The Transformer architecture departed from previous sequential processing methods. Instead of reading words one after another, it leveraged parallel processing and an "attention mechanism" to weigh the importance of different parts of the input simultaneously. This allowed the model to understand context and relationships within the data more effectively, leading to significant improvements in tasks like language translation and text generation.
According to Polosukhin, the initial prototype that he built at Google was the first transformer. The idea was that you don't need to read all the words sequentially, and you can leverage the fact that computers are highly paralyzable, especially GPUs and TPUs, to read everything in parallel. Then, you can reason about it with the attention mechanism.
Despite the Transformer's monumental impact, Polosukhin left Google shortly before the paper's publication. He co-founded NEAR, an AI startup and blockchain ecosystem. His vision was to create a platform for "user-owned AI," where individuals have more control over their data and the AI models they use.
Polosukhin believed in the power of open source. He advocated for it within Google and built NEAR on open-source principles. He felt that open-sourcing the Transformer would allow it to be leveraged as a platform for further innovation.
At NVIDIA's GTC AI Conference in San Jose, California, in March 2025, Polosukhin presented new research from NEAR AI on decentralized confidential machine learning. This approach enables users to harness customized intelligence without relinquishing data to closed, proprietary models. This work provides a technical foundation for an AI economy. Polosukhin's presentation addressed critical challenges such as privacy, the need for open-source models, sustainable economics for model creators, and community collaboration in AI development.
While the Transformer architecture has achieved remarkable success, Polosukhin and other researchers are exploring potential limitations and alternatives. For example, transformers can struggle with intellectual tasks needing multiple steps of combining information and often resort to finding patterns and shortcuts instead of truly understanding the problem.
The rise of "ChatGPT wrapper" companies, which build product layers on top of existing models like OpenAI's GPT-4, has also taken him by surprise. He believes that the next generation of models will likely be able to perform those tasks natively, rendering many of these wrappers obsolete.
Beyond the Transformer, researchers are exploring new AI architectures that could revolutionize large language models. Google's "Titans" and Sakana AI's "Transformers Squared" are two such examples, drawing inspiration from the human brain to improve AI systems' reasoning and efficiency.
Polosukhin's journey from co-creating the Transformer to championing user-owned AI reflects a deep commitment to shaping the future of technology. His work continues to push the boundaries of what's possible with AI, ensuring that its benefits are widely accessible and ethically grounded.
