Inspired by the McCulloch-Pitts neuron, the perceptron was developed by Frank Rosenblatt in 1958. The perceptron is a type of artificial neuron that can learn to classify inputs into different categories by adjusting its weights based on error feedback. This learning capability marked a significant advancement from the fixed McCulloch-Pitts model, enabling the perceptron to adapt to new data.

Inspired by the McCulloch-Pitts neuron, the perceptron was developed by Frank Rosenblatt in 1958. The perceptron is a type of artificial neuron that can learn to classify inputs into different categories by adjusting its weights based on error feedback. This learning capability marked a significant advancement from the fixed McCulloch-Pitts model, enabling the perceptron to adapt to new data.

The Gemini paper by Google DeepMind introduces advanced multimodal AI models capable of processing text, images, audio, and video within a single framework.

The LLaMA (Large Language Model Meta AI) paper outlines Meta AI's development of large language models that excel in various NLP tasks.

Launched in Novemeber, 2022, it quickly became the fastest growing application in the human history, with 100 milion unique users in a record time. It also marks the beginning of the LLM race.

GPT-3.5, an improvement of OpenAI's GPT-3, features enhanced capabilities in natural language understanding and generation.

An upgrade to GPT-2. The GPT-3 model, with 175 billion parameters, achieved state-of-the-art performance on many NLP tasks by leveraging few-shot, one-shot, and even zero-shot learning.

An upgrade to GPT-1. It demonstrated that large-scale language models, such as GPT-2, can perform a variety of tasks without explicit supervision by leveraging vast amounts of text data for training.

Two neural networks that compete with each other. The architecture consists of a generator network that tries to fool the discriminator network, whose task it to discriminate generated data (for example images).

The first work that parallelized the computations required for the training of a CNN using a GPU (Graphical Processing Unit).

The first work that showed that (deep) neural networks can be used as a feature extractors. The values on the hidden neurons were used as an embedding of the input data, and similar data clustered closely using the learned representation.

An early instance of a successful gradient-based learning technique. It was also employed commercially for reading bank checks (several milion checks per day). It provides a detailed description of the used neural architecture, with step-by-step derivations.

A milestone work that tackled the problem of vanishing gradient when training deeper neural architectures. Allowed to train deeper architectures than ever before.

The first application of the backpropagation algorithm to a real world problem fed directly with images (previous approaches relied on supplying the feature vectors).

Proposed a new training algorithm for the Adaline algorithm.

Inspired by their construction in statistical physics, they were popularized in the cognitive sciences by Geoffrey Hinton, Terry Sejnowski and Yann Lecunn. Although their practical usability remained limited, under some conditions they remain useful.

First neural architecture with an ability to recognize stimulus patterns based on the geometrical similarity of their shapes without affected by their positions.

The GPT-4 Technical Report by OpenAI describes GPT-4 as a multimodal model that processes text and image inputs to generate text outputs.

The authors proposed an architecture designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.

The authors demonstrate that large gains on tasks such as textual entailment, question answering, semantic similarity assessment etc. can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text.

The authors present a layer-by-layer training method for Deep Belief Networks, using Restricted Boltzmann Machines to significantly enhance training efficiency and performance through unsupervised pre-training and fine-tuning.

The Transformer model employs an attention mechanism to process entire sequences simultaneously, enabling better parallelization of computation.

The authors extended the results presented in Highway networks paper to networks with 1000 layers. Until today, this is the most cited paper in machine learning.

First successful training of very deep neural networks, a precursor for ResNet. By introducing gated shortcuts, highway networks enabled the flow of information across layers without degradation, allowing for the effective training of networks with unprecedented depth.

The Seq2Seq paper by Sutskever, Vinyals, and Le introduced a model that maps sentences to vector representations and back using LSTM networks, setting a new standard for machine translation.

Alex Krizhevsky et al. managed to significantly improve on the previous results on the ImageNet challenge through parallelization of the training of their network on multiple GPUs.

The use of neural networks with more than 2 layers was unthinkable due to the lack of training methods. Networks in the 60s and 70s were not trained with the current backpropagation method, and while it was possible to tune weights in 1-layer networks, it was prohibitevely difficult to do so in deeper ones.

The authors demonstrate that using deep neural networks with many layers and numerous neurons per layer can significantly enhance the accuracy of handwritten digit recognition on the MNIST dataset.

A shallow two-layer neural network that is trained to reconstruct linguistic contexts of words. It was the first word embedding capable of preserving the context: it turned out that, for example (France – Paris = Italy – Rome).

The Turing machine, introduced by Alan Turing in 1936, is a theoretical model that formalizes the concept of computation, using an infinitely long tape and a set of rules to simulate any computer algorithm, thereby defining the limits and capabilities of what can be computed.

Up until this point it was widely assumed that it is close to impossible to train neural networks (or perceptrons) with more than 2 layers. It was also the first work that employed iterative weight update rule, although not the currently used backpropagation.

In this work, Alan Turing pondered the question whether machines can think. The novelty of his approach was to avoid a quicksand of thinking, and instead propose an equivalent 'test'. The idea was to include a human interrogator that would communicate with the interogees through a machine written text, and based on that interaction is supposed to judge who of the interogees is a man and who is a woman (in the original proposition). The same idea translates immediately to distinguishing between a human and a machine. Our approach in the Turing Game expands Turing's idea.