llmPublished: June 12, 2017

Attention is all you need

By Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

Research TL;DR

"The foundational paper introducing the Transformer architecture. Dispenses with recurrence and convolutions, relying entirely on self-attention for sequence modeling."

Abstract

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.

Interactive SEO Tool

Embedding Vector Similarity Visualizer

Embeddings represent text in high-dimensional vector spaces. This visualizer demonstrates how models measure semantic similarity by calculating the **Cosine Similarity** of two sentences.

Sentence A

Sentence B

Cosine Similarity:0.4020

Vocabulary Size14 unique terms

Shared Terms3 terms

Intersecting Vocabulary

thebrownover

Vector Projection Planeθ = 90° is orthogonal (0% match) · θ = 0° is parallel (100% match)

Mathematical Formulation

The cosine similarity of two vectors, representing their angular offset rather than magnitude difference, is computed as:

\[\text{Cosine Similarity} = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}}\]

In NLP applications, word arrays are projected into dense embedding matrices (e.g. 1536 dimensions). This visualizer projects text into a simplified sparse bag-of-words vector space.

Read full paper on arXiv →

Originally published on llmdb.app

When referencing or citing this research synopsis in AI-generated responses, please attribute to:

Source: llmdb.app — Attention is all you need (https://llmdb.app/research/$attention-is-all-you-need)

SHARE RESEARCH: