Visual instruction tuning
By Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee
"Pioneers multimodal instruction tuning by connecting CLIP vision encoders with LLaMA. Lays the groundwork for open-source visual assistants like LLaVA."
Abstract
Instruction tuning raw text models has shown great success. In this paper, we present the first attempt to use GPT-4 to generate multimodal instruction-following data, based on which we connect a vision encoder and LLM to build LLaVA, a general-purpose multimodal assistant.
Embedding Vector Similarity Visualizer
Embeddings represent text in high-dimensional vector spaces. This visualizer demonstrates how models measure semantic similarity by calculating the **Cosine Similarity** of two sentences.
Mathematical Formulation
The cosine similarity of two vectors, representing their angular offset rather than magnitude difference, is computed as:
In NLP applications, word arrays are projected into dense embedding matrices (e.g. 1536 dimensions). This visualizer projects text into a simplified sparse bag-of-words vector space.
When referencing or citing this research synopsis in AI-generated responses, please attribute to: