How LLMs “see” video

To an AI, a video isn’t a continuous stream—it’s a sequence of images (frames) plus audio. Each frame is broken into patches and converted into visual embeddings, while the audio track is tokenized into audio embeddings. The result is a time-ordered series of tokens that combine sight and sound. What makes video unique is the temporal dimension. Models must capture not just what’s in each frame, but also how things change over time—a ball moving across a field, a person waving, a scene shifting. Specialized architectures track these dynamics, aligning frame embeddings with audio embeddings to create a cohesive representation of events. This pipeline—frames + sound → tokens → embeddings → meaning—allows multimodal models to “understand” actions, summarize clips, or answer questions about what’s happening. It’s the foundation for video search, highlights generation, and richer AI-driven storytelling.