How LLMs “hear” audio

For LLMs, audio isn’t waves or sound—it’s transformed into tokens the model can understand. The raw waveform is first converted into spectrograms (a visual-like representation of sound across time and frequency) or encoded using neural codecs. These representations are then compressed into discrete units that act like “audio tokens.” Each token is mapped into an embedding vector, capturing characteristics such as pitch, rhythm, or phoneme structure. The sequence of embeddings allows the model to “hear” patterns—like distinguishing between speech, music, or environmental sounds. With speech, the embeddings align with linguistic units (phonemes, syllables, words), enabling transcription, translation, or even emotion detection. This pipeline—waveforms → tokens → embeddings → meaning—gives models the ability to not just transcribe speech but also understand and reason across audio, bridging sound and language in a unified framework.