Module 17 — Speech Transformers

Introduction

Speech Transformers extend the Transformer architecture to process audio signals instead of text or images. They convert speech into meaningful representations for tasks such as speech recognition, translation, synthesis, and audio classification.

Traditional Automatic Speech Recognition (ASR) systems relied on Hidden Markov Models (HMMs) and recurrent neural networks. Modern speech models use Transformers to capture long-range dependencies in audio sequences and achieve state-of-the-art performance.

Speech Transformer models power applications such as:

• Speech-to-Text (ASR)
• Speech Translation
• Text-to-Speech (TTS)
• Speaker Recognition
• Audio Classification
• Voice Assistants
• Meeting Transcription
• Podcast Captioning

In this module, you'll learn:

• Whisper
• Wav2Vec2
• SpeechT5
• Audio Spectrogram Transformer (AST)
• Speech Recognition
• Speech Translation
• Build a Speech-to-Text application

Speech Transformer Pipeline

Most speech Transformer models follow this workflow:

textCopy
1                Audio Input
2                     │
3             Waveform (.wav)
4                     │
5        Feature Extraction
6 (Log-Mel Spectrogram / Raw Audio)
7                     │
8            Transformer Encoder
9                     │
10      (Optional Decoder for Seq2Seq)
11                     │
12             Text Generation
13                     │
14          Transcribed Speech

Understanding Audio Representation

Unlike text models that process tokens, speech models first convert audio into numerical features.

Typical pipeline

textCopy
1Audio Waveform
2
3↓
4
5Sampling
6
7↓
8
9Spectrogram
10
11↓
12
13Feature Extraction
14
15↓
16
17Transformer

Example

textCopy
1"Hello World"
2
3↓
4
5Waveform
6
7↓
8
9Mel Spectrogram
10
11↓
12
13Transformer
14
15↓
16
17"Hello World"

1. Whisper

What is Whisper?

Whisper is an encoder-decoder Transformer developed by OpenAI for robust speech recognition and speech translation.

It is trained on hundreds of thousands of hours of multilingual audio and supports many languages.

Applications

• Automatic Speech Recognition (ASR)
• Speech Translation
• Subtitle Generation
• Podcast Transcription
• Meeting Notes

Architecture

textCopy
1Audio
2
3↓
4
5Log-Mel Spectrogram
6
7↓
8
9Transformer Encoder
10
11↓
12
13Transformer Decoder
14
15↓
16
17Generated Text

Load Whisper

pythonCopy
1from transformers import (
2    AutoProcessor,
3    WhisperForConditionalGeneration
4)
5
6model_name = "openai/whisper-small"
7
8processor = AutoProcessor.from_pretrained(model_name)
9
10model = WhisperForConditionalGeneration.from_pretrained(model_name)

Speech Recognition Example

pythonCopy
1import torch
2import librosa
3
4audio, sr = librosa.load(
5    "speech.wav",
6    sr=16000
7)
8
9inputs = processor(
10    audio,
11    sampling_rate=16000,
12    return_tensors="pt"
13)
14
15predicted = model.generate(
16    inputs.input_features
17)
18
19text = processor.batch_decode(
20    predicted,
21    skip_special_tokens=True
22)
23
24print(text[0])

2. Wav2Vec2

What is Wav2Vec2?

Wav2Vec2 is a self-supervised speech representation model developed by Meta.

Unlike Whisper, it learns directly from raw audio waveforms without requiring large amounts of labeled data.

Pipeline

textCopy
1Raw Audio
2
3↓
4
5Feature Encoder
6
7↓
8
9Transformer Encoder
10
11↓
12
13CTC Head
14
15↓
16
17Text

Advantages

• Self-supervised learning
• Strong ASR performance
• Efficient fine-tuning
• Low-resource language support

Load Wav2Vec2

pythonCopy
1from transformers import (
2    AutoProcessor,
3    Wav2Vec2ForCTC
4)
5
6model_name = "facebook/wav2vec2-base-960h"
7
8processor = AutoProcessor.from_pretrained(model_name)
9
10model = Wav2Vec2ForCTC.from_pretrained(model_name)

Speech Recognition Example

pythonCopy
1import librosa
2import torch
3
4speech, rate = librosa.load(
5    "speech.wav",
6    sr=16000
7)
8
9inputs = processor(
10    speech,
11    sampling_rate=16000,
12    return_tensors="pt"
13)
14
15with torch.no_grad():
16    logits = model(**inputs).logits
17
18predicted_ids = torch.argmax(
19    logits,
20    dim=-1
21)
22
23transcription = processor.batch_decode(
24    predicted_ids
25)
26
27print(transcription[0])

3. SpeechT5

What is SpeechT5?

SpeechT5 is a unified Transformer model that supports multiple speech and text tasks.

Capabilities

• Speech-to-Text
• Text-to-Speech
• Speech Translation
• Voice Conversion
• Speech Enhancement

Architecture

textCopy
1Speech
2
3↓
4
5Shared Encoder
6
7↓
8
9Shared Decoder
10
11↓
12
13Task Head
14
15↓
16
17Output

Advantages

• Multi-task learning
• Shared representations
• Flexible architecture

Load SpeechT5

pythonCopy
1from transformers import (
2    SpeechT5Processor,
3    SpeechT5ForSpeechToText
4)
5
6processor = SpeechT5Processor.from_pretrained(
7    "microsoft/speecht5_asr"
8)
9
10model = SpeechT5ForSpeechToText.from_pretrained(
11    "microsoft/speecht5_asr"
12)

4. Audio Spectrogram Transformer (AST)

What is AST?

The Audio Spectrogram Transformer (AST) applies the Vision Transformer (ViT) concept to audio.

Instead of processing raw waveforms, AST processes spectrogram images.

Pipeline

textCopy
1Audio
2
3↓
4
5Spectrogram
6
7↓
8
9Image Patches
10
11↓
12
13Transformer Encoder
14
15↓
16
17Audio Classification

Applications

• Sound Classification
• Environmental Audio
• Music Classification
• Acoustic Event Detection

Load AST

pythonCopy
1from transformers import (
2    AutoFeatureExtractor,
3    ASTForAudioClassification
4)
5
6feature_extractor = AutoFeatureExtractor.from_pretrained(
7    "MIT/ast-finetuned-audioset-10-10-0.4593"
8)
9
10model = ASTForAudioClassification.from_pretrained(
11    "MIT/ast-finetuned-audioset-10-10-0.4593"
12)

5. Speech Recognition (ASR)

Automatic Speech Recognition converts spoken language into text.

Pipeline

textCopy
1Microphone
2
3↓
4
5Audio Features
6
7↓
8
9Transformer
10
11↓
12
13Decoder
14
15↓
16
17Text

Example

textCopy
1Audio
2
3↓
4
5"Welcome to Transformers"
6
7↓
8
9Transcription
10
11↓
12
13Welcome to Transformers

Common Models

• Whisper
• Wav2Vec2
• SpeechT5

6. Speech Translation

Speech Translation converts speech in one language into text (or speech) in another language.

Pipeline

textCopy
1Spanish Audio
2
3↓
4
5Speech Encoder
6
7↓
8
9Translation Decoder
10
11↓
12
13English Text

Example

textCopy
1Input
2
3Hola, ¿cómo estás?
4
5↓
6
7Output
8
9Hello, how are you?

Whisper supports multilingual speech translation directly.

Comparison of Speech Transformer Models

Practice 1 — Speech-to-Text with Whisper

pythonCopy
1from transformers import pipeline
2
3transcriber = pipeline(
4    task="automatic-speech-recognition",
5    model="openai/whisper-small"
6)
7
8result = transcriber("speech.wav")
9
10print(result["text"])

What You'll Learn

• Load a pretrained ASR pipeline
• Transcribe speech into text
• Use Whisper for multilingual recognition

Practice 2 — Speech Recognition with Wav2Vec2

pythonCopy
1from transformers import pipeline
2
3asr = pipeline(
4    task="automatic-speech-recognition",
5    model="facebook/wav2vec2-base-960h"
6)
7
8result = asr("speech.wav")
9
10print(result["text"])

What You'll Learn

• Perform speech recognition using Wav2Vec2
• Compare CTC-based recognition with Whisper
• Understand raw audio processing

Mini Project — Speech-to-Text Application

pythonCopy
1from transformers import pipeline
2
3speech_to_text = pipeline(
4    task="automatic-speech-recognition",
5    model="openai/whisper-base"
6)
7
8audio_file = "meeting.wav"
9
10result = speech_to_text(audio_file)
11
12print("Transcript:")
13print(result["text"])

What You'll Learn

• Build a complete speech transcription application
• Process audio files with a pretrained Transformer
• Generate readable transcripts from spoken language

Choosing the Right Speech Model

Module Summary

After completing this module, you will be able to:

• Explain how Transformer architectures are applied to speech processing.
• Understand the differences between Whisper, Wav2Vec2, SpeechT5, and Audio Spectrogram Transformer (AST).
• Describe the role of spectrograms and raw waveforms as model inputs.
• Build Automatic Speech Recognition (ASR) systems using Whisper and Wav2Vec2.
• Understand multilingual speech translation using encoder-decoder models.
• Apply SpeechT5 to unified speech and text tasks.
• Use AST for audio classification tasks.
• Build end-to-end speech-to-text applications with Hugging Face Transformers and PyTorch.

Next Module: Module 18 – Fine-Tuning Transformers, where you'll learn parameter-efficient fine-tuning (PEFT), LoRA, QLoRA, Hugging Face Trainer, datasets, evaluation metrics, checkpointing, and fine-tuning LLMs, Vision Transformers, and Speech Transformers on custom datasets.