Audio Models
Voice is the most natural human interface. Audio models let applications accept spoken input, generate spoken output, and analyze audio content — critical for accessibility, hands-free use, and processing podcasts/meetings/calls.
A universal translator earpiece: hears any language and converts to text (STT), reads any text aloud naturally (TTS), and understands what's being discussed (audio understanding).
Visual Workflows
Start here — study each diagram, then use + / − to zoom if needed.
Scroll inside the frame · use + / − to zoom
Key Takeaways
- 1.Audio models handle speech-to-text (transcription), text-to-speech (synthesis), and audio understanding — enabling voice interfaces, meeting transcription, and accessible AI applications. Speech-to-Text: OpenAI Whisper (open-source, multilingual, robust).
- 2.API: audio file → text transcript with timestamps. Text-to-Speech: OpenAI TTS (natural voices, multiple styles), ElevenLabs (high quality cloning).
- 3.API: text → audio stream. Audio understanding: GPT-4o audio (process audio directly, understand tone/emotion).
- 4.Key specs: sample rate (16kHz typical), formats (mp3, wav, webm), language detection (Whisper auto-detects 99 languages). Real-time: streaming STT for live transcription.
Real Example
Scenario
Meeting transcription pipeline: upload 1-hour meeting recording → Whisper transcribes with speaker timestamps → GPT-4o summarizes key decisions and action items → TTS generates audio summary for team members who missed the meeting.
What you would do
In LLM Engineering & APIs, apply Audio Models to this scenario: Meeting transcription pipeline: upload 1-hour meeting recording → Whisper transcribes with speaker timestamps → GPT-4o summarizes key decisions and action items → TTS generates audio summary for team members who missed the meeting. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.
Practice Task
Open the Code Walkthrough below and run it locally. Change one parameter related to Audio Models (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.
Code Walkthrough
Highlighted lines show where Audio Models happens in the code.
1from openai import OpenAI # import dependencies2
3client = OpenAI() # create API client4
5# Speech-to-Text with Whisper6with open("meeting.mp3", "rb") as audio_file: # key line for Audio Models7 transcript = client.audio.transcriptions.create( # core API call for Audio Models8 model="whisper-1",9 file=audio_file, # key line for Audio Models10 response_format="verbose_json",11 timestamp_granularities=["segment"],12 )13print(transcript.text) # show output for debugging14
15# Text-to-Speech16response = client.audio.speech.create( # core API call for Audio Models17 model="tts-1",18 voice="nova",19 input="Hello! This is an AI-generated voice.",20)21response.stream_to_file("output.mp3")Cheat Sheet
Quick recap
quick ref- •audio.transcriptions.create()
- •model: whisper-1
- •audio.speech.create()
- •voices: nova, alloy, echo, fable
- •tts-1-hd for quality
- •verbose_json for timestamps
Common Mistakes
- ✕Sending wrong audio format — check API supported formats
- ✕Not handling long audio files — split into chunks
- ✕Ignoring transcription errors — always validate output
- ✕High TTS latency in voice agents — need streaming pipeline