Whisper — OpenAI 通用语音识别模型

{/* 此页面由 website/scripts/generate-skill-docs.py 从技能的 SKILL.md 自动生成。请编辑源文件 SKILL.md，而非此页面。 */}

Whisper

OpenAI’s general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.

技能元数据

来源	可选 — 通过 hermes skills install official/mlops/whisper
路径	optional-skills/mlops/whisper
版本	1.0.0
作者	Orchestra Research
许可证	MIT
依赖项	openai-whisper, transformers, torch
平台	linux, macos
标签	Whisper, Speech Recognition, ASR, Multimodal, Multilingual, OpenAI, Speech-To-Text, Transcription, Translation, Audio Processing

参考：完整 SKILL.md

:::info 以下是 Hermes 在触发此技能时加载的完整技能定义。这是技能激活时代理所看到的指令。 :::

Whisper - Robust Speech Recognition

OpenAI’s multilingual speech recognition model.

何时使用 Whisper

适用场景：

• Speech-to-text transcription (99 languages)
• Podcast/video transcription
• Meeting notes automation
• Translation to English
• Noisy audio transcription
• Multilingual audio processing

数据指标：

• 72,900+ GitHub stars
• 99 languages supported
• Trained on 680,000 hours of audio
• MIT License

Use alternatives instead:

• AssemblyAI: Managed API, speaker diarization
• Deepgram: Real-time streaming ASR
• Google Speech-to-Text: Cloud-based

快速开始

安装

# Requires Python 3.8-3.11
pip install -U openai-whisper
 
# Requires ffmpeg
# macOS: brew install ffmpeg
# Ubuntu: sudo apt install ffmpeg
# Windows: choco install ffmpeg

Basic transcription

import whisper
 
# Load model
model = whisper.load_model("base")
 
# Transcribe
result = model.transcribe("audio.mp3")
 
# Print text
print(result["text"])
 
# Access segments
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")

模型尺寸

# Available models
models = ["tiny", "base", "small", "medium", "large", "turbo"]
 
# Load specific model
model = whisper.load_model("turbo")  # Fastest, good quality

模型	参数	仅英语	多语言	速度	VRAM
tiny	39M	✓	✓	~32x	~1 GB
base	74M	✓	✓	~16x	~1 GB
small	244M	✓	✓	~6x	~2 GB
medium	769M	✓	✓	~2x	~5 GB
large	1550M	✗	✓	1x	~10 GB
turbo	809M	✗	✓	~8x	~6 GB

建议： Use turbo for best speed/quality, base for prototyping

Transcription options

Language specification

# Auto-detect language
result = model.transcribe("audio.mp3")
 
# Specify language (faster)
result = model.transcribe("audio.mp3", language="en")
 
# Supported: en, es, fr, de, it, pt, ru, ja, ko, zh, and 89 more

Task selection

# Transcription (default)
result = model.transcribe("audio.mp3", task="transcribe")
 
# Translation to English
result = model.transcribe("spanish.mp3", task="translate")
# Input: Spanish audio → Output: English text

Initial prompt

# Improve accuracy with context
result = model.transcribe(
    "audio.mp3",
    initial_prompt="This is a technical podcast about machine learning and AI."
)
 
# Helps with:
# - Technical terms
# - Proper nouns
# - Domain-specific vocabulary

Timestamps

# Word-level timestamps
result = model.transcribe("audio.mp3", word_timestamps=True)
 
for segment in result["segments"]:
    for word in segment["words"]:
        print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")

Temperature fallback

# Retry with different temperatures if confidence low
result = model.transcribe(
    "audio.mp3",
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
)

命令行使用

# Basic transcription
whisper audio.mp3
 
# Specify model
whisper audio.mp3 --model turbo
 
# Output formats
whisper audio.mp3 --output_format txt     # Plain text
whisper audio.mp3 --output_format srt     # Subtitles
whisper audio.mp3 --output_format vtt     # WebVTT
whisper audio.mp3 --output_format json    # JSON with timestamps
 
# Language
whisper audio.mp3 --language Spanish
 
# Translation
whisper spanish.mp3 --task translate

批量处理

import os
 
audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]
 
for audio_file in audio_files:
    print(f"Transcribing {audio_file}...")
    result = model.transcribe(audio_file)
 
    # Save to file
    output_file = audio_file.replace(".mp3", ".txt")
    with open(output_file, "w") as f:
        f.write(result["text"])

实时转录

# For streaming audio, use faster-whisper
# pip install faster-whisper
 
from faster_whisper import WhisperModel
 
model = WhisperModel("base", device="cuda", compute_type="float16")
 
# Transcribe with streaming
segments, info = model.transcribe("audio.mp3", beam_size=5)
 
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

GPU 加速

import whisper
 
# Automatically uses GPU if available
model = whisper.load_model("turbo")
 
# Force CPU
model = whisper.load_model("turbo", device="cpu")
 
# Force GPU
model = whisper.load_model("turbo", device="cuda")
 
# 10-20× faster on GPU

与其他工具集成

Subtitle generation

# Generate SRT subtitles
whisper video.mp4 --output_format srt --language English
 
# Output: video.srt

With LangChain

from langchain.document_loaders import WhisperTranscriptionLoader
 
loader = WhisperTranscriptionLoader(file_path="audio.mp3")
docs = loader.load()
 
# Use transcription in RAG
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
 
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

Extract audio from video

# Use ffmpeg to extract audio
ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav
 
# Then transcribe
whisper audio.wav

最佳实践

1. Use turbo model - Best speed/quality for English
2. Specify language - Faster than auto-detect
3. Add initial prompt - Improves technical terms
4. Use GPU - 10-20× faster
5. Batch process - More efficient
6. Convert to WAV - Better compatibility
7. Split long audio - <30 min chunks
8. Check language support - Quality varies by language
9. Use faster-whisper - 4× faster than openai-whisper
10. Monitor VRAM - Scale model size to hardware

性能

Model	Real-time factor (CPU)	Real-time factor (GPU)
tiny	~0.32	~0.01
base	~0.16	~0.01
turbo	~0.08	~0.01
large	~1.0	~0.05

Real-time factor: 0.1 = 10× faster than real-time

语言支持

Top-supported languages:

• English (en)
• Spanish (es)
• French (fr)
• German (de)
• Italian (it)
• Portuguese (pt)
• Russian (ru)
• Japanese (ja)
• Korean (ko)
• Chinese (zh)

完整列表： 99 languages total

局限性

1. Hallucinations - May repeat or invent text
2. Long-form accuracy - Degrades on >30 min audio
3. Speaker identification - No diarization
4. Accents - Quality varies
5. Background noise - Can affect accuracy
6. Real-time latency - Not suitable for live captioning

资源

• GitHub： https://github.com/openai/whisper ⭐ 72,900+
• 论文： https://arxiv.org/abs/2212.04356
• Model Card: https://github.com/openai/whisper/blob/main/model-card.md
• Colab: Available in repo
• 许可证： MIT