dashed/whisperx-subtitles-replicate ✓📝🔢🖼️ → ❓

▶️ 19.1K runs 📅 Sep 2024 ⚙️ Cog 0.9.24 🔗 GitHub

speaker-diarization speech-to-text subtitle-generation

About

Generates subtitles from audio using whisperX (faster-whisper-large-v3)

Example Output

Output

2
00:00:10,097 --> 00:00:11,097
That's not my business.

3
00:00:12,258 --> 00:00:14,258
I don't want to rule or conquer anyone.

4
00:00:14,678 --> 00:00:16,569
I should like to help everyone if
possible.

5
00:00:16,579 --> 00:00:18,959
Jew, gentile, black man, white.

6
00:00:19,900 --> 00:00:22,900
We all want to help one another. Human
beings are like that.

7
00:00:23,301 --> 00:00:26,471
We want to live by each other's happiness,
not by each other's misery.

8
00:00:26,481 --> 00:00:29,352
We don't want to hate and despise one
another. In this world,

9
00:00:29,362 --> 00:00:31,963
there's room for everyone and the good
earth is rich

10
00:00:32,063 --> 00:00:33,343
and can provide for everyone.

11
00:00:33,923 --> 00:00:36,173
The way of life can be free and beautiful.

12
00:00:36,869 --> 00:00:38,369
but we have lost the way.

13
00:00:38,950 --> 00:00:40,790
Greed has poisoned men's souls,

14
00:00:41,530 --> 00:00:43,071
has barricaded the world with hate,

15
00:00:43,771 --> 00:00:46,552
has goose-stepped us into misery and
bloodshed.

16
00:00:46,652 --> 00:00:49,333
We have developed speed, but we have shut
ourselves in.

17
00:00:49,993 --> 00:00:52,243
Machinery that gives abundance has left us
in want.

18
00:00:52,894 --> 00:00:56,195
Our knowledge has made us cynical, our
cleverness hard and unkind.

19
00:00:56,795 --> 00:00:58,856
We think too much and feel too little.

20
00:00:59,585 --> 00:01:01,746
More than machinery, we need humanity.

21
00:01:02,226 --> 00:01:05,108
More than cleverness, we need kindness and
gentleness.

22
00:01:05,728 --> 00:01:09,090
Without these qualities, life will be
violent and all will be lost.

23
00:01:09,851 --> 00:01:12,351
The aeroplane and the radio have brought
us closer together.

24
00:01:12,532 --> 00:01:15,744
The very nature of these inventions cries
out for the goodness in men.

25
00:01:15,754 --> 00:01:18,504
Cries out for universal brotherhood, for
the unity of us all.

26
00:01:19,016 --> 00:01:21,516
Even now, my voice is reaching millions
throughout the world.

27
00:01:21,997 --> 00:01:24,318
Millions of despairing men, women and
children.

28
00:01:24,939 --> 00:01:28,941
Victims of a system that makes men torture
and imprison innocent people.

29
00:01:30,080 --> 00:01:32,841
To those who can hear me, I say, do not
despair.

30
00:01:32,861 --> 00:01:34,862
The misery that is now upon us is

31
00:01:34,882 --> 00:01:36,143
but the passing of greed,

32
00:01:36,663 --> 00:01:39,645
the bitterness of men who fear the way of
human progress.

33
00:01:40,245 --> 00:01:43,106
The hate of men will pass and dictators
die.

34
00:01:43,547 --> 00:01:46,797
And the power they took from the people
will return to the people.

35
00:01:46,988 --> 00:01:49,910
And so long as men die, liberty will never
perish.

36
00:01:51,997 --> 00:01:54,560
Soldiers, don't give yourselves to brutes,

37
00:01:54,660 --> 00:01:56,623
men who despise you, enslave you,

38
00:01:56,723 --> 00:01:59,356
who regiment your lives, tell you what to
do, what to think

39
00:01:59,366 --> 00:02:01,769
and what to feel, who drill you, diet you,

40
00:02:01,869 --> 00:02:04,553
treat you like cattle, use you as cannon
fodder.

41
00:02:04,653 --> 00:02:06,795
Don't give yourselves to these unnatural
men,

42
00:02:07,336 --> 00:02:10,336
machine men with machine minds and machine
hearts.

43
00:02:10,720 --> 00:02:14,842
you are not machines you are not cattle
you are men you have the

44
00:02:14,982 --> 00:02:19,664
love of humanity in your hearts you don't
hate only the unloved hate the unloved

45
00:02:19,704 --> 00:02:23,966
and the unnatural soldiers don't fight for
slavery fight for liberty

46
00:02:24,346 --> 00:02:25,717
in the 17th chapter of saint luke

47
00:02:25,727 --> 00:02:29,938
it is written the kingdom of god is within
man not one man nor a group of men

48
00:02:29,948 --> 00:02:33,250
but in all men in you you the people have
the power

49
00:02:33,610 --> 00:02:36,332
the power to create machines, the power to
create happiness.

50
00:02:36,552 --> 00:02:39,184
You, the people, have the power to make
this life free

51
00:02:39,194 --> 00:02:42,194
and beautiful, to make this life a
wonderful adventure.

52
00:02:42,377 --> 00:02:45,199
Then in the name of democracy, let us use
that power!

53
00:02:45,439 --> 00:02:46,520
Let us all unite!

54
00:02:47,060 --> 00:02:48,810
Let us fight for a new world!

55
00:02:49,102 --> 00:02:51,852
A decent world that will give men a chance
to work,

56
00:02:52,043 --> 00:02:54,964
that will give youth a future and old age
a security.

57
00:02:55,064 --> 00:02:57,814
By the promise of these things, brutes
have risen to power.

58
00:02:57,986 --> 00:03:01,447
But they lie. They do not fulfill that
promise. They never will.

59
00:03:01,687 --> 00:03:05,029
Dictators free themselves, but they
enslave the people.

60
00:03:05,489 --> 00:03:08,090
Now let us fight to fulfill that promise.

61
00:03:08,470 --> 00:03:10,531
Let us fight to free the world,

62
00:03:10,791 --> 00:03:12,432
to do away with national barriers,

63
00:03:12,812 --> 00:03:15,812
to do away with greed, with hate and
intolerance.

64
00:03:16,326 --> 00:03:17,900
Let us fight for a world of reason.

65
00:03:17,910 --> 00:03:22,400
A world where science and progress will
lead to all men's happiness.

66
00:03:23,181 --> 00:03:26,809
Soldiers, in the name of democracy, let us
all unite!

","detected_language":"en"}

Performance Metrics

9.63s Prediction Time

79.36s Total Time

All Input Parameters

{
  "debug": false,
  "language": "en",
  "vad_onset": 0.5,
  "audio_file": "https://replicate.delivery/pbxt/Lj3F130lpxtKYucKpix3vQ9NDnoUGcoZByurQhgjGmIZ4As1/The_Greatest_Speech_Ever_Made_-_Original.m4a",
  "batch_size": 64,
  "vad_offset": 0.363,
  "diarization": false,
  "temperature": 0,
  "align_output": true,
  "language_detection_min_prob": 0,
  "language_detection_max_tries": 5
}

Input Parameters

debug Type: booleanDefault: false: Print out compute/inference times and memory usage information
language Type: string: ISO code of the language spoken in the audio, specify None to perform language detection
vad_onset Type: numberDefault: 0.5: VAD onset
audio_file (required) Type: string: Audio file
batch_size Type: integerDefault: 64: Parallelization of input audio transcription
vad_offset Type: numberDefault: 0.363: VAD offset
diarization Type: booleanDefault: false: Assign speaker ID labels
temperature Type: numberDefault: 0: Temperature to use for sampling
align_output Type: booleanDefault: true: Aligns whisper output to get accurate word-level timestamps
max_speakers Type: integer: Maximum number of speakers if diarization is activated (leave blank if unknown)
min_speakers Type: integer: Minimum number of speakers if diarization is activated (leave blank if unknown)
initial_prompt Type: string: Optional text to provide as a prompt for the first window
huggingface_access_token Type: string: To enable diarization, please enter your HuggingFace token (read). You need to accept the user agreement for the models specified in the README.
language_detection_min_prob Type: numberDefault: 0: If language is not specified, then the language will be detected recursively on different parts of the file until it reaches the given probability
language_detection_max_tries Type: integerDefault: 5: If language is not specified, then the language will be detected following the logic of language_detection_min_prob parameter, but will stop after the given max retries. If max retries is reached, the most probable language is kept.

Output Schema

segments: Segments
srt_file Type: stringFormat: uri: Srt File
srt_output Type: string: Srt Output
detected_language Type: string: Detected Language

Example Execution Logs

Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.4.0. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../root/.cache/torch/whisperx-vad-segmentation.bin`
Model was trained with pyannote.audio 0.0.1, yours is 3.1.1. Bad things might happen unless you revert pyannote.audio to 0.x.
Model was trained with torch 1.10.0+cu102, yours is 2.1.0+cu121. Bad things might happen unless you revert torch to 1.x.
/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/pyannote/audio/utils/reproducibility.py:74: ReproducibilityWarning: TensorFloat-32 (TF32) has been disabled as it might lead to reproducibility issues and lower accuracy.
It can be re-enabled by calling
>>> import torch
>>> torch.backends.cuda.matmul.allow_tf32 = True
>>> torch.backends.cudnn.allow_tf32 = True
See https://github.com/pyannote/pyannote-audio/issues/1370 for more details.
warnings.warn(
Downloading: "https://download.pytorch.org/torchaudio/models/wav2vec2_fairseq_base_ls960_asr_ls960.pth" to /root/.cache/torch/hub/checkpoints/wav2vec2_fairseq_base_ls960_asr_ls960.pth
  0%|          | 0.00/360M [00:00<?, ?B/s]
  8%|▊         | 27.8M/360M [00:00<00:01, 292MB/s]
 21%|██▏       | 77.4M/360M [00:00<00:00, 426MB/s]
 35%|███▍      | 126M/360M [00:00<00:00, 462MB/s] 
 49%|████▊     | 176M/360M [00:00<00:00, 486MB/s]
 63%|██████▎   | 226M/360M [00:00<00:00, 504MB/s]
 77%|███████▋  | 277M/360M [00:00<00:00, 512MB/s]
 91%|█████████ | 327M/360M [00:00<00:00, 516MB/s]
100%|██████████| 360M/360M [00:00<00:00, 492MB/s]

Version Details

Version ID: 2054b259f3f66c44fec5c16daa4cd263488af8e779e3228d9c2b379f3c9843ed
Version Created: October 4, 2024

Run on Replicate →