OmniPlay

Dubbing a video is two problems that people usually solve separately: the audio you hear, and the text you see. OmniPlay does both in one pass, on one machine, with no paid APIs, measured against a 4 GB RTX 3050, because that's the card I had.

That last clause is the whole design constraint, not a disclaimer. env_info.json records the machine it was built against: Python 3.9, torch 2.7.1 on CUDA 12.6, one cuda:0 device with 4095 MiB of VRAM. The obvious shape for a pipeline like this is to load every model and walk the video top to bottom, and on this card that dies inside the first stage. Demucs will not take an episode in one bite, the ASR and diarization models cannot both be resident, and LaMa wants exactly the memory the translation stack is holding. So the pipeline is written as a sequence of stages that each own the GPU alone and hand off through files on disk. Almost everything below follows from that.

The audio track

1. Separate the stems. Demucs v4 (htdemucs) splits speech from everything else, so the music and effects survive untouched. This replaced an earlier Spleeter path. Measured on the same input, Demucs scored 0.984 correlation against 0.96, and the difference is audible on anything with a score under the dialogue. It runs in 7-second windows, because an episode does not fit on this card at once.
2. Transcribe and diarize. faster-whisper large-v3 under CTranslate2 in float16 for the words, pyannote 3.1 for who said them, loaded once the ASR models are freed.
3. Build a voice reference. Rather than slicing the first three seconds, the profiler ranks a speaker's turns and assembles twenty seconds of clean audio.
4. Translate, fit, clone. Google Translate carries the first pass, a local Gemma model repairs mis-heard ASR, and the line is condensed to fit its slot before Chatterbox or XTTS reproduces the speaker in the target language.
5. Re-mux against the preserved backing track, copying the video stream untouched.

Between the translation and the synthesizer sits a normalization pass that exists because TTS models read what they are handed. 1947 comes out as "one nine four seven"; the trailing period in Dr. reads as a sentence boundary and the model inhales mid-phrase; typographic quotes and ellipses are voiced as noise. The expander stops at four digits and refuses currency, ordinals and dates, on the grounds that a wrong expansion in a dub with no second take is worse than leaving the digits alone.

The visual track, and why Hindi got its own route

YOLO12, trained on the AnimeText dataset of 735K images and 4.2M annotations, locates text regions at 1280 px with the NMS IoU dropped to 0.35. manga-ocr reads them, EasyOCR handles Devanagari with matra-aware padding, and a vision model arbitrates disagreements. LaMa inpaints the original out, and the renderer redraws the translation with the colour, outline, stroke weight, opacity and rotation angle taken from the original glyphs.

Hindi is a parallel route rather than a language flag, and the reason is worth stating. manga-ocr's decoder vocabulary is Japanese, so it cannot emit Devanagari at all: hand it a Hindi crop and it returns confident, well-formed Japanese that nothing downstream can detect as wrong. The ja path runs a measured transform order, dakuten scoring and vertical right-to-left reading; the hi path runs gentler preprocessing that does not break the shirorekha, and reads left to right. They share no code.

One context, two tracks

Both halves resolve names against one shared context file, and the audio half runs first on purpose so its naming decisions reach the visual half. Without it a character's name is transliterated one way by the speech model and another by the OCR pass, and the subtitle disagrees with the sign in the same frame. Deciding once that he is Shinnosuke, then carrying that across a process boundary into a different Python environment, is what makes the output read as one translation rather than two.

Mechanically it is a JSON record of TermDecision rows, each holding the source string, the chosen target, a kind of term, name or title, which stage decided it, an occurrence count and any conflicting alternatives. A term has to appear twice before it counts as recurring vocabulary, so a one-off phrasing never constrains later text. Proper nouns are harvested from the target side of the transcript, because that is where the spelling decision actually lives. The second stage gets the top forty rows by occurrence as a prompt block, and its output is then run through a repair pass that only rewrites tokens with a difflib similarity of 0.82 or better against an established name. Anything less similar is left alone, because silently rewriting unrelated words is the same over-reach as a glossary that inserts terms that were never there. Conflicts are recorded rather than resolved, so a disagreement shows up in the run report instead of vanishing.

This is deliberately not the glossary. glossary/*.json is authored ahead of a run and holds what is known about a series; the shared context holds what was decided during this run, including names no glossary contains because they belong to one episode.

Inside the engine

• Six orchestrated stages with a one-line @@PROGRESS JSON protocol on stdout.
• A PyQt5 front end that never imports pipeline code, only spawns subprocesses, so a stage that OOMs cannot take the GUI with it. Cancel kills the whole process tree.
• Four environments because the pins conflict: the text stage needs numpy 2.x and a modern transformers, Coqui TTS 0.22 pins numpy 1.22.0 and transformers 4.36.2, and the three want torch 2.8, 2.7.1 and 2.6.0.
• Independent audio and text languages on the CLI, so dubbing into Hindi while subtitling signage into English is one command rather than two runs.
• A benchmark harness scoring OCR variants by character error rate against hand-verified truth, rather than guessing which preprocessing helps.
• Character-level voting across frames and transforms, with dakuten disambiguation scored from pixel evidence rather than left to the model.
• Scene batching grouping regions by pause gap and sorting them into reading order.
• Reference-free quality scoring: hard fails, a length-ratio band, degeneration detection, optional LaBSE adequacy on CPU.
• A glossary injecting only entries present in the current batch, then post-editing each engine's output.
• A run report recording every vote and the winning consensus, so a wrong subtitle is attributable without re-running.
• Explicit VRAM choreography: the inpainter is not built until analysis is done, the translation stack is freed before LaMa, and Ollama is told to drop the model it holds resident at 3.2 GB.
• A FastAPI wrapper over the single-image path: GET /health reports the device and which models are resident, POST /detect takes a frame plus a YOLO variant and an optional confidence override and returns boxes with OCR text, POST /translate runs detect, OCR, translate, inpaint and render and returns the result as base64 JPEG. Models lazy-load on first use and stay cached, and the docs warn to keep uvicorn at one worker because each extra worker tries to load the same CUDA weights.
• Three regression test files named after the defects they pin down, including the tracker dropout where a single-frame detector miss left one un-inpainted Japanese frame in the output because the flow-predicted box was computed and thrown away.

Measured, not assumed

The benchmark exists because intuitions kept losing. manga-ocr scores 0.10 CER on a 2.6:1 title card and 0.51 on a 6.0:1 caption, where the vision model gets 0.29, which is where the 4.0 aspect-ratio switch came from. Blending the two readings scored 0.37 against 0.35 for picking one. Gemma 12b halves the repair error of 4b for twice the wall clock. The four price signs on the test clip score 0.37, 0.23, 0.10 and 0.09, so the stock 0.25 threshold keeps one of them.

bench_ocr.py has a second mode that ranks whole OCR engines rather than preprocessing variants, and that turned out to matter more. Five variants are five views of one model, so their errors correlate: on a real title card the vote pool agreed twenty-one times on the same wrong dakuten. Voting cannot fix a mistake every voter shares. On a handwritten price card all five variants read 大葉 as 菜ｇ－大＼＼ at around 60 percent confidence, which is exactly the case the vision model was never consulted on, and why the arbitration default was flipped to always-on.

The same discipline runs on the dub. measure_legibility.py feeds finished speech back through ASR and scores CER against the line that was meant to be spoken, which turns "not legible enough" into a number comparable across backends: XTTS 0.009, Chatterbox 0.000 on the test clip. Take ranking uses the same metric in batch, in the ASR environment, so the recogniser loads once for a whole set of takes rather than once each.

Model size was measured too, and lost. yolo12x has 59.1M parameters against yolo12n's 2.57M, and on the test clip it found 7 regions instead of 5 while failing to fix the misread handwritten sign, then pushed host RAM past what was free and died with a MemoryError inside ultralytics. Detection capacity was not the bottleneck, so auto sizes the detector to the machine and the confidence floor moved from 0.25 to 0.10 instead.

Running six stages across four interpreters

studio.py is the orchestrator and it never imports a model. Each stage is a subprocess launched under a named interpreter: an ASR env for faster-whisper and pyannote, a TTS env pinned to Coqui's numpy 1.22 and transformers 4.36.2, a Chatterbox env on torch 2.6.0, and a text env on numpy 2.x with ultralytics. The text interpreter is resolved by capability rather than inherited, because the GUI runs in whichever environment has PyQt5 and that environment has neither Demucs nor the OCR stack; candidates are probed with import cv2, torch, ultralytics and the first that survives wins, with STUDIO_TEXT_PYTHON as the override. Child output is streamed rather than captured, since a ten-minute silent terminal is indistinguishable from a hang. Intermediates under temp/ are kept by default because they are the only record of what each stage decided.

Limits and lineage

Chatterbox scores better and runs roughly ten times slower than XTTS, about 7.7x realtime on this card, so the default is a speed choice rather than a quality one. Hindi OCR mis-orders pre-base matras, reading दिन as दनि; that is detected and handed to the LLM, because deciding which consonant owns a matra is a lexical question with no mechanical fix. Emotion intensity is scored relative to a speaker's own range, so it means nothing until a character has several lines. And the licences do not all match the code: XTTS v2 is CPML and non-commercial, pyannote is gated behind a Hugging Face token, and some LaMa weights carry their own restriction.

Rewritten twice: loose scripts, then OmniPlay as a PyQt app over Whisper, Spleeter and FFmpeg, and now this engine, whose on-screen-text half did not exist before. The text stage alone is about 16,000 lines of Python across forty-odd modules. Eighteen finished runs sit in the output folder.