Cohere Transcribe vs Wit.ai through Tafrigh

Executive findings

A negative delta favors Wit because every delta is Wit minus Cohere. Confidence intervals that exclude zero are labelled decisive for this benchmark. Statistical significance does not by itself establish practical significance or generalization beyond these datasets.

Domain WER at a glance

July 12 production release

The default median is based on 36.58s and 36.39s process-wall observations. The current pipeline used 182 merged model rows from 378 raw speech spans and retained 4080.224s of the 4160.679s recording. Its output contains 9,852 words and 1,198 subtitle cues. The adaptive transcript has the same word count but a different hash and 12 lexical token edits relative to static; batching is not assumed text-invariant under finite-precision GPU execution.

This long lecture has no human transcript, so its 9,852 versus 9,779 output-word counts and audible disagreements are not WER. The comparison separates one-file latency from fleet throughput: 64 truly independent app quotas across eight Tafrigh processes can scale different files concurrently, while this 131.79-second measurement is one file through one eight-app process.

The reviewer’s warning about TorchAudio forced_align removal was outdated: the operation was retained during the TorchAudio maintenance transition. Production therefore checks matching Torch/TorchAudio release lines and executes an operation smoke test rather than replacing a working dependency. The release passed 54 transcription-focused tests and 71 benchmark tests, Ruff lint/format, and bytecode compilation. Mypy was unavailable, so no mypy result is claimed.

The portable final/ bundle contains the production implementation as transcribe.py with simplified input.txt/input.srt/input.vtt/input.json output names, bundled vectorized Silero ONNX runtime and applicable licenses, portable runtime requirements, a device-oriented setup guide, version/model provenance, and an installation validator. Model weights remain upstream and download at first use after model-access approval.

Release summary artifact: production_release_20260712.json. Production audit: PRODUCTION_REVIEW_20260712.md. Both are hashed in the artifact inventory below.

Evaluation suite

The frozen suite contains one audio path and reference per clip. Exact waveform samples returned by the configured decoder, rather than source metadata alone, provide the timing denominator.

Common Voice, SADA22, and Casablanca references were matched to the exact rows published by the Arabic ASR leaderboard at commit 10cf2c8. The Quran subset is 12 evenly spaced blocks of 50 rows from a three-reciter-held-out split. Its canonical verse references make it a Classical Arabic proxy, not an official Quranic leaderboard score.

Decoded duration was 36.393282 hours versus 36.301534 hours in source metadata, a 330.292-second difference dominated by Common Voice MP3 duration reporting. All 24,414 paths decoded successfully and IDs, references, grouping metadata, and audio paths matched across compared configurations.

Full-suite accuracy

These are corpus-aggregated lexical-normalized rates. S/D/I means substitutions, deletions, and insertions. The bootstrap resamples 14,319 speaker/utterance clusters overall, with speakers namespaced by dataset where speaker IDs exist.

Normalization-profile audit

Paired confidence intervals were computed only for the selected lexical-normalized profile. Deltas in the other profile tables are descriptive corpus-rate differences, not separately bootstrapped estimates.

Dialect and variety detail

Labels are retained from their source datasets. “MSA” is a reference-register label, “More than 1 speaker” and “Unknown” are annotation categories, and tiny categories should not be ranked.

Timing and throughput

Cohere generation alone was 11m 21s, but excluding decode and feature preparation makes it an invalid direct boundary against Wit end-to-end wall. Wit active time excludes credential entry, validation, debugging downtime, exact-duration analysis, and WER scoring. Its first recovered component is accurate to roughly one second; later components came from sanitized ledgers.

Continuous 1.wav case study

In the frozen July 11 timestamp experiment, all four output/timestamp modes shared the same ASR inputs and produced the same joined text. That statement does not compare later sample-exact or merged segmentation policies. --text-only writes one non-empty VAD/ASR segment per line and skips word allocation, cue grouping, and MMS alignment; its historical 50.61-second wall was effectively tied with segment mode because ASR and VAD dominated. A frozen, word-indexed rerun paired 9,755 words between FP16 and FP32. The combined-boundary absolute median/p95/p99 were 0.00/0.00/0.00 ms, and the maximum was 1420.00 ms. This supersedes the earlier ordinal-SRT comparison: cue reflow changed which words an ordinal cue contained, so its apparent 3.12-second maximum was not a valid acoustic-boundary comparison. FP32 remains the implementation-reference word-alignment mode; --alignment segment is the production reliable/fast recommendation when exact word boundaries are unnecessary.

The native rows use the reference quiet-boundary planner rather than Silero and write no timestamps. Native Q8 differed from native F16 by 28 lexical word edits (0.2834%); Q4 differed by 116 (1.1740%). Because 1.wav has no human transcript, these are model disagreements, not WER. The frozen 500-clip reference benchmark, not this file, determines the precision recommendation.

VAD and text-only segmentation modes

--text-only removes timestamp and subtitle work; --vad controls which waveform regions reach the ASR model. Changing VAD can therefore change recognition text and accuracy, even though every mode writes the same plain-text output format. The production default remains 30-second maximum spans. The pinned processor accepts one row through 35 seconds, and the script verifies the actual processor row count at runtime; 35 seconds is supported as an experiment, not presumed faster or more accurate.

The same already-segmented corpus scored 22.9551% WER on the direct ASR benchmark path, outside this end-to-end script timing boundary. Silero completed in 48.70s, Auditok in 49.75s, and no-VAD in 49.58s. The cheaper VAD stages did not reduce complete folder wall in this single observation because preprocessing overlaps model loading/GPU work and each segmentation policy changed the ASR inputs.

Silero remains the accuracy-conservative default for arbitrary continuous audio. Auditok threshold 50 and --vad none --max-dur 30 are explicit speed experiments whose suitability should be measured on representative, continuously recorded ground truth.

Timestamp-mode benchmark

--text-only completed in 48.70s versus 48.52s for segment timing: a 0.18-second difference inside run-to-run variation. It produces only .aligned.txt, with one non-empty VAD/ASR segment per line. The periodic collection fix discovered during this benchmark reduced FP16 wall from 192.24s to 119.07s (38.1%) and FP32 from 302.15s to 233.26s (22.8%). Full heap collection now runs every 64 aligned files instead of after every file.

On identical frozen audio, boundaries, and ASR text, FP16 timing compute was 2.67× faster than FP32. The complete production ratio is smaller because both modes pay the same ASR, VAD, startup, decode, and output costs.

FP16 changed only a small tail: 99.665% of word boundaries were within 20 ms and 99.853% were within 100 ms, but four boundaries exceeded 500 ms and two exceeded one second. Both one-second outliers were in the Quran proxy, where a repeated phrase selected a different CTC path. Median, p95, and p99 alone would hide that case. Segment timing is intentionally approximate: it spreads words uniformly inside each VAD segment.

All 500 hypotheses and normalized texts matched across all four modes; the three timing modes also matched every structured word key. FP16 and FP32 each used uniform fallback timing for the same 97 words in two pathological repetition segments; fallback rows are included in the overall comparison and separately identified in the machine-readable report. Cue timing was matched by underlying word span, never by cue ordinal.

Cohere optimization evidence

Four complete BF16 batch/order configurations were evaluated on the same 24,414 clips. Length-sorted batch 24 was selected as the foundation. The later projection cache and conservative repetition-loop stop produced the final Cohere result used throughout the engine comparison.

• Length sorting reduced the 365-segment ASR-stage probe from 64.64s to 26.61s (2.43× faster). Exact strings changed on 14 of 365 segments; full-suite paired WER changed by −0.043 pp, 95% CI [−0.214, +0.081].
• FP16 generation was slower than BF16 on this hardware (34.26s vs 26.61s) and changed 24 segment strings.
• Moving representative features to GPU saved little (8.82s to 8.65s); eager per-item processing was much slower (16.08s).
• Batch 32 consumed more memory and split once without beating batch 24; batch 16 avoided OOM but was slower. Batch 24 length sorting was the pragmatic winner.

Production-pipeline changes

Rejected or non-default trials: torch.compile recompiled dynamic shapes and hurt this one-shot workload; FP16 ASR was slower and changed text; eager attention, oversized adaptive batches, pinned host memory, and shorter CTC variants did not provide a reliable measured win. FP16 alignment remains available explicitly, but it can shift word timestamps and is therefore not the parity default. The actual decoder is selected per installed environment and recorded in profile provenance; the final 1.wav gate resolved to FFmpeg.

Native GGUF batch engine

The new cohere-batch executable is a separate C++/GGML research runtime. It loads one GGUF model for every supplied file and directory, plans long-audio windows globally, computes deterministic CPU features in parallel, batches encoder and ragged decoder work, can select greedy tokens on the GPU, and writes collision-safe plain-text outputs atomically. It does not use Silero, forced alignment, or the Python model implementation.

The cross-engine table uses the same 500 source clips and references, but it is not a controlled decoder-only comparison: the Python row uses fixed 30-second no-VAD chunks, native uses the pinned quiet-boundary planner up to 35 seconds, and Tafrigh uses Auditok plus MP3 cloud requests. On this workload the established Python BF16 path remained faster than every native variant. Native Q8 is therefore an implementation and deployment alternative, not a new overall speed record.

• After applying the production repetition cleanup, batched F16 matched corrected serial native F16 on 500/500 raw transcripts.
• On-device argmax preserves greedy decoding but omits confidence logits. Q8 1.wav was text-identical with and without full-logit readback.
• Parallel native feature extraction reduced the fixed-window 1.wav frontend from 11.822s at one thread to 4.121s at four and 3.347s at twelve; all feature buffers were byte-identical.
• Stable encoder frame buckets were slower. A persistent decoder graph was 0.7% faster on a 20-minute Q8 trial but inserted one punctuation token, so both experiments remain non-default.
• A repaired shared-weight CUDA multistream probe was byte-exact, but decoder-like B2 overlap reached only 1.084x and B4 1.147x; the predefined 1.15x B2-first gate failed, so no model/session refactor was attempted.
• The native executable writes text only. It provides neither segment nor word timestamps, so the timestamp-error tables above do not apply to it.

Upstream contribution opportunities

The final script was mapped against current default-branch source in four upstream repositories. GitHub searches covered open and closed issues and pull requests, exact symbols, broader concepts, and nearby changes. “No duplicate found” means no matching public indexed work was found as of July 12, 2026; it cannot exclude an unlinked draft or private branch.

Recommended contribution order: first the stable DeskPai log-softmax because it is a tiny correctness patch; then the Cohere projection change with a profiler/call-count test; then the generic repetition criterion on the already accepted feature request. Keep the projection and mask patches separate so training, direct decoder calls, beam expansion, compilation, and multi-device behavior can be reviewed independently. For Silero, contribute reproducible export code and exact 8/16 kHz parity tests rather than only the ONNX binary.

The complete audit, pinned upstream commit hashes, exact local line mappings, search queries, neighboring PR analysis, and proposed tests are in UPSTREAM_CANDIDATES_20260712.md.

Wit/Tafrigh configuration probe

The duration-stratified probe contains 500 clips, 100 from each dataset (1.398810 decoded hours). It tests Tafrigh-compatible 15/16-second cuts, its actual generated-noise padding, true silence, and no padding. The baseline is exact Tafrigh v1.7.8 behavior.

Tafrigh calls WhiteNoise(..., volume=0). In Pydub, 0 means 0 dB gain rather than zero amplitude; measured output was roughly −4.8 dBFS. “True silence” used −120 dB. Wit outputs were not deterministic across calls/apps, including one minor Quran variant on the same encoded payload.

Wit/Tafrigh pipeline and telemetry

Compatibility boundary

1. Non-MP3 clips are converted in full to MP3 with Pydub/FFmpeg.
2. Tafrigh's Auditok splitter uses min_dur=0.5, max_silence=0.5, energy_threshold=50, and a 15-second maximum region.
3. Each region is re-encoded after one second of Tafrigh-generated noise is added at both sides.
4. Requests use POST /speech, audio/mpeg3, Wit media version 20200513, Arabic question-mark substitution, and chronological segment concatenation.

Scheduler

Stock Tafrigh loops over files sequentially and creates a manager/process pool for every file. On 24,414 short clips, even a one-request-per-file admission lower bound is 6h 47m. The benchmark preserves Tafrigh's recognition inputs but shares a bounded scheduler across files, uses 8 distinct Arabic apps in 2 inferred user groups, limits each app to one start every 1.02s, caps each user group at 4 starts/s, and checkpoints a clip only after every segment succeeds. Every retry reacquires both quota limits.

The operator's production topology is eight Tafrigh processes with eight keys each, and the 64 keys have independent app/user quotas. That can materially outperform a single eight-app run on a folder when enough independent files exist: the processes can upload and recognize separate regions concurrently. It does not make one unsharded file automatically eight times faster, and actual aggregate scaling still depends on Auditok/encoding CPU, upload bandwidth, network latency, API service time, and per-process scheduling. Cohere has the opposite shape: one GPU model is best kept resident and globally batched across files rather than duplicated eight times on a 12 GB card.

Methodology and reproducibility

Validation gates

• All audio paths decoded successfully; there were 24,414 unique paths.
• IDs, references, dataset/domain/dialect/speaker metadata, and paths matched across configurations.
• Hypothesis-row fingerprints matched their summaries.
• Declared decoder provenance matched where present; the Wit runner did not declare local decoder provenance, so exact duration was independently replayed.
• Wit failures were explicit and resumable; exhausted requests were never silently converted to empty hypotheses.
• Credentials were accepted only through a hidden prompt and were not written to reports, summaries, hypotheses, or logs.
• The July 12 release passed repeated 1.wav timing, three current 500-clip production modes, real FP32 word alignment, ONNX/JIT VAD parity, live truncation retry, 71 automated benchmark tests, Ruff, and bytecode compilation.

Environment

Wit used Python 3.12.12, tafrigh[wit]==1.7.8, and Requests 2.34.2 in an isolated environment without PyTorch, Transformers, Whisper, or Cohere dependencies. The final production profile used Python 3.12.12, PyTorch/TorchAudio 2.11.0+cu128, Transformers 5.13.0, ONNX Runtime 1.27.0, and CUDA BF16 on an NVIDIA GeForce RTX 3060. Exact tested package versions remain recorded in the production profile and release summary; the portable bundle intentionally constrains only compatibility-sensitive packages and leaves device-specific Torch wheels to the installer.

Limitations and correct interpretation

• The 24,414-clip recognition benchmark is pre-segmented and does not measure Silero long-form boundary quality. The separate 500-clip production probe includes Silero, subtitle cueing, and forced alignment. Wit applies Tafrigh/Auditok VAD inside each clip, so its WER includes that package behavior.
• Wit is a cloud service and showed output variation across calls/apps. Cohere timing depends on the tested local GPU and software stack. Each engine has one primary timing observation, so speed has no confidence interval.
• The Wit wall was accumulated across resumed invocations. One component was recovered from progress and checkpoint timestamps to roughly one-second precision; the uncertainty is negligible relative to 4,397 seconds but should still be disclosed.
• The eight-app result is workload-specific. Quota scheduling, network latency, clip count, and average clip duration mean the 2.578× ratio should not be extrapolated directly to one continuous recording.
• The user's 64-key/eight-process Tafrigh topology was not benchmarked as one controlled run. Independent quotas make substantial folder-level scaling plausible, but a linear eightfold extrapolation from the eight-app result would ignore preprocessing, bandwidth, service latency, file distribution, and retry overhead.
• The Quran set has 600 selected ayat but only three reciters, producing a wide cluster-bootstrap interval. Canonical verse text is not an independent verbatim transcript, so this is a Classical Arabic recitation proxy only.
• MSA denotes reference register, not guaranteed speaker accent. Dialect labels combine Casablanca country varieties with SADA annotations and include unknown/multi-speaker categories.
• Very small labels such as Yemeni (7 clips) and Shamali (18 clips) have unstable rates and broad intervals. CER can exceed 100% when insertions outnumber reference characters.
• The lexical normalizer is the closest disclosed approximation to Cohere's internal evaluation normalizer, not a verified byte-identical copy. The repo-exact leaderboard profile intentionally preserves a known punctuation-regex bug.
• None of the five evaluation datasets contains human word-level timestamps. FP32-relative drift measures implementation agreement only; it is not absolute timestamp accuracy and does not prove that FP32 chose the acoustically correct boundary.
• The native GGUF runtime is text-only and uses a different long-audio planner from both the Python fixed-window and Silero paths. Its 500-clip cross-engine table shares sources and references but is not a controlled decoder-only comparison.
• Native quantization intervals resample utterances, not speaker or corpus clusters. Both aggregate intervals include zero; the Q8 recommendation also relies on its much smaller output-change count and domain deltas.
• Three native F16 lanes and three Q8 lanes reached the 445-token generation cap in the 500-clip gate; Q4 had one. The CLI warns and publishes those texts, so maximum-token warnings require review.
• Cloud processing changes the data boundary. Wit states that voice data and transcripts may be retained for up to 90 days. Several evaluation datasets have noncommercial or no-derivatives terms; do not redistribute their audio.
• WER measures transcript edit distance, not semantic fidelity, punctuation quality, named-entity accuracy, timestamp quality, hallucination severity, or downstream task utility.
• The July 12 production patch was gated on the balanced 500 clips rather than rerunning the 24,414-clip suite. It improves production mechanics and sample-exact VAD behavior, but subgroup movement on the 500 clips was not uniform.
• The 36.485-second 1.wav figure is the median of two external process-wall runs on one RTX 3060 system. Model cache state, storage, CPU, driver, thermal state, other GPU users, container decoding, and segment distribution can change results on another device.
• Adaptive growth is memory-aware, not a complete throughput search. Its equal word count but 12-token difference on 1.wav demonstrates why finite-precision configuration changes must retain transcript-quality gates.

Sources, versions, and artifact integrity

Public references

• Cohere Arabic Transcribe technical release and normalization disclosure
• Official Cohere Arabic GGUF variants and quantization notes
• Pinned Transformers Cohere ASR implementation
• TorchAudio maintenance transition update
• CrispASR native runtime source
• Silero VAD and ONNX runtime guidance
• TorchAudio CTC forced-alignment API
• Tafrigh v1.7.8 source
• Wit.ai HTTP /speech documentation
• Wit.ai FAQ
• Wit.ai privacy policy
• Synchronized Cohere/Wit 1.wav playback comparison
• 1.wav source video
• 1.wav hosted audio fallback
• Pinned Arabic ASR leaderboard evaluator and manifests
• Common Voice 18 Arabic mirror
• FLEURS
• SADA22
• Casablanca
• Casablanca corpus paper and utterance-boundary method
• Quran Ayah Corpus used for the CA proxy

Machine-readable source artifacts

This HTML is a derived, static rendering. Percentages are rounded for display; conclusions and intervals come from the unrounded JSON values. Local absolute paths, credentials, transcript text, and app identifiers are intentionally excluded.