# vachaka vachaka (वाचक, “reciter”) turns Sanskrit text in Devanagari into recited audio, and tells you where every line and every word falls in that audio. It is a thin layer over two models by Prathosh A. P. [Vāgdhenu](https://github.com/prathoshap/vagdhenu) is a text-to-speech model that chants: given a verse and a reference recording in the verse’s metre, it sings the verse to that metre’s tune. It is a fine-tune of IndicF5 with a BigVGAN-v2 vocoder, the part that turns the model’s output into a waveform. [Su-śrotā](https://huggingface.co/prathoshap/sushrota-sanskrit-asr) is a Sanskrit speech recogniser, IndicConformer fine-tuned on śāstric Sanskrit. Vāgdhenu does the speaking. Su-śrotā listens back: it transcribes each render and compares the transcript with the text, so a bad render can be thrown away and tried again, and it finds where each word starts and ends. vachaka was written for [VedicReader](https://vedicreader.com), which has thousands of Sanskrit verses and sūtras that nobody has recorded. There the input is a work’s lines as they are displayed, and the output must be one audio file plus a `[start, end]` time for every line and word, so the reader can highlight the text as it plays. `vachaka.recite` does exactly that. ## Listen These clips come from a trial run over VedicReader’s texts. The voice is synthetic, and it is that of Vāgdhenu’s author. CER, the character error rate, is the share of characters Su-śrotā got wrong when it transcribed the clip, compared with the text: 0.0 means every character was heard as written. The audio players work on the [docs site](https://vedicreader.github.io/vachaka/); GitHub and PyPI show only the text. Each clip says whether it was chanted or read, and which reference recording it was rendered against. ### Annapūrṇāṣṭakam, verse 2 (śārdūlavikrīḍita) नानारत्नविचित्रभूषणकरी हेमाम्बराडम्बरी
मुक्ताहारविलम्बमानविलसद्वक्षोजकुम्भान्तरी ।
काश्मीरागरुवासिताङ्गरुचिरे काशीपुराधीश्वरी
भिक्षां देहि कृपावलम्बनकरी मातान्नपूर्णेश्वरी ॥ २ ॥ chant, reference `shardulavikridita`, CER 0.031 ### Bṛhat Parāśara Horā Śāstra 1.3 (anuṣṭubh) इति स्तुत्या सुसंहृष्टो मुनिस्तत्त्वविदां वरः।
अथादिदेश सच्छास्त्रं सारं यज्ज्योतिषां शुभम्॥ ३॥ chant, reference `anushtubh`, CER 0.0 ### Jaimini Sūtras 1.1.1 (prose) उपदेशं व्याख्यास्यामः॥ read, reference `anuṣṭubh`, CER 0.0 ### One sūtra, two references Nyāyasūtra 1.1.2 rendered twice. Vāgdhenu ships two reference recordings for prose (`gadya`), but its author renders prose against the anuṣṭubh reference instead, having found that the gadya references drop the starts of words. When [`recite`](https://vedicreader.github.io/vachaka/recite.html#recite) reads prose it follows the author and uses anuṣṭubh. Judge for yourself: anuṣṭubh reference gadya reference दुःखजन्मप्रवृत्तिदोषमिथ्याज्ञानानामुत्तरोत्तरापाये तदनन्तरापायादपवर्गः ## Numbers Measured on an Apple M3 Max with 36 GB of memory, running Vāgdhenu’s main network (a diffusion transformer, or DiT) on the GPU through MPS and the vocoder on the CPU.
loading Vāgdhenu 10 to 17 s
rendering, warm about 5.5 s per second of audio (a 9.5 s anuṣṭubh verse in 50 to 57 s)
share of render time in the DiT 77% (the rest is BigVGAN on the CPU)
loading Su-śrotā 9 s
transcribing a 10 s clip on the CPU 0.4 s
alignment resolution 40 ms (Su-śrotā’s frame)
At that speed an hour of recitation takes five to six hours on a laptop. A CUDA GPU is much faster, and Vāgdhenu’s author renders on one. The trial rendered 27 sections, 293 s of audio in all, from a stotra, three jyotiṣa books, the Jaimini Sūtras and the Nyāyasūtra, keeping the best of up to three renders per section. “Exact” counts the sections Su-śrotā transcribed with no error.
voice sections mean CER worst exact
chant, reference for the detected metre 9 0.030 0.086 2
prose, anuṣṭubh reference 14 0.016 0.118 9
prose, gadya reference 4 0.010 0.039 3
Two cautions. Su-śrotā comes from the same lab as Vāgdhenu and its training data includes recordings of a TTS speaker, so it may hear Vāgdhenu more kindly than a person would. A low CER says the words are there, not that the chant sounds good. And a section whose metre is not detected (an āryā, or a verse split over three lines) is read as prose, which sounds flatter than a chant. Listen before you publish. ## Install ``` sh pip install vachaka # rendering pip install 'vachaka[check]' # Su-śrotā: transcripts, error rate, word timings ``` vachaka needs Python 3.10 to 3.12. The first [`Chanter()`](https://vedicreader.github.io/vachaka/core.html#chanter) downloads about 1.5 GB of weights from the Hugging Face Hub, and fetches the source code of Vāgdhenu, BigVGAN and IndicF5 at fixed commits into `~/.cache/vachaka` (set `VACHAKA_CACHE` to put it elsewhere). None of the three is on PyPI, and the package called `f5-tts` there is not IndicF5. Su-śrotā is another 520 MB. Writing mp3 files needs `ffmpeg`. ## Use [`Chanter`](https://vedicreader.github.io/vachaka/core.html#chanter) loads Vāgdhenu. `chant` sings a verse given as a list of its lines, picking the tune from the verse’s metre, and `read` reads prose. Both return a [`Recitation`](https://vedicreader.github.io/vachaka/core.html#recitation), which holds the audio and one `(start, end)` span in milliseconds for each line or chunk of prose, and can be saved as wav or mp3: ``` python from vachaka.core import Chanter ch = Chanter() # cuda, then mps, then cpu r = ch.chant(['वसुदेवसुतं देवं कंसचाणूरमर्दनम्', 'देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुम्']) r.spans # [(0, 4230), (4980, 9550)]: one span per line, in ms r.save('krishna.mp3') ch.read('अथातो धर्मजिज्ञासा । चोदनालक्षणोऽर्थो धर्मः ।', ref='anuṣṭubh').save('mimamsa.mp3') ``` [`Ear`](https://vedicreader.github.io/vachaka/check.html#ear) loads Su-śrotā. [`best_of`](https://vedicreader.github.io/vachaka/check.html#best_of) renders up to three times with different random seeds and keeps the render whose transcript is closest to the text, `ear.hear` shows what Su-śrotā heard, and `ear.words` gives each word’s start and end in milliseconds: ``` python from vachaka.check import Ear, best_of ear = Ear() lines = ['वसुदेवसुतं देवं कंसचाणूरमर्दनम्', 'देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुम्'] r = best_of(lambda seed: ch.chant(lines, seed=seed), ear, ' '.join(lines)) r.cer # 0.0 ear.hear(r.audio, r.sr) # 'वसुदेवसुतं देवं कंसचाणूरमर्दनं देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुं' ear.words(r, [l.split() for l in lines]) # [[['वसुदेवसुतं', 120, 1279], ['देवं', 1279, 1878], ['कंसचाणूरमर्दनम्', 1878, 3756]], # [['देवकीपरमानन्दं', 5100, 7018], ['कृष्णं', 7018, 7697], ['वन्दे', 7697, 8217], ['जगद्गुरुम्', 8217, 9056]]] ``` [`recite`](https://vedicreader.github.io/vachaka/recite.html#recite) does all of this for whole sections of a text. You give it each section’s lines exactly as displayed, with daṇḍas (the `।` and `॥` that end half-verses and verses), verse numbers and accent marks left in. It decides for each verse whether to chant or read it, and returns the audio together with every line’s span and words: ``` python from vachaka.recite import recite rec, info = recite(ch, ear, [dict(lines=['नित्यानन्दकरी वराभयकरी सौन्दर्यरत्नाकरी', 'निर्धूताखिलघोरपावनकरी प्रत्यक्षमाहेश्वरी ।', 'प्रालेयाचलवंशपावनकरी काशीपुराधीश्वरी', 'भिक्षां देहि कृपावलम्बनकरी मातान्नपूर्णेश्वरी ॥ १ ॥'])]) info[0]['kind'], info[0]['ref'], info[0]['lines'][0] # 'chant', 'shardulavikridita', {'s': ..., 'e': ..., 'w': [...]} ``` For batches there is a command-line tool that reads and writes JSON lines and can be stopped and restarted: `vachaka-recite jobs.jsonl results.jsonl`, where each job is `dict(id, out, sections)`. See [recite](recite.html#cli). ## How it works [`recite`](https://vedicreader.github.io/vachaka/recite.html#recite) first groups the display lines into pieces, the units Vāgdhenu renders. A piece ends after a word carrying a daṇḍa, so the two pādas (quarter-verses) of a half-verse printed on two lines render together, as Vāgdhenu expects. Each verse, which ends at a double daṇḍa, is scanned by [ganapati](https://github.com/vedicreader/ganapati) to find its metre. A verse in one of the metres Vāgdhenu has a reference recording for is chanted against that recording. Any other syllable-counted metre (a vṛtta) is chanted against the vasantatilakā reference. An āryā, whose metre counts syllable lengths instead, or anything that does not scan, is read as prose in chunks of at most 24 akṣaras (syllables). So a commentary that quotes a verse chants the verse and reads the rest. Vāgdhenu renders each piece separately and joins them with short silences, and vachaka records where each piece landed. Su-śrotā then transcribes the render one piece at a time, and if the character error rate is above 8%, the part is rendered again with another seed, keeping the best of three. Finally the words of each piece are force-aligned, matched against the audio with torchaudio’s CTC aligner over Su-śrotā’s output, inside that piece’s span, so one bad line cannot pull the rest out of place. Two things had to be patched to run this on a Mac. BigVGAN uses a transposed convolution with more than 65,536 output channels, which MPS does not support, so the vocoder runs on the CPU. And Su-śrotā’s checkpoint is configured for AI4Bharat’s fork of NeMo, so vachaka rewrites its configuration for the standard NeMo (the tokenizer type becomes `agg`, and the fork’s `multisoftmax` settings are removed) and uses only the encoder and the CTC output layer, reading the Sanskrit columns 4096 to 4351. ## Limits There are no Vedic svaras (pitch accents): Vāgdhenu chants in the classical (laukika) style, and accent marks are removed before rendering. There is one voice, the model author’s; the reference recording sets the pace and melody but not the timbre. A metre outside Vāgdhenu’s set of references is chanted to vasantatilakā when ganapati recognises it as some other vṛtta, and read as prose when it is not recognised at all. Prose sounds like careful reading, not chant. ## Credits and licences vachaka is Apache-2.0. Vāgdhenu (code Apache-2.0; weights redistributed under IndicF5’s MIT licence and NVIDIA BigVGAN’s licence) and Su-śrotā (fine-tuned from AI4Bharat’s IndicConformer, MIT) are by Prathosh A. P.; please cite Vāgdhenu if you use its audio. The model card asks that the voice not be used to impersonate anyone. Say that the audio is synthetic wherever you publish it.