vachaka

Sanskrit recitation from text: Vāgdhenu chant and prose voices with line and word timings.

vachaka (वाचक, “reciter”) turns Sanskrit text in Devanagari into recited audio, and tells you where every line and every word falls in that audio. It is a thin layer over two models by Prathosh A. P. Vāgdhenu is a text-to-speech model that chants: given a verse and a reference recording in the verse’s metre, it sings the verse to that metre’s tune. It is a fine-tune of IndicF5 with a BigVGAN-v2 vocoder, the part that turns the model’s output into a waveform. Su-śrotā is a Sanskrit speech recogniser, IndicConformer fine-tuned on śāstric Sanskrit. Vāgdhenu does the speaking. Su-śrotā listens back: it transcribes each render and compares the transcript with the text, so a bad render can be thrown away and tried again, and it finds where each word starts and ends.

vachaka was written for VedicReader, which has thousands of Sanskrit verses and sūtras that nobody has recorded. There the input is a work’s lines as they are displayed, and the output must be one audio file plus a [start, end] time for every line and word, so the reader can highlight the text as it plays. vachaka.recite does exactly that.

Listen

These clips come from a trial run over VedicReader’s texts. The voice is synthetic, and it is that of Vāgdhenu’s author. CER, the character error rate, is the share of characters Su-śrotā got wrong when it transcribed the clip, compared with the text: 0.0 means every character was heard as written. The audio players work on the docs site; GitHub and PyPI show only the text. Each clip says whether it was chanted or read, and which reference recording it was rendered against.

Annapūrṇāṣṭakam, verse 2 (śārdūlavikrīḍita)

नानारत्नविचित्रभूषणकरी हेमाम्बराडम्बरी
मुक्ताहारविलम्बमानविलसद्वक्षोजकुम्भान्तरी ।
काश्मीरागरुवासिताङ्गरुचिरे काशीपुराधीश्वरी
भिक्षां देहि कृपावलम्बनकरी मातान्नपूर्णेश्वरी ॥ २ ॥

chant, reference shardulavikridita, CER 0.031

Bṛhat Parāśara Horā Śāstra 1.3 (anuṣṭubh)

इति स्तुत्या सुसंहृष्टो मुनिस्तत्त्वविदां वरः।
अथादिदेश सच्छास्त्रं सारं यज्ज्योतिषां शुभम्॥ ३॥

chant, reference anushtubh, CER 0.0

Jaimini Sūtras 1.1.1 (prose)

उपदेशं व्याख्यास्यामः॥

read, reference anuṣṭubh, CER 0.0

One sūtra, two references

Nyāyasūtra 1.1.2 rendered twice. Vāgdhenu ships two reference recordings for prose (gadya), but its author renders prose against the anuṣṭubh reference instead, having found that the gadya references drop the starts of words. When recite reads prose it follows the author and uses anuṣṭubh. Judge for yourself:

anuṣṭubh reference

gadya reference

दुःखजन्मप्रवृत्तिदोषमिथ्याज्ञानानामुत्तरोत्तरापाये तदनन्तरापायादपवर्गः

Numbers

Measured on an Apple M3 Max with 36 GB of memory, running Vāgdhenu’s main network (a diffusion transformer, or DiT) on the GPU through MPS and the vocoder on the CPU.

loading Vāgdhenu	10 to 17 s
rendering, warm	about 5.5 s per second of audio (a 9.5 s anuṣṭubh verse in 50 to 57 s)
share of render time in the DiT	77% (the rest is BigVGAN on the CPU)
loading Su-śrotā	9 s
transcribing a 10 s clip on the CPU	0.4 s
alignment resolution	40 ms (Su-śrotā’s frame)

At that speed an hour of recitation takes five to six hours on a laptop. A CUDA GPU is much faster, and Vāgdhenu’s author renders on one.

The trial rendered 27 sections, 293 s of audio in all, from a stotra, three jyotiṣa books, the Jaimini Sūtras and the Nyāyasūtra, keeping the best of up to three renders per section. “Exact” counts the sections Su-śrotā transcribed with no error.

voice	sections	mean CER	worst	exact
chant, reference for the detected metre	9	0.030	0.086	2
prose, anuṣṭubh reference	14	0.016	0.118	9
prose, gadya reference	4	0.010	0.039	3

Two cautions. Su-śrotā comes from the same lab as Vāgdhenu and its training data includes recordings of a TTS speaker, so it may hear Vāgdhenu more kindly than a person would. A low CER says the words are there, not that the chant sounds good. And a section whose metre is not detected (an āryā, or a verse split over three lines) is read as prose, which sounds flatter than a chant. Listen before you publish.

Install

pip install vachaka            # rendering
pip install 'vachaka[check]'   # Su-śrotā: transcripts, error rate, word timings

vachaka needs Python 3.10 to 3.12. The first Chanter() downloads about 1.5 GB of weights from the Hugging Face Hub, and fetches the source code of Vāgdhenu, BigVGAN and IndicF5 at fixed commits into ~/.cache/vachaka (set VACHAKA_CACHE to put it elsewhere). None of the three is on PyPI, and the package called f5-tts there is not IndicF5. Su-śrotā is another 520 MB. Writing mp3 files needs ffmpeg.

Use

Chanter loads Vāgdhenu. chant sings a verse given as a list of its lines, picking the tune from the verse’s metre, and read reads prose. Both return a Recitation, which holds the audio and one (start, end) span in milliseconds for each line or chunk of prose, and can be saved as wav or mp3:

from vachaka.core import Chanter
ch = Chanter()                        # cuda, then mps, then cpu
r = ch.chant(['वसुदेवसुतं देवं कंसचाणूरमर्दनम्', 'देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुम्'])
r.spans                               # [(0, 4230), (4980, 9550)]: one span per line, in ms
r.save('krishna.mp3')
ch.read('अथातो धर्मजिज्ञासा । चोदनालक्षणोऽर्थो धर्मः ।', ref='anuṣṭubh').save('mimamsa.mp3')

Ear loads Su-śrotā. best_of renders up to three times with different random seeds and keeps the render whose transcript is closest to the text, ear.hear shows what Su-śrotā heard, and ear.words gives each word’s start and end in milliseconds:

from vachaka.check import Ear, best_of
ear = Ear()
lines = ['वसुदेवसुतं देवं कंसचाणूरमर्दनम्', 'देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुम्']
r = best_of(lambda seed: ch.chant(lines, seed=seed), ear, ' '.join(lines))
r.cer                                 # 0.0
ear.hear(r.audio, r.sr)               # 'वसुदेवसुतं देवं कंसचाणूरमर्दनं देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुं'
ear.words(r, [l.split() for l in lines])
# [[['वसुदेवसुतं', 120, 1279], ['देवं', 1279, 1878], ['कंसचाणूरमर्दनम्', 1878, 3756]],
#  [['देवकीपरमानन्दं', 5100, 7018], ['कृष्णं', 7018, 7697], ['वन्दे', 7697, 8217], ['जगद्गुरुम्', 8217, 9056]]]

recite does all of this for whole sections of a text. You give it each section’s lines exactly as displayed, with daṇḍas (the । and ॥ that end half-verses and verses), verse numbers and accent marks left in. It decides for each verse whether to chant or read it, and returns the audio together with every line’s span and words:

from vachaka.recite import recite
rec, info = recite(ch, ear, [dict(lines=['नित्यानन्दकरी वराभयकरी सौन्दर्यरत्नाकरी', 'निर्धूताखिलघोरपावनकरी प्रत्यक्षमाहेश्वरी ।',
                                         'प्रालेयाचलवंशपावनकरी काशीपुराधीश्वरी', 'भिक्षां देहि कृपावलम्बनकरी मातान्नपूर्णेश्वरी ॥ १ ॥'])])
info[0]['kind'], info[0]['ref'], info[0]['lines'][0]   # 'chant', 'shardulavikridita', {'s': ..., 'e': ..., 'w': [...]}

For batches there is a command-line tool that reads and writes JSON lines and can be stopped and restarted: vachaka-recite jobs.jsonl results.jsonl, where each job is dict(id, out, sections). See recite.

How it works

recite first groups the display lines into pieces, the units Vāgdhenu renders. A piece ends after a word carrying a daṇḍa, so the two pādas (quarter-verses) of a half-verse printed on two lines render together, as Vāgdhenu expects. Each verse, which ends at a double daṇḍa, is scanned by ganapati to find its metre. A verse in one of the metres Vāgdhenu has a reference recording for is chanted against that recording. Any other syllable-counted metre (a vṛtta) is chanted against the vasantatilakā reference. An āryā, whose metre counts syllable lengths instead, or anything that does not scan, is read as prose in chunks of at most 24 akṣaras (syllables). So a commentary that quotes a verse chants the verse and reads the rest.

Vāgdhenu renders each piece separately and joins them with short silences, and vachaka records where each piece landed. Su-śrotā then transcribes the render one piece at a time, and if the character error rate is above 8%, the part is rendered again with another seed, keeping the best of three. Finally the words of each piece are force-aligned, matched against the audio with torchaudio’s CTC aligner over Su-śrotā’s output, inside that piece’s span, so one bad line cannot pull the rest out of place.

Two things had to be patched to run this on a Mac. BigVGAN uses a transposed convolution with more than 65,536 output channels, which MPS does not support, so the vocoder runs on the CPU. And Su-śrotā’s checkpoint is configured for AI4Bharat’s fork of NeMo, so vachaka rewrites its configuration for the standard NeMo (the tokenizer type becomes agg, and the fork’s multisoftmax settings are removed) and uses only the encoder and the CTC output layer, reading the Sanskrit columns 4096 to 4351.

Limits

There are no Vedic svaras (pitch accents): Vāgdhenu chants in the classical (laukika) style, and accent marks are removed before rendering. There is one voice, the model author’s; the reference recording sets the pace and melody but not the timbre. A metre outside Vāgdhenu’s set of references is chanted to vasantatilakā when ganapati recognises it as some other vṛtta, and read as prose when it is not recognised at all. Prose sounds like careful reading, not chant.

Credits and licences

vachaka is Apache-2.0. Vāgdhenu (code Apache-2.0; weights redistributed under IndicF5’s MIT licence and NVIDIA BigVGAN’s licence) and Su-śrotā (fine-tuned from AI4Bharat’s IndicConformer, MIT) are by Prathosh A. P.; please cite Vāgdhenu if you use its audio. The model card asks that the voice not be used to impersonate anyone. Say that the audio is synthetic wherever you publish it.