check

Listen back with Su-śrotā: what a render says, how far it is from the text, the best of a few seeds, and word timings.

A text-to-speech model sometimes mumbles, skips a word or garbles a line, and nobody can listen to thousands of renders. This module uses a speech recogniser to check them instead: it transcribes each render, measures how far the transcript is from the text, re-renders the bad ones, and finds where each word sits in the audio.

The recogniser is Su-śrotā, IndicConformer fine-tuned on śāstric Sanskrit. Its CTC output layer covers all 22 IndicConformer languages, and Sanskrit is the 256 columns starting at 4096, with the blank token at 5632. The checkpoint is configured for AI4Bharat’s fork of NeMo (a multilingual tokenizer and a multisoftmax decoder), so Ear rewrites the configuration for the standard NeMo: the tokenizer type becomes agg, the settings only the fork understands are removed, and the weights are loaded without requiring every key to match. Only the encoder and the CTC output layer are used. Install it with pip install 'vachaka[check]'.

Comparing text

Su-śrotā does not write exactly what the text says. It leaves out daṇḍas, digits and the avagraha (ऽ), and writes a word-final म् as an anusvāra (ं). norm(s) brings any text to that form, so a transcript can be compared with its source fairly. cer(ref, hyp) is the character error rate: the edit distance (lev, the number of single-character insertions, deletions and substitutions) between the two normalised texts, divided by the length of the reference. Spaces are ignored, because Sanskrit texts differ in where they split words joined by sandhi.

---

source

norm

def norm(
    s
):

Text in the form Su-śrotā writes it: NFC, without accents, digits, daṇḍas or avagraha, and with word-final म् as anusvāra.

test_eq(norm('चोदनालक्षणोऽर्थो धर्मः ॥ २ ॥'), 'चोदनालक्षणो र्थो धर्मः')
test_eq(norm('कंसचाणूरमर्दनम्'), 'कंसचाणूरमर्दनं')

---

source

cer

def cer(
    ref, hyp
):

The character error rate of hyp against ref, both normalised and with spaces ignored.

---

source

lev

def lev(
    a, b
):

Edit distance between sequences a and b.

test_eq(lev('kitten', 'sitting'), 3)
test_eq(cer('अथातो धर्मजिज्ञासा', 'अथातो धर्मजिज्ञासा'), 0)
assert 0 < cer('तस्य निमित्तपरीष्टिः', 'तस्य निमित्तपरेष्टिः') < 0.1

Ear

Ear() loads Su-śrotā on the CPU. hear(audio, sr) transcribes audio and returns the normalised text. post(audio, sr) returns the model’s raw output for each 40 ms frame, the log-probability of the blank and of each of the 256 Sanskrit tokens, together with the frame length in milliseconds; words below uses it to align.

---

source

Ear

def Ear():

Su-śrotā on the CPU, for transcripts and word timings.

Best of a few seeds

Each render uses a random seed, and a different seed gives a slightly different performance. best_of(render, ear, text) calls render(seed=...) up to tries (3) times with different seeds, measures each result’s character error rate against text, and keeps the best, stopping early once one is at or below max_cer (0.08). The rate is stored on the result as Recitation.cer.

ear.heard(rec) transcribes a render one piece at a time, using its spans, and joins the results. Transcribing the whole render at once fails on long runs of nearly identical lines, such as a tarpaṇa or the eight directions of a digvandana: the recogniser loses its place, and good renders scored between 0.47 and 0.76.

---

source

Ear.heard

def heard(
    rec
):

Transcribe rec one piece of its spans at a time and join the transcripts.

class _E(Ear):
    def __init__(self): self.got = []
    def hear(self, audio, sr): self.got.append(len(audio)); return 'क'
e, r = _E(), AttrDict(audio=np.zeros(16000*3, np.float32), sr=16000, spans=[(0, 1000), (1500, 3000)])
test_eq(e.heard(r), 'क क'); test_eq(e.got, [16000, 24000])
test_eq(_E().heard(AttrDict(audio=np.zeros(800, np.float32), sr=16000)), 'क')

---

source

best_of

def best_of(
    render, ear, text, tries:int=3, max_cer:float=0.08, seed:int=60
):

Call render(seed=...) up to tries times and keep the render whose transcript is closest to text.

Word timings

ear.words(rec, words) finds the start and end of each word, given a render and, for each of its pieces, the list of words as displayed. It force-aligns them, matching the known word sequence against the audio with torchaudio’s CTC aligner, inside each piece’s own span, so a badly rendered line cannot drag the timings of the others. Each word is tokenised from its cleaned form. A word with no Sanskrit tokens, such as a lone daṇḍa, gets an empty span at the end of the word before it. The result has one list per piece, of [word, start_ms, end_ms].

---

source

Ear.words

def words(
    rec, words
):

The [word, start_ms, end_ms] of each displayed word, for each piece of rec.

These cells need the Su-śrotā checkpoint (520 MB) and the Vāgdhenu weights, so they do not run in the tests.

from vachaka.core import Chanter
ch, ear = Chanter(), Ear()
lines = ['वसुदेवसुतं देवं कंसचाणूरमर्दनम्', 'देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुम्']
r = best_of(partial(ch.chant, lines), ear, ' '.join(lines))
r.cer, ear.words(r, [l.split() for l in lines])