# check
A text-to-speech model sometimes mumbles, skips a word or garbles a
line, and nobody can listen to thousands of renders. This module uses a
speech recogniser to check them instead: it transcribes each render,
measures how far the transcript is from the text, re-renders the bad
ones, and finds where each word sits in the audio.
The recogniser is
[Su-śrotā](https://huggingface.co/prathoshap/sushrota-sanskrit-asr),
IndicConformer fine-tuned on śāstric Sanskrit. Its CTC output layer
covers all 22 IndicConformer languages, and Sanskrit is the 256 columns
starting at 4096, with the blank token at 5632. The checkpoint is
configured for AI4Bharat’s fork of NeMo (a `multilingual` tokenizer and
a `multisoftmax` decoder), so
[`Ear`](https://vedicreader.github.io/vachaka/check.html#ear) rewrites
the configuration for the standard NeMo: the tokenizer type becomes
`agg`, the settings only the fork understands are removed, and the
weights are loaded without requiring every key to match. Only the
encoder and the CTC output layer are used. Install it with
`pip install 'vachaka[check]'`.
## Comparing text
Su-śrotā does not write exactly what the text says. It leaves out
daṇḍas, digits and the avagraha (`ऽ`), and writes a word-final `म्` as an
anusvāra (`ं`). `norm(s)` brings any text to that form, so a transcript
can be compared with its source fairly. `cer(ref, hyp)` is the character
error rate: the edit distance
([`lev`](https://vedicreader.github.io/vachaka/check.html#lev), the
number of single-character insertions, deletions and substitutions)
between the two normalised texts, divided by the length of the
reference. Spaces are ignored, because Sanskrit texts differ in where
they split words joined by sandhi.
------------------------------------------------------------------------
source
### norm
``` python
def norm(
s
):
```
*Text in the form Su-śrotā writes it: NFC, without accents, digits,
daṇḍas or avagraha, and with word-final म् as anusvāra.*
``` python
test_eq(norm('चोदनालक्षणोऽर्थो धर्मः ॥ २ ॥'), 'चोदनालक्षणो र्थो धर्मः')
test_eq(norm('कंसचाणूरमर्दनम्'), 'कंसचाणूरमर्दनं')
```
------------------------------------------------------------------------
source
### cer
``` python
def cer(
ref, hyp
):
```
*The character error rate of `hyp` against `ref`, both normalised and
with spaces ignored.*
------------------------------------------------------------------------
source
### lev
``` python
def lev(
a, b
):
```
*Edit distance between sequences `a` and `b`.*
``` python
test_eq(lev('kitten', 'sitting'), 3)
test_eq(cer('अथातो धर्मजिज्ञासा', 'अथातो धर्मजिज्ञासा'), 0)
assert 0 < cer('तस्य निमित्तपरीष्टिः', 'तस्य निमित्तपरेष्टिः') < 0.1
```
## Ear
[`Ear()`](https://vedicreader.github.io/vachaka/check.html#ear) loads
Su-śrotā on the CPU. `hear(audio, sr)` transcribes audio and returns the
normalised text. `post(audio, sr)` returns the model’s raw output for
each 40 ms frame, the log-probability of the blank and of each of the
256 Sanskrit tokens, together with the frame length in milliseconds;
`words` below uses it to align.
------------------------------------------------------------------------
source
### Ear
``` python
def Ear():
```
*Su-śrotā on the CPU, for transcripts and word timings.*
## Best of a few seeds
Each render uses a random seed, and a different seed gives a slightly
different performance. `best_of(render, ear, text)` calls
`render(seed=...)` up to `tries` (3) times with different seeds,
measures each result’s character error rate against `text`, and keeps
the best, stopping early once one is at or below `max_cer` (0.08). The
rate is stored on the result as `Recitation.cer`.
`ear.heard(rec)` transcribes a render one piece at a time, using its
`spans`, and joins the results. Transcribing the whole render at once
fails on long runs of nearly identical lines, such as a tarpaṇa or the
eight directions of a digvandana: the recogniser loses its place, and
good renders scored between 0.47 and 0.76.
------------------------------------------------------------------------
source
### Ear.heard
``` python
def heard(
rec
):
```
*Transcribe `rec` one piece of its `spans` at a time and join the
transcripts.*
``` python
class _E(Ear):
def __init__(self): self.got = []
def hear(self, audio, sr): self.got.append(len(audio)); return 'क'
e, r = _E(), AttrDict(audio=np.zeros(16000*3, np.float32), sr=16000, spans=[(0, 1000), (1500, 3000)])
test_eq(e.heard(r), 'क क'); test_eq(e.got, [16000, 24000])
test_eq(_E().heard(AttrDict(audio=np.zeros(800, np.float32), sr=16000)), 'क')
```
------------------------------------------------------------------------
source
### best_of
``` python
def best_of(
render, ear, text, tries:int=3, max_cer:float=0.08, seed:int=60
):
```
*Call `render(seed=...)` up to `tries` times and keep the render whose
transcript is closest to `text`.*
## Word timings
`ear.words(rec, words)` finds the start and end of each word, given a
render and, for each of its pieces, the list of words as displayed. It
force-aligns them, matching the known word sequence against the audio
with torchaudio’s CTC aligner, inside each piece’s own span, so a badly
rendered line cannot drag the timings of the others. Each word is
tokenised from its cleaned form. A word with no Sanskrit tokens, such as
a lone daṇḍa, gets an empty span at the end of the word before it. The
result has one list per piece, of `[word, start_ms, end_ms]`.
------------------------------------------------------------------------
source
### Ear.words
``` python
def words(
rec, words
):
```
*The `[word, start_ms, end_ms]` of each displayed word, for each piece
of `rec`.*
These cells need the Su-śrotā checkpoint (520 MB) and the Vāgdhenu
weights, so they do not run in the tests.
``` python
from vachaka.core import Chanter
ch, ear = Chanter(), Ear()
lines = ['वसुदेवसुतं देवं कंसचाणूरमर्दनम्', 'देवकीपरमानन्दं कृष्णं वन्दे जगद्गुरुम्']
r = best_of(partial(ch.chant, lines), ear, ' '.join(lines))
r.cer, ear.words(r, [l.split() for l in lines])
```