editions

Lay verses out as titles for publishing, report a run’s numbering, check a reading against another edition, and measure how far two readings differ

These functions work on the chapters assemble returns, [{heading, colophon, verses}], where a colophon is the line that closes a chapter. A few work on a single page of blocks, the pieces of a page a vision model returns: verses, headings, colophons, commentary and prose. Several compare a reading with another edition of the same work, called a witness. You fetch or load the witness yourself and pass it in; nothing here reads a file or goes to the web.

Titles

vedicreader publishes a work as a set of titles, each a page of verses with its own name. cut divides chapters into titles, each {name, title, chapter, verses}, in one of three ways chosen by how:

how='per', the default, makes one title for every per verses (10 by default) across the whole work, by the verses’ own numbers: verses 1 to 10, then 11 to 20, and so on. An unnumbered verse goes with the numbered verse before it, and unnumbered verses before the first numbered one go with the first title. A title’s range stops at the last number it actually holds, and every title has chapter 1.
how='chapters' makes one title for each chapter, named from book, the chapter’s number and its heading.
how='one' puts the whole work in a single title.

name is a file name built from stem, such as work_011_012 or work_02, and title is the heading a reader sees, with its numbers written by digits (Devanagari by default).

A verse with no number, such as a maṅgala (a verse of blessing that opens a book) or a verse a commentary quotes, still needs a label. cut gives it a letter, क, ख, ग and so on, starting again from क after each numbered verse, and sets its word to open_word (मङ्गलम् by default). An unnumbered verse therefore never takes a number that belongs to a numbered verse.

numbering reports what a run of verses’ own numbers say: how many verses are numbered and unnumbered, which numbers are skipped or repeated, and the first and last.

source

numbering

def numbering(
    vs
)->dict:

Report what a run of verses’ own numbers say: how many are numbered and unnumbered, which are skipped or repeated, and the first and last.

source

cut

def cut(
    chs, how:str='per', per:int=10, open_word:str='मङ्गलम्', stem:str='work', book:str='', digits:function=deva,
    letters:str='कखगघङचछजझञ'
)->list:

Divide chapters’ verses into titles [{name, title, chapter, verses}]: one for every per verses, one for each chapter, or one for the whole work.

V = lambda n, t='x': dict(n=n, text=t, page=1, heading=None)
_cs = [dict(heading='प्रथमः', verses=[V(None), V(None), V('१'), V(2)]), dict(heading='द्वितीयः', verses=[V(3), V(None), V('௧௧'), V(12)])]
test_eq([(t['name'], t['title'], [v['n'] for v in t['verses']]) for t in cut(_cs, per=10, book='शतकम्')],
        [('work_001_003', 'शतकम् १–३', ['क', 'ख', 1, 2, 3, 'क']), ('work_011_012', 'शतकम् ११–१२', [11, 12])])
test_eq(cut(_cs)[0]['verses'][0]['word'], 'मङ्गलम्'); test_eq('word' in cut(_cs)[0]['verses'][2], False)
test_eq([(t['name'], t['title']) for t in cut(_cs, 'chapters', stem='s', book='ग्रन्थः')], [('s_01', 'ग्रन्थः १ प्रथमः'), ('s_02', 'ग्रन्थः २ द्वितीयः')])
test_eq(len(cut(_cs, 'one')[0]['verses']), 8)
test_eq(numbering([V(1), V(None), V('३'), V('௩'), V(6)]), dict(numbered=4, unnumbered=1, skipped=[2, 4, 5], repeated=[3], first=1, last=6))

A witness

A witness is another edition of the same text, used to check a reading. It might be a page from sanskritdocuments.org or a GRETIL file. witness_verses takes the raw page, HTML or plain text, removes the markup, and cuts it into verses at each verse number written as ॥ n ॥ or || n ||, returning [(n, text)]. GRETIL writes each number with a reference, as in || BharSt_1.12 ||, and the verse number is taken from its end. Only the last six lines before each number are kept, so a page header does not end up in the first verse. If the page is in IAST (Roman letters with diacritics) instead of Devanagari, pass a translit function; it is used when the page has fewer than 200 Devanagari characters. A witness shows where to look again; the reading should still follow the scan, not the witness.

Two editions differ in punctuation, numbering and notes even where their words agree. letters reduces a text to its Devanagari letters alone, removing the verse number between ॥ marks, bracketed notes, spaces, daṇḍas and digits. The checks below compare texts in this form.

text_changes proposes corrections and does not make them. It does two checks.

On each line it looks for a pāda-final virāma that was dropped, using virama. A pāda is a quarter of a verse, usually ending in a daṇḍa, and a virāma (्) is the mark that removes a consonant’s vowel. These changes have by set to ['rule'].
When you pass wit, the output of witness_verses, it finds the witness verse whose letters are most like the verse’s, if they are at least at alike (0.7 by default), and compares the two word by word. Where one or two words are spelled differently, it proposes the witness’s spelling, with by set to ['witness']. You can also pass metre, a function that takes a verse’s lines and returns the metre’s name if they scan, or None. A verse that already scans is not compared, and if a proposed change makes a verse scan, by also lists 'metre'.

Each change is a dict {id, chapter, n, page, read, print, why, by}. A proofreader accepts or rejects each one, and an accepted change can be applied as a fix row for fix_chapters, with its read and print as the read and printed text.

source

text_changes

def text_changes(
    chs, wit:tuple=(), metre:NoneType=None, at:float=0.7, key:NoneType=None
)->list:

Propose changes for dropped pāda-final virāmas, and for words the nearest witness verse spells differently in verses that metre does not find scanning.

source

letters

def letters(
    s:str
)->str:

Reduce s to its Devanagari letters, removing the closing ॥ n ॥, bracketed notes, spaces, daṇḍas and digits.

source

witness_verses

def witness_verses(
    raw:str, translit:NoneType=None
)->list:

Cut a witness edition’s raw page into verses [(n, text)] at each ॥ n ॥ or || n ||, with the markup removed; translit turns an IAST page into Devanagari.

_raw = '<html><script>x()</script><p>नीतिशतकम्</p><p>दिक्कालाद्यनवच्छिन्नानन्तचिन्मात्रमूर्तये ।<br>स्वानुभूत्येकमानाय नमः शान्ताय तेजसे ॥ १॥</p><p>बोद्धारो मत्सरग्रस्ताः प्रभवः स्मयदूषिताः ।<br>अबोधोपहताश्चान्ये जीर्णमङ्गे सुभाषितम् ॥ 2 ॥</p></html>'
_w = witness_verses(_raw)
test_eq([n for n, _ in _w], [1, 2]); test_eq(_w[1][1], 'बोद्धारो मत्सरग्रस्ताः प्रभवः स्मयदूषिताः ।\nअबोधोपहताश्चान्ये जीर्णमङ्गे सुभाषितम्')
test_eq([n for n, _ in witness_verses('a b c || BharSt_1.12 || d e f || BharSt_1.13 ||')], [12, 13])
test_eq([t for _, t in witness_verses('rama || 1 ||', translit=str.upper)], ['RAMA'])   # a short Latin page goes through `translit`
test_eq(letters('राम (रामः) ॥ १ ॥ सीता ।'), 'रामसीता')
_ch = [dict(verses=[dict(n=1, page='p1', text='दिक्कालाद्यनवच्छिन्नानन्तचिन्मात्रमूर्तये ।\nस्वानुभूत्येकमानाय नमः शान्ताय तेजसे ॥ १॥'),
                    dict(n=2, page='p1', text='बोद्धारो मत्सरग्रस्ताः प्रभवः स्मयदूषिताः ।\nअबोधोपहताश्चान्ये जीर्णमङ्गे सुभाषितम ॥ २॥')])]
_x = text_changes(_ch, _w)
test_eq([(x['id'], x['read'], x['print'], x['by']) for x in _x], [('1.2.w6', 'सुभाषितम', 'सुभाषितम्', ['witness'])])
test_eq([(x['id'], x['read'], x['print'], x['by']) for x in text_changes([dict(verses=[dict(n=1, text='नमो भगवत ।\nवासुदेवाय ॥ १॥')])])], [('1.1.v0', 'भगवत', 'भगवत्', ['rule'])])
test_eq([x['by'] for x in text_changes(_ch, _w, metre=lambda ls: 'anuṣṭubh' if 'सुभाषितम्' in ls[-1] else None)], [['witness', 'metre']])

Against a reference edition

A reference edition is a complete text of the same work, close enough to compare whole verses with. For ref_check, ref is {chapter: [letters of each verse]}, built with letters as in the test below.

ref_check reports problems and changes nothing. It compares each verse with the reference verses near it, meaning within win (6) places of its position or of its number, and flags it when even the best match is less than at (0.8) alike. That usually means a misread or a lost line. It also flags a verse containing a lacuna mark (·· or …), where the print itself shows missing text. Verses past the end of a shorter reference are not checked.

The model sometimes puts the boundary between two verses in the wrong place, so that the last line of verse 4 is read as the first line of verse 5. move_lines repairs this by comparing with a reference given as {(chapter, n): text}, where each text has been reduced by fold (letters by default, or a function of your own for a reference in IAST). When a verse matches its reference much better without its first lines, move_lines does one of three things. If adding those lines to the verse before makes that verse match its own reference better, it moves them back. If the rest of the verse then matches closely and the lines match no reference verse from n-2 to n+2, they are taken as a quotation the editor printed, and dropped. Otherwise it changes nothing and asks for a check by hand. Each case is recorded in probs.

In both functions key(i, chapter) gives the name a chapter has in the reference; by default it is the chapter’s position counting from 1.

source

move_lines

def move_lines(
    chs, ref, probs:list=None, fold:function=letters, key:NoneType=None
)->list:

Move the leading lines of verse n back to verse n-1 when they belong there, or drop them when they match no nearby verse, judged against ref.

source

ref_check

def ref_check(
    chs, ref, probs:list=None, at:float=0.8, win:int=6, key:NoneType=None
)->list:

List the verses that read unlike every nearby verse of the reference edition, and verses with a lacuna, without changing anything.

import json
from lekhaka.verses import assemble

_bj, _ = assemble(json.load(open('samples/brihat_jataka.json')))
_ref = {ci: [letters(v['text']) for v in c['verses']] for ci, c in enumerate(_bj, 1)}
test_eq(ref_check(_bj, _ref), [])
_v = dict(_bj[0]['verses'][3], text='पृथिव्यापस्तेजो वायुराकाशमेव च ।\nएतानि पञ्च भूतानि ॥')   # a verse the reference does not have
_p = ref_check([dict(verses=_bj[0]['verses'][:3] + [_v, dict(_v, n=5, text='क ···· ख')])], _ref); test_eq(len(_p), 3); assert 'lacuna' in _p[1]
_ref2 = {(ci, v['n']): letters(v['text']) for ci, c in enumerate(_bj, 1) for v in c['verses']}
_a, _b = _bj[0]['verses'][0], _bj[0]['verses'][1]
_moved = [dict(_a, text=_a['text'].split('\n')[0]), dict(_b, text=_a['text'].split('\n')[1] + '\n' + _b['text'])]   # verse 1's last line read as verse 2's first
_p = []; _m = move_lines([dict(verses=_moved)], _ref2, _p)[0]['verses']
test_eq([v['text'] for v in _m], [_a['text'], _b['text']]); test_eq(_p, ['1.2: 1 leading line(s) moved back to 1.1'])
_q = [_a, dict(_b, text='पृथिव्यापस्तेजो वायुराकाशमेव च ।\n' + _b['text'])]
_p = []; test_eq(move_lines([dict(verses=_q)], _ref2, _p)[0]['verses'][1]['text'], _b['text']); assert 'quotation' in _p[0]

Two readings of a page

When you try a second model, or read a book again, you want to know where the two readings differ. These functions measure that page by page.

verse_text joins the text of a page’s verse blocks as bare letters, removing numbers, daṇḍas, spaces, punctuation and the avagraha (ऽ). With numbered=True it keeps only the verses read with a number, which leaves out the part of a verse carried over a page break and verses a commentary quotes. lev counts the character edits (insertions, deletions and substitutions) that turn one string into another. disagree is the share of a page’s text that differs between two readings: the edits divided by the length of the longer text. It is 1 when only one reading found any verse on the page, and None when neither did or one reading lacks the page.

compare takes named readings, each a list of pages or a dict from page name to page, and pairs of names to compare. It returns a markdown table and a list of flagged pages, the pages where the first pair differs by at (10%) or more. The table shows only the flagged pages, followed by one summary line per pair giving the number of pages, the mean difference and how many pages reach at. If you pass pages, the table shows exactly those pages instead.

source

compare

def compare(
    readings:dict, pairs, at:float=0.1, pages:NoneType=None, numbered:bool=False
)->tuple:

A markdown table of how far each pair of named readings differs, page by page, and the pages where the first pair differs by at or more.

source

disagree

def disagree(
    a, b, numbered:bool=False
)->float | None:

The share of a page’s letters that differ between two readings: 1 when only one found verses, None when neither did or a page is missing.

source

lev

def lev(
    a:str, b:str
)->int:

The number of character edits (the Levenshtein distance) between two strings.

source

verse_text

def verse_text(
    page, numbered:bool=False
)->str:

A page’s verse text as bare letters; numbered=True keeps only the verses read with a number.

test_eq(lev('kitten', 'sitting'), 3); test_eq(verse_text(dict(items=[dict(kind='verse', text='राम ॥ १ ॥'), dict(kind='prose', text='x'), dict(kind='verse', text='सीता')]), numbered=True), '')
_one = json.load(open('samples/brihat_jataka.json'))
_two = [dict(p, blocks=[dict(b, text=b['text'].replace('ग्र', 'ग')) for b in p['blocks']]) for p in _one]
_two[3] = dict(_two[3], blocks=[dict(b, text='अ') if b['kind'] == 'verse' else b for b in _two[3]['blocks']])
test_eq(disagree(_one[0], _one[0]), 0); assert 0 < disagree(_one[0], _two[0]) < .1; test_eq(disagree(_one[0], None), None)
_md, _fl = compare(dict(a=_one, b=_two), [('a', 'b')])
test_eq(_fl, [_one[3]['page']]); assert _md.startswith('| page | a vs b |') and '16 pages' in _md