repairs

Apply corrections a person makes from the scan, number verses the model left unnumbered, mark chapter breaks, drop pages that are not the text, and tidy sūtras

Some mistakes in a reading cannot be fixed by rule. A person has to look at the scan, decide what the page really says, and write that decision down. The functions here apply those decisions, along with a few repairs that need a hint from you, such as where a chapter starts.

Most of them work on pages of blocks, [{page, blocks}] (or items), before assemble puts the verses into chapters. A block is one piece of a page as the vision model read it: a verse, a heading, a colophon (the line that closes a chapter), a commentary or prose. These functions change the pages in place and also return them. fix_chapters and sutras work later, on the chapters assemble returns. Every change goes into probs, a list you pass in, so the list of problems also says what was changed and why.

They were written for books vedicreader read and had to repair again and again: the fix tables for the Bṛhat Saṃhitā, the Muhūrta Cintāmaṇi and the Nārada Saṃhitā, the unnumbered runs of the Phaladīpikā, the second readings of the Jātaka Pārijāta, and the chapter plans of the Grahacintāmaṇi.

A table of fixes

Each correction is a fix row, (page, item, kind, read, printed, why). page and item say where the mistake is: the page’s name and the block’s position on that page, counting from 0. read is the text as the model read it, printed is what the page prints, and why is your reason, which goes into probs with the change. kind says what to do:

fix replaces the first occurrence of read in the block’s text with printed, and fix_all replaces every occurrence.
doubt is a fix you are not sure of. It is applied only when witness, a dict keyed by (page, item, read), holds 'agrees' for it, meaning another edition reads the same way. Otherwise the text is kept and the doubt is recorded.
slip records a mistake that the printed book itself makes. The text is kept, since it matches the print.
number sets the block’s number to printed.
drop removes the block.
missing adds a block the model skipped. printed is the whole block as a dict, and it goes in after position item, or first on the page when item is -1.

Blocks are removed and added only after every row has been read, so the positions in all the rows refer to the page as the model read it. A row for a page that is not in the reading is ignored. A row whose position is past the end of the page, or whose read text is not in the block, is not applied, and that is recorded in probs. The test below applies rows of several kinds to one small page.

source

apply_fixes

def apply_fixes(
    pages, rows, witness:dict=None, probs:list=None
)->list:

Apply a table of fix rows (page, item, kind, read, printed, why) to pages of blocks, recording each change in probs.

_pg = lambda: [dict(page='p1', blocks=[dict(kind='verse', n=1, text='रामः वनं गच्छति ॥१॥'), dict(kind='verse', n=3, text='सीता ॥३॥'), dict(kind='prose', text='a note')])]
_p = []
_r = apply_fixes(_pg(), [('p1', 0, 'fix', 'गच्छति', 'गच्छत्', 'the print'), ('p1', 1, 'number', 3, 2, ''), ('p1', 2, 'drop', None, None, 'a footnote'),
                         ('p1', 1, 'missing', None, dict(kind='verse', n=3, text='लक्ष्मणः ॥३॥'), 'at the foot'), ('p1', 0, 'doubt', 'रामः', 'राम', 'faint')], probs=_p)
test_eq([(b['n'], b['text']) for b in _r[0]['blocks']], [(1, 'रामः वनं गच्छत् ॥१॥'), (2, 'सीता ॥३॥'), (3, 'लक्ष्मणः ॥३॥')])
test_eq(len(_p), 5); assert any('doubt' in x for x in _p)
test_eq(apply_fixes(_pg(), [('p1', 0, 'doubt', 'रामः', 'राम', '')], witness={('p1', 0, 'रामः'): 'agrees'})[0]['blocks'][0]['text'][:3], 'राम')
_p = []; test_eq(apply_fixes([dict(page=0, blocks=[dict(kind='verse', text='क क ख')])], [(0, 0, 'fix_all', 'क', 'ग', '')], probs=_p)[0]['blocks'][0]['text'], 'ग ग ख')
test_eq(_p, ['0: read `क`, printed `ग`, 2 times'])

Numbers

Models often miss a verse number, especially one printed small or faint, or at the edge of the page. Two functions recover such numbers without asking a person.

If verse 4 follows verse 1 with two unnumbered verse blocks between them, those two must be verses 2 and 3. number_runs numbers a run of unnumbered verses when it exactly fills the gap between two numbered ones, and leaves it alone otherwise. Only blocks of kind verse count. The count starts again from 0 after a colophon, and after a heading that matches reset (ध्याय by default, as in अध्यायः), so the first verses of a new chapter can be numbered from 1.

borrow_numbers uses a second reading of the same pages, by another model or another run. For each unnumbered verse, it finds the most similar numbered verse on the same page of the other reading and takes its number, if the two texts are at least at alike (0.6 by default, where 1 means identical) and no verse on this page already has that number.

source

borrow_numbers

def borrow_numbers(
    pages, other, probs:list=None, at:float=0.6
)->list:

Give each unnumbered verse the number of the most similar verse on the same page of other, a second reading of the same pages.

source

number_runs

def number_runs(
    pages, probs:list=None, reset:str='ध्याय'
)->list:

Number each run of unnumbered verses that exactly fills the gap between two numbered ones.

_ps = [dict(page=0, blocks=[dict(kind='verse', n=1, text='a'), dict(kind='verse', text='b'), dict(kind='verse', text='c'), dict(kind='verse', n=4, text='d'),
                             dict(kind='colophon', text='इति'), dict(kind='verse', text='e'), dict(kind='verse', n=3, text='f')])]
test_eq([b.get('n') for b in number_runs(_ps)[0]['blocks']], [1, 2, 3, 4, None, None, 3])   # after the colophon one verse cannot fill 1..2
_mine = [dict(page=0, blocks=[dict(kind='verse', text='धर्मक्षेत्रे कुरुक्षेत्रे'), dict(kind='verse', n=2, text='दृष्ट्वा तु')])]
_theirs = [dict(page=0, blocks=[dict(kind='verse', n='१', text='धर्मक्षेत्रे कुरुक्षेत्रे समवेता'), dict(kind='verse', n='२', text='दृष्ट्वा तु')])]
test_eq([b['n'] for b in borrow_numbers(_mine, _theirs)[0]['blocks']], [1, 2])

_it = [dict(page=0, items=[dict(kind='verse', n=1, text='a'), dict(kind='verse', text='b'), dict(kind='head', text='द्वितीयोऽध्यायः'), dict(kind='verse', text='c'), dict(kind='verse', n=2, text='d')])]
test_eq([b.get('n') for b in number_runs(_it)[0]['items']], [1, None, None, 1, 2])   # `items` read as blocks, a `head` as a heading
test_eq([b['n'] for b in borrow_numbers([dict(page=0, items=[dict(kind='verse', text='धर्मक्षेत्रे कुरुक्षेत्रे')])], _theirs)[0]['items']], [1])
test_eq(len(apply_fixes([dict(page=0, items=[dict(kind='verse', text='x')])], [(0, 0, 'drop', None, None, '')])[0]['items']), 0)

Chapters by plan

Some books start a chapter in a way assemble cannot see: with a heading the model did not mark as one, or with no mark at all because a leaf is missing. cut_by_plan takes a plan, a list of (page, item, why) rows saying where each such chapter begins. item is either a block’s position on the page or the first words of a block’s text, usually a heading, and why is optional. Before that block it puts a block of kind start, and assemble begins a new chapter at a start block. Each cut is recorded in probs, and so is a row whose page or block cannot be found.

source

cut_by_plan

def cut_by_plan(
    pages, plan, probs:list=None
)->list:

Insert a chapter-opening start block at each (page, item, why) row of plan, where item is a block’s position or the first words of its text.

from lekhaka.verses import assemble
_pl = [dict(page=0, blocks=[dict(kind='heading', text='கடவுள் வணக்கம்'), dict(kind='verse', n=1, text='x'), dict(kind='heading', text='வாரநயம்'), dict(kind='verse', n=2, text='y')])]
_p = []; _c, _ = assemble(cut_by_plan(_pl, [(0, 'வாரநயம்')], _p)); test_eq(_p, ['0: a chapter starts at `வாரநயம்`, by the plan'])
test_eq([(c['heading'], c['colophon'], [v['n'] for v in c['verses']]) for c in _c], [('கடவுள் வணக்கம்', None, [1]), ('வாரநயம்', None, [2])])
_p = []; cut_by_plan(_pl, [(5, 0), (0, 'அது')], _p); test_eq(len(_p), 2)
_p = []; cut_by_plan([dict(page='p2', items=[dict(kind='verse', n='२', text='x')])], [('p2', 0, 'the leaf before it lost')], _p)
test_eq(_p, ['p2: a chapter starts at verse 2, by the plan (the leaf before it lost)'])

Pages that are not the text

Printed books often carry pages whose verses are not part of the work, such as a preface, an index, or an appendix quoting another work. The Phaladīpikā edition, for example, prints verses of the Jātakābharaṇa after verse 26.48. drop_pages takes a list of (first, last) page ranges, both ends included, and removes the verse blocks on those pages. Headings and other blocks stay.

source

drop_pages

def drop_pages(
    pages, ranges, probs:list=None
)->list:

Remove the verse blocks on every page in each (first, last) range, for pages that are not part of the text.

_p = []
_d = drop_pages([dict(page=k, items=[dict(kind='heading', text='h'), dict(kind='verse', n=k, text='v')]) for k in range(1, 5)], [(2, 3)], _p)
test_eq([len(p['items']) for p in _d], [2, 1, 1, 2]); test_eq(_p, ['2: verses dropped, not the text', '3: verses dropped, not the text'])

After assembly

Once assemble has built the chapters, it is easier to name a verse by its chapter and number than by its page and position. fix_chapters takes fix rows (chapter, verse, kind, read, printed, why), where verse is the verse’s number. chapter is the chapter’s position counting from 1, unless you pass key, a function key(i, chapter) that names chapters some other way, such as the edition’s own chapter numbers or a (part, no) pair. A chapter of None applies the row to every chapter that has that verse. The rows are applied in order, so after a start row has split a chapter, the rows that follow count the chapters as split.

The kinds are:

fix and fix_all replace read with printed in the verse, once or every time, as in apply_fixes.
slip and doubt are recorded and the text is kept. A doubt is never applied here, since fix_chapters takes no witness.
note records why as a remark on the verse and changes nothing.
number sets the verse’s number to printed, and drop removes the verse.
missing adds a verse the reading lacks, perhaps copied by hand from another copy of the book. printed is its text, or a dict such as {text, page}, and it goes in at its place by number. If the chapter already has a verse with that number, nothing is added.
recut moves the boundary between a verse and the one before it, for a case mend cannot see, such as a four-line verse whose last line has no daṇḍa, so the model ran it into the next verse. The two verses are joined and split again so that the earlier one keeps the first printed lines.
start begins a new chapter at the verse. This is for a lost leaf that took the end of one chapter and the start of the next, so the numbering does not restart where assemble would notice. The verse cannot be the first of its chapter. If a chapter holds two verses with that number, printed says which one: the page it is on, or 1 for the first and 2 for the second. The first part keeps the chapter’s heading and the second keeps its colophon.

Any other kind raises a ValueError.

sutras is for works written in sūtras, short prose aphorisms, rather than verse. It puts each sūtra on one line, joining wrapped lines and removing a hyphen at a line break. Where a sūtra does not end in close (॥ by default), it removes a final । or | and adds close, recording that in probs.

source

sutras

def sutras(
    chs, probs:list=None, close:str='॥', key:NoneType=None
)->list:

Put each sūtra on one line, rejoining words hyphenated at a line break, and end it with close where the reading lost it.

source

fix_chapters

def fix_chapters(
    chs, rows, probs:list=None, key:NoneType=None
)->list:

Apply a table of fix rows (chapter, verse, kind, read, printed, why) to assembled chapters; a chapter of None means every chapter with that verse.

V = lambda n, t, pg=1: dict(n=n, text=t, page=pg, heading=None)
_ch = lambda: [dict(heading='h1', colophon='c1', verses=[V(1, 'रामः गच्छति ॥१॥'), V(2, 'सीता ॥२॥'), V(4, 'क\nख\nग ॥४॥'), V(5, 'घ ॥५॥')]), dict(heading='h2', colophon='c2', verses=[V(1, 'रामः ॥१॥')])]
_p = []
_c = fix_chapters(_ch(), [(1, 1, 'fix', 'गच्छति', 'गच्छत्', 'typed'), (None, 1, 'fix', 'रामः', 'राम', ''), (1, 3, 'missing', None, 'लक्ष्मणः ॥३॥', 'from the 1905 print'),
                          (1, 5, 'recut', None, 2, ''), (1, 2, 'slip', 'सीता', None, 'so printed'), (2, 9, 'fix', 'x', 'y', ''), (1, 1, 'note', None, None, 'a lacuna'),
                          (1, 4, 'start', None, None, 'a leaf lost'), (2, 5, 'drop', None, None, 'a quotation')], _p)
test_eq([(c['heading'], c['colophon'], [(v['n'], v['text']) for v in c['verses']]) for c in _c],
        [('h1', None, [(1, 'राम गच्छत् ॥१॥'), (2, 'सीता ॥२॥'), (3, 'लक्ष्मणः ॥३॥')]), (None, 'c1', [(4, 'क\nख')]), ('h2', 'c2', [(1, 'राम ॥१॥')])])
test_eq(_p[:3], ['1.1 (1): read `गच्छति`, printed `गच्छत्` (typed)', '1.1 (1): read `रामः`, printed `राम`', '2.1 (1): read `रामः`, printed `राम`'])
test_eq(len(_p), 10); assert '2.9: verse 9 is gone from the reading; fix x → y not applied' in _p
test_eq([v['n'] for v in fix_chapters(_ch(), [((1, 'x'), 2, 'drop', None, None, '')], key=lambda i, c: (i, 'x'))[0]['verses']], [1, 4, 5])
test_fail(lambda: fix_chapters(_ch(), [(1, 1, 'mend', 'a', 'b', '')]))
_p = []; _c = fix_chapters([dict(heading=None, colophon=None, verses=[V(1, 'क क ॥१॥')])], [(1, 1, 'fix_all', 'क', 'ग', 'twice')], _p)
test_eq((_c[0]['verses'][0]['text'], _p), ('ग ग ॥१॥', ['1.1 (1): read `क`, printed `ग`, 2 times (twice)']))
_two = lambda: [dict(heading='h7', colophon='c8', verses=[V(1, 'a'), V(2, 'b'), V(25, 'y', 'p0107a'), V(2, 'z', 'p0108a'), V(3, 'w', 'p0108a')])]
_p = []; _c = fix_chapters(_two(), [(None, 2, 'start', None, 'p0108a', 'the leaf before it lost')], _p)   # two verses 2 in one chapter: the page picks
test_eq([[v['n'] for v in c['verses']] for c in _c], [[1, 2, 25], [2, 3]]); test_eq(_p, ['p0108a: a chapter starts at verse 2 (the leaf before it lost)'])
test_eq([[v['n'] for v in c['verses']] for c in fix_chapters(_two(), [(1, 2, 'start', None, 2, '')])], [[1, 2, 25], [2, 3]])   # or which one, from 1
test_eq([[v['n'] for v in c['verses']] for c in fix_chapters(_two(), [(1, 2, 'start', None, None, '')])], [[1], [2, 25, 2, 3]])   # else the first
_p = []; fix_chapters(_two(), [(1, 2, 'start', None, 'p0999a', '')], _p); test_eq(_p, ['1.2: no verse 2 after the first of a chapter (p0999a); the start not applied'])
_p = []
test_eq(sutras([dict(verses=[V(1, 'अथातो धर्म-\nजिज्ञासा ।'), V(2, 'चोदनालक्षणोऽर्थो धर्मः ॥')])], _p)[0]['verses'][0]['text'], 'अथातो धर्मजिज्ञासा ॥'); test_eq(_p, ['1.1 (1): closing ॥ supplied'])