blocks
def blocks(
p
)->list:A page’s list of blocks, whether the page keeps them under blocks or under items.
verses
Read printed verse numbers, put verses into chapters, mend their numbering, and repair malformed Devanagari
A vision model reads each page of a book and returns a list of blocks. A block is one piece of the page as the model saw it: a verse, a chapter heading, a commentary, a colophon (the line that closes a chapter, इति … अध्यायः), or a stretch of prose. Each block has a kind that says which of these it is, the printed number n if it has one, and its text. A page is a dict {page, blocks}, which is what vishalakshi.read_doc returns.
Readers do not all use the same names. vedicreader’s saved readings keep a page’s blocks under items instead of blocks, and some models call a heading head. blocks returns a page’s blocks under either key, and kind returns a block’s kind with head read as heading, through the KINDS table.
The functions in this module put verses into chapters. A verse comes out as {n, text, page, heading}: its number, its text, the page it was printed on, and the heading printed just before it, if any. Whenever a function changes a number or a text, it adds a line to probs, a list you pass in, so that a proofreader can check each change against the scan.
kind
def kind(
b
)->str | None:A block’s kind, with an alias from KINDS, such as head, read as the kind it stands for.
test_eq(blocks(dict(page=1, items=[1])), [1]); test_eq(blocks(dict(page=1, blocks=[2], items=[1])), [2]); test_eq(blocks(dict(page=1)), [])
test_eq(kind(dict(kind='head')), 'heading'); test_eq(kind(dict(kind='verse')), 'verse'); test_eq(kind({}), None)Verse numbers
A verse number can be printed in ASCII, Devanagari or Tamil digits, and is often wrapped in daṇḍas or brackets, as in ॥ २६ ॥ or (௪௦). vnum finds the digits and returns the number as an int, or None when there are no digits. Older Tamil prints write numbers with the signs ௰ (ten), ௱ (hundred) and ௲ (thousand), and vnum reads those too: ௬௱௨ is 602.
Sometimes a block carries two numbers, as in २६॥२७. This usually means the model read two verses as one block, so vnum returns the last number, 27. If you pass probs, it also records that verses 26 and 27 were printed as one. Some books print a running number followed by the verse’s number within its section, as in २६॥ १. For those, pick='first' returns the running number. The tests below show each case.
vnum
def vnum(
n, page:NoneType=None, probs:list=None, pick:str='last'
)->int | None:A printed verse number as an int, or None when it has no digits; when several are printed together, the last one, or the first with pick='first'.
_p = []
test_eq(vnum('॥ २६ ॥'), 26); test_eq(vnum('(௪௦)'), 40); test_eq(vnum(12), 12); test_eq(vnum(None), None)
test_eq(vnum('२६॥२७', 'p5', _p), 27); test_eq(_p, ['p5: verses 26-27 printed as one, numbered 27'])
test_eq(vnum('௬௱௨'), 602); test_eq(vnum('௨௱௩'), 203); test_eq(vnum('௰'), 10)
test_eq(vnum('२६॥ १', pick='first'), 26) # a running number, then the count within its partMending a run
Within a chapter the numbers should run 1, 2, 3 without gaps. Two common mistakes break that run in a way the neighbouring numbers can repair, and mend repairs both.
The first is a number read twice. If the model reads 148, 150, 150, 151, the verse after 148 has to be 149, so mend takes the first 150 as 149. The second is two verses read as one block. If the model reads verse 148 and then a long block numbered 150, verse 149 is missing and is probably the first half of that block. mend splits the block after its last inner line that ends with a daṇḍa (।, ॥ or |) or a full stop, keeping at least three lines in the first part, and numbers that part 149. Both repairs happen only when the verse before is numbered exactly two less.
That rule suits verses printed on four lines. In a book of two-line verses, both lines of a verse end in a daṇḍa, so the last such line is the wrong place to cut. pair=True lets the first part be two lines long and cuts where the two parts come out closest in length.
renumber repairs the mistakes found in long chapters. One is a run read at a constant offset: in 52, 63, 64, 65, 56 the model misread the tens digit of three verses, and the sequence returns to normal at 56. Another is a single number that disagrees with both neighbours, such as 9 between 2 and 4, or a number that lost its leading digit, such as 1 read for 101 between 100 and 102. renumber only changes numbered verses. A verse with no number stays where it is.
renumber
def renumber(
vs, where, probs:list
)->list:Correct misread verse numbers in a chapter from the numbers around them, leaving unnumbered verses where they are.
mend
def mend(
vs, probs:list, pair:bool=False
)->list:Correct a verse number read twice and split two verses read as one block, where the numbers around them show what is right.
V = lambda n, t='a ।\nb ॥', pg=1: dict(n=n, text=t, page=pg, heading=None)
_p = []
test_eq([v['n'] for v in mend([V(148), V(150), V(150), V(151)], _p)], [148, 149, 150, 151]); test_eq(len(_p), 1)
_p = []
_m = mend([V(1), V(3, 'a\nb\nc ॥\nd\ne ॥')], _p)
test_eq([(v['n'], v['text']) for v in _m], [(1, 'a ।\nb ॥'), (2, 'a\nb\nc ॥'), (3, 'd\ne ॥')])
_p = []
test_eq([v['n'] for v in renumber([V(n) for n in (52, 63, 64, 65, 56, 57)], 'ch 3', _p)], [52, 53, 54, 55, 56, 57]); test_eq(len(_p), 3)
test_eq([v['n'] for v in renumber([V(n) for n in (99, 100, 1, 102)], 'ch 9', [])], [99, 100, 101, 102])
test_eq([v['n'] for v in renumber([V(None), V(99), V(100), V(1), V(102)], 'ch 9', [])], [None, 99, 100, 101, 102]) # an unnumbered opener stays put
test_eq([v['n'] for v in mend([V(None), V(1), V(3, 'a\nb\nc ॥\nd\ne ॥')], [])], [None, 1, 2, 3])Chapters
assemble reads the pages in order and builds chapters, each a dict {heading, colophon, verses}. It returns the chapters and a list of problems.
The kinds argument says which blocks count as verse. By default these are blocks of kind verse and blocks with no kind. With kinds=None, every block except headings and colophons counts as verse, which suits a reading in which the model gave verse text kind names of its own.
A heading becomes the name of the open chapter if that chapter has no name yet, or opens a new chapter if none is open. Each verse also keeps the heading printed just before it in its own heading field. A verse numbered 1 after a higher number starts a new chapter. A colophon ends the open chapter and is stored as its colophon.
A verse that starts at the foot of one page often has its number printed only at its end, on the next page, so the model returns the first part as a block with no number. assemble holds every unnumbered block and adds its text to the next numbered verse. If a colophon comes while text is still held, that text becomes the chapter’s last verse, numbered one more than the verse before it, and this is recorded in the problems.
A book photographed as two-page spreads may show the same page twice, and then the model reads the same verse twice. A numbered verse whose number and last line match a verse already in the chapter is dropped, and an unnumbered block that repeats the one held before it is kept once. As verses are added, any break in the numbering, such as verse 5 after verse 3, is listed in the problems. When every page is in, mend runs on each chapter.
Many books print a commentary after each verse; in Tamil books this is the urai. The model gives such a block a kind of its own, such as urai, gloss or bhashya. Name those kinds in notes, and assemble keeps each such block with the verse it follows, in verse['notes'][kind], as a list of texts. A commentary that arrives while verse text is still held, with no number yet, is kept for the next verse that is added. With the default kinds and no notes, these blocks are skipped. Some books, the Vidhānamālai among them, print the number at the end of the commentary and leave the verse unnumbered. With number_from_note=True, a numbered commentary block gives its number to the verse text held before it, and this is recorded in the problems.
A page break can also cut a block in two, so that its second half opens the next page. Some readings mark that second half with cont. Only when you pass cont=True does assemble first join each such half to the block it continues, using join_cont on a copy of the pages.
Not every unnumbered block belongs to the next verse. Many books open with a maṅgala, a verse of blessing printed before verse 1 without a number, and commentaries quote whole verses from other works. Such a block ends in ॥ like a finished verse. By default assemble treats it like any other unnumbered block and adds its text to the next numbered verse, which then holds two verses. With openers=True, an unnumbered block that ends in ॥ is kept as a verse of its own, with n set to None. It goes into the chapter of the numbered verse that follows it, just before that verse; if a colophon comes first, it goes at the end of the chapter the colophon closes. mend and renumber leave such verses where they are.
join_cont is the joining step on its own. When the first block on a page is marked cont and has the same kind as the last block on the page before, its text is added to that block: after a line break for a verse, and after a space for anything else. If the earlier text ends in a hyphen, the hyphen is removed and the two parts are joined with no space, so a word split across the page break is whole again. If the second half carries a number, the joined block takes it. join_cont changes the pages it is given and returns them.
The tests below build small pages for each of these cases: a chapter closed by a colophon, a verse read twice, commentary kept as notes, a reading with items and cont, and maṅgala verses kept with openers.
join_cont
def join_cont(
pages
)->list:Join each block marked cont at the top of a page to the block it continues on the page before, rejoining a word hyphenated at the break.
_jc = [dict(page=1, items=[dict(kind='prose', text='a long com-'), dict(kind='verse', text='क ।')]),
dict(page=2, items=[dict(kind='verse', text='ख ॥२॥', n=2, cont=True), dict(kind='prose', text='next')]),
dict(page=3, items=[dict(kind='prose', text='one', cont=True)])]
test_eq([(b['kind'], b['text'], b.get('n')) for p in join_cont(_jc) for b in p['items']], [('prose', 'a long com-', None), ('verse', 'क ।\nख ॥२॥', 2), ('prose', 'next one', None)])
test_eq(join_cont([dict(page=1, blocks=[dict(kind='prose', text='com-')]), dict(page=2, blocks=[dict(kind='prose', text='mentary', cont=True)])])[0]['blocks'][0]['text'], 'commentary')check_counts
def check_counts(
chs, counts
)->list:List the chapters whose number of verses differs from the expected count, and a difference in the number of chapters.
assemble
def assemble(
pages, pair:bool=False, kinds:tuple=('verse', None), notes:tuple=(), number_from_note:bool=False,
cont:bool=False, openers:bool=False
)->tuple:Build chapters [{heading, colophon, verses}] from pages of blocks, and return them with the problems found. kinds names the block kinds that count as verse, and None means every kind except headings and colophons. A block whose kind is in notes is kept in the notes of its verse, and number_from_note=True numbers a verse from its commentary. cont=True first joins blocks cut by a page break, and openers=True keeps an unnumbered block ending in ॥ as a verse of its own with n None.
B = lambda kind, text, n=None: dict(kind=kind, text=text, n=n)
_pages = [dict(page=0, blocks=[B('heading', 'अथ प्रथमोऽध्यायः'), B('verse', 'क ।\nख ॥१॥', '१'), B('verse', 'ग ।', None)]),
dict(page=1, blocks=[B('verse', 'घ ॥२॥', '॥२॥'), B('verse', 'ङ ॥३॥', 3), B('colophon', 'इति प्रथमोऽध्यायः')]),
dict(page=2, blocks=[B('verse', 'च ॥१॥', 1), B('verse', 'छ ॥२॥', 2), B('verse', 'छ ॥२॥', 2), B('prose', 'an English note')])]
_chs, _pr = assemble(_pages)
test_eq(len(_chs), 2); test_eq(_chs[0]['heading'], 'अथ प्रथमोऽध्यायः'); test_eq(_chs[0]['colophon'], 'इति प्रथमोऽध्यायः')
test_eq([(v['n'], v['text']) for v in _chs[0]['verses']], [(1, 'क ।\nख ॥१॥'), (2, 'ग ।\nघ ॥२॥'), (3, 'ङ ॥३॥')])
test_eq([v['n'] for v in _chs[1]['verses']], [1, 2]); test_eq(_pr, [])
test_eq(check_counts(_chs, [3, 2]), []); test_eq(check_counts(_chs, [3, 3]), ['chapter 2: 2 verses, the plan has 3'])_u = [dict(page=0, blocks=[B('verse', 'பா', 1), B('urai', 'உரை', 1)])]
test_eq(len(assemble(_u)[0][0]['verses']), 1); test_eq([v['text'] for v in assemble(_u, kinds=None)[0][0]['verses']], ['பா', 'உரை']) # every kind but headings and colophons_vm = [dict(page=0, blocks=[B('verse', 'பா ஒன்று', None), B('urai', 'உரை ஒன்று', 1), B('verse', 'பா இரண்டு', None), B('urai', 'உரை இரண்டு', 2)])]
_c, _p = assemble(_vm, notes=('urai',), number_from_note=True)
test_eq([(v['n'], v['text'], v['notes']['urai']) for v in _c[0]['verses']], [(1, 'பா ஒன்று', ['உரை ஒன்று']), (2, 'பா இரண்டு', ['உரை இரண்டு'])])
_g = [dict(page=0, blocks=[B('verse', 'a ॥१॥', 1), B('gloss', 'its gloss'), B('verse', 'b ॥२॥', 2)])]
test_eq(assemble(_g, notes=('gloss',))[0][0]['verses'][0]['notes'], {'gloss': ['its gloss']})
test_eq('notes' in assemble(_g)[0][0]['verses'][0], False) # without `notes` a gloss is not a verse and is not kept_it = [dict(page=0, items=[B('head', 'अथ प्रथमोऽध्यायः'), B('verse', 'क ।\nख ॥१॥', 1), dict(kind='verse', text='ग ।', open=True)]),
dict(page=1, items=[dict(kind='verse', text='घ ॥२॥', n=2, cont=True)])]
_c, _p = assemble(_it, cont=True)
test_eq((_c[0]['heading'], [(v['n'], v['text']) for v in _c[0]['verses']]), ('अथ प्रथमोऽध्यायः', [(1, 'क ।\nख ॥१॥'), (2, 'ग ।\nघ ॥२॥')]))
test_eq(_it[1]['items'][0]['text'], 'घ ॥२॥') # the pages are left as they were_op = [dict(page=0, blocks=[B('heading', 'अथ प्रथमोऽध्यायः'), B('verse', 'वन्दे गुरुम् ॥'), B('verse', 'क ।\nख ॥१॥', 1), B('verse', 'ग ॥२॥', 2)]),
dict(page=1, blocks=[B('verse', 'नमः शिवाय ॥'), B('verse', 'धर्मक्षेत्रे ॥१॥', 1), B('verse', 'समवेता ।'), B('verse', 'मामकाः ॥२॥', 2), B('verse', 'श्रीः ॥'), B('colophon', 'इति')])]
_c, _p = assemble(_op, openers=True)
test_eq([[(v['n'], v['text'], v['heading']) for v in c['verses']] for c in _c],
[[(None, 'वन्दे गुरुम् ॥', 'अथ प्रथमोऽध्यायः'), (1, 'क ।\nख ॥१॥', None), (2, 'ग ॥२॥', None)],
[(None, 'नमः शिवाय ॥', None), (1, 'धर्मक्षेत्रे ॥१॥', None), (2, 'समवेता ।\nमामकाः ॥२॥', None), (None, 'श्रीः ॥', None)]])
test_eq(_p, [])
test_eq([(v['n'], v['text'][:5]) for v in assemble(_op)[0][0]['verses']], [(1, 'वन्दे'), (2, 'ग ॥२॥')]) # by default an opener joins the verse after itChapters cut at their colophons
Some editions print no usable chapter headings and number each chapter only in its colophon. The Khemraj edition of the Bṛhat Parāśara Horā Śāstra is one. It has a pūrva khaṇḍa (first part) of 51 adhyāyas (chapters) and an uttara bhāga (second part) of 20, and a chapter’s number appears only in its closing line, such as इति … राशिभेदनामाध्यायः ॥ ३ ॥. assemble_colophon cuts the pages into chapters at these colophons. Each chapter comes out as {part, no, name, heading, colophon, verses}. By default every block other than a heading or a colophon counts as verse (kinds=None).
A colophon closes a chapter only if it contains adh, which is ध्याय by default, as in अध्यायः. The chapter’s number is the first number after the last adh in the colophon. Failing that, it is the colophon block’s own n, and failing that, one more than the previous chapter in the same part. The chapter’s name is the text after one of the anchor words, such as जातके or होरायां, and before नाम, or before the ordinal word and अध्याय. If the colophon gives no name, the chapter takes one from an earlier heading of the form अथ … अध्यायः.
parts lists the book’s parts in order, each with a pattern. The pages start in the first part, and a heading or colophon that matches a later part’s pattern moves the count into that part, so chapter numbers can start again at 1 in the uttara bhāga.
These editions also print things that are not the text, and there is a pattern for each. A colophon matching contents closes a table of contents printed before the text, so everything collected up to it is thrown away. A heading matching skip starts the commentator’s own verses, such as his maṅgala or his praise of the author, and verses are skipped until a heading matching resume. Commentators also write colophons of their own. A colophon matching tika (टीका or व्याख्या) that repeats the number of the chapter just closed marks the verses between the two colophons as commentary, and they are dropped. An ordinary colophon that repeats the number adds those verses to that chapter instead.
If the verses before a colophon start again at 1 partway through, the chapter before them lost its colophon. The function then splits the verses where the numbering restarts, giving the first part the number before and no colophon or name, unless a chapter with that number already exists. Unnumbered blocks are held and joined to the next numbered verse, and repeated verses are dropped, as in assemble. With pair=True, text still held when a colophon arrives becomes the chapter’s last verse, and mend uses its rule for two-line verses. Finally each chapter goes through renumber and mend, and any break left in its numbering is listed in the problems.
The test below runs a small two-page book through these cases.
assemble_colophon
def assemble_colophon(
pages, pair:bool=False, kinds:NoneType=None,
parts:tuple=(('पूर्व', None), ('उत्तर', 'त्तरभाग|त्तरार्ध|त्तरार्द्ध|उ० भाग|उ० भा')), adh:str='ध्याय',
tika:str='टीका|व्याख्या', skip:str='टीकाकर्तु|टीकाकार|ग्रंथकर्तृप्रशंसा|ग्रन्थकर्तृप्रशंसा',
resume:str='ग्रंथारंभ|ग्रन्थारम्भ|ध्याय', contents:str='नुक्रमणिका',
anchor:str='(?:सारांशे|सारांश|सा०|भागे|उत्तरार्धे|उत्तरार्द्धे|शास्त्रे|होरायाम्|होरायां|चिन्तामणौ|जातके)'
)->tuple:Cut pages into chapters [{part, no, name, heading, colophon, verses}] at the colophons that number them, and return them with the problems found.
_bj = [dict(page=1, items=[B('colophon', 'इति ग्रन्थानुक्रमणिका'), B('head', 'टीकाकर्तुर्मङ्गलम्'), B('verse', 'टीका ॥१॥', 1), B('head', 'अथ ग्रन्थारम्भः'),
B('verse', 'क ॥१॥', 1), B('verse', 'ख ॥२॥', 2), B('colophon', 'इति होरायां संज्ञाध्यायः प्रथमः ॥१॥')]),
dict(page=2, items=[B('verse', 'रामो राजमणिः ॥१॥', 1), B('verse', 'सदा विजयते ॥२॥', 2), B('verse', 'गणेशं नमामि ॥१॥', 1), B('colophon', 'इति शास्त्रे राशिभेदनामाध्यायः॥ ३॥'),
B('head', 'अथोत्तरभागः'), B('verse', 'च ॥१॥', '१'), B('colophon', 'इति उत्तरभागे फलनाम प्रथमोऽध्यायः ॥१॥'),
B('colophon', 'इति टीकायां प्रथमोऽध्यायः ॥१॥')])]
_c, _p = assemble_colophon(_bj)
test_eq([(c['part'], c['no'], c['name'], [v['n'] for v in c['verses']]) for c in _c],
[('पूर्व', 1, 'संज्ञ', [1, 2]), ('पूर्व', 2, None, [1, 2]), ('पूर्व', 3, 'राशिभेद', [1]), ('उत्तर', 1, 'फल', [1])])
assert any('no colophon; split' in x for x in _p)
test_eq(_cname('इति श्रीवराहमिहिरकृते बृहज्जातके वियोनिजन्माध्यायस्तृतीयः॥३॥', _ANCHOR, 'ध्याय'), 'वियोनिजन्म'); test_eq(_cnum('निषेकाध्यायश्चतुर्थः॥ ४॥', 'ध्याय'), 4)Akṣaras
An akṣara is one written syllable: a consonant or a cluster of consonants with its vowel sign, or a vowel letter on its own. Some sequences of Unicode characters can never make a correct akṣara, and they turn up when a model writes Devanagari or when a text was typed or converted from an old font. One example is a virāma (्, the mark that removes a consonant’s vowel) followed by a vowel sign, as in ध्ा. Another is two vowel signs on one consonant.
AKSHARA_BAD names five such faults, each with a pattern, and bad_aksharas lists the names of those found in a text. fix_aksharas repairs the ones whose right form is not in doubt:
- an i-sign (ि) typed after a virāma is moved to the end of its consonant cluster, so श्िच becomes श्चि;
- a virāma before a vowel sign or an anusvāra is removed, and so is a visarga after a virāma;
- ट्ः followed by a vowel letter becomes ठ with that vowel’s sign, so कण्ट्ःए becomes कण्ठे;
- a nukta after a vowel sign is removed;
- े or ै followed by ा becomes ो or ौ;
- a vowel letter followed by a sign that makes another vowel, such as अ with ा, becomes that vowel’s letter, here आ;
- a vowel sign typed twice, and a few pairs such as ु followed by ू, become one sign.
The Taittirīya texts write ग्ं for a nasal, so a virāma before an anusvāra after ग is left as it is.
fix_aksharas
def fix_aksharas(
t:str
)->str:Return the Devanagari text t with the malformed akṣaras whose right form is certain repaired.
bad_aksharas
def bad_aksharas(
t:str
)->list:The names, from AKSHARA_BAD, of the malformed akṣaras found in t.
test_eq(fix_aksharas('श्िच'), 'श्चि'); test_eq(fix_aksharas('भक्ित'), 'भक्ति'); test_eq(fix_aksharas('ध्ार्म'), 'धार्म')
test_eq(fix_aksharas('कण्ट्ःए'), 'कण्ठे'); test_eq(fix_aksharas('अाम'), 'आम'); test_eq(fix_aksharas('ग्ं'), 'ग्ं')
test_eq(bad_aksharas('ध्ार्म'), ['virāma before a vowel sign']); test_eq(bad_aksharas(fix_aksharas('ध्ार्म')), [])Typing slips
Three small helpers fix slips in Devanagari typed by a person or written by a model.
norm turns a colon written after a Devanagari letter into a visarga (ः), writes ASCII digits between ॥ marks in Devanagari digits, and removes spaces before a daṇḍa (। or ॥). So राम: ॥ 1 ॥ becomes रामः॥ १॥.
A pāda is a quarter of a verse, usually printed as a line or half line ending in a daṇḍa. A word at the end of a pāda often ends in a bare consonant marked with a virāma, as in जगत्, and typists drop the virāma. virama puts it back when a line ends with one of a few endings that need it, either at the end of the line or before a ।. The endings are त after ा, े, ृ, व or य (-ात्, -ेत्, -ृत्, -वत्, -यत्), न after ा or ि (-ान्, -िन्), and the whole words जगत and सत. A word before ॥ is not changed.
deva writes a number in Devanagari digits.
virama
def virama(
l:str
)->str:Restore the virāma a typist drops at the end of a pāda, for the endings -ात्, -ेत्, -ृत्, -वत्, -यत्, -ान्, -िन् and the words जगत्, सत्, before । or the end of the line.
norm
def norm(
t:str
)->str:Fix typing slips in t: a colon typed for a visarga, ASCII digits inside ॥ n ॥, and spaces before a daṇḍa.
test_eq(deva(18), '१८'); test_eq(norm('रामः वनं गच्छति: ।\nसीता ॥ 12 ॥'), 'रामः वनं गच्छतिः।\nसीता॥ १२॥')
test_eq(virama('भगवत ।'), 'भगवत् ।'); test_eq(virama('महान'), 'महान्'); test_eq(virama('जगत'), 'जगत्'); test_eq(virama('राम ।'), 'राम ।')