lekhaka

Turn the pages of a printed book of verses into numbered chapters.

lekhaka (लेखक, “scribe”) takes a printed book of verses, such as a Sanskrit or Tamil text, and gives back its chapters with every verse numbered.

A vision model reads each page through vishalakshi and returns it as a list of blocks. A block is one piece of the page: a verse with its printed number, a chapter heading, a commentary, or a colophon, the line that closes a chapter (इति … अध्यायः). Models make mistakes on old prints. They miss a verse number, read two verses as one block, or lose half a verse at a page break. They also write Devanagari no printer would set. lekhaka finds and fixes these slips, puts the verses into chapters, and lists every change it made so that a person can check it against the scan.

Install

pip install lekhaka

How to use

from lekhaka import read_book, book_md
from vishalakshi import Vault

b = read_book('gita.pdf', Vault('books.db'), model='claude/sonnet', title='Bhagavad Gita',
              hint='Each verse ends in ॥ n ॥.', counts=[47, 72])
b.chapters[0]['verses'][:2]     # [{n, text, page, heading}, ...]
b.probs                         # what was changed or needs checking, page by page
print(book_md(b.chapters, 'Bhagavad Gita'))

counts is the number of verses you expect in each chapter, usually from the book’s table of contents. lekhaka reports any chapter that comes out with a different number.

If you pass a vault, each page is stored as a version of one document. You can then have other models read the pages again with vault.proofread(b.doc_id, readers), and apply the corrections you agree with using vault.accept. Without a vault, nothing is stored.

Each function also works on its own. The input is a list of pages, each {page, blocks}, which is what vishalakshi.read_doc returns. The rest of this page runs every public function on real pages.

The samples

nbs/samples holds pages from three books that vedicreader has read, saved exactly as the model returned them.

name	book	pages
bj	Varāhamihira’s Bṛhajjātaka, chapters 3 and 4	16
vm	the Tamil Vidhānamālai, with its commentary (urai)	6
uro	Uromarishi’s Pañcapakṣi, printed in 1917	4

import json
from copy import deepcopy

bj, vm, uro = [json.load(open(f'samples/{k}.json')) for k in ('brihat_jataka', 'vidhanamalai', 'uromarishi')]
len(bj), len(vm), len(uro)

(16, 6, 4)

verses

Models do not label blocks consistently. One calls a heading head and another heading; vedicreader stores a page’s blocks under items and vishalakshi under blocks. kind and blocks accept both.

vnum reads a printed verse number in any digits, Devanagari, Tamil or ASCII. The first verse on this Pañcapakṣi page is numbered 205 in Tamil numerals. Some books print two numbers on a verse, such as २६॥२७; vnum takes the last unless you ask for the first.

kind(dict(kind='head')), len(blocks(uro[0])), vnum(uro[0]['blocks'][1]['n']), vnum('२६॥२७'), vnum('२६॥ १', pick='first')

('heading', 8, 205, 27, 26)

assemble puts the verses into chapters. A colophon closes the current chapter, and a chapter also ends when the numbering drops back to 1. A heading names the chapter it falls in, if that chapter has no name yet; it does not start a chapter by itself. A block with no number is taken to be the start of the next numbered verse, because a verse that begins at the foot of one page often gets its number only on the next. The sixteen Bṛhajjātaka pages give two chapters, of 8 and 22 verses, with nothing to report:

chs, probs = assemble(bj)
[(c['heading'], len(c['verses']), c['colophon'][:30]) for c in chs], probs

([('वियोनिजन्माध्यायः', 8, 'इति श्रीवराहमिहिरकृते बृहज्जात'),
  ('निषेकाध्यायः', 22, 'इति श्रीवराहमिहिरकृते बृहज्जात')],
 [])

Some blocks without a number are whole verses, not pieces of one. A book may open with a maṅgala, a verse of blessing printed before verse 1, and a commentary may quote a verse from another work. Such a block ends in ॥ like any verse but carries no number, and adding it to the next verse is wrong.

Page p0025a of the Bṛhajjātaka has one. Between verses 3 and 4 the commentary quotes a verse in the āryā metre. By default assemble adds it to verse 4, which ends up with four lines instead of two. With openers=True, a block that ends in ॥ and has no number stays a separate verse, with n set to None:

op, _ = assemble(bj, openers=True)
[(v['n'], v['page'], v['text'][:25]) for v in op[1]['verses'][2:5]], len(chs[1]['verses'][3]['text'].split('\n'))

([(3, 'p0025a', 'रवीन्दुशुक्रावनिजैः स्वभा'),
  (None, 'p0025a', 'बलयुतौ स्वगृहांशेष्वर्कसि'),
  (4, 'p0025a', 'दिवाकरेन्द्वोः स्मरगौ कुज')],
 4)

Many books print a commentary under each verse. notes=('urai',) keeps blocks of that kind with the verse they explain, in verse['notes'], instead of counting them as verse text. Some books number the commentary and leave the verse unnumbered, and the Vidhānamālai is one of them. number_from_note=True gives each verse the number of its commentary.

join_cont joins a block that a page break cut in two. The model marks the second half with cont, and a word hyphenated across the break is put back together. assemble(cont=True) does this before anything else.

vc, _ = assemble(vm, notes=('urai',), number_from_note=True)
v = vc[0]['verses'][0]
len(vc[0]['verses']), v['n'], v['notes']['urai'][0][:40]

(11, 1, '(என்றது.) ஸ்ரீமகாலக்ஷ்மிக்கு நாயகரும், ப')

join_cont([dict(page=1, blocks=[dict(kind='prose', text='a com-')]), dict(page=2, blocks=[dict(kind='prose', text='mentary', cont=True)])])[0]['blocks'][0]['text']

'a commentary'

mend fixes two common slips in a chapter’s numbering. When the model reads the same number twice, as in 7, 9, 9, 10, it takes the first 9 as 8. When the model reads two verses as one block, it splits them. renumber fixes a run of numbers that are wrong the same way: in 52, 63, 64, 65, 56 the model misread the tens digit three times. check_counts compares each chapter’s number of verses with the count you expect:

V = lambda n: dict(n=n, text='a ।\nb ॥', page='p1')
p = []; [v['n'] for v in mend([V(7), V(9), V(9), V(10)], p)], p

([7, 8, 9, 10], ['p1: first of two 9s taken as 8'])

[v['n'] for v in renumber([V(n) for n in (52, 63, 64, 65, 56)], 'ch 3', [])], check_counts(chs, [8, 21])

([52, 53, 54, 55, 56], ['chapter 2: 22 verses, the plan has 21'])

Some books print no chapter headings, and the only marker is the colophon at the end of each chapter, such as इति श्रीवराहमिहिरकृते बृहज्जातके निषेकाध्यायश्चतुर्थः. assemble_colophon starts a new chapter after each colophon and reads the chapter’s part, number and name from it. It ignores colophons that belong to a commentary.

cc, _ = assemble_colophon(bj)
[(c['part'], c['no'], c['name'], len(c['verses'])) for c in cc]

[('पूर्व', 3, 'वियोनिजन्म', 8), ('पूर्व', 4, 'निषेक', 22)]

Models sometimes write akṣaras that cannot occur, such as a virāma followed by a vowel sign (ध्ा). bad_aksharas names each fault and fix_aksharas repairs it. Three smaller helpers deal with typing slips: deva writes a number in Devanagari digits, norm replaces a colon used for a visarga and ASCII digits inside ॥ ॥, and virama puts back a virāma dropped at the end of a pāda.

bad_aksharas('ध्ार्म'), fix_aksharas('ध्ार्म'), deva(18), norm('राम: ॥ 1 ॥'), virama('जगत ।')

(['virāma before a vowel sign'], 'धार्म', '१८', 'रामः॥ १॥', 'जगत् ।')

book

read_book does the whole job for a PDF. It reads the pages through vishalakshi and passes them to assemble_book, which applies your fix rows, gives a verse read without its number the number it ends on (॥ १२ ॥), assembles the chapters, repairs akṣaras, and compares the chapters with counts. read_book takes renumber=True to renumber each chapter. Joining blocks split by a page break (cont=True) is an option of assemble_book only, for pages you already have. book_md writes chapters as markdown:

print(book_md(chs[:1], 'बृहज्जातकम्')[:160])

# बृहज्जातकम्

## वियोनिजन्माध्यायः

क्रूरग्रहैस्तु बलिभिर्विंबलैश्च सौम्यैः क्लीबे चतुष्टयगतेतद्वेक्षणाद्वा।
चन्द्रोपगद्विरसभागसमानरूपं सत्त्वं वदेद्यदि भवेत्स

repairs

Some mistakes need a person. You look at the scan, decide what the page really says, and write that down as a row. lekhaka applies the rows and records each change in probs, so the list of problems also says what was changed and why.

apply_fixes works on pages, before assemble. A row is (page, block, kind, read, printed, why): the text the model read, the text the page prints, and your reason. A fix row replaces the first match and a fix_all row replaces every match.

apply_fixes(deepcopy(bj), [('p0022b', 2, 'fix', 'शेभनाः', 'शोभनाः', 'a typo')], probs=(p := [])); p

['p0022b: read `शेभनाः`, printed `शोभनाः` (a typo)']

number_runs numbers verses printed without numbers when they sit between two numbered verses, since the gap says what the missing numbers must be. borrow_numbers takes a missing number from a second reading of the same page, when both readings have the same verse.

number_runs([dict(page='p1', blocks=[dict(kind='verse', n=1, text='a'), dict(kind='verse', text='b'), dict(kind='verse', n=3, text='c')])], p := [])
mine = deepcopy(bj[:1]); mine[0]['blocks'][2]['n'] = None
borrow_numbers(mine, bj, p); p

['p1: verse printed without its number taken as 2, between 1 and 3',
 "p0022b: verse 2's number, lost in this reading, taken from the other (1.00)"]

cut_by_plan starts a chapter where you say one starts, given as (page, where, why), where where is a block’s position on the page or the first words of a heading. Use it when a book prints neither a heading nor a colophon that assemble can find. drop_pages removes the verses on pages that are not part of the text, such as a preface or an index.

cut_by_plan(deepcopy(vm), [('p0020a', 'பதிகம்.', 'the contents open a chapter')], p := [])
drop_pages(deepcopy(bj), [('p0029b', 'p0030a')], p); p

['p0020a: a chapter starts at `பதிகம்.`, by the plan (the contents open a chapter)',
 'p0029b: verses dropped, not the text',
 'p0030a: verses dropped, not the text']

fix_chapters works on chapters, after assemble, and finds a verse by its chapter and number instead of its page and position. Its rows are (chapter, verse, kind, read, printed, why). Besides fix and fix_all, a note row records a remark without changing the text, and a start row starts a new chapter at a verse.

If a chapter holds two verses with the same number, a start row’s printed says which one is meant: the page it is on, or 1 for the first and 2 for the second.

rows = [(1, 2, 'fix', 'शेभनाः', 'शोभनाः', 'a typo'), (2, 19, 'fix_all', '।।', '॥', 'doubled daṇḍas'), (2, 15, 'note', None, None, 'its first two lines are a quotation')]
fix_chapters(deepcopy(chs), rows, p := []); p

['1.2 (p0022b): read `शेभनाः`, printed `शोभनाः` (a typo)',
 '2.19 (p0029a): read `।।`, printed `॥`, 2 times (doubled daṇḍas)',
 '2.15: its first two lines are a quotation']

one = [dict(heading=None, colophon=None, verses=chs[0]['verses'] + chs[1]['verses'])]
p = []; [len(c['verses']) for c in fix_chapters(one, [(1, 1, 'start', None, 'p0024b', 'by hand')], p)], p

([8, 22], ['p0024b: a chapter starts at verse 1.1 (by hand)'])

Works in sūtra form need different handling. sutras joins each sūtra onto one line and closes it with ॥.

sutras([dict(verses=[dict(n=1, text='अथातो धर्म-\nजिज्ञासा ।', page='p1')])], p := [])[0]['verses'][0]['text'], p

('अथातो धर्मजिज्ञासा ॥', ['1.1 (p1): closing ॥ supplied'])

editions

vedicreader publishes a work as a set of titles. cut divides chapters into titles: one per chapter, one every per verses, or the whole work as one. A verse without a number, such as the quoted āryā above, gets a letter (क, ख and so on) in place of a number, with the word open_word (मङ्गलम्) as its label.

numbering summarises a run of verses: how many are numbered, which numbers are skipped or repeated, and the first and last.

[t['title'] for t in cut(cc, 'chapters', book='बृहज्जातकम्')], [(v['n'], v.get('word')) for v in cut(op, 'chapters')[1]['verses'][2:5]]

(['बृहज्जातकम् १ वियोनिजन्म', 'बृहज्जातकम् २ निषेक'],
 [(3, None), ('क', 'मङ्गलम्'), (4, None)])

numbering(op[1]['verses'])

{'numbered': 22,
 'unnumbered': 1,
 'skipped': [],
 'repeated': [],
 'first': 1,
 'last': 22}

The next functions compare a reading with another edition of the same text, called a witness. witness_verses takes the raw HTML of a witness page, from GRETIL for example, and returns its numbered verses. letters reduces a verse to its letters, dropping numbers, punctuation and bracketed notes, so that two editions can be compared on the words alone. text_changes proposes a change wherever the closest verse in the witness spells a word differently, and also flags dropped virāmas.

witness_verses('<p>क ख ॥ १ ॥</p><p>ग घ ॥ २ ॥</p>'), letters('रामः ॥ १ ॥ (टि)')

([(1, 'क ख'), (2, 'ग घ')], 'रामः')

wit = [(v['n'], v['text'].replace('शेभनाः', 'शोभनाः')) for v in chs[0]['verses']]
[(c['id'], c['read'], c['print'], c['why']) for c in text_changes(chs[:1], wit)]

[('1.2.w4', 'शेभनाः', 'शोभनाः', 'the witness (2) reads so')]

ref_check flags verses that read very differently from a reference edition, which usually means a misread or a lost line. move_lines fixes a common slip, where the last line of one verse was read as the first line of the next, by moving the line back.

ref = {1: [letters(v['text']) for v in chs[0]['verses']]}; ref[1][4] = ''
ref_check(chs[:1], ref)

['1.5 (p0023a): reads unlike the reference (0.29), check against the scan']

mv = deepcopy(chs[:1]); a, b = mv[0]['verses'][:2]; ls = a['text'].split('\n')
a['text'], b['text'] = ls[0], ls[1] + '\n' + b['text']
move_lines(mv, {(1, v['n']): letters(v['text']) for v in chs[0]['verses']}, p := []); p, mv[0]['verses'][0]['text'] == chs[0]['verses'][0]['text']

(['1.2: 1 leading line(s) moved back to 1.1'], True)

The last group measures how far apart two readings of the same pages are, which is useful when you try a second model. verse_text collects a page’s verse text, lev is the edit distance between two strings, and disagree is the share of a page’s text that differs. compare puts the result for each page in a table and summarises it. Here the second reading lost the first verse on p0023a, and that page stands out:

b2 = deepcopy(bj); b2[1]['blocks'] = b2[1]['blocks'][1:]   # a second reading that lost p0023a's first verse
verse_text(bj[1])[:12], lev('क्रियः', 'क्रिया'), round(disagree(bj[1], b2[1]), 2)

('क्रियःशिरोवक', 1, 0.34)

md, flag = compare({'opus': bj, 'other': b2}, [('opus', 'other')])
print(md)

| page | opus vs other |
|---|---|
| p0023a | 34.1% |

opus vs other: 16 pages, mean 2.1%, 1 at 10% or more