# lekhaka lekhaka (लेखक, “scribe”) takes a printed book of verses, such as a Sanskrit or Tamil text, and gives back its chapters with every verse numbered. A vision model reads each page through [vishalakshi](https://github.com/vedicreader/vishalakshi) and returns it as a list of blocks. A block is one piece of the page: a verse with its printed number, a chapter heading, a commentary, or a colophon, the line that closes a chapter (इति … अध्यायः). Models make mistakes on old prints. They miss a verse number, read two verses as one block, or lose half a verse at a page break. They also write Devanagari no printer would set. lekhaka finds and fixes these slips, puts the verses into chapters, and lists every change it made so that a person can check it against the scan. ## Install ``` sh pip install lekhaka ``` ## How to use ``` python from lekhaka import read_book, book_md from vishalakshi import Vault b = read_book('gita.pdf', Vault('books.db'), model='claude/sonnet', title='Bhagavad Gita', hint='Each verse ends in ॥ n ॥.', counts=[47, 72]) b.chapters[0]['verses'][:2] # [{n, text, page, heading}, ...] b.probs # what was changed or needs checking, page by page print(book_md(b.chapters, 'Bhagavad Gita')) ``` `counts` is the number of verses you expect in each chapter, usually from the book’s table of contents. lekhaka reports any chapter that comes out with a different number. If you pass a vault, each page is stored as a version of one document. You can then have other models read the pages again with `vault.proofread(b.doc_id, readers)`, and apply the corrections you agree with using `vault.accept`. Without a vault, nothing is stored. Each function also works on its own. The input is a list of pages, each `{page, blocks}`, which is what `vishalakshi.read_doc` returns. The rest of this page runs every public function on real pages. ## The samples `nbs/samples` holds pages from three books that vedicreader has read, saved exactly as the model returned them.
name book pages
bj Varāhamihira’s Bṛhajjātaka, chapters 3 and 4 16
vm the Tamil Vidhānamālai, with its commentary (urai) 6
uro Uromarishi’s Pañcapakṣi, printed in 1917 4
``` python import json from copy import deepcopy ``` ``` python bj, vm, uro = [json.load(open(f'samples/{k}.json')) for k in ('brihat_jataka', 'vidhanamalai', 'uromarishi')] len(bj), len(vm), len(uro) ``` (16, 6, 4) ## verses Models do not label blocks consistently. One calls a heading `head` and another `heading`; vedicreader stores a page’s blocks under `items` and vishalakshi under [`blocks`](https://vedicreader.github.io/lekhaka/verses.html#blocks). [`kind`](https://vedicreader.github.io/lekhaka/verses.html#kind) and [`blocks`](https://vedicreader.github.io/lekhaka/verses.html#blocks) accept both. [`vnum`](https://vedicreader.github.io/lekhaka/verses.html#vnum) reads a printed verse number in any digits, Devanagari, Tamil or ASCII. The first verse on this Pañcapakṣi page is numbered 205 in Tamil numerals. Some books print two numbers on a verse, such as २६॥२७; [`vnum`](https://vedicreader.github.io/lekhaka/verses.html#vnum) takes the last unless you ask for the first. ``` python kind(dict(kind='head')), len(blocks(uro[0])), vnum(uro[0]['blocks'][1]['n']), vnum('२६॥२७'), vnum('२६॥ १', pick='first') ``` ('heading', 8, 205, 27, 26) [`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble) puts the verses into chapters. A colophon closes the current chapter, and a chapter also ends when the numbering drops back to 1. A heading names the chapter it falls in, if that chapter has no name yet; it does not start a chapter by itself. A block with no number is taken to be the start of the next numbered verse, because a verse that begins at the foot of one page often gets its number only on the next. The sixteen Bṛhajjātaka pages give two chapters, of 8 and 22 verses, with nothing to report: ``` python chs, probs = assemble(bj) [(c['heading'], len(c['verses']), c['colophon'][:30]) for c in chs], probs ``` ([('वियोनिजन्माध्यायः', 8, 'इति श्रीवराहमिहिरकृते बृहज्जात'), ('निषेकाध्यायः', 22, 'इति श्रीवराहमिहिरकृते बृहज्जात')], []) Some blocks without a number are whole verses, not pieces of one. A book may open with a maṅgala, a verse of blessing printed before verse 1, and a commentary may quote a verse from another work. Such a block ends in ॥ like any verse but carries no number, and adding it to the next verse is wrong. Page p0025a of the Bṛhajjātaka has one. Between verses 3 and 4 the commentary quotes a verse in the āryā metre. By default [`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble) adds it to verse 4, which ends up with four lines instead of two. With `openers=True`, a block that ends in ॥ and has no number stays a separate verse, with `n` set to None: ``` python op, _ = assemble(bj, openers=True) [(v['n'], v['page'], v['text'][:25]) for v in op[1]['verses'][2:5]], len(chs[1]['verses'][3]['text'].split('\n')) ``` ([(3, 'p0025a', 'रवीन्दुशुक्रावनिजैः स्वभा'), (None, 'p0025a', 'बलयुतौ स्वगृहांशेष्वर्कसि'), (4, 'p0025a', 'दिवाकरेन्द्वोः स्मरगौ कुज')], 4) Many books print a commentary under each verse. `notes=('urai',)` keeps blocks of that kind with the verse they explain, in `verse['notes']`, instead of counting them as verse text. Some books number the commentary and leave the verse unnumbered, and the Vidhānamālai is one of them. `number_from_note=True` gives each verse the number of its commentary. [`join_cont`](https://vedicreader.github.io/lekhaka/verses.html#join_cont) joins a block that a page break cut in two. The model marks the second half with `cont`, and a word hyphenated across the break is put back together. `assemble(cont=True)` does this before anything else. ``` python vc, _ = assemble(vm, notes=('urai',), number_from_note=True) v = vc[0]['verses'][0] len(vc[0]['verses']), v['n'], v['notes']['urai'][0][:40] ``` (11, 1, '(என்றது.) ஸ்ரீமகாலக்ஷ்மிக்கு நாயகரும், ப') ``` python join_cont([dict(page=1, blocks=[dict(kind='prose', text='a com-')]), dict(page=2, blocks=[dict(kind='prose', text='mentary', cont=True)])])[0]['blocks'][0]['text'] ``` 'a commentary' [`mend`](https://vedicreader.github.io/lekhaka/verses.html#mend) fixes two common slips in a chapter’s numbering. When the model reads the same number twice, as in 7, 9, 9, 10, it takes the first 9 as 8. When the model reads two verses as one block, it splits them. [`renumber`](https://vedicreader.github.io/lekhaka/verses.html#renumber) fixes a run of numbers that are wrong the same way: in 52, 63, 64, 65, 56 the model misread the tens digit three times. [`check_counts`](https://vedicreader.github.io/lekhaka/verses.html#check_counts) compares each chapter’s number of verses with the count you expect: ``` python V = lambda n: dict(n=n, text='a ।\nb ॥', page='p1') p = []; [v['n'] for v in mend([V(7), V(9), V(9), V(10)], p)], p ``` ([7, 8, 9, 10], ['p1: first of two 9s taken as 8']) ``` python [v['n'] for v in renumber([V(n) for n in (52, 63, 64, 65, 56)], 'ch 3', [])], check_counts(chs, [8, 21]) ``` ([52, 53, 54, 55, 56], ['chapter 2: 22 verses, the plan has 21']) Some books print no chapter headings, and the only marker is the colophon at the end of each chapter, such as इति श्रीवराहमिहिरकृते बृहज्जातके निषेकाध्यायश्चतुर्थः. [`assemble_colophon`](https://vedicreader.github.io/lekhaka/verses.html#assemble_colophon) starts a new chapter after each colophon and reads the chapter’s part, number and name from it. It ignores colophons that belong to a commentary. ``` python cc, _ = assemble_colophon(bj) [(c['part'], c['no'], c['name'], len(c['verses'])) for c in cc] ``` [('पूर्व', 3, 'वियोनिजन्म', 8), ('पूर्व', 4, 'निषेक', 22)] Models sometimes write akṣaras that cannot occur, such as a virāma followed by a vowel sign (ध्ा). [`bad_aksharas`](https://vedicreader.github.io/lekhaka/verses.html#bad_aksharas) names each fault and [`fix_aksharas`](https://vedicreader.github.io/lekhaka/verses.html#fix_aksharas) repairs it. Three smaller helpers deal with typing slips: [`deva`](https://vedicreader.github.io/lekhaka/verses.html#deva) writes a number in Devanagari digits, [`norm`](https://vedicreader.github.io/lekhaka/verses.html#norm) replaces a colon used for a visarga and ASCII digits inside ॥ ॥, and [`virama`](https://vedicreader.github.io/lekhaka/verses.html#virama) puts back a virāma dropped at the end of a pāda. ``` python bad_aksharas('ध्ार्म'), fix_aksharas('ध्ार्म'), deva(18), norm('राम: ॥ 1 ॥'), virama('जगत ।') ``` (['virāma before a vowel sign'], 'धार्म', '१८', 'रामः॥ १॥', 'जगत् ।') ## book [`read_book`](https://vedicreader.github.io/lekhaka/book.html#read_book) does the whole job for a PDF. It reads the pages through vishalakshi and passes them to [`assemble_book`](https://vedicreader.github.io/lekhaka/book.html#assemble_book), which applies your fix rows, gives a verse read without its number the number it ends on (॥ १२ ॥), assembles the chapters, repairs akṣaras, and compares the chapters with `counts`. [`read_book`](https://vedicreader.github.io/lekhaka/book.html#read_book) takes `renumber=True` to renumber each chapter. Joining blocks split by a page break (`cont=True`) is an option of [`assemble_book`](https://vedicreader.github.io/lekhaka/book.html#assemble_book) only, for pages you already have. [`book_md`](https://vedicreader.github.io/lekhaka/book.html#book_md) writes chapters as markdown: ``` python print(book_md(chs[:1], 'बृहज्जातकम्')[:160]) ``` # बृहज्जातकम् ## वियोनिजन्माध्यायः क्रूरग्रहैस्तु बलिभिर्विंबलैश्च सौम्यैः क्लीबे चतुष्टयगतेतद्वेक्षणाद्वा। चन्द्रोपगद्विरसभागसमानरूपं सत्त्वं वदेद्यदि भवेत्स ## repairs Some mistakes need a person. You look at the scan, decide what the page really says, and write that down as a row. lekhaka applies the rows and records each change in `probs`, so the list of problems also says what was changed and why. [`apply_fixes`](https://vedicreader.github.io/lekhaka/repairs.html#apply_fixes) works on pages, before [`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble). A row is `(page, block, kind, read, printed, why)`: the text the model read, the text the page prints, and your reason. A `fix` row replaces the first match and a `fix_all` row replaces every match. ``` python apply_fixes(deepcopy(bj), [('p0022b', 2, 'fix', 'शेभनाः', 'शोभनाः', 'a typo')], probs=(p := [])); p ``` ['p0022b: read `शेभनाः`, printed `शोभनाः` (a typo)'] [`number_runs`](https://vedicreader.github.io/lekhaka/repairs.html#number_runs) numbers verses printed without numbers when they sit between two numbered verses, since the gap says what the missing numbers must be. [`borrow_numbers`](https://vedicreader.github.io/lekhaka/repairs.html#borrow_numbers) takes a missing number from a second reading of the same page, when both readings have the same verse. ``` python number_runs([dict(page='p1', blocks=[dict(kind='verse', n=1, text='a'), dict(kind='verse', text='b'), dict(kind='verse', n=3, text='c')])], p := []) mine = deepcopy(bj[:1]); mine[0]['blocks'][2]['n'] = None borrow_numbers(mine, bj, p); p ``` ['p1: verse printed without its number taken as 2, between 1 and 3', "p0022b: verse 2's number, lost in this reading, taken from the other (1.00)"] [`cut_by_plan`](https://vedicreader.github.io/lekhaka/repairs.html#cut_by_plan) starts a chapter where you say one starts, given as `(page, where, why)`, where `where` is a block’s position on the page or the first words of a heading. Use it when a book prints neither a heading nor a colophon that [`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble) can find. [`drop_pages`](https://vedicreader.github.io/lekhaka/repairs.html#drop_pages) removes the verses on pages that are not part of the text, such as a preface or an index. ``` python cut_by_plan(deepcopy(vm), [('p0020a', 'பதிகம்.', 'the contents open a chapter')], p := []) drop_pages(deepcopy(bj), [('p0029b', 'p0030a')], p); p ``` ['p0020a: a chapter starts at `பதிகம்.`, by the plan (the contents open a chapter)', 'p0029b: verses dropped, not the text', 'p0030a: verses dropped, not the text'] [`fix_chapters`](https://vedicreader.github.io/lekhaka/repairs.html#fix_chapters) works on chapters, after [`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble), and finds a verse by its chapter and number instead of its page and position. Its rows are `(chapter, verse, kind, read, printed, why)`. Besides `fix` and `fix_all`, a `note` row records a remark without changing the text, and a `start` row starts a new chapter at a verse. If a chapter holds two verses with the same number, a `start` row’s `printed` says which one is meant: the page it is on, or 1 for the first and 2 for the second. ``` python rows = [(1, 2, 'fix', 'शेभनाः', 'शोभनाः', 'a typo'), (2, 19, 'fix_all', '।।', '॥', 'doubled daṇḍas'), (2, 15, 'note', None, None, 'its first two lines are a quotation')] fix_chapters(deepcopy(chs), rows, p := []); p ``` ['1.2 (p0022b): read `शेभनाः`, printed `शोभनाः` (a typo)', '2.19 (p0029a): read `।।`, printed `॥`, 2 times (doubled daṇḍas)', '2.15: its first two lines are a quotation'] ``` python one = [dict(heading=None, colophon=None, verses=chs[0]['verses'] + chs[1]['verses'])] p = []; [len(c['verses']) for c in fix_chapters(one, [(1, 1, 'start', None, 'p0024b', 'by hand')], p)], p ``` ([8, 22], ['p0024b: a chapter starts at verse 1.1 (by hand)']) Works in sūtra form need different handling. [`sutras`](https://vedicreader.github.io/lekhaka/repairs.html#sutras) joins each sūtra onto one line and closes it with ॥. ``` python sutras([dict(verses=[dict(n=1, text='अथातो धर्म-\nजिज्ञासा ।', page='p1')])], p := [])[0]['verses'][0]['text'], p ``` ('अथातो धर्मजिज्ञासा ॥', ['1.1 (p1): closing ॥ supplied']) ## editions vedicreader publishes a work as a set of titles. [`cut`](https://vedicreader.github.io/lekhaka/editions.html#cut) divides chapters into titles: one per chapter, one every `per` verses, or the whole work as one. A verse without a number, such as the quoted āryā above, gets a letter (क, ख and so on) in place of a number, with the word `open_word` (मङ्गलम्) as its label. [`numbering`](https://vedicreader.github.io/lekhaka/editions.html#numbering) summarises a run of verses: how many are numbered, which numbers are skipped or repeated, and the first and last. ``` python [t['title'] for t in cut(cc, 'chapters', book='बृहज्जातकम्')], [(v['n'], v.get('word')) for v in cut(op, 'chapters')[1]['verses'][2:5]] ``` (['बृहज्जातकम् १ वियोनिजन्म', 'बृहज्जातकम् २ निषेक'], [(3, None), ('क', 'मङ्गलम्'), (4, None)]) ``` python numbering(op[1]['verses']) ``` {'numbered': 22, 'unnumbered': 1, 'skipped': [], 'repeated': [], 'first': 1, 'last': 22} The next functions compare a reading with another edition of the same text, called a witness. [`witness_verses`](https://vedicreader.github.io/lekhaka/editions.html#witness_verses) takes the raw HTML of a witness page, from GRETIL for example, and returns its numbered verses. [`letters`](https://vedicreader.github.io/lekhaka/editions.html#letters) reduces a verse to its letters, dropping numbers, punctuation and bracketed notes, so that two editions can be compared on the words alone. [`text_changes`](https://vedicreader.github.io/lekhaka/editions.html#text_changes) proposes a change wherever the closest verse in the witness spells a word differently, and also flags dropped virāmas. ``` python witness_verses('

क ख ॥ १ ॥

ग घ ॥ २ ॥

'), letters('रामः ॥ १ ॥ (टि)') ``` ([(1, 'क ख'), (2, 'ग घ')], 'रामः') ``` python wit = [(v['n'], v['text'].replace('शेभनाः', 'शोभनाः')) for v in chs[0]['verses']] [(c['id'], c['read'], c['print'], c['why']) for c in text_changes(chs[:1], wit)] ``` [('1.2.w4', 'शेभनाः', 'शोभनाः', 'the witness (2) reads so')] [`ref_check`](https://vedicreader.github.io/lekhaka/editions.html#ref_check) flags verses that read very differently from a reference edition, which usually means a misread or a lost line. [`move_lines`](https://vedicreader.github.io/lekhaka/editions.html#move_lines) fixes a common slip, where the last line of one verse was read as the first line of the next, by moving the line back. ``` python ref = {1: [letters(v['text']) for v in chs[0]['verses']]}; ref[1][4] = '' ref_check(chs[:1], ref) ``` ['1.5 (p0023a): reads unlike the reference (0.29), check against the scan'] ``` python mv = deepcopy(chs[:1]); a, b = mv[0]['verses'][:2]; ls = a['text'].split('\n') a['text'], b['text'] = ls[0], ls[1] + '\n' + b['text'] move_lines(mv, {(1, v['n']): letters(v['text']) for v in chs[0]['verses']}, p := []); p, mv[0]['verses'][0]['text'] == chs[0]['verses'][0]['text'] ``` (['1.2: 1 leading line(s) moved back to 1.1'], True) The last group measures how far apart two readings of the same pages are, which is useful when you try a second model. [`verse_text`](https://vedicreader.github.io/lekhaka/editions.html#verse_text) collects a page’s verse text, [`lev`](https://vedicreader.github.io/lekhaka/editions.html#lev) is the edit distance between two strings, and [`disagree`](https://vedicreader.github.io/lekhaka/editions.html#disagree) is the share of a page’s text that differs. [`compare`](https://vedicreader.github.io/lekhaka/editions.html#compare) puts the result for each page in a table and summarises it. Here the second reading lost the first verse on p0023a, and that page stands out: ``` python b2 = deepcopy(bj); b2[1]['blocks'] = b2[1]['blocks'][1:] # a second reading that lost p0023a's first verse verse_text(bj[1])[:12], lev('क्रियः', 'क्रिया'), round(disagree(bj[1], b2[1]), 2) ``` ('क्रियःशिरोवक', 1, 0.34) ``` python md, flag = compare({'opus': bj, 'other': b2}, [('opus', 'other')]) print(md) ``` | page | opus vs other | |---|---| | p0023a | 34.1% | opus vs other: 16 pages, mean 2.1%, 1 at 10% or more