# repairs
Some mistakes in a reading cannot be fixed by rule. A person has to look
at the scan, decide what the page really says, and write that decision
down. The functions here apply those decisions, along with a few repairs
that need a hint from you, such as where a chapter starts.
Most of them work on pages of blocks, `[{page, blocks}]` (or `items`),
before
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
puts the verses into chapters. A block is one piece of a page as the
vision model read it: a verse, a heading, a colophon (the line that
closes a chapter), a commentary or prose. These functions change the
pages in place and also return them.
[`fix_chapters`](https://vedicreader.github.io/lekhaka/repairs.html#fix_chapters)
and
[`sutras`](https://vedicreader.github.io/lekhaka/repairs.html#sutras)
work later, on the chapters
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
returns. Every change goes into `probs`, a list you pass in, so the list
of problems also says what was changed and why.
They were written for books vedicreader read and had to repair again and
again: the fix tables for the Bṛhat Saṃhitā, the Muhūrta Cintāmaṇi and
the Nārada Saṃhitā, the unnumbered runs of the Phaladīpikā, the second
readings of the Jātaka Pārijāta, and the chapter plans of the
Grahacintāmaṇi.
## A table of fixes
Each correction is a fix row, `(page, item, kind, read, printed, why)`.
`page` and `item` say where the mistake is: the page’s name and the
block’s position on that page, counting from 0. `read` is the text as
the model read it, `printed` is what the page prints, and `why` is your
reason, which goes into `probs` with the change.
[`kind`](https://vedicreader.github.io/lekhaka/verses.html#kind) says
what to do:
- `fix` replaces the first occurrence of `read` in the block’s text with
`printed`, and `fix_all` replaces every occurrence.
- `doubt` is a fix you are not sure of. It is applied only when
`witness`, a dict keyed by `(page, item, read)`, holds `'agrees'` for
it, meaning another edition reads the same way. Otherwise the text is
kept and the doubt is recorded.
- `slip` records a mistake that the printed book itself makes. The text
is kept, since it matches the print.
- `number` sets the block’s number to `printed`.
- `drop` removes the block.
- `missing` adds a block the model skipped. `printed` is the whole block
as a dict, and it goes in after position `item`, or first on the page
when `item` is -1.
Blocks are removed and added only after every row has been read, so the
positions in all the rows refer to the page as the model read it. A row
for a page that is not in the reading is ignored. A row whose position
is past the end of the page, or whose `read` text is not in the block,
is not applied, and that is recorded in `probs`. The test below applies
rows of several kinds to one small page.
------------------------------------------------------------------------
source
### apply_fixes
``` python
def apply_fixes(
pages, rows, witness:dict=None, probs:list=None
)->list:
```
*Apply a table of fix rows `(page, item, kind, read, printed, why)` to
pages of blocks, recording each change in `probs`.*
``` python
_pg = lambda: [dict(page='p1', blocks=[dict(kind='verse', n=1, text='रामः वनं गच्छति ॥१॥'), dict(kind='verse', n=3, text='सीता ॥३॥'), dict(kind='prose', text='a note')])]
_p = []
_r = apply_fixes(_pg(), [('p1', 0, 'fix', 'गच्छति', 'गच्छत्', 'the print'), ('p1', 1, 'number', 3, 2, ''), ('p1', 2, 'drop', None, None, 'a footnote'),
('p1', 1, 'missing', None, dict(kind='verse', n=3, text='लक्ष्मणः ॥३॥'), 'at the foot'), ('p1', 0, 'doubt', 'रामः', 'राम', 'faint')], probs=_p)
test_eq([(b['n'], b['text']) for b in _r[0]['blocks']], [(1, 'रामः वनं गच्छत् ॥१॥'), (2, 'सीता ॥३॥'), (3, 'लक्ष्मणः ॥३॥')])
test_eq(len(_p), 5); assert any('doubt' in x for x in _p)
test_eq(apply_fixes(_pg(), [('p1', 0, 'doubt', 'रामः', 'राम', '')], witness={('p1', 0, 'रामः'): 'agrees'})[0]['blocks'][0]['text'][:3], 'राम')
_p = []; test_eq(apply_fixes([dict(page=0, blocks=[dict(kind='verse', text='क क ख')])], [(0, 0, 'fix_all', 'क', 'ग', '')], probs=_p)[0]['blocks'][0]['text'], 'ग ग ख')
test_eq(_p, ['0: read `क`, printed `ग`, 2 times'])
```
## Numbers
Models often miss a verse number, especially one printed small or faint,
or at the edge of the page. Two functions recover such numbers without
asking a person.
If verse 4 follows verse 1 with two unnumbered verse blocks between
them, those two must be verses 2 and 3.
[`number_runs`](https://vedicreader.github.io/lekhaka/repairs.html#number_runs)
numbers a run of unnumbered verses when it exactly fills the gap between
two numbered ones, and leaves it alone otherwise. Only blocks of kind
`verse` count. The count starts again from 0 after a colophon, and after
a heading that matches `reset` (ध्याय by default, as in अध्यायः), so the
first verses of a new chapter can be numbered from 1.
[`borrow_numbers`](https://vedicreader.github.io/lekhaka/repairs.html#borrow_numbers)
uses a second reading of the same pages, by another model or another
run. For each unnumbered verse, it finds the most similar numbered verse
on the same page of the other reading and takes its number, if the two
texts are at least `at` alike (0.6 by default, where 1 means identical)
and no verse on this page already has that number.
------------------------------------------------------------------------
source
### borrow_numbers
``` python
def borrow_numbers(
pages, other, probs:list=None, at:float=0.6
)->list:
```
*Give each unnumbered verse the number of the most similar verse on the
same page of `other`, a second reading of the same pages.*
------------------------------------------------------------------------
source
### number_runs
``` python
def number_runs(
pages, probs:list=None, reset:str='ध्याय'
)->list:
```
*Number each run of unnumbered verses that exactly fills the gap between
two numbered ones.*
``` python
_ps = [dict(page=0, blocks=[dict(kind='verse', n=1, text='a'), dict(kind='verse', text='b'), dict(kind='verse', text='c'), dict(kind='verse', n=4, text='d'),
dict(kind='colophon', text='इति'), dict(kind='verse', text='e'), dict(kind='verse', n=3, text='f')])]
test_eq([b.get('n') for b in number_runs(_ps)[0]['blocks']], [1, 2, 3, 4, None, None, 3]) # after the colophon one verse cannot fill 1..2
_mine = [dict(page=0, blocks=[dict(kind='verse', text='धर्मक्षेत्रे कुरुक्षेत्रे'), dict(kind='verse', n=2, text='दृष्ट्वा तु')])]
_theirs = [dict(page=0, blocks=[dict(kind='verse', n='१', text='धर्मक्षेत्रे कुरुक्षेत्रे समवेता'), dict(kind='verse', n='२', text='दृष्ट्वा तु')])]
test_eq([b['n'] for b in borrow_numbers(_mine, _theirs)[0]['blocks']], [1, 2])
```
``` python
_it = [dict(page=0, items=[dict(kind='verse', n=1, text='a'), dict(kind='verse', text='b'), dict(kind='head', text='द्वितीयोऽध्यायः'), dict(kind='verse', text='c'), dict(kind='verse', n=2, text='d')])]
test_eq([b.get('n') for b in number_runs(_it)[0]['items']], [1, None, None, 1, 2]) # `items` read as blocks, a `head` as a heading
test_eq([b['n'] for b in borrow_numbers([dict(page=0, items=[dict(kind='verse', text='धर्मक्षेत्रे कुरुक्षेत्रे')])], _theirs)[0]['items']], [1])
test_eq(len(apply_fixes([dict(page=0, items=[dict(kind='verse', text='x')])], [(0, 0, 'drop', None, None, '')])[0]['items']), 0)
```
## Chapters by plan
Some books start a chapter in a way
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
cannot see: with a heading the model did not mark as one, or with no
mark at all because a leaf is missing.
[`cut_by_plan`](https://vedicreader.github.io/lekhaka/repairs.html#cut_by_plan)
takes a plan, a list of `(page, item, why)` rows saying where each such
chapter begins. `item` is either a block’s position on the page or the
first words of a block’s text, usually a heading, and `why` is optional.
Before that block it puts a block of kind `start`, and
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
begins a new chapter at a `start` block. Each cut is recorded in
`probs`, and so is a row whose page or block cannot be found.
------------------------------------------------------------------------
source
### cut_by_plan
``` python
def cut_by_plan(
pages, plan, probs:list=None
)->list:
```
*Insert a chapter-opening `start` block at each `(page, item, why)` row
of `plan`, where `item` is a block’s position or the first words of its
text.*
``` python
from lekhaka.verses import assemble
_pl = [dict(page=0, blocks=[dict(kind='heading', text='கடவுள் வணக்கம்'), dict(kind='verse', n=1, text='x'), dict(kind='heading', text='வாரநயம்'), dict(kind='verse', n=2, text='y')])]
_p = []; _c, _ = assemble(cut_by_plan(_pl, [(0, 'வாரநயம்')], _p)); test_eq(_p, ['0: a chapter starts at `வாரநயம்`, by the plan'])
test_eq([(c['heading'], c['colophon'], [v['n'] for v in c['verses']]) for c in _c], [('கடவுள் வணக்கம்', None, [1]), ('வாரநயம்', None, [2])])
_p = []; cut_by_plan(_pl, [(5, 0), (0, 'அது')], _p); test_eq(len(_p), 2)
_p = []; cut_by_plan([dict(page='p2', items=[dict(kind='verse', n='२', text='x')])], [('p2', 0, 'the leaf before it lost')], _p)
test_eq(_p, ['p2: a chapter starts at verse 2, by the plan (the leaf before it lost)'])
```
## Pages that are not the text
Printed books often carry pages whose verses are not part of the work,
such as a preface, an index, or an appendix quoting another work. The
Phaladīpikā edition, for example, prints verses of the Jātakābharaṇa
after verse 26.48.
[`drop_pages`](https://vedicreader.github.io/lekhaka/repairs.html#drop_pages)
takes a list of `(first, last)` page ranges, both ends included, and
removes the verse blocks on those pages. Headings and other blocks stay.
------------------------------------------------------------------------
source
### drop_pages
``` python
def drop_pages(
pages, ranges, probs:list=None
)->list:
```
*Remove the verse blocks on every page in each `(first, last)` range,
for pages that are not part of the text.*
``` python
_p = []
_d = drop_pages([dict(page=k, items=[dict(kind='heading', text='h'), dict(kind='verse', n=k, text='v')]) for k in range(1, 5)], [(2, 3)], _p)
test_eq([len(p['items']) for p in _d], [2, 1, 1, 2]); test_eq(_p, ['2: verses dropped, not the text', '3: verses dropped, not the text'])
```
## After assembly
Once
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
has built the chapters, it is easier to name a verse by its chapter and
number than by its page and position.
[`fix_chapters`](https://vedicreader.github.io/lekhaka/repairs.html#fix_chapters)
takes fix rows `(chapter, verse, kind, read, printed, why)`, where
`verse` is the verse’s number. `chapter` is the chapter’s position
counting from 1, unless you pass `key`, a function `key(i, chapter)`
that names chapters some other way, such as the edition’s own chapter
numbers or a `(part, no)` pair. A `chapter` of None applies the row to
every chapter that has that verse. The rows are applied in order, so
after a `start` row has split a chapter, the rows that follow count the
chapters as split.
The kinds are:
- `fix` and `fix_all` replace `read` with `printed` in the verse, once
or every time, as in
[`apply_fixes`](https://vedicreader.github.io/lekhaka/repairs.html#apply_fixes).
- `slip` and `doubt` are recorded and the text is kept. A `doubt` is
never applied here, since
[`fix_chapters`](https://vedicreader.github.io/lekhaka/repairs.html#fix_chapters)
takes no witness.
- `note` records `why` as a remark on the verse and changes nothing.
- `number` sets the verse’s number to `printed`, and `drop` removes the
verse.
- `missing` adds a verse the reading lacks, perhaps copied by hand from
another copy of the book. `printed` is its text, or a dict such as
`{text, page}`, and it goes in at its place by number. If the chapter
already has a verse with that number, nothing is added.
- `recut` moves the boundary between a verse and the one before it, for
a case
[`mend`](https://vedicreader.github.io/lekhaka/verses.html#mend)
cannot see, such as a four-line verse whose last line has no daṇḍa, so
the model ran it into the next verse. The two verses are joined and
split again so that the earlier one keeps the first `printed` lines.
- `start` begins a new chapter at the verse. This is for a lost leaf
that took the end of one chapter and the start of the next, so the
numbering does not restart where
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
would notice. The verse cannot be the first of its chapter. If a
chapter holds two verses with that number, `printed` says which one:
the page it is on, or 1 for the first and 2 for the second. The first
part keeps the chapter’s heading and the second keeps its colophon.
Any other kind raises a `ValueError`.
[`sutras`](https://vedicreader.github.io/lekhaka/repairs.html#sutras) is
for works written in sūtras, short prose aphorisms, rather than verse.
It puts each sūtra on one line, joining wrapped lines and removing a
hyphen at a line break. Where a sūtra does not end in `close` (॥ by
default), it removes a final । or | and adds `close`, recording that in
`probs`.
------------------------------------------------------------------------
source
### sutras
``` python
def sutras(
chs, probs:list=None, close:str='॥', key:NoneType=None
)->list:
```
*Put each sūtra on one line, rejoining words hyphenated at a line break,
and end it with `close` where the reading lost it.*
------------------------------------------------------------------------
source
### fix_chapters
``` python
def fix_chapters(
chs, rows, probs:list=None, key:NoneType=None
)->list:
```
*Apply a table of fix rows `(chapter, verse, kind, read, printed, why)`
to assembled chapters; a `chapter` of None means every chapter with that
verse.*
``` python
V = lambda n, t, pg=1: dict(n=n, text=t, page=pg, heading=None)
_ch = lambda: [dict(heading='h1', colophon='c1', verses=[V(1, 'रामः गच्छति ॥१॥'), V(2, 'सीता ॥२॥'), V(4, 'क\nख\nग ॥४॥'), V(5, 'घ ॥५॥')]), dict(heading='h2', colophon='c2', verses=[V(1, 'रामः ॥१॥')])]
_p = []
_c = fix_chapters(_ch(), [(1, 1, 'fix', 'गच्छति', 'गच्छत्', 'typed'), (None, 1, 'fix', 'रामः', 'राम', ''), (1, 3, 'missing', None, 'लक्ष्मणः ॥३॥', 'from the 1905 print'),
(1, 5, 'recut', None, 2, ''), (1, 2, 'slip', 'सीता', None, 'so printed'), (2, 9, 'fix', 'x', 'y', ''), (1, 1, 'note', None, None, 'a lacuna'),
(1, 4, 'start', None, None, 'a leaf lost'), (2, 5, 'drop', None, None, 'a quotation')], _p)
test_eq([(c['heading'], c['colophon'], [(v['n'], v['text']) for v in c['verses']]) for c in _c],
[('h1', None, [(1, 'राम गच्छत् ॥१॥'), (2, 'सीता ॥२॥'), (3, 'लक्ष्मणः ॥३॥')]), (None, 'c1', [(4, 'क\nख')]), ('h2', 'c2', [(1, 'राम ॥१॥')])])
test_eq(_p[:3], ['1.1 (1): read `गच्छति`, printed `गच्छत्` (typed)', '1.1 (1): read `रामः`, printed `राम`', '2.1 (1): read `रामः`, printed `राम`'])
test_eq(len(_p), 10); assert '2.9: verse 9 is gone from the reading; fix x → y not applied' in _p
test_eq([v['n'] for v in fix_chapters(_ch(), [((1, 'x'), 2, 'drop', None, None, '')], key=lambda i, c: (i, 'x'))[0]['verses']], [1, 4, 5])
test_fail(lambda: fix_chapters(_ch(), [(1, 1, 'mend', 'a', 'b', '')]))
_p = []; _c = fix_chapters([dict(heading=None, colophon=None, verses=[V(1, 'क क ॥१॥')])], [(1, 1, 'fix_all', 'क', 'ग', 'twice')], _p)
test_eq((_c[0]['verses'][0]['text'], _p), ('ग ग ॥१॥', ['1.1 (1): read `क`, printed `ग`, 2 times (twice)']))
_two = lambda: [dict(heading='h7', colophon='c8', verses=[V(1, 'a'), V(2, 'b'), V(25, 'y', 'p0107a'), V(2, 'z', 'p0108a'), V(3, 'w', 'p0108a')])]
_p = []; _c = fix_chapters(_two(), [(None, 2, 'start', None, 'p0108a', 'the leaf before it lost')], _p) # two verses 2 in one chapter: the page picks
test_eq([[v['n'] for v in c['verses']] for c in _c], [[1, 2, 25], [2, 3]]); test_eq(_p, ['p0108a: a chapter starts at verse 2 (the leaf before it lost)'])
test_eq([[v['n'] for v in c['verses']] for c in fix_chapters(_two(), [(1, 2, 'start', None, 2, '')])], [[1, 2, 25], [2, 3]]) # or which one, from 1
test_eq([[v['n'] for v in c['verses']] for c in fix_chapters(_two(), [(1, 2, 'start', None, None, '')])], [[1], [2, 25, 2, 3]]) # else the first
_p = []; fix_chapters(_two(), [(1, 2, 'start', None, 'p0999a', '')], _p); test_eq(_p, ['1.2: no verse 2 after the first of a chapter (p0999a); the start not applied'])
_p = []
test_eq(sutras([dict(verses=[V(1, 'अथातो धर्म-\nजिज्ञासा ।'), V(2, 'चोदनालक्षणोऽर्थो धर्मः ॥')])], _p)[0]['verses'][0]['text'], 'अथातो धर्मजिज्ञासा ॥'); test_eq(_p, ['1.1 (1): closing ॥ supplied'])
```