# editions
These functions work on the chapters
[`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
returns, `[{heading, colophon, verses}]`, where a colophon is the line
that closes a chapter. A few work on a single page of blocks, the pieces
of a page a vision model returns: verses, headings, colophons,
commentary and prose. Several compare a reading with another edition of
the same work, called a witness. You fetch or load the witness yourself
and pass it in; nothing here reads a file or goes to the web.
## Titles
vedicreader publishes a work as a set of titles, each a page of verses
with its own name.
[`cut`](https://vedicreader.github.io/lekhaka/editions.html#cut) divides
chapters into titles, each `{name, title, chapter, verses}`, in one of
three ways chosen by `how`:
- `how='per'`, the default, makes one title for every `per` verses (10
by default) across the whole work, by the verses’ own numbers: verses
1 to 10, then 11 to 20, and so on. An unnumbered verse goes with the
numbered verse before it, and unnumbered verses before the first
numbered one go with the first title. A title’s range stops at the
last number it actually holds, and every title has `chapter` 1.
- `how='chapters'` makes one title for each chapter, named from `book`,
the chapter’s number and its heading.
- `how='one'` puts the whole work in a single title.
`name` is a file name built from `stem`, such as `work_011_012` or
`work_02`, and `title` is the heading a reader sees, with its numbers
written by `digits` (Devanagari by default).
A verse with no number, such as a maṅgala (a verse of blessing that
opens a book) or a verse a commentary quotes, still needs a label.
[`cut`](https://vedicreader.github.io/lekhaka/editions.html#cut) gives
it a letter, क, ख, ग and so on, starting again from क after each
numbered verse, and sets its `word` to `open_word` (मङ्गलम् by default).
An unnumbered verse therefore never takes a number that belongs to a
numbered verse.
[`numbering`](https://vedicreader.github.io/lekhaka/editions.html#numbering)
reports what a run of verses’ own numbers say: how many verses are
numbered and unnumbered, which numbers are skipped or repeated, and the
first and last.
------------------------------------------------------------------------
source
### numbering
``` python
def numbering(
vs
)->dict:
```
*Report what a run of verses’ own numbers say: how many are numbered and
unnumbered, which are skipped or repeated, and the first and last.*
------------------------------------------------------------------------
source
### cut
``` python
def cut(
chs, how:str='per', per:int=10, open_word:str='मङ्गलम्', stem:str='work', book:str='', digits:function=deva,
letters:str='कखगघङचछजझञ'
)->list:
```
*Divide chapters’ verses into titles `[{name, title, chapter, verses}]`:
one for every `per` verses, one for each chapter, or one for the whole
work.*
``` python
V = lambda n, t='x': dict(n=n, text=t, page=1, heading=None)
_cs = [dict(heading='प्रथमः', verses=[V(None), V(None), V('१'), V(2)]), dict(heading='द्वितीयः', verses=[V(3), V(None), V('௧௧'), V(12)])]
test_eq([(t['name'], t['title'], [v['n'] for v in t['verses']]) for t in cut(_cs, per=10, book='शतकम्')],
[('work_001_003', 'शतकम् १–३', ['क', 'ख', 1, 2, 3, 'क']), ('work_011_012', 'शतकम् ११–१२', [11, 12])])
test_eq(cut(_cs)[0]['verses'][0]['word'], 'मङ्गलम्'); test_eq('word' in cut(_cs)[0]['verses'][2], False)
test_eq([(t['name'], t['title']) for t in cut(_cs, 'chapters', stem='s', book='ग्रन्थः')], [('s_01', 'ग्रन्थः १ प्रथमः'), ('s_02', 'ग्रन्थः २ द्वितीयः')])
test_eq(len(cut(_cs, 'one')[0]['verses']), 8)
test_eq(numbering([V(1), V(None), V('३'), V('௩'), V(6)]), dict(numbered=4, unnumbered=1, skipped=[2, 4, 5], repeated=[3], first=1, last=6))
```
## A witness
A witness is another edition of the same text, used to check a reading.
It might be a page from sanskritdocuments.org or a GRETIL file.
[`witness_verses`](https://vedicreader.github.io/lekhaka/editions.html#witness_verses)
takes the raw page, HTML or plain text, removes the markup, and cuts it
into verses at each verse number written as `॥ n ॥` or `|| n ||`,
returning `[(n, text)]`. GRETIL writes each number with a reference, as
in `|| BharSt_1.12 ||`, and the verse number is taken from its end. Only
the last six lines before each number are kept, so a page header does
not end up in the first verse. If the page is in IAST (Roman letters
with diacritics) instead of Devanagari, pass a `translit` function; it
is used when the page has fewer than 200 Devanagari characters. A
witness shows where to look again; the reading should still follow the
scan, not the witness.
Two editions differ in punctuation, numbering and notes even where their
words agree.
[`letters`](https://vedicreader.github.io/lekhaka/editions.html#letters)
reduces a text to its Devanagari letters alone, removing the verse
number between ॥ marks, bracketed notes, spaces, daṇḍas and digits. The
checks below compare texts in this form.
[`text_changes`](https://vedicreader.github.io/lekhaka/editions.html#text_changes)
proposes corrections and does not make them. It does two checks.
1. On each line it looks for a pāda-final virāma that was dropped,
using
[`virama`](https://vedicreader.github.io/lekhaka/verses.html#virama).
A pāda is a quarter of a verse, usually ending in a daṇḍa, and a
virāma (्) is the mark that removes a consonant’s vowel. These
changes have `by` set to `['rule']`.
2. When you pass `wit`, the output of
[`witness_verses`](https://vedicreader.github.io/lekhaka/editions.html#witness_verses),
it finds the witness verse whose letters are most like the verse’s,
if they are at least `at` alike (0.7 by default), and compares the
two word by word. Where one or two words are spelled differently, it
proposes the witness’s spelling, with `by` set to `['witness']`. You
can also pass `metre`, a function that takes a verse’s lines and
returns the metre’s name if they scan, or None. A verse that already
scans is not compared, and if a proposed change makes a verse scan,
`by` also lists `'metre'`.
Each change is a dict `{id, chapter, n, page, read, print, why, by}`. A
proofreader accepts or rejects each one, and an accepted change can be
applied as a `fix` row for
[`fix_chapters`](https://vedicreader.github.io/lekhaka/repairs.html#fix_chapters),
with its `read` and `print` as the read and printed text.
------------------------------------------------------------------------
source
### text_changes
``` python
def text_changes(
chs, wit:tuple=(), metre:NoneType=None, at:float=0.7, key:NoneType=None
)->list:
```
*Propose changes for dropped pāda-final virāmas, and for words the
nearest witness verse spells differently in verses that `metre` does not
find scanning.*
------------------------------------------------------------------------
source
### letters
``` python
def letters(
s:str
)->str:
```
*Reduce `s` to its Devanagari letters, removing the closing `॥ n ॥`,
bracketed notes, spaces, daṇḍas and digits.*
------------------------------------------------------------------------
source
### witness_verses
``` python
def witness_verses(
raw:str, translit:NoneType=None
)->list:
```
*Cut a witness edition’s raw page into verses `[(n, text)]` at each
`॥ n ॥` or `|| n ||`, with the markup removed; `translit` turns an IAST
page into Devanagari.*
``` python
_raw = 'नीतिशतकम्
दिक्कालाद्यनवच्छिन्नानन्तचिन्मात्रमूर्तये ।
स्वानुभूत्येकमानाय नमः शान्ताय तेजसे ॥ १॥
बोद्धारो मत्सरग्रस्ताः प्रभवः स्मयदूषिताः ।
अबोधोपहताश्चान्ये जीर्णमङ्गे सुभाषितम् ॥ 2 ॥
'
_w = witness_verses(_raw)
test_eq([n for n, _ in _w], [1, 2]); test_eq(_w[1][1], 'बोद्धारो मत्सरग्रस्ताः प्रभवः स्मयदूषिताः ।\nअबोधोपहताश्चान्ये जीर्णमङ्गे सुभाषितम्')
test_eq([n for n, _ in witness_verses('a b c || BharSt_1.12 || d e f || BharSt_1.13 ||')], [12, 13])
test_eq([t for _, t in witness_verses('rama || 1 ||', translit=str.upper)], ['RAMA']) # a short Latin page goes through `translit`
test_eq(letters('राम (रामः) ॥ १ ॥ सीता ।'), 'रामसीता')
_ch = [dict(verses=[dict(n=1, page='p1', text='दिक्कालाद्यनवच्छिन्नानन्तचिन्मात्रमूर्तये ।\nस्वानुभूत्येकमानाय नमः शान्ताय तेजसे ॥ १॥'),
dict(n=2, page='p1', text='बोद्धारो मत्सरग्रस्ताः प्रभवः स्मयदूषिताः ।\nअबोधोपहताश्चान्ये जीर्णमङ्गे सुभाषितम ॥ २॥')])]
_x = text_changes(_ch, _w)
test_eq([(x['id'], x['read'], x['print'], x['by']) for x in _x], [('1.2.w6', 'सुभाषितम', 'सुभाषितम्', ['witness'])])
test_eq([(x['id'], x['read'], x['print'], x['by']) for x in text_changes([dict(verses=[dict(n=1, text='नमो भगवत ।\nवासुदेवाय ॥ १॥')])])], [('1.1.v0', 'भगवत', 'भगवत्', ['rule'])])
test_eq([x['by'] for x in text_changes(_ch, _w, metre=lambda ls: 'anuṣṭubh' if 'सुभाषितम्' in ls[-1] else None)], [['witness', 'metre']])
```
## Against a reference edition
A reference edition is a complete text of the same work, close enough to
compare whole verses with. For
[`ref_check`](https://vedicreader.github.io/lekhaka/editions.html#ref_check),
`ref` is `{chapter: [letters of each verse]}`, built with
[`letters`](https://vedicreader.github.io/lekhaka/editions.html#letters)
as in the test below.
[`ref_check`](https://vedicreader.github.io/lekhaka/editions.html#ref_check)
reports problems and changes nothing. It compares each verse with the
reference verses near it, meaning within `win` (6) places of its
position or of its number, and flags it when even the best match is less
than `at` (0.8) alike. That usually means a misread or a lost line. It
also flags a verse containing a lacuna mark (·· or …), where the print
itself shows missing text. Verses past the end of a shorter reference
are not checked.
The model sometimes puts the boundary between two verses in the wrong
place, so that the last line of verse 4 is read as the first line of
verse 5.
[`move_lines`](https://vedicreader.github.io/lekhaka/editions.html#move_lines)
repairs this by comparing with a reference given as
`{(chapter, n): text}`, where each text has been reduced by `fold`
([`letters`](https://vedicreader.github.io/lekhaka/editions.html#letters)
by default, or a function of your own for a reference in IAST). When a
verse matches its reference much better without its first lines,
[`move_lines`](https://vedicreader.github.io/lekhaka/editions.html#move_lines)
does one of three things. If adding those lines to the verse before
makes that verse match its own reference better, it moves them back. If
the rest of the verse then matches closely and the lines match no
reference verse from n-2 to n+2, they are taken as a quotation the
editor printed, and dropped. Otherwise it changes nothing and asks for a
check by hand. Each case is recorded in `probs`.
In both functions `key(i, chapter)` gives the name a chapter has in the
reference; by default it is the chapter’s position counting from 1.
------------------------------------------------------------------------
source
### move_lines
``` python
def move_lines(
chs, ref, probs:list=None, fold:function=letters, key:NoneType=None
)->list:
```
*Move the leading lines of verse n back to verse n-1 when they belong
there, or drop them when they match no nearby verse, judged against
`ref`.*
------------------------------------------------------------------------
source
### ref_check
``` python
def ref_check(
chs, ref, probs:list=None, at:float=0.8, win:int=6, key:NoneType=None
)->list:
```
*List the verses that read unlike every nearby verse of the reference
edition, and verses with a lacuna, without changing anything.*
``` python
import json
from lekhaka.verses import assemble
```
``` python
_bj, _ = assemble(json.load(open('samples/brihat_jataka.json')))
_ref = {ci: [letters(v['text']) for v in c['verses']] for ci, c in enumerate(_bj, 1)}
test_eq(ref_check(_bj, _ref), [])
_v = dict(_bj[0]['verses'][3], text='पृथिव्यापस्तेजो वायुराकाशमेव च ।\nएतानि पञ्च भूतानि ॥') # a verse the reference does not have
_p = ref_check([dict(verses=_bj[0]['verses'][:3] + [_v, dict(_v, n=5, text='क ···· ख')])], _ref); test_eq(len(_p), 3); assert 'lacuna' in _p[1]
_ref2 = {(ci, v['n']): letters(v['text']) for ci, c in enumerate(_bj, 1) for v in c['verses']}
_a, _b = _bj[0]['verses'][0], _bj[0]['verses'][1]
_moved = [dict(_a, text=_a['text'].split('\n')[0]), dict(_b, text=_a['text'].split('\n')[1] + '\n' + _b['text'])] # verse 1's last line read as verse 2's first
_p = []; _m = move_lines([dict(verses=_moved)], _ref2, _p)[0]['verses']
test_eq([v['text'] for v in _m], [_a['text'], _b['text']]); test_eq(_p, ['1.2: 1 leading line(s) moved back to 1.1'])
_q = [_a, dict(_b, text='पृथिव्यापस्तेजो वायुराकाशमेव च ।\n' + _b['text'])]
_p = []; test_eq(move_lines([dict(verses=_q)], _ref2, _p)[0]['verses'][1]['text'], _b['text']); assert 'quotation' in _p[0]
```
## Two readings of a page
When you try a second model, or read a book again, you want to know
where the two readings differ. These functions measure that page by
page.
[`verse_text`](https://vedicreader.github.io/lekhaka/editions.html#verse_text)
joins the text of a page’s verse blocks as bare letters, removing
numbers, daṇḍas, spaces, punctuation and the avagraha (ऽ). With
`numbered=True` it keeps only the verses read with a number, which
leaves out the part of a verse carried over a page break and verses a
commentary quotes.
[`lev`](https://vedicreader.github.io/lekhaka/editions.html#lev) counts
the character edits (insertions, deletions and substitutions) that turn
one string into another.
[`disagree`](https://vedicreader.github.io/lekhaka/editions.html#disagree)
is the share of a page’s text that differs between two readings: the
edits divided by the length of the longer text. It is 1 when only one
reading found any verse on the page, and None when neither did or one
reading lacks the page.
[`compare`](https://vedicreader.github.io/lekhaka/editions.html#compare)
takes named readings, each a list of pages or a dict from page name to
page, and `pairs` of names to compare. It returns a markdown table and a
list of flagged pages, the pages where the first pair differs by `at`
(10%) or more. The table shows only the flagged pages, followed by one
summary line per pair giving the number of pages, the mean difference
and how many pages reach `at`. If you pass `pages`, the table shows
exactly those pages instead.
------------------------------------------------------------------------
source
### compare
``` python
def compare(
readings:dict, pairs, at:float=0.1, pages:NoneType=None, numbered:bool=False
)->tuple:
```
*A markdown table of how far each pair of named readings differs, page
by page, and the pages where the first pair differs by `at` or more.*
------------------------------------------------------------------------
source
### disagree
``` python
def disagree(
a, b, numbered:bool=False
)->float | None:
```
*The share of a page’s letters that differ between two readings: 1 when
only one found verses, None when neither did or a page is missing.*
------------------------------------------------------------------------
source
### lev
``` python
def lev(
a:str, b:str
)->int:
```
*The number of character edits (the Levenshtein distance) between two
strings.*
------------------------------------------------------------------------
source
### verse_text
``` python
def verse_text(
page, numbered:bool=False
)->str:
```
*A page’s verse text as bare letters; `numbered=True` keeps only the
verses read with a number.*
``` python
test_eq(lev('kitten', 'sitting'), 3); test_eq(verse_text(dict(items=[dict(kind='verse', text='राम ॥ १ ॥'), dict(kind='prose', text='x'), dict(kind='verse', text='सीता')]), numbered=True), '')
_one = json.load(open('samples/brihat_jataka.json'))
_two = [dict(p, blocks=[dict(b, text=b['text'].replace('ग्र', 'ग')) for b in p['blocks']]) for p in _one]
_two[3] = dict(_two[3], blocks=[dict(b, text='अ') if b['kind'] == 'verse' else b for b in _two[3]['blocks']])
test_eq(disagree(_one[0], _one[0]), 0); assert 0 < disagree(_one[0], _two[0]) < .1; test_eq(disagree(_one[0], None), None)
_md, _fl = compare(dict(a=_one, b=_two), [('a', 'b')])
test_eq(_fl, [_one[3]['page']]); assert _md.startswith('| page | a vs b |') and '16 pages' in _md
```