book

Read a book of verses with a vision model and return its chapters, with numbers mended, akṣaras repaired and verse counts checked

read_book takes a PDF or a page image of a book of verses and does the whole job. It sends every page to the vision model model through vishalakshi’s read_doc, with the reading prompt named by preset (sanskrit by default) and any hint you add, such as “Each verse ends in ॥ n ॥.”. The model returns each page as a list of blocks, the pieces it found on the page: verses with their printed numbers, headings, colophons (the line that closes a chapter, इति … अध्यायः) and prose. read_book then passes the pages to assemble_book, described below.

If you pass a vishalakshi vault, the pages are stored there as versions of one document, and the result’s doc_id names it. Without a vault nothing is stored and doc_id is None. Other keyword arguments go on to read_doc. The most useful is cache, a folder where each page’s reading is saved as JSON, so that reading the book a second time sends nothing to the model. tries, progress, spread, strips and the rest work as they do in read_doc. chat replaces the model with a function of your own, which is how the tests below run without one.

The result holds the chapters, the probs, the prose blocks left out of the verses, the number of pages, the cost of the reading, and the pages vishalakshi reports as failed, unread or blocked. probs is a list of lines, one for every change lekhaka made and every place a proofreader should look at the scan.

assemble_book is the second half on its own, for pages that have already been read. A common case is a vault whose pages other models have proofread, with the corrections accepted: you take the pages back out of the vault and assemble them again. It works in this order:

A verse block read without a number takes the number its text ends on, if that is a number between ॥ marks (॥ १२ ॥). Blocks with no text are removed, except table blocks, which carry rows.
fixes, a table of fix rows for apply_fixes (see repairs), is applied.
With cont=True, blocks that a page break cut in two are joined (join_cont).
assemble builds the chapters, with pair, kinds, notes, number_from_note and openers passed to it.
With renumber=True, each chapter goes through renumber.
When fix is on, as it is by default, and script is Devanagari, each verse with malformed akṣaras (syllables written in a form no printer would set) is repaired with fix_aksharas, and the repair is listed in probs.
If you give counts, the number of verses expected in each chapter, every chapter with a different number is listed in probs.

The prose blocks, which are not verse, come back in left. read_book takes the same options except cont, which only assemble_book has.

book_md writes chapters out as markdown: a heading for each chapter, each verse followed by its number in italics, and the colophon at the end.

source

book_md

def book_md(
    chapters, title:str=''
)->str:

Write chapters as markdown: a heading for each chapter, each verse followed by its number, and the colophon at the end.

source

read_book

def read_book(
    path, vault:NoneType=None, model:str=None, preset:str='sanskrit', title:str='', script:str='Devanagari',
    hint:str='', pages:NoneType=None, counts:NoneType=None, fix:bool=True, pair:bool=False,
    kinds:tuple=('verse', None), notes:tuple=(), number_from_note:bool=False, fixes:NoneType=None, dpi:int=300,
    workers:int=4, chat:NoneType=None, openers:bool=False, renumber:bool=False, **kw
)->fastcore.basics.AttrDict: # passed on to vishalakshi's `read_doc`: `cache`, `redo`, `clean`, `tries`, `spread`, `strips`, `progress`, `pii`

Read a book with a vision model and return its chapters [{heading, colophon, verses}], the problems a proofreader should see, and what the reading cost and left unread.

source

assemble_book

def assemble_book(
    pages, script:str='Devanagari', pair:bool=False, kinds:tuple=('verse', None), notes:tuple=(),
    number_from_note:bool=False, fixes:NoneType=None, counts:NoneType=None, fix:bool=True, cont:bool=False,
    openers:bool=False, renumber:bool=False
)->fastcore.basics.AttrDict:

Assemble pages already read, [{page, blocks}], into chapters [{heading, colophon, verses}], and return them with the problems a proofreader should see and the prose left out. A verse read without its n takes the number its text ends on (॥ १२ ॥), and renumber=True corrects each chapter’s numbers.

from pdflite import pdf_doc
_src = Path(tempfile.mkdtemp())/'two.pdf'
from pdf_oxide import Pdf
_fx = Path(tempfile.mkdtemp())
for _i, _t in enumerate(['# Attention Is All You Need\n\nThe dominant sequence transduction models are based on recurrent or convolutional networks.',
                         '# Background\n\nThe goal of reducing sequential computation also forms the foundation of the Extended Neural GPU.']):
    Pdf.from_markdown(_t).save(str(_fx/f'p{_i}.pdf'))
Pdf.merge([str(_fx/'p0.pdf'), str(_fx/'p1.pdf')]).save(str(_fx/'two.pdf'))
_att = _fx/'two.pdf'
_src.write_bytes(pdf_doc(pdf_doc(_att).extract_pages_to_bytes([0, 1])).flatten_to_images(dpi=60))
_reads = iter([
    dict(items=[dict(kind='heading', text='अथ प्रथमोऽध्यायः'), dict(kind='verse', n='१', text='धर्मक्षेत्रे कुरुक्षेत्रे ।\nसमवेता युयुत्सवः ॥१॥'),
                dict(kind='verse', n=None, text='मामकाः पाण्डवाश्चैव ।')]),
    dict(items=[dict(kind='verse', n='॥२॥', text='किमकुर्वत सञ्जय ॥२॥'), dict(kind='colophon', text='इति प्रथमोऽध्यायः')])])
class Book:
    "Reads the two pages in order, whatever order the threads ask in."
    model, lock = 'fake/vision', __import__('threading').Lock()
    def __call__(self, msg):
        with Book.lock: return dict(content=json.dumps(next(_reads), ensure_ascii=False))
_b = read_book(_src, chat=Book(), workers=1)
test_eq(_b.pages, 2); test_eq(_b.doc_id, None); test_eq(_b.probs, [])
_c = _b.chapters[0]
test_eq((_c['heading'], _c['colophon']), ('अथ प्रथमोऽध्यायः', 'इति प्रथमोऽध्यायः'))
test_eq([v['n'] for v in _c['verses']], [1, 2]); assert _c['verses'][1]['text'].startswith('मामकाः')
_md = book_md(_b.chapters, 'Gita')
assert _md.startswith('# Gita') and '## अथ प्रथमोऽध्यायः' in _md and '*2*' in _md
test_eq(check_counts(_b.chapters, [3]), ['chapter 1: 2 verses, the plan has 3'])

from vishalakshi import Vault
_reads = iter([dict(items=[dict(kind='verse', n=1, text='क ॥१॥')]), dict(items=[dict(kind='verse', n=2, text='ध्ार्म ॥२॥')])])
_v = Vault(Path(tempfile.mkdtemp())/'v.db', offline=True)
_vb = read_book(_src, _v, chat=Book(), workers=1)
assert _vb.doc_id and len(_v.pages(_vb.doc_id)) == 2
test_eq([v['text'] for v in _vb.chapters[0]['verses']], ['क ॥१॥', 'धार्म ॥२॥'])   # repaired
test_eq((_vb.probs, _vb.cost, _vb.failed), (['1: 1.2: akṣaras mended'], 0, []))
# a cache: read again, nothing is sent and the vault's pages stay as they were
_cdir = Path(tempfile.mkdtemp())
_reads = iter([dict(items=[dict(kind='verse', n=1, text='क ॥१॥')]), dict(items=[dict(kind='verse', n=2, text='ख ॥२॥')])])
_cb = read_book(_src, _v, chat=Book(), workers=1, cache=_cdir)
test_eq(read_book(_src, _v, chat=Book(), workers=1, cache=_cdir).chapters, _cb.chapters)   # `_reads` is spent: a second ask would raise
test_eq(sorted(f.name for f in _cdir.iterdir()), ['p0001.json', 'p0002.json'])

# assembled again from the vault's pages after a proofread is accepted: its change is in the verses
class Says:
    def __init__(self, t): self.t, self.model = t, 'fake'
    def __call__(self, msg): return dict(content=json.dumps(dict(items=[dict(kind='verse', n=None, text=self.t)]), ensure_ascii=False))
_v.proofread(_cb.doc_id, ['a', 'b'], pages=[1], chats=dict(a=Says('ग ॥२॥'), b=Says('ग ॥२॥')))
_v.accept(_cb.doc_id)
_ab = assemble_book([dict(page=p.page, blocks=p.blocks) for p in _v.pages(_cb.doc_id)])
test_eq([(v['n'], v['text']) for v in _ab.chapters[0]['verses']], [(1, 'क ॥१॥'), (2, 'ग ॥२॥')])
# a verse read without its number takes the one it closes on; a maṅgala kept with `openers`; renumbered; the prose left out
_ps = [dict(page=1, blocks=[dict(kind='verse', text='श्रीगणेशाय नमः ॥'), dict(kind='verse', text='धर्मो रक्षति ॥ ௧ ॥'), dict(kind='prose', text='टीका'),
                            dict(kind='verse', text=''), dict(kind='verse', text='सत्यं वद ॥ ३ ॥')])]
_ab = assemble_book(_ps, kinds=('verse',), openers=True, renumber=True)
test_eq([v['n'] for v in _ab.chapters[0]['verses']], [None, 1, 3])   # ௧ read as 1; the gap is the print's, noted
test_eq([b['text'] for b in _ab.left], ['टीका']); assert any('3' in x for x in _ab.probs), _ab.probs

Other Formats

book_md

read_book

assemble_book