# book
[`read_book`](https://vedicreader.github.io/lekhaka/book.html#read_book)
takes a PDF or a page image of a book of verses and does the whole job.
It sends every page to the vision model `model` through vishalakshi’s
`read_doc`, with the reading prompt named by `preset` (`sanskrit` by
default) and any `hint` you add, such as “Each verse ends in ॥ n ॥.”.
The model returns each page as a list of blocks, the pieces it found on
the page: verses with their printed numbers, headings, colophons (the
line that closes a chapter, इति … अध्यायः) and prose.
[`read_book`](https://vedicreader.github.io/lekhaka/book.html#read_book)
then passes the pages to
[`assemble_book`](https://vedicreader.github.io/lekhaka/book.html#assemble_book),
described below.
If you pass a vishalakshi `vault`, the pages are stored there as
versions of one document, and the result’s `doc_id` names it. Without a
vault nothing is stored and `doc_id` is None. Other keyword arguments go
on to `read_doc`. The most useful is `cache`, a folder where each page’s
reading is saved as JSON, so that reading the book a second time sends
nothing to the model. `tries`, `progress`, `spread`, `strips` and the
rest work as they do in `read_doc`. `chat` replaces the model with a
function of your own, which is how the tests below run without one.
The result holds the `chapters`, the `probs`, the prose blocks `left`
out of the verses, the number of `pages`, the `cost` of the reading, and
the pages vishalakshi reports as `failed`, `unread` or `blocked`.
`probs` is a list of lines, one for every change lekhaka made and every
place a proofreader should look at the scan.
[`assemble_book`](https://vedicreader.github.io/lekhaka/book.html#assemble_book)
is the second half on its own, for pages that have already been read. A
common case is a vault whose pages other models have proofread, with the
corrections accepted: you take the pages back out of the vault and
assemble them again. It works in this order:
1. A verse block read without a number takes the number its text ends
on, if that is a number between ॥ marks (॥ १२ ॥). Blocks with no
text are removed, except table blocks, which carry `rows`.
2. `fixes`, a table of fix rows for
[`apply_fixes`](https://vedicreader.github.io/lekhaka/repairs.html#apply_fixes)
(see repairs), is applied.
3. With `cont=True`, blocks that a page break cut in two are joined
([`join_cont`](https://vedicreader.github.io/lekhaka/verses.html#join_cont)).
4. [`assemble`](https://vedicreader.github.io/lekhaka/verses.html#assemble)
builds the chapters, with `pair`, `kinds`, `notes`,
`number_from_note` and `openers` passed to it.
5. With `renumber=True`, each chapter goes through
[`renumber`](https://vedicreader.github.io/lekhaka/verses.html#renumber).
6. When `fix` is on, as it is by default, and `script` is Devanagari,
each verse with malformed akṣaras (syllables written in a form no
printer would set) is repaired with
[`fix_aksharas`](https://vedicreader.github.io/lekhaka/verses.html#fix_aksharas),
and the repair is listed in `probs`.
7. If you give `counts`, the number of verses expected in each chapter,
every chapter with a different number is listed in `probs`.
The prose blocks, which are not verse, come back in `left`.
[`read_book`](https://vedicreader.github.io/lekhaka/book.html#read_book)
takes the same options except `cont`, which only
[`assemble_book`](https://vedicreader.github.io/lekhaka/book.html#assemble_book)
has.
[`book_md`](https://vedicreader.github.io/lekhaka/book.html#book_md)
writes chapters out as markdown: a heading for each chapter, each verse
followed by its number in italics, and the colophon at the end.
------------------------------------------------------------------------
source
### book_md
``` python
def book_md(
chapters, title:str=''
)->str:
```
*Write chapters as markdown: a heading for each chapter, each verse
followed by its number, and the colophon at the end.*
------------------------------------------------------------------------
source
### read_book
``` python
def read_book(
path, vault:NoneType=None, model:str=None, preset:str='sanskrit', title:str='', script:str='Devanagari',
hint:str='', pages:NoneType=None, counts:NoneType=None, fix:bool=True, pair:bool=False,
kinds:tuple=('verse', None), notes:tuple=(), number_from_note:bool=False, fixes:NoneType=None, dpi:int=300,
workers:int=4, chat:NoneType=None, openers:bool=False, renumber:bool=False, **kw
)->fastcore.basics.AttrDict: # passed on to vishalakshi's `read_doc`: `cache`, `redo`, `clean`, `tries`, `spread`, `strips`, `progress`, `pii`
```
*Read a book with a vision model and return its chapters
`[{heading, colophon, verses}]`, the problems a proofreader should see,
and what the reading cost and left unread.*
------------------------------------------------------------------------
source
### assemble_book
``` python
def assemble_book(
pages, script:str='Devanagari', pair:bool=False, kinds:tuple=('verse', None), notes:tuple=(),
number_from_note:bool=False, fixes:NoneType=None, counts:NoneType=None, fix:bool=True, cont:bool=False,
openers:bool=False, renumber:bool=False
)->fastcore.basics.AttrDict:
```
*Assemble pages already read, `[{page, blocks}]`, into chapters
`[{heading, colophon, verses}]`, and return them with the problems a*
proofreader should see and the prose `left` out. A verse read without
its `n` takes the number its text ends on (`॥ १२ ॥`), and
`renumber=True` corrects each chapter’s numbers.
``` python
from pdflite import pdf_doc
_src = Path(tempfile.mkdtemp())/'two.pdf'
from pdf_oxide import Pdf
_fx = Path(tempfile.mkdtemp())
for _i, _t in enumerate(['# Attention Is All You Need\n\nThe dominant sequence transduction models are based on recurrent or convolutional networks.',
'# Background\n\nThe goal of reducing sequential computation also forms the foundation of the Extended Neural GPU.']):
Pdf.from_markdown(_t).save(str(_fx/f'p{_i}.pdf'))
Pdf.merge([str(_fx/'p0.pdf'), str(_fx/'p1.pdf')]).save(str(_fx/'two.pdf'))
_att = _fx/'two.pdf'
_src.write_bytes(pdf_doc(pdf_doc(_att).extract_pages_to_bytes([0, 1])).flatten_to_images(dpi=60))
_reads = iter([
dict(items=[dict(kind='heading', text='अथ प्रथमोऽध्यायः'), dict(kind='verse', n='१', text='धर्मक्षेत्रे कुरुक्षेत्रे ।\nसमवेता युयुत्सवः ॥१॥'),
dict(kind='verse', n=None, text='मामकाः पाण्डवाश्चैव ।')]),
dict(items=[dict(kind='verse', n='॥२॥', text='किमकुर्वत सञ्जय ॥२॥'), dict(kind='colophon', text='इति प्रथमोऽध्यायः')])])
class Book:
"Reads the two pages in order, whatever order the threads ask in."
model, lock = 'fake/vision', __import__('threading').Lock()
def __call__(self, msg):
with Book.lock: return dict(content=json.dumps(next(_reads), ensure_ascii=False))
_b = read_book(_src, chat=Book(), workers=1)
test_eq(_b.pages, 2); test_eq(_b.doc_id, None); test_eq(_b.probs, [])
_c = _b.chapters[0]
test_eq((_c['heading'], _c['colophon']), ('अथ प्रथमोऽध्यायः', 'इति प्रथमोऽध्यायः'))
test_eq([v['n'] for v in _c['verses']], [1, 2]); assert _c['verses'][1]['text'].startswith('मामकाः')
_md = book_md(_b.chapters, 'Gita')
assert _md.startswith('# Gita') and '## अथ प्रथमोऽध्यायः' in _md and '*2*' in _md
test_eq(check_counts(_b.chapters, [3]), ['chapter 1: 2 verses, the plan has 3'])
```
``` python
from vishalakshi import Vault
_reads = iter([dict(items=[dict(kind='verse', n=1, text='क ॥१॥')]), dict(items=[dict(kind='verse', n=2, text='ध्ार्म ॥२॥')])])
_v = Vault(Path(tempfile.mkdtemp())/'v.db', offline=True)
_vb = read_book(_src, _v, chat=Book(), workers=1)
assert _vb.doc_id and len(_v.pages(_vb.doc_id)) == 2
test_eq([v['text'] for v in _vb.chapters[0]['verses']], ['क ॥१॥', 'धार्म ॥२॥']) # repaired
test_eq((_vb.probs, _vb.cost, _vb.failed), (['1: 1.2: akṣaras mended'], 0, []))
# a cache: read again, nothing is sent and the vault's pages stay as they were
_cdir = Path(tempfile.mkdtemp())
_reads = iter([dict(items=[dict(kind='verse', n=1, text='क ॥१॥')]), dict(items=[dict(kind='verse', n=2, text='ख ॥२॥')])])
_cb = read_book(_src, _v, chat=Book(), workers=1, cache=_cdir)
test_eq(read_book(_src, _v, chat=Book(), workers=1, cache=_cdir).chapters, _cb.chapters) # `_reads` is spent: a second ask would raise
test_eq(sorted(f.name for f in _cdir.iterdir()), ['p0001.json', 'p0002.json'])
```
``` python
# assembled again from the vault's pages after a proofread is accepted: its change is in the verses
class Says:
def __init__(self, t): self.t, self.model = t, 'fake'
def __call__(self, msg): return dict(content=json.dumps(dict(items=[dict(kind='verse', n=None, text=self.t)]), ensure_ascii=False))
_v.proofread(_cb.doc_id, ['a', 'b'], pages=[1], chats=dict(a=Says('ग ॥२॥'), b=Says('ग ॥२॥')))
_v.accept(_cb.doc_id)
_ab = assemble_book([dict(page=p.page, blocks=p.blocks) for p in _v.pages(_cb.doc_id)])
test_eq([(v['n'], v['text']) for v in _ab.chapters[0]['verses']], [(1, 'क ॥१॥'), (2, 'ग ॥२॥')])
# a verse read without its number takes the one it closes on; a maṅgala kept with `openers`; renumbered; the prose left out
_ps = [dict(page=1, blocks=[dict(kind='verse', text='श्रीगणेशाय नमः ॥'), dict(kind='verse', text='धर्मो रक्षति ॥ ௧ ॥'), dict(kind='prose', text='टीका'),
dict(kind='verse', text=''), dict(kind='verse', text='सत्यं वद ॥ ३ ॥')])]
_ab = assemble_book(_ps, kinds=('verse',), openers=True, renumber=True)
test_eq([v['n'] for v in _ab.chapters[0]['verses']], [None, 1, 3]) # ௧ read as 1; the gap is the print's, noted
test_eq([b['text'] for b in _ab.left], ['टीका']); assert any('3' in x for x in _ab.probs), _ab.probs
```