# proof


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

A vision model’s reading of a page can contain slips. Other models,
called readers on this page, can read the same pages again and compare
their readings with the stored text. Each difference is filed as a
change. A change is a run of words to replace, insert or delete, with
the names of the readers who made it. Nothing in the document changes
until a person decides. This page calls that person the owner.

`Vault.proofread(doc_id, readers)` has each reader read the pages, then
files the changes. A change that at least `agree` readers (2 by default)
make identically has the state `proposed`. A change that fewer readers
make has the state `doubt`. A reader named twice counts once. The owner
then works through the review queue with these methods:

<table>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr>
<th>method</th>
<th>what it does</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>changes(doc_id, state)</code></td>
<td>lists the changes, all of them or those in one state:
<code>proposed</code>, <code>doubt</code>, <code>accepted</code> or
<code>rejected</code></td>
</tr>
<tr>
<td><code>accept(doc_id, ids)</code></td>
<td>applies changes, all the <code>proposed</code> ones when
<code>ids</code> is None, as a new version of each page they touch
(route <code>proof</code>), and re-indexes the document</td>
</tr>
<tr>
<td><code>reject(doc_id, ids)</code></td>
<td>sets changes aside. They stay in the record as not taken</td>
</tr>
<tr>
<td><code>edit_change(doc_id, id, after)</code></td>
<td>rewrites the proposed text of a change that is still open</td>
</tr>
<tr>
<td><code>undo_decision(doc_id, ids)</code></td>
<td>takes a decision back. A rejected change returns to the state it
had. The page of an accepted change returns to its text before the
accept, and every change that the accept applied is
<code>proposed</code> again</td>
</tr>
<tr>
<td><code>revert_page(doc_id, page, version)</code></td>
<td>files any old version of a page again as its newest version (route
<code>revert</code>)</td>
</tr>
</tbody>
</table>

A change filed against an older version of its page cannot be accepted.
`accept` raises an error that asks for a new proofread. A person can
undo an accept only while its page has not changed since. `edit_change`,
`undo_decision` and `revert_page` are for a person only. They raise
`PermissionError` when called with `by='agent'`.

`proof_md` writes the record of a proofread as markdown, page by page.
It lists what changed, what is proposed, what is in doubt and what was
not taken.

## Word changes

`word_changes(a, b)` compares two texts word by word. It returns each
run of words that `b` replaces, inserts or deletes, as
`{at, n, before, after}`. `at` is the index of the first word in `a`.
`n` is the number of words of `a` that the change replaces (0 for an
insertion). `before` and `after` are the words themselves.

[`apply_changes`](https://vedicreader.github.io/vishalakshi/proof.html#apply_changes)
applies a list of such changes from the last to the first. It keeps the
spacing and line breaks of the text everywhere else.

A stored page also has blocks, and the blocks must match its text.
[`block_changes`](https://vedicreader.github.io/vishalakshi/proof.html#block_changes)
applies each change to the block that holds its words. It splits a
change that crosses the end of a block. The cells of a table and the
`##` of a heading stay as read.
[`change_box`](https://vedicreader.github.io/vishalakshi/proof.html#change_box)
finds the `box` of the block that holds a change. A review screen uses
it to show the change on the page image.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L54"
target="_blank" style="float:right; font-size:smaller">source</a>

### block_changes

``` python
def block_changes(
    blocks, text:str, changes
)->list | None:
```

*Make `changes` to `text` in the blocks their words fall in, splitting a
change at a block’s end, or return None when the blocks do not read as
`text`.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L36"
target="_blank" style="float:right; font-size:smaller">source</a>

### apply_changes

``` python
def apply_changes(
    t:str, changes
)->str:
```

*Make `changes` in `t`, each replacing `n` words from word `at` with
`after`, and keep the text’s own spacing elsewhere.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L30"
target="_blank" style="float:right; font-size:smaller">source</a>

### change_box

``` python
def change_box(
    blocks, text:str, at:int
):
```

*The `box` of the block that word `at` of `text` falls in, the last
block for a position past the end, or None when the blocks do not read
as `text`.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L24"
target="_blank" style="float:right; font-size:smaller">source</a>

### word_changes

``` python
def word_changes(
    a:str, b:str
)->list:
```

*Compare `a` and `b` word by word and return each run of words `b`
changes, as `{at, n, before, after}`.*

``` python
_a, _b = 'धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवः', 'धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवाः ॥१॥'
_c = word_changes(_a, _b)
test_eq(_c, [dict(at=3, n=1, before='युयुत्सवः', after='युयुत्सवाः ॥१॥')])
test_eq(apply_changes(_a, _c), _b)
test_eq(apply_changes('a b c', word_changes('a b c', 'a c')), 'a c')
test_eq(apply_changes('a c', word_changes('a c', 'a b c')), 'a b c')
test_eq(apply_changes('line one\nline two', word_changes('line one\nline two', 'line 1\nline two')), 'line 1\nline two')
_bl = [dict(kind='heading', text='Title here', box=[0, 0, 1, .1]), dict(kind='prose', text='one two three', box=[0, .2, 1, .4]), dict(kind='prose', text='four')]
_tx = _md(_bl)
test_eq([change_box(_bl, _tx, i) for i in (0, 2, 3, 5, 6, 9)], [[0, 0, 1, .1], [0, 0, 1, .1], [0, .2, 1, .4], [0, .2, 1, .4], None, None])   # the heading is 3 words; past the end is the last block
test_eq((change_box(_bl, 'other words', 0), change_box(None, _tx, 0)), (None, None))
for x, y in [('', 'new words'), ('a', ''), ('x y z', 'z y x'), ('one two', 'one two three')]: test_eq(apply_changes(x, word_changes(x, y)), y if x or y else '')
test_eq([b['text'] for b in block_changes(_bl, _tx, word_changes(_tx, '## Title there\n\none 2 three\n\nfour'))], ['Title there', 'one 2 three', 'four'])
test_eq([b['text'] for b in block_changes(_bl, _tx, word_changes(_tx, '## Title here\n\none two 3 4'))], ['Title here', 'one two 3 4', ''])   # a run across a block's end splits there
test_eq([b['text'] for b in block_changes(_bl, _tx, word_changes(_tx, '## Title here\n\none two three\n\nfour five'))], ['Title here', 'one two three', 'four five'])
test_eq(block_changes(_bl, _tx, word_changes(_tx, 'Title here one two three four'))[0]['text'], 'Title here')   # the ## goes from the text, the heading stays
_tb = [dict(kind='table', rows=[['a', 'b']]), dict(kind='prose', text='after it')]
test_eq(block_changes(_tb, _md(_tb), word_changes(_md(_tb), _md(_tb).replace('a', 'x', 1).replace('after', 'later'))), [_tb[0], dict(kind='prose', text='later it')])
test_eq(block_changes(_bl, 'not the blocks', []), None)
```

## Proofreading

`readers` are model ids. `chats` maps a reader to a stand-in callable.
The tests use it to run without models.

Each reader reads the page image through
[`read_page`](https://vedicreader.github.io/vishalakshi/read.html#read_page),
so the privacy rules are the rules of
[`read_page`](https://vedicreader.github.io/vishalakshi/read.html#read_page).
A reader that may not see a page skips it, and `blocked` reports the
skip. A hosted reader may not see a Personal page without `allow=True`.
No hosted reader sees a Local only page. The vault writes each hosted
reading to the send log.

A second run of `proofread` replaces the open changes (`proposed` and
`doubt`) on each page it reads. It leaves decided changes alone. A
change keeps the `box` of the block that holds it when the blocks of the
page have boxes.

`progress(done, total, page, route, cost)` is called once with `done=0`
and again after each page. If it returns False, the rest of the pages
are not read. The result carries `cancelled`, `unread` and the summed
`cost`.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L225"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.proof_md

``` python
def proof_md(
    doc_id:str
)->str:
```

*The record of a document’s proofreading as markdown, by page: what
changed, what is proposed or in doubt, and what was not taken.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L203"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.undo_decision

``` python
def undo_decision(
    doc_id:str, ids, by:str='person'
)->dict:
```

*Take decisions back: a rejected change returns to its old state, and an
accepted change’s page returns to its text before the accept, with that
accept’s changes `proposed` again; only a person may.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L194"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.revert_page

``` python
def revert_page(
    doc_id:str, page:int, version:int, by:str='person'
)->dict:
```

*File an old `version` of a page as its newest version, route `revert`,
and re-index the document; only a person may.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L175"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.edit_change

``` python
def edit_change(
    doc_id:str, id:str, after:str, by:str='person'
)->dict:
```

*Rewrite the proposed text of a change that is still open; only a person
may.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L147"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.accept

``` python
def accept(
    doc_id:str, ids:NoneType=None
)->dict:
```

*Apply changes, all the `proposed` ones when `ids` is None, as a new
version of each page they touch, with its blocks changed to match, and
re-index the document.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L136"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.reject

``` python
def reject(
    doc_id:str, ids
)->int:
```

*Set this document’s open changes aside as not taken, remembering the
state each had so that `undo_decision` can restore it, and return how
many were set aside.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L128"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.changes

``` python
def changes(
    doc_id:str, state:str=None, page:int=None
)->fastcore.foundation.L:
```

*List a document’s filed changes in page and word order, or only those
in one `state`: `proposed`, `doubt`, `accepted` or `rejected`.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/proof.py#L84"
target="_blank" style="float:right; font-size:smaller">source</a>

### Vault.proofread

``` python
def proofread(
    doc_id:str, readers, pages:NoneType=None, agree:int=2, preset:str=None, script:str=None, hint:str=None,
    dpi:int=200, chats:dict=None,
    pii:str='redact', # what `read_page` applies: a reader that may not see a page is skipped for it and reported in `blocked`
    allow:bool=False, # allow once, for Personal pages
    progress:NoneType=None, # `progress(done, total, page, route, cost)` after each page; False stops the rest
)->dict:
```

*Have each of `readers` read the pages again and file their word changes
to the current text, `proposed` where at least `agree` readers make the
same one.*

``` python
from vishalakshi.read import read_doc
from pdflite import pdf_doc
class Says:
    "A reader that reads every page as `text`."
    def __init__(self, text): self.text, self.model = text, 'fake'
    def __call__(self, msg): return dict(content=json.dumps(dict(items=[dict(kind='prose', text=self.text)])))
_d = Path(tempfile.mkdtemp())
from pdf_oxide import Pdf
_fx = Path(tempfile.mkdtemp())
for _i, _t in enumerate(['# Attention Is All You Need\n\nThe dominant sequence transduction models are based on recurrent or convolutional networks.',
                         '# Background\n\nThe goal of reducing sequential computation also forms the foundation of the Extended Neural GPU.']):
    Pdf.from_markdown(_t).save(str(_fx/f'p{_i}.pdf'))
Pdf.merge([str(_fx/'p0.pdf'), str(_fx/'p1.pdf')]).save(str(_fx/'two.pdf'))
_src = _fx/'two.pdf'
_scan = _d/'scan.pdf'; _scan.write_bytes(pdf_doc(pdf_doc(_src).extract_pages_to_bytes([0])).flatten_to_images(dpi=72))
_v = Vault(Path(tempfile.mkdtemp())/'v.db', offline=True)
_o = _v.read_doc(_scan, parse='llm', chat=Says('the dominat sequence transduction models'))
_id = _o['doc_id']
_r = _v.proofread(_id, ['a', 'b', 'c'], chats=dict(a=Says('the dominant sequence transduction models'), b=Says('the dominant sequence transduction models'),
                                                   c=Says('the dominat sequence transduktion models')))
test_eq(_r['changes'], 2); test_eq(_r['proposed'], 1)
_p = _v.changes(_id, 'proposed')[0]; test_eq((_p.before, _p.after, _p.by), ('dominat', 'dominant', ['a', 'b']))
_dt = _v.changes(_id, 'doubt')[0]; test_eq((_dt.after, _dt.by), ('transduktion', ['c']))
test_eq(_v.reject(_id, [_dt.id]), 1)
test_eq(_v.accept(_id)['accepted'], 1)
test_eq(_v.reject(_id, [_p.id, 'nope']), 0)   # an accepted change and an unknown id are not set aside
test_eq(_v.pages(_id)[0].text, 'the dominant sequence transduction models'); test_eq(_v.pages(_id)[0].route, 'proof')
assert 'dominant' in _v.search('dominant sequence')[0]['snippet']
_md_ = _v.proof_md(_id)
assert '## Changed' in _md_ and 'dominat → dominant (a, b)' in _md_ and '## Not taken' in _md_
for _i, _a in enumerate('xy'): _v._proof_t().insert(dict(id=f'ov{_i}', doc_id=_id, store=_v.name, page=0, version=_v.pages(_id)[0].version, at=1 + _i, n=2, before='', after=_a, by='["a", "b"]', state='proposed', ts=0.))
test_fail(lambda: _v.accept(_id, ['ov0', 'ov1']), contains='overlap')   # overlapping ranges are refused before any page changes
test_eq(_v.pages(_id)[0].text, 'the dominant sequence transduction models')
_v.proofread(_id, ['a'], chats=dict(a=Says('the dominant sequence transduction model')))
_v.reparse(_id, chat=Says('a different reading entirely'))
test_fail(lambda: _v.accept(_id, [c.id for c in _v.changes(_id, 'doubt')]), contains='older version')
```