# extract
Search finds documents. This page covers what is in them.
A doctype is the type of a document, such as an invoice, a receipt, a
contract, a paper, meeting notes or a passport. There are 31 doctypes.
The doctype is separate from the document’s `kind`, which records how
the document arrived (`web`, `pdf`, `note` and so on).
The doctype decides two things. It decides the schema that `extract`
uses, which is the list of fields that `extract` asks a model for. An
invoice schema has a number, a vendor, a total and line items. It also
decides the shelf that `reshelf` moves the document to.
[varga](https://vedicreader.github.io/varga/) holds the cue table that
recognises each doctype, the schemas, and the code that makes a model
answer in the form of a schema. This page stores the varga verdicts in
the vault. It runs extraction through the vault’s model and its privacy
checks.
`categorize` decides the doctype of a document and writes it into the
document’s `meta`.
`categorize` first scores the document against the varga cue table. The
cues are phrases such as “invoice no” or “total due”, amounts of money,
dates and organisation names. `score` is the score of the winning
doctype. `decisive` is True when the winning score is clearly higher
than the runner-up.
`categorize` asks a model only when the cues are not decisive. `llm=`
sets this:
llm=
when categorize asks a model
'auto' (default)
When the cues are not decisive. A local model must already be
downloaded, because categorize never starts a
download.
'always'
Every time.
'never'
Never.
With `llm='auto'`, a model error leaves the cue verdict in place, and
`by` names the error type. With `llm='always'`, `categorize` raises the
error.
The model receives the document’s title and text through `send_check`. A
hosted model receives the title and text of a private document masked,
or not at all. A file read off disk is judged on its name as well as its
text. `by` says how the verdict was reached.
When the cues are decisive, or a model gave the verdict, `categorize`
writes it as `doctype`. An undecided verdict with a score of at least
`REVIEW_SCORE` (0.2) is only proposed. `categorize` writes it as
`doctype_proposed`, and a person confirms it. `proposed_types` lists the
proposals, surest first. `accept_type`, `accept_types` and `reject_type`
settle them. `suggest_noisy` and `accept_noisy` use the same two-step
review for noise.
Below 0.2, `categorize` writes nothing. `reshelf` never moves a document
on a proposal. `categorize_all` types every untyped document on the
shelf.
------------------------------------------------------------------------
source
### Vault.ner
``` python
def ner(
ref:str, # doc_id, source, title substring, or a path on disk
limit:int=40, # entities returned
max_chars:int=20000
)->fastcore.basics.AttrDict:
```
*List the entities one document names, such as organisations, people,
places and products, with their counts.*
------------------------------------------------------------------------
source
### Vault.reshelf
``` python
def reshelf(
ref, # doc_id, source, or a title substring
shelf:str=None, # where to put it; None -> whatever its doctype says
llm:str='auto', # the LLM leg of the categorisation that picks the shelf
max_chars:int=2000000, # longest text carried over; a longer document is refused
**kw
)->fastcore.basics.AttrDict: # forwarded to categorize (model=, chat_kw=, ner=)
```
*Move a document to `shelf`, or to the shelf its doctype belongs on:
papers to `papers`, code to `code`, catalogues to `data`.*
------------------------------------------------------------------------
source
### Vault.of_type
``` python
def of_type(
doctype:str, kind:str=None
)->fastcore.foundation.L:
```
*Every document categorised as `doctype`, newest first.*
------------------------------------------------------------------------
source
### Vault.doctypes
``` python
def doctypes(
kind:str=None
)->dict:
```
*Count this shelf’s documents by doctype, with `untyped` for those that
have none.*
------------------------------------------------------------------------
source
### Vault.accept_types
``` python
def accept_types(
k:int=None, kind:str=None
)->fastcore.foundation.L:
```
*Confirm the `k` surest proposals. Confirm all of them when `k` is
None.*
------------------------------------------------------------------------
source
### Vault.reject_type
``` python
def reject_type(
ref
)->dict:
```
*Remove the proposed doctype of a document. The document stays untyped.*
------------------------------------------------------------------------
source
### Vault.accept_type
``` python
def accept_type(
ref, doctype:str=None
)->dict:
```
*Confirm the proposed doctype of a document, or set `doctype` instead.*
------------------------------------------------------------------------
source
### Vault.proposed_types
``` python
def proposed_types(
kind:str=None
)->fastcore.foundation.L:
```
*Return the documents with a proposed doctype that a person has not
confirmed, surest first.*
------------------------------------------------------------------------
source
### Vault.categorize_all
``` python
def categorize_all(
kind:str=None, # restrict to one or more KINDS
force:bool=False, # re-type documents that already carry a doctype
limit:int=None, # stop after this many
**kw
)->fastcore.basics.AttrDict: # forwarded to categorize (model=, chat_kw=, llm=, ner=, max_chars=)
```
*Categorise every document on this shelf that has no doctype yet, and
count the results by doctype.*
------------------------------------------------------------------------
source
### Vault.categorize
``` python
def categorize(
ref, # doc_id, source, title, a path on disk, or a loaded `document()`
model:str=None, # an id, a path, `mlx/…`; None -> $VISHALAKSHI_MODEL
chat_kw:dict=None, # anything else Urai's `Chat` takes: temp, runtime, think, …
llm:str='auto', # 'auto' -> only when the cues cannot decide | 'always' | 'never'
labels:str=None, # comma-separated labels to choose from; None -> DOCTYPES
max_chars:int=6000, # chars of the document the model and the cues see
ner:bool=True, # run entity extraction alongside the regex signals
save:bool=True, # write the verdict into the document's meta
review:float=0.2, # an undecided verdict at or above this is proposed, not written
pii:str='redact', # off|local|redact|refuse: what a hosted model gets of a private document, via `send_check`
)->fastcore.basics.AttrDict:
```
*Decide a document’s doctype (invoice, contract, paper and so on) from
varga’s cues, asking a model only when the cues cannot decide.*
------------------------------------------------------------------------
source
### model_cached
``` python
def model_cached(
mid:str
)->bool:
```
*Return True when model `mid` is in the Hugging Face cache. The check
scans the cache and never downloads.*
## Extraction
`extract` reads a whole document, up to `max_chars`, and returns its
fields as a dict.
Without a `schema`, `extract` categorises the document first and uses
the schema of its doctype. An invoice uses `Invoice`. A doctype with no
schema of its own uses `Summary`. A schema can also be a `SCHEMAS` key,
a dataclass, or a string such as `'vendor:str, total:float'`.
The model answers in the form of the schema through a tool call, a
grammar or parsed JSON. The backend decides which of the three is used.
`check=True` reads every value back through `check_fields`. `errors`
lists the values that did not fit. `src` gives the page and block that
each field was read from. `cite=True` adds the quote.
`extract_all(doctype='invoice')` runs over every invoice and returns one
row per document.
Privacy works as in `ask`, with `pii='local'` as the default here.
`send_check` makes the decision:
- A private document is read by the local `pii_model` or by no model.
- `redact` masks the title, source and text of a private document for a
hosted model.
- `refuse` reads nothing.
- A Local only document goes to no hosted model under any policy, `off`
included.
- A model on this machine reads the document whole.
`extract` writes each hosted send to the send log. It scans and masks
the fields read from a private document before it returns or saves them.
------------------------------------------------------------------------
source
### Vault.extract_all
``` python
def extract_all(
doctype:str=None, # only documents already categorised as this
kind:str=None, # only these KINDS
schema:NoneType=None, # one shape for all of them; None -> each document's own doctype
limit:int=None, # stop after this many
**kw
)->fastcore.basics.AttrDict: # forwarded to extract (model=, chat_kw=, max_chars=, save=, sp=)
```
*Run `extract` over many documents, such as every invoice, and return
one row per document and a list of errors.*
------------------------------------------------------------------------
source
### Vault.extract
``` python
def extract(
ref:str, # doc_id, source, title substring, or a path on disk
schema:NoneType=None, # a SCHEMAS key, a dataclass, or a 'field:type, …' spec; None -> from the doctype
model:str=None, # an id, a path, `mlx/…`; None -> $VISHALAKSHI_MODEL
chat_kw:dict=None, # anything else Urai's `Chat` takes: temp, runtime, think, …
max_chars:int=8000, # chars of the document the model sees
sp:str='You pull structured fields out of one document.\n\nRules:\n- Copy what the document states. Do not compute, convert, round or tidy a value.\n- A field the document does not state stays empty. An invented number is worse than a blank one.\n- Numbers are bare: 1240.50, not "$1,240.50". The currency belongs in its own field.\n- Dates are ISO: 2024-03-01.\n- A list field takes every entry the document has, in the order it has them.', # system prompt for the extraction
save:bool=False, # write the fields into the document's meta
llm:str='auto', # the LLM leg of the categorisation, when picking the schema
pii:str='local', # local|redact|refuse|off, as in `ask`; a Local only document reaches no hosted model even under `off`
pii_model:str=None, # local model when private; None -> $VISHALAKSHI_PII_MODEL
check:bool=True, # read the values back through `check_fields` and report what did not
cite:bool=False, # ask for the quote each value was read from; `src` carries it
)->fastcore.basics.AttrDict:
```
*Pull structured fields out of one document, such as an invoice’s
number, vendor and totals, in the shape of a schema.*
## Asking about one whole document
`ask_doc(ref, question)` is `ask` with `ref=`. The document is section
\[1\], whole up to `max_chars`. `related` sections from the rest of the
vault follow it. `schema=` returns the answer as fields instead of
prose. The privacy rules are the rules of `ask`.
------------------------------------------------------------------------
source
### Vault.ask_doc
``` python
def ask_doc(
ref:str, # doc_id, source, title substring, or a path on disk
question:str, # what you want to know about it
schema:NoneType=None, # answer as this shape instead of prose
max_chars:int=8000, # chars of the document the model sees
related:int=3, # sections from the *rest* of the vault to add as context
**kw
)->fastcore.basics.AttrDict: # forwarded to `ask` (model=, chat_kw=, sp=)
```
*Answer a question about one whole document: `ask` with that document as
section \[1\].*
## Try it
The cue table recognises these two documents without a model. The
schemas name the fields that each document gives.
``` python
from vishalakshi import Vault
INVOICE = '''# INVOICE
Invoice No: ACM-2024-0117
Date: 2024-03-01
Payment terms: Net 30
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd
## Line items
| Description | Qty | Unit price | Amount |
|---|---|---|---|
| Widget, steel | 12 | $8.50 | $102.00 |
| Gasket, nitrile | 40 | $1.20 | $48.00 |
Subtotal: $150.00
VAT (20%): $30.00
Total due: $180.00
'''
CATALOGUE = '''# Spring price list
## Widget, steel
SKU: WS-100. Price: $8.50 per unit. In stock.
Specifications: 40mm, zinc plated. Dimensions 40x12x4mm.
## Gasket, nitrile
SKU: GN-220. Price: $1.20 per unit. Out of stock.
Add to cart to be notified.
'''
v = Vault(':memory:')
v.add(INVOICE, 'Acme invoice ACM-2024-0117', source='/inbox/acme-0117.md')
v.add(CATALOGUE, 'Spring price list', source='/inbox/spring-list.md')
v.doctypes()
```
{'untyped': 2}
``` python
r = v.categorize('Acme invoice ACM-2024-0117', llm='never')
r.doctype, r.score, r.decisive, r.by
```
('invoice', 1.0, True, 'cues (ner+regex)')
``` python
test_eq(r.doctype, 'invoice')
assert r.decisive, r.scores # the cues are sure enough that no model is needed
test_eq(v.doc(r.doc_id)['meta']['doctype'], 'invoice') # written back into the vault
test_eq(v.categorize('Spring price list', llm='never').doctype, 'catalogue')
test_eq(v.doctypes(), {'invoice': 1, 'catalogue': 1})
test_eq(v.of_type('invoice').attrgot('title'), ['Acme invoice ACM-2024-0117'])
```
The cells above need no model and no API key. Pass `model=` and
`chat_kw=` to select the model and its settings, as for `ask`. The
hidden cells below replay answers recorded from a local model. `evals/`
has the measurements.