Search finds documents. This page covers what is in them.
A doctype is the type of a document, such as an invoice, a receipt, a contract, a paper, meeting notes or a passport. There are 31 doctypes. The doctype is separate from the document’s kind, which records how the document arrived (web, pdf, note and so on).
The doctype decides two things. It decides the schema that extract uses, which is the list of fields that extract asks a model for. An invoice schema has a number, a vendor, a total and line items. It also decides the shelf that reshelf moves the document to.
varga holds the cue table that recognises each doctype, the schemas, and the code that makes a model answer in the form of a schema. This page stores the varga verdicts in the vault. It runs extraction through the vault’s model and its privacy checks.
categorize decides the doctype of a document and writes it into the document’s meta.
categorize first scores the document against the varga cue table. The cues are phrases such as “invoice no” or “total due”, amounts of money, dates and organisation names. score is the score of the winning doctype. decisive is True when the winning score is clearly higher than the runner-up.
categorize asks a model only when the cues are not decisive. llm= sets this:
llm= |
when categorize asks a model |
|---|---|
'auto' (default) |
When the cues are not decisive. A local model must already be downloaded, because categorize never starts a download. |
'always' |
Every time. |
'never' |
Never. |
With llm='auto', a model error leaves the cue verdict in place, and by names the error type. With llm='always', categorize raises the error.
The model receives the document’s title and text through send_check. A hosted model receives the title and text of a private document masked, or not at all. A file read off disk is judged on its name as well as its text. by says how the verdict was reached.
When the cues are decisive, or a model gave the verdict, categorize writes it as doctype. An undecided verdict with a score of at least REVIEW_SCORE (0.2) is only proposed. categorize writes it as doctype_proposed, and a person confirms it. proposed_types lists the proposals, surest first. accept_type, accept_types and reject_type settle them. suggest_noisy and accept_noisy use the same two-step review for noise.
Below 0.2, categorize writes nothing. reshelf never moves a document on a proposal. categorize_all types every untyped document on the shelf.
List the entities one document names, such as organisations, people, places and products, with their counts.
def reshelf(
ref, # doc_id, source, or a title substring
shelf:str=None, # where to put it; None -> whatever its doctype says
llm:str='auto', # the LLM leg of the categorisation that picks the shelf
max_chars:int=2000000, # longest text carried over; a longer document is refused
**kw
)->fastcore.basics.AttrDict: # forwarded to categorize (model=, chat_kw=, ner=)Move a document to shelf, or to the shelf its doctype belongs on: papers to papers, code to code, catalogues to data.
Every document categorised as doctype, newest first.
Count this shelf’s documents by doctype, with untyped for those that have none.
Confirm the k surest proposals. Confirm all of them when k is None.
Remove the proposed doctype of a document. The document stays untyped.
Confirm the proposed doctype of a document, or set doctype instead.
Return the documents with a proposed doctype that a person has not confirmed, surest first.
Categorise every document on this shelf that has no doctype yet, and count the results by doctype.
def categorize(
ref, # doc_id, source, title, a path on disk, or a loaded `document()`
model:str=None, # an id, a path, `mlx/…`; None -> $VISHALAKSHI_MODEL
chat_kw:dict=None, # anything else Urai's `Chat` takes: temp, runtime, think, …
llm:str='auto', # 'auto' -> only when the cues cannot decide | 'always' | 'never'
labels:str=None, # comma-separated labels to choose from; None -> DOCTYPES
max_chars:int=6000, # chars of the document the model and the cues see
ner:bool=True, # run entity extraction alongside the regex signals
save:bool=True, # write the verdict into the document's meta
review:float=0.2, # an undecided verdict at or above this is proposed, not written
pii:str='redact', # off|local|redact|refuse: what a hosted model gets of a private document, via `send_check`
)->fastcore.basics.AttrDict:Decide a document’s doctype (invoice, contract, paper and so on) from varga’s cues, asking a model only when the cues cannot decide.
Return True when model mid is in the Hugging Face cache. The check scans the cache and never downloads.
extract reads a whole document, up to max_chars, and returns its fields as a dict.
Without a schema, extract categorises the document first and uses the schema of its doctype. An invoice uses Invoice. A doctype with no schema of its own uses Summary. A schema can also be a SCHEMAS key, a dataclass, or a string such as 'vendor:str, total:float'.
The model answers in the form of the schema through a tool call, a grammar or parsed JSON. The backend decides which of the three is used.
check=True reads every value back through check_fields. errors lists the values that did not fit. src gives the page and block that each field was read from. cite=True adds the quote.
extract_all(doctype='invoice') runs over every invoice and returns one row per document.
Privacy works as in ask, with pii='local' as the default here. send_check makes the decision:
pii_model or by no model.redact masks the title, source and text of a private document for a hosted model.refuse reads nothing.off included.extract writes each hosted send to the send log. It scans and masks the fields read from a private document before it returns or saves them.
def extract_all(
doctype:str=None, # only documents already categorised as this
kind:str=None, # only these KINDS
schema:NoneType=None, # one shape for all of them; None -> each document's own doctype
limit:int=None, # stop after this many
**kw
)->fastcore.basics.AttrDict: # forwarded to extract (model=, chat_kw=, max_chars=, save=, sp=)Run extract over many documents, such as every invoice, and return one row per document and a list of errors.
def extract(
ref:str, # doc_id, source, title substring, or a path on disk
schema:NoneType=None, # a SCHEMAS key, a dataclass, or a 'field:type, …' spec; None -> from the doctype
model:str=None, # an id, a path, `mlx/…`; None -> $VISHALAKSHI_MODEL
chat_kw:dict=None, # anything else Urai's `Chat` takes: temp, runtime, think, …
max_chars:int=8000, # chars of the document the model sees
sp:str='You pull structured fields out of one document.\n\nRules:\n- Copy what the document states. Do not compute, convert, round or tidy a value.\n- A field the document does not state stays empty. An invented number is worse than a blank one.\n- Numbers are bare: 1240.50, not "$1,240.50". The currency belongs in its own field.\n- Dates are ISO: 2024-03-01.\n- A list field takes every entry the document has, in the order it has them.', # system prompt for the extraction
save:bool=False, # write the fields into the document's meta
llm:str='auto', # the LLM leg of the categorisation, when picking the schema
pii:str='local', # local|redact|refuse|off, as in `ask`; a Local only document reaches no hosted model even under `off`
pii_model:str=None, # local model when private; None -> $VISHALAKSHI_PII_MODEL
check:bool=True, # read the values back through `check_fields` and report what did not
cite:bool=False, # ask for the quote each value was read from; `src` carries it
)->fastcore.basics.AttrDict:Pull structured fields out of one document, such as an invoice’s number, vendor and totals, in the shape of a schema.
ask_doc(ref, question) is ask with ref=. The document is section [1], whole up to max_chars. related sections from the rest of the vault follow it. schema= returns the answer as fields instead of prose. The privacy rules are the rules of ask.
def ask_doc(
ref:str, # doc_id, source, title substring, or a path on disk
question:str, # what you want to know about it
schema:NoneType=None, # answer as this shape instead of prose
max_chars:int=8000, # chars of the document the model sees
related:int=3, # sections from the *rest* of the vault to add as context
**kw
)->fastcore.basics.AttrDict: # forwarded to `ask` (model=, chat_kw=, sp=)Answer a question about one whole document: ask with that document as section [1].
The cue table recognises these two documents without a model. The schemas name the fields that each document gives.
from vishalakshi import Vault
INVOICE = '''# INVOICE
Invoice No: ACM-2024-0117
Date: 2024-03-01
Payment terms: Net 30
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd
## Line items
| Description | Qty | Unit price | Amount |
|---|---|---|---|
| Widget, steel | 12 | $8.50 | $102.00 |
| Gasket, nitrile | 40 | $1.20 | $48.00 |
Subtotal: $150.00
VAT (20%): $30.00
Total due: $180.00
'''
CATALOGUE = '''# Spring price list
## Widget, steel
SKU: WS-100. Price: $8.50 per unit. In stock.
Specifications: 40mm, zinc plated. Dimensions 40x12x4mm.
## Gasket, nitrile
SKU: GN-220. Price: $1.20 per unit. Out of stock.
Add to cart to be notified.
'''
v = Vault(':memory:')
v.add(INVOICE, 'Acme invoice ACM-2024-0117', source='/inbox/acme-0117.md')
v.add(CATALOGUE, 'Spring price list', source='/inbox/spring-list.md')
v.doctypes(){'untyped': 2}('invoice', 1.0, True, 'cues (ner+regex)')test_eq(r.doctype, 'invoice')
assert r.decisive, r.scores # the cues are sure enough that no model is needed
test_eq(v.doc(r.doc_id)['meta']['doctype'], 'invoice') # written back into the vault
test_eq(v.categorize('Spring price list', llm='never').doctype, 'catalogue')
test_eq(v.doctypes(), {'invoice': 1, 'catalogue': 1})
test_eq(v.of_type('invoice').attrgot('title'), ['Acme invoice ACM-2024-0117'])The cells above need no model and no API key. Pass model= and chat_kw= to select the model and its settings, as for ask. The hidden cells below replay answers recorded from a local model. evals/ has the measurements.