core

the vault: one SQLite file holding everything you have read, and the retrieval over it

A Vault is a litesearch document tree — docs → nodes → chunks, FTS5 and a usearch HNSW index — with an entity graph over it, in one SQLite file. Everything you read goes in under a kind, and one query crosses all of them. Nothing here reimplements litesearch: the vault holds an encoder and a store name, and hands both to it.

kinds

def kinds(
    kind
)->L:

A kind filter as a list — 'note', 'note,web' and ['note','web'] all work.

tidy_bc

def tidy_bc(
    bc:str
)->str:

Drop build_tree’s Pages n–m: window placeholders from a breadcrumb: real nodes, noise in a citation.

mk_encoder

def mk_encoder(
    model:str=None, # model2vec/HF id; None -> the retrieval default
    dims:int=256, # dims for the hashing fallback only
    offline:bool=False, # skip the download attempt entirely
    dtype:type=float16, # stored width; litesearch's default everywhere
)->AttrDict:

The best encoder available as AttrDict(model, doc, query, dtype, dims, method, note).

Degrades to litesearch’s hash_embed rather than failing, and always says which answered: the gap between the two is the gap between a vault that answers questions and one that can only keyword-match. model is the embedder object itself, so kosha can share it (see code).

Both encoders are cast to dtype because litesearch’s own default is float16 and not every entry point takes a dtype=: Database.context does not thread one down to its section search, so a float32 store would be read back as float16 there — the vectors survive, the distances do not, and context() silently degrades to keyword ranking. Half precision costs nothing at these magnitudes; a mismatched width costs the whole semantic leg.

Vault

def Vault(
    path:str=None, # vault file; None -> ~/.vishalakshi/vault.db
    encoder:NoneType=None, # an mk_encoder() AttrDict, a model id, or None
    store:str='store', # chunk store name
    offline:bool=False, # never attempt a model download
    dims:int=256, # dims for the hashing fallback
):

Everything you have read, in one SQLite file, searchable as one corpus.

Web pages, PDFs, papers, transcripts, local files, code and your own notes land in the same litesearch store under different kinds, which is the point: one query crosses all of them and context() hands back sections rather than fragments. Acquisition lives in vishalakshi.acquire, answering in .ask, code in .code — all optional, since the vault itself needs neither a network nor an LLM.

Vault.note

def note(
    text:str, # what you want to remember
    title:str=None, # defaults to the first line
    tags:list=None, # free-form tags, kept in the doc's meta
)->dict:

Write a note into the vault so it is searched alongside the corpus.

Notes are ordinary documents with kind='note', which is deliberate: the graph, the clusters and context() all see them for free, so what you concluded about a corpus comes back next to the evidence you concluded it from.

Vault.add_dir

def add_dir(
    dir:str, types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb', kind:str=None, **kw
)->L:

Ingest every document under a directory. dir2files skips dotfiles, tests, build and dist.

Vault.add_file

def add_file(
    path:str, title:str=None, kind:str=None, **kw
)->dict:

Ingest one local file: PDFs page by page, everything else through litesearch’s parsers.

PDFs are parsed here rather than through litesearch.add_file for one reason: pdf-oxide writes extracted images relative to its out_path, which defaults to ./pdfs/, so ingesting a paper would silently litter whatever directory you happened to be in. They go next to the vault.

Vault.assets

def assets(
    name:str=None
)->Path:

Where extracted assets (PDF images) go: beside the vault file, never the working directory.

Vault.add

def add(
    pages, # markdown/text, or [(page_no, text)]
    title:str, # document title
    source:str=None, # url or path; defaults to the title. Identity is hashed over it
    kind:str='file', # one of KINDS — the facet you filter and report on
    meta:dict=None, # provenance: the query that found it, when, which tier fetched it
    force:bool=False, # re-ingest a source already present
    **kw
)->dict: # forwarded to litesearch add_doc (chunker, summarize, with_heading)

Ingest one document into the vault: tree, chunks, embeddings, ANN index.

Identity is content-addressed over source|title, so re-adding the same page is a no-op rather than a duplicate — which is what makes it safe to re-run a search whose results overlap what you already have.

Vault.toc

def toc(
    **kw
)->list:

The table of contents across every document in the vault.

Vault.read

def read(
    node_id:str, max_chars:int=6000
)->dict:

Assemble a whole section back out of its chunks.

Vault.related

def related(
    node_id:str, limit:int=8
)->L:

Sections nearest an existing one — “what else in the vault reads like this”.

Reuses the vectors usearch already holds, so nothing is re-embedded.

Vault.context

def context(
    q:str, # the question
    sections:int=6, # operative sections returned
    related:int=8, # related sections reached by graph + vector
    kind:str=None, # restrict to one or more KINDS ('note' or 'note,web')
    max_read:int=6000, # chars of assembled text per section
    **kw
)->AttrDict: # forwarded to litesearch context

The retrieval an LLM should be handed: whole sections plus what they connect to.

Operative sections carry text, breadcrumb, pages, filename and their tree neighbourhood; related holds sections reached by the entity graph (via='graph') and by embedding similarity (via='vector'). kind filters after retrieval here, because litesearch’s context does not thread a where down to both legs — the over-fetch covers the common case, but a filter matching very little of a large vault can still come back short.

Vault.sections

def sections(
    q:str, limit:int=5, kind:str=None, per:int=3, **kw
)->list:

Ranked sections* rather than chunks — the unit worth reading, each with a read handle.*

Vault.find

def find(
    q:str, # query
    limit:int=10, # hits to return
    kind:str=None, # restrict to one or more KINDS ('note' or 'note,web')
    **kw
)->list: # forwarded to litesearch doc_search

Chunk-level hybrid search (FTS5 + vectors, RRF-fused), each hit carrying its breadcrumb.

Vault.stats

def stats()->dict:

Row counts across the vault, by kind.

Vault.forget

def forget(
    doc_id:str
):

Remove a document, its sections and its chunks, and rebuild the ANN index.

Vault.sources

def sources(
    kind:str=None
)->L:

Every document in the vault with its provenance, newest first.

Vault.map

def map(
    min_count:int=2, **kw
)->AttrDict:

Cluster the corpus into labelled topics — the shape of what you have collected.

Vault.connect

def connect(
    resolve:bool=True, **kw
)->dict:

(Re)build the entity graph over everything in the vault.

Try it

offline=True skips the model download and uses litesearch’s hash_embed, which is what you want in CI and for a quick look: retrieval is lexical, and stats()['encoder'] says so.

v = Vault(':memory:')
v.note('Late chunking beats naive chunking because context survives the split.', tags=['retrieval'])
v.add('# Attention\n\nScaled dot-product attention weights values by query-key similarity.\n\n'
      '## Multi-head\n\nHeads attend to different subspaces in parallel.', 'Attention', kind='note')
v.stats()

{'docs': 2,
 'nodes': 5,
 'chunks': 3,
 'entities': 0,
 'by_kind': {'note': 2},
 'encoder': 'model2vec',
 'path': ':memory:'}

test_eq(v.stats()['docs'], 2)
test_eq(v.stats()['encoder'], 'model2vec')
assert v.find('chunking')[0]['content']
test_eq([d['kind'] for d in v.sources()], ['note', 'note'])
test_eq(len(v.sources(kind='web')), 0)
test_eq(len(v.find('chunking', kind='web')), 0)

test_eq(tidy_bc('Attention › Pages 1–1: Scaled dot-product › Multi-head'), 'Attention › Multi-head')
test_eq(tidy_bc(None), '')

r = v.connect()
assert r['entities'] > 0 and r['resolved']['resolvable'] == r['entities']
test_eq(v.stats()['entities'], r['entities'])