# core
A [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault)
is a [litesearch](https://github.com/Karthik777/litesearch) document
tree — docs → nodes → chunks, FTS5 and a usearch HNSW index — with an
entity graph over it, in one SQLite file. Everything you read goes in
under a `kind`, and one query crosses all of them. Nothing here
reimplements litesearch: the vault holds an encoder and a store name,
and hands both to it.
------------------------------------------------------------------------
source
### kinds
``` python
def kinds(
kind
)->L:
```
*A kind filter as a list — `'note'`, `'note,web'` and `['note','web']`
all work.*
------------------------------------------------------------------------
source
### tidy_bc
``` python
def tidy_bc(
bc:str
)->str:
```
*Drop `build_tree`’s `Pages n–m:` window placeholders from a breadcrumb:
real nodes, noise in a citation.*
------------------------------------------------------------------------
source
### mk_encoder
``` python
def mk_encoder(
model:str=None, # model2vec/HF id; None -> the retrieval default
dims:int=256, # dims for the hashing fallback only
offline:bool=False, # skip the download attempt entirely
dtype:type=float16, # stored width; litesearch's default everywhere
)->AttrDict:
```
*The best encoder available as
`AttrDict(model, doc, query, dtype, dims, method, note)`.*
Degrades to litesearch’s `hash_embed` rather than failing, and always
says which answered: the gap between the two is the gap between a vault
that answers questions and one that can only keyword-match. `model` is
the embedder object itself, so kosha can share it (see `code`).
Both encoders are cast to `dtype` because litesearch’s own default is
float16 and not every entry point takes a `dtype=`: `Database.context`
does not thread one down to its section search, so a float32 store would
be read back as float16 there — the vectors survive, the *distances* do
not, and `context()` silently degrades to keyword ranking. Half
precision costs nothing at these magnitudes; a mismatched width costs
the whole semantic leg.
------------------------------------------------------------------------
source
### Vault
``` python
def Vault(
path:str=None, # vault file; None -> ~/.vishalakshi/vault.db
encoder:NoneType=None, # an mk_encoder() AttrDict, a model id, or None
store:str='store', # chunk store name
offline:bool=False, # never attempt a model download
dims:int=256, # dims for the hashing fallback
):
```
*Everything you have read, in one SQLite file, searchable as one
corpus.*
Web pages, PDFs, papers, transcripts, local files, code and your own
notes land in the same litesearch store under different `kind`s, which
is the point: one query crosses all of them and `context()` hands back
sections rather than fragments. Acquisition lives in
`vishalakshi.acquire`, answering in `.ask`, code in `.code` — all
optional, since the vault itself needs neither a network nor an LLM.
------------------------------------------------------------------------
source
### Vault.note
``` python
def note(
text:str, # what you want to remember
title:str=None, # defaults to the first line
tags:list=None, # free-form tags, kept in the doc's meta
)->dict:
```
*Write a note into the vault so it is searched alongside the corpus.*
Notes are ordinary documents with `kind='note'`, which is deliberate:
the graph, the clusters and `context()` all see them for free, so what
you concluded about a corpus comes back next to the evidence you
concluded it from.
------------------------------------------------------------------------
source
### Vault.add_dir
``` python
def add_dir(
dir:str, types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb', kind:str=None, **kw
)->L:
```
*Ingest every document under a directory. `dir2files` skips dotfiles,
tests, build and dist.*
------------------------------------------------------------------------
source
### Vault.add_file
``` python
def add_file(
path:str, title:str=None, kind:str=None, **kw
)->dict:
```
*Ingest one local file: PDFs page by page, everything else through
litesearch’s parsers.*
PDFs are parsed here rather than through `litesearch.add_file` for one
reason: pdf-oxide writes extracted images relative to its `out_path`,
which defaults to `./pdfs/`, so ingesting a paper would silently litter
whatever directory you happened to be in. They go next to the vault.
------------------------------------------------------------------------
source
### Vault.assets
``` python
def assets(
name:str=None
)->Path:
```
*Where extracted assets (PDF images) go: beside the vault file, never
the working directory.*
------------------------------------------------------------------------
source
### Vault.add
``` python
def add(
pages, # markdown/text, or [(page_no, text)]
title:str, # document title
source:str=None, # url or path; defaults to the title. Identity is hashed over it
kind:str='file', # one of KINDS — the facet you filter and report on
meta:dict=None, # provenance: the query that found it, when, which tier fetched it
force:bool=False, # re-ingest a source already present
**kw
)->dict: # forwarded to litesearch add_doc (chunker, summarize, with_heading)
```
*Ingest one document into the vault: tree, chunks, embeddings, ANN
index.*
Identity is content-addressed over `source|title`, so re-adding the same
page is a no-op rather than a duplicate — which is what makes it safe to
re-run a search whose results overlap what you already have.
------------------------------------------------------------------------
source
### Vault.toc
``` python
def toc(
**kw
)->list:
```
*The table of contents across every document in the vault.*
------------------------------------------------------------------------
source
### Vault.read
``` python
def read(
node_id:str, max_chars:int=6000
)->dict:
```
*Assemble a whole section back out of its chunks.*
------------------------------------------------------------------------
source
### Vault.related
``` python
def related(
node_id:str, limit:int=8
)->L:
```
*Sections nearest an existing one — “what else in the vault reads like
this”.*
Reuses the vectors usearch already holds, so nothing is re-embedded.
------------------------------------------------------------------------
source
### Vault.context
``` python
def context(
q:str, # the question
sections:int=6, # operative sections returned
related:int=8, # related sections reached by graph + vector
kind:str=None, # restrict to one or more KINDS ('note' or 'note,web')
max_read:int=6000, # chars of assembled text per section
**kw
)->AttrDict: # forwarded to litesearch context
```
*The retrieval an LLM should be handed: whole sections plus what they
connect to.*
Operative sections carry `text, breadcrumb, pages, filename` and their
tree neighbourhood; `related` holds sections reached by the entity graph
(`via='graph'`) and by embedding similarity (`via='vector'`). `kind`
filters *after* retrieval here, because litesearch’s `context` does not
thread a `where` down to both legs — the over-fetch covers the common
case, but a filter matching very little of a large vault can still come
back short.
------------------------------------------------------------------------
source
### Vault.sections
``` python
def sections(
q:str, limit:int=5, kind:str=None, per:int=3, **kw
)->list:
```
*Ranked* sections\* rather than chunks — the unit worth reading, each
with a `read` handle.\*
------------------------------------------------------------------------
source
### Vault.find
``` python
def find(
q:str, # query
limit:int=10, # hits to return
kind:str=None, # restrict to one or more KINDS ('note' or 'note,web')
**kw
)->list: # forwarded to litesearch doc_search
```
*Chunk-level hybrid search (FTS5 + vectors, RRF-fused), each hit
carrying its breadcrumb.*
------------------------------------------------------------------------
source
### Vault.stats
``` python
def stats()->dict:
```
*Row counts across the vault, by kind.*
------------------------------------------------------------------------
source
### Vault.forget
``` python
def forget(
doc_id:str
):
```
*Remove a document, its sections and its chunks, and rebuild the ANN
index.*
------------------------------------------------------------------------
source
### Vault.sources
``` python
def sources(
kind:str=None
)->L:
```
*Every document in the vault with its provenance, newest first.*
------------------------------------------------------------------------
source
### Vault.map
``` python
def map(
min_count:int=2, **kw
)->AttrDict:
```
*Cluster the corpus into labelled topics — the shape of what you have
collected.*
------------------------------------------------------------------------
source
### Vault.connect
``` python
def connect(
resolve:bool=True, **kw
)->dict:
```
*(Re)build the entity graph over everything in the vault.*
## Try it
`offline=True` skips the model download and uses litesearch’s
`hash_embed`, which is what you want in CI and for a quick look:
retrieval is lexical, and `stats()['encoder']` says so.
``` python
v = Vault(':memory:')
v.note('Late chunking beats naive chunking because context survives the split.', tags=['retrieval'])
v.add('# Attention\n\nScaled dot-product attention weights values by query-key similarity.\n\n'
'## Multi-head\n\nHeads attend to different subspaces in parallel.', 'Attention', kind='note')
v.stats()
```
{'docs': 2,
'nodes': 5,
'chunks': 3,
'entities': 0,
'by_kind': {'note': 2},
'encoder': 'model2vec',
'path': ':memory:'}
``` python
test_eq(v.stats()['docs'], 2)
test_eq(v.stats()['encoder'], 'model2vec')
assert v.find('chunking')[0]['content']
test_eq([d['kind'] for d in v.sources()], ['note', 'note'])
test_eq(len(v.sources(kind='web')), 0)
test_eq(len(v.find('chunking', kind='web')), 0)
```
``` python
test_eq(tidy_bc('Attention › Pages 1–1: Scaled dot-product › Multi-head'), 'Attention › Multi-head')
test_eq(tidy_bc(None), '')
```
``` python
r = v.connect()
assert r['entities'] > 0 and r['resolved']['resolvable'] == r['entities']
test_eq(v.stats()['entities'], r['entities'])
```