concepts

Embedding models, model backends and shelves, and when to change the defaults

The defaults work for most vaults. This page explains the three choices behind them:

• the embedding model that turns text into vectors
• the language model that answers
• the shelf that holds a document

Use it to decide when a default no longer fits.

from tempfile import mkdtemp
from vishalakshi import Vault
from functools import partial
from vishalakshi.ask import CHAT_CACHE, CachedChat, use_chat

v = Vault(Path(mkdtemp())/'vault.db')
v.note('federate fuses the legs by rank because they share no vector space: the vault embeds '
       'prose, kosha embeds identifiers, ripgrep embeds nothing.', tags=['retrieval'])
v.add('# Late chunking\n\n## Method\n\nEmbed the whole document, then pool per chunk, so a chunk '
      'keeps the context around it.\n\n## Results\n\nEvaluated on BEIR.', 'Late chunking', kind='note')
with use_chat(partial(CachedChat, path=CHAT_CACHE)): v.connect()   # replays a recorded extraction; plain `v.connect()` asks the model
v.stats()

/Users/71293/code/personal/orgs/vishalakshi/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

{'docs': 2,
 'nodes': 6,
 'chunks': 3,
 'encoder': 'model2vec',
 'entities': 21,
 'path': '/var/folders/kg/9vdw4mdd1fs58svgh4k1qhr09x7dqh/T/tmph5ysy_7x/vault.db',
 'by_kind': {'note': 2}}

What retrieval actually returns

v.context(q) is the retrieval that ask uses. It returns whole sections. Each section has a breadcrumb and a node_id that read opens again.

The first cell below prints the length and breadcrumb of each section in results. The code hits at the end come from kosha and ripgrep, because this repository is indexed.

c = v.context('why are rankings fused instead of distances?', sections=4, related=4)
for s in c.results: print(f'{len(s.text):5}  {s.breadcrumb}')

  141  federate fuses the legs by rank because they share no vector space: the vault em
   86  Late chunking › Method
   18  Late chunking › Results
  234  repo › /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/code.py:57
  154  grep › index.ipynb:161
  600  repo › /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/extract.py:179
   89  grep › 07_concepts.ipynb:101

for s in c.related: print(f'{s.via:6}  {s.breadcrumb}')

results are the sections that answer the question. related are the sections nearest to them in vector space. They give leads before you know what to search for. They do not change the ranking of results. The via field of each related section says how it was reached.

c.encoder

'minishlab/potion-multilingual-128M (256d, float16, model2vec)'

c.encoder names the embedding model that produced the vectors. With offline=True, the vault hashes character sequences and uses no model. evals/encoder.py measures the cost of this against the default.

Models and backends

A model id names a model. Rishi works out which backend runs it. The local backends are LiteRT, MLX, llama.cpp and Ollama. A hosted API needs its API key. LiteRT is the default local backend.

Rishi owns the backends and their options. They change with Rishi. This page shows only the two arguments that the vault passes on: model= and chat_kw=.

import rishi.core
from urai import Chat, infer_runtime
from vishalakshi.ask import dflt_model

# Rishi names and registers the backends. Urai resolves their model ids.
gemma4_e2b = 'litert-community/gemma-4-E2B-it-litert-lm'
dflt_model, gemma4_e2b     # $VISHALAKSHI_MODEL if set, else the small local default

{m: infer_runtime(m) for m in (gemma4_e2b, 'mlx-community/Qwen3-4B-4bit', 'my-local.gguf',
                               'gpt-5.6-luna', 'gemma-3-4b-it-int4')}

v.ask('why are rankings fused?', model=gemma4_e2b)             # an id: rishi knows what runs it
v.ask('why are rankings fused?', model='llama/my-local.gguf')  # a prefix when the id cannot say
v.ask('why are rankings fused?', model='mlx-community/Qwen3-4B-4bit',
      chat_kw=dict(temp=0, think=True))          # the rest of rishi's constructor

from fastcore.test import test_fail
# `model=` plus `chat_kw=` is the model API. Urai reports an id that names no registered backend.
test_fail(lambda: Chat('gemma-3-4b-it-int4'), contains='backend')

A reasoning model writes its reasoning in a <think> block before the answer. ask puts that block in r.thinking and the rest in r.answer. The block often mentions sections that the model then sets aside. cited reads citations from the answer only.

Retrieval never needs the network. Only answering with a hosted model does.

Encoders

An encoder, or embedding model, turns text into a vector. Texts with similar meanings get vectors that are close together.

The default encoder is a static multilingual model. It produces 256 numbers per text, stored as 16-bit floats.

offline=True hashes character sequences instead. This needs no download. It matches spelling and not meaning.

A shelf records the encoder that wrote it.

Vault(':memory:', offline=True).enc.note      # never attempt a download; also the CI default

'char-n-gram hashing (256d): lexical only; pass encoder= or restore network access for real semantics'

Vault(encoder='minishlab/potion-science-32M')   # pick a different one

Vault('/Users/71293/.vishalakshi/vault.db': 0 docs, 0 chunks, 0 entities, encoder=model2vec)

16-bit floats are the default width in litesearch. One vector index holds the vectors of one encoder. Documents embedded by different encoders need different shelves.

ENCODERS maps three aliases to model ids:

alias	use
multilingual	The default.
code	The model that kosha uses for identifiers.
gemma	The slowest and most faithful.

Any other model2vec id also works by name.

from vishalakshi.core import ENCODERS

ENCODERS

A static model is a lookup table from words to vectors. It embeds a document in milliseconds without a GPU.

Weighted mean reciprocal rank (MRR) measures how highly retrieval places relevant results. Higher scores are better. litesearch measured weighted MRR across four encoders (evals/encoder.py).

measure	value
Encoders compared	4
Lowest measured score	0.018
Highest measured score	0.046

The static default trades some score for speed.

gemma is an ONNX model. Choosing it loads onnxruntime.

# `encoder=` also takes any object with an `.encode` method; `mk_encoder` wraps it for a shelf
import numpy as np
from vishalakshi.core import mk_encoder

class Eight:
    def encode(self, xs, **kw): return np.zeros((len(xs), 8), dtype=np.float16)

e = mk_encoder(Eight())
e.dims, e.method, e.name

One vector index is one vector space. Different encoders need different shelves. SHELVES names the shelves that the vault uses itself. KIND_SHELF maps each kind that acquisition adds to its own shelf.

from vishalakshi.core import SHELVES, KIND_SHELF

SHELVES, KIND_SHELF

papers = v.shelf('papers')        # uses the live encoder unless a registry entry names another
papers.add('# Late chunking\n\nWe evaluate contextual chunk embeddings on BEIR.', 'a paper')

[(s['store'], s['encoder'], s['docs']) for s in v.shelves()]

KIND_SHELF routes documents as they arrive. reshelf moves a document later, when its doctype is known. elsewhere searches the other shelves.

v.route('arxiv').name, v.route('web').name

('papers', 'store')

A PDF may be an invoice or a paper. Its kind does not say which. categorize decides after the document is filed. reshelf then moves the document to the shelf of its doctype.

v.add('# Attention is all you need\n\n## Abstract\n\nWe propose the Transformer.\n\n'
      '## Introduction\n\nRelated work [1] et al.\n\n## References\n\ndoi:10.1/x',
      'attention', source='/inbox/attention.pdf')

r = v.reshelf('/inbox/attention.pdf', llm='never')
r.doctype, r.was, r.store, r.moved

('paper', 'store', 'papers', True)

v.doc('/inbox/attention.pdf'), v.shelf('papers').doc('/inbox/attention.pdf')['title']

(None, 'attention')

Moving a document adds it to the new shelf from scratch. The encoder of the new shelf makes the vectors again. reshelf deletes the copy on the old shelf.

The meta of the document moves with it. It includes the doctype and the shelf that the document came from.

Marks do not move. A mark belongs to a document on its own shelf.

federate runs a search on each shelf. It merges the results by rank, together with the kosha and ripgrep results. elsewhere adds a few sections from the neighbouring shelves to context and ask.

v.federate('contextual chunk embeddings', repo=False, grep=False).legs

{'prose': 3, 'shelf:papers': 4}

Each row from elsewhere has a store field that names its shelf. Pass store to read(node_id, store=...) when the section is not on the current shelf.

e = v.elsewhere('contextual chunk embeddings')
L(e).map(lambda r: (r.store, r.breadcrumb))

[('papers', 'papers › a paper'),
 ('papers', 'papers › attention › Attention is all you need › Abstract')]

v.read(e[0].node_id, store=e[0].store)['text'][:80]

'# Late chunking\n\nWe evaluate contextual chunk embeddings on BEIR.'

A shelf records the encoder that wrote it and reopens with the same encoder. Opening a shelf with a different encoder raises an error, so two vector spaces never mix. To change the encoder of a shelf, drop the shelf and add its documents again.