# vault


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## Why a vault

`LocalHost`, the host the terminal runs on, can read web pages and index
the repository, but it keeps neither past the end of the process. A page
read in one session is gone in the next, and the code index and the
pages read live in separate stores that never meet.

A [vishalakshi](https://vedicreader.github.io/vishalakshi/) `Vault` is
one SQLite file that holds documents of every kind: notes, web pages,
research.
[`VaultHost`](https://vedicreader.github.io/ramabana/vault.html#vaulthost)
puts one behind the host, and three things follow.

- **Memory.** `LocalHost` leaves the memory group out of the groups it
  provides, so
  [`tools_for`](https://vedicreader.github.io/ramabana/tools.html#tools_for)
  builds none of its tools. With a vault the memory tools appear
  (`memory_search`, `memory_read`, `remember`, `memory_forget`),
  together with `ask_memory`, and they answer from everything the vault
  has stored.
- **One search over everything.** `search_code` asks three indexes at
  once: the vault for prose, kosha for identifiers and ripgrep for
  literal text. The three score results on unrelated scales, so
  `Vault.federate` merges them by rank rather than by score.
- **Watches.** A watch is a standing job the vault runs on a schedule,
  such as re-reading a page every week or filing a reminder. With a
  vault the watch tools appear too (`watch`, `list_watches`,
  `cancel_watch`).

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ramabana/blob/main/ramabana/vault.py#L27"
target="_blank" style="float:right; font-size:smaller">source</a>

### safe_shelf

``` python
def safe_shelf(
    name:NoneType=None, dflt:str='store'
):
```

*Return a valid SQL table-prefix identifier for a shelf name.*

## The host

[`VaultHost`](https://vedicreader.github.io/ramabana/vault.html#vaulthost)
is a `LocalHost` with a vishalakshi vault. The constructor opens the
vault in a background thread, so building the tool list never waits for
an embedding model to load. It declares the memory, ask and watch groups
at construction, by taking them out of `without`, the set of groups a
host switches off.

`vault` is a `Vault`, a path to one, or `None` for the shared default at
`~/.vishalakshi/vault.db`. `shelf` names a separate store inside the
vault for this host’s writes. `remember_reads` files every page
`read_url` fetches, and `federate` turns on the merged search. `pii` and
`pii_ner` gate personal data and are covered below. `mk_chat` builds the
chats the vault answers questions with, and `graph_chat` is the chat
that builds its entity graph.

[`WorkspaceHost`](https://vedicreader.github.io/ramabana/vault.html#workspacehost),
in the same cell, is the host
[`mk_agent`](https://vedicreader.github.io/ramabana/agent.html#mk_agent)
builds. It is a
[`VaultHost`](https://vedicreader.github.io/ramabana/vault.html#vaulthost)
and a
[`SpecHost`](https://vedicreader.github.io/ramabana/spec.html#spechost)
at once, with both providers off unless asked for: `vault=True` (or a
path, or a `Vault`) turns memory on, and `spec=True` turns on the API
tools from the spec notebook. When a provider is off, its groups stay
switched off and the calls fall through to `LocalHost`. It adds two
calls of its own. `add_worktree(repo, branch)` makes a git worktree for
a job and opens it as a root. `run_cmd_bg` starts a background command
that logs to a file rather than opening a tmux pane.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ramabana/blob/main/ramabana/vault.py#L236"
target="_blank" style="float:right; font-size:smaller">source</a>

### WorkspaceHost

``` python
def WorkspaceHost(
    roots:tuple=('.',), vault:bool=False, # False, True (default vault), a path, or a Vault
    spec:bool=False, # whether Ramabana's API provider is available
    pii:NoneType=None, pii_ner:NoneType=None, warm:bool=True, **kwargs
):
```

*One workspace host configured with optional memory and API providers.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ramabana/blob/main/ramabana/vault.py#L331"
target="_blank" style="float:right; font-size:smaller">source</a>

### WorkspaceHost.open_index

``` python
def open_index():
```

*Search each root’s index as the last sync left it, loaded by the first
search;
[`sync_index`](https://vedicreader.github.io/ramabana/cli.html#sync_index)
skips here when nothing changed.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ramabana/blob/main/ramabana/vault.py#L319"
target="_blank" style="float:right; font-size:smaller">source</a>

### LazyIndex

``` python
def LazyIndex(
    root
):
```

*A root’s Kosha index as it stands on disk, opened without a sync on
first use.*

``` python
with tempfile.TemporaryDirectory() as d:
    h = WorkspaceHost([d], index=False, web=False).open_index()
    test_eq((h.index_ready, h.indexed), (True, h.roots))
    test_eq(h._indexes[0]._k, None)
    assert not any(hasattr(h, k) for k in ('mk_chat', 'pii', 'pii_ner'))
```

`poll` and `connect` work on a second connection to the vault file, made
by `_worker_vault`, so a background job never shares the connection a
turn is using. The next two cells check that the poll gets a connection
of its own.

``` python
from tempfile import TemporaryDirectory

with TemporaryDirectory() as d:
    host = VaultHost(vault=Path(d)/'vault.db', warm=False)
    host.vault.add('A report about polonium.', title='report')

    worker = host._worker_vault()

    assert worker is not host.vault
    assert worker.db is not host.vault.db
    assert worker.doc('report')

    worker.db.conn.close()
```

    Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

    Updating packages:   0%|          | 0/31 [00:00<?, ?pkg/s]

    parse files from /home/runner/work/ramabana/ramabana:   0%|          | 0/103 [00:00<?, ?it/s]

    parse files from /home/runner/work/ramabana/ramabana:   2%|1         | 2/103 [00:00<00:40,  2.47it/s]

    parse files from /home/runner/work/ramabana/ramabana:   3%|2         | 3/103 [00:03<02:14,  1.34s/it]


    parse files from httpx:   0%|          | 0/23 [00:00<?, ?it/s]

    parse files from /home/runner/work/ramabana/ramabana:   7%|6         | 7/103 [00:03<00:39,  2.40it/s]


    parse files from httpx:  26%|##6       | 6/23 [00:00<00:00, 51.82it/s]

    parse files from /home/runner/work/ramabana/ramabana:  13%|#2        | 13/103 [00:03<00:16,  5.49it/s]

    parse files from /home/runner/work/ramabana/ramabana:  16%|#5        | 16/103 [00:04<00:16,  5.43it/s]


    parse files from httpx:  52%|#####2    | 12/23 [00:00<00:00, 13.22it/s]
    parse files from httpx: 100%|##########| 23/23 [00:00<00:00, 25.75it/s]


    parse files from /home/runner/work/ramabana/ramabana:  32%|###2      | 33/103 [00:04<00:04, 17.24it/s]

    parse files from /home/runner/work/ramabana/ramabana:  63%|######3   | 65/103 [00:04<00:00, 45.17it/s]

    parse files from /home/runner/work/ramabana/ramabana: 100%|##########| 103/103 [00:04<00:00, 84.16it/s]
    parse files from /home/runner/work/ramabana/ramabana: 100%|##########| 103/103 [00:04<00:00, 22.32it/s]

``` python
from ramabana.testing import no_model

with TemporaryDirectory() as d:
    host = VaultHost(vault=Path(d)/'vault.db', warm=False, graph_chat=no_model())
    host.watch('scheduled report', action='remind', every='1d')

    root = host.vault
    seen = []

    original = host._worker_vault
    def worker():
        v = original()
        seen.append(v)
        return v

    host._worker_vault = worker
    host.poll()

    assert seen
    assert seen[0] is not root
    assert seen[0].db is not root.db
```

    SQLITE_LOG: disk I/O error in "pragma "optimize"(65538)" (1802) SQLITE_IOERR_FSTAT
    Exception in thread ramabana-vault-connect:
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/threading.py", line 1075, in _bootstrap_inner

The vault builds its entity graph with `graph_chat` and with nothing
else. When `graph_chat` is `None`, the host builds no graph at all
unless `$VISHALAKSHI_MODEL` names a model, because vishalakshi’s own
default would load a local model. `mk_chat` answers the session’s
questions and may be a hosted model. Every chunk in the vault is private
material, so `mk_chat` never sees the graph build, as the next cell
checks.

``` python
from ramabana.testing import no_model

def _connect_arg(**kw):
    "What `Vault.connect` was handed as `chat`, by a host built with `kw`."
    with TemporaryDirectory() as d:
        h = VaultHost(vault=Path(d)/'vault.db', warm=False, **kw)
        h.vault.add('A report about polonium.', title='report')
        got = []
        original = h._worker_vault
        def worker():
            v = original()
            v.connect = lambda *a, **k: got.append(k.get('chat'))
            return v
        h._worker_vault = worker
        h.connect(wait=True)
        return got

test_eq(_connect_arg(mk_chat=lambda model=None, **kw: None), [None])
quiet = no_model()
test_eq(_connect_arg(graph_chat=quiet), [quiet])
```

    Updating packages:   0%|          | 0/29 [00:00<?, ?pkg/s]


    parse files from /home/runner/work/ramabana/ramabana:   0%|          | 0/103 [00:00<?, ?it/s]


    parse files from /home/runner/work/ramabana/ramabana:   2%|1         | 2/103 [00:00<00:05, 17.22it/s]


    parse files from /home/runner/work/ramabana/ramabana:  13%|#2        | 13/103 [00:00<00:01, 64.06it/s]


    parse files from /home/runner/work/ramabana/ramabana:  28%|##8       | 29/103 [00:00<00:00, 104.49it/s]


    parse files from /home/runner/work/ramabana/ramabana:  61%|######1   | 63/103 [00:00<00:00, 193.01it/s]
    parse files from /home/runner/work/ramabana/ramabana: 100%|##########| 103/103 [00:00<00:00, 196.00it/s]

The rest of the page uses a vault of its own in a temporary folder, so
nothing here touches the real one.

``` python
tmp = Path(tempfile.mkdtemp())
from ramabana.testing import no_model
host = VaultHost(roots=[tmp], vault=tmp/'vault.db', index=False, web=False, graph_chat=no_model())
host.open_vault(wait=True)
host.vault.stats()['encoder']
```

    'model2vec'

## Memory

Memory search returns whole sections of stored documents, plus related
sections found through the vault’s graph. Each section carries its `age`
in seconds and whether it is `stale`. `memory_tree` lists the heading
tree of each stored document, `memory_read` returns one section by its
node id, and `memory_forget` deletes a document. These pass straight
through to the vault.

`remember` stores a note. Notes are ordinary documents in the same store
as everything else, so a conclusion the agent reached comes back next to
the evidence for it. Here two notes go in, and a search in different
words finds them.

``` python
host.remember('The weekly Coles order is milk, sourdough, eggs, baby spinach and olive oil.',
              title='weekly Coles order', tags=['groceries'])
host.remember('Ceres Fair Food delivers a mixed seasonal fruit box on Thursdays.',
              title='Ceres fruit box', tags=['groceries'])

r = host.memory_search('what do I need from the supermarket', limit=2)
[s['title'] for s in r['results']], {k: r['results'][0][k] for k in ('age', 'stale')}
```

``` python
test_eq(len(host.memory_tree()), 2)
test_eq(len(host.memory_tree('ceres')), 1)

node = r['results'][0]['node_id']
'olive oil' in host.memory_read(node)['text'] or 'fruit box' in host.memory_read(node)['text']
```

With a vault behind it,
[`tools_for`](https://vedicreader.github.io/ramabana/tools.html#tools_for)
offers the memory group, which
[`NullHost`](https://vedicreader.github.io/ramabana/tools.html#nullhost)
does not get.

``` python
from ramabana.tools import tools_for, NullHost
have = lambda h: {getattr(t, '__name__', '') for t in tools_for(h)}
test_eq('memory_search' in have(NullHost()), False)
test_eq('memory_search' in have(host), True)
```

## Gating what memory returns

Some of what a vault holds is personal: names, emails, phone numbers.
`pii` sets what happens when a section that the vault hands back
contains such data. `off`, the default, returns it as stored. `redact`
masks the personal parts, and `refuse` withholds the section.
`pii_ner=True` also catches titled names, not only what the patterns
match. vishalakshi does the gating, and the host passes the setting into
every vault read:

- `memory_search`
- `memory_read`
- `memory_tree`
- `search`
- `research`

Either setting can be a function instead of a value. The host calls it
on every read, so a settings panel can change it mid-session, and a
function that raises counts as `off`. The hidden cells check each mode
against a note with an email and a phone number in it.

## Federated search

`search`, the host call behind `search_code`, asks the vault for prose,
kosha for identifiers and ripgrep for literal text, and merges the three
lists with `litesearch.rrf_all` through `Vault.federate`. If the vault
fails to open, the host falls back to `LocalHost.search`.

In an empty temporary folder only the vault has anything to find: there
is no kosha index, and no text file for ripgrep to match. `search_note`
reports how many hits each index gave.

``` python
hits = host.search('seasonal fruit box', limit=5)
host.search_note
```

A prose hit’s `symbol` is its section’s node id, so
`memory_read(hit.symbol)` reads the whole section. A repository hit’s
`symbol` is its module name, for `symbols`. One field serves whichever
follow-up suits the kind of hit.

``` python
prose = [h for h in hits if h.symbol.count('#') == 1]
prose and host.memory_read(prose[0].symbol)['text'][:60]
```

`federate=False` turns the merging off and keeps `LocalHost`’s own
search, which is what a host gets when its vault will not open.

``` python
plain = VaultHost(roots=[tmp], vault=tmp/'vault.db', index=False, web=False,
                  federate=False, warm=False)
test_eq(plain.search_note, plain.__class__.__mro__[1].search_note.fget(plain))
```

## Watches and reminders

Watches live in the vault, so they survive between sessions. `poll` runs
the watches that are due. It does not rebuild the entity graph while a
turn waits; when something ran, it starts `connect`, which rebuilds the
graph in the background.

Each watch has an action and an interval. The simplest action is
`remind`. When a reminder comes due, the vault files its text back into
itself as a note, so the reminder can be found by memory search next to
everything else instead of sitting in a notification queue the harness
does not own.

``` python
w = host.watch('Add the weekly groceries to the Coles trolley, and a Ceres fruit box.',
               action='remind', every='1w', note='weekly shop')
[(x['id'], x['action'], int(x['every'])) for x in host.watches()]
```

A new watch is first due at the moment it is created, so a poll at any
earlier moment finds nothing to run.

``` python
test_eq(host.vault.poll(at=w['next_run'] - 1, connect=False)['ran'], 0)
```

A poll after that moment runs it, and a memory search then finds what it
filed.

``` python
fired = host.vault.poll(at=w['next_run'] + 1, connect=False)
test_eq(fired['ran'], 1)
test_eq(fired['results'][0]['status'], 'ok')

hit = host.memory_search('what am I supposed to be doing about groceries', limit=3)
[s['title'] for s in hit['results']]
```

One failing watch must not stop the others. vishalakshi’s `run_watch`
records a failure on the watch and returns it instead of raising, so a
poll over five due watches runs all five. The vault runs each watch by
calling its method named after the action, so replacing `host.vault.url`
with a function that always fails makes every `url` watch fail. Here one
such watch sits next to a working reminder.

``` python
def _down(target, **kw): raise ConnectionError('name not resolved')
host.vault.url = _down

dead = host.watch('https://example.invalid/x', action='url', every='1d')
live = host.watch('check which day the Ceres box arrives', action='remind', every='1d')
out = host.vault.poll(at=max(dead['next_run'], live['next_run']) + 1, connect=False)
sorted(r['status'] for r in out['results'])
```

The failed watch is still there, rescheduled, with its last status
recorded. A single failure is no reason to delete it, so the cleanup
cancels both by hand.

``` python
host.unwatch(dead['id'])
host.unwatch(live['id'])
test_eq(len(host.watches()), 1)
```

The watch tools follow the capability, the same way the memory tools do.

``` python
tools = have(host)
test_eq({'remember', 'watch', 'list_watches', 'cancel_watch'} <= tools, True)
test_eq(any(t.startswith('watch') for t in have(NullHost())), False)
```

`watch` is one tool for every kind of watch. Its `kind` is `url` for a
page, `remind` for a reminder, `search` for a web query and `folder` for
a path. Polling is the harness’s job, not the model’s. At the start of
each turn
[`Agent.poll_watches`](https://vedicreader.github.io/ramabana/agent.html#agent.poll_watches)
calls `host.poll()` in a background thread, at most once every
`poll_every` seconds, and reports what fired. The host returns failures
instead of raising, so a turn that starts by asking what is outstanding
does not end on an error.

``` python
from ramabana.tools import watch_tools
watch = next(t for t in watch_tools(host) if t.__name__ == 'watch')
print(watch('renew the domain', kind='remind', every='1w'))
host.poll()
```

    watching remind renew the domain as 6a93995604c6, every 1w

    {'checked': 1,
     'ran': 1,
     'results': [{'job_id': '1990f53ec5af',
       'kind': 'watch',
       'status': 'ok',
       'took': 0.238,
       'error': None,
       'result': {'doc_id': '36a48543e42596a7',
        'title': 'renew the domain',
        'kind': 'note',
        'nodes': 2,
        'chunks': 1,
        'changed': True,
        'source': 'note:reminder:6a93995604c6',
        'replaced': False}}],
     'reclaimed': 0,
     'dead': 0,
     'next_due': 1793079498,
     'housekeeping': {'checked': 1, 'stale': 0, 'pruned': 0, 'reasons': {}}}

## Using it

``` python
from ramabana import Agent
from ramabana.vault import VaultHost

host = VaultHost(roots=['~/code/myproject'])     # the shared ~/.vishalakshi/vault.db
agent = Agent(host, model='gpt-mini')
```

That agent gets eight tools a `LocalHost` agent does not have: the four
memory tools, `ask_memory`, and the three watch tools. Its `search_code`
also answers from everything the vault has read.
[`Agent`](https://vedicreader.github.io/ramabana/agent.html#agent)
itself did not change. The tools appeared because the host declares the
groups they belong to.

## Answering from material the model must not see

`memory_search` hands the model whole sections to read. That suits
research and is wrong for a bank statement. To answer from a section,
the section has to be in the context of whatever model runs the turn,
and that model often runs on someone else’s servers.

`ask`, which the `ask_memory` tool calls, works the other way round. The
vault answers and the caller gets only the answer. When the sections it
retrieved hold personal information, which `vishalakshi.pii` detects by
pattern rather than by asking a model to read them first, the vault
answers with a local model under a briefing that forbids repeating any
personal detail. These cells are marked `eval: false`, so the test suite
skips them.

In a real session
[`Agent.lend_model`](https://vedicreader.github.io/ramabana/agent.html#agent.lend_model)
supplies `mk_chat`, and it honours the model it is asked for. The vault
asks for its local model and then checks that the chat it got back runs
on this machine. Without that check a factory could send a statement to
a hosted API. Here the factory returns a chat that claims a remote
runtime, so the vault refuses and sends nothing.

``` python
_h.mk_chat = _chat('remote', 'THIS MUST NEVER BE SENT')
_sent.clear()
r = _h.ask('what is on invoice 4471?')
test_eq(_sent, [])
test_eq(r.refused, True)
r.answer
```

A chat on a local runtime does receive the question, under a different
briefing, and returns an answer that keeps the personal details back.

``` python
_h.mk_chat = _chat('litert', 'One invoice, 240.00 GBP, due in September. Holding back the name, '
                             'email, phone and card. Tell me if you need a total, a date, or a yes/no.')
_sent.clear()
print(ask_memory('what is on invoice 4471?'))
```

That reply is half of a two-step exchange. The model reading it decides
what it needs and sends an `instruction` back, which runs the local
model again over the same sections. The useful answer arrives and the
private material never leaves the machine.

``` python
_h.mk_chat = _chat('litert', 'Total 240.00 GBP, due 2026-09-01.')
_sent.clear()
out = ask_memory('what is on invoice 4471?', instruction='Give me the total and the due date only.')
test_eq('Instruction from the questioner' in _sent[0][1], True)
out
```