vault

Memory that outlives the process: a vishalakshi vault behind the host,

Why a vault

LocalHost, the host the terminal runs on, can read web pages and index the repository, but it keeps neither past the end of the process. A page read in one session is gone in the next, and the code index and the pages read live in separate stores that never meet.

A vishalakshi Vault is one SQLite file that holds documents of every kind: notes, web pages, research. VaultHost puts one behind the host, and three things follow.

  • Memory. LocalHost leaves the memory group out of the groups it provides, so tools_for builds none of its tools. With a vault the memory tools appear (memory_search, memory_read, remember, memory_forget), together with ask_memory, and they answer from everything the vault has stored.
  • One search over everything. search_code asks three indexes at once: the vault for prose, kosha for identifiers and ripgrep for literal text. The three score results on unrelated scales, so Vault.federate merges them by rank rather than by score.
  • Watches. A watch is a standing job the vault runs on a schedule, such as re-reading a page every week or filing a reminder. With a vault the watch tools appear too (watch, list_watches, cancel_watch).

source

safe_shelf

def safe_shelf(
    name:NoneType=None, dflt:str='store'
):

Return a valid SQL table-prefix identifier for a shelf name.

The host

VaultHost is a LocalHost with a vishalakshi vault. The constructor opens the vault in a background thread, so building the tool list never waits for an embedding model to load. It declares the memory, ask and watch groups at construction, by taking them out of without, the set of groups a host switches off.

vault is a Vault, a path to one, or None for the shared default at ~/.vishalakshi/vault.db. shelf names a separate store inside the vault for this host’s writes. remember_reads files every page read_url fetches, and federate turns on the merged search. pii and pii_ner gate personal data and are covered below. mk_chat builds the chats the vault answers questions with, and graph_chat is the chat that builds its entity graph.

WorkspaceHost, in the same cell, is the host mk_agent builds. It is a VaultHost and a SpecHost at once, with both providers off unless asked for: vault=True (or a path, or a Vault) turns memory on, and spec=True turns on the API tools from the spec notebook. When a provider is off, its groups stay switched off and the calls fall through to LocalHost. It adds two calls of its own. add_worktree(repo, branch) makes a git worktree for a job and opens it as a root. run_cmd_bg starts a background command that logs to a file rather than opening a tmux pane.

source

WorkspaceHost

def WorkspaceHost(
    roots:tuple=('.',), vault:bool=False, # False, True (default vault), a path, or a Vault
    spec:bool=False, # whether Ramabana's API provider is available
    pii:NoneType=None, pii_ner:NoneType=None, warm:bool=True, **kwargs
):

One workspace host configured with optional memory and API providers.

source

WorkspaceHost.open_index

def open_index():

Search each root’s index as the last sync left it, loaded by the first search; sync_index skips here when nothing changed.

source

LazyIndex

def LazyIndex(
    root
):

A root’s Kosha index as it stands on disk, opened without a sync on first use.

with tempfile.TemporaryDirectory() as d:
    h = WorkspaceHost([d], index=False, web=False).open_index()
    test_eq((h.index_ready, h.indexed), (True, h.roots))
    test_eq(h._indexes[0]._k, None)
    assert not any(hasattr(h, k) for k in ('mk_chat', 'pii', 'pii_ner'))

poll and connect work on a second connection to the vault file, made by _worker_vault, so a background job never shares the connection a turn is using. The next two cells check that the poll gets a connection of its own.

from tempfile import TemporaryDirectory

with TemporaryDirectory() as d:
    host = VaultHost(vault=Path(d)/'vault.db', warm=False)
    host.vault.add('A report about polonium.', title='report')

    worker = host._worker_vault()

    assert worker is not host.vault
    assert worker.db is not host.vault.db
    assert worker.doc('report')

    worker.db.conn.close()
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Updating packages:   0%|          | 0/31 [00:00<?, ?pkg/s]

parse files from /home/runner/work/ramabana/ramabana:   0%|          | 0/103 [00:00<?, ?it/s]

parse files from /home/runner/work/ramabana/ramabana:   2%|1         | 2/103 [00:00<00:40,  2.47it/s]

parse files from /home/runner/work/ramabana/ramabana:   3%|2         | 3/103 [00:03<02:14,  1.34s/it]


parse files from httpx:   0%|          | 0/23 [00:00<?, ?it/s]

parse files from /home/runner/work/ramabana/ramabana:   7%|6         | 7/103 [00:03<00:39,  2.40it/s]


parse files from httpx:  26%|##6       | 6/23 [00:00<00:00, 51.82it/s]

parse files from /home/runner/work/ramabana/ramabana:  13%|#2        | 13/103 [00:03<00:16,  5.49it/s]

parse files from /home/runner/work/ramabana/ramabana:  16%|#5        | 16/103 [00:04<00:16,  5.43it/s]


parse files from httpx:  52%|#####2    | 12/23 [00:00<00:00, 13.22it/s]
parse files from httpx: 100%|##########| 23/23 [00:00<00:00, 25.75it/s]


parse files from /home/runner/work/ramabana/ramabana:  32%|###2      | 33/103 [00:04<00:04, 17.24it/s]

parse files from /home/runner/work/ramabana/ramabana:  63%|######3   | 65/103 [00:04<00:00, 45.17it/s]

parse files from /home/runner/work/ramabana/ramabana: 100%|##########| 103/103 [00:04<00:00, 84.16it/s]
parse files from /home/runner/work/ramabana/ramabana: 100%|##########| 103/103 [00:04<00:00, 22.32it/s]
from ramabana.testing import no_model

with TemporaryDirectory() as d:
    host = VaultHost(vault=Path(d)/'vault.db', warm=False, graph_chat=no_model())
    host.watch('scheduled report', action='remind', every='1d')

    root = host.vault
    seen = []

    original = host._worker_vault
    def worker():
        v = original()
        seen.append(v)
        return v

    host._worker_vault = worker
    host.poll()

    assert seen
    assert seen[0] is not root
    assert seen[0].db is not root.db
SQLITE_LOG: disk I/O error in "pragma "optimize"(65538)" (1802) SQLITE_IOERR_FSTAT
Exception in thread ramabana-vault-connect:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/threading.py", line 1075, in _bootstrap_inner

The vault builds its entity graph with graph_chat and with nothing else. When graph_chat is None, the host builds no graph at all unless $VISHALAKSHI_MODEL names a model, because vishalakshi’s own default would load a local model. mk_chat answers the session’s questions and may be a hosted model. Every chunk in the vault is private material, so mk_chat never sees the graph build, as the next cell checks.

from ramabana.testing import no_model

def _connect_arg(**kw):
    "What `Vault.connect` was handed as `chat`, by a host built with `kw`."
    with TemporaryDirectory() as d:
        h = VaultHost(vault=Path(d)/'vault.db', warm=False, **kw)
        h.vault.add('A report about polonium.', title='report')
        got = []
        original = h._worker_vault
        def worker():
            v = original()
            v.connect = lambda *a, **k: got.append(k.get('chat'))
            return v
        h._worker_vault = worker
        h.connect(wait=True)
        return got

test_eq(_connect_arg(mk_chat=lambda model=None, **kw: None), [None])
quiet = no_model()
test_eq(_connect_arg(graph_chat=quiet), [quiet])


Updating packages:   0%|          | 0/29 [00:00<?, ?pkg/s]


parse files from /home/runner/work/ramabana/ramabana:   0%|          | 0/103 [00:00<?, ?it/s]


parse files from /home/runner/work/ramabana/ramabana:   2%|1         | 2/103 [00:00<00:05, 17.22it/s]


parse files from /home/runner/work/ramabana/ramabana:  13%|#2        | 13/103 [00:00<00:01, 64.06it/s]


parse files from /home/runner/work/ramabana/ramabana:  28%|##8       | 29/103 [00:00<00:00, 104.49it/s]


parse files from /home/runner/work/ramabana/ramabana:  61%|######1   | 63/103 [00:00<00:00, 193.01it/s]
parse files from /home/runner/work/ramabana/ramabana: 100%|##########| 103/103 [00:00<00:00, 196.00it/s]

The rest of the page uses a vault of its own in a temporary folder, so nothing here touches the real one.

tmp = Path(tempfile.mkdtemp())
from ramabana.testing import no_model
host = VaultHost(roots=[tmp], vault=tmp/'vault.db', index=False, web=False, graph_chat=no_model())
host.open_vault(wait=True)
host.vault.stats()['encoder']
'model2vec'

Memory

Memory search returns whole sections of stored documents, plus related sections found through the vault’s graph. Each section carries its age in seconds and whether it is stale. memory_tree lists the heading tree of each stored document, memory_read returns one section by its node id, and memory_forget deletes a document. These pass straight through to the vault.

remember stores a note. Notes are ordinary documents in the same store as everything else, so a conclusion the agent reached comes back next to the evidence for it. Here two notes go in, and a search in different words finds them.

host.remember('The weekly Coles order is milk, sourdough, eggs, baby spinach and olive oil.',
              title='weekly Coles order', tags=['groceries'])
host.remember('Ceres Fair Food delivers a mixed seasonal fruit box on Thursdays.',
              title='Ceres fruit box', tags=['groceries'])

r = host.memory_search('what do I need from the supermarket', limit=2)
[s['title'] for s in r['results']], {k: r['results'][0][k] for k in ('age', 'stale')}
test_eq(len(host.memory_tree()), 2)
test_eq(len(host.memory_tree('ceres')), 1)

node = r['results'][0]['node_id']
'olive oil' in host.memory_read(node)['text'] or 'fruit box' in host.memory_read(node)['text']

With a vault behind it, tools_for offers the memory group, which NullHost does not get.

from ramabana.tools import tools_for, NullHost
have = lambda h: {getattr(t, '__name__', '') for t in tools_for(h)}
test_eq('memory_search' in have(NullHost()), False)
test_eq('memory_search' in have(host), True)

Gating what memory returns

Some of what a vault holds is personal: names, emails, phone numbers. pii sets what happens when a section that the vault hands back contains such data. off, the default, returns it as stored. redact masks the personal parts, and refuse withholds the section. pii_ner=True also catches titled names, not only what the patterns match. vishalakshi does the gating, and the host passes the setting into every vault read:

  • memory_search
  • memory_read
  • memory_tree
  • search
  • research

Either setting can be a function instead of a value. The host calls it on every read, so a settings panel can change it mid-session, and a function that raises counts as off. The hidden cells check each mode against a note with an email and a phone number in it.

Watches and reminders

Watches live in the vault, so they survive between sessions. poll runs the watches that are due. It does not rebuild the entity graph while a turn waits; when something ran, it starts connect, which rebuilds the graph in the background.

Each watch has an action and an interval. The simplest action is remind. When a reminder comes due, the vault files its text back into itself as a note, so the reminder can be found by memory search next to everything else instead of sitting in a notification queue the harness does not own.

w = host.watch('Add the weekly groceries to the Coles trolley, and a Ceres fruit box.',
               action='remind', every='1w', note='weekly shop')
[(x['id'], x['action'], int(x['every'])) for x in host.watches()]

A new watch is first due at the moment it is created, so a poll at any earlier moment finds nothing to run.

test_eq(host.vault.poll(at=w['next_run'] - 1, connect=False)['ran'], 0)

A poll after that moment runs it, and a memory search then finds what it filed.

fired = host.vault.poll(at=w['next_run'] + 1, connect=False)
test_eq(fired['ran'], 1)
test_eq(fired['results'][0]['status'], 'ok')

hit = host.memory_search('what am I supposed to be doing about groceries', limit=3)
[s['title'] for s in hit['results']]

One failing watch must not stop the others. vishalakshi’s run_watch records a failure on the watch and returns it instead of raising, so a poll over five due watches runs all five. The vault runs each watch by calling its method named after the action, so replacing host.vault.url with a function that always fails makes every url watch fail. Here one such watch sits next to a working reminder.

def _down(target, **kw): raise ConnectionError('name not resolved')
host.vault.url = _down

dead = host.watch('https://example.invalid/x', action='url', every='1d')
live = host.watch('check which day the Ceres box arrives', action='remind', every='1d')
out = host.vault.poll(at=max(dead['next_run'], live['next_run']) + 1, connect=False)
sorted(r['status'] for r in out['results'])

The failed watch is still there, rescheduled, with its last status recorded. A single failure is no reason to delete it, so the cleanup cancels both by hand.

host.unwatch(dead['id'])
host.unwatch(live['id'])
test_eq(len(host.watches()), 1)

The watch tools follow the capability, the same way the memory tools do.

tools = have(host)
test_eq({'remember', 'watch', 'list_watches', 'cancel_watch'} <= tools, True)
test_eq(any(t.startswith('watch') for t in have(NullHost())), False)

watch is one tool for every kind of watch. Its kind is url for a page, remind for a reminder, search for a web query and folder for a path. Polling is the harness’s job, not the model’s. At the start of each turn Agent.poll_watches calls host.poll() in a background thread, at most once every poll_every seconds, and reports what fired. The host returns failures instead of raising, so a turn that starts by asking what is outstanding does not end on an error.

from ramabana.tools import watch_tools
watch = next(t for t in watch_tools(host) if t.__name__ == 'watch')
print(watch('renew the domain', kind='remind', every='1w'))
host.poll()
watching remind renew the domain as 6a93995604c6, every 1w
{'checked': 1,
 'ran': 1,
 'results': [{'job_id': '1990f53ec5af',
   'kind': 'watch',
   'status': 'ok',
   'took': 0.238,
   'error': None,
   'result': {'doc_id': '36a48543e42596a7',
    'title': 'renew the domain',
    'kind': 'note',
    'nodes': 2,
    'chunks': 1,
    'changed': True,
    'source': 'note:reminder:6a93995604c6',
    'replaced': False}}],
 'reclaimed': 0,
 'dead': 0,
 'next_due': 1793079498,
 'housekeeping': {'checked': 1, 'stale': 0, 'pruned': 0, 'reasons': {}}}

Using it

from ramabana import Agent
from ramabana.vault import VaultHost

host = VaultHost(roots=['~/code/myproject'])     # the shared ~/.vishalakshi/vault.db
agent = Agent(host, model='gpt-mini')

That agent gets eight tools a LocalHost agent does not have: the four memory tools, ask_memory, and the three watch tools. Its search_code also answers from everything the vault has read. Agent itself did not change. The tools appeared because the host declares the groups they belong to.

Answering from material the model must not see

memory_search hands the model whole sections to read. That suits research and is wrong for a bank statement. To answer from a section, the section has to be in the context of whatever model runs the turn, and that model often runs on someone else’s servers.

ask, which the ask_memory tool calls, works the other way round. The vault answers and the caller gets only the answer. When the sections it retrieved hold personal information, which vishalakshi.pii detects by pattern rather than by asking a model to read them first, the vault answers with a local model under a briefing that forbids repeating any personal detail. These cells are marked eval: false, so the test suite skips them.

In a real session Agent.lend_model supplies mk_chat, and it honours the model it is asked for. The vault asks for its local model and then checks that the chat it got back runs on this machine. Without that check a factory could send a statement to a hosted API. Here the factory returns a chat that claims a remote runtime, so the vault refuses and sends nothing.

_h.mk_chat = _chat('remote', 'THIS MUST NEVER BE SENT')
_sent.clear()
r = _h.ask('what is on invoice 4471?')
test_eq(_sent, [])
test_eq(r.refused, True)
r.answer

A chat on a local runtime does receive the question, under a different briefing, and returns an answer that keeps the personal details back.

_h.mk_chat = _chat('litert', 'One invoice, 240.00 GBP, due in September. Holding back the name, '
                             'email, phone and card. Tell me if you need a total, a date, or a yes/no.')
_sent.clear()
print(ask_memory('what is on invoice 4471?'))

That reply is half of a two-step exchange. The model reading it decides what it needs and sends an instruction back, which runs the local model again over the same sections. The useful answer arrives and the private material never leaves the machine.

_h.mk_chat = _chat('litert', 'Total 240.00 GBP, due 2026-09-01.')
_sent.clear()
out = ask_memory('what is on invoice 4471?', instruction='Give me the total and the due date only.')
test_eq('Instruction from the questioner' in _sent[0][1], True)
out