parse
def parse(
code:NoneType=None, p:NoneType=None
):Parse code or the file p and return (tree, imports, top-level defs, defs plus call-assigned names, import paths).
core
Index your repository and its installed packages, then search them by meaning or by keyword.
This module builds the index that shows whether your code or one of your dependencies already has a helper you need. It splits source files into chunks, one per function, class or method. It stores each chunk with its metadata and an embedding. An embedding is a vector of numbers computed from the code, and chunks with similar meaning have similar vectors. A search combines two methods. Keyword search (SQLite’s FTS5) finds chunks that contain the words of the query. Vector search finds chunks whose embedding is close to the query’s embedding. No language model is involved, and everything runs locally.
There are two indexes. The repo index covers your own code and lives in .kosha/code.db inside the repository. The package index covers installed packages and lives in your user data directory (~/.local/share/kosha/env.db on most systems). Every project on the machine shares it, and a package version is indexed once. The Kosha class owns both indexes. The graph module holds the call graph built from them.
Utilities
These helpers support the indexing code below. Most users never call them directly.
parse reads Python source, from a string or a file path, and returns its syntax tree along with what kosha needs from it: a map from each imported name to its top-level package, the names of top-level functions and classes, the same plus module-level names assigned from a call, and a map from each imported name to its full dotted path. Results are cached, so calling it again on the same input is free. Source that does not parse gives None and empty collections.
The package includes a SKILL.md file that tells coding agents how to use kosha. mv_skill_md copies it to .agents/skills/kosha/SKILL.md and .claude/skills/kosha/SKILL.md in the repository, or under dir if you give one. By default it prints the target paths and copies nothing. Pass dry_run=False to write the files. Kosha(install_skill=True) and the kosha install command call it for you.
mv_skill_md
def mv_skill_md(
dry_run:bool=True, dir:NoneType=None
)->None:Copy the bundled SKILL.md into .agents/skills/kosha/ and .claude/skills/kosha/; dry_run only prints the targets.
arun runs a coroutine from ordinary code. asyncio.run fails inside a running event loop, and Jupyter notebook code runs inside one. In that case arun runs the coroutine in a worker thread with its own loop. arun returns the result wrapped in a fastcore L. The two tests below show both paths.
arun
def arun(
coro
)-><built-in function any>:Run coro to completion from synchronous code, even inside a running event loop, and return its result in an L.
async def _add(a, b): return a + b
assert arun(_add(1, 2)) == L(3)
print('arun: sync context ok')arun: sync context okasync def _test_nested():
result = arun(_add(10, 20))
assert first(result) == 30, f'got {result}'
arun(_test_nested())
print('arun: nested-loop context ok')arun: nested-loop context okmv_skill_md()mv_skill_md(dir='.')To decide which packages to index, kosha starts from the dependencies listed in your pyproject.toml and follows their own requirements. pkg_trans_deps(seeds, depth) does the walk. It returns the seed packages plus the packages they require, depth levels down. It keeps only installed packages and skips optional extras. With depth=0 it returns the seeds.
env_pkg_versions returns {package name: installed version} for the packages to index. With pyproject=True, the default, it starts from the dependencies in pyproject.toml and expands depth levels with pkg_trans_deps. It returns an empty dict when there is no repository root or no pyproject.toml. With pyproject=False it returns every installed package. The examples show both.
pkg_trans_deps
def pkg_trans_deps(
seeds:list, depth:int=2, path:list=None
)->fastcore.foundation.L:The seeds plus the installed packages they require, depth levels down, ignoring optional extras.
env_path
def env_path(
root
)->list:The site-packages folders of the environment in root: its .venv, else $VIRTUAL_ENV when that lies inside root; empty when there is none.
pyproject_deps
def pyproject_deps(
root:NoneType=None, xtras:str=None
)->list:Names of the dependencies in root/pyproject.toml, plus the comma-separated xtras groups; None when root (default: the git repository of the current folder) has no such file.
env_pkg_versions
def env_pkg_versions(
pyproject:bool=True, depth:int=1, xtras:NoneType=None, path:list=None, root:NoneType=None
)->dict:{name: version} of the packages to index: the dependencies of the pyproject.toml in root expanded depth levels, or all installed packages.
installed_packages(pyproject=True)pkg_trans_deps(['litesearch'], depth=1)['notebook', 'usearch', 'codesigs', 'model2vec', 'tokenizers', 'flashrank', 'onnxruntime', 'pdf-oxide', 'chonkie', 'litesearch', 'pillow', 'liteparse', 'spacy', 'fastlite', 'pandas', 'onnx', 'yake']d1 = pkg_trans_deps(['httpx'], depth=1)
assert 'httpx' in d1, "seeds must be in result"
assert 'httpcore' in d1, f"depth=1 should include httpcore (direct dep of httpx): {d1}"
assert list(pkg_trans_deps(['httpx'], depth=0)) == ['httpx'], "depth=0 should return only seeds"
pyp = installed_packages(pyproject=True)
if pyp:
deep = pkg_trans_deps(list(pyp), depth=2)
assert set(pyp).issubset(set(deep)), "seeds must be a subset of result"
assert len(deep) >= len(pyp), "depth=2 should expand beyond pyproject direct deps"
print(f"pkg_trans_deps ok: {len(pyp)} pyproject seeds → {len(deep)} packages at depth=2")
else:
print("pkg_trans_deps ok (no pyproject deps to test)")pkg_trans_deps ok: 5 pyproject seeds → 80 packages at depth=2d=env_pkg_versions(xtras=None); {k:d[k] for k in list(d.keys())[-5:]}{'fastlite': '0.2.4',
'pyskills': '0.0.27',
'pandas': '3.0.5',
'onnx': '1.22.0',
'yake': '0.7.3'}d=env_pkg_versions(); {k:d[k] for k in list(d.keys())[-5:]}{'fastaudit': '0.2.7',
'liteparse': '2.11.0',
'asttokens': '3.0.2',
'curl_cffi': '0.16.0',
'watchdog': '6.0.0'}Kosha
Kosha is the object you work with. Create one in your repository with Kosha(). It finds the repository root, opens the repo index in .kosha/code.db and the shared package index in the user data directory, and loads the embedding model. Its arguments let you point it somewhere else: dir sets the repository root, xdg_dir the data directory that holds the package index, and efn the embedding model, by default litesearch’s static code embedder. install_skill=True also installs SKILL.md. Set busy_timeout (in milliseconds) when several processes or threads write to the index at once, as update_pkgs(parallel=True) does.
Inside, the object holds three stores: code_st for repo chunks, env_st for package chunks, and pkg_st for package descriptions. Four plain tables sit beside them. packages records each indexed package version, and pkg_deps and repo_deps count which packages each package or repo module imports. repo_file_imports keeps the packages each repo file imports, so an incremental update_repo recounts repo_deps without parsing unchanged files.
cast_emb makes sure every vector kosha stores has the data type the search code reads back (16-bit floats), whatever the embedding model returns.
Kosha
def Kosha(
dir:pathlib.Path=None, install_skill:bool=False, xdg_dir:pathlib.Path=None,
efn:fastcore.basics.bind=<fastcore.basics.bind object at 0x7fd17bf2a600>,
busy_timeout:int=None, # ms to wait for a locked database; set it for parallel indexing
):Search index over a repository and its installed packages.
cast_emb
def cast_emb(
f, dtype:type=float16
):Wrap encoder f so the vectors it returns have the dtype the store reads back.
k=Kosha()/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdmk.emb_doc('def add(a, b): return a + b')[:5]array([[ 1.0394e-01, 4.9438e-03, -1.2079e-01, 6.4026e-02, 6.0699e-02,
-1.9971e-01, 7.8125e-02, -2.9053e-02, 1.3100e-02, 3.9398e-02,
1.1307e-02, -8.7830e-02, 1.8845e-02, 4.8309e-02, -8.6670e-02,
8.5083e-02, -1.0788e-02, 4.5197e-02, -1.8661e-02, 4.2084e-02,
2.5284e-02, -1.0582e-02, -2.4757e-03, -2.8519e-02, 3.7109e-02,
-2.4109e-02, -5.1514e-02, -3.3875e-02, -3.2867e-02, 1.3110e-01,
-7.6342e-04, -5.1331e-02, -2.2522e-02, -8.9233e-02, -1.1131e-02,
-1.0872e-02, -5.7861e-02, -8.6975e-02, -3.4790e-02, -1.3191e-02,
7.6965e-02, -3.4210e-02, -3.2532e-02, 3.8605e-02, 4.3304e-02,
-9.4360e-02, -4.0497e-02, -5.2490e-02, 2.0142e-02, -1.2463e-01,
-1.7975e-02, 4.4464e-02, 5.1788e-02, 1.8530e-01, 9.9976e-02,
7.2571e-02, 7.6233e-02, 1.0156e-01, 6.5727e-03, 1.8646e-02,
3.8109e-03, 6.3232e-02, 4.9774e-02, 4.4373e-02, 8.8196e-03,
1.0364e-01, 2.7695e-02, -1.1450e-01, 4.9866e-02, -4.1779e-02,
-6.3904e-02, 1.0948e-02, -1.0107e-01, 2.2339e-02, 1.6312e-02,
2.0538e-02, 3.3691e-02, 4.0192e-02, 4.7882e-02, 9.2590e-02,
2.7359e-02, 2.4353e-02, 8.8196e-02, -5.7312e-02, 1.2067e-01,
-1.1833e-02, -7.1533e-02, -1.3000e-01, -2.2934e-02, 2.9678e-02,
1.0124e-02, -5.9174e-02, -8.7891e-02, -1.3328e-04, 7.9468e-02,
7.2815e-02, 8.6853e-02, -2.7161e-02, 4.9713e-02, 6.1798e-02,
7.4829e-02, -2.1210e-02, -6.4087e-02, 3.5706e-02, -1.1444e-01,
-1.5778e-02, -7.2021e-02, -5.9937e-02, -4.8523e-02, -1.0864e-01,
9.4238e-02, 9.2239e-03, -7.1533e-02, -4.8828e-02, -5.2216e-02,
-8.4717e-02, -8.4290e-02, 2.0569e-02, -5.8502e-02, 2.7908e-02,
1.5350e-02, 7.9529e-02, -8.6594e-03, -2.2354e-02, 2.4918e-02,
3.7323e-02, 1.5961e-02, -6.4453e-02, 9.7168e-02, 6.5552e-02,
-1.1163e-01, -1.6711e-01, 1.5190e-02, -3.5400e-02, 1.2323e-01,
7.2693e-02, -3.7964e-02, -3.4821e-02, 1.9360e-01, -4.2725e-02,
-8.1848e-02, 4.2145e-02, -3.4302e-02, 8.3466e-03, 1.8509e-02,
-1.5808e-02, 5.4970e-03, 6.9885e-02, 5.5817e-02, 6.7749e-02,
-2.2705e-02, -3.4332e-02, -6.4758e-02, -1.4481e-02, 8.5327e-02,
8.7738e-04, -4.2328e-02, 7.2327e-02, 4.8309e-02, 9.3018e-02,
-2.1210e-02, -5.6763e-02, 8.1726e-02, 3.0060e-02, 5.3070e-02,
-1.6449e-02, -1.5579e-02, -7.2327e-03, 5.1636e-02, 1.1925e-02,
-8.4167e-02, -3.1921e-02, -7.5745e-02, 1.5356e-01, 1.4246e-01,
-9.9792e-03, 2.1652e-02, 1.0718e-01, 6.3416e-02, 6.4850e-03,
-1.2825e-02, -5.4077e-02, 7.2708e-03, -2.0111e-02, 9.6558e-02,
-4.5868e-02, 9.1187e-02, 4.5441e-02, 2.5040e-02, -1.2030e-01,
-1.0168e-01, -3.1281e-02, -2.1194e-02, -1.2444e-02, -1.3023e-02,
-1.5839e-02, 2.8793e-02, 7.4463e-03, -5.8502e-02, 5.7030e-03,
-1.7670e-02, 6.3171e-02, -6.8787e-02, 2.2537e-02, -7.9727e-03,
-7.9224e-02, 3.8269e-02, -4.7638e-02, -4.1168e-02, -3.0106e-02,
-2.2339e-02, 2.7359e-02, -1.0269e-02, -2.3163e-02, 1.7654e-02,
5.1147e-02, -5.9998e-02, 5.3497e-02, 1.8448e-02, -7.0923e-02,
2.4246e-02, 4.7760e-02, -5.9509e-02, 1.4519e-02, -3.8422e-02,
-9.5139e-03, -5.0201e-03, 8.6899e-03, -7.0801e-02, 3.2135e-02,
-1.4062e-01, -1.4954e-01, 1.9623e-02, -9.6130e-02, -1.3374e-02,
5.7922e-02, 5.0171e-02, 3.8940e-02, 5.6488e-02, 9.1629e-03,
1.3130e-02, -5.2246e-02, -6.3416e-02, 3.0396e-02, 1.9089e-02,
8.6487e-02, -3.2715e-02, -5.2734e-02, 5.3802e-02, 9.0408e-03,
-1.0633e-03, 4.0405e-02, -1.7990e-02, 5.1056e-02, 4.7089e-02,
-8.9905e-02]], dtype=float16)pkg_url returns the best web address for an installed package from its metadata: the source code link if there is one, then the repository, then the home page, then any project URL. Hand it to a web search or fetch tool when you need a package’s changelog or documentation. pkg_doc builds a description record of a package. The content is its summary and the first 2,000 characters of its README. The metadata holds name, version, keywords, requirements, summary and URL.
pkg_url
def pkg_url(
pkg:str, path:list=None
)->str | None:The best web address for installed package pkg: source code, then repository, then home page, then any project URL.
pkg_doc
def pkg_doc(
pkg, path:list=None
)->dict:A description record of pkg: summary and the start of its README as content, package details as metadata.
pkg_doc('httpx'){'content': 'The next generation HTTP client.\n<p align="center">\n <a href="https://www.python-httpx.org/"><img width="350" height="208" src="https://raw.githubusercontent.com/encode/httpx/master/docs/img/butterfly.png" alt=\'HTTPX\'></a>\n</p>\n\n<p align="center"><strong>HTTPX</strong> <em>- A next-generation HTTP client for Python.</em></p>\n\n<p align="center">\n<a href="https://github.com/encode/httpx/actions">\n <img src="https://github.com/encode/httpx/workflows/Test%20Suite/badge.svg" alt="Test Suite">\n</a>\n<a href="https://pypi.org/project/httpx/">\n <img src="https://badge.fury.io/py/httpx.svg" alt="Package version">\n</a>\n</p>\n\nHTTPX is a fully featured HTTP client library for Python 3. It includes **an integrated command line client**, has support for both **HTTP/1.1 and HTTP/2**, and provides both **sync and async APIs**.\n\n---\n\nInstall HTTPX using pip:\n\n```shell\n$ pip install httpx\n```\n\nNow, let\'s get started:\n\n```pycon\n>>> import httpx\n>>> r = httpx.get(\'https://www.example.org/\')\n>>> r\n<Response [200 OK]>\n>>> r.status_code\n200\n>>> r.headers[\'content-type\']\n\'text/html; charset=UTF-8\'\n>>> r.text\n\'<!doctype html>\\n<html>\\n<head>\\n<title>Example Domain</title>...\'\n```\n\nOr, using the command-line client.\n\n```shell\n$ pip install \'httpx[cli]\' # The command line client is an optional dependency.\n```\n\nWhich now allows us to use HTTPX directly from the command-line...\n\n<p align="center">\n <img width="700" src="https://raw.githubusercontent.com/encode/httpx/master/docs/img/httpx-help.png" alt=\'httpx --help\'>\n</p>\n\nSending a request...\n\n<p align="center">\n <img width="700" src="https://raw.githubusercontent.com/encode/httpx/master/docs/img/httpx-request.png" alt=\'httpx http://httpbin.org/json\'>\n</p>\n\n## Features\n\nHTTPX builds on the well-established usability of `requests`, and gives you:\n\n* A broadly [requests-compatible API](https://www.python-httpx.org/compatibility/).\n* An integrated command-line client.\n* HTTP/1.1 [and HTTP/2 support](https://www.python-httpx.org/http2/).\n* Standard synchronous interface,',
'metadata': {'name': 'httpx',
'version': '0.28.1',
'keywords': '',
'requires': 'certifi anyio pygments idna brotlicffi h2 httpcore socksio rich click brotli zstandard',
'summary': 'The next generation HTTP client.',
'url': 'https://github.com/encode/httpx'}}u = pkg_url('httpx')
assert u and u.startswith('http'), f'expected a URL, got {u!r}'
print('pkg_url httpx:', u)
assert pkg_url('fastcore') is not None, 'fastcore should have a URL'
print('pkg_url fastcore:', pkg_url('fastcore'))pkg_url httpx: https://github.com/encode/httpx
pkg_url fastcore: https://github.com/AnswerDotAI/fastcore/Chunk names are dotted paths like fastcore.basics.merge. To build them, kosha needs a file’s import root: the directory a Python import starts from. has_init checks whether a directory is a package (it holds an __init__.py or a compiled __init__ module), and imp_root climbs from a file to the nearest ancestor that is not a package.
imp_root
def imp_root(
p:pathlib.Path
)->pathlib.Path:The directory imports of p start from: its nearest ancestor that is not a package.
has_init
def has_init(
d:pathlib.Path
)->bool:True if d is a Python package directory: it holds __init__.py or a compiled __init__*.so.
litesearch’s file_parse splits a file into chunks, one per top-level function, class or assignment. enrich_chunks prepares them for the index. It marks each chunk as public API or not. A name is public when it is in the module’s __all__, or, when the module has no __all__, when it has no leading underscore. It records the docstring. It adds every method of a class as a chunk of its own, so a search can return Table.insert_all and not only Table. enrich_chunks reads assignment chunks only to find __all__ and does not return them.
enrich_chunks
def enrich_chunks(
content:fastcore.foundation.L
)->fastcore.foundation.L:Mark chunks as public or private, record their docstrings, add one chunk per class method, and drop assignment chunks.
Indexing packages
update_pkg indexes one installed package. It parses the package’s source files into chunks, stores them in the package index with an embedding, and records in pkg_deps how many of its modules import each other package. A package already indexed at its installed version is not parsed again, unless you pass force=True, but any of its chunks still without an embedding are embedded. update_pkgs does the same for a list. Use parallel=True to index several packages in threads at once; that needs a Kosha built with a busy_timeout, and without one it raises an error, as the test below shows. rm_pkg removes a package, or one version of it, from the index.
process_env computes embeddings for package chunks that were stored without one, for example after update_pkgs(embed=False); reembed=True recomputes all of them. It works through embed_missing, which does the same for any store and keeps its HNSW index in step. nuke deletes the repo’s .kosha directory, which holds the repo index and the call graph, and with env=True also the shared data directory holding the package and docs indexes. count_imp counts, for a list of files, how many import each top-level package.
count_imp
def count_imp(
files, own:str=''
)->collections.Counter:For a list of files, the number of files importing each top-level package other than own.
Kosha.process_env
def process_env(
pkg:str=None, # only this package's chunks; None embeds every package
reembed:bool=False, # embed chunks that already have an embedding too
batch:int=5000, # chunks embedded, committed and indexed together
progress:bool=False, # show a progress bar
)->int:Embed package chunks and descriptions that have no embedding yet, or all of them with reembed=True, and return how many chunks were embedded.
embed_missing
def embed_missing(
st, # hash-id store, HNSW-indexed or not
emb_fn, # list of texts -> vectors
where:str=None, # extra SQL condition on the rows
where_args:dict=None, # named parameters for `where`
reembed:bool=False, # embed rows that already have an embedding too
batch:int=5000, # rows embedded, committed and indexed together
progress:bool=False, # show a progress bar
)->int:Embed the rows of st that have no embedding, batch at a time, adding each batch to the HNSW index; returns how many were embedded.
Kosha.nuke
def nuke(
env:bool=False
):Delete the repo index and call graph, and with env=True the shared package and docs indexes too.
Kosha.update_pkgs
def update_pkgs(
pkgs:str | list=None, # package name or list of names to index; nothing happens if empty
embed:bool=True, # embed chunks after indexing
exts:str='.py,.js,.ts,.jsx,.tsx,.java,.go,.cs,.ruby,.php,.swift,.kt,.kts,.rs,.scala,.lua', # file extensions to include
verbose:bool=True, # print progress
force:bool=False, # reindex even if package version already loaded
parallel:bool=False, # ingest packages concurrently; requires Kosha(busy_timeout=...)
n_workers:NoneType=None, # threads when parallel=True; fastcore's default if None
chunk:int=5000, # rows per write transaction when parallel=True
**kwargs
):Index each package in pkgs, in turn or in parallel threads.
Kosha.rm_pkg
def rm_pkg(
pkg:str, ver:str=None
):Remove package pkg, or only version ver, and its chunks from the index.
Kosha.update_pkg
def update_pkg(
pkg:str, embed:bool=True,
exts:str='.py,.js,.ts,.jsx,.tsx,.java,.go,.cs,.ruby,.php,.swift,.kt,.kts,.rs,.scala,.lua', verbose:bool=True,
force:bool=False, imports:bool=False, chunk:int=5000, *, n_workers:int=None, threadpool:bool=False,
path:list=None
):Index installed package pkg: parse, store and embed its chunks, and record its imports and version.
Kosha.put_pkg_doc
def put_pkg_doc(
pkg:str, embed:bool=True
)->dict:Store the pkg_doc record of pkg in pkg_store, replacing older ones, and return it.
# update_pkgs(parallel=True) refuses without a busy timeout. Concurrent database() opens would hit
# apsw's 100ms default and raise SQLITE_BUSY inside the open hook. That drops whole packages and no
# exception reaches the caller. A silently incomplete index is worse than a loud failure.
import tempfile
from pathlib import Path
_gt = tempfile.mkdtemp(); _gk = Kosha(dir=_gt, xdg_dir=Path(_gt))
assert _gk.busy_timeout is None
try:
_gk.update_pkgs(['certifi'], parallel=True, embed=False, verbose=False)
assert False, 'parallel=True without busy_timeout must raise'
except ValueError as e: assert 'busy timeout' in str(e)
_gk.update_pkgs(['certifi'], parallel=False, embed=False, verbose=False) # serial path is unguarded
assert len(_gk.env_st()) > 0
# with a timeout set, threaded ingestion must store exactly what the serial path stores
_pk = ['certifi', 'idna', 'fastlite']
_ta, _tb = tempfile.mkdtemp(), tempfile.mkdtemp()
_ka = Kosha(dir=_ta, xdg_dir=Path(_ta), busy_timeout=30_000)
_kb = Kosha(dir=_tb, xdg_dir=Path(_tb), busy_timeout=30_000)
_ka.update_pkgs(_pk, parallel=True, embed=False, verbose=False)
_kb.update_pkgs(_pk, parallel=False, embed=False, verbose=False)
assert {r['name'] for r in _ka.pkgs(select='name')} == set(_pk), 'parallel lost a package'
assert len(_ka.env_st()) == len(_kb.env_st()), 'parallel dropped rows vs serial'
Fetching 6 files: 0%| | 0/6 [00:00<?, ?it/s]Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Fetching 6 files: 100%|##########| 6/6 [00:00<00:00, 13.65it/s]
Fetching 6 files: 100%|##########| 6/6 [00:00<00:00, 13.62it/s]
Updating packages: 0%| | 0/1 [00:00<?, ?pkg/s]
parse files from certifi: 0%| | 0/3 [00:00<?, ?it/s]
parse files from certifi: 100%|##########| 3/3 [00:00<00:00, 1817.03it/s]
Updating packages: 100%|##########| 1/1 [00:00<00:00, 46.53pkg/s]
parse files from idna: 0%| | 0/10 [00:00<?, ?it/s]
parse files from idna: 100%|##########| 10/10 [00:00<00:00, 130.09it/s]
parse files from certifi: 0%| | 0/3 [00:00<?, ?it/s]
parse files from certifi: 100%|##########| 3/3 [00:00<00:00, 964.06it/s]
parse files from fastlite: 0%| | 0/4 [00:00<?, ?it/s]
parse files from fastlite: 100%|##########| 4/4 [00:00<00:00, 489.96it/s]
Updating packages: 0%| | 0/3 [00:00<?, ?pkg/s]
parse files from certifi: 0%| | 0/3 [00:00<?, ?it/s]
parse files from certifi: 100%|##########| 3/3 [00:00<00:00, 3014.59it/s]
parse files from idna: 0%| | 0/10 [00:00<?, ?it/s]
parse files from idna: 100%|##########| 10/10 [00:00<00:00, 158.93it/s]
Updating packages: 67%|######6 | 2/3 [00:00<00:00, 8.15pkg/s]
parse files from fastlite: 0%| | 0/4 [00:00<?, ?it/s]
parse files from fastlite: 100%|##########| 4/4 [00:00<00:00, 479.12it/s]
Updating packages: 100%|##########| 3/3 [00:00<00:00, 9.63pkg/s]Kosha finds the project’s own environment once, in Kosha.env_path: the .venv in its root, else $VIRTUAL_ENV when that lies inside the root. kosha reads packages from that environment and not from the interpreter that runs kosha. A package installed only in the project’s .venv appears in status and update_pkg indexes it, with the version and file paths of that copy. When there is no such environment, env_path is empty and kosha uses the running interpreter.
from fastcore.test import test_eqsync(embed=False) stores chunks for keyword search only. Indexing the same package again with embeddings on embeds those chunks too, even though the package is already loaded at its installed version. A first sync with embed=False does not leave the package invisible to vector search.
_et = Path(tempfile.mkdtemp()); _er, _esp = _fake_proj(_et, 'proj')
_fake_dist(_esp, 'kkledger', '1.0', 'def parse_ledger(path): return open(path).read().split()\n\ndef sum_ledger(rows): return sum(map(float, rows))\n')
_ek = Kosha(dir=_er, xdg_dir=_et/'xdg')
_ek.update_pkg('kkledger', embed=False, verbose=False)
_nulls = lambda: (len(_ek.env_st(where="package='kkledger' AND embedding IS NULL")), len(_ek.pkg_st(where='embedding IS NULL')))
_unembedded = _nulls()
_ek.update_pkg('kkledger', verbose=False)
test_eq((_unembedded, _nulls()), ((2, 1), (0, 0)))
_unembedded, _nulls()Chunks stored without an embedding before that behaviour existed need a backfill. process_env embeds every such chunk, batch at a time. It commits each batch and adds it to the HNSW index in the same step. An interrupted backfill resumes where it stopped. Vector search sees each chunk as soon as the chunk has a vector:
_fake_dist(_esp, 'kkinvoice', '2.0', 'def render_invoice(order): return f"Invoice {order}"\n\ndef email_invoice(addr, pdf): pass\n\ndef void_invoice(n): return -n\n')
_ek.update_pkg('kkinvoice', embed=False, verbose=False)
_idx = lambda: _ek.envdb.get_index(_ek.env_st.name)
_size = _idx().size
_done = _ek.process_env(batch=2)
test_eq((_done, _idx().size - _size, len(_ek.env_st(where='embedding IS NULL'))), (3, 3, 0))
_nearest(_ek, 'render an invoice for an order')Indexing the repository
update_repo indexes your own code. It walks the repository, skipping folders whose names start with . or _ and the build and dist folders, and skipping files whose names start with . or _ as well as setup.py and conftest.py. It compares each file’s modification time with the time recorded at the last run. It re-parses only files that are new or changed, and deletes chunks of files that are gone. It also records in repo_deps how many repo files import each installed package. force=True re-indexes every file. files= limits the run to the listed paths, as watch_repo does for each change event. update_repo deletes the chunks of a listed path that no longer exists. process_repo embeds repo chunks that were stored without an embedding.
Kosha.update_repo
def update_repo(
dir:pathlib.Path=None, # directory to index; defaults to repo root
embed:bool=True, # embed chunks after parsing
exts:str='.py,.js,.ts,.jsx,.tsx,.java,.go,.cs,.ruby,.php,.swift,.kt,.kts,.rs,.scala,.lua', # file extensions to index
verbose:bool=True, # print progress
force:bool=False, # re-index every file, changed or not
files:list=None, # only these paths, as from a watch event; listed paths that are gone are dropped
*, skip_file_re:str='^[.]|^(?:setup\\.py|conftest\\.py)$',
skip_folder_re:str='^[.]|^(?:tests?|examples?|docs?|build|dist|_proc|_docs|_site|_site_libs|node_modules|__pycache__)$',
func:type=Path, path:pathlib.Path | str='.', recursive:bool=True, maxdepth:int=None, symlinks:bool=True,
file_glob:str=None, file_re:str=None, path_glob:str=None, path_re:str=None, folder_re:str=None,
skip_file_glob:str=None, ret_folders:bool=False, sort:bool=True, types:str | list=None
):Index new and changed repo files under dir and drop the chunks of deleted ones.
Kosha.process_repo
def process_repo(
content:NoneType=None, reembed:bool=False, *, embed:bool=True, emb_fn:NoneType=None,
hash_id_columns:tuple=('content',), parallel:bool=False, chunk:int=1000
):Embed repo chunks that have no embedding yet, or all of them with reembed=True.
# update_repo(files=...) as watch_repo calls it: only listed paths are touched, and a listed path that is gone is dropped.
_wt = Path(tempfile.mkdtemp()); _wk = Kosha(dir=_wt, xdg_dir=_wt)
for _n in 'abc': (_wt/f'{_n}.py').write_text(f'def {_n}(): pass\n')
_wk.update_repo(embed=False, verbose=False)
(_wt/'a.py').write_text('def a2(): pass\n'); (_wt/'b.py').unlink(); (_wt/'c.py').write_text('def c2(): pass\n')
_wk.update_repo(files=[_wt/'a.py', _wt/'b.py'], embed=False, verbose=False)
assert set(L(_wk.code_st(select='metadata')).map(lambda r: jl(r['metadata'])['name'])) == {'a2','c'}
(_wt/'sub').mkdir(); (_wt/'sub'/'d.py').write_text('def d(): pass\n')
_wk.update_repo(dir=_wt/'sub', embed=False, verbose=False)
assert {'a2','c','d'} <= set(L(_wk.code_st(select='metadata')).map(lambda r: jl(r['metadata'])['name'])), 'indexing a subdirectory keeps records outside it'A file that fails to parse, such as an empty notebook, is skipped and the rest are indexed. An edit above a function moves it without changing its text. update_repo records the new lineno. A lookup by name reads the json_extract(metadata,'$.name') index and does not scan the store. update_repo drops the chunks of a deleted file only when the file lies under dir. An index written before CODE_FORMAT 2 can hold old line numbers and lacks the stored imports of each file. When Kosha opens such an index, it marks every indexed file for re-parsing and sets the format to 2. The next update_repo re-parses those files.
_mt = Path(tempfile.mkdtemp()); _mk = Kosha(dir=_mt, xdg_dir=_mt)
(_mt/'m.py').write_text('def f(): pass\n\ndef g(): pass\n'); (_mt/'empty.ipynb').write_text('')
_mk.update_repo(embed=False, verbose=False, exts=code_exts+',.ipynb')
assert _mk.code_st(select='count(*) n')[0]['n'] == 2, 'an empty notebook beside it stops nothing'
(_mt/'m.py').write_text('X = 1\ndef f(): pass\n\ndef g2(): pass\n')
_mk.update_repo(embed=False, verbose=False)
_lines = lambda k: {jl(r['metadata'])['name']: jl(r['metadata'])['lineno'] for r in k.code_st(select='metadata')}
assert _lines(_mk)['f'] == 2, _lines(_mk)
_plan = _mk.codedb.q(f"explain query plan select id from [{_mk.code_st.name}] where json_extract(metadata, '$.name') = ?", ['f'])
assert any('_name' in r['detail'] for r in _plan), _plan
_mk.codedb.execute(f"update [{_mk.code_st.name}] set metadata=json_set(metadata,'$.lineno',99)")
_mk.codedb.execute('pragma user_version=0')
_mk2 = Kosha(dir=_mt, xdg_dir=_mt); _mk2.update_repo(embed=False, verbose=False)
assert _lines(_mk2)['f'] == 2, _lines(_mk2)
_fixed = _lines(_mk2)
(_mt/'m.py').write_text('from os import path\n__all__ = ["path"]\n'); _mk2.update_repo(embed=False, verbose=False)
assert _lines(_mk2) == {}, 'a file whose definitions all moved away keeps none of them'
_deps = lambda: {r['to_pkg']: r['n_files'] for r in _mk2.code_rd()}
(_mt/'a.py').write_text('import json\n'); (_mt/'b.py').write_text('import json\n'); _mk2.update_repo(embed=False, verbose=False)
assert _deps()['json'] == 2
(_mt/'a.py').write_text('import json\nx = 1\n'); os.utime(_mt/'a.py', (2e9, 2e9)); _mk2.update_repo(embed=False, verbose=False)
assert _deps()['json'] == 2, 'editing one importer keeps the other in the count'
for _n in 'ab': (_mt/f'{_n}.py').write_text('x = 1\n'); os.utime(_mt/f'{_n}.py', (2e9+1, 2e9+1))
_mk2.update_repo(embed=False, verbose=False)
assert 'json' not in _deps(), 'a package no file imports any more is dropped'
_fixedWhen you upgrade a package, the index can hold rows for both versions. prune_old_versions keeps only the highest version of one package. It applies to the package’s chunks, its description and its packages row. prune_old_pkgs does the same for every package. pkgs_in_env returns the indexed packages whose version matches the installed one.
The cells after this one are checks and examples: module names built from the import root, the file lists the repo walk produces with and without the strict skip rules, and direct queries on the stores. Cells marked eval: false index real packages. The docs build does not run them.
Kosha.pkgs_in_env
def pkgs_in_env(
pyproject:bool=False, depth:int=1
)->list:Indexed packages whose indexed version is the one installed.
Kosha.prune_old_pkgs
def prune_old_pkgs():Keep only the highest indexed version of every package.
Kosha.prune_old_versions
def prune_old_versions(
pkg:str
):Keep only the highest indexed version of pkg.
# prune_old_versions drops chunks by their `package` column; metadata.name is the chunk's symbol name.
_vt = Path(tempfile.mkdtemp()); _vk = Kosha(dir=_vt, xdg_dir=_vt)
_vk.update_pkg('certifi', embed=False, verbose=False)
process_content(_vk.env_st, [dict(content='def old(): pass', package='certifi', metadata=dict(name='old', version='0.0.1'))], embed=False)
_vk.pkgs.insert(dict(name='certifi', version='0.0.1'))
_n = len(_vk.env_st(where="package='certifi'"))
_vk.prune_old_versions('certifi')
assert len(_vk.env_st(where="package='certifi'")) == _n-1
assert [r['version'] for r in _vk.pkgs(where="name='certifi'")] == [_dist('certifi').version]Every project on the machine shares the package index. The index can therefore hold packages that no environment has any more, such as a dependency you removed or every package of a project folder you deleted. prune_env removes them. It judges what is installed by the *.dist-info folders in each site-packages folder it knows. It lists folders and never imports a package. It knows these folders: the folders the indexed chunks were read from, the nearest .venv above an editable install’s source, and the current project’s environment. A folder that no longer exists holds nothing.
A package keeps its chunks while any of those folders holds it at any version. A package has one set of chunks in the index, from whichever version was synced last. A chunk’s version records the version at which the chunk was first stored. An unchanged function from an old release is still current code. A chunk’s version does not identify stale chunks. Each packages row, one per indexed version, stays while some folder holds that exact version. Removing a package also drops its description and import counts. prune_env then runs VACUUM once to shrink the file. After that it rebuilds the chunk index and the package-description index once each. Each of these two indexes is an approximate nearest-neighbour index of the HNSW kind.
Kosha.prune_env
def prune_env(
dry_run:bool=False
)->dict:Remove the packages no known site-packages folder holds and the versions none holds, compact env.db, and return what was removed.
installed_dists
def installed_dists(
root
)->dict:{normalized name: {version: metadata folder}} of the distributions installed in site-packages folder root; empty when it is gone.
Two projects share one index here. The first has kktax 1.0 and kkfx 1.0 installed, the second has kktax 2.0, so the index holds three package versions:
_pt = Path(tempfile.mkdtemp()); _pa, _spa = _fake_proj(_pt, 'old'); _pb, _spb = _fake_proj(_pt, 'new')
_fake_dist(_spa, 'kktax', '1.0', 'def vat(x): return x*0.2\n')
_fake_dist(_spb, 'kktax', '2.0', 'def vat(x, rate=0.2): return x*rate\n')
_fxdi = _fake_dist(_spa, 'kkfx', '1.0', 'def convert_currency(amount, rate): return amount*rate\n')
_kold, _knew = Kosha(dir=_pa, xdg_dir=_pt/'xdg'), Kosha(dir=_pb, xdg_dir=_pt/'xdg')
for _p in ('kktax', 'kkfx'): _kold.update_pkg(_p, verbose=False)
_knew.update_pkg('kktax', verbose=False)
_vers = lambda: sorted((r['name'], r['version']) for r in _kold.pkgs(select='name, version'))
_vers()Uninstalling kkfx from the first project deletes its dist-info folder. A dry run then lists kkfx 1.0 as stale and removes nothing. kktax 1.0 is not listed. The second project has a newer kktax, and the first project’s site-packages still holds 1.0.
_fxdi.delete()
_dry = _kold.prune_env(dry_run=True)
test_eq(_pruned(_dry), (['kkfx'], [('kkfx', '1.0')]))
test_eq(len(_vers()), 3)
_dryThe real run removes kkfx and its chunks, keeps both kktax versions, and leaves the HNSW index holding exactly the vectors still in the table. Vector search keeps working:
test_eq(_pruned(_kold.prune_env()), _pruned(_dry))
_n_vec = lambda: (_kold.envdb.get_index(_kold.env_st.name).size, len(_kold.env_st(where='embedding IS NOT NULL')))
test_eq((_vers(), len(_kold.env_st(where="package='kkfx'"))), ([('kktax', '1.0'), ('kktax', '2.0')], 0))
test_eq(*_n_vec())
_nearest(_kold, 'value added tax rate')Deleting the second project removes its site-packages folder, the only place kktax 2.0 was installed. The 2.0 record goes. kktax keeps its chunks because the first project still has it:
_pb.delete()
test_eq(_pruned(_kold.prune_env()), ([], [('kktax', '2.0')]))
test_eq(*_n_vec())
_vers(), len(_kold.env_st(where="package='kktax'"))import kosha.core as _kc, inspect_f = Path(inspect.getfile(_kc))
_anchor = _f.parent
while has_init(_anchor): _anchor = _anchor.parent
_mod = '.'.join(_f.relative_to(_anchor).with_suffix('').parts)
assert _mod == 'kosha.core', f'expected kosha.core, got {_mod}'
print('mod_name anchor fix: ok —', _mod)mod_name anchor fix: ok — kosha.coredir2files(repo_root(), None, skip_folder_re,exts=code_exts)[Path('/Users/71293/code/personal/orgs/kosha/kosha/__init__.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/_modidx.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/cli.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/core.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/graph.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/mcp.py')]dir2files(repo_root(), strict_skip_file_re, skip_folder_re,exts=code_exts)[Path('/Users/71293/code/personal/orgs/kosha/kosha/cli.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/core.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/graph.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/mcp.py')]pkg='fastcore'
k._is_pkg_ingested(pkg){'name': 'fastcore', 'version': '2.1.17'}k.env_st(select='content, embedding, json_extract(metadata, "$.mod_name","$.docstring", "$.public_api") as pa',
where='package="httpx" and json_extract(metadata, "$.public_api")=1', limit=1)[{'content': 'def close(self) -> None:\n pass',
'embedding': b'\xd90\xd0%c\xac\xd8-\x91(\xac\x17\xee\x1f \xa9\x87\xa6N\x1a\xc5,\x11(\xec)o)-$k*\xf1\xa8^\xa6p\xa9\xa1\xa1\x0e\xa5\xe5\xa0\xa4+\'*\xe9(h\xa2\x90,\xe8\x9d\x80\xa3\x16\x91X\'5/\x82\x1bC&\x99\xa5\xf7\xa6\xc5\x17\xa9"="\xf2\xaa~(\n\xa4\xf8%\x9e\xa8\xa5\xa8\xd8/\xb9\xa9\xd7+@\xb1\xf2\xa7\xfe-\xa9\xa9g#\x85\xae\x91\xad\xb3\x1c\xf9\xb1\x18.\x06 9\xaa\x10/l%F\x1d\x9c*\xac\xac\x98)\xaf+\x00\xa6G,\x80\xac\xe7/\xfb\x1cI\xa7\xf4\xacu+\xd9\xac\xfc\xa1\xd3\xa8\xf4(U\x94\xd0,\x88\xac0\xac\xdd\xb0y-\xf7*N\xa6\xb6\x116,(&\xfa+u\xa4Y\xadY&\x14+&\xa8\x8e.B0Q)\x9f*B(\x95\xac\xfa(>%\xf7\xa40\xa4\xae$\x99.G\xab\x1f+\xba\xabE*\x18\xa7\xd9\xa8\x10)}\xadS\xa4J\x1f@/\xfb\xae\xd2\xae\xe5&\xd9*\x831y\'\xea\xa1\xb3\xa7E\xae\x9f\xa6\xcb,\xde\xa7Y%=\xa4\xbc\xaa\xb0\x99\xb6\xab\xc4-O&J\xa8\xe5.O.\xd1)\xcb\xa0\x83,\x04\x9a\x87\xad\xf1\'m)\xb7-,\xaaE)\x9d$\xf3)\xed\xa3\x9a)\x7f&\x1e-Q)\x97#\x88+x\xaa\xe3\xa2\xd7"n\'.0\xeb\x18\x10\xabN+\x9d+\x03!\x82\xaa\xc6\x94\xb4&/\xa9\xd7\x9b\x81\xac\xb0%_\'H\xa5\x85\'\x92-\xfc)\\\xb3\xb0(4\xad\xbb\xaa\xf9\x99/\xa9\xdd \xbc\xa0\xa8(\x17\xa4H\xaa\x8b\xadv(\x8f\xaft%3\xaf\xe3\x99-\'Z\xac],\xa6\xa4\xc0\x9d\xad.\xda\xa8 $\xfe(#.G+\x1d\xac\xf7\xaa\xa4-\xf314\xa8w\xaeJ\xad\xb2&\x0c-`\xaa\xc5\x9d\xda\x1e\xd7\x1f\x17\xa0\xb8,|-\xbc\xaa\xc0%M\x9f\x97/\xaf\xadb(\x19,\xc3.+.r,9.p(\x0e&\x0c,\x06)!\xa8B _\xab\xe6\xa8K*\x1a\x18\x89(\x0c#[&\xc5\xafJ,\x13\xa4h&\xc6*\x85\xaa',
'pa': '["httpx._transports.base.BaseTransport.close",null,true]'}]k.pkgs(order_by='name', limit=1)[{'name': 'FlashRank',
'version': '0.2.10',
'summary': 'Ultra lite & Super fast SoTA cross-encoder based re-ranking for your search & retrieval pipelines.',
'uploaded_at': '2026-08-03 02:24:34'}]k.update_pkg('protobuf')updating pkg: protobuf ...
package {'name': 'protobuf', 'version': '7.35.1'} already loaded.k.update_pkg('fastlite', force=True)updating pkg: fastlite ...
package {'name': 'fastlite', 'version': '0.2.4'} already loaded.
forcing an update to pkg {'name': 'fastlite', 'version': '0.2.4'}
parse files from fastlite: 0%| | 0/4 [00:00<?, ?it/s]
parse files from fastlite: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 261.68it/s]{'changed': 64, 'same': 0, 'removed': 0}{}k.update_repo(skip_file_glob='*.ipynb', force=True)syncing files [Path('/Users/71293/code/personal/orgs/kosha/kosha/cli.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/core.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/graph.py'), Path('/Users/71293/code/personal/orgs/kosha/kosha/mcp.py')] .....
parse files from /Users/71293/code/personal/orgs/kosha: 0%| | 0/4 [00:00<?, ?it/s]
parse files from /Users/71293/code/personal/orgs/kosha: 75%|██████████████████████████████████████████████▌ | 3/4 [00:00<00:00, 28.69it/s]
parse files from /Users/71293/code/personal/orgs/kosha: 100%|██████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 36.64it/s]{'changed': 163, 'same': 0, 'removed': 0}
synced repok.code_st(select='json_extract(metadata, "$.mod_name","$.docstring", "$.public_api") as pa',
where='json_extract(metadata, "$.public_api")=1')k.code_st(select='content',where='json_extract(metadata, "$.mod_name")="kosha.core.Kosha.__init__"')[{'content': "def __init__(self, dir: Path = None, install_skill: bool = False, xdg_dir: Path = None, efn=static_code_embedder):\n\t\tself.root, self.xdg, self.efn = Path(dir or repo_root() or '.'), xdg_dir or xdg_data_home(), efn()\n\t\tself.emb_doc, self.emb_query = doc_encoder(self.efn), query_encoder(self.efn)\n\t\tif install_skill: mv_skill_md(dir=self.root, dry_run=False)\n\t\tself.cp, self.ce = self.root.joinpath('.kosha','code.db'), self.xdg.joinpath('kosha','env.db')\n\t\tfor p in (self.cp, self.ce): p.parent.mkdir(parents=True, exist_ok=True)\n\t\tself.codedb, self.envdb, kw = database(self.cp), database(self.ce), dict(hash=True,ann=True)\n\t\tself.code_st,self.env_st = self.codedb.get_store(path=str,**kw),self.envdb.get_store(package=str,**kw)\n\t\tself.pkg_st, self.pkgs = self.envdb.get_store('pkg_store', **kw), self.envdb.t.packages\n\t\tself.env_pd, self.code_rd = self.envdb.t.pkg_deps, self.codedb.t.repo_deps\n\t\tself.pkgs.create(name=str, version=str, summary=str, uploaded_at=float, pk=['name','version'],\n\t\t\t defaults=dict(uploaded_at='CURRENT_TIMESTAMP'), if_not_exists=True)\n\t\tself.env_pd.create(from_pkg=str, to_pkg=str, n_modules=int, if_not_exists=True, pk=['from_pkg','to_pkg'])\n\t\tself.code_rd.create(from_module=str, to_pkg=str, n_files=int, if_not_exists=True, pk=['from_module','to_pkg'])"}]k.update_pkgs(set(env_pkg_versions()))loading pkgs {'notebook', 'usearch', 'codesigs', 'model2vec', 'anyio', 'ddgs', 'scrapling', 'execnb', 'Jinja2', 'nbdev', 'watchfiles', 'onnxruntime', 'setuptools', 'mcp', 'FlashRank', 'ghapi', 'pdf_oxide', 'tornado', 'litesearch', 'jupyter_builder', 'fossick', 'pillow', 'html2text', 'spacy', 'fastlite', 'jupyterlab', 'pyskills', 'pandas', 'diskcache', 'onnx', 'PyYAML', 'yake', 'yt-dlp', 'tokenizers', 'notebook_shim', 'fastprogress', 'curl_cffi', 'fastgit', 'fastcdp', 'jupyter_server', 'build', 'astunparse', 'readability-lxml', 'chonkie', 'fastcore', 'pyan3', 'jupyterlab_server', 'python-fasthtml', 'fastaudit', 'liteparse', 'asttokens', 'watchdog'} ...
Updating packages: 0%| | 0/52 [00:00<?, ?pkg/s]
Updating packages: 4%|███▋ | 2/52 [00:00<00:03, 14.11pkg/s]updating pkg: notebook ...
package {'name': 'notebook', 'version': '7.6.1'} already loaded.
updating pkg: usearch ...
package {'name': 'usearch', 'version': '2.26.0'} already loaded.
updating pkg: codesigs ...
Updating packages: 8%|███████▍ | 4/52 [00:00<00:03, 13.57pkg/s]
Updating packages: 12%|███████████▏ | 6/52 [00:00<00:03, 14.48pkg/s]package {'name': 'codesigs', 'version': '0.0.3'} already loaded.
updating pkg: model2vec ...
package {'name': 'model2vec', 'version': '0.8.2'} already loaded.
updating pkg: anyio ...
package {'name': 'anyio', 'version': '4.14.2'} already loaded.
updating pkg: ddgs ...
package {'name': 'ddgs', 'version': '9.14.4'} already loaded.
updating pkg: scrapling ...
Updating packages: 15%|██████████████▉ | 8/52 [00:00<00:02, 15.25pkg/s]
Updating packages: 19%|██████████████████▍ | 10/52 [00:00<00:02, 15.77pkg/s]package {'name': 'scrapling', 'version': '0.4.12'} already loaded.
updating pkg: execnb ...
package {'name': 'execnb', 'version': '0.2.9'} already loaded.
updating pkg: Jinja2 ...
package {'name': 'Jinja2', 'version': '3.1.6'} already loaded.
updating pkg: nbdev ...
package {'name': 'nbdev', 'version': '3.3.3'} already loaded.
updating pkg: watchfiles ...
Updating packages: 23%|██████████████████████▏ | 12/52 [00:00<00:02, 16.35pkg/s]
Updating packages: 27%|█████████████████████████▊ | 14/52 [00:00<00:02, 16.40pkg/s]package {'name': 'watchfiles', 'version': '1.2.0'} already loaded.
updating pkg: onnxruntime ...
package {'name': 'onnxruntime', 'version': '1.28.0'} already loaded.
updating pkg: setuptools ...
package {'name': 'setuptools', 'version': '83.0.0'} already loaded.
updating pkg: mcp ...
package {'name': 'mcp', 'version': '2.0.0'} already loaded.
updating pkg: FlashRank ...
Updating packages: 31%|█████████████████████████████▌ | 16/52 [00:00<00:02, 17.29pkg/s]
Updating packages: 35%|█████████████████████████████████▏ | 18/52 [00:01<00:01, 17.82pkg/s]package {'name': 'FlashRank', 'version': '0.2.10'} already loaded.
updating pkg: ghapi ...
package {'name': 'ghapi', 'version': '2.0.5'} already loaded.
updating pkg: pdf_oxide ...
package {'name': 'pdf_oxide', 'version': '0.3.77'} already loaded.
updating pkg: tornado ...
package {'name': 'tornado', 'version': '6.5.7'} already loaded.
updating pkg: litesearch ...
Updating packages: 38%|████████████████████████████████████▉ | 20/52 [00:01<00:01, 18.13pkg/s]
Updating packages: 42%|████████████████████████████████████████▌ | 22/52 [00:01<00:01, 18.37pkg/s]package {'name': 'litesearch', 'version': '0.1.5'} already loaded.
updating pkg: jupyter_builder ...
package {'name': 'jupyter_builder', 'version': '1.2.1'} already loaded.
updating pkg: fossick ...
package {'name': 'fossick', 'version': '0.0.15'} already loaded.
updating pkg: pillow ...
package {'name': 'pillow', 'version': '12.3.0'} already loaded.
updating pkg: html2text ...
Updating packages: 46%|████████████████████████████████████████████▎ | 24/52 [00:01<00:01, 18.40pkg/s]
Updating packages: 50%|████████████████████████████████████████████████ | 26/52 [00:01<00:01, 17.92pkg/s]package {'name': 'html2text', 'version': '2025.4.15'} already loaded.
updating pkg: spacy ...
package {'name': 'spacy', 'version': '3.8.14'} already loaded.
updating pkg: fastlite ...
package {'name': 'fastlite', 'version': '0.2.4'} already loaded.
updating pkg: jupyterlab ...
package {'name': 'jupyterlab', 'version': '4.6.2'} already loaded.
updating pkg: pyskills ...
Updating packages: 54%|███████████████████████████████████████████████████▋ | 28/52 [00:01<00:01, 17.30pkg/s]
Updating packages: 58%|███████████████████████████████████████████████████████▍ | 30/52 [00:01<00:01, 17.75pkg/s]package {'name': 'pyskills', 'version': '0.0.27'} already loaded.
updating pkg: pandas ...
package {'name': 'pandas', 'version': '3.0.5'} already loaded.
updating pkg: diskcache ...
package {'name': 'diskcache', 'version': '5.6.3'} already loaded.
updating pkg: onnx ...
package {'name': 'onnx', 'version': '1.22.0'} already loaded.
updating pkg: PyYAML ...
parse files from PyYAML: 0%| | 0/1 [00:00<?, ?it/s]
parse files from PyYAML: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 803.81it/s]
Updating packages: 62%|███████████████████████████████████████████████████████████ | 32/52 [00:01<00:01, 14.82pkg/s]
Updating packages: 65%|██████████████████████████████████████████████████████████████▊ | 34/52 [00:02<00:01, 15.39pkg/s]{'changed': 0, 'same': 0, 'removed': 0}
updating pkg: yake ...
package {'name': 'yake', 'version': '0.7.3'} already loaded.
updating pkg: yt-dlp ...
package {'name': 'yt-dlp', 'version': '2026.7.4'} already loaded.
updating pkg: tokenizers ...
package {'name': 'tokenizers', 'version': '0.23.1'} already loaded.
updating pkg: notebook_shim ...
Updating packages: 69%|██████████████████████████████████████████████████████████████████▍ | 36/52 [00:02<00:00, 16.44pkg/s]package {'name': 'notebook_shim', 'version': '0.2.4'} already loaded.
updating pkg: fastprogress ...
package {'name': 'fastprogress', 'version': '1.1.6'} already loaded.
updating pkg: curl_cffi ...
package {'name': 'curl_cffi', 'version': '0.16.0'} already loaded.
updating pkg: fastgit ...
package {'name': 'fastgit', 'version': '0.1.1'} already loaded.
updating pkg: fastcdp ...
package {'name': 'fastcdp', 'version': '0.0.7'} already loaded.
Updating packages: 75%|████████████████████████████████████████████████████████████████████████ | 39/52 [00:02<00:00, 17.66pkg/s]
Updating packages: 81%|█████████████████████████████████████████████████████████████████████████████▌ | 42/52 [00:02<00:00, 18.41pkg/s]updating pkg: jupyter_server ...
package {'name': 'jupyter_server', 'version': '2.20.0'} already loaded.
updating pkg: build ...
package {'name': 'build', 'version': '1.5.0'} already loaded.
updating pkg: astunparse ...
package {'name': 'astunparse', 'version': '1.6.3'} already loaded.
updating pkg: readability-lxml ...
Updating packages: 85%|█████████████████████████████████████████████████████████████████████████████████▏ | 44/52 [00:02<00:00, 18.58pkg/s]
Updating packages: 88%|████████████████████████████████████████████████████████████████████████████████████▉ | 46/52 [00:02<00:00, 18.63pkg/s]package {'name': 'readability-lxml', 'version': '0.8.4.1'} already loaded.
updating pkg: chonkie ...
package {'name': 'chonkie', 'version': '1.7.0'} already loaded.
updating pkg: fastcore ...
package {'name': 'fastcore', 'version': '2.1.17'} already loaded.
updating pkg: pyan3 ...
package {'name': 'pyan3', 'version': '2.6.2'} already loaded.
updating pkg: jupyterlab_server ...
Updating packages: 92%|████████████████████████████████████████████████████████████████████████████████████████▌ | 48/52 [00:02<00:00, 18.92pkg/s]
Updating packages: 96%|████████████████████████████████████████████████████████████████████████████████████████████▎ | 50/52 [00:02<00:00, 19.06pkg/s]package {'name': 'jupyterlab_server', 'version': '2.28.0'} already loaded.
updating pkg: python-fasthtml ...
package {'name': 'python-fasthtml', 'version': '0.14.11'} already loaded.
updating pkg: fastaudit ...
package {'name': 'fastaudit', 'version': '0.2.7'} already loaded.
updating pkg: liteparse ...
package {'name': 'liteparse', 'version': '2.11.0'} already loaded.
updating pkg: asttokens ...
Updating packages: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:02<00:00, 19.16pkg/s]
Updating packages: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:02<00:00, 17.35pkg/s]package {'name': 'asttokens', 'version': '3.0.2'} already loaded.
updating pkg: watchdog ...
package {'name': 'watchdog', 'version': '6.0.0'} already loaded.k.env_st(where='embedding is NULL', select='count(*)')[{'count(*)': 0}]k.process_env()k.update_pkgs(['fastcore','litesearch','fastlite','apsw','ghapi','httpx'])loading pkgs ['fastcore', 'litesearch', 'fastlite', 'apsw', 'ghapi', 'httpx'] ...
Updating packages: 0%| | 0/6 [00:00<?, ?pkg/s]
Updating packages: 33%|████████████████████████████████▋ | 2/6 [00:00<00:00, 17.10pkg/s]updating pkg: litesearch ...
package {'name': 'litesearch', 'version': '0.1.5'} already loaded.
updating pkg: fastcore ...
package {'name': 'fastcore', 'version': '2.1.17'} already loaded.
updating pkg: apsw ...
package {'name': 'apsw', 'version': '3.53.4.0'} already loaded.
updating pkg: fastlite ...
Updating packages: 67%|█████████████████████████████████████████████████████████████████▎ | 4/6 [00:00<00:00, 18.56pkg/s]
Updating packages: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 18.58pkg/s]
Updating packages: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 18.39pkg/s]package {'name': 'fastlite', 'version': '0.2.4'} already loaded.
updating pkg: httpx ...
package {'name': 'httpx', 'version': '0.28.1'} already loaded.
updating pkg: ghapi ...
package {'name': 'ghapi', 'version': '2.0.5'} already loaded.Searching
A query can carry filters as key:value tokens next to the words, as in merge dicts package:fastcore path:basics. parseq splits a query into the plain text and a dict of filters. The keys are file, path, package, lang and type, and the short forms people tend to type first also work: pkg, dir, folder, ext, plurals such as packages, and so on. A value can list several options separated by commas. A trailing !, as in package!:fastcore, is accepted and means the same as without it, because every filter already excludes everything that does not match. A token that looks like a filter but has an unknown key, such as libr:litesearch, is reported under _unknown, so a caller can warn about it instead of silently searching for the literal text. If a query holds nothing but filters, the whole query is kept as the search text.
parseq
def parseq(
q:str
)->tuple:Split query q into its plain text and a dict of key:value filters, with unknown keys under _unknown.
# aliases and the `!` suffix resolve to the canonical key
for tok in ['package:litesearch', 'pkg:litesearch', 'pkgs:litesearch', 'package!:litesearch', 'pkg!:litesearch']:
raw, fs = parseq(f'how do I add indexes with {tok}')
assert fs['package'] == ['litesearch'], (tok, dict(fs))
assert raw == 'how do I add indexes with', (tok, raw)
assert parseq('atomic save dir:xtras')[1]['path'] == ['xtras']
assert parseq('atomic save ext:py')[1]['lang'] == ['py']
assert parseq('search package:httpx,requests')[1]['package'] == ['httpx', 'requests']
# an unknown filter key is reported, not silently searched as text
assert parseq('add indexes with libr:litesearch')[1]['_unknown'] == ['libr']
assert parseq('add indexes with module:foo')[1]['_unknown'] == ['module']
# ...and known keys, urls and bare prose never trip it
for q in ['add indexes with package:litesearch', 'see https://example.com/x', 'how do I add indexes']:
assert '_unknown' not in parseq(q)[1], qparseq('how do I merge dicts package:fastcore path:data*')('how do I merge dicts',
defaultdict(list, {'package': ['fastcore'], 'path': ['data*']}))assert parseq('stripe payments package:fasthtml file:routes*') == ('stripe payments', {'package': ['fasthtml'], 'file': ['routes*']})
assert parseq('package:monsterui package:fasthtml create a table') == ('create a table', {'package': ['monsterui', 'fasthtml']})
assert parseq('no filters here') == ('no filters here', {})
assert parseq('type:FunctionDef lang:py') == ('type:FunctionDef lang:py', {'type': ['FunctionDef'], 'lang': ['py']})
assert parseq('package:only')[0] == 'package:only'
# plural forms + comma-separated values
assert parseq('merge dicts packages:fastcore,litesearch') == ('merge dicts', {'package': ['fastcore', 'litesearch']})
assert parseq('search paths:basics,core,data') == ('search', {'path': ['basics', 'core', 'data']})
assert parseq('query packages:fastcore,litesearch paths:basics,core') == ('query', {'package': ['fastcore', 'litesearch'], 'path': ['basics', 'core']})
assert parseq('langs:py,js files:routes*,api*') == ('langs:py,js files:routes*,api*', {'lang': ['py', 'js'], 'file': ['routes*', 'api*']})
print('parse_query: all assertions pass')parse_query: all assertions passfrom litesearch import pre, cleanassert clean('with_uv') == 'with_uv'filt2wh turns those filters into an SQL WHERE clause for one of the two indexes. file and path values are globs (* and ?) matched anywhere in the path. lang takes an extension or a language name (py or python). type matches the chunk’s syntax type, such as FunctionDef or ClassDef. package applies only to the package index, so on the repo index a filter dict holding only a package gives None, meaning no filter.
filt2wh
def filt2wh(
filters:dict, store:str='code'
)->str | None:An SQL WHERE clause for filters (from parseq) on the code or env store, or None if nothing applies.
sql_str
def sql_str(
v
)->str:v as an SQLite string literal: single-quoted, with each internal quote doubled.
# a `lang:` filter is answered from idx_store_lang on both stores, not by reading every row's metadata JSON
for _db, _s in ((k.codedb, 'code'), (k.envdb, 'env')):
_plan = ' '.join(r['detail'] for r in _db.q(f"explain query plan select rowid from store where {filt2wh(parseq('x lang:rs')[1], _s)}"))
assert 'idx_store_lang' in _plan, _planfilt2wh(parseq('how do I merge dicts package:fastcore path:data*')[1], 'env')"(json_extract(metadata, '$.path') like '%data%') AND (package='fastcore')"filt2wh({'type': ['FunctionDef']}, 'code')"(json_extract(metadata, '$.type') = 'FunctionDef')"assert filt2wh({'file': ['routes*']}, 'code') == "(path LIKE '%routes%')"
assert filt2wh({'path': ['api/*']}, 'code') == "(path LIKE '%api/%')"
assert filt2wh({'lang': ['py']}, 'code') == "(json_extract(metadata, '$.lang') = '.py')"
assert filt2wh({'lang': ['python']}, 'code') == "(json_extract(metadata, '$.lang') = '.py')"
assert filt2wh({'package': ['fasthtml']}, 'env') == "(package='fasthtml')"
assert filt2wh({'type': ['FunctionDef']}, 'code') == "(json_extract(metadata, '$.type') = 'FunctionDef')"
assert filt2wh({}, 'code') is None
assert filt2wh({}, 'env') is None
assert filt2wh({'package': ['fasthtml']}, 'code') is None
print('filters_to_where: all assertions pass')filters_to_where: all assertions passfilt2wh quotes every filter value with sql_str. A value is always one SQL string literal, whatever quotes it holds. A value that holds a quote of either kind matches that literal text and never changes the clause. sql_str raises ValueError for a value that holds a NUL character. Ordinary values give the same text as before.
import sqlite3
from fastcore.test import expect_fail_qdb = sqlite3.connect(':memory:')
_qdb.execute('create table store(package text, metadata text, path text)')
_qdb.executemany('insert into store values (?, ?, ?)', [('x', '{}', 'a.py'), ('y', '{}', 'b.py')])
_qrows = lambda fs: [r[0] for r in _qdb.execute(f"select package from store where {filt2wh(fs, 'env')}")]
_qodd = ['x"\' OR 1=1 --', "it's", parseq('q package:x"\' OR 1=1 --')[1]['package'][0]]
test_eq([_qrows({'package': [v]}) for v in _qodd], [[], [], []])
test_eq(_qrows({'package': ['x']}), ['x'])
test_eq(filt2wh({'package': ['x', 'y']}, 'env'), "(package='x' OR package='y')")
with expect_fail(ValueError, contains='NUL'): sql_str('a\x00b')
filt2wh({'package': [_qodd[0]]}, 'env')env_context searches the package index and repo_context searches the repo index. Each parses the filters, embeds the query, runs one search that combines keyword and vector matching, and returns hits with their content and metadata. Use one of them when you know which index you want. Kosha.context, defined in the graph module, searches both indexes, reranks the hits for code and adds call-graph details. Use Kosha.context by default.
Both take limit, an extra SQL where, and rerank=True to reorder the hits with a flashrank cross-encoder, a small model that scores each hit against the query. When reranking, kosha first retrieves a wider pool of candidates (fanout) for the reranker to choose from. It then cuts the result to limit. pkgs2consider returns the names of indexed packages, or with sys_wide=False only those indexed at the version your project uses.
Kosha.pkgs2consider
def pkgs2consider(
sys_wide:bool=True
)->set:Names of indexed packages, or with sys_wide=False only those indexed at the installed version.
Kosha.env_context
def env_context(
q:str, # query to search (supports key:value filters)
emb_q:str=None, # query to embed; defaults to bare query after filter parsing
columns:str='content,metadata,package', # comma separated columns string to return from search
where:str=None, # additional where clause to filter search results
ann:bool=True, # HNSW Approximate Nearest Neighbors (ann) search; defaults to True
rerank:bool=False, # reorder the merged hits with a flashrank cross-encoder
rerank_model:str=None, # flashrank model name (None -> fast default)
**kw
)->fastcore.foundation.L: # additional args to pass to db.searchKeyword and vector search over the package index, with optional key:value filters.
fanout
def fanout(
kw, limit, rerank
):Set the retrieval limit in kw, widened to at least RERANK_FANOUT when reranking.
# `sys_wide=False` keeps a package indexed at the installed version, also when the index holds two versions of it
from types import SimpleNamespace
_fk = SimpleNamespace(env_path=None, root=None, pkgs=lambda select: [dict(name='a', version='1'), dict(name='b', version='1'), dict(name='b', version='2')])
_epv, env_pkg_versions = env_pkg_versions, lambda path=None, root=None: {'a': '1', 'b': '2', 'c': '1'}
try:
test_eq(Kosha.pkgs2consider(_fk, sys_wide=False), {'a', 'b'})
test_eq(set(Kosha.pkgs2consider(_fk)), {'a', 'b'})
finally: env_pkg_versions = _epvKosha.watch_repo
def watch_repo(
dir:pathlib.Path=None, **kw
):Watch the repo and re-index files as they change, until you press Ctrl-C.
Kosha.awatch_repo
async def awatch_repo(
dir:pathlib.Path=None, **kw
):Watch dir asynchronously and re-index the files in each change event.
Kosha.repo_context
def repo_context(
q:str, # query to search (supports key:value filters)
emb_q:str=None, # query to embed; defaults to bare query after filter parsing
columns:str='content,path,metadata', # columns to return
where:str=None, # extra SQL filter
ann:bool=True, # HNSW Approximate Nearest Neighbors (ann) search; defaults to True
rerank:bool=False, # reorder the merged hits with a flashrank cross-encoder
rerank_model:str=None, # flashrank model name (None -> fast default)
**kw
)->fastcore.foundation.L: # additional args to pass to db.searchKeyword and vector search over the repo index, with optional key:value filters.
import inspect# `wide` was accepted and ignored; the FTS leg always ORs the query terms through litesearch's `pre`.
assert 'wide' not in inspect.signature(Kosha.env_context).parameters
assert 'wide' not in inspect.signature(Kosha.repo_context).parameterswatch_repo keeps the repo index current while you work. It blocks and watches the repository with watchfiles. After every change it calls update_repo with the changed files. It runs until you press Ctrl-C. awatch_repo is the same loop as a coroutine, for code that already runs an event loop.
One row shape
A tool that shows kosha hits next to hits from other sources, such as documentation or notes, needs every hit in one format. code_row turns one hit from repo_context or env_context into an AttrDict with six keys: source (which index the hit came from), ref (the dotted name, or the path), title, where (path:line), text (the first 600 characters of the chunk) and open. open is a hint for how to open the hit, such as symbol('pkg.mod'). rows runs the searches and returns the hits in that format. It searches the repo index by default. Pass env=True to add the package index.
Kosha.rows
def rows(
q:str, # query, with kosha's key:value filters
limit:int=12, repo:bool=True, # the repo index
env:bool=False, # the installed-package index
**kw
)->fastcore.foundation.L: # forwarded to repo_context / env_contextSearch hits for q from the repo and, with env=True, the package index, each as a code_row.
code_row
def code_row(
r:dict, # a hit from `repo_context` or `env_context`
leg:str='repo', # which index it came from
)->fastcore.basics.AttrDict:Hit r as an AttrDict with the keys source, ref, title, where, text and open.
pkg_context searches package descriptions, the records pkg_doc builds, to find a package by what it does. It reads the pkg_store table, which update_pkg fills with one record per package through put_pkg_doc.
Kosha.pkg_context
def pkg_context(
q:str, # the search query
limit:int=10, # number of hits to return
rerank:bool=False, # reorder the merged hits with a flashrank cross-encoder
rerank_model:str=None, # flashrank model name (None -> fast default)
)->fastcore.foundation.L:Keyword and vector search over the package descriptions in pkg_store.
# update_pkg fills pkg_store with one description per package; reindexing replaces it rather than adding another.
_pt = Path(tempfile.mkdtemp()); _pkk = Kosha(dir=_pt, xdg_dir=_pt)
_pkk.update_pkg('certifi', verbose=False); _pkk.update_pkg('certifi', verbose=False, force=True)
assert len(_pkk.pkg_st()) == 1
_r = _pkk.pkg_context('Mozilla CA bundle certificates', limit=3)
assert _r and _r[0]['metadata']['name'] == 'certifi', _r# `cast_emb` pins the stored dtype to what `Database.search` reads back (float16), so a
# custom `efn` emitting float32 can no longer corrupt the vector leg silently.
emb = k.emb_doc(['def f(): pass'])
qry = k.emb_query('a function')
assert emb.dtype == DTYPE == np.float16, emb.dtype
assert qry.dtype == DTYPE, qry.dtype
f32 = cast_emb(lambda txts, **kw: np.zeros((len(txts), 4), dtype=np.float32))
assert f32(['a','b']).dtype == DTYPE# rerank fans out to RERANK_FANOUT candidates, then the cross-encoder cuts back to `limit`.
assert fanout({}, 5, True)['limit'] == RERANK_FANOUT
assert fanout({}, 50, True)['limit'] == 50
assert fanout({}, 5, False)['limit'] == 5
# the live leg needs flashrank's model on disk; skip rather than fail on a cold/offline machine.
try: rr = k.env_context('atomic file write', limit=5, rerank=True)
except Exception as e: print('rerank skipped:', type(e).__name__)
else:
assert len(rr) <= 5
wide = k.env_context('atomic file write', limit=RERANK_FANOUT)
assert {r['content'] for r in rr} <= {r['content'] for r in wide}, 'reranked hits must come from the fanout'
print(rr.attrgot('metadata').attrgot('mod_name'))Docs index
Code chunks suit code, but a README or a docs/ folder reads better as whole sections. kosha keeps a third index for such prose, built on litesearch’s Index, which splits documents by their headings and can search, list and return whole sections. It lives in the shared data directory (docs.db), next to the package index, and is created the first time you use it.
add_docs adds a folder of documents, a single file, or a dict of {title: text}, and returns the number of chunks added. add_pkg_docs adds a package’s full README. docs_context searches the docs index. By default it returns ranked sections, each with a node_id you can pass to k.docs.read to get the whole section; sections=False returns the raw chunks. The docs index uses the same embedding model as the code indexes, so only one model is loaded.
Kosha.docs_context
def docs_context(
q:str, # the search query
limit:int=5, # hits to return
sections:bool=True, # ranked sections (with `read` handles) vs raw chunks
**kw
)->fastcore.foundation.L: # forwarded to Index.sections / Index.searchSearch the docs index, returning whole sections by default or raw chunks with sections=False.
Kosha.add_pkg_docs
def add_pkg_docs(
pkg:str
)->int:Add the full README of installed package pkg to the docs index.
Kosha.add_docs
def add_docs(
src, # a docs directory, a file, or {title: text}
**kw
)->int: # forwarded to Index.addAdd documents to the docs index and return the number of chunks added.
Kosha.docs
def docs()->litesearch.api.Index:The docs index, a litesearch Index over READMEs and other prose, opened on first use.
from tempfile import TemporaryDirectory
with TemporaryDirectory() as td:
_dk = Kosha(dir=Path(td), xdg_dir=Path(td))
n = _dk.add_docs({'install': '# Install\n\nRun `uv add kosha` to install kosha into your project.',
'search': '# Search\n\nCall `env_context` to search installed packages by meaning.'})
assert n > 0, 'docs index ingested nothing'
assert len(_dk.docs) == n
hits = _dk.docs_context('how do I install this', sections=False)
assert hits, 'no docs hits'
assert 'uv add kosha' in hits[0]['content']
secs = _dk.docs_context('search installed packages')
assert secs and _dk.docs.read(secs[0]['node_id'])['text']
print(repr(_dk.docs))Index(path='/tmp/tmprazt862s/kosha/docs.db', chunks=2, docs=2)/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)k.env_context('payment page with fasthtml package:python-fasthtml', limit=10)[{'rowid': 35281, 'content': 'def account_management(sess):\n user_email = sess[\'auth\']\n user_payments = payments("email=?", (user_email,))\n # Create table rows for each payment\n payment_rows = []\n for payment in user_payments:\n action_button = ""\n if payment.payment_status == \'paid\':\n action_button = Button("Request Refund", hx_post=f"/refund?checkout_sid={payment.checkout_session_id}",hx_target="#refund-status")\n elif payment.payment_status == \'refunded\': action_button = "Refunded"\n \n # Add row to table\n payment_rows.append(\n Tr(*map(Td, (payment.created_at, payment.amount, payment.payment_status, action_button))))\n \n # Create payment history table\n payment_history = Table(\n Thead(Tr(*map(Th, ("Date", "Amount", "Status", "Action")))),\n Tbody(*payment_rows))\n \n return Titled(\n "Account Management",\n Div(H2(f"Account: {user_email}"),\n H3("Payment History"),\n payment_history,\n Div(id="refund-status"), # Target for refund status messages\n A("Back to Home", href="/")))', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/stripe_otp.py', 'uploaded_at': 1785635980.530908, 'name': 'account_management', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 185, 'end_lineno': 211, 'mod_name': 'fasthtml.stripe_otp.account_management', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': None, 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.48605406284332275, '_rrf_score': 0.016666666666666666}, {'rowid': 35278, 'content': 'def success(sess, checkout_sid:str): \n # Get payment record from database (saved in the webhook)\n payment = payments[checkout_sid]\n\n if not payment or payment.payment_status != \'paid\': \n return Titled("Error", P("Payment not found"))\n\n return Titled(\n "Success",\n Div(H2("Payment Successful!"),\n P(f"Thank you for your purchase, {sess[\'auth\']}"),\n P(f"Amount Paid: ${payment.amount / 100:.2f}"),\n P(f"Status: {payment.payment_status}"),\n P(f"Transaction ID: {payment.checkout_session_id}"),\n A("Back to Home", href="/")))', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/stripe_otp.py', 'uploaded_at': 1785635980.530908, 'name': 'success', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 133, 'end_lineno': 147, 'mod_name': 'fasthtml.stripe_otp.success', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': None, 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.5118991136550903, '_rrf_score': 0.01639344262295082}, {'rowid': 35227, 'content': 'def show(*s, **kwargs):\n "Same as fasthtml.components.show, but also adds `htmx.process()`"\n if IN_NOTEBOOK: return _show(*s, Script(\'if (window.htmx) htmx.process(document.body)\'), **kwargs)\n return _show(*s, **kwargs)', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/jupyter.py', 'uploaded_at': 1785635980.4980536, 'name': 'show', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 41, 'end_lineno': 44, 'mod_name': 'fasthtml.jupyter.show', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': 'Same as fasthtml.components.show, but also adds `htmx.process()`', 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.5131857395172119, '_rrf_score': 0.016129032258064516}, {'rowid': 35216, 'content': 'def _app_factory(*args, **kwargs):\n "Creates a FastHTML or FastHTMLWithLiveReload app instance"\n from .live_reload import FastHTMLWithLiveReload\n if kwargs.pop(\'live\', False): return FastHTMLWithLiveReload(*args, **kwargs)\n kwargs.pop(\'reload_attempts\', None)\n kwargs.pop(\'reload_interval\', None)\n return FastHTML(*args, **kwargs)', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/fastapp.py', 'uploaded_at': 1785635980.4972112, 'name': '_app_factory', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 19, 'end_lineno': 25, 'mod_name': 'fasthtml.fastapp._app_factory', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': 'Creates a FastHTML or FastHTMLWithLiveReload app instance', 'public_api': False}, 'package': 'python-fasthtml', '_dist': 0.5457382202148438, '_rrf_score': 0.015873015873015872}, {'rowid': 35228, 'content': 'def render_ft(**kw):\n "Call once in a notebook or solveit dialog to auto-render components with HTMX support"\n @patch\n def _repr_html_(self:FT):\n scr_proc = Script(\'if (window.htmx) htmx.process(document.body)\')\n return to_xml(Div(self, scr_proc, **kw))', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/jupyter.py', 'uploaded_at': 1785635980.4980536, 'name': 'render_ft', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 47, 'end_lineno': 52, 'mod_name': 'fasthtml.jupyter.render_ft', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': 'Call once in a notebook or solveit dialog to auto-render components with HTMX support', 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.5844467282295227, '_rrf_score': 0.015625}, {'rowid': 35112, 'content': 'def show(ft, *rest, iframe=False, height=\'auto\', style=None):\n "Renders FT Components into HTML within a Jupyter notebook."\n from IPython import display\n if isinstance(ft, str): ft = Safe(ft)\n if rest: ft = (ft,)+rest\n res = to_xml(ft)\n if iframe:\n style = "border: none; " + (style or "")\n cfg = dict(frameborder=0, width=\'100%\', height=height, style=style)\n res = to_xml(Iframe(srcdoc=res, **cfg))\n with warnings.catch_warnings():\n warnings.simplefilter("ignore", UserWarning)\n display.display(display.HTML(res))', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/components.py', 'uploaded_at': 1785635980.4775577, 'name': 'show', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 126, 'end_lineno': 138, 'mod_name': 'fasthtml.components.show', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': 'Renders FT Components into HTML within a Jupyter notebook.', 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.6196398138999939, '_rrf_score': 0.015384615384615385}, {'rowid': 35280, 'content': 'async def refund(sess, checkout_sid:str):\n # Get payment record from database\n payment = payments[checkout_sid]\n \n if not payment or payment.payment_status != \'paid\':\n return P("Error: Payment not found or not eligible for refund")\n \n try:\n # Get the payment intent ID from the checkout session\n checkout_session = stripe.checkout.Session.retrieve(checkout_sid)\n \n # Process the refund\n refund = stripe.Refund.create(payment_intent=checkout_session.payment_intent, reason="requested_by_customer")\n \n # Update payment status in database\n payments.update(Payment(checkout_session_id= checkout_sid, payment_status="refunded"))\n \n return Div(\n P("Refund processed successfully!"),\n P(f"Refund ID: {refund.id}"))\n \n except Exception as e: return P(f"Refund failed: {str(e)}")', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/stripe_otp.py', 'uploaded_at': 1785635980.530908, 'name': 'refund', 'lang': '.py', 'type': 'AsyncFunctionDef', 'lineno': 160, 'end_lineno': 181, 'mod_name': 'fasthtml.stripe_otp.refund', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': None, 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.6311675310134888, '_rrf_score': 0.015151515151515152}, {'rowid': 35160, 'content': 'def is_full_page(req, resp):\n "Check if response should be rendered as full page or fragment"\n if resp and any(getattr(o, \'tag\', \'\')==\'html\' for o in resp): return True\n return \'hx-request\' in req.headers and \'hx-history-restore-request\' not in req.headers', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/core.py', 'uploaded_at': 1785635980.496347, 'name': 'is_full_page', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 418, 'end_lineno': 421, 'mod_name': 'fasthtml.core.is_full_page', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': 'Check if response should be rendered as full page or fragment', 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.6416814923286438, '_rrf_score': 0.014925373134328358}, {'rowid': 35123, 'content': 'def _get_htmx(h):\n res = {k:h.get(v.lower(), None) for k,v in htmx_hdrs.items()}\n return HtmxHeaders(**res)', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/core.py', 'uploaded_at': 1785635980.496347, 'name': '_get_htmx', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 77, 'end_lineno': 79, 'mod_name': 'fasthtml.core._get_htmx', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': None, 'public_api': False}, 'package': 'python-fasthtml', '_dist': 0.6486342549324036, '_rrf_score': 0.014705882352941176}, {'rowid': 35130, 'content': 'def HtmxResponseHeaders(**kwargs):\n "HTMX response headers"\n res = tuple(HttpHeader(_to_htmx_header(k), v) for k,v in kwargs.items())\n return res[0] if len(res)==1 else res', 'metadata': {'path': '/Users/71293/code/personal/orgs/kosha/.venv/lib/python3.13/site-packages/fasthtml/core.py', 'uploaded_at': 1785635980.496347, 'name': 'HtmxResponseHeaders', 'lang': '.py', 'type': 'FunctionDef', 'lineno': 130, 'end_lineno': 133, 'mod_name': 'fasthtml.core.HtmxResponseHeaders', 'package': 'python-fasthtml', 'version': '0.14.11', 'docstring': 'HTMX response headers', 'public_api': True}, 'package': 'python-fasthtml', '_dist': 0.6605545282363892, '_rrf_score': 0.014492753623188406}]Keeping syncs fast
Indexing is meant to fit in an edit and search loop. Repo and package updates touch only what changed. The call graph is optional, and sync(graph=False) skips it when you need only search. The default graph update reads changed Python files in one fast pass. The slower full extraction runs only with graph_mode='full'. PageRank is recomputed only when the number of edges has changed by more than 10%. graph_metrics=False stops the recomputation.
_root = Path(tempfile.mkdtemp())/'proj'; (_root/'.git').mkdir(parents=True)
(_root/'pyproject.toml').write_text('[project]\nname = "proj"\nversion = "0"\ndependencies = ["kkfake"]\n')
_sp = _root/'.venv/lib/python3.13/site-packages'; (_sp/'kkfake').mkdir(parents=True)
(_sp/'kkfake'/'__init__.py').write_text('def hello(): return 1\n')
(_sp/'kkfake-1.0.dist-info').mkdir()
(_sp/'kkfake-1.0.dist-info'/'METADATA').write_text('Metadata-Version: 2.1\nName: kkfake\nVersion: 1.0\nAuthor-email: a@b.co\n')
_kp, _cwd = Kosha(dir=_root, xdg_dir=_root/'xdg'), os.getcwd()
os.chdir(_root)
try:
test_eq(_kp.env_path, [str(_sp.resolve())])
_kp.update_pkg('kkfake', embed=False, verbose=False)
_md = L(_kp.env_st(where="package='kkfake'")).map(lambda r: jl(r['metadata']) if isinstance(r['metadata'], str) else r['metadata'])
finally: os.chdir(_cwd)
test_eq(set(_md.attrgot('version')), {'1.0'})
assert all(Path(m['path']).resolve().is_relative_to(_sp.resolve()) for m in _md)
_md.map(lambda m: (m['mod_name'], m['version']))