# acquire
Every method here is one
[fossick](https://github.com/vedicreader/fossick) call plus
[`Vault.add`](https://vedicreader.github.io/vishalakshi/core.html#vault.add).
fossick knows how to get past bot walls, read arXiv and YouTube, and
sniff a page’s JSON API; the vault’s job is only to file what comes back
with the provenance that explains why it is there.
------------------------------------------------------------------------
source
### md_title
``` python
def md_title(
md:str, fallback:str=''
)->str:
```
\*First markdown heading in `md`, else `fallback` — scraped
s are often junk.\*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L18"
target="_blank" style="float:right; font-size:smaller">source</a>
### clip
``` python
def clip(
s:str, n:int=120
)->str:
```
*Collapse whitespace and clip a scraped title to something a breadcrumb
can carry.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L63"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.web
``` python
def web(
query:str, # what to search for
n:int=5, # top results to read
google:bool=False, # real Google ranking via a stealth browser (slower)
chars:int=60000, # max markdown chars kept per source
verify:bool=False, **kw
)->AttrDict: # forwarded to fossick.research
```
*Search the web, read the top `n` results, and file all of them in the
vault.*
This is the loop the vault exists for: the query that found a page is
kept in its metadata, so months later `sources()` still says *why* a
document is in your corpus. Results already present are skipped rather
than duplicated, so re-running an overlapping search is cheap.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L54"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.crawl
``` python
def crawl(
start_url:str, max_pages:int=10, sel:str=None, verify:bool=False, **kw
)->L:
```
*Crawl a docs site or blog from a start URL and file every page in the
vault.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L29"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.url
``` python
def url(
url:str, # page to read
title:str=None, # defaults to the page's first heading, else its path
sel:str=None, # CSS selector to narrow the page before conversion
kind:str='web', auto:bool=True, # escalate plain -> heavy -> stealthy -> logged-in Chrome past bot walls
meta:dict=None, force:bool=False, verify:bool=False, **kw
)->dict: # forwarded to fossick.fetch
```
*Fetch one URL, convert it to markdown and file it in the vault.*
`auto=True` is the default because a bot wall returns HTTP 200 with a
challenge page, which would otherwise be indexed as if it were the
article.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L138"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.code
``` python
def code(
dir:str, types:str='.py,.js,.ts,.jsx,.tsx,.java,.go,.cs,.ruby,.php,.swift,.kt,.kts,.rs,.scala,.lua', **kw
)->L:
```
*File a source tree into the vault as `kind='code'`, so code and prose
answer one query.* This is deliberately the shallow path: files as
documents, headings from the text. For call graphs, PageRank over
symbols and `where_to_add`, use `index_code` — kosha builds an
AST-derived index that this cannot, and `federate` searches both.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L121"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.grab
``` python
def grab(
target:str, # a URL, an arXiv id, a YouTube link, a PDF, a local file or a directory
title:str=None, sel:str=None, # CSS selector, for the web cases
**kw
):
```
*File anything, by looking at what it is — the one call a CLI or an
agent needs.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L109"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.youtube
``` python
def youtube(
url:str, force:bool=False
)->dict:
```
*Read a YouTube video’s transcript and metadata into the vault.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L96"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.pdf
``` python
def pdf(
path_or_url:str, title:str=None, force:bool=False, verify:bool=False, **kw
)->dict:
```
*Read a PDF (local path or URL) into the vault, one tree node per
heading.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L86"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.arxiv
``` python
def arxiv(
id_or_url:str, save_dir:str=None, force:bool=False, verify:bool=False, **kw
)->dict:
```
*Read an arXiv paper (metadata + full text) into the vault as
`kind="arxiv"`.*
### Harvest: read a page’s API, not its HTML
Listing, product and dashboard pages render from an internal JSON API.
Reading that API is faster, paginates cleanly and survives redesigns,
where scraping the DOM does none of those. One document per harvest, one
`##` section per record — so `build_tree` gives every record its own
node and breadcrumb, and a catalogue becomes individually retrievable
rows sitting next to your notes.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L163"
target="_blank" style="float:right; font-size:smaller">source</a>
### records_md
``` python
def records_md(
recs, title_keys:tuple=('name', 'title', 'displayName', 'productName', 'label', 'sku', 'id')
)->str:
```
*Records as markdown, one `##` section per record, so each becomes its
own retrievable node.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L146"
target="_blank" style="float:right; font-size:smaller">source</a>
### records
``` python
def records(
data, min_len:int=2
)->list:
```
*The longest list of dicts inside an arbitrary JSON response.* APIs bury
their payload at different depths (`results`, `data.products.items`, a
bare array), so this walks for the longest list of dicts rather than
guessing a key name.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L222"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.add_records
``` python
def add_records(
recs:list, title:str, source:str=None, kind:str='data', force:bool=False, meta:dict=None
)->dict:
```
*File a list of dicts you already have (any API, any export) as one
document, one section each.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L188"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.harvest
``` python
def harvest(
url:str, # the page whose API you want
pattern:str='*', # which captured request URLs to keep
title:str=None, # document title; defaults to the page host + path
capture:int=None, # replay a specific endpoint from the last apis() call
pages:int=1, # pages to pull; >1 paginates the endpoint
page_field:str='page', # query/body key incremented per page
session:bool=False, # capture through the logged-in Chrome
force:bool=False, **kw
)->dict:
```
*Sniff a page’s JSON API, pull the records, and file them in the vault
as `kind='data'`.* Re-harvesting the same source replaces rather than
duplicates when `force=True`; otherwise it is a no-op, which is what
makes this safe to put behind a `watch`.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L172"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.apis
``` python
def apis(
url:str, # page to watch
pattern:str='*', # glob/regex filtering captured request URLs
session:bool=False, # capture through the logged-in debug Chrome
preview:int=240, # chars of each response shown
**kw
)->L: # forwarded to fossick.find_xhr
```
*Discover the JSON endpoints a page calls, so you can read its data
instead of its HTML.*
### Watches: keeping it current
A watch is what turns the vault from an archive into something that
stays current. `action` names an acquisition method, so anything you can
file once you can file on a schedule; `remind` writes a note instead of
fetching, which is the recurring-reminder case with no network involved.
`poll()` is the tick — cron, a scheduler, or a frontend button.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L296"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.poll
``` python
def poll(
at:float=None, limit:int=None, connect:bool=True
)->dict:
```
*Run every watch that is due. This is the tick a scheduler, a cron or a
frontend calls.*
Rebuilds the entity graph once at the end rather than per watch, because
`connect()` reads the whole store and a poll that fired five watches
would otherwise pay for it five times.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L278"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.run_watch
``` python
def run_watch(
w:dict
)->dict:
```
*Fire one watch and record the outcome.*
A failure is recorded on the row and returned, never raised: one dead
URL must not stop a polling loop from servicing every other watch.
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L273"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.pause
``` python
def pause(
watch_id:str, enabled:bool=False
):
```
*Disable (or re-enable) a watch without losing it.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L268"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.unwatch
``` python
def unwatch(
watch_id:str
):
```
*Delete a watch. The documents it already filed stay in the vault.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L261"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.watches
``` python
def watches(
due_only:bool=False, at:float=None
)->L:
```
*Every registered watch, soonest first; `due_only` keeps the ones whose
next run has arrived.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L245"
target="_blank" style="float:right; font-size:smaller">source</a>
### Vault.watch
``` python
def watch(
target:str, # URL, query, arXiv id, or the text of a reminder
action:str='url', # one of ACTIONS — what to do when it fires
every:str='1d', # interval: '30m', '6h', '1d', '1w', or seconds
note:str=None, # why you are watching
start:float=None, # first run time (epoch); defaults to now
**params
)->dict: # forwarded to the action (n=, pattern=, pages=, sel=, ...)
```
*Register a recurring job: re-read a page, re-run a search, re-harvest
an API, or remind you.*
------------------------------------------------------------------------
<a
href="https://github.com/vedicreader/vishalakshi/blob/main/vishalakshi/acquire.py#L231"
target="_blank" style="float:right; font-size:smaller">source</a>
### secs
``` python
def secs(
every
)->float:
```
*Seconds from `'30m'`, `'6h'`, `'2 days'`, `'1w'`, or a number of
seconds.*
## Try it
``` python
v = Vault(':memory:')
v.add_records([dict(sku='A1', name='Free range eggs', price=4.5),
dict(sku='B2', name='Oat milk', price=2.1)], 'dairy')
v.find('eggs')[0]['breadcrumb']
```
/Users/71293/code/personal/orgs/vishalakshi/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
'dairy › Free range eggs'
``` python
test_eq(len(records({'data': {'items': [{'a': 1}, {'a': 2}, {'a': 3}]}})), 3)
test_eq(secs('6h'), 21600); test_eq(secs('1w'), 604800); test_eq(secs(90), 90)
w = v.watch('late chunking', action='web', every='1d', n=3)
test_eq(w['params'], dict(n=3))
test_eq(len(v.watches(due_only=True)), 1)
v.unwatch(w['id'])
test_eq(len(v.watches()), 0)
```
/var/folders/kg/9vdw4mdd1fs58svgh4k1qhr09x7dqh/T/ipykernel_76490/529824922.py:7: Pandas4Warning: 'w' is deprecated and will be removed in a future version. Please use 'W' instead of 'w'.
return float(every) if isinstance(every, (int, float)) else pd.Timedelta(every).total_seconds()
/var/folders/kg/9vdw4mdd1fs58svgh4k1qhr09x7dqh/T/ipykernel_76490/529824922.py:7: Pandas4Warning: 'd' is deprecated and will be removed in a future version. Please use 'D' instead of 'd'.
return float(every) if isinstance(every, (int, float)) else pd.Timedelta(every).total_seconds()