Installing fossick adds a fossick command. A shell script, or an agent with only a terminal, can use the library through it. The command takes a command name and then that command’s options:
fossick search "sqlite wal mode"--n 5fossick fetch https://example.com --autofossick read-arxiv 1706.03762 --sourcefossick research "what is the raft consensus algorithm"--n 3
Most commands wrap the Python function of the same name. Their options are that function’s arguments, with underscores written as hyphens: max_pages becomes --max-pages. A command offers only some of the function’s arguments. Output is readable markdown by default. --as-json prints JSON instead, for piping into other programs. paginate-api always prints JSON. fossick <command> --help lists a command’s options. fossick alone lists the commands.
Two defaults differ from Python. fossick search reranks with BM25 (--rerank lexical), and search in Python keeps the merged order. fossick search uses --region us-en, and search in Python uses region='auto'.
def search( query:str, # search query n:int=10, # number of results region:str='us-en', # ddgs region (us-en, uk-en, ru-ru, ...) google:bool=False, # use direct Google via stealth browser instead of ddgs metasearch backend:str=None, # restrict to one ddgs backend (google, brave, duckduckgo, mojeek, yahoo, startpage) pages:int=1, # result pages to pull from each backend (a bigger candidate pool for large --n) rerank:str='lexical', # rerank the fused hits: lexical | flashrank | auto | none as_json:bool=False, # output JSON):
Search the web and print each result’s title, URL and snippet.
def read_arxiv( url:str, # arXiv URL or paper ID source:bool=False, # include full paper text chars:int=4000, # max source chars to include force:bool=False, # re-download even if cached as_json:bool=False, # output JSON):
Print an arXiv paper’s title, authors, date and summary, and optionally its full text.
def paginate_api( url:str, # API endpoint URL payload:str=None, # request body/params as a JSON string page_field:str='pageNumber', # payload key to increment per page results_field:str=None, # response key with the items list (auto-detect if None) method:str='POST', # HTTP method max_pages:int=10, # max pages to fetch):
Read every page of a JSON API and print all the items as JSON.
def collect( url:str, # URL to open in Chrome save_dir:str='.', # directory to save screenshots tout:int=None, # stop after this many seconds count:int=None, # stop after this many screenshots every_n:int=None, # auto-capture every N seconds):
Open a URL in the debug Chrome and save a screenshot each time you press the capture button.
def annotate( url:str, # URL to open in Chrome save_dir:str='.', # directory to save annotated screenshot as_json:bool=False, # output JSON list of selected elements):
Open a URL in the debug Chrome, let you click elements to number them, and save a labelled screenshot.
def pdf2nb( url:str, # PDF URL or local path path:str=None, # output notebook path (default: derived from PDF) ocr:str='auto', # OCR mode: auto | on | off force:bool=False, # re-convert even if notebook exists as_json:bool=False, # output JSON with notebook path):
Save a PDF, from a URL or a local path, as a Jupyter notebook and print its path.
def download_yt( url:str, # YouTube URL or video IDformat:str='audio', # audio | video | yt-dlp format string save_dir:str='.', # directory to save to as_json:bool=False, # output JSON with saved path):
Download a YouTube video’s audio or video and print the saved path.
def ax( url:str, # URL to open in the debug Chrome port:int=9223, # debug Chrome remote-debugging port full:bool=False, # show the full accessibility tree instead of interactive-only):
Open a URL in the debug Chrome and print a short snapshot of the controls on the page.
def research( query:str, # search query n:int=5, # number of top results to read google:bool=False, # use Google's own ranking through the stealth browser instead of ddgs metasearch region:str='auto', # ddgs 'country-lang' region, or 'auto' to read the country off the query timelimit:str=None, # d | w | m | y, only sources from the last day, week, month or year sel:str=None, # CSS selector to narrow each page before markdown chars:int=4000, # max markdown chars per source pages:int=1, # result pages to pull from each backend when searching as_json:bool=False, # output JSON):
Search, read the top results, and print them as one markdown digest that cites each source.
def shop( url:str, # store URL; the command reuses the tab the last call left open on that site search:str=None, # search the store for this term and list the products found add:str=None, # add a product by index (from a previous --search) or by title qty:int=1, # quantity to add variant:str=None, # variant/size to pick, for products that need one cart:bool=False, # print the cart and its lines fields:bool=False, # print the form fields on this page port:int=9223, # debug Chrome remote-debugging port as_json:bool=False, # output JSON):
Search a store, add a product, or read the cart in the debug Chrome, keeping the page open between calls.
The command table
CMDS maps each command name to its function. main is the entry point of the fossick script. It removes the command name from the command line, parses the remaining options against that function’s signature, and runs the function. An unknown command prints the list of commands and exits with status 1. When you supply no command, main prints the command list and exits with status 0.
Run the fossick command named on the command line.
Tests
The first cell checks that every command is registered and that the options agents rely on exist. The second runs research against live services and is skipped by the tests.
# end-to-end: the research CLI runs against live services and prints a cited corpusimport iofrom contextlib import redirect_stdoutbuf = io.StringIO()with redirect_stdout(buf): research.__wrapped__('what is the raft consensus algorithm', n=2)out = buf.getvalue()assert out.startswith('# Research:') and'http'in outprint(out[:300])
from pathlib import Path
# The CLI's `search`, `fetch`, `crawl` and `research` share names with the library's, and nbdev's doc links resolve a# bare name to the CLI page first, so index.ipynb links them by hand; the README must not send readers to cli.html.assert'cli.html#'notin Path('../README.md').read_text()