core

fetch, markdown, crawl, and specialized readers

fetch() reads URLs with plain HTTP or a browser-backed tier. to_md() cleans HTML. The module also crawls sites, discovers XHR calls, and reads YouTube, arXiv, and GitHub sources.

---

source

fossick_cache

def fossick_cache(
    path:NoneType=None
):

Get cache path for name (e.g. ‘arxiv’ or ‘fetch’)

---

source

Path.mk_bytes

def mk_bytes(
    data, mode:int=511, uid:int=-1, gid:int=-1
):

Make all parent dirs of self, and write data

---

source

syncy

def syncy(
    coro, tout:int=60
):

Run coro on the shared background loop from sync code, even inside a running loop.

Fetch

Responses expose url, status, html, data (JSON), and xhr. Use heavy, stealthy, session, or auto to select the fetch tier.

---

source

to_md

def to_md(
    res_or_html, # Page dict (from fetch/crawl) or raw HTML string
    sel:str=None, # CSS selector to extract before conversion; returns '' if no match
    multi:bool=False, # Return all selector matches joined
    wrap_tag:str=None, # Wrap each multi-result in <wrap_tag>...</wrap_tag>; only used when multi=True
    ignore_links:bool=True, ignore_images:bool=False,
    readability:bool=True, # False keeps the whole document, for local files and saved pages
)->str:

Convert a Page dict or HTML string to clean markdown

---

source

html2md

def html2md(
    s:str, ignore_links:bool=True, ignore_images:bool=False,
    readability:bool=True, # Extract the article (right for fetched pages); False converts the whole document
):

Convert s from HTML to markdown

---

source

html_title

def html_title(
    s:str
)->str:

The document’s own title: <title> if present, else the first <h1>

to_md/html2md run readability’s article extractor by default — right for a fetched page, since it drops nav, ads and furniture. For a document somebody saved deliberately, pass readability=False: it converts the whole document, minus script/style/noscript/template and comments.

SSRF guard

check_url blocks private/link-local/metadata targets unless FOSSICK_ALLOW_PRIVATE or allow_private=True.

---

source

check_url

def check_url(
    url:str, # the URL about to be fetched
    allow_private:bool=None, # None = honour $FOSSICK_ALLOW_PRIVATE (default: off)
)->str:

Raise BlockedURL if url targets a private/link-local/metadata host.

---

source

BlockedURL

def BlockedURL(
    *args, **kwargs
):

A fetch target that points inside the network fossick is running in.

---

source

get_page

def get_page(
    url, method:str='GET', payload:NoneType=None, heavy:bool=False, stealthy:bool=False, session:bool=False,
    allow_private:NoneType=None, **kw
)->Response:

Fetch url. session=True (or a port int) drives the persistent debug Chrome via CDP, reusing its logged-in cookies.

Every fetch in fossick arrives here, which is why the private-address check lives here and not in fetch: crawl, replay_xhr, download_files and http_get would each have to remember.

---

source

http_page

def http_page(
    url, method:str='GET', payload:NoneType=None, verify:bool=False, **kw
)->Response:

Call self as a function.

---

source

BrowserUnavailable

def BrowserUnavailable(
    *args, **kwargs
):

scrapling’s headless/stealth browsers could not be loaded.

r1=get_page('https://httpbun.com/get', verify=False)
r1.__dict__.keys()

from fastcore.test import test_fail

try: get_page('https://httpbun.com/put', method='PUT', verify=False)
except AssertionError as e: test_fail(str(e) == 'Only GET and POST methods are supported')

http_page('https://httpbun.com/put', method='PUT', verify=False).status

---

source

browser_session

def browser_session(
    stealthy:bool=False, headless:bool=True, **init_kw
):

Open ONE browser and yield a fetch(url, sel=None, **) func that reuses it across calls (skips per-URL relaunch).

---

source

fetch

def fetch(
    url:str, # URL to fetch
    sel:str=None, # CSS selector to extract (None = full page)
    method:str='GET', # HTTP method; 'POST' sends payload as JSON body
    payload:dict=None, # POST body (JSON) or GET query params
    heavy:bool=False, # Full JS rendering via headless browser
    stealthy:bool=False, # Anti-bot stealth fetcher (Cloudflare etc.)
    capture_xhr:bool=False, # Capture XHR calls made by the page (only works with heavy or stealthy)
    session:bool=False, # Route through the persistent debug Chrome (reuses its logged-in cookies); True or a port int
    auto:bool=False, # Auto-escalate plain->heavy->stealthy->session on bot-block detection
    allow_private:bool=None, # Allow localhost/intranet targets (default: refuse; $FOSSICK_ALLOW_PRIVATE=1)
    **kw
)->Response: # Extra kwargs passed to scrapling (e.g. verify, headers)

Fetch url, return a Response with .html (raw or sel-extracted), .xhr, .status.

Bot walls

_blocked() detects Cloudflare/Anubis-style interstitials. auto=True escalates tiers until one is clean.

# A site behind Cloudflare: a plain fetch hits the bot wall, and _blocked() flags it for escalation
cf = fetch('https://www.scrapingcourse.com/cloudflare-challenge', verify=False)
assert cf.status == 403, f'expected a Cloudflare 403 challenge, got {cf.status}'
assert _blocked(cf) is True            # fetch(auto=True) escalates from here (heavy -> stealthy -> session)
print('cloudflare:', cf.status, '| blocked?', _blocked(cf))

# A site behind Anubis (proof-of-work bot wall): _blocked() now recognizes it, so fetch(auto=True) escalates
an = fetch('https://anubis.techaro.lol/', verify=False)
assert 'making sure' in an.html.lower(), 'expected the Anubis challenge page'
assert _blocked(an) is True            # Anubis markers are in _BLOCK_MARKERS -> auto escalates past this wall
print('anubis wall detected by _blocked?', _blocked(an))

# regression: a normal page that merely *mentions* captcha must NOT read as a bot wall.
# Wikipedia embeds hCaptcha config keys in its page JS -- the word is there, but there's no challenge widget.
wiki = fetch('https://en.wikipedia.org/wiki/Anubis', verify=False)
assert wiki.status == 200 and 'captcha' in wiki.html.lower()   # the word IS present on the page...
assert _blocked(wiki) is False   # ...but _BLOCK_MARKERS matches captcha *widgets* (g-recaptcha/h-captcha/...), not the word

# fetch(auto=True): walks plain->heavy->stealthy->session, stops at the first unblocked tier, tags .tier
plain = fetch('https://example.com', auto=True, verify=False)
assert plain.status == 200 and plain.tier == 'plain'   # a static page needs no escalation
print('example.com ->', plain.tier)

# browser_session(): open ONE browser, reuse it across fetches (skips per-URL relaunch)
with browser_session() as bf:
    a = bf('https://example.com')
    b = bf('https://httpbin.org/html')
assert a.status == 200 and b.status == 200
assert len(a.html) > 0 and len(b.html) > 0

# fetch(session=True): route through the persistent debug Chrome, reusing its logged-in cookies
pg = fetch('https://httpbin.org/headers', session=True)
assert pg.status == 200
print(pg.html[:200])

---

source

crawl

def crawl(
    start_url:str, # URL to start from
    sel:str=None, # CSS selector to extract per page
    follow_sel:str='a[href]', # CSS selector for links to follow
    same_domain:bool=True, # Only follow links on same domain
    max_pages:int=10, # Max pages to visit
    delay:float=0, # Seconds to wait between requests (polite crawling)
    heavy:bool=False, # Full JS rendering via headless browser
    stealthy:bool=False, # Anti-bot stealth fetcher (Cloudflare etc.)
    reuse:bool=True, # For heavy/stealthy crawls, keep one browser open across pages
    **kw
)->list: # Extra kwargs passed to scrapling (e.g. verify, timeout)

Crawl from start_url, following follow_sel links, return list of Page dicts

---

source

fetch_all

def fetch_all(
    urls:list, # URLs to fetch
    sel:str=None, # CSS selector to extract per page (None = full page)
    concurrency:int=8, # Max parallel fetches
    heavy:bool=False, # Full JS rendering via headless browser
    stealthy:bool=False, # Anti-bot stealth fetcher (Cloudflare etc.)
    **kw
)->list: # Extra kwargs passed to fetch()

Fetch a list of URLs in parallel; returns Page dicts in the same order as urls

---

source

get_options

def get_options(
    page_or_html, # Page dict (from fetch) or raw HTML string
    sel:str, # CSS selector for the <select> element
)->list:

Extract options from a element; returns [{‘value’: …, ‘text’: …}] _sel_html = '''<html><body> <select id="kanda"> <option value="1">Balakanda</option> <option value="2">Ayodhyakanda</option> <option value="3">Aranyakanda</option> </select> </body></html>''' opts = get_options(_sel_html, '#kanda') assert opts == [{'value': '1', 'text': 'Balakanda'}, {'value': '2', 'text': 'Ayodhyakanda'}, {'value': '3', 'text': 'Aranyakanda'}] arXiv source get_pdf def get_pdf( url_or_path:str, # URL or local path to PDF save_path:NoneType=None, # optional path to save the PDF (directory or full path) **scrapling_kw )->pdf_oxide.pdf_oxide.PdfDocument | None: # extra kwargs passed to scrapling fetcher (e.g. verify, headers, timeout) Fetch PDF from URL or path and return as PdfDocument, or None if it is not a PDF. Identity comes from the %PDF- magic bytes, not the URL suffix. arXiv serves its PDFs from extensionless URLs (arxiv.org/pdf/1706.03762v7), so a suffix check rejected every paper fossick is most used to read. NCBI PMC’s download page is a proof-of-work interstitial; its challenge is solved once here rather than mistaken for a failed PDF. import tempfile sp = Path(tempfile.mkdtemp()) sdt = 'https://selfdeterminationtheory.org/SDT/documents/2000_RyanDeci_SDT.pdf' sdt_p = get_pdf(sdt, save_path=sp, verify=False) sdt_p.to_markdown_all()[:200] gdr = 'https://selfdeterminationtheory.org/wp-content/uploads/2020/10/1997_GrolnickDeciRyan.pdf' gcr = get_pdf(gdr, save_path=sp, verify=False) gcr.to_markdown_all()[:200] source read_arxiv def read_arxiv( url:str, # arxiv PDF URL, or arxiv abstract URL, or arxiv ID save_pdf:bool=True, # if True, saves the downloaded PDF to disk save_dir:str='.', # directory in which to save the PDF source:bool=False, # if True, also extract the paper's full text into `source` force:bool=False, # if True, forces re-download even if cached **kw )->dict: # http_get kwargs (e.g. verify=False to skip SSL verification, headers=... etc.) *Paper metadata from an arXiv URL or ID: title, authors, summary, published, link, pdf_url,* pdf_path and, with source=True, the full text in source. The live PdfDocument comes back under doc when this call downloaded one. Only JSON-able fields are cached — a PdfDocument is a live handle, not cache content. read_arxiv('https://arxiv.org/abs/2306.14881')['summary'][:200] source lookup_doi def lookup_doi( title:str, # Paper title to search )->str | None: Return doi.org URL for first Crossref match on title lookup_doi('Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being') source pdf2nb def pdf2nb( u_or_p, # URL or local path to PDF nb_path:NoneType=None, # optional path to save the notebook; defaults to <pdf-stem>.ipynb force:bool=False, # if True, forces re-download and conversion even if notebook exists ocr_selection:str='auto', # choosing ocr (auto, off, on). auto uses pdf-oxide's ocr detection sel:str=None, method:str='GET', payload:dict=None, heavy:bool=False, stealthy:bool=False, capture_xhr:bool=False, session:bool=False, auto:bool=False, allow_private:bool=None )->Path: # extra kwargs passed to fetch() (e.g. verify=False to skip SSL verification) Convert PDF to a Jupyter notebook; one markdown cell per page + empty code cell source url2nb def url2nb( url, # URL to convert (PDF, arxiv, or HTML page) nb_path:NoneType=None, # optional path to save the notebook; defaults to <url-stem>.ipynb force:bool=False, # if True, forces re-download and conversion even if notebook exists sel:str=None, method:str='GET', payload:dict=None, heavy:bool=False, stealthy:bool=False, capture_xhr:bool=False, session:bool=False, auto:bool=False, allow_private:bool=None )->Path: # extra kwargs passed to fetch() (e.g. verify=False to skip SSL verification) Convert any URL (PDF, arxiv, or HTML page) to a Jupyter notebook source pdf2md def pdf2md( pdf, # a PdfDocument, path, bytes or URL out_path:str | pathlib.Path, # notebook path; images go under its stem image_dir:str='images', pages:bool=False, ocr:str='auto', preserve_layout:bool=False, **kw )->str: # forwarded to pdflite.pdf_md A PDF as one markdown string, pages joined by ---. get_pdf('https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf', save_path=sp, verify=False) url2nb('https://squiddev.medium.com/continuing-continuations-cps-in-python-47bba90c8d1e', nb_path=sp, verify=False) pdf2nb(sdt, nb_path=Path(sp)/'test.ipynb', verify=False) GitHub source read_gh_file def read_gh_file( url:str, # GitHub blob URL of the file to read ): Read raw contents of a file from its GitHub URL source read_gh_repo def read_gh_repo( path_or_url:str, # GitHub URL, SSH address, or local path globs:tuple=None, # file glob patterns (default: README*, pyproject.toml, *.py) limit:int=None, # max files to return as_list:bool=False, # return list of Paths instead of {path: content} dict )->dict: Read files from a GitHub repo filtered by glob patterns source gh_clone def gh_clone( path_or_url:str, # GitHub URL, SSH address, or a local path (returned as-is) )->Path: The local clone for a GitHub repo, cloned or refreshed on demand. A local path passes through. read_gh_file('https://github.com/Karthik777/litesearch/blob/main/README.md')[:200] list(read_gh_repo('https://github.com/vedicreader/gheasy')) API Discovery & Pagination XHR & pagination find_xhr captures XHR/fetch; paginate_api / json_records walk JSON list endpoints. source replay_xhr def replay_xhr( capture:dict, # a request dict from cdp.calls() or find_xhr(session=True)[i].capture data:NoneType=None, # override request body (str/dict); defaults to the captured body port:int=9223, # debug Chrome port to pull cookies from use_cookies:bool=True, # attach the browser's cookies for this domain (keeps auth) **kw )->Response: # Extra kwargs passed to the underlying Fetcher (headers, params, verify) Replay a browser-captured request as a fast plain-HTTP call, authenticated via the debug Chrome’s cookies. source find_xhr def find_xhr( url:str, # URL to visit with browser pattern:str='*', # Glob or regex pattern to filter captured XHR URLs session:bool=False, # Use the persistent debug Chrome (authenticated) instead of a throwaway browser port:int=9223, # debug Chrome port (only when session=True) tail:int=3, # seconds to keep listening after navigation (only when session=True) **kw )->list: # Extra kwargs passed to fetch (verify, network_idle, etc.) Visit url, return [{url, content_type, data, capture}] for each XHR/fetch call made. capture replays via replay_xhr. source compile_pattern def compile_pattern( pattern ): Compile pattern as regex; if invalid (e.g. bare glob like foo), convert via fnmatch first assert compile_pattern('.*products.*').search('https://api.example.com/products?q=1') assert compile_pattern('*products*').search('https://api.example.com/products?q=1') assert not compile_pattern('*products*').search('https://api.example.com/search') assert compile_pattern('.*[Ss]earch.*').search('https://api.example.com/Search') source paginate_api def paginate_api( url:str, # API endpoint URL payload:dict=None, # Request body (POST) or params (GET) page_field:str='pageNumber', # Payload key to increment for each page size_field:str='pageSize', # Payload key for page size (detects last page) results_field:str=None, # Response key with items list (auto-detect if None) method:str='POST', # HTTP method max_pages:int=10, page_size:int=24, # Page size to request (only used if not in payload) page_start:int=1, # Starting page number (default 1) save:bool=False, # If True, saves each page's items to disk save_file:str='{url}_page_{page}.json', # Filepath pattern for saving (only used if save=True) force:bool=False, # If True, forces re-fetching even if saved file exists **kw )->list: # Extra kwargs passed to fetch() (verify, headers, etc.) Paginate through a JSON API, collecting all results. Auto-detects the items list in response. source json_records def json_records( data, # any decoded JSON payload min_len:int=2, # shortest list of dicts worth calling records key:str=None, # only accept a list held under this key, at any depth with_key:bool=False, # also return the key the records were found under max_depth:int=6 ): The longest list of dicts anywhere in data — how an API’s records are found when no key is named. A JSON API puts its rows where it likes: at the top level, under results, or under data.products.items. Scanning only the top level finds the first two and silently misses the third, which reads as an empty response rather than as a payload nobody looked into. source download_files def download_files( url:str, pattern:str='*', # Glob or regex to match download hrefs save_dir:str='.', # Directory to save files follow:bool=False, # Crawl linked pages looking for more files follow_sel:str='a[href]', # Selector for page links to follow when crawling file_sel:str='a[href]', # Selector to scan for download links on each page same_domain:bool=True, max_pages:int=10, delay:float=0, force:bool=False, # Re-download even if file already exists heavy:bool=False, stealthy:bool=False, **kw )->list: # Extra kwargs passed to fetch/http_get (e.g. verify) Download all hrefs matching pattern from url; optionally crawl linked pages first jax, fs = 'https://docs.jax.dev/en/latest/notebooks/thinking_in_jax.html', 'a[class="right-next"]' download_files(jax, follow=True, follow_sel=fs, file_sel='a[href$=".ipynb"]', save_dir=sp, verify=False) from fastcore.test import test_eq _pages = crawl('https://lego.sankalpa.sh/', max_pages=2, verify=False) assert isinstance(_pages, list), f"Expected list, got {type(_pages)}" assert len(_pages) > 0, "Expected at least one page" assert all(p.status == 200 for p in _pages), "Non-200 pages should be skipped" assert all(len(p.html) > 0 for p in _pages), "html should be non-empty" assert len({p.url for p in _pages}) == len(_pages), "url values should be unique" # crawl(reuse=True): a JS-rendered crawl keeps ONE browser open across pages (no per-page relaunch) _jsp = crawl('https://quotes.toscrape.com/js/', max_pages=2, heavy=True, reuse=True) assert _jsp and all(p.status == 200 for p in _jsp) assert all(len(p.html) > 500 for p in _jsp) # JS actually rendered content into the DOM # Step 1: visit the listing page with a headless browser, capture all XHR/fetch calls apis = find_xhr('https://www.danmurphys.com.au/list/wine-all', verify=False) # paginate_api: test with JSONPlaceholder (free public REST API, GET-based) posts = paginate_api( 'https://jsonplaceholder.typicode.com/posts', payload={'_page': 1, '_limit': 5}, page_field='_page', size_field='_limit', method='GET', verify=False, ) assert len(posts) >= 5 assert 'title' in posts[0] find_xhr(session=True) uses the authenticated debug Chrome. # find_xhr(session=True) -> replay_xhr: capture a request via the authenticated debug Chrome, # then replay it as a fast plain-HTTP call using the browser's cookies hits = find_xhr('https://httpbin.org/json', pattern='.*json', session=True) assert hits, 'no matching request captured' assert hits[0].capture, 'each hit should carry a replayable capture' r = replay_xhr(hits[0].capture) assert r.status == 200 print(hits[0].url, '->', r.status) YouTube search_yt / read_yt / download_yt via yt-dlp. Cached by video id. source search_yt def search_yt( q:str, # search query n:int=10, # number of results to return **kw )->list: # extra opts forwarded to YoutubeDL; verify= is translated to nocheckcertificate, proxy= passed as-is Search YouTube; returns L of dicts: id, title, url, duration, view_count, channel, description, thumbnail results = search_yt('3blue1brown neural networks', n=3) assert len(results) >= 1, f"expected results, got {len(results)}" assert any(kw in results[0]['title'].lower() for kw in ('3blue1brown', 'neural', 'network')), \ f"unexpected title: {results[0]['title']}" assert results[0]['url'].startswith('https://www.youtube.com'), f"bad url: {results[0]['url']}" print(results[0]['title'], '|', results[0]['url']) source read_yt def read_yt( url:str, # YouTube URL or video ID force:bool=False, # if True, forces re-fetching even if cached **kw )->dict: # SSL/proxy opts (e.g. **ddgs_env()); verify= translated to nocheckcertificate Fetch YouTube metadata + English transcript (auto-captions); result disk-cached by video ID meta = read_yt('https://www.youtube.com/watch?v=aircAruvnKk') assert meta['title'], "title should be non-empty" assert isinstance(meta['source'], str), "source should be a string" assert len(meta['source']) > 100, "transcript should be substantial" source download_yt def download_yt( url:str, # YouTube URL or video ID format:str='audio', # 'audio'|'video'|yt-dlp format string save_dir:str='.', # directory to save file quality:str=None, # preferred audio quality (e.g. '192' for 192kbps, only used if format='audio') **kw )->Path: # SSL/proxy opts (e.g. **ddgs_env()); verify= translated to nocheckcertificate Download YouTube media; format=‘audio’|‘video’|yt-dlp format string. Returns Path to saved file. p = download_yt('https://www.youtube.com/watch?v=aircAruvnKk', format='audio', save_dir='/tmp/fossick_test') assert p.exists(), f"file not found: {p}" assert p.suffix == '.mp3', f"expected .mp3, got {p.suffix}" print(f"saved to: {p} ({p.stat().st_size // 1024} KB)") One door read accepts every supported target and returns the same fields: ok, kind, title, source, text, skipped, and meta. what_is selects the reader. Local paths take precedence over URL patterns, and GitHub file URLs take precedence over repository URLs. This gives collectors and indexes one input contract for web pages, PDFs, arXiv papers, YouTube videos, GitHub content, and local files. source what_is def what_is( target:str, # a URL, an arXiv id, a YouTube link, a GitHub repo or file, a PDF, a path )->str: Return the reader kind for target. source md_title def md_title( md:str, # markdown to take a title from fallback:str='', # what to use when it has no heading and no first line )->str: Return the first heading, first non-empty line, or fallback. source pdf_pages def pdf_pages( pdf:PdfDocument, # an open PDF out_path:str | pathlib.Path, # where extracted images go, as `pdf2md` takes it **kw )->list: # forwarded to `pdf2md` Return a PDF as (page_number, markdown) pairs. source read def read( target:str, # a URL, an arXiv id, a YouTube link, a GitHub repo or file, a PDF, a local path sel:str=None, # CSS selector, for the web cases auto:bool=True, # escalate plain -> heavy -> stealthy -> logged-in Chrome past bot walls pages:bool=False, # PDFs as `[(page_no, text)]` rather than one string save_dir:str='.', # where a downloaded PDF is kept force:bool=False, # ignore whatever the reader has cached **kw )->AttrDict: # `(ok, kind, title, source, text, skipped, meta)` Read a supported target and return a common result. Read failures set ok=False and give the reason in skipped. Directories and GitHub repositories set meta['path'] and leave text empty. # the network cases, which the CI run does not reach r = read('https://arxiv.org/abs/1706.03762') r.ok, r.kind, r.title, r.meta['authors'][:2] p = read('https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf', pages=True) p.ok, p.kind, len(p.text), p.text[0][0] Install source mv_skill_md def mv_skill_md( dry_run:bool=True, dir:NoneType=None )->None: Copy bundled SKILL.md to skill directories. source repo_root def repo_root()->Path: Find the root of the current git repository, or None if not in a repo. root = repo_root() assert root is not None and (root/'.git').exists(), f"Expected git root, got {root}" mv_skill_md(dry_run=True) # a fetch may not be aimed inside the network fossick is running in for _bad in ('http://169.254.169.254/latest/meta-data/', # cloud metadata 'http://127.0.0.1:8000/admin', 'http://localhost/x', 'http://10.0.0.5/x', 'http://192.168.1.1/', 'http://[::1]/', 'http://metadata.google.internal/', 'http://0.0.0.0/', 'file:///etc/passwd'): try: check_url(_bad); raise SystemExit(f'not blocked: {_bad}') except BlockedURL: pass assert check_url('https://example.com/a') == 'https://example.com/a' assert check_url('http://127.0.0.1:8000/x', allow_private=True) # the escape hatch is per-call # the message has to name the way out, or the guard just gets deleted try: check_url('http://localhost/x') except BlockedURL as e: assert 'allow_private' in str(e) and 'FOSSICK_ALLOW_PRIVATE' in str(e) # and the env var turns it off process-wide os.environ['FOSSICK_ALLOW_PRIVATE'] = '1' try: assert check_url('http://127.0.0.1/x') finally: del os.environ['FOSSICK_ALLOW_PRIVATE'] fossick fossick cdp fossick fossick core cdp search cli mcp shop quality core core – fossick core – fossick core – fossick fossick fetch, markdown, crawl, and specialized readers fetch, markdown, crawl, and specialized readers Web search, fetch, crawl, and browser automation for humans and agents