search

private web search through a local SearXNG instance

The search module routes queries through SearXNG, a self-hosted meta-search engine running in Docker. fossick starts the container automatically on the first call. Results are cached in Redis, so repeated queries return immediately.

SearXNG

---

source

searxng_stop

def searxng_stop(
    
):

Stop SearXNG + Redis if we started them. No-op otherwise.

---

source

searxng_start

def searxng_start(
    settings:Path=None
)->str:

Start SearXNG + Redis via Compose. settings overrides the bundled searxng_settings.yml. Idempotent.

Result type and query routing

_intent() scores the query against keyword lists for each category. At least two signals must match before routing away from the default general category, which keeps single generic words like “python” from sending a query to GitHub and Stack Overflow instead of Google.

---

source

Result

def Result(
    args:VAR_POSITIONAL, default_:NoneType=None, kwargs:VAR_KEYWORD
):

One search result. Fields: title, url, snippet, score, engines, provider, ts.

SearXNG search

---

source

search

def search(
    q:str, n:int=10, # max results
    category:str=None, # override SearXNG category (general/science/it/news)
    engines:str=None, # comma-sep engine names passed to SearXNG
)->L:

Search the web. Returns L of Result sorted by relevance.

Tests

# Verify intent routing
assert _intent('arxiv transformer paper')[0] == 'academic'
assert _intent('latest news today')[0] == 'recent'
assert _intent('what is the capital of France')[0] == 'default'
# Requires >=2 code signals — single match falls through to general
assert _intent('github python error')[0] == 'default'        # only 1 signal (github)
assert _intent('github npm package install')[0] == 'code'    # 2 signals (github + npm)
# Generic web-dev terms alone must not trigger code routing
assert _intent('fasthtml python web framework')[0] == 'default'

searxng_start()

'http://localhost:8080'

# Live test — requires Docker
url = searxng_start()
print('SearXNG at', url)
results = search('fasthtml python web framework', n=5)
for r in results: print(r.title, r.url)

SearXNG at http://localhost:8080
FastHTML - Modern web applications in pure Python https://fastht.ml/
GitHub - AnswerDotAI/fasthtml: The fastest way to create an HTML app https://github.com/answerdotai/fasthtml
FastHTML: a hypermedia-first python framework w/htmx available ootb https://www.reddit.com/r/htmx/comments/1efizmn/fasthtml_a_hypermediafirst_python_framework_whtmx/
FastHTML Tutorial: Build Modern Web Applications with Pure Python https://www.youtube.com/watch?v=AxA8YH_UyBo
First encounter with FastHTML: Building a FastHTML assistant https://medium.com/@mrsirsh/first-encounter-with-fasthtml-building-a-fasthtml-assistant-fe896d3a3e60

search('what is the capital of France')[0]

{ 'default_': None,
  'engines': ['google', 'startpage'],
  'provider': 'searxng',
  'score': 8.0,
  'snippet': 'Paris is the capital and largest city of France, with an '
             'estimated city population of 2.04 million in an area of 105.4 km '
             '2 (40.7 sq mi), and a metropolitan ...',
  'title': 'Paris - Wikipedia',
  'ts': 1780439809.467083,
  'url': 'https://en.wikipedia.org/wiki/Paris'}