# quality


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

[`search()`](https://vedicreader.github.io/fossick/cli.html#search)
ranks matches. This module decides what is worth reading: drop mirrors,
boost authority for the query’s intent, and stop one publisher owning
the page. Used by
[`research()`](https://vedicreader.github.io/fossick/cli.html#research)
when `curated=True`.

## Domain identity

[`registrable_domain`](https://vedicreader.github.io/fossick/quality.html#registrable_domain)
/ [`host`](https://vedicreader.github.io/fossick/quality.html#host)
answer whose site is this? `MULTI_LABEL_SUFFIXES` covers the country
registries where that answer changes (`bbc.co.uk`, `nsw.gov.au`).

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L44"
target="_blank" style="float:right; font-size:smaller">source</a>

### registrable_domain

``` python
def registrable_domain(
    url:str
)->str:
```

*Site behind `url` (`docs.python.org`→`python.org`,
`bbc.co.uk`→`bbc.co.uk`). IP/single-label unchanged.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L37"
target="_blank" style="float:right; font-size:smaller">source</a>

### host

``` python
def host(
    url:str
)->str:
```

*The lowercase hostname of `url`, without `www.` or a trailing dot. `''`
when there isn’t one.*

## [`norm_url`](https://vedicreader.github.io/fossick/quality.html#norm_url)

One page → one key: drop scheme, fragment, tracking params, `www.`,
default port; sort query. Used as the dedup key everywhere below.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L60"
target="_blank" style="float:right; font-size:smaller">source</a>

### norm_url

``` python
def norm_url(
    u:str
)->str:
```

*Dedup key: no scheme/fragment/tracking/`www.`/default port/trailing
slash; query sorted.*

## Authority by intent

Rules are keyed by intent (`docs`, `policy`, `release`, …). Shape:
`docs.` (label), `.gov.au` (suffix), `sec.gov` (domain). No bare
`startswith`: that would let `evil.example` inherit boosts.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L122"
target="_blank" style="float:right; font-size:smaller">source</a>

### blocked_matches

``` python
def blocked_matches(
    domain:str, rule:str
)->bool:
```

*True if host is in the blocked list (exact or subdomain).*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L112"
target="_blank" style="float:right; font-size:smaller">source</a>

### domain_matches

``` python
def domain_matches(
    domain:str, rule:str
)->bool:
```

*Match `docs.` (label), `.gov.au` (suffix), or exact/subdomain — never
bare startswith.*

## [`classify`](https://vedicreader.github.io/fossick/quality.html#classify)

Cue table → intent, or `None` when cues disagree. Abstain \> guess.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L153"
target="_blank" style="float:right; font-size:smaller">source</a>

### classify

``` python
def classify(
    q:str
)->str:
```

*Intent class from cue table, or None when cues disagree/tie.*

## Mirrors & `site:`

`SPAM_MIRRORS` are removed. Queries with `site:` skip spam filter and
diversity caps, the caller already chose the domain.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L189"
target="_blank" style="float:right; font-size:smaller">source</a>

### drop_spam

``` python
def drop_spam(
    hits:list, blocked:tuple=(), allowed:tuple=()
)->tuple:
```

*Remove hits from mirror domains. Returns `(kept, removed_domains)`;
`allowed` rescues a domain.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L181"
target="_blank" style="float:right; font-size:smaller">source</a>

### site_domains

``` python
def site_domains(
    q:str, include:tuple=()
)->list:
```

*Domains from `site:` ops in `q` plus any explicit `include` list.*

## [`plan`](https://vedicreader.github.io/fossick/quality.html#plan)

Split a multi-part question into searches. Original query is always
first; abstains to `[q]` when there is no second question.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L266"
target="_blank" style="float:right; font-size:smaller">source</a>

### interleave

``` python
def interleave(
    lists:list, key:NoneType=None
)->list:
```

*Round-robin merge of result lists; dedup by `key` (default norm_url).*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L240"
target="_blank" style="float:right; font-size:smaller">source</a>

### plan

``` python
def plan(
    q:str, # the question as asked
    max_queries:int=4, # total searches including the original; 1 disables planning
    min_words:int=6, # below this a query has no room for two questions
    rewrite:NoneType=None, # optional `f(q) -> list[str]`, e.g. a model. Overrides the heuristic.
)->list:
```

*Split multi-part `q` into searches; always `[q]` first. Abstains when
no second question.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L235"
target="_blank" style="float:right; font-size:smaller">source</a>

### facets

``` python
def facets(
    q:str
)->list:
```

*The clauses of `q` as content-word lists, lead-in stripped. `[]` when
there is nothing to split.*

## Rerank

[`authority_rerank`](https://vedicreader.github.io/fossick/quality.html#authority_rerank)
and
[`cap_per_domain`](https://vedicreader.github.io/fossick/quality.html#cap_per_domain)
are stable and lossless: nothing dropped, demoted items keep relative
order.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L300"
target="_blank" style="float:right; font-size:smaller">source</a>

### cap_per_domain

``` python
def cap_per_domain(
    hits:list, max_per:int=2
)->tuple:
```

*Stable per-domain soft cap: overflow keeps relative order in the tail.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L283"
target="_blank" style="float:right; font-size:smaller">source</a>

### authority_rerank

``` python
def authority_rerank(
    hits:list, cls:str
)->tuple:
```

*Stable boost/demote by DOMAIN_RULES for `cls`; lossless.*

## [`cluster_sources`](https://vedicreader.github.io/fossick/quality.html#cluster_sources)

Union-find over canonical URL + Jaccard text similarity. Counts
independent stories, not URLs. Exact Jaccard (not MinHash), faster at
research-scale `n`.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L363"
target="_blank" style="float:right; font-size:smaller">source</a>

### independence

``` python
def independence(
    hits:list, threshold:float=0.6
)->dict:
```

*Cluster count / n — fraction of independent stories.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L334"
target="_blank" style="float:right; font-size:smaller">source</a>

### cluster_sources

``` python
def cluster_sources(
    hits:list, # search hits or research sources
    threshold:float=0.6, # trigram Jaccard above which two texts are one story
)->list:
```

*Union-find clusters by canonical URL + Jaccard≥threshold;
representative is first in input order.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L317"
target="_blank" style="float:right; font-size:smaller">source</a>

### jaccard

``` python
def jaccard(
    a:set, b:set
)->float:
```

*|a ∩ b| / |a ∪ b|, and 0.0 when either side is empty — absence of
evidence is not similarity.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L312"
target="_blank" style="float:right; font-size:smaller">source</a>

### shingles

``` python
def shingles(
    text:str, n:int=3
)->set:
```

*Word n-grams of `text`. Fewer than `n` words has no evidence and yields
nothing, never a match.*

## [`diversity`](https://vedicreader.github.io/fossick/quality.html#diversity)

Describe only: `dominant_domain`, near-dup pairs, independent `sources`.
Never reorders.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L399"
target="_blank" style="float:right; font-size:smaller">source</a>

### diversity

``` python
def diversity(
    hits:list, threshold:float=0.6
)->dict:
```

*Describe dominant domain, near-dups, and independent source count.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L390"
target="_blank" style="float:right; font-size:smaller">source</a>

### snippet_similarity

``` python
def snippet_similarity(
    a:str, b:str
)->float:
```

*Jaccard over word trigrams. Text shorter than three words scores 0 — no
evidence, not a match.*

## [`curate`](https://vedicreader.github.io/fossick/quality.html#curate)

drop spam → authority → cap → report. Returns the report even with
`rerank=False`.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/fossick/blob/main/fossick/quality.py#L423"
target="_blank" style="float:right; font-size:smaller">source</a>

### curate

``` python
def curate(
    q:str, # the query, read for intent and `site:` constraints
    hits:list, # search hits (dicts with `href`/`url`, `title`, `body`)
    intent:str='auto', # an intent class, 'auto' to classify, or None to skip authority
    max_per_domain:int=2, # per-registrable-domain cap; 0 disables
    blocked:tuple=(), allowed:tuple=(), # extra mirror domains, and rescues from the built-in list
    rerank:bool=True, # False: describe only, return `hits` untouched
)->tuple:
```

*Filter/rerank/describe hits → `(hits, report)`. `site:` skips spam drop
and domain cap.*

## Tests

``` python
# registrable domain: the site behind a url, with the three-label registries handled
assert registrable_domain('https://docs.python.org/3/library/functions.html') == 'python.org'
assert registrable_domain('https://www.bbc.co.uk/news') == 'bbc.co.uk'
assert registrable_domain('https://planning.nsw.gov.au/x') == 'nsw.gov.au'
assert registrable_domain('https://wko.at/') == 'wko.at'
assert registrable_domain('https://x.com') == 'x.com' and registrable_domain('') == ''
assert registrable_domain('http://192.168.1.1/a') == '192.168.1.1'      # an ip has nothing to strip
assert host('https://WWW.Example.COM./p') == 'example.com'
```

``` python
# norm_url: one page, one key -- and query order is not identity
assert norm_url('https://www.x.com/a/?utm_source=t#frag') == norm_url('http://x.com/a')
assert norm_url('https://x.com/a') != norm_url('https://x.com/b')
assert norm_url('https://x.com/a?b=2&a=1') == norm_url('https://x.com/a?a=1&b=2')   # sorted
assert norm_url('https://x.com:443/a') == norm_url('https://x.com/a')               # default port
assert norm_url('https://x.com:8443/a') != norm_url('https://x.com/a')              # but not any port
assert norm_url('https://x.com/a?mkt_tok=z&_ga=1&keep=2') == norm_url('https://x.com/a?keep=2')
```

``` python
# rule matching is the security boundary, not just the ranking one
assert domain_matches('docs.python.org', 'docs.') and not domain_matches('notdocs.com', 'docs.')
assert domain_matches('planning.nsw.gov.au', '.gov.au')          # suffix rules cover a whole registry
assert domain_matches('gov.uk', '.gov.uk')                       # ...including the registry itself
assert domain_matches('abcb.gov.au', '.gov.au')
assert not domain_matches('hipages.com.au', '.gov.au')
assert domain_matches('sec.gov', 'sec.gov') and domain_matches('www.sec.gov', 'sec.gov')
assert not domain_matches('openai.com.evil.example', 'openai.com')   # never a bare startswith
assert blocked_matches('de.newbedev.com', 'newbedev.com')
assert not blocked_matches('newbedev.com.evil.example', 'newbedev.com')
```

``` python
# intent: read it off the query, and abstain rather than guess
assert classify('rebuild my garage in australia, what is the regulation on it') == 'policy'
assert classify('planning permission for a garage') == 'policy'
assert classify('latest Tailscale release notes') == 'release'
assert classify('python asyncio api reference') == 'docs'
assert classify('CVE-2024-1234 advisory') is None or classify('log4j vulnerability advisory') == 'security'
assert classify('what do people on reddit think of it') == 'community'
assert classify('best turntables under 1000 euro') is None          # says nothing -> no reranking
assert classify('reddit release notes') is None                     # says two things -> no reranking
assert classify('') is None and classify(None) is None
```

``` python
# spam mirrors are removed; an explicit site: constraint switches the whole thing off
_h = [dict(href='https://stackoverflow.com/questions/1', title='real'),
      dict(href='https://newbedev.com/questions/1', title='mirror'),
      dict(href='https://de.newbedev.com/q/1', title='mirror subdomain')]
_kept, _rm = drop_spam(_h)
assert [h['title'] for h in _kept] == ['real'] and _rm == ['de.newbedev.com', 'newbedev.com']
assert drop_spam(_h, allowed=['newbedev.com'])[0] == _h            # allow rescues the subdomain too

assert site_domains('site:reddit.com/r/LocalLLaMA best server') == ['reddit.com']
assert site_domains('plain query', include=['NSW.gov.au']) == ['nsw.gov.au']
_, _r = curate('site:reddit.com best local llm', [dict(href=f'https://reddit.com/{i}') for i in range(5)])
assert _r['site_constrained'] == ['reddit.com'] and _r['demoted'] == 0   # not diversified away
```

``` python
# authority: the garage query, which is what this is all for
_garage = [dict(href='https://hipages.com.au/article/garage-cost', title='How much to build a garage'),
           dict(href='https://oneflare.com.au/costs/garage', title='Garage costs 2026'),
           dict(href='https://www.planning.nsw.gov.au/exempt-development', title='Exempt development'),
           dict(href='https://airtasker.com/garage', title='Get quotes'),
           dict(href='https://abcb.gov.au/ncc', title='National Construction Code')]
_out, _rep = curate('rebuild my garage in australia, what is the regulation on it', _garage)
assert _rep['intent'] == 'policy'
assert [host(_url(h)) for h in _out][:2] == ['planning.nsw.gov.au', 'abcb.gov.au'], [_url(h) for h in _out]
assert _rep['authority']['applied'] and _rep['authority']['top_before'] == 'hipages.com.au'

# nothing is dropped by reranking -- a lead-gen page is still a price signal, just not the answer
assert len(_out) == len(_garage)

# an unknown or absent intent is a no-op, not an error
assert authority_rerank(_garage, None)[0] == _garage
assert authority_rerank(_garage, 'nonsense')[1]['applied'] is False
```

``` python
# capping and diversity describe the five-listicles-from-one-site failure
_mono = [dict(href=f'https://hipages.com.au/a/{i}', title='Garage cost guide', body='what a garage costs')
         for i in range(5)] + [dict(href='https://abcb.gov.au/ncc', title='NCC', body='construction code')]
_capped, _n = cap_per_domain(_mono, 2)
assert _n == 3 and [host(_url(h)) for h in _capped][:3] == ['hipages.com.au']*2 + ['abcb.gov.au']
assert len(_capped) == len(_mono)                                    # moved, never dropped

_d = diversity(_mono)
assert _d['n'] == 6 and _d['domains'] == 2
assert _d['dominant_domain'] == dict(domain='hipages.com.au', share=round(5/6, 4))
assert _d['near_dups'] > 0                                           # the five guides read alike
assert diversity([])['n'] == 0 and diversity([])['score'] == 0.0

# a genuinely diverse set scores higher than a monopolised one
_div = [dict(href=f'https://s{i}.com/a', title=f'topic {i}', body=f'quite different text number {i}')
        for i in range(6)]
assert diversity(_div)['score'] > _d['score']
assert snippet_similarity('one two three four', 'one two three four') == 1.0
assert snippet_similarity('too short', 'too short') == 0.0           # no trigram evidence -> not a match
```

``` python
# curate is safe on the boring paths
assert curate('anything', [])[0] == []
_h2 = [dict(href='https://a.com/1', title='t')]
assert curate('q', _h2, rerank=False)[0] is _h2                      # describe-only changes nothing
assert 'diversity' in curate('q', _h2, rerank=False)[1]
assert curate('q', _h2, intent=None)[1]['intent'] is None
```

``` python
# planning: one question containing three
_GARAGE = ('I want to rebuild my garage in australia, cheap options, '
           'what is the regulation on it, material options')
_p = plan(_GARAGE)
assert _p[0] == _GARAGE                                   # the baseline is never given up
assert len(_p) == 4                                       # all three facets, plus the original
assert any('regulation' in s for s in _p[1:]), _p
assert any('cheap' in s for s in _p[1:]) and any('material' in s for s in _p[1:]), _p
assert all('garage' in s and 'australia' in s for s in _p[1:]), _p   # subject re-attached to each facet
assert 'want' not in _p[1] and 'what' not in _p[1]        # lead-in and filler stripped

# ...and constant abstention, which is what makes it safe to leave on
assert plan('sqlite wal mode vs journal mode performance') == ['sqlite wal mode vs journal mode performance']
assert plan('garage cost') == ['garage cost']                        # too short to hold two questions
assert plan('site:reddit.com best local llm, cheap options') == ['site:reddit.com best local llm, cheap options']
assert plan(_GARAGE, max_queries=1) == [_GARAGE]                     # explicitly disabled
assert plan('') == []
assert len(plan(_GARAGE, max_queries=2)) == 2                        # the cap is honoured

# a facet that adds no words to the subject is not a search
assert plan('rebuild my garage in australia, the garage, australia') == \
       ['rebuild my garage in australia, the garage, australia']

# an injected rewriter (a model, in ramabana) replaces the heuristic and still keeps the original
assert plan('anything at all here', rewrite=lambda q: ['one', 'two']) == ['anything at all here', 'one', 'two']
```

``` python
# interleaving is coverage, not consensus: the generalist that ranks for every facet must not
# outrank the specialist that owns one
_cost = [dict(href='https://hipages.com.au/guide', title='complete garage guide'),
         dict(href='https://oneflare.com.au/cost', title='costs')]
_reg  = [dict(href='https://hipages.com.au/guide', title='complete garage guide'),
         dict(href='https://planning.nsw.gov.au/exempt', title='exempt development')]
_mat  = [dict(href='https://hipages.com.au/guide', title='complete garage guide'),
         dict(href='https://bunnings.com.au/materials', title='materials')]
_il = interleave([_cost, _reg, _mat])
assert [host(_url(h)) for h in _il][:2] == ['hipages.com.au', 'oneflare.com.au']
assert 'planning.nsw.gov.au' in [host(_url(h)) for h in _il[:4]], [_url(h) for h in _il]
assert len(_il) == 4                                      # the guide appears once, not three times

# ragged lists and empties are fine
assert interleave([]) == [] and interleave([[], []]) == []
assert len(interleave([_cost, []])) == 2
```

``` python
# clustering: four urls, one story -- and transitivity is what makes that visible
_wire = ('OpenAI said today that it is lowering prices for its newest model across all tiers, '
         'effective immediately for every API customer.')
_syn = [dict(href='https://a.invalid/x', title='OpenAI cuts prices', body=_wire),
        dict(href='https://b.invalid/y', title='OpenAI cuts prices', body=_wire),
        dict(href='https://c.invalid/z', title='OpenAI cuts prices', body=_wire),
        dict(href='https://openai.com/index/new-pricing', title='New pricing for our latest model',
             body='We are updating the per-token price of our newest model from today.')]
_cl = cluster_sources(_syn)
assert len(_cl) == 2, _cl                                  # three copies collapse, the original stands
assert _cl[0]['size'] == 3 and _cl[1]['size'] == 1
assert _cl[0]['canonical'] == 'https://a.invalid/x'        # named by its first (highest-ranked) member

_ind = independence(_syn)
assert _ind['n'] == 4 and _ind['sources'] == 2             # the number a reader actually needs
assert _ind['method'] == 'url+text' and _ind['confidence'] == 'medium'
assert _ind['limitations']                                 # always says what it cannot see

# transitivity: A~B and B~C is one family even where A and C alone fall short of the threshold
_chain = [dict(href='https://a.invalid/1', body='alpha beta gamma delta epsilon zeta eta theta'),
          dict(href='https://b.invalid/2', body='beta gamma delta epsilon zeta eta theta iota'),
          dict(href='https://c.invalid/3', body='gamma delta epsilon zeta eta theta iota kappa')]
assert jaccard(shingles(_chain[0]['body']), shingles(_chain[2]['body'])) < 0.6      # not a pair...
assert len(cluster_sources(_chain, threshold=0.5)) == 1, cluster_sources(_chain, 0.5)  # ...but one chain

# the same url twice is one source however differently it is described
assert len(cluster_sources([dict(href='https://x.com/a?utm_source=t', body='one two three four'),
                            dict(href='http://www.x.com/a', body='utterly different words here')])) == 1

# genuinely independent results stay independent
_div = [dict(href=f'https://s{i}.invalid/a', body=f'entirely separate subject number {i} here')
        for i in range(5)]
assert independence(_div)['sources'] == 5 and independence(_div)['score'] > independence(_syn)['score']

# no text to compare: degrade to url identity and say so rather than claiming confidence
_bare = [dict(href='https://a.invalid/1'), dict(href='https://b.invalid/2')]
assert independence(_bare)['method'] == 'url' and independence(_bare)['confidence'] == 'low'
assert any('too little text' in l for l in independence(_bare)['limitations'])
assert independence([]) == dict(n=0, sources=0, score=0.0, clusters=[], method='none',
                                confidence='low', limitations=[])

# and diversity carries the honest source count next to the url count
assert diversity(_syn)['n'] == 4 and diversity(_syn)['sources'] == 2
assert shingles('two words') == set() and jaccard(set(), {('a','b','c')}) == 0.0
```