quality

Choosing which search results are worth reading, and what to search for

A search engine ranks pages by how well they match a query. That is not the same as how worth reading they are. A question about building rules in Australia can return five cost guides from one trade directory ahead of the government page that sets the rules. A programming question can return sites that copy Stack Overflow answers.

This module reorders results before any page is downloaded. It does these things:

It removes copycat sites.
It moves sources that have authority on the kind of question up the list.
It limits how many results one website can have near the top.
It splits a question that asks several things into one search per part.

research in fossick.search calls curate when curated=True, its default. It also uses plan and interleave. Each function here also works on its own, on any list of result dictionaries with an href or url.

Which site a URL belongs to

Two URLs belong to the same site when they share a registrable domain, the part of the name someone registered: docs.python.org and www.python.org both belong to python.org. Usually that is the last two parts of the host name. In countries whose registries sell names one level down, such as bbc.co.uk or nsw.gov.au, it is the last three. MULTI_LABEL_SUFFIXES lists those registry endings for the countries fossick handles. host returns a URL’s host name in lower case without www., and registrable_domain returns the site.

registrable_domain

def registrable_domain(
    url:str
)->str:

Return the registered site behind url, such as python.org for docs.python.org or bbc.co.uk for www.bbc.co.uk.

host

def host(
    url:str
)->str:

Return the host name of url in lower case, without www. or a trailing dot, or '' when it has none.

One key per page

The same page often appears under several URLs: with and without www., with a trailing slash, or with tracking parameters such as utm_source added by whoever shared it. norm_url reduces a URL to one key. It drops the scheme, the www., the fragment, tracking parameters, a default port and a trailing slash. It sorts the query parameters. A port that is not the default stays in the key. interleave, cluster_sources and diversity use the key to decide when two results are the same page.

norm_url

def norm_url(
    u:str
)->str:

Return a key for url that is the same for every URL of one page.

Authority depends on the question

The source to trust depends on the question. For a question about the law, a government site is the authority. For a question about a library, its documentation is. For a question about what users think, a forum is. DOMAIN_RULES lists, for each kind of question, the sites to move up (boost) and the sites to move down (demote). The kinds are docs, policy, release, security and community.

domain_matches checks one host name against one rule. A rule has one of four forms:

Rule	Matches	Example
Ends in a dot	Host names that start with it	`docs.`
Starts with a dot	The ending itself and every host under it	`.gov.au`
Contains a `/`	Only the host name before the `/`, with no subdomains	`github.com/advisories`
Any other text	That domain and its subdomains	`sec.gov`

Only a rule that ends in a dot matches the start of a host name. openai.com.evil.example does not match openai.com. blocked_matches applies the last form, the domain and its subdomains, to a blocked list.

blocked_matches

def blocked_matches(
    domain:str, rule:str
)->bool:

Return True when domain is the domain rule or one of its subdomains.

domain_matches

def domain_matches(
    domain:str, rule:str
)->bool:

Return True when domain matches rule: a leading label, a domain ending, a host before a /, or a domain and its subdomains.

What kind of question it is

classify reads the kind of question from the cue words in INTENT_CUES. “regulation” and “permit” indicate policy. “release notes” indicates release. classify returns the kind with the most cue matches. It returns None when the query has no cues or when two kinds tie. With None, curate does not reorder the results by authority.

classify

def classify(
    q:str
)->str:

Return the kind of question q asks, read from its cue words, or None when the cues are absent or tie.

Copycat sites and `site:`

SPAM_MIRRORS lists sites that copy the content of other sites, such as Stack Overflow scrapers and GitHub mirrors. They add nothing to the original. Searching several engines tends to promote them, because they are built to rank on all of them.

drop_spam removes results from these sites and from any extra domains in blocked. It returns the remaining results and the domains removed. A domain in allowed is kept even when it is on a list. allowed also covers the subdomains of that domain.

A query that contains site:reddit.com has already chosen its site. site_domains returns the domains named by site: in a query. For such a query, curate skips the copycat removal and the per-site limit, which would undo the choice. curate still reorders by authority.

drop_spam

def drop_spam(
    hits:list, blocked:tuple=(), allowed:tuple=()
)->tuple:

Remove results from copycat sites and return the remaining results and the domains removed.

site_domains

def site_domains(
    q:str, include:tuple=()
)->list:

Return the domains named by site: in q, together with any in include.

Splitting a question into searches

A question such as “I want to rebuild my garage in australia, cheap options, what is the regulation on it, material options” asks three things. One search for all of it mostly finds pages about the first. plan splits it into one search per part.

plan removes openings such as “I want to” (LEAD_INS) and empty phrases (FILLER). It splits the rest at commas, semicolons, question marks and words such as “and”. It adds the topic words of the first part to each later part. Each search then still says “garage australia”. The original question is always the first search.

plan returns [q] when there is nothing to split. The table lists the conditions.

Condition	Result
`max_queries` is below 2	`[q]`
The query has fewer than `min_words` words	`[q]`
The query has `site:`	`[q]`
The query has only one part	`[q]`
The later parts add no new words to the first	`[q]`

max_queries caps the number of searches. rewrite is a function, such as a call to a language model, that takes the question and returns a list of searches. When you pass it, plan uses it instead of these rules. facets returns the parts as lists of topic words.

interleave merges the result lists of several searches. It takes one result from each list in turn and keeps a page that several searches found once. For such a page it joins the queries lists of the duplicate results. Taking turns gives each part of the question its own sources. Merging by score would favour the general guide that ranks for every part over the specific source that answers one part.

interleave

def interleave(
    lists:list, key:NoneType=None
)->list:

Merge result lists by taking one from each in turn, keeping each page once.

plan

def plan(
    q:str, # the question as asked
    max_queries:int=4, # total searches including the original; 1 disables planning
    min_words:int=6, # below this a query has no room for two questions
    rewrite:NoneType=None, # optional `f(q) -> list[str]`, e.g. a model. Overrides the heuristic.
)->list:

Split a question that asks several things into one search per part, with the original question first.

Reordering

authority_rerank moves results from the sites that DOMAIN_RULES boosts for a kind of question to the front. It moves results from the sites that DOMAIN_RULES demotes towards the back. cap_per_domain lets each site keep at most max_per results near the top and moves the rest to the end of the list in their original order. A list of fewer than three results is returned unchanged.

Neither function drops a result. Results that are not moved keep their order. Both return the reordered list and a second value. For authority_rerank it is a report dictionary with intent, applied, boosted and demoted. For cap_per_domain it is the number of results moved to the end.

cap_per_domain

def cap_per_domain(
    hits:list, max_per:int=2
)->tuple:

Keep at most max_per results per site near the top and move the rest, in order, to the end.

authority_rerank

def authority_rerank(
    hits:list, cls:str
)->tuple:

Move results from sites favoured for the kind of question cls up and disfavoured ones down, dropping none.

How many stories, not how many URLs

Ten results can be one story, for example a news agency report that nine sites reprinted. cluster_sources groups results that have the same norm_url key or that share most of their text.

shingles returns the set of three-word sequences in a text. jaccard divides the size of the overlap of two sets by the size of their union. cluster_sources applies jaccard to the shingles of each result’s title and text. Two results with a similarity at or above threshold (0.6 by default) are one story. The grouping is transitive. When A matches B and B matches C, all three are one group. Each group is named after its first member in the input order.

independence returns the groups with a score. The score rises with the number of separate stories and the number of separate sites. The result also lists its limits. Two sites that rewrite one press release in their own words do not share enough text to be grouped.

independence

def independence(
    hits:list, threshold:float=0.6
)->dict:

Return how many separate stories and sites a result list holds, with a score and the method’s limits.

cluster_sources

def cluster_sources(
    hits:list, # search hits or research sources
    threshold:float=0.6, # trigram Jaccard at or above which two texts are one story
)->list:

Group results that are the same page or share most of their text, and return the groups.

jaccard

def jaccard(
    a:set, b:set
)->float:

Return the size of the overlap of two sets divided by the size of their union, or 0.0 when either is empty.

shingles

def shingles(
    text:str, n:int=3
)->set:

Return the set of n-word sequences in text, empty when it has fewer than n words.

Describing a result list

diversity describes a list and does not change it. The result has these keys:

Key	Meaning
`n`, `domains`	Number of results and number of sites
`dominant_domain`	The site with the most results, and its `share` of the results
`dup_urls`	Number of results whose `norm_url` key repeats an earlier one
`near_dups`	Number of pairs of results that read alike
`sources`	Number of separate stories
`score`	A number from 0 to 1. A higher number means a more varied list.

The score is the weighted sum of three components.

Component	Weight
Share of separate sites	0.5
Share of unique URLs	0.3
Share of pairs that do not read alike	0.2

snippet_similarity returns the Jaccard similarity of the three-word sequences of two texts.

diversity

def diversity(
    hits:list, threshold:float=0.6
)->dict:

Describe how varied a result list is: its sites, repeated URLs, look-alike pairs and separate stories.

snippet_similarity

def snippet_similarity(
    a:str, b:str
)->float:

Return the Jaccard similarity of two texts’ three-word sequences, or 0.0 when either has fewer than three words.

`curate`

curate runs these steps in order:

drop_spam removes copycat sites.
authority_rerank reorders by authority for the kind of question. classify reads the kind from the query, or you pass intent. intent=None skips this step.
cap_per_domain applies the per-site limit. max_per_domain=0 skips this step.
diversity describes the result.

A query with site: skips steps 1 and 3. curate returns the reordered results and a report of what each step did. With rerank=False it returns the results untouched. The report then has dropped=[], demoted=0 and authority=None, and still has intent and diversity.

curate

def curate(
    q:str, # the query, read for intent and `site:` constraints
    hits:list, # search hits (dicts with `href`/`url`, `title`, `body`)
    intent:str='auto', # an intent class, 'auto' to classify, or None to skip authority
    max_per_domain:int=2, # per-registrable-domain cap; 0 disables
    blocked:tuple=(), allowed:tuple=(), # extra mirror domains, and rescues from the built-in list
    rerank:bool=True, # False: describe only, return `hits` untouched
)->tuple:

Remove copycat sites, reorder by authority, limit results per site, and return the results with a report.

Tests

The cells below test each function offline, mostly on the garage question, which is the case this module was written for.

# registrable domain: the site behind a url, with the three-label registries handled
assert registrable_domain('https://docs.python.org/3/library/functions.html') == 'python.org'
assert registrable_domain('https://www.bbc.co.uk/news') == 'bbc.co.uk'
assert registrable_domain('https://planning.nsw.gov.au/x') == 'nsw.gov.au'
assert registrable_domain('https://wko.at/') == 'wko.at'
assert registrable_domain('https://x.com') == 'x.com' and registrable_domain('') == ''
assert registrable_domain('http://192.168.1.1/a') == '192.168.1.1'      # an ip has nothing to strip
assert host('https://WWW.Example.COM./p') == 'example.com'

# norm_url: one page has one key, and query order does not change the key
assert norm_url('https://www.x.com/a/?utm_source=t#frag') == norm_url('http://x.com/a')
assert norm_url('https://x.com/a') != norm_url('https://x.com/b')
assert norm_url('https://x.com/a?b=2&a=1') == norm_url('https://x.com/a?a=1&b=2')   # sorted
assert norm_url('https://x.com:443/a') == norm_url('https://x.com/a')               # default port
assert norm_url('https://x.com:8443/a') != norm_url('https://x.com/a')              # but not any port
assert norm_url('https://x.com/a?mkt_tok=z&_ga=1&keep=2') == norm_url('https://x.com/a?keep=2')

# rule matching is the security boundary, not just the ranking one
assert domain_matches('docs.python.org', 'docs.') and not domain_matches('notdocs.com', 'docs.')
assert domain_matches('planning.nsw.gov.au', '.gov.au')          # suffix rules cover a whole registry
assert domain_matches('gov.uk', '.gov.uk')                       # ...including the registry itself
assert domain_matches('abcb.gov.au', '.gov.au')
assert not domain_matches('hipages.com.au', '.gov.au')
assert domain_matches('sec.gov', 'sec.gov') and domain_matches('www.sec.gov', 'sec.gov')
assert not domain_matches('openai.com.evil.example', 'openai.com')   # never a bare startswith
assert blocked_matches('de.newbedev.com', 'newbedev.com')
assert not blocked_matches('newbedev.com.evil.example', 'newbedev.com')

# intent: read it off the query, and abstain rather than guess
assert classify('rebuild my garage in australia, what is the regulation on it') == 'policy'
assert classify('planning permission for a garage') == 'policy'
assert classify('latest Tailscale release notes') == 'release'
assert classify('python asyncio api reference') == 'docs'
assert classify('CVE-2024-1234 advisory') is None or classify('log4j vulnerability advisory') == 'security'
assert classify('what do people on reddit think of it') == 'community'
assert classify('best turntables under 1000 euro') is None          # says nothing -> no reranking
assert classify('reddit release notes') is None                     # says two things -> no reranking
assert classify('') is None and classify(None) is None

# spam mirrors are removed; an explicit site: constraint skips the copycat removal and the per-site limit
_h = [dict(href='https://stackoverflow.com/questions/1', title='real'),
      dict(href='https://newbedev.com/questions/1', title='mirror'),
      dict(href='https://de.newbedev.com/q/1', title='mirror subdomain')]
_kept, _rm = drop_spam(_h)
assert [h['title'] for h in _kept] == ['real'] and _rm == ['de.newbedev.com', 'newbedev.com']
assert drop_spam(_h, allowed=['newbedev.com'])[0] == _h            # allow rescues the subdomain too

assert site_domains('site:reddit.com/r/LocalLLaMA best server') == ['reddit.com']
assert site_domains('plain query', include=['NSW.gov.au']) == ['nsw.gov.au']
_, _r = curate('site:reddit.com best local llm', [dict(href=f'https://reddit.com/{i}') for i in range(5)])
assert _r['site_constrained'] == ['reddit.com'] and _r['demoted'] == 0   # not diversified away

# authority: the garage query, which is what this is all for
_garage = [dict(href='https://hipages.com.au/article/garage-cost', title='How much to build a garage'),
           dict(href='https://oneflare.com.au/costs/garage', title='Garage costs 2026'),
           dict(href='https://www.planning.nsw.gov.au/exempt-development', title='Exempt development'),
           dict(href='https://airtasker.com/garage', title='Get quotes'),
           dict(href='https://abcb.gov.au/ncc', title='National Construction Code')]
_out, _rep = curate('rebuild my garage in australia, what is the regulation on it', _garage)
assert _rep['intent'] == 'policy'
assert [host(_url(h)) for h in _out][:2] == ['planning.nsw.gov.au', 'abcb.gov.au'], [_url(h) for h in _out]
assert _rep['authority']['applied'] and _rep['authority']['top_before'] == 'hipages.com.au'

# reranking drops nothing
assert len(_out) == len(_garage)

# an unknown or absent intent is a no-op, not an error
assert authority_rerank(_garage, None)[0] == _garage
assert authority_rerank(_garage, 'nonsense')[1]['applied'] is False

# capping and diversity describe the five-listicles-from-one-site failure
_mono = [dict(href=f'https://hipages.com.au/a/{i}', title='Garage cost guide', body='what a garage costs')
         for i in range(5)] + [dict(href='https://abcb.gov.au/ncc', title='NCC', body='construction code')]
_capped, _n = cap_per_domain(_mono, 2)
assert _n == 3 and [host(_url(h)) for h in _capped][:3] == ['hipages.com.au']*2 + ['abcb.gov.au']
assert len(_capped) == len(_mono)                                    # moved, never dropped

_d = diversity(_mono)
assert _d['n'] == 6 and _d['domains'] == 2
assert _d['dominant_domain'] == dict(domain='hipages.com.au', share=round(5/6, 4))
assert _d['near_dups'] > 0                                           # the five guides read alike
assert diversity([])['n'] == 0 and diversity([])['score'] == 0.0

# a genuinely diverse set scores higher than a monopolised one
_div = [dict(href=f'https://s{i}.com/a', title=f'topic {i}', body=f'quite different text number {i}')
        for i in range(6)]
assert diversity(_div)['score'] > _d['score']
assert snippet_similarity('one two three four', 'one two three four') == 1.0
assert snippet_similarity('too short', 'too short') == 0.0           # no trigram evidence -> not a match

# curate is safe on the boring paths
assert curate('anything', [])[0] == []
_h2 = [dict(href='https://a.com/1', title='t')]
assert curate('q', _h2, rerank=False)[0] is _h2                      # describe-only changes nothing
assert 'diversity' in curate('q', _h2, rerank=False)[1]
assert curate('q', _h2, intent=None)[1]['intent'] is None

# planning: one question containing three
_GARAGE = ('I want to rebuild my garage in australia, cheap options, '
           'what is the regulation on it, material options')
_p = plan(_GARAGE)
assert _p[0] == _GARAGE                                   # the baseline is never given up
assert len(_p) == 4                                       # all three facets, plus the original
assert any('regulation' in s for s in _p[1:]), _p
assert any('cheap' in s for s in _p[1:]) and any('material' in s for s in _p[1:]), _p
assert all('garage' in s and 'australia' in s for s in _p[1:]), _p   # subject re-attached to each facet
assert 'want' not in _p[1] and 'what' not in _p[1]        # lead-in and filler stripped

# ...and constant abstention, which is what makes it safe to leave on
assert plan('sqlite wal mode vs journal mode performance') == ['sqlite wal mode vs journal mode performance']
assert plan('garage cost') == ['garage cost']                        # too short to hold two questions
assert plan('site:reddit.com best local llm, cheap options') == ['site:reddit.com best local llm, cheap options']
assert plan(_GARAGE, max_queries=1) == [_GARAGE]                     # explicitly disabled
assert plan('') == []
assert len(plan(_GARAGE, max_queries=2)) == 2                        # the cap is honoured

# a facet that adds no words to the subject is not a search
assert plan('rebuild my garage in australia, the garage, australia') == \
       ['rebuild my garage in australia, the garage, australia']

# an injected rewriter (a model, in ramabana) replaces the heuristic and still keeps the original
assert plan('anything at all here', rewrite=lambda q: ['one', 'two']) == ['anything at all here', 'one', 'two']

# interleaving is coverage, not consensus: the generalist that ranks for every facet must not
# outrank the specialist that owns one
_cost = [dict(href='https://hipages.com.au/guide', title='complete garage guide'),
         dict(href='https://oneflare.com.au/cost', title='costs')]
_reg  = [dict(href='https://hipages.com.au/guide', title='complete garage guide'),
         dict(href='https://planning.nsw.gov.au/exempt', title='exempt development')]
_mat  = [dict(href='https://hipages.com.au/guide', title='complete garage guide'),
         dict(href='https://bunnings.com.au/materials', title='materials')]
_il = interleave([_cost, _reg, _mat])
assert [host(_url(h)) for h in _il][:2] == ['hipages.com.au', 'oneflare.com.au']
assert 'planning.nsw.gov.au' in [host(_url(h)) for h in _il[:4]], [_url(h) for h in _il]
assert len(_il) == 4                                      # the guide appears once, not three times

# ragged lists and empties are fine
assert interleave([]) == [] and interleave([[], []]) == []
assert len(interleave([_cost, []])) == 2

# clustering: three copies and one original make two stories, and clusters are transitive
_wire = ('OpenAI said today that it is lowering prices for its newest model across all tiers, '
         'effective immediately for every API customer.')
_syn = [dict(href='https://a.invalid/x', title='OpenAI cuts prices', body=_wire),
        dict(href='https://b.invalid/y', title='OpenAI cuts prices', body=_wire),
        dict(href='https://c.invalid/z', title='OpenAI cuts prices', body=_wire),
        dict(href='https://openai.com/index/new-pricing', title='New pricing for our latest model',
             body='We are updating the per-token price of our newest model from today.')]
_cl = cluster_sources(_syn)
assert len(_cl) == 2, _cl                                  # three copies collapse, the original stands
assert _cl[0]['size'] == 3 and _cl[1]['size'] == 1
assert _cl[0]['canonical'] == 'https://a.invalid/x'        # named by its first (highest-ranked) member

_ind = independence(_syn)
assert _ind['n'] == 4 and _ind['sources'] == 2             # the number a reader actually needs
assert _ind['method'] == 'url+text' and _ind['confidence'] == 'medium'
assert _ind['limitations']                                 # always says what it cannot see

# transitivity: A~B and B~C is one family even where A and C alone fall short of the threshold
_chain = [dict(href='https://a.invalid/1', body='alpha beta gamma delta epsilon zeta eta theta'),
          dict(href='https://b.invalid/2', body='beta gamma delta epsilon zeta eta theta iota'),
          dict(href='https://c.invalid/3', body='gamma delta epsilon zeta eta theta iota kappa')]
assert jaccard(shingles(_chain[0]['body']), shingles(_chain[2]['body'])) < 0.6      # not a pair...
assert len(cluster_sources(_chain, threshold=0.5)) == 1, cluster_sources(_chain, 0.5)  # ...but one chain

# the same url twice is one source however differently it is described
assert len(cluster_sources([dict(href='https://x.com/a?utm_source=t', body='one two three four'),
                            dict(href='http://www.x.com/a', body='utterly different words here')])) == 1

# genuinely independent results stay independent
_div = [dict(href=f'https://s{i}.invalid/a', body=f'entirely separate subject number {i} here')
        for i in range(5)]
assert independence(_div)['sources'] == 5 and independence(_div)['score'] > independence(_syn)['score']

# no text to compare: degrade to url identity and say so rather than claiming confidence
_bare = [dict(href='https://a.invalid/1'), dict(href='https://b.invalid/2')]
assert independence(_bare)['method'] == 'url' and independence(_bare)['confidence'] == 'low'
assert any('too little text' in l for l in independence(_bare)['limitations'])
assert independence([]) == dict(n=0, sources=0, score=0.0, clusters=[], method='none',
                                confidence='low', limitations=[])

# and diversity carries the honest source count next to the url count
assert diversity(_syn)['n'] == 4 and diversity(_syn)['sources'] == 2
assert shingles('two words') == set() and jaccard(set(), {('a','b','c')}) == 0.0