quality

Find junk documents with a noise score, record which sections were useful, and optionally fit a ranker from that record

A vault fills up with things that match many searches and answer none: cookie banners, site footers, navigation pages, licence boilerplate. This page describes three tools for that problem. All three are off until you turn them on.

A noise score ranks the documents that are most likely junk. It needs no labels. On the project’s test corpora it scores 0.988 AUC, and 0.996 after it is fitted to a person’s marks. AUC is the area under the ROC curve. A score of 1.0 is a perfect ranking and 0.5 is chance. The score only suggests. A person reviews suggest_noisy, then marks documents with mark_noisy or accept_noisy.
A feedback log records which sections an answer cited and which sections a person rated. It records them after learn() is on.
A ranker is fitted from that log and reorders retrieval. No learned ranker beat plain reciprocal rank fusion in a repeatable way (evals/RESULTS.md). Leave the ranker off unless it helps on your own collection.

What was good

rate records one judgement about one section for one question. SIGNALS maps each signal to a label (1 good, 0 bad) and a weight.

signal	label	weight
`up`, `good`	1	3
`down`, `bad`	0	3
`read`: the person read the section	1	1.5
`cited`: an answer cited the section	1	1
`shown`: the section was offered and not cited	0	0.35

ratings returns the log, newest first.

source

Vault.ratings

def ratings(
    doc_id:str=None, limit:int=None
)->fastcore.foundation.L:

The feedback log for this shelf, newest first.

source

Vault.rate

def rate(
    q:str, # the question this judgement is about
    node_id:str=None, # the section being judged
    doc_id:str=None, # its document; derived from `node_id` when not given
    signal:str='up', # one of SIGNALS
    rank:int=None, # where it was ranked when it was shown
    score:float=None, # the retrieval score it was shown with
    ask_id:str=None, # groups the rows from one question together
)->dict:

Record one judgement about one section for one question.

import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as d:
    v = Vault(Path(d)/'test.db', offline=True)

    assert 'feedback' in v.db.t
    assert v.ratings() == []

    v.rate('question', node_id='doc#1', signal='up')
    assert len(v.ratings()) == 1

Citations are the label

ask returns the sections its answer cited. After learn(), every ask calls log_ask. log_ask writes the cited sections to the log as positives and the other sections as weak negatives. An answer that cites nothing writes nothing, unless the caller passes up or down. The ranker and the document prior learn from these rows.

source

Vault.log_ask

def log_ask(
    out, # what `ask` returned
    up:list=None, # section numbers you thought were good, as `[1]`-style numbers
    down:list=None, # section numbers you thought were bad
    weak:bool=True, # log the uncited sections as weak negatives
)->int:

Write feedback rows for one answered question from what ask returned; learn() does this for every ask.

The Beta prior

doc_prior turns the log into a multiplier on the retrieval score of each document. It models the usefulness of a document as a Beta distribution. The weighted wins and losses of the document update the distribution. A document with little evidence stays near neutral. A document cited once does not move far ahead.

gamma sets how strongly the prior changes a score. explore is the chance, from 0 to 1, that doc_prior samples from the distribution (Thompson sampling) and does not use its mean. Sampling keeps documents with little evidence in play. lo and hi clamp every multiplier, so no amount of evidence can bury a document.

source

Vault.doc_prior

def doc_prior(
    gamma:float=0.35, # how hard the prior is allowed to push; 0 disables it
    explore:float=1.0, # 1 -> Thompson sample the posterior, 0 -> use its mean
    lo:float=0.6, hi:float=1.6, # clamp, so no amount of evidence can bury a document
    seed:int=None, exclude_ask:NoneType=None, # ignore these questions' rows; for an out-of-fold prior
    stats:dict=None, # `{doc_id: (wins, losses)}` to use instead of reading the log
)->dict:

Each document’s multiplier on its retrieval score, from a Beta distribution over its feedback, as {doc_id: multiplier}.

source

Vault.doc_stats

def doc_stats(
    exclude_ask:NoneType=None
)->dict:

Each document’s weighted wins and losses in this shelf’s feedback log, as {doc_id: (wins, losses)}.

Noise candidates

suggest_noisy ranks the documents that look most like junk. It leaves out each document that a person has already judged for noise. mark_noisy records the decision. search, sections, context and ask leave out a document marked as noise. accept_noisy marks the current top suggestions in one call. mark_noisy_many marks a list of documents.

The features and the Ranker come from litesearch, which computes them from the chunk store alone. The 0.988 AUC is measured there. The features include these:

feature	what it measures
`hub`	how often a chunk appears among the nearest neighbours of other chunks
`spread_chunk`	how evenly a chunk sits between topics
`off_centre`	how far a chunk is from the middle of the collection
`dup_out`	how much of a chunk is duplicated in other documents

This page adds the vault’s side. It adds the documents that a person marked and the questions each document was retrieved for (promiscuity). It also adds the rule that the score only suggests.

source

Vault.chunk_matrix

def chunk_matrix(
    limit:int=None, seed:int=0
)->tuple:

(ids, doc_ids, texts, V) for this shelf: litesearch holds the vectors, this names the store.

source

Vault.noise_scores

def noise_scores(
    weights:dict=None, ranker:NoneType=None, **kw
)->fastcore.foundation.L:

Score every document for noise, most suspicious first, with the features behind each score.

source

Vault.noise_features

def noise_features(
    **kw
)->fastcore.basics.AttrDict:

litesearch’s noise features for this shelf’s documents, with promiscuity taken from the vault’s feedback log.

source

Vault.accept_noisy

def accept_noisy(
    k:int=20, reason:str='suggested', **kw
)->fastcore.foundation.L:

Mark the current top k noise suggestions as noise.

source

Vault.mark_noisy_many

def mark_noisy_many(
    refs, noisy:bool=True, reason:str=''
)->fastcore.foundation.L:

Mark many documents as noise at once, for example a reviewed suggest_noisy list.

source

Vault.suggest_noisy

def suggest_noisy(
    k:int=20, min_score:float=1.0, **kw
)->fastcore.foundation.L:

The k documents most likely to be noise, leaving out those a person has already judged.

Learning from marks

The fixed weights need no labels. After a person marks a few documents as noisy and a few as worth keeping, fit_noise fits the weights to those marks. On held-out documents the fitted weights scored 0.996 AUC. They were fitted from six noisy documents and eighteen confirmed keepers (evals/RESULTS.md). A fitted blend changes nothing until you call use_noise(True).

The ranker

The ranker is a pairwise linear model trained on the feedback log. For each question it learns which of two sections ranks higher. fit_ranker(save=True) fits the ranker and stores it. use_ranker(True) is a separate call. It makes context use the ranker, and ask uses context. The ranker is off by default. In the measurements no learned reranker beat plain reciprocal rank fusion in a repeatable way.

The features

PAIR_FEATURES are the signals that exist when a section is retrieved. They are the retrieval score and rank, the gap to the top score, the words the section shares with the question (raw and weighted by rarity), the cosine similarity to the question, the section length, the document prior, the kind and age of the document, and the noise features. The method is in evals/.

source

Vault.pair_X

def pair_X(
    q:str, hits, prior:dict=None
)->numpy.ndarray:

The design matrix for one question’s hits, in PAIR_FEATURES order.

Fitting, storing, switching on

Logging, fitting and switching on are separate calls. Nothing changes retrieval until you call use_ranker(True). learn() starts the log. training_data turns the log into rows grouped by question. fit_ranker and fit_noise fit the models. use_ranker and use_noise switch the stored models on. use_noise affects only suggest_noisy. evals/ holds the measurements on whether a collection should use a ranker.

source

Vault.retune

def retune(
    q:str, hits, ranker:NoneType=None, alpha:float=0.25, rrf_k:int=60
)->fastcore.foundation.L:

Reorder hits by baseline reciprocal rank fusion plus the ranker’s order, weighted by alpha. At alpha=0.25 nDCG@10 changed by -0.001. Replacing the baseline order with the ranker’s order changed it by -0.127.

source

Vault.use_ranker

def use_ranker(
    on:bool=True
)->dict:

Switch the stored ranker on (or off) in context, and so in ask.

source

Vault.use_noise

def use_noise(
    on:bool=True
)->dict:

Switch the stored noise weights on (or off) for suggest_noisy.

source

Vault.fit_noise

def fit_noise(
    labels:dict=None, l2:float=1.0, save:bool=False, **kw
)->litesearch.quality.Ranker:

Fit the noise weights to the documents a person marked; save=True stores them, and use_noise(True) switches them on.

source

Vault.fit_ranker

def fit_ranker(
    l2:float=1.0, save:bool=False, **kw
)->litesearch.quality.Ranker:

Fit a ranker on the feedback logged so far; save=True stores it without switching it on.

source

Vault.training_data

def training_data(
    min_group:int=2
)->fastcore.basics.AttrDict:

Turn the feedback log into (X, groups, y, weights) for the ranker, dropping questions whose rows are all one label.

source

Vault.learn

def learn(
    on:bool=True
)->dict:

Start (or with on=False stop) logging every ask as feedback; nothing is logged until this is called.

A worked check

The cells below only check that the parts connect, on a small made-up collection; they measure nothing. The measurements are in evals/.