# quality A vault fills up with things that match many searches and answer none: cookie banners, site footers, navigation pages, licence boilerplate. This page describes three tools for that problem. All three are off until you turn them on. 1. A noise score ranks the documents that are most likely junk. It needs no labels. On the project’s test corpora it scores 0.988 AUC, and 0.996 after it is fitted to a person’s marks. AUC is the area under the ROC curve. A score of 1.0 is a perfect ranking and 0.5 is chance. The score only suggests. A person reviews `suggest_noisy`, then marks documents with `mark_noisy` or `accept_noisy`. 2. A feedback log records which sections an answer cited and which sections a person rated. It records them after `learn()` is on. 3. A ranker is fitted from that log and reorders retrieval. No learned ranker beat plain reciprocal rank fusion in a repeatable way (`evals/RESULTS.md`). Leave the ranker off unless it helps on your own collection. ## What was good `rate` records one judgement about one section for one question. `SIGNALS` maps each signal to a label (1 good, 0 bad) and a weight.

signal	label	weight
`up`, `good`	1	3
`down`, `bad`	0	3
`read`: the person read the section	1	1.5
`cited`: an answer cited the section	1	1
`shown`: the section was offered and not cited	0	0.35

`ratings` returns the log, newest first. ------------------------------------------------------------------------ source ### Vault.ratings ``` python def ratings( doc_id:str=None, limit:int=None )->fastcore.foundation.L: ``` *The feedback log for this shelf, newest first.* ------------------------------------------------------------------------ source ### Vault.rate ``` python def rate( q:str, # the question this judgement is about node_id:str=None, # the section being judged doc_id:str=None, # its document; derived from `node_id` when not given signal:str='up', # one of SIGNALS rank:int=None, # where it was ranked when it was shown score:float=None, # the retrieval score it was shown with ask_id:str=None, # groups the rows from one question together )->dict: ``` *Record one judgement about one section for one question.* ``` python import tempfile from pathlib import Path with tempfile.TemporaryDirectory() as d: v = Vault(Path(d)/'test.db', offline=True) assert 'feedback' in v.db.t assert v.ratings() == [] v.rate('question', node_id='doc#1', signal='up') assert len(v.ratings()) == 1 ``` ### Citations are the label `ask` returns the sections its answer cited. After `learn()`, every `ask` calls `log_ask`. `log_ask` writes the cited sections to the log as positives and the other sections as weak negatives. An answer that cites nothing writes nothing, unless the caller passes `up` or `down`. The ranker and the document prior learn from these rows. ------------------------------------------------------------------------ source ### Vault.log_ask ``` python def log_ask( out, # what `ask` returned up:list=None, # section numbers you thought were good, as `[1]`-style numbers down:list=None, # section numbers you thought were bad weak:bool=True, # log the uncited sections as weak negatives )->int: ``` *Write feedback rows for one answered question from what `ask` returned; `learn()` does this for every `ask`.* ## The Beta prior `doc_prior` turns the log into a multiplier on the retrieval score of each document. It models the usefulness of a document as a Beta distribution. The weighted wins and losses of the document update the distribution. A document with little evidence stays near neutral. A document cited once does not move far ahead. `gamma` sets how strongly the prior changes a score. `explore` is the chance, from 0 to 1, that `doc_prior` samples from the distribution (Thompson sampling) and does not use its mean. Sampling keeps documents with little evidence in play. `lo` and `hi` clamp every multiplier, so no amount of evidence can bury a document. ------------------------------------------------------------------------ source ### Vault.doc_prior ``` python def doc_prior( gamma:float=0.35, # how hard the prior is allowed to push; 0 disables it explore:float=1.0, # 1 -> Thompson sample the posterior, 0 -> use its mean lo:float=0.6, hi:float=1.6, # clamp, so no amount of evidence can bury a document seed:int=None, exclude_ask:NoneType=None, # ignore these questions' rows; for an out-of-fold prior stats:dict=None, # `{doc_id: (wins, losses)}` to use instead of reading the log )->dict: ``` *Each document’s multiplier on its retrieval score, from a Beta distribution over its feedback, as `{doc_id: multiplier}`.* ------------------------------------------------------------------------ source ### Vault.doc_stats ``` python def doc_stats( exclude_ask:NoneType=None )->dict: ``` *Each document’s weighted wins and losses in this shelf’s feedback log, as `{doc_id: (wins, losses)}`.* ## Noise candidates `suggest_noisy` ranks the documents that look most like junk. It leaves out each document that a person has already judged for noise. `mark_noisy` records the decision. `search`, `sections`, `context` and `ask` leave out a document marked as noise. `accept_noisy` marks the current top suggestions in one call. `mark_noisy_many` marks a list of documents. The features and the `Ranker` come from [litesearch](https://Karthik777.github.io/litesearch/quality.html), which computes them from the chunk store alone. The 0.988 AUC is measured there. The features include these:

feature	what it measures
`hub`	how often a chunk appears among the nearest neighbours of other chunks
`spread_chunk`	how evenly a chunk sits between topics
`off_centre`	how far a chunk is from the middle of the collection
`dup_out`	how much of a chunk is duplicated in other documents

This page adds the vault’s side. It adds the documents that a person marked and the questions each document was retrieved for (`promiscuity`). It also adds the rule that the score only suggests. ------------------------------------------------------------------------ source ### Vault.chunk_matrix ``` python def chunk_matrix( limit:int=None, seed:int=0 )->tuple: ``` *`(ids, doc_ids, texts, V)` for this shelf: litesearch holds the vectors, this names the store.* ------------------------------------------------------------------------ source ### Vault.noise_scores ``` python def noise_scores( weights:dict=None, ranker:NoneType=None, **kw )->fastcore.foundation.L: ``` *Score every document for noise, most suspicious first, with the features behind each score.* ------------------------------------------------------------------------ source ### Vault.noise_features ``` python def noise_features( **kw )->fastcore.basics.AttrDict: ``` *litesearch’s noise features for this shelf’s documents, with `promiscuity` taken from the vault’s feedback log.* ------------------------------------------------------------------------ source ### Vault.accept_noisy ``` python def accept_noisy( k:int=20, reason:str='suggested', **kw )->fastcore.foundation.L: ``` *Mark the current top `k` noise suggestions as noise.* ------------------------------------------------------------------------ source ### Vault.mark_noisy_many ``` python def mark_noisy_many( refs, noisy:bool=True, reason:str='' )->fastcore.foundation.L: ``` *Mark many documents as noise at once, for example a reviewed `suggest_noisy` list.* ------------------------------------------------------------------------ source ### Vault.suggest_noisy ``` python def suggest_noisy( k:int=20, min_score:float=1.0, **kw )->fastcore.foundation.L: ``` *The `k` documents most likely to be noise, leaving out those a person has already judged.* ### Learning from marks The fixed weights need no labels. After a person marks a few documents as noisy and a few as worth keeping, `fit_noise` fits the weights to those marks. On held-out documents the fitted weights scored 0.996 AUC. They were fitted from six noisy documents and eighteen confirmed keepers (`evals/RESULTS.md`). A fitted blend changes nothing until you call `use_noise(True)`. ## The ranker The ranker is a pairwise linear model trained on the feedback log. For each question it learns which of two sections ranks higher. `fit_ranker(save=True)` fits the ranker and stores it. `use_ranker(True)` is a separate call. It makes `context` use the ranker, and `ask` uses `context`. The ranker is off by default. In the measurements no learned reranker beat plain reciprocal rank fusion in a repeatable way. ### The features `PAIR_FEATURES` are the signals that exist when a section is retrieved. They are the retrieval score and rank, the gap to the top score, the words the section shares with the question (raw and weighted by rarity), the cosine similarity to the question, the section length, the document prior, the kind and age of the document, and the noise features. The method is in `evals/`. ------------------------------------------------------------------------ source ### Vault.pair_X ``` python def pair_X( q:str, hits, prior:dict=None )->numpy.ndarray: ``` *The design matrix for one question’s hits, in `PAIR_FEATURES` order.* ## Fitting, storing, switching on Logging, fitting and switching on are separate calls. Nothing changes retrieval until you call `use_ranker(True)`. `learn()` starts the log. `training_data` turns the log into rows grouped by question. `fit_ranker` and `fit_noise` fit the models. `use_ranker` and `use_noise` switch the stored models on. `use_noise` affects only `suggest_noisy`. `evals/` holds the measurements on whether a collection should use a ranker. ------------------------------------------------------------------------ source ### Vault.retune ``` python def retune( q:str, hits, ranker:NoneType=None, alpha:float=0.25, rrf_k:int=60 )->fastcore.foundation.L: ``` *Reorder `hits` by baseline reciprocal rank fusion plus the ranker’s order, weighted by `alpha`. At `alpha=0.25` nDCG@10 changed by -0.001. Replacing the baseline order with the ranker’s order changed it by -0.127.* ------------------------------------------------------------------------ source ### Vault.use_ranker ``` python def use_ranker( on:bool=True )->dict: ``` *Switch the stored ranker on (or off) in `context`, and so in `ask`.* ------------------------------------------------------------------------ source ### Vault.use_noise ``` python def use_noise( on:bool=True )->dict: ``` *Switch the stored noise weights on (or off) for `suggest_noisy`.* ------------------------------------------------------------------------ source ### Vault.fit_noise ``` python def fit_noise( labels:dict=None, l2:float=1.0, save:bool=False, **kw )->litesearch.quality.Ranker: ``` *Fit the noise weights to the documents a person marked; `save=True` stores them, and `use_noise(True)` switches them on.* ------------------------------------------------------------------------ source ### Vault.fit_ranker ``` python def fit_ranker( l2:float=1.0, save:bool=False, **kw )->litesearch.quality.Ranker: ``` *Fit a ranker on the feedback logged so far; `save=True` stores it without switching it on.* ------------------------------------------------------------------------ source ### Vault.training_data ``` python def training_data( min_group:int=2 )->fastcore.basics.AttrDict: ``` *Turn the feedback log into `(X, groups, y, weights)` for the ranker, dropping questions whose rows are all one label.* ------------------------------------------------------------------------ source ### Vault.learn ``` python def learn( on:bool=True )->dict: ``` *Start (or with `on=False` stop) logging every `ask` as feedback; nothing is logged until this is called.* ## A worked check The cells below only check that the parts connect, on a small made-up collection; they measure nothing. The measurements are in `evals/`.