# lemma
## Lemmas
A lemma is the dictionary form of a word: the root gam behind gacchati,
or the stem rāma behind rāmeṇa.
[`fold_token`](https://vedicreader.github.io/ganapati/sanskrit.html#fold_token)
merges two spellings of one form, but it cannot connect two forms of one
word, so a search for `gam` will not find `gacchati`. That needs a
lexicon of inflected forms. ganapati uses the one in vidyut, a Sanskrit
toolkit from the Ambuda project. vidyut is installed with ganapati, but
its data is an 81 MB download, made only the first time you ask for a
lemma. The rest of ganapati works without it.
[`sanskrit_home`](https://vedicreader.github.io/ganapati/lemma.html#sanskrit_home)
is the folder for downloaded data: `$LITESEARCH_HOME/sanskrit` if that
variable is set, otherwise `litesearch/sanskrit` in the XDG cache folder
(`~/.cache` by default).
[`vidyut_data`](https://vedicreader.github.io/ganapati/lemma.html#vidyut_data)
downloads vidyut’s data there. vidyut keys its lexicon, which it calls
the kosha, on SLP1, an ASCII transliteration with one letter for each
sound.
[`to_slp1`](https://vedicreader.github.io/ganapati/lemma.html#to_slp1)
and
[`from_slp1`](https://vedicreader.github.io/ganapati/lemma.html#from_slp1)
convert to and from it and need no download.
[`vidyut_pipe`](https://vedicreader.github.io/ganapati/lemma.html#vidyut_pipe)
returns a lemmatiser: a function that takes text and returns one token
for each lemma it finds, with `.text` (the written word), `.lemma_` (the
lemma in IAST) and `.lemma_slp` (the lemma in SLP1). The tokens look
like spaCy’s, so code written for a spaCy pipeline accepts them. A
written word that is not in the kosha is often two or more words joined
by sandhi, the sound changes where words meet, and with `split=True` the
lemmatiser tries to split it into pieces the kosha knows.
[`sanskrit_terms`](https://vedicreader.github.io/ganapati/lemma.html#sanskrit_terms)
is built the same way for litesearch’s `build_graph`: it returns the
most frequent words of a chunk that are not finite verbs, which are
mostly nouns and adjectives.
For English,
[`mw_lexicon`](https://vedicreader.github.io/ganapati/lemma.html#mw_lexicon)
downloads the Monier-Williams dictionary from the Cologne Sanskrit
Lexicon and reduces each entry to a short gloss, keyed by its SLP1
headword.
[`gloss_facets`](https://vedicreader.github.io/ganapati/lemma.html#gloss_facets)
uses it with a lemmatiser to list the English words behind a chunk’s
Sanskrit.
/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
------------------------------------------------------------------------
source
### gloss_facets
``` python
def gloss_facets(
text:str, # chunk text
nlp, # a `vidyut_pipe()`; tokens must carry `.lemma_slp`
mw:dict, # a `mw_lexicon()`
max_terms:int=24, # cap, so the gloss cannot outgrow the verse
)->dict:
```
*Return `{'gloss': ...}`, the English words behind a chunk’s Sanskrit,
from a lemmatiser and
[`mw_lexicon`](https://vedicreader.github.io/ganapati/lemma.html#mw_lexicon).*
------------------------------------------------------------------------
source
### mw_lexicon
``` python
def mw_lexicon(
path:NoneType=None, force:bool=False, maxlen:int=110
)->dict:
```
*Return Monier-Williams as `{SLP1 headword: short English gloss}`,
downloading and reducing it on first use.*
------------------------------------------------------------------------
source
### sanskrit_terms
``` python
def sanskrit_terms(
path:NoneType=None, # vidyut data dir; None -> `sanskrit_home()`
topk:int=12, # number of words kept per chunk, most frequent first
min_len:int=3, # skip particles and one-syllable words
split:bool=True, # split a word the kosha does not know into the words sandhi joined
):
```
*Return a `terms_fn` for litesearch’s `build_graph` that lists the most
frequent words of a chunk that are not finite verbs.*
------------------------------------------------------------------------
source
### vidyut_pipe
``` python
def vidyut_pipe(
path:NoneType=None, split:bool=True
):
```
*Return a lemmatiser backed by vidyut’s kosha that turns text into
spaCy-style tokens with `.lemma_` and `.lemma_slp`.*
------------------------------------------------------------------------
source
### from_slp1
``` python
def from_slp1(
text:str, iast:bool=True
)->str:
```
*Transliterate SLP1 to IAST, or to Devanagari when `iast=False`.*
------------------------------------------------------------------------
source
### to_slp1
``` python
def to_slp1(
text:str
)->str:
```
*Transliterate Devanagari or IAST to SLP1, the scheme the kosha is keyed
on, with no downloaded data.*
------------------------------------------------------------------------
source
### vidyut_data
``` python
def vidyut_data(
path:NoneType=None, force:bool=False
)->pathlib.Path:
```
*Return the folder of vidyut’s data, downloading it (about 81 MB) on
first use.*
------------------------------------------------------------------------
source
### sanskrit_home
``` python
def sanskrit_home(
name:str=None
)->pathlib.Path:
```
*Return the folder for downloaded Sanskrit data, or the file `name` in
it, under `$LITESEARCH_HOME` or else the XDG cache.*
## Facets
A facet is one field of a chunk’s metadata, such as `meter`, `lemma` or
`gloss`, that a search can filter on.
[`lemma_facets`](https://vedicreader.github.io/ganapati/lemma.html#lemma_facets)
runs a lemmatiser over a chunk and returns `{'lemma': ...}` with the
lemmas it found. It leaves out a lemma whose
[`fold_token`](https://vedicreader.github.io/ganapati/sanskrit.html#fold_token)
key is the same as the written word’s, because the ordinary text index
finds that word already.
Some sources already carry a word-by-word analysis, and then no
lemmatiser is needed. vedicreader files write it on `> etym:` lines, one
entry per word in the form `surface, lemma, grammar..., gloss`, for
example
`गणानां, gaṇa, genitive case, masculine gender, plural number, of troops.`
[`etym_entries`](https://vedicreader.github.io/ganapati/lemma.html#etym_entries)
reads such lines into `AttrDict(w, lemma, gram, g)` records and skips
lines of any other shape.
[`etym_facets`](https://vedicreader.github.io/ganapati/lemma.html#etym_facets)
collects the lemmas and the English gloss words from a chunk’s `> etym:`
lines, leaving out grammar terms and common English words. It also
handles two looser shapes, `word: gloss` entries and free English prose.
[`source_meta`](https://vedicreader.github.io/ganapati/lemma.html#source_meta)
combines everything the source supplies itself: the metre facets of
[`verse_meta`](https://vedicreader.github.io/ganapati/metre.html#verse_meta)
and the facets of
[`etym_facets`](https://vedicreader.github.io/ganapati/lemma.html#etym_facets).
[`sanskrit_meta`](https://vedicreader.github.io/ganapati/lemma.html#sanskrit_meta)
builds the function litesearch calls to fill a chunk’s metadata. With no
arguments it returns
[`source_meta`](https://vedicreader.github.io/ganapati/lemma.html#source_meta)
itself, which needs neither vidyut nor its download. Given a lemmatiser,
and optionally a
[`mw_lexicon()`](https://vedicreader.github.io/ganapati/lemma.html#mw_lexicon),
it also runs vidyut, but only on the chunks whose source did not supply
lemmas or glosses.
------------------------------------------------------------------------
source
### Database.by_meter
``` python
def by_meter(
meter:ganapati.metre.Meter | str=None, # metre, e.g. `Meter.MANDAKRANTA`
gana:str=None, # gaṇa signature, `ma bha na ta ta ga ga` or `ma_bha_na_...`
store:str='store', prefix:str=None, columns:list=None, limit:int=50
)->list:
```
*Return the chunks with a verse in `meter` or with the gaṇa signature
`gana`, unranked.*
------------------------------------------------------------------------
source
### Database.by_lemma
``` python
def by_lemma(
lemma:str, # a dictionary form, e.g. `gam`
store:str='store', prefix:str=None, columns:list=None, limit:int=50
)->list:
```
*Return the chunks with a form of `lemma` in them, from an index built
after `register_profiles(nlp=...)` or from sources with their own
analysis.*
------------------------------------------------------------------------
source
### sanskrit_meta
``` python
def sanskrit_meta(
nlp:NoneType=None, mw:dict=None
):
```
*Build the function that fills a chunk’s metadata: the source’s own
facets always, and vidyut’s only where the source has none.*
------------------------------------------------------------------------
source
### source_meta
``` python
def source_meta(
text:str
)->dict:
```
*Return the facets the source supplies itself: metre, audio timings and
its own word-by-word analysis.*
------------------------------------------------------------------------
source
### etym_facets
``` python
def etym_facets(
text:str, # chunk text
max_lemmas:int=96, # cap on the Sanskrit side
max_terms:int=24, # cap on the English side
)->dict:
```
*Return `{'lemma': ..., 'gloss': ...}` from the source’s own `> etym:`
lines, without vidyut.*
------------------------------------------------------------------------
source
### etym_entries
``` python
def etym_entries(
text:str
)->fastcore.foundation.L:
```
*Read the lines of a word-by-word analysis in the form
`surface, lemma, grammar…, gloss` into `AttrDict(w, lemma, gram, g)`
records, skipping other lines.*
------------------------------------------------------------------------
source
### lemma_facets
``` python
def lemma_facets(
text:str, # chunk text
nlp, # a `vidyut_pipe()` (or any spaCy-shaped pipeline)
max_terms:int=96, # cap, so one chunk's metadata cannot outgrow its content
)->dict:
```
*Return `{'lemma': ...}`, the dictionary forms behind a chunk’s
inflected words, leaving out those the text index already finds.*
The example gives a vedicreader line with an audio span and a
`word: gloss` analysis, and reads the lemmas, the gloss words and the
mātrā rate out of it.
``` python
from ganapati.text import vr_xml_parse
```
``` python
VR = '''
॥ श्रीरुद्रचमकप्रश्नः ॥
अग्नाविष्णू सजोषसेमा
'''
f = etym_facets(vr_xml_parse(VR)[0][-1][1])
assert f['lemma'] == 'agni viṣṇu' # the headword of each `word: gloss` entry
assert f['gloss'] == 'fire god all-pervader' # and no grammar, no stopwords
assert sanskrit_meta() is source_meta # the cheap path, and no vidyut in it
assert source_meta(vr_xml_parse(VR)[0][-1][1])['mora_rate'] == '112'
```
## Searching by metre and lemma
`by_lemma` and `by_meter` are added to fastlite’s `Database`, so you
call them on an index’s `db`, as in
`ix.db.by_meter(meter=Meter.ANUSTUBH)`. They filter chunks on the
metadata the facets wrote and return rows with `content`, `metadata`,
`node_id`, `doc_id` and `page`. They narrow the set of chunks and do not
rank it; ordering results is the job of litesearch’s search. `by_meter`
takes a `Meter` or a metre’s name, a gaṇa signature such as
`ma bha na ta ta ga ga`, or both, and returns `[]` if given neither.
`by_lemma` takes a lemma in IAST, such as `gam`, and finds something
only if the chunks have `lemma` facets.
## Registering with litesearch
litesearch decides how to read a file from a profile: the extensions it
covers, a test for files whose extension other formats share, the
chunker, and the function that writes each chunk’s metadata.
[`register_profiles`](https://vedicreader.github.io/ganapati/lemma.html#register_profiles)
registers two profiles. `sanskrit_verse` covers `.xml`, `.json`, `.tei`,
`.htm`, `.html`, `.conllu` and `.txt` files whose contents look like
Sanskrit, and chunks them with
[`VerseChunker`](https://vedicreader.github.io/ganapati/text.html#versechunker).
`sanskrit_prose` uses
[`ProseChunker`](https://vedicreader.github.io/ganapati/text.html#prosechunker);
it has no extensions and never matches a file on its own, so it applies
only where a caller chooses it.
You don’t normally call
[`register_profiles`](https://vedicreader.github.io/ganapati/lemma.html#register_profiles)
yourself. ganapati declares it as an entry point in litesearch’s
`litesearch.profiles` group, and litesearch 0.1.38 and later calls it
the first time it looks up a profile, so ganapati never imports
litesearch to register. Install both with
`pip install 'ganapati[search]'`. Call
`register_profiles(nlp=vidyut_pipe(), mw=mw_lexicon())` to add lemmas
and glosses to every document indexed after that. It is safe to call
more than once.
------------------------------------------------------------------------
source
### register_profiles
``` python
def register_profiles(
nlp:NoneType=None, mw:dict=None
):
```
*Register the `sanskrit_verse` and `sanskrit_prose` profiles with
litesearch, which calls this through its `litesearch.profiles` entry
point.*
The cells below check that litesearch picks the verse profile for a
GRETIL file and that the metadata function works without vidyut. They
then read word-by-word entries in the corpus’s own shape and in the two
looser shapes.
``` python
from litesearch.data import profile_for
from litesearch import Index
import json, numpy as np
from fastcore.all import Path, first
```
``` python
gretil = "atha prathamo'dhyāyaḥ || MBh_1,1.1 ||\n" * 10
p = profile_for('mahabharata_u.htm', gretil)
assert p and p.name == 'sanskrit_verse'
assert p.chunker is VerseChunker and p.kind == 'sanskrit'
assert sanskrit_meta(None) is source_meta # no pipeline, no vidyut import
# and it still carries the metre: a real verse in, its metre out
assert sanskrit_meta(None)('dharmakṣetre kurukṣetre samavetā yuyutsavaḥ '
'māmakāḥ pāṇḍavāścaiva kimakurvata sañjaya')['meter'] == 'anuṣṭubh'
assert sanskrit_meta(None)('') == {}
```
``` python
# the corpus's own shape: `surface, lemma, grammar…, gloss`, one entry per line
E = """> etym: गणानां, gaṇa, genitive case, masculine gender, plural number, of troops.
> etym: हवामहे, hvā, present tense, indicative mood, first person, plural number, we invoke.
> etym: विष्णुं, viṣṇu, case=accusative, gender=masculine, number=singular, Viṣṇu, the object of the meditation."""
f = etym_facets(E)
assert 'gaṇa' in f['lemma'] and 'hvā' in f['lemma'] and 'गणानां' in f['lemma']
# the grammar fields go to neither side: the lemma already implies them
assert 'troops' in f['gloss'] and 'invoke' in f['gloss']
assert not {'genitive', 'masculine', 'plural', 'accusative', 'tense', 'case'} & set(f['gloss'].split())
# the two other shapes a source may use still split
assert 'agnā' in etym_facets('> etym: agnā: agni, voc. du.')['lemma']
assert 'dharma' in etym_facets('> etym: The compound joins dharma with kshetra.')['gloss']
```
``` python
# the same lines read as records: surface, lemma, the grammar fields, the gloss
es = etym_entries('गणानां, gaṇa, genitive case, masculine gender, plural number, of troops.\n'
'विष्णुं, viṣṇu, case=accusative, gender=masculine, number=singular, Viṣṇu, the object of the meditation.\n'
'तत्, tad, nominal stem, form of the pronoun, that.\n'
'- agnā: agni, voc. du.') # a `word: gloss` bullet is the other shape, not an entry
assert len(es) == 3
assert es[0].w == 'गणानां' and es[0].lemma == 'gaṇa' and es[0].gram == 'genitive case, masculine gender, plural number' and es[0].g == 'of troops.'
assert es[1].gram == 'case=accusative, gender=masculine, number=singular' and es[1].g == 'Viṣṇu, the object of the meditation.'
assert es[2].gram == 'nominal stem, form of the pronoun' and es[2].g == 'that.' # a gloss with a comma in it stays whole
assert etym_entries('') == [] and etym_entries('just prose, with commas, in it') == []
```
## Glosses written in the wrong script
A language model asked for word-by-word entries with Devanagari
headwords sometimes writes the English gloss in Devanagari too, as
though the English were IAST, so `an ox; cattle` comes out as
`अन् ओक्ष्; चत्त्ले`.
[`delatinize`](https://vedicreader.github.io/ganapati/lemma.html#delatinize)
finds such a line and turns it back into English. It transliterates the
Devanagari back into a roman scheme (IAST, ITRANS, Harvard-Kyoto or
Velthuis) and keeps the result only if enough of it is English. English
here means the words in
[`gloss_vocab`](https://vedicreader.github.io/ganapati/lemma.html#gloss_vocab),
the vocabulary of the corpus’s own well-formed glosses, so a word only
counts if the corpus has used it in a gloss before. A correct entry and
a line of real Sanskrit come back unchanged.
[`latin_frac`](https://vedicreader.github.io/ganapati/lemma.html#latin_frac)
gives the share of a line that reads back as English words, which is the
test
[`delatinize`](https://vedicreader.github.io/ganapati/lemma.html#delatinize)
applies.
[`fix_etym`](https://vedicreader.github.io/ganapati/lemma.html#fix_etym)
runs
[`delatinize`](https://vedicreader.github.io/ganapati/lemma.html#delatinize)
on every line of an analysis and counts the lines it changed.
Transliterating back needs aksharamukha, which is not installed with
ganapati; the first call tells you to install it.
------------------------------------------------------------------------
source
### fix_etym
``` python
def fix_etym(
text:str, vocab:set
)->tuple:
```
*Run
[`delatinize`](https://vedicreader.github.io/ganapati/lemma.html#delatinize)
on each line of a word-by-word analysis, returning `(text, n_fixed)`.*
------------------------------------------------------------------------
source
### delatinize
``` python
def delatinize(
s:str, vocab:set
)->str:
```
*Turn an English gloss wrongly written in Devanagari back into English,
and return any other line unchanged.*
------------------------------------------------------------------------
source
### latin_frac
``` python
def latin_frac(
s:str, # one Devanagari line
vocab:set, # the corpus's English gloss words, from `gloss_vocab`
scheme:str='IAST'
)->tuple:
```
*Return the share of a Devanagari line that reads back as English,
counting only runs of two or more words, as `(fraction, n_words)`.*
------------------------------------------------------------------------
source
### gloss_vocab
``` python
def gloss_vocab(
texts
)->set:
```
*Collect the English words in a corpus’s well-formed glosses, the
reference
[`delatinize`](https://vedicreader.github.io/ganapati/lemma.html#delatinize)
checks against.*
``` python
v = gloss_vocab(['गणानां, gaṇa, genitive case, of troops.',
'गौः, go, nominative case, an ox; cattle; the earth.',
'सूर्यः, sūrya, nominative case, the sun; light of the sun.'])
assert v == {'an', 'cattle', 'earth', 'light', 'of', 'ox', 'sun', 'the', 'troops'}
bad = 'थे सुन्; लिघ्त् ओफ़् थे सुन्' # `the sun; light of the sun`, written as if it were IAST
assert latin_frac(bad, v) == (1.0, 6)
assert delatinize(bad, v) == 'the sun; light of the sun'
# left alone: real Sanskrit, a well-formed entry, a header, a `word: gloss` bullet
for s in ['गणानां त्वा गणपतिं हवामहे', 'गणानां, gaṇa, genitive case, of troops.', 'ॐ भूः', '- ॐ: Sacred syllable', '']:
assert delatinize(s, v) == s
assert latin_frac('गणानां त्वा गणपतिं हवामहे', v) == (0.0, 4)
t, n = fix_etym(f'ॐ भूः\n{bad}\nगणानां, gaṇa, genitive case, of troops.', v)
assert n == 1 and t.splitlines()[1] == 'the sun; light of the sun' and t.splitlines()[0] == 'ॐ भूः'
```
``` python
# a real numbered HTML edition follows the same profile through Index.add.
# `mahabharata.htm` is GRETIL's Mahābhārata 1.1, 210 verses:
# gretil.sub.uni-goettingen.de/gretil/1_sanskr/2_epic/mbh/sas/b01/b01c001.htm
mbh = first(p for p in (Path('mahabharata.htm'), Path('nbs/mahabharata.htm')) if p.exists())
ix = Index()
assert ix.add(mbh) > 0
ix.docs
```
/Users/71293/code/personal/orgs/ganapati/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
[{'id': '73a63025c7fc3d14',
'title': 'Mahabharata online',
'source': 'mahabharata.htm',
'kind': 'sanskrit',
'pages': 215,
'meta': '{"fmt": "gretil", "title": "Mahabharata online", "header": "Mahabharata online"}',
'added_at': '2026-08-30 02:25:00'}]
``` python
assert ix.docs[0]['title'] == 'Mahabharata online'
assert ix.docs[0]['kind'] == 'sanskrit' and ix.docs[0]['pages'] > 1
```
``` python
hits = ix.db.by_meter(meter=Meter.ANUSTUBH)
assert hits and json.loads(hits[0]['metadata'])['meter'] == Meter.ANUSTUBH.value
```
``` python
nlp = vidyut_pipe()
f = lemma_facets('gacchati rāmaḥ', nlp)
assert 'gam' in f['lemma']
```