lemma

Find the dictionary forms and English glosses behind Sanskrit words, and register ganapati’s readers with litesearch

Lemmas

A lemma is the dictionary form of a word: the root gam behind gacchati, or the stem rāma behind rāmeṇa. fold_token merges two spellings of one form, but it cannot connect two forms of one word, so a search for gam will not find gacchati. That needs a lexicon of inflected forms. ganapati uses the one in vidyut, a Sanskrit toolkit from the Ambuda project. vidyut is installed with ganapati, but its data is an 81 MB download, made only the first time you ask for a lemma. The rest of ganapati works without it.

sanskrit_home is the folder for downloaded data: $LITESEARCH_HOME/sanskrit if that variable is set, otherwise litesearch/sanskrit in the XDG cache folder (~/.cache by default). vidyut_data downloads vidyut’s data there. vidyut keys its lexicon, which it calls the kosha, on SLP1, an ASCII transliteration with one letter for each sound. to_slp1 and from_slp1 convert to and from it and need no download.

vidyut_pipe returns a lemmatiser: a function that takes text and returns one token for each lemma it finds, with .text (the written word), .lemma_ (the lemma in IAST) and .lemma_slp (the lemma in SLP1). The tokens look like spaCy’s, so code written for a spaCy pipeline accepts them. A written word that is not in the kosha is often two or more words joined by sandhi, the sound changes where words meet, and with split=True the lemmatiser tries to split it into pieces the kosha knows. sanskrit_terms is built the same way for litesearch’s build_graph: it returns the most frequent words of a chunk that are not finite verbs, which are mostly nouns and adjectives.

For English, mw_lexicon downloads the Monier-Williams dictionary from the Cologne Sanskrit Lexicon and reduces each entry to a short gloss, keyed by its SLP1 headword. gloss_facets uses it with a lemmatiser to list the English words behind a chunk’s Sanskrit.

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)

gloss_facets

def gloss_facets(
    text:str, # chunk text
    nlp, # a `vidyut_pipe()`; tokens must carry `.lemma_slp`
    mw:dict, # a `mw_lexicon()`
    max_terms:int=24, # cap, so the gloss cannot outgrow the verse
)->dict:

Return {'gloss': ...}, the English words behind a chunk’s Sanskrit, from a lemmatiser and mw_lexicon.

mw_lexicon

def mw_lexicon(
    path:NoneType=None, force:bool=False, maxlen:int=110
)->dict:

Return Monier-Williams as {SLP1 headword: short English gloss}, downloading and reducing it on first use.

sanskrit_terms

def sanskrit_terms(
    path:NoneType=None, # vidyut data dir; None -> `sanskrit_home()`
    topk:int=12, # number of words kept per chunk, most frequent first
    min_len:int=3, # skip particles and one-syllable words
    split:bool=True, # split a word the kosha does not know into the words sandhi joined
):

Return a terms_fn for litesearch’s build_graph that lists the most frequent words of a chunk that are not finite verbs.

vidyut_pipe

def vidyut_pipe(
    path:NoneType=None, split:bool=True
):

Return a lemmatiser backed by vidyut’s kosha that turns text into spaCy-style tokens with .lemma_ and .lemma_slp.

from_slp1

def from_slp1(
    text:str, iast:bool=True
)->str:

Transliterate SLP1 to IAST, or to Devanagari when iast=False.

to_slp1

def to_slp1(
    text:str
)->str:

Transliterate Devanagari or IAST to SLP1, the scheme the kosha is keyed on, with no downloaded data.

vidyut_data

def vidyut_data(
    path:NoneType=None, force:bool=False
)->pathlib.Path:

Return the folder of vidyut’s data, downloading it (about 81 MB) on first use.

sanskrit_home

def sanskrit_home(
    name:str=None
)->pathlib.Path:

Return the folder for downloaded Sanskrit data, or the file name in it, under $LITESEARCH_HOME or else the XDG cache.

Facets

A facet is one field of a chunk’s metadata, such as meter, lemma or gloss, that a search can filter on. lemma_facets runs a lemmatiser over a chunk and returns {'lemma': ...} with the lemmas it found. It leaves out a lemma whose fold_token key is the same as the written word’s, because the ordinary text index finds that word already.

Some sources already carry a word-by-word analysis, and then no lemmatiser is needed. vedicreader files write it on > etym: lines, one entry per word in the form surface, lemma, grammar..., gloss, for example गणानां, gaṇa, genitive case, masculine gender, plural number, of troops. etym_entries reads such lines into AttrDict(w, lemma, gram, g) records and skips lines of any other shape. etym_facets collects the lemmas and the English gloss words from a chunk’s > etym: lines, leaving out grammar terms and common English words. It also handles two looser shapes, word: gloss entries and free English prose.

source_meta combines everything the source supplies itself: the metre facets of verse_meta and the facets of etym_facets. sanskrit_meta builds the function litesearch calls to fill a chunk’s metadata. With no arguments it returns source_meta itself, which needs neither vidyut nor its download. Given a lemmatiser, and optionally a mw_lexicon(), it also runs vidyut, but only on the chunks whose source did not supply lemmas or glosses.

Database.by_meter

def by_meter(
    meter:ganapati.metre.Meter | str=None, # metre, e.g. `Meter.MANDAKRANTA`
    gana:str=None, # gaṇa signature, `ma bha na ta ta ga ga` or `ma_bha_na_...`
    store:str='store', prefix:str=None, columns:list=None, limit:int=50
)->list:

Return the chunks with a verse in meter or with the gaṇa signature gana, unranked.

Database.by_lemma

def by_lemma(
    lemma:str, # a dictionary form, e.g. `gam`
    store:str='store', prefix:str=None, columns:list=None, limit:int=50
)->list:

Return the chunks with a form of lemma in them, from an index built after register_profiles(nlp=...) or from sources with their own analysis.

sanskrit_meta

def sanskrit_meta(
    nlp:NoneType=None, mw:dict=None
):

Build the function that fills a chunk’s metadata: the source’s own facets always, and vidyut’s only where the source has none.

source_meta

def source_meta(
    text:str
)->dict:

Return the facets the source supplies itself: metre, audio timings and its own word-by-word analysis.

etym_facets

def etym_facets(
    text:str, # chunk text
    max_lemmas:int=96, # cap on the Sanskrit side
    max_terms:int=24, # cap on the English side
)->dict:

Return {'lemma': ..., 'gloss': ...} from the source’s own > etym: lines, without vidyut.

etym_entries

def etym_entries(
    text:str
)->fastcore.foundation.L:

Read the lines of a word-by-word analysis in the form surface, lemma, grammar…, gloss into AttrDict(w, lemma, gram, g) records, skipping other lines.

lemma_facets

def lemma_facets(
    text:str, # chunk text
    nlp, # a `vidyut_pipe()` (or any spaCy-shaped pipeline)
    max_terms:int=96, # cap, so one chunk's metadata cannot outgrow its content
)->dict:

Return {'lemma': ...}, the dictionary forms behind a chunk’s inflected words, leaving out those the text index already finds.

The example gives a vedicreader line with an audio span and a word: gloss analysis, and reads the lemmas, the gloss words and the mātrā rate out of it.

from ganapati.text import vr_xml_parse

VR = '''<lyrics><section name="anuvaka 1">
  <line ignore="true">॥ श्रीरुद्रचमकप्रश्नः ॥</line>
  <line start_time_ms="0" end_time_ms="1800"
        etymology="agni: fire, the god of fire; viṣṇu: voc. du., the all-pervader">अग्नाविष्णू सजोषसेमा</line>
</section></lyrics>'''

f = etym_facets(vr_xml_parse(VR)[0][-1][1])
assert f['lemma'] == 'agni viṣṇu'                        # the headword of each `word: gloss` entry
assert f['gloss'] == 'fire god all-pervader'             # and no grammar, no stopwords
assert sanskrit_meta() is source_meta                    # the cheap path, and no vidyut in it
assert source_meta(vr_xml_parse(VR)[0][-1][1])['mora_rate'] == '112'

Searching by metre and lemma

by_lemma and by_meter are added to fastlite’s Database, so you call them on an index’s db, as in ix.db.by_meter(meter=Meter.ANUSTUBH). They filter chunks on the metadata the facets wrote and return rows with content, metadata, node_id, doc_id and page. They narrow the set of chunks and do not rank it; ordering results is the job of litesearch’s search. by_meter takes a Meter or a metre’s name, a gaṇa signature such as ma bha na ta ta ga ga, or both, and returns [] if given neither. by_lemma takes a lemma in IAST, such as gam, and finds something only if the chunks have lemma facets.

Registering with litesearch

litesearch decides how to read a file from a profile: the extensions it covers, a test for files whose extension other formats share, the chunker, and the function that writes each chunk’s metadata. register_profiles registers two profiles. sanskrit_verse covers .xml, .json, .tei, .htm, .html, .conllu and .txt files whose contents look like Sanskrit, and chunks them with VerseChunker. sanskrit_prose uses ProseChunker; it has no extensions and never matches a file on its own, so it applies only where a caller chooses it.

You don’t normally call register_profiles yourself. ganapati declares it as an entry point in litesearch’s litesearch.profiles group, and litesearch 0.1.38 and later calls it the first time it looks up a profile, so ganapati never imports litesearch to register. Install both with pip install 'ganapati[search]'. Call register_profiles(nlp=vidyut_pipe(), mw=mw_lexicon()) to add lemmas and glosses to every document indexed after that. It is safe to call more than once.

register_profiles

def register_profiles(
    nlp:NoneType=None, mw:dict=None
):

Register the sanskrit_verse and sanskrit_prose profiles with litesearch, which calls this through its litesearch.profiles entry point.

The cells below check that litesearch picks the verse profile for a GRETIL file and that the metadata function works without vidyut. They then read word-by-word entries in the corpus’s own shape and in the two looser shapes.

from litesearch.data import profile_for
from litesearch import Index
import json, numpy as np
from fastcore.all import Path, first

gretil = "atha prathamo'dhyāyaḥ || MBh_1,1.1 ||\n" * 10
p = profile_for('mahabharata_u.htm', gretil)
assert p and p.name == 'sanskrit_verse'
assert p.chunker is VerseChunker and p.kind == 'sanskrit'

assert sanskrit_meta(None) is source_meta          # no pipeline, no vidyut import
# and it still carries the metre: a real verse in, its metre out
assert sanskrit_meta(None)('dharmakṣetre kurukṣetre samavetā yuyutsavaḥ '
                          'māmakāḥ pāṇḍavāścaiva kimakurvata sañjaya')['meter'] == 'anuṣṭubh'
assert sanskrit_meta(None)('') == {}

# the corpus's own shape: `surface, lemma, grammar…, gloss`, one entry per line
E = """> etym: गणानां, gaṇa, genitive case, masculine gender, plural number, of troops.
> etym: हवामहे, hvā, present tense, indicative mood, first person, plural number, we invoke.
> etym: विष्णुं, viṣṇu, case=accusative, gender=masculine, number=singular, Viṣṇu, the object of the meditation."""
f = etym_facets(E)
assert 'gaṇa' in f['lemma'] and 'hvā' in f['lemma'] and 'गणानां' in f['lemma']
# the grammar fields go to neither side: the lemma already implies them
assert 'troops' in f['gloss'] and 'invoke' in f['gloss']
assert not {'genitive', 'masculine', 'plural', 'accusative', 'tense', 'case'} & set(f['gloss'].split())

# the two other shapes a source may use still split
assert 'agnā' in etym_facets('> etym: agnā: agni, voc. du.')['lemma']
assert 'dharma' in etym_facets('> etym: The compound joins dharma with kshetra.')['gloss']

# the same lines read as records: surface, lemma, the grammar fields, the gloss
es = etym_entries('गणानां, gaṇa, genitive case, masculine gender, plural number, of troops.\n'
                  'विष्णुं, viṣṇu, case=accusative, gender=masculine, number=singular, Viṣṇu, the object of the meditation.\n'
                  'तत्, tad, nominal stem, form of the pronoun, that.\n'
                  '- agnā: agni, voc. du.')            # a `word: gloss` bullet is the other shape, not an entry
assert len(es) == 3
assert es[0].w == 'गणानां' and es[0].lemma == 'gaṇa' and es[0].gram == 'genitive case, masculine gender, plural number' and es[0].g == 'of troops.'
assert es[1].gram == 'case=accusative, gender=masculine, number=singular' and es[1].g == 'Viṣṇu, the object of the meditation.'
assert es[2].gram == 'nominal stem, form of the pronoun' and es[2].g == 'that.'   # a gloss with a comma in it stays whole
assert etym_entries('') == [] and etym_entries('just prose, with commas, in it') == []

Glosses written in the wrong script

A language model asked for word-by-word entries with Devanagari headwords sometimes writes the English gloss in Devanagari too, as though the English were IAST, so an ox; cattle comes out as अन् ओक्ष्; चत्त्ले. delatinize finds such a line and turns it back into English. It transliterates the Devanagari back into a roman scheme (IAST, ITRANS, Harvard-Kyoto or Velthuis) and keeps the result only if enough of it is English. English here means the words in gloss_vocab, the vocabulary of the corpus’s own well-formed glosses, so a word only counts if the corpus has used it in a gloss before. A correct entry and a line of real Sanskrit come back unchanged.

latin_frac gives the share of a line that reads back as English words, which is the test delatinize applies. fix_etym runs delatinize on every line of an analysis and counts the lines it changed. Transliterating back needs aksharamukha, which is not installed with ganapati; the first call tells you to install it.

fix_etym

def fix_etym(
    text:str, vocab:set
)->tuple:

Run delatinize on each line of a word-by-word analysis, returning (text, n_fixed).

delatinize

def delatinize(
    s:str, vocab:set
)->str:

Turn an English gloss wrongly written in Devanagari back into English, and return any other line unchanged.

latin_frac

def latin_frac(
    s:str, # one Devanagari line
    vocab:set, # the corpus's English gloss words, from `gloss_vocab`
    scheme:str='IAST'
)->tuple:

Return the share of a Devanagari line that reads back as English, counting only runs of two or more words, as (fraction, n_words).

gloss_vocab

def gloss_vocab(
    texts
)->set:

Collect the English words in a corpus’s well-formed glosses, the reference delatinize checks against.

v = gloss_vocab(['गणानां, gaṇa, genitive case, of troops.',
                 'गौः, go, nominative case, an ox; cattle; the earth.',
                 'सूर्यः, sūrya, nominative case, the sun; light of the sun.'])
assert v == {'an', 'cattle', 'earth', 'light', 'of', 'ox', 'sun', 'the', 'troops'}

bad = 'थे सुन्; लिघ्त् ओफ़् थे सुन्'                  # `the sun; light of the sun`, written as if it were IAST
assert latin_frac(bad, v) == (1.0, 6)
assert delatinize(bad, v) == 'the sun; light of the sun'

# left alone: real Sanskrit, a well-formed entry, a header, a `word: gloss` bullet
for s in ['गणानां त्वा गणपतिं हवामहे', 'गणानां, gaṇa, genitive case, of troops.', 'ॐ भूः', '- ॐ: Sacred syllable', '']:
    assert delatinize(s, v) == s
assert latin_frac('गणानां त्वा गणपतिं हवामहे', v) == (0.0, 4)

t, n = fix_etym(f'ॐ भूः\n{bad}\nगणानां, gaṇa, genitive case, of troops.', v)
assert n == 1 and t.splitlines()[1] == 'the sun; light of the sun' and t.splitlines()[0] == 'ॐ भूः'

# a real numbered HTML edition follows the same profile through Index.add.
# `mahabharata.htm` is GRETIL's Mahābhārata 1.1, 210 verses:
# gretil.sub.uni-goettingen.de/gretil/1_sanskr/2_epic/mbh/sas/b01/b01c001.htm
mbh = first(p for p in (Path('mahabharata.htm'), Path('nbs/mahabharata.htm')) if p.exists())
ix = Index()
assert ix.add(mbh) > 0
ix.docs

/Users/71293/code/personal/orgs/ganapati/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

[{'id': '73a63025c7fc3d14',
  'title': 'Mahabharata online',
  'source': 'mahabharata.htm',
  'kind': 'sanskrit',
  'pages': 215,
  'meta': '{"fmt": "gretil", "title": "Mahabharata online", "header": "Mahabharata online"}',
  'added_at': '2026-08-30 02:25:00'}]

assert ix.docs[0]['title'] == 'Mahabharata online'
assert ix.docs[0]['kind'] == 'sanskrit' and ix.docs[0]['pages'] > 1

hits = ix.db.by_meter(meter=Meter.ANUSTUBH)
assert hits and json.loads(hits[0]['metadata'])['meter'] == Meter.ANUSTUBH.value

nlp = vidyut_pipe()
f = lemma_facets('gacchati rāmaḥ', nlp)
assert 'gam' in f['lemma']