Small helpers for Devanagari text: remove Vedic accents, give any spelling of a word one ASCII key, and read citation ids
Every other module in ganapati builds on the constants and functions here. They deal with three things that come up whenever you index Sanskrit: the Vedic accent marks that a search should ignore, the many ways one word can be spelled (Devanagari, IAST, plain ASCII), and the citations that label verses in an edition, such as MBh_1,1.1. None of them needs litesearch.
A few constants are used throughout. DANDA (।) and DDANDA (॥) are the single and double daṇḍa, the marks that close a half-verse and a verse. VIRAMA is the sign that removes a consonant’s built-in a. DEVANAGARI is a regular expression that matches any Devanagari character, including the Vedic extension blocks, and VEDIC_MARKS matches the accent marks alone.
/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
def roman_fold( s:str, doubles:bool=False, # make doubled letters single, so Kārttikeya and Kartikeya give one key endings:bool=False, # cut an English plural or past-tense ending when three letters remain, so duties and duty give one key)->str:
Fold like fold_token, then also merge informal English spellings (sh to s, th to t, ee to i, oo to u, w to v), so Bheeshma and Bhīṣma both give bisma.
Remove Vedic tone marks from Devanagari text, leaving Latin diacritics alone, and return it in NFC form.
strip_vedic removes the udātta, anudātta and other tone marks from Devanagari and keeps the letters, so अ॒ग्निमी॑ळे becomes अग्निमीळे. Latin diacritics such as the macron in ā are left alone.
fold_token gives one token its search key: lowercase ASCII with no diacritics and no vowel length. Devanagari goes through deva2ascii, which writes out each consonant’s implicit a, and Latin text just loses its accents, so भीष्म and Bhīṣma both fold to bhisma. A search index applies the same key to the stored text and to the query, which is how a reader who types bhisma finds भीष्म. The function is cached, since a corpus repeats a small vocabulary many times. detect_script says whether a string is mostly Devanagari, Latin or neither.
roman_fold goes further, for names and words typed in the informal spellings of Indian English. After fold_token it drops the h of an aspirate (sh, th and bh become s, t and b), reads ee and oo as i and u, makes a vowel written twice single, reads ri before a consonant as the vowel ṛ, and treats w as v. Bheeshma and Bhīṣma both become bisma. Two options loosen it more: doubles=True makes doubled letters single, and endings=True cuts an English plural or past-tense ending. Use fold_token for text in a proper transliteration and roman_fold for text typed by someone writing English. The cell below checks each of these.
A citation is how GRETIL and similar editions label a verse. It is an abbreviation for the work, called a siglum, followed by the verse’s place in it: || MBh_1,1.1 || is the Mahābhārata, book 1, chapter 1, verse 1. CITE_RE finds a citation between || or // markers in running text, and cite_parts splits one into the siglum and its list of numbers. A number for another edition in square brackets, as in IsUp_4[57M], is dropped.