text

Find where verses end, lay a verse out by pādas, chunk a text for search, and read five source formats

Where a verse ends

Sanskrit sources rarely put one verse on one line. A GRETIL file runs verses together and closes each with a citation, a recitation text glues the verse number onto the last line, and a plain edition may have nothing but daṇḍas. verse_spans cuts a text into verses and returns each one as (start, end, text, citation), where start and end are character offsets into the input and citation is the verse’s id, or None.

It looks for one kind of marker at a time, and uses the first kind that occurs anywhere in the text:

  1. a citation such as // Mn_1.1 //, which becomes the verse’s id;
  2. a Devanagari verse number such as ॥ १॥, which becomes the id;
  3. a double daṇḍa, || or // at the end of a line, which gives no id.

A text with none of these but with numbers at the start of its lines is cut at each number, and the number becomes the id. In the other cases a markdown heading also starts a new verse. A line that begins with > holds a gloss or a translation, and it stays with the verse before it.

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)

source

verse_spans

def verse_spans(
    text:str
)->fastcore.foundation.L:

Split text into verses, returning (start, end, text, citation) for each.

The examples below split two verses of the Īśa Upaniṣad by their citations, and a numbered text with no daṇḍas by its line numbers. They also check cite_parts from ganapati.sanskrit.

from ganapati.text import cite_parts, verse_spans
assert cite_parts('BrhUp_1,1.2') == ('BrhUp', ['1','1','2'])
assert cite_parts('Mn_1.1') == ('Mn', ['1','1'])
assert cite_parts('IsUp_4[57M]') == ('IsUp', ['4'])      # the alternate-edition number is dropped
assert cite_parts('not a citation') == ('not a citation', [])

two = ('īśā vāsyam idaṃ sarvaṃ | tena tyaktena bhuñjīthāḥ || IsUp_1 ||\n'
       'kurvann eveha karmāṇi | evaṃ tvayi nānyatheto || IsUp_2 ||\n')
sp = verse_spans(two)
assert len(sp) == 2
assert [c for *_, c in sp] == ['IsUp_1', 'IsUp_2']
assert sp[0][2].startswith('īśā') and sp[0][2].endswith('||')

numbered = ('0nārāyaṇaṃ namaskṛtya\n'
            '  1\n'
            'lomaharṣaṇaputra ugraśravāḥ\n'
            '2samāsīnān abhyagacchat')
sp = verse_spans(numbered)
assert len(sp) == 3
assert [s for *_, s, _ in sp] == [
    'nārāyaṇaṃ namaskṛtya', 'lomaharṣaṇaputra ugraśravāḥ', 'samāsīnān abhyagacchat']

Laying out a verse

A pāda is a quarter of a verse. A printed verse usually shows two pādas on each line, a single daṇḍa (।) after the first half, and a double daṇḍa with the verse number after the second: ॥ १ ॥. Recitation sources often send the whole verse as one line, with a speaker tag such as धृतराष्ट्र उवाच (“Dhṛtarāṣṭra said”) glued to the front and the number in ASCII digits. These functions put such a verse back into the printed shape.

padas cuts one verse at each single daṇḍa, so each output line is a half-verse in the usual case. It puts a speaker tag on a line of its own and closes the last line with ॥ n ॥ in Devanagari digits. Pass num to set the number, or leave it out to keep the number the verse already carries. When you have a whole saṃhitā or chapter in one string, call split_mantras first: it cuts the text at each ॥ n ॥ and returns (number, body) pairs that you can pass to padas one at a time. A single daṇḍa before a number does not count as a marker, because the paryāya hymns of the Atharvaveda print a running count that way. deva_num writes a number in Devanagari digits, and is_word tells a word from a daṇḍa, a verse number or a stray punctuation mark.

The examples below use Gītā 1.1 as the DharmicData corpus prints it, a verse of the Kumārasambhava, the opening of the Yajurveda, and edge cases from the Atharvaveda, the Chāndogya Upaniṣad and the Gītā.


source

padas

def padas(
    verse:str, # one verse, on one line or several, with or without its end-marker
    num:NoneType=None, # the verse number to close it with; None keeps the one the verse carries
)->fastcore.foundation.L:

Lay out one verse as lines cut at its daṇḍas, with a glued speaker tag (… उवाच) on a line of its own and the last line closed by ॥ num ॥.


source

split_mantras

def split_mantras(
    text:str
)->fastcore.foundation.L:

Cut a text at its ॥ n ॥ verse markers and return (number, body) pairs, with each number as written.


source

deva_num

def deva_num(
    n
)->str:

Write a verse number in Devanagari digits: 18 gives १८ and 1.1 gives १.१.


source

is_word

def is_word(
    t:str
)->bool:

Return True if a token contains a letter, so a daṇḍa, a verse number or a lone punctuation mark is not a word.

from ganapati.text import is_word, deva_num, split_mantras, padas
assert is_word('धर्मक्षेत्रे') and is_word('kṣetre') and is_word('yuyutsavaḥ,')
assert not is_word('॥') and not is_word('।') and not is_word('१८') and not is_word('1.1') and not is_word('')
assert deva_num(18) == '१८' and deva_num('1.1') == '१.१'

# Gītā 1.1 as DharmicData prints it: one line, the speaker glued on, the chapter.verse marker at the end
g11 = 'धृतराष्ट्र उवाच धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवः । मामकाः पाण्डवाश्चैव किमकुर्वत सञ्जय ।। 1.1 ।।'
assert padas(g11, 1) == ['धृतराष्ट्र उवाच', 'धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवः ।', 'मामकाः पाण्डवाश्चैव किमकुर्वत सञ्जय ॥ १ ॥']
assert padas(g11)[-1].endswith('॥ १.१ ॥')             # no number given: the one the verse carries, in Devanagari
assert padas('मामकाः पाण्डवाश्चैव किमकुर्वत सञ्जय')[-1].endswith(' ॥')   # no number anywhere: a bare close

# Kumārasambhava 4.1 across two lines with its own daṇḍas: the lines are only re-cut
K41 = 'अथ मोहपरायणा सती विवशा कामवधूर्विबोधिता ।\nविधिना प्रतिपादयिष्यता नववैधव्यमसह्यवेदनम् ॥'
assert padas(K41, 1) == ['अथ मोहपरायणा सती विवशा कामवधूर्विबोधिता ।', 'विधिना प्रतिपादयिष्यता नववैधव्यमसह्यवेदनम् ॥ १ ॥']

# a saṃhitā blob: two mantras, numbered as written, each then split into its pādas
blob = 'इषे त्वोर्जे त्वा वायव स्थ देवो वः सविता प्रार्पयतु ॥ १ ॥ वसोः पवित्रमसि शतधारं । वसोः पवित्रमसि सहस्रधारम् ॥ २ ॥'
ms = split_mantras(blob)
assert [n for n, _ in ms] == ['१', '२'] and ms[0][1].startswith('इषे')
assert padas(ms[1][1], ms[1][0]) == ['वसोः पवित्रमसि शतधारं ।', 'वसोः पवित्रमसि सहस्रधारम् ॥ २ ॥']
assert split_mantras('') == [] and padas('') == []

# DharmicData prints AV 2.36.7's close as `॥७` with no closing daṇḍa and a newline: still a verse end, or 7 merges into 8 and `७` is left glued mid-line
av = 'अन्यं वेत्तवे ॥७\nआ ते नयतु सविता ॥ ८ ॥'
assert split_mantras(av) == [('७', 'अन्यं वेत्तवे'), ('८', 'आ ते नयतु सविता')]
assert padas('अन्यं वेत्तवे ॥७') == ['अन्यं वेत्तवे ॥ ७ ॥']
assert split_mantras('क । १० ख ॥ ११ ॥') == [('११', 'क । १० ख')]                 # a number after a daṇḍa mid-line is text, not a marker

# a paryāya sūkta (AV 16.8) writes a running count after the marker, `॥४॥ ४। १`: a single daṇḍa before a bare number is not a marker
py = 'पादयामि॥४॥ ४। १\nजितमस्माकम्॥१॥'
assert split_mantras(py) == [('४', 'पादयामि'), ('१', '४। १\nजितमस्माकम्')]

# the speaker tag is cut only where `वाच` ends the word: Chāndogya 4.2.3's `प्रत्युवाचाह` goes on, and so do `वाचं`, `वाचः`, `वाचा`
ch = 'तमु ह परः प्रत्युवाचाह हारेत्वा शूद्र तवैव सह गोभिरस्त्विति ।'
assert padas(ch) == ['तमु ह परः प्रत्युवाचाह हारेत्वा शूद्र तवैव सह गोभिरस्त्विति ॥']
assert padas('श्रीभगवानुवाच कुतस्त्वा कश्मलमिदं विषमे समुपस्थितम् ।')[:2] == ['श्रीभगवानुवाच', 'कुतस्त्वा कश्मलमिदं विषमे समुपस्थितम् ॥']
assert padas(g11)[0] == 'धृतराष्ट्र उवाच' and padas('अर्जुन उवाच ।') == ['अर्जुन उवाच']
for w in ('गुरुवाचं श्रुत्वा', 'तदुवाचः स आह', 'मधुवाचा वदति', 'अनुवाचम् इति'): assert padas(w) == [f'{w} ॥'], w

# DharmicData glues 11 Gītā speaker tags to the pāda with no space (12.1, 12.2, 13.2, 14.1, 14.21, 14.22, 15.1, 17.1, 17.2, 18.73, 18.74): a letter after `वाच` starts the pāda
glued = [
    ('अर्जुन उवाचएवं सततयुक्ता ये भक्तास्त्वां पर्युपासते।येचाप्यक्षरमव्यक्तं तेषां के योगवित्तमाः।।12.1।।', 1, ['अर्जुन उवाच', 'एवं सततयुक्ता ये भक्तास्त्वां पर्युपासते ।', 'येचाप्यक्षरमव्यक्तं तेषां के योगवित्तमाः ॥ १ ॥']),
    ('श्री भगवानुवाचमय्यावेश्य मनो ये मां नित्ययुक्ता उपासते।श्रद्धया परयोपेतास्ते मे युक्ततमा मताः।।12.2।।', 2, ['श्री भगवानुवाच', 'मय्यावेश्य मनो ये मां नित्ययुक्ता उपासते ।', 'श्रद्धया परयोपेतास्ते मे युक्ततमा मताः ॥ २ ॥']),
    ('श्री भगवानुवाचइदं शरीरं कौन्तेय क्षेत्रमित्यभिधीयते।एतद्यो वेत्ति तं प्राहुः क्षेत्रज्ञ इति तद्विदः।।13.2।।', 2, ['श्री भगवानुवाच', 'इदं शरीरं कौन्तेय क्षेत्रमित्यभिधीयते ।', 'एतद्यो वेत्ति तं प्राहुः क्षेत्रज्ञ इति तद्विदः ॥ २ ॥']),
    ('श्री भगवानुवाचपरं भूयः प्रवक्ष्यामि ज्ञानानां ज्ञानमुत्तमम्।यज्ज्ञात्वा मुनयः सर्वे परां सिद्धिमितो गताः।।14.1।।', 1, ['श्री भगवानुवाच', 'परं भूयः प्रवक्ष्यामि ज्ञानानां ज्ञानमुत्तमम् ।', 'यज्ज्ञात्वा मुनयः सर्वे परां सिद्धिमितो गताः ॥ १ ॥']),
    ('अर्जुन उवाचकैर्लिंगैस्त्रीन्गुणानेतानतीतो भवति प्रभो।किमाचारः कथं चैतांस्त्रीन्गुणानतिवर्तते।।14.21।।', 21, ['अर्जुन उवाच', 'कैर्लिंगैस्त्रीन्गुणानेतानतीतो भवति प्रभो ।', 'किमाचारः कथं चैतांस्त्रीन्गुणानतिवर्तते ॥ २१ ॥']),
    ('श्री भगवानुवाचप्रकाशं च प्रवृत्तिं च मोहमेव च पाण्डव।न द्वेष्टि सम्प्रवृत्तानि न निवृत्तानि काङ्क्षति।।14.22।।', 22, ['श्री भगवानुवाच', 'प्रकाशं च प्रवृत्तिं च मोहमेव च पाण्डव ।', 'न द्वेष्टि सम्प्रवृत्तानि न निवृत्तानि काङ्क्षति ॥ २२ ॥']),
    ('श्री भगवानुवाचऊर्ध्वमूलमधःशाखमश्वत्थं प्राहुरव्ययम्।छन्दांसि यस्य पर्णानि यस्तं वेद स वेदवित्।।15.1।।', 1, ['श्री भगवानुवाच', 'ऊर्ध्वमूलमधःशाखमश्वत्थं प्राहुरव्ययम् ।', 'छन्दांसि यस्य पर्णानि यस्तं वेद स वेदवित् ॥ १ ॥']),
    ('अर्जुन उवाचये शास्त्रविधिमुत्सृज्य यजन्ते श्रद्धयाऽन्विताः।तेषां निष्ठा तु का कृष्ण सत्त्वमाहो रजस्तमः।।17.1।।', 1, ['अर्जुन उवाच', 'ये शास्त्रविधिमुत्सृज्य यजन्ते श्रद्धयाऽन्विताः ।', 'तेषां निष्ठा तु का कृष्ण सत्त्वमाहो रजस्तमः ॥ १ ॥']),
    ('श्री भगवानुवाचत्रिविधा भवति श्रद्धा देहिनां सा स्वभावजा।सात्त्विकी राजसी चैव तामसी चेति तां श्रृणु।।17.2।।', 2, ['श्री भगवानुवाच', 'त्रिविधा भवति श्रद्धा देहिनां सा स्वभावजा ।', 'सात्त्विकी राजसी चैव तामसी चेति तां श्रृणु ॥ २ ॥']),
    ('अर्जुन उवाचनष्टो मोहः स्मृतिर्लब्धा त्वत्प्रसादान्मयाच्युत।स्थितोऽस्मि गतसन्देहः करिष्ये वचनं तव।।18.73।।', 73, ['अर्जुन उवाच', 'नष्टो मोहः स्मृतिर्लब्धा त्वत्प्रसादान्मयाच्युत ।', 'स्थितोऽस्मि गतसन्देहः करिष्ये वचनं तव ॥ ७३ ॥']),
    ('सञ्जय उवाचइत्यहं वासुदेवस्य पार्थस्य च महात्मनः।संवादमिममश्रौषमद्भुतं रोमहर्षणम्।।18.74।।', 74, ['सञ्जय उवाच', 'इत्यहं वासुदेवस्य पार्थस्य च महात्मनः ।', 'संवादमिममश्रौषमद्भुतं रोमहर्षणम् ॥ ७४ ॥']),
]
for t, n, exp in glued: assert padas(t, n) == exp, (n, padas(t, n))

# no piece opens on a dependent vowel sign, virama, anusvāra or visarga, over verses as vedicreader ships them: Gītā 1.2, 2.1, 2.2, 2.4; Kaṭha 1.1.1, 1.1.4; Kena 1.1, 1.2, 1.5; Chāndogya 4.1.3, 4.1.5, 4.2.3-5
DEP = re.compile('^[\u0900-\u0903\u093a-\u094f\u0951-\u0957\u0962\u0963]')
real = [g11, K41, blob,
        'सञ्जय उवाच दृष्ट्वा तु पाण्डवानीकं व्यूढं दुर्योधनस्तदा । आचार्यमुपसङ्गम्य राजा वचनमब्रवीत् ॥ २ ॥',
        'सञ्जय उवाच तं तथा कृपयाऽविष्टमश्रुपूर्णाकुलेक्षणम् । विषीदन्तमिदं वाक्यमुवाच मधुसूदनः ॥ १ ॥',
        'श्री भगवानुवाच कुतस्त्वा कश्मलमिदं विषमे समुपस्थितम् । अनार्यजुष्टमस्वर्ग्यमकीर्तिकरमर्जुन ॥ २ ॥',
        'अर्जुन उवाच कथं भीष्ममहं संख्ये द्रोणं च मधुसूदन । इषुभिः प्रतियोत्स्यामि पूजार्हावरिसूदन ॥ ४ ॥',
        'ॐ उशन् ह वै वाजश्रवसः सर्ववेदसं ददौ । तस्य ह नचिकेता नाम पुत्र आस ॥ १ ॥',
        'स होवाच पितरं तत कस्मै मां दास्यसीति । द्वितीयं तृतीयं तँ होवाच मृत्यवे त्वा ददामीति ॥ ४ ॥',
        'ॐ केनेषितं पतति प्रेषितं मनः केन प्राणः प्रथमः प्रैति युक्तः । केनेषितां वाचमिमां वदन्ति चक्षुः श्रोत्रं क उ देवो युनक्ति ॥ १ ॥',
        'श्रोत्रस्य श्रोत्रं मनसो मनो यद् वाचो ह वाचं स उ प्राणस्य प्राणः । चक्षुषश्चक्षुरतिमुच्य धीराः प्रेत्यास्माल्लोकादमृता भवन्ति ॥ २ ॥',
        'यद्वाचाऽनभ्युदितं येन वागभ्युद्यते । तदेव ब्रह्म त्वं विद्धि नेदं यदिदमुपासते ॥ ५ ॥',
        'तमु ह परः प्रत्युवाच कं वर एनमेतत् सन्तं सयुग्वानमिव रैक्वमात्थ इति । यो नु कथं सयुग्वा रैक्व इति ॥ ३ ॥',
        'तदु ह जानश्रुतिः पौत्रायण उपशुश्राव । स ह संजिहान एव क्षत्तारमुवाचाङ्गारे ह सयुग्वानमिव रैक्वमात्थेति । यो नु कथं सयुग्वा रैक्व इति ॥ ५ ॥',
        'तमु ह परः प्रत्युवाचाह हारेत्वा शूद्र तवैव सह गोभिरस्त्विति । तदु ह पुनरेव जानश्रुतिः पौत्रायणः सहस्रं गवां निष्कमश्वतरीरथं दुहितरं तदादाय प्रतिचक्रमे ॥ ३ ॥',
        'तं हाभ्युवाद । रैक्वेदं सहस्रं गवामयं निष्कोऽयमश्वतरीरथ इयं जायायं ग्रामो यस्मिन्नास्सेऽन्वेव मा भगवः शाधीति ॥ ४ ॥',
        'तस्या ह मुखमुपोद्गृह्णन्नुवाचाजहारेमाः शूद्रानेनैव मुखेनालापयिष्यथा इति । ते हैते रैक्वपर्णा नाम महावृषेषु यत्रास्मा उवास । तस्मै होवाच ॥ ५ ॥']
assert all(not DEP.match(p) for v in real for p in padas(v)), [p for v in real for p in padas(v) if DEP.match(p)]

Chunkers

A search index stores a text as chunks, and the chunk boundaries decide what a search result looks like: a chunk that ends mid-verse returns half a verse. chunk_verses cuts on the verse boundaries that verse_spans finds. It joins short verses until a chunk reaches target characters, and splits a verse longer than max_chars at its daṇḍas, never inside a clause. A heading always starts a new chunk. A chunk normally ends as soon as it holds a verse with a citation, so that each cited verse is its own result; pass pack_cited=True to let cited verses share a chunk up to target.

VerseChunker and ProseChunker wrap chunk_verses in the interface of a chonkie chunker, so you can pass either to litesearch’s add_doc(chunker=...). Each chunk carries start_index and end_index, its offsets in the source. VerseChunker aims for 300 characters and allows 900. ProseChunker is for prose such as commentaries, bhāṣyas and the prose Upaniṣads: it aims for 700, allows 1400, and packs cited units together. litesearch reads the chunker’s settings too, and packs text across pages only for a chunker that packs cited units, which the last two checks below confirm.


source

ProseChunker

def ProseChunker(
    target:int=700, max_chars:int=1400, pack_cited:bool=True
):

A VerseChunker with larger chunks that packs cited units together, for commentaries, bhāṣyas and the prose Upaniṣads.


source

VerseChunker

def VerseChunker(
    target:int=300, max_chars:int=900, pack_cited:bool=False
):

A chunker with the chonkie interface that cuts on verse boundaries, for short verses.


source

chunk_verses

def chunk_verses(
    text:str, # source text
    target:int=300, # pack short units forward up to this many chars
    max_chars:int=900, # split a unit longer than this at clause boundaries
    pack_cited:bool=False, # let consecutive cited units share a chunk up to `target`
)->fastcore.foundation.L:

Cut a text into chunks on verse boundaries, joining short verses up to target and splitting long ones at their daṇḍas.

from litesearch.tree import pack_target
cs = VerseChunker().chunk(two)
assert len(cs) == 2
assert two[cs[0].start_index:cs[0].end_index] == cs[0].text   # offsets point back at the source

# ProseChunker packs cited units instead, which is what commentary wants
assert len(ProseChunker().chunk(two)) == 1

# with no citation to cut on, the budget decides. 3,040 characters of daṇḍa-separated prose:
long = ' | '.join(['padam yatra tatra sarvatra gacchati'] * 80) + ' ||'
assert len(chunk_verses(long, 300, 900)) == 4
assert len(chunk_verses(long, 700, 1400)) == 3

# litesearch packs across segments only for a chunker that asks; the verse profile does not,
# because a reader that emits one verse per page is already the right size
assert pack_target(VerseChunker()) == 0
assert pack_target(ProseChunker()) == 700

Reading source files

Sanskrit texts are published in several file formats, and ganapati reads five. Each reader takes a path or the file’s contents as a string and returns (pages, meta). pages is a list of (index, text) pairs, usually one per verse or section, and meta is a dict with the format, the title and whatever else the file states. When you don’t know the format, call sanskrit_parse: it looks at the extension and the first 4,000 characters and calls the right reader.

reader source
gretil_parse GRETIL plain-text editions, the *_u.htm files
tei_parse TEI XML, as GRETIL and SARIT publish it
vr_xml_parse vedicreader <lyrics> XML, which carries audio timings and a word-by-word analysis
vr_json_parse vedicreader content JSON, which carries the same information
dcs_parse CoNLL-U files from the Digital Corpus of Sanskrit (DCS), which give each word’s lemma

is_sanskrit answers a simpler question: whether a piece of text looks like Sanskrit at all, judged by its script, its IAST diacritics, or its citations and daṇḍas.

The vedicreader formats need a few more words. A line marked ignore="true" is printed but not recited, such as a title or a rubric, so the reader leaves it out of the text; the first line of the title section becomes meta['title']. A line aligned to a recording carries start_time_ms, end_time_ms and alignment_id. time_marker writes these into the text as [t 0-1800 a1] so that they survive chunking, and line_times reads them back from a chunk as AttrDict(text, start, end, aid, ms). A line’s translation is written on a following line that starts with >, and its word-by-word analysis on lines that start with > etym:, which line_etyms returns. The metre code ignores > lines and anything in square brackets, so neither a gloss nor a timing is scanned as verse.


source

sanskrit_parse

def sanskrit_parse(
    src
)->tuple:

Read any supported Sanskrit source into (pages, meta), choosing the reader from the extension and the first 4,000 characters.


source

is_sanskrit

def is_sanskrit(
    text:str, thresh:float=0.15
)->bool:

Guess whether text is Sanskrit from its script, its IAST diacritics, or its citations and daṇḍas.


source

dcs_parse

def dcs_parse(
    src
)->tuple:

Read a DCS CoNLL-U file into (pages, meta), one page per sentence with its lemmas on a > lemmas: line.


source

vr_json_parse

def vr_json_parse(
    src
)->tuple:

Read vedicreader content JSON into (pages, meta), in the same shape as vr_xml_parse.


source

vr_xml_parse

def vr_xml_parse(
    src
)->tuple:

Read vedicreader <lyrics> XML into (pages, meta), keeping audio timings and the word-by-word analysis.


source

line_etyms

def line_etyms(
    text:str
)->fastcore.foundation.L:

Return the word-by-word analysis a source supplies on its > etym: lines, without the prefix.


source

line_times

def line_times(
    text:str
)->fastcore.foundation.L:

Return every timed line in a chunk as AttrDict(text, start, end, aid, ms), in order.


source

time_marker

def time_marker(
    start:int, # line start in ms
    end:int, # line end in ms
    aid:str=None, # the aligner's id for the span
)->str:

Write one line’s audio span as a text marker such as [t 0-1800 a1], so the timing survives chunking.


source

tei_parse

def tei_parse(
    src
)->tuple:

Read a TEI XML edition from GRETIL or SARIT into (pages, meta), with one page per division heading, verse group or paragraph.


source

gretil_parse

def gretil_parse(
    src
)->tuple:

Read a GRETIL plain-text edition (*_u.htm) into (pages, meta), keeping its header in meta.

The first example below parses a short vedicreader XML file with a title line, a section translation, and two lines aligned to audio, one with its word-by-word analysis. The second gives the same content as JSON. The third shows is_sanskrit on IAST, Devanagari and English, and the last reads GRETIL’s edition of Mahābhārata 1.1 from mahabharata.htm.

VR = """<lyrics>
  <category>vedic</category><tags>rudram</tags>
  <section name="title">
    <line ignore="true">॥ श्रीरुद्रचमकप्रश्नः ॥</line>
  </section>
  <section name="anuvaka 1" meaning="A dual invocation of Agni and Viṣṇu.">
    <line start_time_ms="0" end_time_ms="1800" alignment_id="a1" caption="of one accord"
          etymology="अग्नाविष्णू, agni-viṣṇu, vocative case, dual number, O Agni and Viṣṇu.&#10;सजोषसा, sajoṣas, instrumental case, singular number, of one accord.">अग्नाविष्णू सजोषसेमा<w start_time_ms="0" end_time_ms="900">अग्नाविष्णू</w><w start_time_ms="900" end_time_ms="1800">सजोषसेमा</w></line>
    <line start_time_ms="1800" end_time_ms="3400" alignment_id="a2">वर्धन्तु वां गिरः ॥</line>
  </section>
</lyrics>"""

pages, meta = vr_xml_parse(VR)
assert meta['title'] == '॥ श्रीरुद्रचमकप्रश्नः ॥'        # kept as the title
assert not any('श्रीरुद्रचमकप्रश्नः' in t for _, t in pages)   # and dropped from the content

ts = line_times(pages[-1][1])
assert [t.start for t in ts] == [0, 1800] and [t.ms for t in ts] == [1800, 1600]
assert ts[0].aid == 'a1' and ts[0].text == 'अग्नाविष्णू सजोषसेमा'   # `<w>` children stay out of the line
# the corpus writes a whole word-by-word analysis into one attribute, so every physical line needs the
# `> etym:` prefix: an unprefixed continuation line reaches the scansion as if it were a pāda
es = line_etyms(pages[-1][1])
assert len(es) == 2 and es[0].startswith('अग्नाविष्णू,') and es[1].startswith('सजोषसा,')
quoted = lambda pg: all(ln.startswith(('>', '##')) or '[t ' in ln for ln in pg.splitlines() if ln.strip())
assert quoted(pages[-1][1])
VRJ = """{"title": "॥ श्रीरुद्रचमकप्रश्नः ॥", "category": "vedic", "tags": "rudram",
 "sections": [{"name": "anuvaka 1", "meaning": "A dual invocation of Agni and Viṣṇu.",
   "lines": [{"t": "अग्नाविष्णू सजोषसेमा", "s": 0, "e": 1800, "cap": "of one accord",
              "etym": "अग्नाविष्णू, agni-viṣṇu, vocative case, dual number, O Agni and Viṣṇu.\\nसजोषसा, sajoṣas, instrumental case, singular number, of one accord.",
              "w": [["अग्नाविष्णू", 0, 900], ["सजोषसेमा", 900, 1800]]},
             {"t": "वर्धन्तु वां गिरः ॥", "s": 1800, "e": 3400},
             {"t": "[bell]", "role": "audio", "s": 3400, "e": 4000}]}]}"""

pj, mj = vr_json_parse(VRJ)
assert mj['title'] == meta['title'] and mj['category'] == 'vedic'
assert [t.ms for t in line_times(pj[-1][1])] == [1800, 1600]   # `audio` is recited but not printed, so not verse
assert len(line_etyms(pj[-1][1])) == 2                         # one entry per physical line, same as the XML reader
assert quoted(pj[-1][1])
assert sanskrit_parse(VRJ)[1]['fmt'] == 'vedicreader'          # picked by shape, not by extension
assert is_sanskrit('śrīmātā śrīmahārājñī || LSN_1 ||\nśrīmatsiṃhāsaneśvarī || LSN_2 ||\n'
                   'cidagnikuṇḍasambhūtā || LSN_3 ||')
assert is_sanskrit('अथ योगानुशासनम् ॥ १॥')
assert not is_sanskrit('This is an ordinary English paragraph about nothing in particular.')
assert not is_sanskrit('')
assert _detag('dharma&nbsp;artha') == 'dharma artha'
sanskrit_parse('mahabharata.htm')[0][0]
(0,
 'nārāyaṇaṃ namaskṛtya naraṃ caiva narottamam\ndevīṃ sarasvatīṃ caiva tato jayam udīrayet')