# text


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## Where a verse ends

Sanskrit sources rarely put one verse on one line. A GRETIL file runs
verses together and closes each with a citation, a recitation text glues
the verse number onto the last line, and a plain edition may have
nothing but daṇḍas.
[`verse_spans`](https://vedicreader.github.io/ganapati/text.html#verse_spans)
cuts a text into verses and returns each one as
`(start, end, text, citation)`, where `start` and `end` are character
offsets into the input and `citation` is the verse’s id, or None.

It looks for one kind of marker at a time, and uses the first kind that
occurs anywhere in the text:

1.  a citation such as `// Mn_1.1 //`, which becomes the verse’s id;
2.  a Devanagari verse number such as `॥ १॥`, which becomes the id;
3.  a double daṇḍa, `||` or `//` at the end of a line, which gives no
    id.

A text with none of these but with numbers at the start of its lines is
cut at each number, and the number becomes the id. In the other cases a
markdown heading also starts a new verse. A line that begins with `>`
holds a gloss or a translation, and it stays with the verse before it.

    /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
      warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L26"
target="_blank" style="float:right; font-size:smaller">source</a>

### verse_spans

``` python
def verse_spans(
    text:str
)->fastcore.foundation.L:
```

*Split `text` into verses, returning `(start, end, text, citation)` for
each.*

The examples below split two verses of the Īśa Upaniṣad by their
citations, and a numbered text with no daṇḍas by its line numbers. They
also check
[`cite_parts`](https://vedicreader.github.io/ganapati/sanskrit.html#cite_parts)
from `ganapati.sanskrit`.

``` python
from ganapati.text import cite_parts, verse_spans
```

``` python
assert cite_parts('BrhUp_1,1.2') == ('BrhUp', ['1','1','2'])
assert cite_parts('Mn_1.1') == ('Mn', ['1','1'])
assert cite_parts('IsUp_4[57M]') == ('IsUp', ['4'])      # the alternate-edition number is dropped
assert cite_parts('not a citation') == ('not a citation', [])

two = ('īśā vāsyam idaṃ sarvaṃ | tena tyaktena bhuñjīthāḥ || IsUp_1 ||\n'
       'kurvann eveha karmāṇi | evaṃ tvayi nānyatheto || IsUp_2 ||\n')
sp = verse_spans(two)
assert len(sp) == 2
assert [c for *_, c in sp] == ['IsUp_1', 'IsUp_2']
assert sp[0][2].startswith('īśā') and sp[0][2].endswith('||')

numbered = ('0nārāyaṇaṃ namaskṛtya\n'
            '  1\n'
            'lomaharṣaṇaputra ugraśravāḥ\n'
            '2samāsīnān abhyagacchat')
sp = verse_spans(numbered)
assert len(sp) == 3
assert [s for *_, s, _ in sp] == [
    'nārāyaṇaṃ namaskṛtya', 'lomaharṣaṇaputra ugraśravāḥ', 'samāsīnān abhyagacchat']
```

## Laying out a verse

A pāda is a quarter of a verse. A printed verse usually shows two pādas
on each line, a single daṇḍa (।) after the first half, and a double
daṇḍa with the verse number after the second: `॥ १ ॥`. Recitation
sources often send the whole verse as one line, with a speaker tag such
as धृतराष्ट्र उवाच (“Dhṛtarāṣṭra said”) glued to the front and the number
in ASCII digits. These functions put such a verse back into the printed
shape.

[`padas`](https://vedicreader.github.io/ganapati/text.html#padas) cuts
one verse at each single daṇḍa, so each output line is a half-verse in
the usual case. It puts a speaker tag on a line of its own and closes
the last line with `॥ n ॥` in Devanagari digits. Pass `num` to set the
number, or leave it out to keep the number the verse already carries.
When you have a whole saṃhitā or chapter in one string, call
[`split_mantras`](https://vedicreader.github.io/ganapati/text.html#split_mantras)
first: it cuts the text at each `॥ n ॥` and returns `(number, body)`
pairs that you can pass to
[`padas`](https://vedicreader.github.io/ganapati/text.html#padas) one at
a time. A single daṇḍa before a number does not count as a marker,
because the paryāya hymns of the Atharvaveda print a running count that
way.
[`deva_num`](https://vedicreader.github.io/ganapati/text.html#deva_num)
writes a number in Devanagari digits, and
[`is_word`](https://vedicreader.github.io/ganapati/text.html#is_word)
tells a word from a daṇḍa, a verse number or a stray punctuation mark.

The examples below use Gītā 1.1 as the DharmicData corpus prints it, a
verse of the Kumārasambhava, the opening of the Yajurveda, and edge
cases from the Atharvaveda, the Chāndogya Upaniṣad and the Gītā.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L98"
target="_blank" style="float:right; font-size:smaller">source</a>

### padas

``` python
def padas(
    verse:str, # one verse, on one line or several, with or without its end-marker
    num:NoneType=None, # the verse number to close it with; None keeps the one the verse carries
)->fastcore.foundation.L:
```

*Lay out one verse as lines cut at its daṇḍas, with a glued speaker tag
(`… उवाच`) on a line of its own and the last line closed by `॥ num ॥`.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L90"
target="_blank" style="float:right; font-size:smaller">source</a>

### split_mantras

``` python
def split_mantras(
    text:str
)->fastcore.foundation.L:
```

*Cut a text at its `॥ n ॥` verse markers and return `(number, body)`
pairs, with each number as written.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L86"
target="_blank" style="float:right; font-size:smaller">source</a>

### deva_num

``` python
def deva_num(
    n
)->str:
```

*Write a verse number in Devanagari digits: `18` gives `१८` and `1.1`
gives `१.१`.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L82"
target="_blank" style="float:right; font-size:smaller">source</a>

### is_word

``` python
def is_word(
    t:str
)->bool:
```

*Return True if a token contains a letter, so a daṇḍa, a verse number or
a lone punctuation mark is not a word.*

``` python
from ganapati.text import is_word, deva_num, split_mantras, padas
```

``` python
assert is_word('धर्मक्षेत्रे') and is_word('kṣetre') and is_word('yuyutsavaḥ,')
assert not is_word('॥') and not is_word('।') and not is_word('१८') and not is_word('1.1') and not is_word('')
assert deva_num(18) == '१८' and deva_num('1.1') == '१.१'

# Gītā 1.1 as DharmicData prints it: one line, the speaker glued on, the chapter.verse marker at the end
g11 = 'धृतराष्ट्र उवाच धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवः । मामकाः पाण्डवाश्चैव किमकुर्वत सञ्जय ।। 1.1 ।।'
assert padas(g11, 1) == ['धृतराष्ट्र उवाच', 'धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवः ।', 'मामकाः पाण्डवाश्चैव किमकुर्वत सञ्जय ॥ १ ॥']
assert padas(g11)[-1].endswith('॥ १.१ ॥')             # no number given: the one the verse carries, in Devanagari
assert padas('मामकाः पाण्डवाश्चैव किमकुर्वत सञ्जय')[-1].endswith(' ॥')   # no number anywhere: a bare close

# Kumārasambhava 4.1 across two lines with its own daṇḍas: the lines are only re-cut
K41 = 'अथ मोहपरायणा सती विवशा कामवधूर्विबोधिता ।\nविधिना प्रतिपादयिष्यता नववैधव्यमसह्यवेदनम् ॥'
assert padas(K41, 1) == ['अथ मोहपरायणा सती विवशा कामवधूर्विबोधिता ।', 'विधिना प्रतिपादयिष्यता नववैधव्यमसह्यवेदनम् ॥ १ ॥']

# a saṃhitā blob: two mantras, numbered as written, each then split into its pādas
blob = 'इषे त्वोर्जे त्वा वायव स्थ देवो वः सविता प्रार्पयतु ॥ १ ॥ वसोः पवित्रमसि शतधारं । वसोः पवित्रमसि सहस्रधारम् ॥ २ ॥'
ms = split_mantras(blob)
assert [n for n, _ in ms] == ['१', '२'] and ms[0][1].startswith('इषे')
assert padas(ms[1][1], ms[1][0]) == ['वसोः पवित्रमसि शतधारं ।', 'वसोः पवित्रमसि सहस्रधारम् ॥ २ ॥']
assert split_mantras('') == [] and padas('') == []

# DharmicData prints AV 2.36.7's close as `॥७` with no closing daṇḍa and a newline: still a verse end, or 7 merges into 8 and `७` is left glued mid-line
av = 'अन्यं वेत्तवे ॥७\nआ ते नयतु सविता ॥ ८ ॥'
assert split_mantras(av) == [('७', 'अन्यं वेत्तवे'), ('८', 'आ ते नयतु सविता')]
assert padas('अन्यं वेत्तवे ॥७') == ['अन्यं वेत्तवे ॥ ७ ॥']
assert split_mantras('क । १० ख ॥ ११ ॥') == [('११', 'क । १० ख')]                 # a number after a daṇḍa mid-line is text, not a marker

# a paryāya sūkta (AV 16.8) writes a running count after the marker, `॥४॥ ४। १`: a single daṇḍa before a bare number is not a marker
py = 'पादयामि॥४॥ ४। १\nजितमस्माकम्॥१॥'
assert split_mantras(py) == [('४', 'पादयामि'), ('१', '४। १\nजितमस्माकम्')]

# the speaker tag is cut only where `वाच` ends the word: Chāndogya 4.2.3's `प्रत्युवाचाह` goes on, and so do `वाचं`, `वाचः`, `वाचा`
ch = 'तमु ह परः प्रत्युवाचाह हारेत्वा शूद्र तवैव सह गोभिरस्त्विति ।'
assert padas(ch) == ['तमु ह परः प्रत्युवाचाह हारेत्वा शूद्र तवैव सह गोभिरस्त्विति ॥']
assert padas('श्रीभगवानुवाच कुतस्त्वा कश्मलमिदं विषमे समुपस्थितम् ।')[:2] == ['श्रीभगवानुवाच', 'कुतस्त्वा कश्मलमिदं विषमे समुपस्थितम् ॥']
assert padas(g11)[0] == 'धृतराष्ट्र उवाच' and padas('अर्जुन उवाच ।') == ['अर्जुन उवाच']
for w in ('गुरुवाचं श्रुत्वा', 'तदुवाचः स आह', 'मधुवाचा वदति', 'अनुवाचम् इति'): assert padas(w) == [f'{w} ॥'], w

# DharmicData glues 11 Gītā speaker tags to the pāda with no space (12.1, 12.2, 13.2, 14.1, 14.21, 14.22, 15.1, 17.1, 17.2, 18.73, 18.74): a letter after `वाच` starts the pāda
glued = [
    ('अर्जुन उवाचएवं सततयुक्ता ये भक्तास्त्वां पर्युपासते।येचाप्यक्षरमव्यक्तं तेषां के योगवित्तमाः।।12.1।।', 1, ['अर्जुन उवाच', 'एवं सततयुक्ता ये भक्तास्त्वां पर्युपासते ।', 'येचाप्यक्षरमव्यक्तं तेषां के योगवित्तमाः ॥ १ ॥']),
    ('श्री भगवानुवाचमय्यावेश्य मनो ये मां नित्ययुक्ता उपासते।श्रद्धया परयोपेतास्ते मे युक्ततमा मताः।।12.2।।', 2, ['श्री भगवानुवाच', 'मय्यावेश्य मनो ये मां नित्ययुक्ता उपासते ।', 'श्रद्धया परयोपेतास्ते मे युक्ततमा मताः ॥ २ ॥']),
    ('श्री भगवानुवाचइदं शरीरं कौन्तेय क्षेत्रमित्यभिधीयते।एतद्यो वेत्ति तं प्राहुः क्षेत्रज्ञ इति तद्विदः।।13.2।।', 2, ['श्री भगवानुवाच', 'इदं शरीरं कौन्तेय क्षेत्रमित्यभिधीयते ।', 'एतद्यो वेत्ति तं प्राहुः क्षेत्रज्ञ इति तद्विदः ॥ २ ॥']),
    ('श्री भगवानुवाचपरं भूयः प्रवक्ष्यामि ज्ञानानां ज्ञानमुत्तमम्।यज्ज्ञात्वा मुनयः सर्वे परां सिद्धिमितो गताः।।14.1।।', 1, ['श्री भगवानुवाच', 'परं भूयः प्रवक्ष्यामि ज्ञानानां ज्ञानमुत्तमम् ।', 'यज्ज्ञात्वा मुनयः सर्वे परां सिद्धिमितो गताः ॥ १ ॥']),
    ('अर्जुन उवाचकैर्लिंगैस्त्रीन्गुणानेतानतीतो भवति प्रभो।किमाचारः कथं चैतांस्त्रीन्गुणानतिवर्तते।।14.21।।', 21, ['अर्जुन उवाच', 'कैर्लिंगैस्त्रीन्गुणानेतानतीतो भवति प्रभो ।', 'किमाचारः कथं चैतांस्त्रीन्गुणानतिवर्तते ॥ २१ ॥']),
    ('श्री भगवानुवाचप्रकाशं च प्रवृत्तिं च मोहमेव च पाण्डव।न द्वेष्टि सम्प्रवृत्तानि न निवृत्तानि काङ्क्षति।।14.22।।', 22, ['श्री भगवानुवाच', 'प्रकाशं च प्रवृत्तिं च मोहमेव च पाण्डव ।', 'न द्वेष्टि सम्प्रवृत्तानि न निवृत्तानि काङ्क्षति ॥ २२ ॥']),
    ('श्री भगवानुवाचऊर्ध्वमूलमधःशाखमश्वत्थं प्राहुरव्ययम्।छन्दांसि यस्य पर्णानि यस्तं वेद स वेदवित्।।15.1।।', 1, ['श्री भगवानुवाच', 'ऊर्ध्वमूलमधःशाखमश्वत्थं प्राहुरव्ययम् ।', 'छन्दांसि यस्य पर्णानि यस्तं वेद स वेदवित् ॥ १ ॥']),
    ('अर्जुन उवाचये शास्त्रविधिमुत्सृज्य यजन्ते श्रद्धयाऽन्विताः।तेषां निष्ठा तु का कृष्ण सत्त्वमाहो रजस्तमः।।17.1।।', 1, ['अर्जुन उवाच', 'ये शास्त्रविधिमुत्सृज्य यजन्ते श्रद्धयाऽन्विताः ।', 'तेषां निष्ठा तु का कृष्ण सत्त्वमाहो रजस्तमः ॥ १ ॥']),
    ('श्री भगवानुवाचत्रिविधा भवति श्रद्धा देहिनां सा स्वभावजा।सात्त्विकी राजसी चैव तामसी चेति तां श्रृणु।।17.2।।', 2, ['श्री भगवानुवाच', 'त्रिविधा भवति श्रद्धा देहिनां सा स्वभावजा ।', 'सात्त्विकी राजसी चैव तामसी चेति तां श्रृणु ॥ २ ॥']),
    ('अर्जुन उवाचनष्टो मोहः स्मृतिर्लब्धा त्वत्प्रसादान्मयाच्युत।स्थितोऽस्मि गतसन्देहः करिष्ये वचनं तव।।18.73।।', 73, ['अर्जुन उवाच', 'नष्टो मोहः स्मृतिर्लब्धा त्वत्प्रसादान्मयाच्युत ।', 'स्थितोऽस्मि गतसन्देहः करिष्ये वचनं तव ॥ ७३ ॥']),
    ('सञ्जय उवाचइत्यहं वासुदेवस्य पार्थस्य च महात्मनः।संवादमिममश्रौषमद्भुतं रोमहर्षणम्।।18.74।।', 74, ['सञ्जय उवाच', 'इत्यहं वासुदेवस्य पार्थस्य च महात्मनः ।', 'संवादमिममश्रौषमद्भुतं रोमहर्षणम् ॥ ७४ ॥']),
]
for t, n, exp in glued: assert padas(t, n) == exp, (n, padas(t, n))

# no piece opens on a dependent vowel sign, virama, anusvāra or visarga, over verses as vedicreader ships them: Gītā 1.2, 2.1, 2.2, 2.4; Kaṭha 1.1.1, 1.1.4; Kena 1.1, 1.2, 1.5; Chāndogya 4.1.3, 4.1.5, 4.2.3-5
DEP = re.compile('^[\u0900-\u0903\u093a-\u094f\u0951-\u0957\u0962\u0963]')
real = [g11, K41, blob,
        'सञ्जय उवाच दृष्ट्वा तु पाण्डवानीकं व्यूढं दुर्योधनस्तदा । आचार्यमुपसङ्गम्य राजा वचनमब्रवीत् ॥ २ ॥',
        'सञ्जय उवाच तं तथा कृपयाऽविष्टमश्रुपूर्णाकुलेक्षणम् । विषीदन्तमिदं वाक्यमुवाच मधुसूदनः ॥ १ ॥',
        'श्री भगवानुवाच कुतस्त्वा कश्मलमिदं विषमे समुपस्थितम् । अनार्यजुष्टमस्वर्ग्यमकीर्तिकरमर्जुन ॥ २ ॥',
        'अर्जुन उवाच कथं भीष्ममहं संख्ये द्रोणं च मधुसूदन । इषुभिः प्रतियोत्स्यामि पूजार्हावरिसूदन ॥ ४ ॥',
        'ॐ उशन् ह वै वाजश्रवसः सर्ववेदसं ददौ । तस्य ह नचिकेता नाम पुत्र आस ॥ १ ॥',
        'स होवाच पितरं तत कस्मै मां दास्यसीति । द्वितीयं तृतीयं तँ होवाच मृत्यवे त्वा ददामीति ॥ ४ ॥',
        'ॐ केनेषितं पतति प्रेषितं मनः केन प्राणः प्रथमः प्रैति युक्तः । केनेषितां वाचमिमां वदन्ति चक्षुः श्रोत्रं क उ देवो युनक्ति ॥ १ ॥',
        'श्रोत्रस्य श्रोत्रं मनसो मनो यद् वाचो ह वाचं स उ प्राणस्य प्राणः । चक्षुषश्चक्षुरतिमुच्य धीराः प्रेत्यास्माल्लोकादमृता भवन्ति ॥ २ ॥',
        'यद्वाचाऽनभ्युदितं येन वागभ्युद्यते । तदेव ब्रह्म त्वं विद्धि नेदं यदिदमुपासते ॥ ५ ॥',
        'तमु ह परः प्रत्युवाच कं वर एनमेतत् सन्तं सयुग्वानमिव रैक्वमात्थ इति । यो नु कथं सयुग्वा रैक्व इति ॥ ३ ॥',
        'तदु ह जानश्रुतिः पौत्रायण उपशुश्राव । स ह संजिहान एव क्षत्तारमुवाचाङ्गारे ह सयुग्वानमिव रैक्वमात्थेति । यो नु कथं सयुग्वा रैक्व इति ॥ ५ ॥',
        'तमु ह परः प्रत्युवाचाह हारेत्वा शूद्र तवैव सह गोभिरस्त्विति । तदु ह पुनरेव जानश्रुतिः पौत्रायणः सहस्रं गवां निष्कमश्वतरीरथं दुहितरं तदादाय प्रतिचक्रमे ॥ ३ ॥',
        'तं हाभ्युवाद । रैक्वेदं सहस्रं गवामयं निष्कोऽयमश्वतरीरथ इयं जायायं ग्रामो यस्मिन्नास्सेऽन्वेव मा भगवः शाधीति ॥ ४ ॥',
        'तस्या ह मुखमुपोद्गृह्णन्नुवाचाजहारेमाः शूद्रानेनैव मुखेनालापयिष्यथा इति । ते हैते रैक्वपर्णा नाम महावृषेषु यत्रास्मा उवास । तस्मै होवाच ॥ ५ ॥']
assert all(not DEP.match(p) for v in real for p in padas(v)), [p for v in real for p in padas(v) if DEP.match(p)]
```

## Chunkers

A search index stores a text as chunks, and the chunk boundaries decide
what a search result looks like: a chunk that ends mid-verse returns
half a verse.
[`chunk_verses`](https://vedicreader.github.io/ganapati/text.html#chunk_verses)
cuts on the verse boundaries that
[`verse_spans`](https://vedicreader.github.io/ganapati/text.html#verse_spans)
finds. It joins short verses until a chunk reaches `target` characters,
and splits a verse longer than `max_chars` at its daṇḍas, never inside a
clause. A heading always starts a new chunk. A chunk normally ends as
soon as it holds a verse with a citation, so that each cited verse is
its own result; pass `pack_cited=True` to let cited verses share a chunk
up to `target`.

[`VerseChunker`](https://vedicreader.github.io/ganapati/text.html#versechunker)
and
[`ProseChunker`](https://vedicreader.github.io/ganapati/text.html#prosechunker)
wrap
[`chunk_verses`](https://vedicreader.github.io/ganapati/text.html#chunk_verses)
in the interface of a chonkie chunker, so you can pass either to
litesearch’s `add_doc(chunker=...)`. Each chunk carries `start_index`
and `end_index`, its offsets in the source.
[`VerseChunker`](https://vedicreader.github.io/ganapati/text.html#versechunker)
aims for 300 characters and allows 900.
[`ProseChunker`](https://vedicreader.github.io/ganapati/text.html#prosechunker)
is for prose such as commentaries, bhāṣyas and the prose Upaniṣads: it
aims for 700, allows 1400, and packs cited units together. litesearch
reads the chunker’s settings too, and packs text across pages only for a
chunker that packs cited units, which the last two checks below confirm.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L176"
target="_blank" style="float:right; font-size:smaller">source</a>

### ProseChunker

``` python
def ProseChunker(
    target:int=700, max_chars:int=1400, pack_cited:bool=True
):
```

*A
[`VerseChunker`](https://vedicreader.github.io/ganapati/text.html#versechunker)
with larger chunks that packs cited units together, for commentaries,
bhāṣyas and the prose Upaniṣads.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L165"
target="_blank" style="float:right; font-size:smaller">source</a>

### VerseChunker

``` python
def VerseChunker(
    target:int=300, max_chars:int=900, pack_cited:bool=False
):
```

*A chunker with the chonkie interface that cuts on verse boundaries, for
short verses.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L117"
target="_blank" style="float:right; font-size:smaller">source</a>

### chunk_verses

``` python
def chunk_verses(
    text:str, # source text
    target:int=300, # pack short units forward up to this many chars
    max_chars:int=900, # split a unit longer than this at clause boundaries
    pack_cited:bool=False, # let consecutive cited units share a chunk up to `target`
)->fastcore.foundation.L:
```

*Cut a text into chunks on verse boundaries, joining short verses up to
`target` and splitting long ones at their daṇḍas.*

``` python
from litesearch.tree import pack_target
```

``` python
cs = VerseChunker().chunk(two)
assert len(cs) == 2
assert two[cs[0].start_index:cs[0].end_index] == cs[0].text   # offsets point back at the source

# ProseChunker packs cited units instead, which is what commentary wants
assert len(ProseChunker().chunk(two)) == 1

# with no citation to cut on, the budget decides. 3,040 characters of daṇḍa-separated prose:
long = ' | '.join(['padam yatra tatra sarvatra gacchati'] * 80) + ' ||'
assert len(chunk_verses(long, 300, 900)) == 4
assert len(chunk_verses(long, 700, 1400)) == 3

# litesearch packs across segments only for a chunker that asks; the verse profile does not,
# because a reader that emits one verse per page is already the right size
assert pack_target(VerseChunker()) == 0
assert pack_target(ProseChunker()) == 700
```

## Reading source files

Sanskrit texts are published in several file formats, and ganapati reads
five. Each reader takes a path or the file’s contents as a string and
returns `(pages, meta)`. `pages` is a list of `(index, text)` pairs,
usually one per verse or section, and `meta` is a dict with the format,
the title and whatever else the file states. When you don’t know the
format, call
[`sanskrit_parse`](https://vedicreader.github.io/ganapati/text.html#sanskrit_parse):
it looks at the extension and the first 4,000 characters and calls the
right reader.

<table>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr>
<th>reader</th>
<th>source</th>
</tr>
</thead>
<tbody>
<tr>
<td><a
href="https://vedicreader.github.io/ganapati/text.html#gretil_parse"><code>gretil_parse</code></a></td>
<td>GRETIL plain-text editions, the <code>*_u.htm</code> files</td>
</tr>
<tr>
<td><a
href="https://vedicreader.github.io/ganapati/text.html#tei_parse"><code>tei_parse</code></a></td>
<td>TEI XML, as GRETIL and SARIT publish it</td>
</tr>
<tr>
<td><a
href="https://vedicreader.github.io/ganapati/text.html#vr_xml_parse"><code>vr_xml_parse</code></a></td>
<td>vedicreader <code>&lt;lyrics&gt;</code> XML, which carries audio
timings and a word-by-word analysis</td>
</tr>
<tr>
<td><a
href="https://vedicreader.github.io/ganapati/text.html#vr_json_parse"><code>vr_json_parse</code></a></td>
<td>vedicreader content JSON, which carries the same information</td>
</tr>
<tr>
<td><a
href="https://vedicreader.github.io/ganapati/text.html#dcs_parse"><code>dcs_parse</code></a></td>
<td>CoNLL-U files from the Digital Corpus of Sanskrit (DCS), which give
each word’s lemma</td>
</tr>
</tbody>
</table>

[`is_sanskrit`](https://vedicreader.github.io/ganapati/text.html#is_sanskrit)
answers a simpler question: whether a piece of text looks like Sanskrit
at all, judged by its script, its IAST diacritics, or its citations and
daṇḍas.

The vedicreader formats need a few more words. A line marked
`ignore="true"` is printed but not recited, such as a title or a rubric,
so the reader leaves it out of the text; the first line of the `title`
section becomes `meta['title']`. A line aligned to a recording carries
`start_time_ms`, `end_time_ms` and `alignment_id`.
[`time_marker`](https://vedicreader.github.io/ganapati/text.html#time_marker)
writes these into the text as `[t 0-1800 a1]` so that they survive
chunking, and
[`line_times`](https://vedicreader.github.io/ganapati/text.html#line_times)
reads them back from a chunk as `AttrDict(text, start, end, aid, ms)`. A
line’s translation is written on a following line that starts with `>`,
and its word-by-word analysis on lines that start with `> etym:`, which
[`line_etyms`](https://vedicreader.github.io/ganapati/text.html#line_etyms)
returns. The metre code ignores `>` lines and anything in square
brackets, so neither a gloss nor a timing is scanned as verse.

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L423"
target="_blank" style="float:right; font-size:smaller">source</a>

### sanskrit_parse

``` python
def sanskrit_parse(
    src
)->tuple:
```

*Read any supported Sanskrit source into `(pages, meta)`, choosing the
reader from the extension and the first 4,000 characters.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L412"
target="_blank" style="float:right; font-size:smaller">source</a>

### is_sanskrit

``` python
def is_sanskrit(
    text:str, thresh:float=0.15
)->bool:
```

*Guess whether `text` is Sanskrit from its script, its IAST diacritics,
or its citations and daṇḍas.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L361"
target="_blank" style="float:right; font-size:smaller">source</a>

### dcs_parse

``` python
def dcs_parse(
    src
)->tuple:
```

*Read a DCS CoNLL-U file into `(pages, meta)`, one page per sentence
with its lemmas on a `> lemmas:` line.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L337"
target="_blank" style="float:right; font-size:smaller">source</a>

### vr_json_parse

``` python
def vr_json_parse(
    src
)->tuple:
```

*Read vedicreader content JSON into `(pages, meta)`, in the same shape
as
[`vr_xml_parse`](https://vedicreader.github.io/ganapati/text.html#vr_xml_parse).*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L296"
target="_blank" style="float:right; font-size:smaller">source</a>

### vr_xml_parse

``` python
def vr_xml_parse(
    src
)->tuple:
```

*Read vedicreader `<lyrics>` XML into `(pages, meta)`, keeping audio
timings and the word-by-word analysis.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L269"
target="_blank" style="float:right; font-size:smaller">source</a>

### line_etyms

``` python
def line_etyms(
    text:str
)->fastcore.foundation.L:
```

*Return the word-by-word analysis a source supplies on its `> etym:`
lines, without the prefix.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L260"
target="_blank" style="float:right; font-size:smaller">source</a>

### line_times

``` python
def line_times(
    text:str
)->fastcore.foundation.L:
```

*Return every timed line in a chunk as
`AttrDict(text, start, end, aid, ms)`, in order.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L253"
target="_blank" style="float:right; font-size:smaller">source</a>

### time_marker

``` python
def time_marker(
    start:int, # line start in ms
    end:int, # line end in ms
    aid:str=None, # the aligner's id for the span
)->str:
```

*Write one line’s audio span as a text marker such as `[t 0-1800 a1]`,
so the timing survives chunking.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L209"
target="_blank" style="float:right; font-size:smaller">source</a>

### tei_parse

``` python
def tei_parse(
    src
)->tuple:
```

*Read a TEI XML edition from GRETIL or SARIT into `(pages, meta)`, with
one page per division heading, verse group or paragraph.*

------------------------------------------------------------------------

<a
href="https://github.com/vedicreader/ganapati/blob/main/ganapati/text.py#L190"
target="_blank" style="float:right; font-size:smaller">source</a>

### gretil_parse

``` python
def gretil_parse(
    src
)->tuple:
```

*Read a GRETIL plain-text edition (`*_u.htm`) into `(pages, meta)`,
keeping its header in `meta`.*

The first example below parses a short vedicreader XML file with a title
line, a section translation, and two lines aligned to audio, one with
its word-by-word analysis. The second gives the same content as JSON.
The third shows
[`is_sanskrit`](https://vedicreader.github.io/ganapati/text.html#is_sanskrit)
on IAST, Devanagari and English, and the last reads GRETIL’s edition of
Mahābhārata 1.1 from `mahabharata.htm`.

``` python
VR = """<lyrics>
  <category>vedic</category><tags>rudram</tags>
  <section name="title">
    <line ignore="true">॥ श्रीरुद्रचमकप्रश्नः ॥</line>
  </section>
  <section name="anuvaka 1" meaning="A dual invocation of Agni and Viṣṇu.">
    <line start_time_ms="0" end_time_ms="1800" alignment_id="a1" caption="of one accord"
          etymology="अग्नाविष्णू, agni-viṣṇu, vocative case, dual number, O Agni and Viṣṇu.&#10;सजोषसा, sajoṣas, instrumental case, singular number, of one accord.">अग्नाविष्णू सजोषसेमा<w start_time_ms="0" end_time_ms="900">अग्नाविष्णू</w><w start_time_ms="900" end_time_ms="1800">सजोषसेमा</w></line>
    <line start_time_ms="1800" end_time_ms="3400" alignment_id="a2">वर्धन्तु वां गिरः ॥</line>
  </section>
</lyrics>"""

pages, meta = vr_xml_parse(VR)
assert meta['title'] == '॥ श्रीरुद्रचमकप्रश्नः ॥'        # kept as the title
assert not any('श्रीरुद्रचमकप्रश्नः' in t for _, t in pages)   # and dropped from the content

ts = line_times(pages[-1][1])
assert [t.start for t in ts] == [0, 1800] and [t.ms for t in ts] == [1800, 1600]
assert ts[0].aid == 'a1' and ts[0].text == 'अग्नाविष्णू सजोषसेमा'   # `<w>` children stay out of the line
# the corpus writes a whole word-by-word analysis into one attribute, so every physical line needs the
# `> etym:` prefix: an unprefixed continuation line reaches the scansion as if it were a pāda
es = line_etyms(pages[-1][1])
assert len(es) == 2 and es[0].startswith('अग्नाविष्णू,') and es[1].startswith('सजोषसा,')
quoted = lambda pg: all(ln.startswith(('>', '##')) or '[t ' in ln for ln in pg.splitlines() if ln.strip())
assert quoted(pages[-1][1])
```

``` python
VRJ = """{"title": "॥ श्रीरुद्रचमकप्रश्नः ॥", "category": "vedic", "tags": "rudram",
 "sections": [{"name": "anuvaka 1", "meaning": "A dual invocation of Agni and Viṣṇu.",
   "lines": [{"t": "अग्नाविष्णू सजोषसेमा", "s": 0, "e": 1800, "cap": "of one accord",
              "etym": "अग्नाविष्णू, agni-viṣṇu, vocative case, dual number, O Agni and Viṣṇu.\\nसजोषसा, sajoṣas, instrumental case, singular number, of one accord.",
              "w": [["अग्नाविष्णू", 0, 900], ["सजोषसेमा", 900, 1800]]},
             {"t": "वर्धन्तु वां गिरः ॥", "s": 1800, "e": 3400},
             {"t": "[bell]", "role": "audio", "s": 3400, "e": 4000}]}]}"""

pj, mj = vr_json_parse(VRJ)
assert mj['title'] == meta['title'] and mj['category'] == 'vedic'
assert [t.ms for t in line_times(pj[-1][1])] == [1800, 1600]   # `audio` is recited but not printed, so not verse
assert len(line_etyms(pj[-1][1])) == 2                         # one entry per physical line, same as the XML reader
assert quoted(pj[-1][1])
assert sanskrit_parse(VRJ)[1]['fmt'] == 'vedicreader'          # picked by shape, not by extension
```

``` python
assert is_sanskrit('śrīmātā śrīmahārājñī || LSN_1 ||\nśrīmatsiṃhāsaneśvarī || LSN_2 ||\n'
                   'cidagnikuṇḍasambhūtā || LSN_3 ||')
assert is_sanskrit('अथ योगानुशासनम् ॥ १॥')
assert not is_sanskrit('This is an ordinary English paragraph about nothing in particular.')
assert not is_sanskrit('')
assert _detag('dharma&nbsp;artha') == 'dharma artha'
```

``` python
sanskrit_parse('mahabharata.htm')[0][0]
```

    (0,
     'nārāyaṇaṃ namaskṛtya naraṃ caiva narottamam\ndevīṃ sarasvatīṃ caiva tato jayam udīrayet')