# journey
A learner often wants to learn more than one concept. “I want to learn
how LLMs work” needs tokenizers, embeddings, attention, transformers and
more, in a given order. A journey keeps that goal and that order on the
learning shelf.
The planner (one model call) proposes the steps. It uses the concepts on
the shelf when they fit. It makes new concepts for the topics that the
shelf does not have. Then the learner can change the path: move, skip,
mark as known, add and split steps.
The journey does not keep progress. Progress comes from the learner
model each time. A step is learned when the floor of its concept reaches
the goal depth. So a concept that the learner learns anywhere counts in
each journey that has it.
Import the modules that this notebook uses.
## The document
A journey is one vault document of kind `journey`. Its meta has the
goal, a title, a subject, and the steps. Each step has a slug, a status
(`todo`, `known` or `skip`) and the reason that it is in the path.
------------------------------------------------------------------------
source
### Learning.drop_journey
``` python
def drop_journey(
jid:str
):
```
*Remove a journey. Its concepts and their state stay.*
------------------------------------------------------------------------
source
### Learning.journeys
``` python
def journeys()->fastcore.foundation.L:
```
*All journeys, the most recently changed first.*
------------------------------------------------------------------------
source
### Learning.journey
``` python
def journey(
jid:str
)->fastcore.basics.AttrDict | None:
```
*One journey by id.*
------------------------------------------------------------------------
source
### Learning.put_journey
``` python
def put_journey(
j:dict
)->fastcore.basics.AttrDict:
```
*Save journey `j`, and replace an earlier one with the same id.*
## Progress
[`step_state`](https://vedicreader.github.io/abhyasa/journey.html#step_state)
names the state of one step from its status and its concept. The states
[`learned`](https://vedicreader.github.io/abhyasa/model.html#learned),
`known`, `skipped` and `missing` (the concept is gone) are done. The
states `new` and `started` are not done.
------------------------------------------------------------------------
source
### progress
``` python
def progress(
lr:abhyasa.store.Learning, j, goal:int=4, by:dict=None
)->fastcore.basics.AttrDict:
```
*Each step with its concept and state, the count done, the count that
counts, and the current step.*
------------------------------------------------------------------------
source
### step_state
``` python
def step_state(
c, status:str='todo', goal:int=4
)->str:
```
*One of new, started, learned, known, skipped or missing.*
[`current`](https://vedicreader.github.io/abhyasa/journey.html#current)
gives the slug of the step to learn now. A step in `skip` waits, unless
no other step is left.
------------------------------------------------------------------------
source
### current
``` python
def current(
lr:abhyasa.store.Learning, j, skip:tuple=(), goal:int=4
)->str | None:
```
*The slug of the first step that is not done and not in `skip`; else the
first step that is not done.*
## Change the path
`journey_step` makes one change to one step. `journey_add` adds a step
at the end, or before another step.
------------------------------------------------------------------------
source
### Learning.journey_add
``` python
def journey_add(
jid:str, slug:str, before:str=None, why:str=''
)->fastcore.basics.AttrDict:
```
*Add concept `slug` as a step, at the end or before step `before`. A
step that is there already stays where it is.*
------------------------------------------------------------------------
source
### Learning.journey_step
``` python
def journey_step(
jid:str, slug:str, act:str
)->fastcore.basics.AttrDict:
```
*Do `act` on step `slug`: up, down, skip, known, todo or remove.*
## The planner
The planner gets the goal and the learner profile as text. It also gets
the concepts on the shelf that are near the goal or that the learner
started. It returns the steps in order.
------------------------------------------------------------------------
source
### near_goal
``` python
def near_goal(
lr:abhyasa.store.Learning, goal:str, k:int=40, most:int=150
)->list:
```
*`(slug, name, floor)` for the concepts that search finds for `goal`,
and the ones the learner started.*
------------------------------------------------------------------------
source
### plan_prompt
``` python
def plan_prompt(
goal:str, who:str='', known:tuple=()
)->str:
```
*The prompt that plans a path to `goal`.*
[`plan`](https://vedicreader.github.io/abhyasa/journey.html#plan) saves
the journey. A step can name a known slug, or a known name or alias.
Then it uses that concept and keeps its state. Each other step becomes a
new concept, with the steps it needs as its prereqs. With `sources`,
[`plan`](https://vedicreader.github.io/abhyasa/journey.html#plan) links
each new concept to the passages that name it in the learner’s own
sources.
------------------------------------------------------------------------
source
### plan
``` python
def plan(
lr:abhyasa.store.Learning, ask, goal:str, who:str='', sources:NoneType=None
)->fastcore.basics.AttrDict:
```
*Plan a journey to `goal`, make the concepts that the shelf does not
have, and save it.*
## Add a topic, split a step
[`add_topic`](https://vedicreader.github.io/abhyasa/journey.html#add_topic)
adds a topic by name. When the shelf knows the name,
[`add_topic`](https://vedicreader.github.io/abhyasa/journey.html#add_topic)
uses that concept. For a new name, the model writes a short statement.
------------------------------------------------------------------------
source
### add_topic
``` python
def add_topic(
lr:abhyasa.store.Learning, ask, jid:str, name:str, before:str=None, sources:NoneType=None
)->fastcore.basics.AttrDict:
```
*Add topic `name` to journey `jid`, as a concept on the shelf. Return
the journey.*
[`name_gaps`](https://vedicreader.github.io/abhyasa/journey.html#name_gaps)
keeps each idea that you say you do not know as a gap of the concept you
learn. It finds the idea on the shelf by name or alias. Or it adds the
idea with a short statement. Next time the gap comes first, but it does
not block the concept.
------------------------------------------------------------------------
source
### name_gaps
``` python
def name_gaps(
lr:abhyasa.store.Learning, ask, slug:str, terms, sources:NoneType=None
)->fastcore.foundation.L:
```
*Find or add a concept for each idea in `terms`, and keep it as a gap of
`slug`. Return the gap slugs.*
A new idea becomes a concept with a statement and the gap provenance.
[`name_gaps`](https://vedicreader.github.io/abhyasa/journey.html#name_gaps)
finds an idea that is already on the shelf by its alias. It does not add
that idea again.
``` python
from abhyasa.store import temp_learning
from abhyasa.llm import Fake
ngl = temp_learning(); ngl.put('Functions and the neuron', 'z = w . x + b.', 'ml'); ngl.put('Bias', 'A constant added to z.', 'ml', aliases=['the bias'])
gask = Fake({'The idea: the loop': dict(name='Weighted sum as a loop', statement='A loop adds each weight times its input.')})
test_eq(name_gaps(ngl, gask, 'functions-and-the-neuron', ['the loop', 'the bias', 'the loop']), ['weighted-sum-as-a-loop', 'bias'])
g = ngl.concept('weighted-sum-as-a-loop')
test_eq((g.aliases, g.provenance[0]['src'], g.provenance[0]['of']), (['the loop'], 'gap', 'functions-and-the-neuron'))
test_eq(ngl.concept('functions-and-the-neuron').gaps, ['weighted-sum-as-a-loop', 'bias'])
```
[`split`](https://vedicreader.github.io/abhyasa/journey.html#split)
divides a big step into smaller steps. It runs
[`extract`](https://vedicreader.github.io/abhyasa/sources.html#extract)
over the statement of the step and the passages from the learner’s
sources. The concepts that it finds go before the step, as its prereqs.
------------------------------------------------------------------------
source
### split
``` python
def split(
lr:abhyasa.store.Learning, ask, jid:str, slug:str, sources:NoneType=None
)->fastcore.foundation.L:
```
*Find the parts of step `slug`, add them before it as steps and prereqs,
and return their slugs.*
## Related topics
[`related`](https://vedicreader.github.io/abhyasa/journey.html#related)
gives the concepts next to `slug` in the graph. First come the concepts
that build on it, then the related ones, then its prereqs. The concepts
that are not learned come first.
[`with_step`](https://vedicreader.github.io/abhyasa/journey.html#with_step)
finds the journeys that have a concept.
[`beyond`](https://vedicreader.github.io/abhyasa/journey.html#beyond)
gives what comes after a journey: the concepts that build on its steps
and are not in it.
------------------------------------------------------------------------
source
### beyond
``` python
def beyond(
lr:abhyasa.store.Learning, j, k:int=5, goal:int=4
)->fastcore.foundation.L:
```
*Up to `k` concepts that build on the steps of journey `j` and are not
in it.*
------------------------------------------------------------------------
source
### with_step
``` python
def with_step(
lr:abhyasa.store.Learning, slug:str
)->fastcore.foundation.L:
```
*`(journey, index)` for each journey that has step `slug`.*
------------------------------------------------------------------------
source
### related
``` python
def related(
lr:abhyasa.store.Learning, slug:str, k:int=6, goal:int=4, by:dict=None
)->fastcore.foundation.L:
```
*Up to `k` concepts next to `slug`: what builds on it, what relates to
it, then what it needs.*
## Tests
A fake model plans a path to LLMs. The shelf has `attention` already, at
depth 3.
``` python
from abhyasa.store import temp_learning
from abhyasa.llm import Fake
lr = temp_learning()
lr.put('Attention', 'Weights from query-key dot products.', 'ml')
lr.update('attention', floor=3, seen=1)
fake = Fake({'You plan a learning path': dict(title='How LLMs work', subject='ml', steps=[
dict(name='Tokenizers', statement='A tokenizer cuts text into tokens.', slug='', why='The model reads tokens.', needs=[]),
dict(name='Embeddings', statement='An embedding maps a token to a vector.', slug='', why='Tokens become vectors.', needs=['Tokenizers']),
dict(name='Attention', statement='', slug='attention', why='The core of a transformer.', needs=['Embeddings']),
dict(name='Mamba', statement='Mamba is a selective state-space model.', slug='', why='Another way to mix tokens.', needs=['Attention'])])})
j = plan(lr, fake, 'I want to learn how LLMs work', who='Background: codes in Python')
```
[`plan`](https://vedicreader.github.io/abhyasa/journey.html#plan) keeps
the order, uses the known concept, and makes the new ones with their
prereqs.
``` python
test_eq([s['slug'] for s in j.steps], ['tokenizers', 'embeddings', 'attention', 'mamba'])
test_eq((lr.concept('embeddings').prereqs, lr.concept('mamba').prereqs, lr.concept('mamba').subject), (['tokenizers'], ['attention'], 'ml'))
test_eq(lr.concept('attention').floor, 3) # a known concept keeps its state
assert 'codes in Python' in fake.calls[-1][0] and 'attention: Attention (depth 3)' in fake.calls[-1][0]
```
[`progress`](https://vedicreader.github.io/abhyasa/journey.html#progress)
names the state of each step. The current step is the first that is not
done.
``` python
pg = progress(lr, j)
test_eq([s.state for s in pg.steps], ['new', 'new', 'started', 'new'])
test_eq((pg.done, pg.total, pg.current.slug), (0, 4, 'tokenizers'))
```
A step that the learner learns anywhere is learned in this journey too.
A step that is marked known or skipped is done. Then
[`current`](https://vedicreader.github.io/abhyasa/journey.html#current)
moves to the next step.
``` python
lr.update('tokenizers', floor=4, seen=2)
lr.journey_step(j.id, 'embeddings', 'known')
pg = progress(lr, lr.journey(j.id))
test_eq([s.state for s in pg.steps], ['learned', 'known', 'started', 'new'])
test_eq((pg.done, current(lr, lr.journey(j.id)), current(lr, lr.journey(j.id), skip={'attention'})), (2, 'attention', 'mamba'))
```
Steps move up and down. A removed step leaves the path. A skipped step
does not count in the total.
``` python
lr.journey_step(j.id, 'mamba', 'up')
test_eq([s['slug'] for s in lr.journey(j.id).steps], ['tokenizers', 'embeddings', 'mamba', 'attention'])
lr.journey_step(j.id, 'mamba', 'skip')
test_eq(progress(lr, lr.journey(j.id)).total, 3)
lr.journey_step(j.id, 'mamba', 'remove')
test_eq(len(lr.journey(j.id).steps), 3)
```
[`add_topic`](https://vedicreader.github.io/abhyasa/journey.html#add_topic)
uses a known name, and asks the model only for a new one.
``` python
fake.rules['The idea: '] = dict(name='PyTorch', statement='PyTorch is a Python library for tensors and autograd.')
add_topic(lr, fake, j.id, 'PyTorch')
add_topic(lr, fake, j.id, 'tokenizers') # known, and in the path already: no change
test_eq([s['slug'] for s in lr.journey(j.id).steps], ['tokenizers', 'embeddings', 'attention', 'pytorch'])
```
[`split`](https://vedicreader.github.io/abhyasa/journey.html#split) adds
the parts of a step before it, as its prereqs.
``` python
fake.rules['Return JSON: {"concepts"'] = dict(concepts=[dict(name='Query and key', statement='Each token has a query and a key.', aliases=[], prereqs=[])])
new = split(lr, fake, j.id, 'attention')
test_eq(list(new), ['query-and-key'])
test_eq([s['slug'] for s in lr.journey(j.id).steps], ['tokenizers', 'embeddings', 'query-and-key', 'attention', 'pytorch'])
assert 'query-and-key' in lr.concept('attention').prereqs
```
[`related`](https://vedicreader.github.io/abhyasa/journey.html#related)
puts what builds on a concept first.
[`with_step`](https://vedicreader.github.io/abhyasa/journey.html#with_step)
finds the journeys of a concept.
[`beyond`](https://vedicreader.github.io/abhyasa/journey.html#beyond)
gives what comes after.
``` python
lr.put('Transformer block', 'Attention, then a feed-forward layer.', 'ml', prereqs=['attention'])
test_eq(related(lr, 'attention').attrgot('slug'), ['mamba', 'transformer-block', 'query-and-key']) # mamba left the path, but it builds on attention
test_eq([(x.id, i) for x, i in with_step(lr, 'attention')], [(j.id, 3)])
test_eq(beyond(lr, lr.journey(j.id)).attrgot('slug'), ['mamba', 'transformer-block'])
```
`journeys` lists the journeys with the newest change first.
`drop_journey` removes one journey.
``` python
j2 = lr.put_journey(dict(id='sanskrit', title='Read the Gita', goal='read', steps=[]))
test_eq(lr.journeys().attrgot('id'), ['sanskrit', j.id])
lr.drop_journey('sanskrit')
test_eq(len(lr.journeys()), 1)
```