pullup

Find out how a project is released, run the release one step at a time, and look after the workflows and servers it ships to.

pullup is the engine behind a release and deploy panel. Point it at a project folder and it tells you what kind of project it is (an nbdev library, a plain Python package, a Rust crate and so on) and which commands release it. It then runs those commands in a terminal you can watch, and stops at the first one that fails. The same machinery runs the deploy of a web app to a virtual private server (VPS). pullup also reads the GitHub Actions workflows in the repository, writes new ones, and lists the Hetzner servers and Cloudflare tunnels the project uses.

Every public call returns plain dictionaries and lists, so a user interface can draw them and a test can compare them. The work that talks to GitHub, Docker, Hetzner and Cloudflare is done by four sibling libraries: gheasy, dockeasy, vpseasy and cfeasy.

Install

pip install pullup

This also installs gheasy, dockeasy, vpseasy and cfeasy. None of them is imported until a call needs it, so reading a project or running a release does not load them.

What kind of project is this

A release flow is the ordered list of commands that publishes one kind of project. An nbdev library is released with nbdev-prepare, nbdev-bump-version, nbdev-release-gh and nbdev-pypi. A plain Python package and a Rust crate each have a different list. Project looks at the files in a folder, picks the flow that fits, and returns its steps. The folder here holds only a pyproject.toml with a [tool.nbdev] table, so it is an nbdev project:

project = Project(root)
project.kind, [step.id for step in project.steps()]

('nbdev', ['prepare', 'bump', 'gh', 'pypi'])

Running the steps

A Pipeline runs a list of steps for one project. Each step runs in a pseudo-terminal (PTY), so a program that asks for a password behaves as it would in your own shell, and every step’s output goes into one shared transcript. When a step passes, the next one starts by itself. When one fails, the pipeline stops there.

save stores the plan and start launches the first step that has not yet passed. start returns as soon as the step is running, not when it finishes, which is why the transcript below shows the command but not yet its output. Call state or tail whenever you want to see how far it has got.

pipeline = Pipeline(root, dir='.demo')
pipeline.save([{'id': 'hello', 'cmd': 'say hello',
                'argv': [sys.executable, '-c', 'print("ready")']}])
await pipeline.start()
pipeline.state()['done'], pipeline.tail().splitlines()[-2:]

(False, ['', '❯ say hello'])

Where plans are kept

A saved plan is a JSON file inside the project folder, in the directory you pass as dir (.pullup unless you say otherwise). pullup never writes a plan outside the project:

custom = Pipeline(root, dir='.myapp')
custom.save([{'id': 'show', 'cmd': 'show', 'argv': [sys.executable, '-c', 'print("saved")']}])
custom.path.relative_to(root)

Path('.myapp/pipeline.json')

Deploying a web app

Deploy is a pipeline that puts a web app on a server. Its steps run the subcommands of a deploy.py script that lives in the project, and Deploy.scaffold can write that script for you. Drive covers the same ground one library call at a time (write the Dockerfile, open the tunnel, provision the server), so you can run and check each part on its own. Both are pipelines, so they start, stop and report like the one above.

Deploy(root).state()['kind'], Drive(root).state()['kind']

('deploy', 'drive')

Workflows

Workflows reads the GitHub Actions files in .github/workflows and turns each one into a row of triggers and jobs. Reading local files needs no GitHub token. Listing runs, starting them and pushing secrets do.

workflow_dir = root/'.github'/'workflows'
workflow_dir.mkdir(parents=True)
(workflow_dir/'ci.yml').write_text('''name: CI
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - run: pytest
''')
workflows = Workflows(root)
[(row['name'], [job['id'] for job in row['jobs']]) for row in workflows.parsed()]

[('CI', ['test'])]

Environment values

Many steps need a token or a setting, such as GITHUB_TOKEN for a GitHub release. A step lists the names it needs under needs, and the pipeline looks them up in an environment store when it starts the step. The values are given to the child process only, and your own process environment is left alone. The default store is an EnvStore, which keeps secrets in the system keychain, but any object with get and values methods will do, like the small one below:

class Values:
    def get(self, key, secret=True): return 'demo-token' if key == 'DEMO_TOKEN' else ''
    def values(self, keys): return {key: self.get(key) for key in keys if self.get(key)}

with_values = Pipeline(root, env=Values(), dir='.env-demo')
with_values.save([{'id': 'env', 'cmd': 'read environment', 'needs': ['DEMO_TOKEN'],
                   'argv': [sys.executable, '-c', 'import os; print(os.environ["DEMO_TOKEN"])']}])
await with_values.start()
with_values.tail().splitlines()[-2:]

['', '❯ read environment']

Which package broke

When a step fails with a Python traceback, the useful question is which of your own packages raised it. blame reads the traceback text, finds the deepest frame that belongs to one of the packages named in family, and returns that package with the file, the line, the function and the error message:

trace = '''Traceback (most recent call last):
  File "/tmp/pullup/demo.py", line 7, in release
    raise RuntimeError("stopped")
RuntimeError: stopped
'''
blame(trace, family=['pullup'])

{'package': 'pullup',
 'file': '/tmp/pullup/demo.py',
 'line': 7,
 'fn': 'release',
 'error': 'RuntimeError: stopped',
 'open': '/tmp/pullup/demo.py',
 'checkout': '',
 'version': '0.0.4'}

Servers and tunnels

Infra lists and deletes Hetzner servers and Cloudflare tunnels, and lists Cloudflare zones and DNS records, using the tokens in the project’s environment store. status says which of the two libraries are installed and which tokens are set, without sending a request. An empty env falls back to the default EnvStore, so the answer below reflects the keychain of the machine that ran it:

infra = Infra(env={})
infra.status()

{'vpseasy': True,
 'cfeasy': True,
 'hcloud_token': True,
 'cf_token': False,
 'keys': ['HCLOUD_TOKEN', 'CLOUDFLARE_API_TOKEN']}