12. Vanes V: Gall and Userspace

This lesson covers Gall: the $agent type, running agents, and userspace software updates.

Gall

Currently, the end user zone of Urbit, denoted “userspace”, is supplied primarily by the /sys/vane/gall vane. Much as Arvo acts as a dispatcher and state manager for a functional OS in terms of transactions between vanes, Gall acts as a dispatcher and state manager for longrunning daemons. These daemons are referred to as “agents”, and comprise the main way that users actually use Urbit. (Other parts of userspace include generators and threads.)

Gall is responsible to manage userspace applications and their state, including subscribers. To fully understand agent userspace, we want to cover a few related topics:

1. Agents

2. /sys/vane/gall

3. Treaty/Docket publication

4. Agent wrappers

5. Historic (dynamic) Gall

We will cover threads in the lesson on Khan and Lick. We covered generators previously in the lesson on Dill and Dojo.

Agents

An agent is a piece of software that is primarily focused on maintaining and distributing a piece of state with a defined structure. It exposes an interface that lets programs read, subscribe to, and manipulate the state. Every event happens in an atomic transaction, so the state is never inconsistent. Since the state is permanent, when the agent is upgraded with a change to the structure of the state, the developer provides a migration function from the old state type to the new state type.

What is an agent in practice? In contemporary static Gall, an agent is a core that hews to the definition:

::                                                      ::::
::::                    ++gall                            ::  (1g) extensions
  ::                                                    ::::
++  gall  ^?
  |%
  ::
  ::  +agent: app core
  ::
  ++  agent
    =<  form
    |%
    +$  step  (quip card form)
    +$  card  (wind note gift)
    +$  note
      $%  [%agent [=ship name=term] =task]
          [%arvo note-arvo]
          [%pyre =tang]
      ::
          [%grow =spur =page]
          [%tomb =case =spur]
          [%cull =case =spur]
      ==
    +$  task
      $%  [%watch =path]
          [%watch-as =mark =path]
          [%leave ~]
          [%poke =cage]
          [%poke-as =mark =cage]
      ==
    +$  gift
      $%  [%fact paths=(list path) =cage]
          [%kick paths=(list path) ship=(unit ship)]
          [%watch-ack p=(unit tang)]
          [%poke-ack p=(unit tang)]
      ==
    +$  sign
      $%  [%poke-ack p=(unit tang)]
          [%watch-ack p=(unit tang)]
          [%fact =cage]
          [%kick ~]
      ==
    ++  form
      $_  ^|
      |_  bowl
      ++  on-init
        *(quip card _^|(..on-init))
      ::
      ++  on-save
        *vase
      ::
      ++  on-load
        |~  old-state=vase
        *(quip card _^|(..on-init))
      ::
      ++  on-poke
        |~  [mark vase]
        *(quip card _^|(..on-init))
      ::
      ++  on-watch
        |~  path
        *(quip card _^|(..on-init))
      ::
      ++  on-leave
        |~  path
        *(quip card _^|(..on-init))
      ::
      ++  on-peek
        |~  path
        *(unit (unit cage))
      ::
      ++  on-agent
        |~  [wire sign]
        *(quip card _^|(..on-init))
      ::
      ++  on-arvo
        |~  [wire sign-arvo]
        *(quip card _^|(..on-init))
      ::
      ++  on-fail
        |~  [term tang]
        *(quip card _^|(..on-init))
      --
    --
  --  ::gall

A Gall agent must have ten arms. (There's a fascinating bit of self-reference in the state definition going on, and to be honest I'm a little surprised that it works, but it is an iron core.) The definitions here are for |~ barsig arms in a ^| ketbar core.

• |~ barsig yields an iron gate.

• ^| ketbar yields an iron core from a gold core.

So we must at last really grapple with the core variance model in Urbit. This is often notorious to understand because we don't have great metaphors or analogues to type variance in real life. Right now, what we need to understand is that an iron/contravariant core is opaque: That is, we use this to define an interface in which the argument can be less specific than the interface and the result can be more specific. Contravariance is useful for flexibility in input values (samples).

An %iron core i has a write-only sample (payload head, +6.i) and an opaque context (payload tail, +7.i). A core j which nests within it must be a %gold or %iron core, such that +6.i nests within +6.j. Hence, contravariant.

The archetypal Gall agents in /sys/lull are composed using iron gates since they will be used as examples for building actual agent cores. Likewise, the +rs and sister gates in /sys/hoon are built using iron doors with specified rounding behavior so when you actually use the core (like +add:rs) the core you are using has been built as an example.

• How are the iron gate runes actually implemented in the Hoon type system? (See +deem:nest:ut and +peel:ut.)

• Try to implement an agent missing an arm, like +on-fail.

• Bonus question: what half-implemented rune produces an %iron core?

We construct an agent explicitly in an /app file by applying %- agent:gall to a correctly-shaped core.

• What does each arm produce?

++  on-init   (quip card _agent)
++  on-save   (vase)
++  on-load   (quip card _agent)
++  on-poke   (quip card _agent)
++  on-watch  (quip card _agent)
++  on-leave  (quip card _agent)
++  on-peek   (unit (unit cage))
++  on-agent  (quip card _agent)
++  on-arvo   (quip card _agent)
++  on-fail   (quip card _agent)

Finally, we can take a gander at what that ubiquitous +quip is:

++  quip
  |$  [item state]
  [(list item) state]

(It's just a wrapper for (list item) state.)

Basically, every arm must produces a list of effects and a state change, if any.

• What does each arm expect?

++  on-init   ::  not a gate, only an arm
++  on-save   ::  not a gate, only an arm
++  on-load   |=  =vase
++  on-poke   |=  =cage
++  on-watch  |=  =path
++  on-leave  |=  =path
++  on-peek   |=  =path
++  on-agent  |=  [=wire =sign:agent:gall]
++  on-arvo   |=  [=wire =sign-arvo]
++  on-fail   |=  [=term =tang]

We'll need to differentiate the Gall $signs and the Arvo $signs in a moment.

When +ford:clay reads in a Gall agent file from /app, it automatically composes cores together using => tisgar. (This leads to a slightly disconcerting situation in which the cores are simply present serially in a file.)

• Compare the definition of $agent:shoe with $agent:gall. How does this correctly extend the Gall agent definition for the type system?

Vane

While Gall facilitates very complex userspace apps, the vane itself is rather modest, weighing in at less than half the size of Clay or Ames. Gall knows how to route events to the handler arms in a standard agent core, and it instruments upgrades and subscriptions.

However, we have to consider Gall at two levels: the vane level, which manages top-level state like the set of running agents and queued moves, and the agent level, which manages agents as doors.

Gall is a landlocked vane. It has no runtime counterpart.

/sys/lull Definition

::                                                      ::::
::::                    ++gall                            ::  (1g) extensions
  ::                                                    ::::
++  gall  ^?
  |%
  +$  boar  (map [=wire =ship =term] nonce=@)           ::  and their nonces
  +$  dude  term                                        ::  server identity
  +$  gill  (pair ship term)                            ::  general contact
  +$  load  (list [=dude =beak =agent])                 ::  loadout
  +$  scar                                              ::  opaque duct
    $:  p=@ud                                           ::  bone sequence
        q=(map duct bone)                               ::  by duct
        r=(map bone duct)                               ::  by bone
    ==                                                  ::
  +$  suss  (trel dude @tas @da)                        ::  config report
  +$  well  (pair desk term)                            ::
  +$  deal
    $%  [%raw-poke =mark =noun]
        task:agent
    ==
  +$  unto
    $%  [%raw-fact =mark =noun]
        sign:agent
    ==
  --  ::gall

Most of the important types have been separated and are called out below.

(Variance again: ^? ketwut is for a lead/bivariant core.)

Vane State

+$  state
  $:  system-duct=duct
      outstanding=(map [wire duct] (qeu remote-request))
      contacts=(set ship)
      yokes=(map term yoke)
      blocked=(map term (qeu blocked-move))
      =bug
  ==

• system-duct is the set of outbound moves to other vanes (like Ames for subscriptions) or remote agent contacts.

• outstanding is the outstanding request queue.

• contacts is the set of other ships with which we are in communication.

• yokes is the set of running agents.

• blocked is the set of moves to agents that haven't been started yet.

• bug is the debug print configuration.

Vane Moves

|%
+$  gift                                              ::  outgoing result
  $%  [%boon payload=*]                               ::  ames response
      [%done error=(unit error:ames)]                 ::  ames message (n)ack
      [%unto p=unto]                                  ::
  ==                                                  ::
+$  task                                              ::  incoming request
  $~  [%vega ~]                                       ::
  $%  [%deal p=sock q=term r=deal]                    ::  full transmission
      [%sear =ship]                                   ::  clear pending queues
      [%jolt =desk =dude]                             ::  (re)start agent
      [%idle =dude]                                   ::  suspend agent
      [%load =load]                                   ::  load agent
      [%nuke =dude]                                   ::  delete agent
      [%doff dude=(unit dude) ship=(unit ship)]       ::  kill subscriptions
      [%rake dude=(unit dude) all=?]                  ::  reclaim old subs
      $>(%init vane-task)                             ::  set owner
      $>(%trim vane-task)                             ::  trim state
      $>(%vega vane-task)                             ::  report upgrade
      $>(%plea vane-task)                             ::  network request
      [%spew veb=(list verb)]                         ::  set verbosity
      [%sift dudes=(list dude)]                       ::  per agent
  ==                                                  ::
--

Agent State

+$  bitt  (map duct (pair ship path))                 ::  incoming subs
+$  boat  (map [=wire =ship =term] [acked=? =path])   ::  outgoing subs
+$  bowl                                              ::  standard app state
  $:  $:  our=ship                                    ::  host
          src=ship                                    ::  guest
          dap=term                                    ::  agent
      ==                                              ::
      $:  wex=boat                                    ::  outgoing subs
          sup=bitt                                    ::  incoming subs
          $=  sky                                     ::  scry bindings
          %+  map  path                               ::
          ((mop @ud (pair @da (each page @uvI))) lte) ::
      ==                                              ::
      $:  act=@ud                                     ::  change number
          eny=@uvJ                                    ::  entropy
          now=@da                                     ::  current time
          byk=beak                                    ::  load source
  ==  ==                                              ::                     

Every agent needs two parts of its state: the $bowl, which is the information outside of the agent that Gall needs to communicate for the vane, and the

Agent Moves

|%
+$  card  (wind note gift)
+$  note
  $%  [%agent [=ship name=term] =task]
      [%arvo note-arvo]
      [%pyre =tang]
  ::
      [%grow =spur =page]
      [%tomb =case =spur]
      [%cull =case =spur]
  ==
+$  task
  $%  [%watch =path]
      [%watch-as =mark =path]
      [%leave ~]
      [%poke =cage]
      [%poke-as =mark =cage]
  ==
+$  gift
  $%  [%fact paths=(list path) =cage]
      [%kick paths=(list path) ship=(unit ship)]
      [%watch-ack p=(unit tang)]
      [%poke-ack p=(unit tang)]
  ==
+$  sign
  $%  [%poke-ack p=(unit tang)]
      [%watch-ack p=(unit tang)]
      [%fact =cage]
      [%kick ~]
  ==
--

A Gall $card differs from an Arvo card:

::  Arvo
+$  card  (cask)                          ::  tagged, untyped event
++  cask  |$  [a]  (pair mark a)          ::  marked data builder
::
::  Gall
+$  card  (wind note gift)
++  wind
  |$  ::  a: forward
      ::  b: reverse
      ::
      [a b]
  $%  ::  %pass: advance
      ::  %slip: lateral
      ::  %give: retreat
      ::
      [%pass p=path q=a]
      [%slip p=a]
      [%give p=b]
  ==
+$  note
  $%  [%agent [=ship name=term] =task]
      [%arvo note-arvo]
      [%pyre =tang]
  ::
      [%grow =spur =page]
      [%tomb =case =spur]
      [%cull =case =spur]
  ==
+$  gift
  $%  [%fact paths=(list path) =cage]
      [%kick paths=(list path) ship=(unit ship)]
      [%watch-ack p=(unit tang)]
      [%poke-ack p=(unit tang)]
  ==

Gall does not permit a %slip, so a card is either:

• [%pass path note]

• [%give gift]

Structure

The two main engine cores within /sys/vane/gall are:

• +mo Arvo move handler

• +ap agent-level core

The +abet pattern used in Gall prefixes each arm with the containing door abbreviation so you can remain more easily oriented within /sys/vane/gall.

+mo Arvo move handler

Many +mo calls resolve into +ap calls. It mainly sets things up around particular per-agent calls.

+ap agent-level core

To run an agent, we have to know the state of the agent, which includes its state and relevant bowl information:

::  $yoke: agent runner state
::
+$  yoke
  $%  [%nuke sky=(map spur @ud)]
      $:  %live
          control-duct=duct
          run-nonce=@t
          sub-nonce=_1
          =stats
          =bitt
          =boat
          =boar
          code=*
          agent=(each agent vase)
          =beak
          marks=(map duct mark)
          sky=(map spur path-state)
          ken=(jug spar:ames wire)
  ==  ==

• control-duct is the duct of

• run-nonce is a unique nonce for each build.

• sub-nonce is global %watch nonce.

• stats

• bitt is the set of incoming subscriptions (for the bowl).

• boat is the set of outgoing subscriptions (for the bowl).

• code is the most recently loaded code as a noun.

• agent is the agent core, possibly as a vase.

• beak is the compilation source.

• marks is the map of mark conversion requests.

• sky is the map of scry bindings.

• ken is the map of sets of open %keen remote scry requests.

A typical call from +mo to +ap will be predicated on +ap setting up a Gall agent with its state and processing the incoming move through the appropriate arm.

For instance, this is the lifecycle of a scry call to Gall:

• A scry handler (roof) produces a call to Gall's +scry arm.

  • +mo as a door needs a duct and a set of moves.

    • +mo-peek sets up a call to +ap-peek along the given path with the appropriate care.

      • +ap-abed sets up the agent noun for evaluation.

        • +ap-yoke loads the actual agent state; an agent is a door with a state and bowl sample.

      • +ap-peek parses the scry path appropriately.

        • +ap-mule-peek evaluates the code using [9 2 0 1] and +mock (see ca01 for a refresher).

          • +ap-agent-core sets up the agent core with its current bowl and state; this includes a +on-peek arm since we know the shape of the $agent core.

            • +ap-construct-bowl produces the agent-ready bowl from Gall-level information.

The lifecycle of a poke looks like this:

• A move is injected targeting Gall's +call arm as a %deal (indicating that the move goes to an agent).

  • +call dispatches to +mo-handle-use for an agent.

    • +mo-handle-local is for running local agents.

      • +mo-apply and +mo-apply-sure prepare to call +ap.

        • +ap-abed sets up the agent noun for evaluation.

          • +ap-apply dispatches several kinds of operations, including pokes.

            • +ap-poke queues a %poke-ack (since it's first among moves) and calls +ap-ingest.

              • +ap-ingest calls the agent arm.

                • +ap-handle-result and +ap-handle-peers take care of watches etc.

                  • +ap-agent-core sets up the agent core with its current bowl and state; this includes a +on-poke arm since we know the shape of the $agent core.

        • +ap-abet yields the list of cards to resolve back to +mo, but also the $yoke, which is the new agent state for Gall's state.

      • +mo-abet finalizes.

What about an agent modification like |nuke? Let's see the lifecycle of that call.

• /gen/hood/nuke → %kiln-nuke

  • /lib/hood/kiln → +poke-nuke

    • [%pass /nuke %arvo %g [%nuke dude]]

    • /sys/vane/gall:

      • +call

        • +mo

        • +mo-nuke

          • +ap

            • +ap-abed

            • +ap-nuke is where the real work is done. Review it.

              • +ap-ingest

            • +ap-abet

        • +mo-abet

Clay actually governs which agents can run on a given desk. How does |install instigate this?

• See +goad in /sys/vane/clay.

There is additionally some plumbing around Gall receiving responses in +take for the vane (/sys) versus for an agent (/use).

We'll need to differentiate the Gall $signs and the Arvo $signs in a moment.

Scry interface

Gall brokers two kinds of scries: vane scries and agent scries.

In order to hit the vane-level endpoints, the beginning of the the spur (e.g. the path after the beak) must be a %$ empty element. For example:

.^(desk %gd /=acme=/$)
.^((set [=dude:gall live=?]) %ge /=base=/$)
.^((list path) %gt /=acme=//foo)

• %d: get desk of app

• %e: running apps

• %f: nonces of apps

• %n: get nonce of subscription

• %t: remote scry subpaths

• %u: check if installed

• %w: latest revision of path

• %x: remote scry file

• %z: hash of value at path

An agent scry has the form /=agent=/path/to/scry and may accept any care. %x cares must include a terminal mark in the path, however.

Gall also dispatches scries to agents' +on-peek arms. This takes place via +mo-peek→+ap-peek→+on-peek→+ap-mule-peek.

See +scry for details of both.

• “Gall Scry Reference”

Treaty and Docket

There are two ways to distribute nouns over Ames today:

1. Mark a desk |public and use Clay to directly synchronize desks.

2. Use the /app/treaty agent from %landscape to discover and install agents.

Landscape (formerly Grid) is formally a Tlon product. It primarily consists of two agents (and associated marks, libraries, etc.):

• /app/treaty handles publishing and advertising application desks.

• /app/docket handles retrieving, validating, and installing application desks.

Together they query a remote %treaty instance to +install a particular desk.

• Examine $alliance in /sur/treaty.

• Examine +publish:so and +watch:tr in /app/treaty.

• Examine +install:ch in /app/docket.

/app/treaty in particular has pretty tight construction and I commend its style to you.

• “Software Distribution · Userspace”

Updates

When Gall receives a newly rebuilt agent from Clay, it calls the gate produced by the +on-load arm of the new agent with the state extracted from the old agent. If there is a crash in any +on-load calls or in the handling of any effects they emit (which can include more agent activations), then the whole event crashes, canceling the commit. This effectively gives any agent the ability to abort a commit by crashing.

• Gall's +call arm receives a %load move with a noun of a core built by +ford:clay.

  • +mo-core is a handle to +mo because no +abet is needed.

    • +mo-load installs agents pretty mechanically, by simply +skimming over the %live agents.

      • +mo-receive-core checks whether the agent is running.

      • If it is, then +ap is invoked to update the agent:

        • +ap-abed

        • +ap-reinstall

          • +on-save:ap-agent-core

          • +ap-install is the install wrapper.

            • +ap-upgrade-state

              • +on-init:ap-agent-core

              • +on-load:ap-agent-core

        • +ap-abet

        • +mo-clear-queue flushes the blocked tasks pending for a new agent.

      • If it isn't, then we have to create it in +ap:

        • +ap-abed

        • +ap-upgrade-state

          • +on-init:ap-agent-core

          • +on-load:ap-agent-core

      • +mo-idle puts the agent to sleep if it's in the kill list (to be retired due to desk/bill).

  • +mo-abet finalizes moves and state changes.

Note that this is independent of Treaty and Docket once the remote desk has been installed.

Agent Wrappers as Core Modifiers

.An agent wrapper (like dbug) is a tool to wrap additional handlers around an agent core. These can wrap the internal Gall agent with new functionality by catching pokes and other standard moves, then re-dispatching to the normal arms if no special behavior is needed.

> :my-agent +dbug [%state 'value']

• Examine /lib/dbug.

  • In particular, how does the +on-poke wrapper arm work?

  • How does the +dbug generator work?

Using agent transformers to extend agents is a very nice conceptual pattern. But in practice, there are three pretty big problems with it:

1. You need to edit the agent code yourself.

2. Stateful transformers can break the agent.

3. The agent's world will also get transformed.

• “Debugging Wrapper”

• ~wicrum-wicrun, “Gall Middleware (Assembly 2022”

Dynamic Gall

An earlier incarnation of Gall, dynamic Gall, specified its arms in terms of the names of the move coming back or the mark of the poke coming in. (This is what I learned on, way back when.)

For instance, this agent was an earlier version of the egg timer app:

::  |start %egg
::  :egg ~s5
|%
+$  effect   (pair bone syscall)
+$  syscall  [%wait path @da]
--
|_  [bowl:gall ~]
++  poke-noun
  |=  t=@dr
  ^+  [*(list effect) +>.$]
  :_  +>.$  :_  ~
  [ost %wait /egg-timer (add now t)]
++  wake
  |=  [=wire error=(unit tang)]
  ^+  [*(list effect) +>.$]
  ~&  "Timer went off!"
  [~ +>.$]
--

Exercises

• Your assignment is to produce a minimalist Gall-like agent handler: a userspace framework for producing “toy” agent-like applications. Let's call them “scamps”.

  The scamp's state is defined in a $state block at the top of its file, e.g.:

  +$  state
    $:  scores=(list @)
        hi-score=@
    ==

  Scamps do not support state upgrades, so no version tag is provided.

  A scamp requires the following arms for the developer:

  |%
  ++  on-init
  ++  on-poke
  ++  on-peek
  --

  You should be able to poke and peek into a scamp. It has no subscription model.

  A scamp specification file is NOT implicitly chained with a running => tisgal. Compose explicitly.

As a final aside, I believe that building an %aqua/%pyro testbed along the lines of Gall should also be feasible for you at this point.