# What is Urbit?

Urbit docs site. Urbit ID, Urbit OS, Hoon, Nock, and self-hosting.

Urbit is a simple personal server (Urbit OS), a peer-to-peer network of those servers, and a decentralized identity standard (Urbit ID) for that network and beyond.

You can [get on Urbit](/get-on-urbit) in a few clicks with cloud hosting or host your server on your own machine at home.

## What is this site?

The Urbit docs are for developers and self-hosted users. Developers can find [courses](/build-on-urbit/environment), guides, and reference material for [Urbit ID](/urbit-id/what-is-urbit-id) and [Urbit OS](/urbit-os/what-is-urbit-os). The [manual](/user-manual/contents) section covers everything that users need to run their own server.

## Developer documentation

* [Build on Urbit](/build-on-urbit/environment) - Courses and tutorials for building on Urbit.
* [Urbit ID](/urbit-id/what-is-urbit-id) - Everything there is to know about Urbit ID, from high-level concepts to smart contract documentation.
* [Urbit OS](/urbit-os/what-is-urbit-os) - Documentation for Urbit OS, including step-by-step guides and kernel module APIs.
* [Hoon](/hoon/why-hoon) - Guides and reference material for Urbit's high-level programming language.
* [Nock](/nock/what-is-nock) - Guides and reference material for Urbit's low-level assembly language.

## User documentation

* [Manual](/user-manual/contents) - How to run your own Urbit, manage your Urbit ID, and more.


# Get on Urbit

Step-by-step guide to getting started with Urbit, including obtaining an Urbit ID, downloading they keyfile, installing the runtime, and booting your first Urbit OS server.

Urbit is two things: [Urbit ID](/urbit-id/what-is-urbit-id), and [Urbit OS](/urbit-os/what-is-urbit-os). To get on the network, you'll need both.

This guide will walk through the steps of getting an Urbit ID, downloading its key, installing the Urbit runtime, and using that to boot up your Urbit OS.

{% hint style="warning" %}

### Get on Urbit faster

If you want a quicker and easier way to get on Urbit, you can skip this guide and use a hosting provider instead. Tlon offers a free Urbit ID and hosting in the cloud that only takes a few clicks to get up and running, [available here](https://join.tlon.io/0v3.r87kb.fjpft.3k7b5.pbsr5.5em17).
{% endhint %}

**Requirements**

* Ability to use a terminal / command line. We'll go through everything you need to do step by step.
* If you want to buy an Urbit ID you will need either...
  * A credit card
  * Some ETH and familiarity with Ethereum wallets, gas fees, security best practices, etc.
* A computer or server running macOS or Linux, with at least 4GB of RAM and around 4GB of disk space. (You can usually get away with 2GB of RAM plus a swapfile and less disk space, but this might become a problem in the future as your Urbit grows.)

(If you want to run Urbit on a Windows computer you should be able to do so with Windows Subsystem for Linux (WSL), but that's outside the scope of this guide.)

{% hint style="info" %}
If you're on mobile, open this page on a desktop and complete the following instructions on that.
{% endhint %}

## 1. Get an Urbit ID <a href="#get-an-urbit-id" id="get-an-urbit-id"></a>

Every Urbit OS server is made unique by its Urbit ID, which others can use to reach you on the network. There are [five ranks of Urbit ID](/user-manual/id/get-id#types-of-id), but the one an ordinary user needs is a planet, which has a four-syllable name like "\~sampel-palnet". Unless you know someone who can gift you one, or you want to get one from a cloud hosting provider like Tlon, you'll need to buy one.

{% hint style="info" %}
If you don't want to buy anything at this stage, you can get on the network with a free, disposable ID called a comet. Comets can do almost everything a planet can do, but select groups on the network may refuse entry to prevent spam and abuse.

If you just want to run an Urbit locally for testing/development purposes, without networking, you can boot a "fake ship" that uses an Urbit ID you don't actually own.

To boot a comet or fake ship, [skip straight to Step 3](#get-the-urbit-runtime).

You can also get a planet for free with [cloud hosting](#get-on-urbit-faster).
{% endhint %}

Here are the best places to buy planets:

| Layer   | Market                                                                                                                                        | Description                                        |
| ------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| Layer 1 | OpenSea [Urbit ID: Planet](https://opensea.io/collection/urbit-id-planet) and [Urbit ID](https://opensea.io/collection/urbit-id) collections. | The largest NFT marketplace. Accepts ETH.          |
| Layer 2 | [Pocwet](https://store.pocwet.com)                                                                                                            | Third-party L2 planet store. Accepts credit cards. |

Originally, all Urbit IDs were ERC-721 NFTs on Ethereum. In 2021, Tlon [introduced a Layer 2 solution](https://urbit.org/blog/rollups) to reduce transaction costs on Ethereum. This means there are two places where ownership of your Urbit ID could be recorded:

* **Layer 1**: These ordinary NFTs can traded on regular NFT marketplaces like [OpenSea](https://opensea.io/), and other smart contracts can interact with them. It'll cost you a bit of [gas](https://ethereum.org/en/gas/#what-is-gas) if you need to do an onchain action like a [factory reset](/user-manual/id/guide-to-resets) or [changing your networking keys](/user-manual/id/hd-wallet). Ownership of these IDs is recorded in Urbit's [Azimuth.eth](/urbit-id/azimuth-eth) smart contract.
* **Layer 2**: With a Layer 2 planet you pay no transaction fee for onchain actions like factory resets and key changes. (Tlon runs the L2 system you'll use by default and they pay the very small L2 fees.) However, *normal Ethereum smart contracts and NFT marketplaces cannot see or interact with L2 IDs.* While there have been proposals to offer an L2 -> L1 bridge that would turn L2 IDs into ordinary NFTs on Ethereum, you should not assume this will come to pass when making a purchase decision. Ownership of your Urbit ID is not recorded in the Azimuth smart contract, but every onchain transaction associated with your ID is [recorded on Ethereum](/urbit-id/l2).

In either case, your ownership of your planet is [secured by your private key](/urbit-id/hd-wallet). Only you can access your Urbit ID, only you can boot up an Urbit server with your ID. Messages attributed to your ID will only ever come from your Urbit. Nobody can take your ID away from you unless they know your private key, and nobody should know your private key. **Your Urbit is yours, forever.**

## 2. Get your keyfile <a href="#get-your-keyfile" id="get-your-keyfile"></a>

Once you've got an Urbit ID, the next step is to download its private [networking key](/urbit-id/hd-wallet) (contained in a "keyfile") so you can boot it up. This process can vary depending on how you obtained your ID and where it's stored.

{% tabs %}
{% tab title="In a wallet" %}
If you got an L1 planet from somewhere like OpenSea, it was likely transferred directly to your Ethereum wallet. Here are the steps to get your keyfile:

1. Go to [bridge.urbit.org](https://bridge.urbit.org).
2. Conect your wallet.
3. Click on your planet, which should be listed there. (If you only have one Urbit ID in your wallet, click "Set Network Keys" on the popup and go to Step 5.)
4. Go to the "OS" section.
5. Click "Initialize" next to "Network Keys". (See the note below if it says something else.)
6. Make sure you have a little ETH to pay the transaction fee, then click on "Set Network Keys".
7. Click "Send Transaction".
8. Approve the transaction in your wallet.
9. Wait until the transaction completes and it says "Network Keys have been set" in [bridge.urbit.org](https://bridge.urbit.org).
10. Click "Download Keyfile". Ignore the prompt about [`|rekey`](/user-manual/os/dojo-tools#rekey).
11. A file called something like `sampel-palnet-1.key` will have been downloaded. Don't lose it; you'll need it for the next step.

{% hint style="info" %}

### Don't see "Initialize"?

If the options next to "Networking Keys" are "Reset" and "View" rather than "Initialize", it means your ID's previous owner at one point generated keys for their planet. In that case, click on "Reset", tick the "Factory Reset" box, and continue from step 6 above.
{% endhint %}
{% endtab %}

{% tab title="Invite link" %}
If you bought an L2 planet, you might have received an invite link that looks like `https://bridge.urbit.org/#foshec-moplec-haddem-poddun-sampel-palnet`. Here are the steps to get your keyfile:

1. Open the link and complete the steps as prompted.
2. At one point, there'll be an option to "Download Backup (Passport)". Click on that, and it'll download a file named something like like `sampel-palnet-passport.zip`.
3. Once you see the a message like "Congratulations, \~sampel-palnet is now yours", you can close the window and return to this page.
4. Unzip the `sampel-palnet-passport.zip` file. It will contain your keyfile, titled something like `sampel-palnet-1.key`.
5. Store your Master Ticket and [Management Proxy](/user-manual/id/proxies#types-of-proxies) somewhere safe. As long as you know your Master Ticket (e.g. "\~sampel-ticlyt-migfun-falmel"), you can always log into [bridge.urbit.org](https://bridge.urbit.org) and retrieve all the other information you need.

{% hint style="info" %}

### Already have a planet?

If you already claimed this planet at some point and forgot to download the passport, you'll need to go and download your keyfile from [bridge.urbit.org](https://bridge.urbit.org):

1. Go to [bridge.urbit.org](https://bridge.urbit.org).
2. Click the "Master Ticket" login option.
3. Enter the planet name and Master Ticket code, and click "Login".
4. Go to the "OS" section.
5. Click on "Download Keyfile" and it'll download a file with a name like `sampel-palnet-2.key`.
   {% endhint %}
   {% endtab %}

{% tab title="Passport / Master Ticket" %}
If you bought an L2 planet you might have received a "passport" or "Master Ticket". (As well as a file titled Master Ticket, there may be a file titled "[Management Proxy](/user-manual/id/proxies#types-of-proxies)" which you can ignore for now.) Here are the steps to get your keyfile:

1. Go to [bridge.urbit.org](https://bridge.urbit.org).
2. Click the "Master Ticket" login option.
3. Enter the Urbit ID (or "Point") and the Master Ticket (e.g. "\~sampel-ticlyt-migfun-falmel") from the file titled Master Ticket.
4. Click the "OS" section and click "Download Keyfile". This file should be named something like `sampel-palnet-1.key`.
5. Store your Master Ticket and Management Proxy somewhere safe. As long as you know your Master Ticket, you can always log into [bridge.urbit.org](https://bridge.urbit.org) and retrieve all the other information you need.
   {% endtab %}
   {% endtabs %}

## 3. Get the Urbit runtime <a href="#get-the-urbit-runtime" id="get-the-urbit-runtime"></a>

To boot up a new Urbit, you'll need the Urbit runtime. There are four pre-built versions of the runtime available, one for each platform we support. Pick the one you're on and copy/paste the provided `curl` command into your terminal.

{% tabs %}
{% tab title="macOS (Apple Silicon)" %}
Open a terminal and run:

```sh
curl -L https://urbit.org/install/macos-aarch64/latest | tar xzk -s '/.*/urbit/'
```

{% endtab %}

{% tab title="macOS (Intel)" %}
Open a terminal and run:

```sh
curl -L https://urbit.org/install/macos-x86_64/latest | tar xzk -s '/.*/urbit/'
```

{% endtab %}

{% tab title="Linux (x86\_64)" %}
Open a terminal and run:

```sh
curl -L https://urbit.org/install/linux-x86_64/latest | tar xzk --transform='s/.*/urbit/g'
```

{% endtab %}

{% tab title="Linux (aarch64)" %}
Open a terminal and run:

```sh
curl -L https://urbit.org/install/linux-aarch64/latest | tar xzk --transform='s/.*/urbit/g'
```

{% endtab %}
{% endtabs %}

## 4. Boot up your Urbit <a href="#boot-up-your-urbit" id="boot-up-your-urbit"></a>

{% tabs %}
{% tab title="Boot a planet" %}
Having acquired a planet and downloaded its keyfile, you can now boot up your Urbit.

The easiest way to run an Urbit is inside a `screen` session. `screen` is installed on macOS and Linux by default; it lets you "detach" from terminal sessions, leave them running in the background, then "attach" to them again later.

To start a new `screen` session and name it `urbit`, run the following command in your terminal:

```sh
screen -S urbit
```

You are now inside a `screen` session. You can boot your Urbit with the command below, replacing `sampel-palnet` with your actual planet name, and `/path/to/sampel-palnet-1.key` with the path to the keyfile you downloaded previously. (If you're on macOS, it's probably `~/Downloads/sampel-palnet-1.key`.)

```sh
./urbit -w sampel-palnet -k /path/to/sampel-palnet-1.key
```

(Very rarely this can result in an error message `pre-boot error: %not-keyed`. If you see this, check your ETH wallet in Etherscan for a "Configure Keys" transaction. If you've confirmed your planet's keys have been set, leave it for a while and try again.)

Your planet will begin to boot up, which will take a few minutes. You'll know your planet has booted when you see something like this in the terminal:

```
~sampel-palnet:dojo>
```

This is the Dojo, Urbit's command-line interface.

For the moment, shut the ship down again by typing `|exit` or pressing `Ctrl+D`. If you see a loading spinner like "<\<behn>>", you won't be able to type until that goes away. Your planet might be installing a couple of apps, but since Urbit OS has no concept of RAM you can safely turn it off at any time without worrying about breaking it.

{% hint style="info" %}

### Linux default port

Linux won't let Urbit's web server bind port 80, and it will instead default to port 8080. If you want it to bind port 80, you'll need to do the following in the terminal:

```sh
# if you're on ubuntu and don't already have setcap
sudo apt-get install libcap2-bin
# replace sampel-palnet with the actual folder name
sudo setcap 'cap_net_bind_service=+ep' sampel-palnet/.run
```

{% endhint %}

For ease of portability the Urbit runtime has made a copy of itself in the `sampel-palnet` folder, so you don't need the separate `urbit` executable anymore. Delete it by typing `rm urbit`.

At this point you can also delete your keyfile because it can never be used again.

Spin up your planet again by running `sampel-palnet/.run`, and you'll be back at the Dojo.

Next, you'll want to get the web login code so you can login to your Urbit's web interface in a browser.

To do so, type `+code` in the Dojo and press `Return`. It'll give you a code that looks like `lidlut-tabwed-pillex-ridrup`. Highlight that and copy it.

{% hint style="info" %}

### Save your code

If you want, you can save your `+code` in a password manager or write it down somewhere safe. Your `+code` isn't your private key, just a web login password. If you think your `+code` has been compromised, you can just [reset it in the Dojo](/user-manual/os/dojo-tools#code) to log out of all browser sessions in your Urbit.
{% endhint %}

Detach from this `screen` session by hitting `Ctrl+A`, then hitting `D`. It should say something like `[detached from 1819892.urbit]`. You can now close or quit the terminal entirely; your Urbit is still running in the `screen` session.

If you need to get back to the Dojo again in the future, just open a terminal and run `screen -r urbit` to reattach to the `urbit` session.
{% endtab %}

{% tab title="Mine a comet" %}
Comets are free, disposable Urbit IDs. Comets are trivial to create, so we use them as anonymous IDs, but to boot an Urbit server with a comet you'll have to "mine" one.

The easiest way to run an Urbit is inside a `screen` session. `screen` is installed on macOS and Linux by default; it lets you "detach" from terminal sessions, leave them running in the background, then "attach" to them again later.

To start a new `screen` session and name it `urbit`, run the following command in your terminal:

```sh
screen -S urbit
```

Now that you're in a `screen` session, you can mine a new comet with the command below. `mycomet` is just the name of the folder, so you can change it to whatever you like.

```sh
./urbit -c mycomet
```

It might take a few minutes to boot up and mine a comet. You'll know your comet has booted when you see something like this in the terminal:

```
~sampel_litzod:dojo>
```

This is the Dojo, Urbit's command-line interface.

For the moment, shut the ship down again by typing `|exit` or pressing `Ctrl+D`. If you see a loading spinner like "<\<behn>>", you won't be able to type until that goes away.

{% hint style="info" %}

### Linux default port

Linux won't let Urbit's web server bind port 80, and it will instead default to port 8080. If you want it to bind port 80, you'll need to do the following in the terminal:

```sh
# if you're on ubuntu and don't already have setcap
sudo apt-get install libcap2-bin
# replace mycomet with the actual folder name
sudo setcap 'cap_net_bind_service=+ep' mycomet/.run
```

{% endhint %}

For ease of portability the Urbit runtime has made a copy of itself in the `mycomet` folder, so you don't need the separate `urbit` executable anymore. Delete it by typing `rm urbit`.

Spin up your comet again by running `mycomet/.run`, and you'll be back at the Dojo.

Comets don't receive software updates by default. If you intend on running this comet for a while and you want to receive updates, run `|ota (sein:title our now our)` in the Dojo.

Next, you'll want to get the web login code so you can login to your Urbit's web interface in a browser.

To do so, type `+code` in the Dojo and press `Return`. It'll give you a code that looks like `lidlut-tabwed-pillex-ridrup`. Highlight that and copy it.

{% hint style="info" %}

### Save your code

If you want, you can save your `+code` in a password manager or write it down somewhere safe. Your `+code` isn't your private key, just a web login password. If you think your `+code` has been compromised, you can just [reset it in the Dojo](/user-manual/os/dojo-tools#code) to log out of all browser sessions in your Urbit.
{% endhint %}

Detach from this `screen` session by hitting `Ctrl+A`, then hitting `D`. It should say something like `[detached from 1819892.urbit]`. You can now close or quit the terminal entirely; your Urbit is still running in the `screen` session.

If you need to get back to the Dojo again in the future, just open a terminal and run `screen -r urbit` to reattach to the `urbit` session.
{% endtab %}

{% tab title="Boot a fake ship" %}
Fake ships are for development purposes only and cannot connect to the live network.

Run the `urbit` executable you previously downloaded with the `-F` flag to boot a new fake ship. You can specify any identity you want. Most people use the galaxy \~zod.

```sh
./urbit -F zod
```

This will take a few minutes. Once it's done, you'll have a responsive prompt that looks like this:

```
~zod:dojo> 
```

You can shut the fake ship down by typing `|exit` or pressing `Ctrl+D`.

For ease of portability the Urbit runtime has made a copy of itself in the `zod` folder, so you don't need the separate `urbit` executable anymore. Delete it by typing `rm urbit`.

Spin up your fake ship again by running `zod/.run`, and you'll be back at the Dojo.

Next, you may want to get the web login code so you can login to your Urbit's web interface in a browser. To do so, type `+code` in the Dojo and press `Return`. It'll give you a code that looks like `lidlut-tabwed-pillex-ridrup`. Highlight that and copy it.

Fake ships are for developers. If you want to learn more about programming, app development, or core development on Urbit, look at our [courses](/build-on-urbit/environment) section to get started.
{% endtab %}
{% endtabs %}

## 5. Login to Landscape <a href="#login-to-landscape" id="login-to-landscape"></a>

With your Urbit now running and your web login code copied, you can open a browser and go to `http://localhost`. (If that doesn't work, try `http://localhost:8080`.) You should see with your Urbit's login screen. Paste in the code you got from running `+code` in the Dojo (it looks like `lidlut-tabwed-pillex-ridrup`) and click "Continue". You'll now be in your Landscape homescreen.

To join your first group, you can open the Tlon app by clicking on its tile, then click the `+` in the sidebar and click "Join a group". Paste in `~halbex-palheb/uf-public`, the Urbit Foundation's official public group, and click "Go", then click "Join group".

Have a look at the Urbit Foundation group for a list of more groups on the network to join.

If you'd like to see what apps are available other than Tlon, click on the "Get Urbit Apps" button at the top of the Landscape homescreen. There are a few suggestions listed there. If you know an app's publisher (e.g. \~bitdeg) or "shortcode" (e.g. `~bitdeg/hits`), you can type that in the searchbar at the top of the "Get Urbit Apps" menu.


# Style Guide

Urbit docs style guide.

## Style & Tone of Voice

The Urbit docs are written for several audiences: new Urbit developers, seasoned Urbit developers, potential investors, large language models, and more. Different sections of the site target different audiences, so they should be written in a tone of voice appropriate for that audience.

### The docs in general

* Write in a clear, formal tone with minimal color.
* Use American English spelling.
* Use the Oxford comma.
* Urbit IDs should be referred to as “Urbit IDs” outside of a Hoon context where it would make more sense to refer to them as “`@p`s”.
* Urbit IDs should not be monospaced or otherwise specially formatted by default, but this is fine if the situation really calls for it.
* Call it "Urbit OS" in top-level landing pages and the like unless you're actually talking about `/sys/arvo.hoon`.
* Avoid em-dashes — they look weird in the monospace code editors we use to edit the docs. They can also be mistaken as a telltale sign of LLM-generated text. Don’t use `--` or `&mdash;` in their place; use semicolons to break up two independent but related clauses and use a pair of commas (or round brackets) as parentheses.
* Try to avoid beginning a paragraph with monospaced text; whatever text you’d put in front of it would usually help the reader follow your thinking. Starting a bullet-point with monospaced text is fine.

### Landing pages, overview pages

* Overview/landing pages should be accessible to a non-technical audience to help them find their way around the site.
* Avoid Urbit-specific jargon wherever possible. A new developer should be able to do a "breadth-first" skim of these navigation pages and roughly understand the contents beneath those pages in the filetree.

### Guides, courses, tutorials

* Guides and courses should be accessible to new Urbit developers.
* Urbit has a strong track record of attracting non- or somewhat-technical people and turning them into productive developers. As such, references to big-brain computer science concepts should be avoided where possible, and contextualized where unavoidable.
  * Comparisons to common programming concepts like functions, Promises, etc. where appropriate are *strongly encouraged*.
* Err on the side of formatting common kernelspace structures and molds as `$path`, `$mark`, `+list` etc., where in other sections of the docs this would be discouraged.

### Reference material

* Reference pages may assume a high level of familiarity with the system.

## Formatting

The purpose of formatting is to disambiguate as quickly and clearly as possible: picoseconds of confusion add up to a poor reading experience, especially when the reader is unfamiliar with the concepts being discussed.

### Formatting prose

* Hoon arms are always `+foo`, never `foo` or `++foo`.
* Hoon paths are always `/foo/bar`.
* Named wings in the Hoon subject are always `.foo`, `.foo.bar`, etc.
  * Occassionally you might want to refer to the name of the wing as distinct from the value of that named wing. In this case, use backticks without prepending a `.`. (For example, the name of the wing `.foo` is `foo`.)
  * Where the name of a wing is also the type of its data, e.g. `=foo` or `foo=foo`, refer to it as `.foo` rather than `$foo`.
* Hoon structures are almost always `$foo`, and never `foo` or `+$foo`.
  * Use `%foo` when referring to the actual `%foo` mark, not when referring to the `$foo` datatype it converts to, nor the `/mar/foo.hoon` file that defines it.
  * When a kernelspace Hoon structure is referred to so many times throughout the docs, or this page, that it would be annoying to call it `$foo` every time, just call it “foo”.
    * Example: *The path ends with a mark.*
* Desks are always `%base`, never `base`.
  * Desks may be referred to as `/base` when referring to the Unix directory.
* Unix directories are always `/foo`, never `foo`.
* Files in a desk are generally referred to as `/app/foo.hoon`, `/lib/foo.hoon`, `/mar/foo.hoon`, `/sur/foo.hoon`, and so on.
* Library files in a desk are always introduced as `/lib/foo.hoon` and never `/lib/foo` or `lib/foo`.
  * When referring to a library like strandio by name, just call it "strandio" rather than `strandio`.
* When writing about non-Hoon languages, use whatever conventions of that language’s first- and third-party documentation that exist to format inline symbols. For example, JavaScript functions and classes should always be `foo()`, never `foo`.
* Vanes like Gall are always Gall, never %gall, `%gall`, or `gall`.
* Keyboard commands should be formatted like `Ctrl+D`.

### Formatting code blocks

* Disambiguate between Unix and Dojo code blocks as clearly as possible.
  * Use `>` and `$` prompts to differentiate between Dojo and Unix input.
  * You have the option of using the code block’s `title` metadata to specify “Dojo” or “Unix” if necessary.
* If the code block is part of a file, always include the name of the file in the code block’s metadata.
  * If the file is in a desk, include the parent directory (e.g. `/app/foo.hoon`).
* Hoon code blocks should never be wrapped.

## Hoon Style

Hoon in the docs should generally follow the [Hoon Style Guide](https://docs.urbit.org/hoon/style), with some extra considerations for new developers and for limited screen real estate:

* A line of Hoon should be no more than 80 columns wide.
* Hoon should strive to be no more than 56 columns wide.
* Backstep as early and often as you reasonably can.


# Contents

Overview of development resources for building on Urbit, including courses for learning Hoon, app development guides, core development, runtime development, and tools for Urbit developers.

Courses and guides for developers looking to learn how to build on Urbit.

* [Environment Setup](/build-on-urbit/environment) - How to set up your developer environment.
* [Hoon School](/build-on-urbit/hoon-school) - Learn the basics of Hoon, Urbit's high-level programming language.
* [App School I](/build-on-urbit/app-school) - A detailed guide to Gall, Urbit's app framework.
* [App School II](/build-on-urbit/app-school-full-stack) - A walkthrough for building an Urbit app with a Gall backend and a React frontend.
* [Core Academy](/build-on-urbit/core-academy) - Guide to Urbit OS and the runtime.
* [Tools](/build-on-urbit/tools) - Useful third-party libraries.
* [Userspace](/build-on-urbit/userspace) - Misc. tutorials and examples for app development on Urbit.
* [Runtime](/build-on-urbit/runtime) - A guide to working in the Urbit runtime.


# Environment Setup

Developer environment setup guide for Urbit development, including text editor configuration, development ship setup, and project organization best practices.

This guide covers best practices for preparing your environment to develop within the Urbit ecosystem.

## Text editors <a href="#text-editors" id="text-editors"></a>

A variety of plugins have been built to provide support for the Hoon language in different text editors. These are listed below.

**Note:** The hoon compiler expects Unix-style line endings (LF) and will fail to parse Windows-style line endings (CRLF). Make sure your editor is set to use LF for line endings, especially if you're developing on Windows.

#### Visual Studio Code

Visual Studio Code is free and open-source and runs on all major operating systems. It is available [here](https://code.visualstudio.com/). Hoon support may be acquired in the Extensions menu within the editor by searching for `Hoon`.

#### Sublime Text

Sublime Text is closed-source, but may be downloaded for free and there is no enforced time limit for evaluation. It runs on all major operating systems. It is available [here](https://www.sublimetext.com/).

#### Emacs

Emacs is free and open-source and runs on all major operating systems. It is available [here](https://www.gnu.org/software/emacs/). Hoon support is available with [hoon-mode.el](https://github.com/urbit/hoon-mode.el).

#### Vim

Vim is free and open-source and runs on all major operating systems. It is available [here](https://www.vim.org/). Hoon support is available with [hoon.vim](https://github.com/urbit/hoon.vim).

## Development ships <a href="#development-ships" id="development-ships"></a>

### Creating a fake ship <a href="#creating-a-fake-ship" id="creating-a-fake-ship"></a>

To do work with Hoon, we recommended using a "fake" ship — one that's not connected to the network.

Because such a ship has no presence on the network, you don't need an Azimuth identity. You just need to have [installed the Urbit binary](/get-on-urbit#get-the-urbit-runtime).

To create a fake ship named \~zod, run the command below. You can replace `zod` with any valid Urbit ID.

```
./urbit -F zod
```

This will take a couple of minutes, during which you should see a block of boot messages, starting with the Urbit version number.

### Fake ship networking <a href="#fake-ship-networking" id="fake-ship-networking"></a>

Fake ships on the same machine can automatically talk to one another. Having created a fake \~zod, you can create a fake \~bus the same way:

```
./urbit -F bus
```

Now in the fake \~bus's dojo, try:

```
> |hi ~bus
>=
hi ~bus successful
```

### Local Networking <a href="#local-networking" id="local-networking"></a>

Fake ships run on their own network using fake keys and do not communicate with live-net ships in any way. Multiple fake ships running on the same machine can network with each other. They don't need to have a "realistic" chain of fake sponsors to communicate.

### Faster fake ship booting <a href="#faster-fake-ship-booting" id="faster-fake-ship-booting"></a>

While working with Hoon, you'll often want to delete an old fake ship and recreate a fresh one. Rather than having to wait a few minutes for the fresh ship to be initialized, you can instead create a backup copy of a fake ship. That way you can just delete the current copy, replace it with the backup, and reboot in a matter of seconds.

To do this, boot a fresh fake ship like usual, but with a different name:

```
./urbit -F zod -c zod.new
```

Once it's finished booting, it's a good idea to mount its desks so you don't have to do it again each time. In the Dojo:

```
> |mount %base
>=
> |mount %landscape
>=
```

Next, shut the ship down with `Ctrl+D`. Then, copy the pier and start using the copy instead:

```
cp -r zod.new zod
./urbit zod
```

Now whenever you want a new fake \~zod, you can just shut it down and do:

```
rm -r zod
cp -r zod.new zod
./urbit zod
```

## Working with desks <a href="#working-with-desks" id="working-with-desks"></a>

If you're just working in the dojo or experimenting with generators, committing to the `%base` desk on a fake ship is fine. If you're working on a Gall agent or developing a desk for distribution, you'll most likely want to work on a separate desk and it's slightly more complicated.

### Mount a desk <a href="#mount-a-desk" id="mount-a-desk"></a>

To mount a desk to Unix so you can add files, you just need to run the `|mount` command in the dojo and specify the name of the desk to mount:

```
|mount %base
```

The desk will now appear in the root of your pier (zod in this case):

```
zod
└── base
```

You can unmount it again by running the `|unmount` command in the dojo:

```
|unmount %base
```

### Create a new desk <a href="#create-a-new-desk" id="create-a-new-desk"></a>

To create a new desk, you can just run:

```
|new-desk %mydesk
```

If you run `|mount %mydesk`, you'll see a `/mydesk` directory in your pier with the following files:

```
mydesk
├── mar
│   ├── hoon.hoon
│   ├── kelvin.hoon
│   ├── noun.hoon
│   └── txt.hoon
└── sys.kelvin
```

The mark files in `/mar` are for handling some basic filetypes, and `sys.kelvin` specifies which kernel version(s) the desk is compatible with. The `|new-desk` generator populates `sys.kelvin` with the current kernel version like `[%zuse 408]`.

You can delete these files, copy in your own and run `|commit %mydesk` in the Dojo.

## `/base-dev`, `/desk-dev` folders

The files included by `|new-desk` are the only the bare minimum necessary to mount the desk. If you're building a full app, you'll almost certainly need a number of mark files and libraries from the `%base` and `%landscape` desks. If your app is going to talk to other apps on your ship, you'll likely need files for those, too.

To make these dependencies easier, the necessary dependencies are included in `/base-dev` and `/desk-dev` folders.

* The [urbit/urbit repo](https://github.com/urbit/urbit) includes [`/base-dev`](https://github.com/urbit/urbit/tree/develop/pkg/base-dev) with the files necessary for interacting with agents on the `%base` desk, among other useful marks and libraries.
* The [tloncorp/landscape repo](https://github.com/tloncorp/landscape) includes [`/desk-dev`](https://github.com/tloncorp/landscape/tree/develop/desk-dev) with marks and libraries for building Landscape apps.

You can clone these repos and copy the contents of their `/*-dev` folders into your own projects. A better alternative is to use the [desk skeleton](#desk-skeleton) described below.

## Project organization <a href="#project-organization" id="project-organization"></a>

When you're developing a desk, it's best to structure your working directory with the same hierarchy as a real desk. For example, `~/project-repo/desk` might look like:

```
desk
├── app
│   └── foo.hoon
├── desk.bill
├── desk.docket-0
├── lib
│   └── foo.hoon
├── mar
│   └── foo
│       ├── action.hoon
│       └── update.hoon
├── sur
│   └── foo.hoon
└── sys.kelvin

```

That way, whenever you want to test your changes, you can just copy it across to your pier like:

```
cp -ruv desk/* /path/to/fake/zod/mydesk
```

And then just commit it in the dojo:

```
|commit %mydesk
```

If you're using [dev folders](#dev-folders) as a base, it's best to keep those files separate from your own code.

## Desk skeleton <a href="#desk-skeleton" id="desk-skeleton"></a>

Dependency management can be inconvenient when building Urbit apps. If you manually copy in `/base-dev` and Landscape's `/desk-dev`, it can be annoying to update them when a new kernel version is released.

For this reason, the Urbit Foundation has published a desk skeleton to use for new projects. It includes a couple of tools to make code organization and dependency management easier.

You can git clone the repo from [urbit/desk-skeleton](https://github.com/urbit/desk-skeleton):

```sh
git clone https://github.com/urbit/desk-skeleton.git my-project
```

Then you can create a `my-project` repo on your Github, set the upstream to that instead, and push it:

```sh
cd my-project
git remote set-url origin https://github.com/YOUR-GITHUB/my-project.git
git push
```

The desk skeleton contains a `/desk` folder with an extremely simple example app. You can delete the `/app/example.hoon` file, add your own files, list your own agents in `desk.bill`, and add your own Docket configuration to `desk.docket-0`.

You can optionally create a separate `/desk-dev` folder for any dependencies another developer would need, or you can just put everything in `/desk`.

You'll notice `/desk` doesn't include `/base-dev` or Landscape's `/desk-dev` files. Instead, they're configured in `peru.yaml` and pulled in by [peru](https://github.com/buildinspace/peru). Peru is a Python app for managing dependencies. You can install it from:

* [pip](https://pypi.org/project/pip/): `pip install peru`
* [Homebrew](https://brew.sh/): `brew install peru`
* The [AUR](https://aur.archlinux.org/packages/peru) if you use Arch Linux

With `peru` installed on your system, you simply need to run `./build.sh`. It'll create a `/dist` folder, copy in all the files from the `/desk` folder (and `/desk-dev` if it exists), and copy in the files from `/base-dev` and Landscape's `/desk-dev` on Github. The `/dist` folder will now contain all the necessary files, and you can copy them across to a mounted desk on a fake ship and `|commit` them in the Dojo.

The `./build.sh` script can be run again any time you make changes.

If there's a kernel update down the line and you need to update the `/base-dev` and Landscape dependencies, you just need to run `peru reup`. This will update `peru.yaml` to use the latest commit on the `master` branch of the `urbit/urbit` and `tloncorp/landscape` GitHub repos. Then you can just run `./build.sh` again.

The default `peru.yaml` only includes the two dependencies mentioned, but you can easily add any others you need. Refer to the [peru documentation](https://github.com/buildinspace/peru) configuration details.


# Hoon School

Beginner's course for learning Hoon, Urbit's functional programming language, with explanations, exercises, and tutorials for developers at all levels.

Hoon School is designed to teach you Hoon without assuming you have an extensive programming background. You should be able to following most of it even if you have no programming experience at all yet, though of course experience helps. We strongly encourage you to try out all the examples of each lesson. Hoon School is meant for the beginner, but it's not meant to be skimmed.

Each lesson consists of:

* **Explanations**, which are prose-heavy commentary on the Hoon fundamentals.
* **Exercises**, which challenge you to clarify or expand your own understanding in practice.
* **Tutorials**, which are line-by-line commentary on example programs.

## Why Hoon? <a href="#why-hoon" id="why-hoon"></a>

The short version is that Hoon uses Urbit's provisions and protocols to enable very fast application development with shared primitives, sensible affordances, and straightforward distribution.

Urbit consists of an identity protocol (Azimuth, or “Urbit ID”) and a system protocol (Arvo, or “Urbit OS”). These two parts work hand-in-hand to build your hundred-year computer.

**Urbit ID (Azimuth)** is a general-purpose public-key infrastructure (PKI) on the Ethereum blockchain, used as a platform for Urbit identities. It provides a system of scarce and immutable identities which are cryptographically secure.

**Urbit OS (Arvo)** is an operating system which provides the software for the personal server platform that constitutes the day-to-day usage of Urbit. Arvo works over a [peer-to-peer](https://en.wikipedia.org/wiki/Peer-to-peer), [end-to-end-encrypted](https://en.wikipedia.org/wiki/End-to-end_encryption) network to interact with other Urbit ships (or unique instances).

Arvo is an axiomatic operating system which restricts itself to pure mathematical functions, making it [deterministic](https://en.wikipedia.org/wiki/Deterministic_algorithm) and [functional-as-in-programming](https://en.wikipedia.org/wiki/Functional_programming). Such strong guarantees require an operating protocol, the [Nock virtual machine](/nock/definition), which will be persistent across hardware changes and always provide an upgrade path for necessary changes.

It's hard to write a purely functional operating system on hardware which doesn't make such guarantees. So Urbit OS uses a new language, Hoon, which compiles to Nock and hews to the necessary conceptual models for a platform like Urbit. [The Hoon overview](/hoon/why-hoon) covers more of the high-level design decisions behind the language.

Hoon School introduces and explains the fundamental concepts you need in order to understand Hoon's semantics. It then introduces a number of key examples and higher-order abstractions which will make you a more fluent Hoon programmer.

Once you have completed Hoon School, you should work through [App School](/build-on-urbit/app-school) to learn how to build full applications on Urbit.

## Environment Setup <a href="#environment-setup" id="environment-setup"></a>

An Urbit ship is a particular realization of an *identity* and an *event log* (state). Both of these are necessary.

Since live network identities ("live ships") are finite, scarce, and valuable, most developers prefer to write new code using fake identities ("fakeships"). A fakeship is also different from a comet, which is an unkeyed live ship.

Two fakeships can communicate with each other on the same machine, but have no awareness of the broader Urbit network. We won't need to use this capability in Hoon School Live, but it will be helpful later when you start developing networked apps.

Before beginning, you'll need to get a development ship running and configure an appropriate editor. See the [Environment Setup](/build-on-urbit/environment) guide for details.

Once you have a "...dojo>" prompt, the system is ready to go and waiting on input.

## Getting started <a href="#getting-started" id="getting-started"></a>

Once you've created your development ship, let's try a basic command. Type `%- add [2 2]` at the prompt and hit `Return`. (Note the double spaces before and after `+add`.) Your screen now shows:

```hoon
fake: ~zod
ames: czar: ~zod on 31337 (localhost only)
http: live (insecure, public) on 80
http: live (insecure, loopback) on 12321
> %-  add  [2 2]
4
~zod:dojo>
```

You just used a function from the Hoon standard library, `+add`, which for reasons that will become clear later is frequently written [`+add`](/hoon/stdlib/1a#add). Next, quit Urbit by entering [|exit](/user-manual/os/dojo-tools#exit) :

```hoon
> %-  add  [2 2]
4
~zod:dojo> |exit
$
```

Your ship isn't running anymore and you're back at your computer's normal terminal prompt. If your ship is \~zod, then you can restart the ship by typing:

```hoon
./urbit zod
```

You've already used a standard library function to produce one value, in the Dojo. Now that your ship is running again, let's try another. Enter the number `17`.

(We won't show the `~zod:dojo>` prompt from here on out. We'll just show the echoed command along with its result.)

You'll see:

```hoon
> 17
17
```

You asked Dojo to evaluate `17` and it echoed the number back at you. This value is a noun. We'll talk more about nouns in the next lesson.

Basically, every Hoon expression operates on the values it is given until it reduces to some form that can't evaluate any farther. This is then returned as the result of the evaluation.

One more:

```hoon
> :-  1  2
[1 2]
```

This `:-` rune takes two values and composes them into a cell, a pair of values.

## Pronouncing Hoon <a href="#pronouncing-hoon" id="pronouncing-hoon"></a>

Hoon uses runes, or two-character ASCII symbols, to describe its structure. (These are analogous to keywords in other programming languages.) Because there has not really been a standard way of pronouncing, say, `#` (hash, pound, number, sharp, hatch) or `!` (exclamation point, bang, shriek, pling), the authors of Urbit decided to adopt a one-syllable mnemonic to uniquely refer to each.

It is highly advisable for you to learn these pronunciations, as the documentation and other developers employ them frequently. For instance, a rune like `|=` is called a “bartis”, and you will find it designated as such in the docs, in the source code, and among the developers.

| Name  | Character |
| ----- | --------- |
| "ace" | `␣`       |
| "gap" | `␣␣`,     |
| "pat" | `@`       |
| "bar" | `\|`      |
| "gar" | `>`       |
| "sel" | `[`       |
| "bas" | `\`       |
| "hax" | `#`       |
| "ser" | `]`       |
| "buc" | `$`       |
| "hep" | `-`       |
| "sig" | `~`       |
| "cab" | `_`       |
| "kel" | `{`       |
| "soq" | `'`       |
| "cen" | `%`       |
| "ker" | `}`       |
| "tar" | `*`       |
| "col" | `:`       |
| "ket" | `^`       |
| "tic" | `` ` ``   |
| "com" | `,`       |
| "lus" | `+`       |
| "tis" | `=`       |
| "doq" | `"`       |
| "mic" | `;`       |
| "wut" | `?`       |
| "dot" | `.`       |
| "pal" | `(`       |
| "zap" | `!`       |
| "fas" | `/`       |
| "pam" | `&`       |
| "gal" | `<`       |
| "par" | `)`       |

Note that the list includes two separate whitespace forms: "ace" for a single space `␣`; "gap" is either two or more spaces `␣␣` or a line break . In Hoon, the only whitespace significance is the distinction between "ace" and "gap": the distinction between "one space" and "more than one space".


# 1. Hoon Syntax

Introduction to Hoon programming language syntax, covering basic elements like nouns (atoms and cells), runes (operators), auras (atom types), and fundamental control flow structures.

*This module will discuss the fundamental data concepts of Hoon and how programs handle control flow.*

The study of Hoon can be divided into two parts: syntax and semantics.

1. The **syntax** of a programming language is the set of rules that determine what counts as admissible code in that language. It determines which characters may be used in the source, and also how these characters may be assembled to constitute a program. Attempting to run a program that doesn’t follow these rules will result in a syntax error.
2. The **semantics** of a programming language concerns the meaning of the various parts of that language’s code.

In this lesson we will give a general overview of Hoon’s syntax. By the end of it, you should be familiar with all the basic elements of Hoon code.

## Hoon Elements <a href="#hoon-elements" id="hoon-elements"></a>

An [**expression**](https://en.wikipedia.org/wiki/Expression_%28computer_science%29) is a combination of characters that a language interprets and evaluates to produce a value. All Hoon programs are built of expressions, rather like mathematical equations. Hoon expressions are built along a backbone of runes, which are two-character symbols that act like keywords in other programming languages to define the syntax, or grammar, of the expression.

Runes are the building blocks of all Hoon code, represented as a pair of non-alphanumeric ASCII characters. Runes form expressions; runes are used how keywords are used in other languages. In other words, all computations in Hoon ultimately require runes. Runes and other Hoon expressions are all separated from one another by either two spaces or a line break.

All runes take a fixed number of “children” or “daughters”. Children can themselves be runes with children, and Hoon programs work by chaining through these until a value, not another rune, is arrived at. For this reason, we very rarely need to close expressions. Keep this scheme in mind when examining Hoon code.

Hoon expressions can be either basic or complex. Basic expressions of Hoon are fundamental, meaning that they can’t be broken down into smaller expressions. Complex expressions are made up of smaller expressions (which are called **subexpressions**).

The Urbit operating system hews to a conceptual model wherein each expression takes place in a certain context (the subject). While sharing a lot of practicality with other programming paradigms and platforms, Urbit's model is mathematically well-defined and unambiguously specified. Every expression of Hoon is evaluated relative to its subject, a piece of data that represents the environment, or the context, of an expression.

At its root, Urbit is completely specified by Nock, sort of a machine language for the Urbit virtual machine layer and event log. However, Nock code is basically unreadable (and unwriteable) for a human. [One worked example](/nock/decrement) yields, for decrementing a value by one, the Nock formula:

```hoon
[8 [1 0] 8 [1 6 [5 [0 7] 4 0 6] [0 6] 9 2 [0 2] [4 0 6] 0 7] 9 2 0 1]
```

This is like reading binary machine code: we mortals need a clearer vernacular.

Hoon serves as Urbit's practical programming language. Everything in Urbit OS is written in Hoon, and many of the ancillary tools as well.

Any operation in Urbit ultimately results in a value. Much like machine language designates any value as a command, an address, or a number, a Hoon value is interpreted per the Nock rules and results in a basic data value at the end. So what are our data values in Hoon, and how do they relate to each other?

## Nouns <a href="#nouns" id="nouns"></a>

Think about a child persistently asking you what a thing is made of. At first, you may respond, “plastic”, or “metal”. Eventually, the child may wear you down to a more fundamental level: atoms and molecules (bonded atoms).

In a very similar sense, everything in a Hoon program is an atom or a bond. Metaphorically, a Hoon program is a complex molecule, a digital chemistry that describes one mathematical representation of data.

The most general data category in Hoon is a noun. This is just about as broad as saying “thing”, so let's be more specific:

> A noun is an atom or a cell.

Progress? We can say, in plain English, that

* An atom is a non-negative integer number (0 to +∞), e.g. `42`.
* A cell is a pair of two nouns, written in square brackets, e.g. `[0 1]`.

*Everything* in Hoon (and Nock, and Urbit) is a noun. The Urbit OS itself is a noun. So given any noun, the Urbit VM simply applies the Nock rules to change the noun in well-defined mechanical ways.

### Atoms <a href="#atoms" id="atoms"></a>

If an atom is a non-negative number, how do we represent anything else? Hoon provides each atom an aura, a tag which lets you treat a number as text, time, date, Urbit address, IP address, and much more.

An aura always begins with `@` ("pat"), which denotes an atom (as opposed to a cell, `^` ("ket"), or the general noun, `*` ("tar")). The next letter or letters tells you what kind of representation you want the value to have.

For instance, to change the representation of a regular decimal number like `32` to a binary representation (i.e. for 2⁵), use `@ub`:

```hoon
> `@ub`32
0b10.0000
```

(The tic marks are a shorthand which we'll explain later.)

Aura values are all designed to be [URL-safe](https://developers.google.com/maps/url-encoding), so the European-style thousands separator `.` dot is used instead of the English `,` com. `1.000` is one thousand, not `1.0` one with a fractional part of zero.

While there are dozens of auras for specialized applications, here are the most important ones for you to know:

| Aura  | Meaning                     | Example                    | Comment                                                         |
| ----- | --------------------------- | -------------------------- | --------------------------------------------------------------- |
| `@`   | Empty aura                  | `100`                      | (displays as `@ud`)                                             |
| `@da` | Date (absolute)             | \~2022.2.8..16.48.20..b53a | Epoch calculated from 292 billion B.C.                          |
| `@p`  | Ship name                   | `~zod`                     |                                                                 |
| `@rs` | Number with fractional part | `.3.1415`                  | Note the preceding `.` dot.                                     |
| `@t`  | Text (“cord”)               | `'hello'`                  | One of Urbit's several text types; only UTF-8 values are valid. |
| `@ub` | Binary value                | `0b1100.0101`              |                                                                 |
| `@ud` | Decimal value               | `100.000`                  | Note that German-style thousands separator is used, `.` dot.    |
| `@ux` | Hexadecimal value           | `0x1f.3c4b`                |                                                                 |

You can see that these are all different-but-equivalent ways of representing the same underlying data values.

There's a special value that recurs in many contexts in Hoon: `~` sig is the null or zero value.

The `^-` [kethep](/hoon/rune/ket#kethep) rune is useful for ensuring that everything in the second child matches the type (aura) of the first, e.g.

```hoon
^-  @ux  0x1ab4
```

We will use `^-` kethep extensively to enforce type constraints, a very useful tool in Hoon code.

### Exercise: Aura Conversions <a href="#exercise-aura-conversions" id="exercise-aura-conversions"></a>

Convert between some of the given auras at the Dojo prompt, e.g.:

* `100` to `@p`
* `0b1100.0101` to `@p`
* `0b1100.0101` to `@ux`
* `0b1100.0101` to `@ud`
* `~` to any other aura

### Cells <a href="#cells" id="cells"></a>

A cell is a pair of nouns. Cells are traditionally written using square brackets: `[]`. For now, just recall the square brackets and that cells are always "pairs" of values.

```hoon
[1 2]
[@p @t]
[[1 2] [3 4]]
```

This is actually a shorthand for a rune as well, `:-` [colhep](/hoon/rune/col#colhep)

```hoon
:-  1  2
```

produces a cell `[1 2]`. You can chain these together:

```hoon
:-  1  :-  2  3
```

to produce `[1 [2 3]]` or `[1 2 3]`.

We deal with cells in more detail below.

{% hint style="info" %}
**Hoon as Noun**

We mentioned earlier that everything in Urbit is a noun, including the program itself. This is true, but getting from the rune expression in Hoon to the numeric expression requires a few more tools than we currently are prepared to introduce.

For now, you can preview the structure of the Urbit OS as a noun by typing `.` dot at the Dojo prompt. This displays a summary of the structure of the operating function itself as a noun.
{% endhint %}

## Verbs (Runes) <a href="#verbs-runes" id="verbs-runes"></a>

The backbone of any Hoon expression is a scaffolding of runes, which are essentially mathematical relationships between daughter components. If nouns are nouns, then runes are verbs: they describe how nouns relate. Runes provide the structural and logical relationship between noun values.

A rune is just a pair of ASCII characters (a digraph). We usually pronounce runes by combining their characters’ names, e.g.: ["kethep"](/hoon/rune/ket#kethep) for `^-`, ["bartis"](/hoon/rune/bar#bartis) for `|=`, and ["barcen"](/hoon/rune/bar#barcen) for `|%`.

For instance, when we called a function earlier (in Hoon parlance, we *slammed a gate*), we needed to provide the `%-` [cenhep](/hoon/rune/cen#cenhep) rune with two bits of information, a function name and the values to associate with it:

```hoon
%-
add  
[1 2]
```

The operation you just completed is straightforward enough: `1 + 2`, in many languages, or `(+ 1 2)` in a [Lisp dialect](https://en.wikipedia.org/wiki/Lisp_%28programming_language%29) like [Clojure](https://en.wikipedia.org/wiki/Clojure). Literally, we can interpret `%- add [1 2]` as “evaluate the `+add` core on the input values `[1 2]`”.

The [`+add`](/hoon/stdlib/1a#add) function expects precisely two values (or *arguments*), which are provided by [%-](/hoon/rune/cen#cenhep) in the neighboring child expression as a cell. There's really no limit to the complexity of Hoon expressions: they can track deep and wide. They also don't care much about layout, which leaves you a lot of latitude. The only hard-and-fast rule is that there are single spaces ("ace"s) and everything else ("gap"s).

```hoon
%-
add
[%-(add [1 2]) 3]
```

(Notice that inside of the `[]` cell notation we are using a slightly different form of the `%-` rune call. In general, there are several ways to use many runes, and we will introduce these gradually. We'll see more expressive ways to write Hoon code after you're comfortable using runes.)

For instance, here are some of the standard library functions which have a similar architecture in common:

* [`+add`](/hoon/stdlib/1a#add) (addition)
* [`+sub`](/hoon/stdlib/1a#sub) (subtraction, positive results only... what happens if you subtract past zero?)
* [`+mul`](/hoon/stdlib/1a#mul) (multiplication)
* [`+div`](/hoon/stdlib/1a#div) (integer division, no remainder)
* [`+pow`](/hoon/stdlib/2g#pow) (power or exponentiation)
* [`+mod`](/hoon/stdlib/1a#mod) (modulus, remainder after integer division)
* [`+dvr`](/hoon/stdlib/1a#dvr) (integer division with remainder)
* [`+max`](/hoon/stdlib/1a#max) (maximum of two numbers)
* [`+min`](/hoon/stdlib/1a#min) (minimum of two numbers)

### Rune Expressions <a href="#rune-expressions" id="rune-expressions"></a>

Any Hoon program is architected around runes. If you have used another programming language, you can see these as analogous to keywords, although they also make explicit what most language syntax parsers leave implicit. Hoon aims at a parsimony of representation while leaving latitude for aesthetics. In other words, Hoon strives to give you a unique characteristic way of writing a correct expression, but it leaves you flexibility in how you lay out the components to maximize readability.

We are only going to introduce a handful of runes in this lesson, but by the time we're done with Hoon School, you'll know the twenty-five or so runes that yield 80% of the capability.

### Exercise: Identifying Unknown Runes <a href="#exercise-identifying-unknown-runes" id="exercise-identifying-unknown-runes"></a>

Here is a lightly-edited snippet of Hoon code. Anything written after a `::` [colcol](/hoon/rune/col#colcol) is a *comment* and is ignored by the computer. (Comments are useful for human-language explanations.)

```hoon
%-  send
::  forwards compatibility with next-dill
?@  p.kyz  [%txt p.kyz ~]
?:  ?=  %hit  -.p.kyz
  [%txt ~]
?.  ?=  %mod  -.p.kyz
  p.kyz
=/  =@c
  ?@  key.p.kyz  key.p.kyz
    ?:  ?=  ?(%bac %del %ret)  -.key.p.kyz 
      `@`-.key.p.kyz
    ~-
?:  ?=  %met  mod.p.kyz  [%met c]  [%ctl c]
```

1. Mark each rune.
2. For each rune, find its corresponding children. (You don't need to know what a rune does to identify how things slot together.)
3. Consider these questions:
   * Is every pair of punctuation marks a rune?
   * How can you tell a rune from other kinds of marks?

One clue: every rune in Hoon (except for one, not in the above code) has *at least one child*.

### Exercise: Inferring Rune Behavior <a href="#exercise-inferring-rune-behavior" id="exercise-inferring-rune-behavior"></a>

Here is a snippet of Hoon code:

```hoon
^-  list
:~  [hen %slip %e %init ~]
    [hen %slip %d %init ~]
    [hen %slip %g %init ~]
    [hen %slip %c %init ~]
    [hen %slip %a %init ~]
==
```

Without looking it up first, what does the `==` [tistis](/hoon/rune/terminators#tistis) do for the `:~` [colsig](/hoon/rune/col#colsig) rune? Hint: some runes can take any number of arguments.

Most runes are used at the beginning of a complex expression, but there are exceptions. For example, the runes `--` [hephep](/hoon/rune/terminators#hephep) and `==` [tistis](/hoon/rune/terminators#tistis) are used at the end of certain expressions.

#### Aside: Writing Incorrect Code

At the Dojo, you can attempt to operate using the wrong values; for instance, `+add` doesn't know how to add three numbers at the same time.

```hoon
> %-  add  [1 2 3]
-need.@
-have.[@ud @ud]
nest-fail
dojo: hoon expression failed
```

So this statement above is *syntactically* correct (for the `%-` rune) but in practice fails because the expected input arguments don't match. Any time you see a "need"/"have" pair, this is what it means.

### Rune Families <a href="#rune-families" id="rune-families"></a>

Runes are classified by family (with the exceptions of `--` hephep and `==` tistis). The first of the two symbols indicates the family; the `^-` kethep rune is in the `^` [ket](/hoon/rune/ket) family of runes, and the `|=` bartis and `|%` barcen runes are in the `|` [bar](/hoon/rune/bar) family. The runes of particular family usually have related meanings. Two simple examples: the runes in the `|` bar family are all used to create cores, and the runes in the `:` [col](/hoon/rune/col) family are all used to create cells.

Rune expressions are usually complex, which means they usually have one or more subexpressions. The appropriate syntax varies from rune to rune; after all, they’re used for different purposes. To see the syntax rules for a particular rune, consult the rune reference. Nevertheless, there are some general principles that hold of all rune expressions.

Runes generally have a fixed number of expected children, and thus do not need to be closed. In other languages you’ll see an abundance of terminators, such as opening and closing parentheses, and this way of doing this is largely absent from Urbit. That’s because all runes take a fixed number of children. Children of runes can themselves be runes (with more children), and Hoon programs work by chaining through these series of children until a value (not another rune) is arrived at. This makes Hoon code nice and neat to look at.

### Tall and Wide Forms <a href="#tall-and-wide-forms" id="tall-and-wide-forms"></a>

We call rune expressions separated by "gap"s **tall form** and those using parentheses **wide form**. Tall form is usually used for multi-line expressions, and wide form is used for one-line expressions. Most runes can be used in either tall or wide form. Tall form expressions may contain wide form subexpressions, but wide form expressions may not contain tall form.

The spacing rules differ in the two forms. In tall form, each rune and subexpression must be separated from the others by a "gap": two or more spaces, or a line break. In wide form the rune is immediately followed by parentheses `( )`, and the various subexpressions inside the parentheses must be separated from the others by an "ace": a single space.

Seeing an example will help you understand the difference. The `:-` colhep rune is used to produce a cell. Accordingly, it is followed by two subexpressions: the first defines the head of the cell, and the second defines the tail. Here are three different ways to write a `:-` colhep expression in tall form:

```hoon
> :-  11  22
[11 22]

> :-  11
  22
[11 22]

> :-
  11
  22
[11 22]
```

All of these expressions do the same thing. The first example shows that, if you want to, you can write tall form code on a single line. Notice that there are two spaces between the `:-` colhep rune and `11`, and also between `11` and `22`. This is the minimum spacing necessary between the various parts of a tall form expression: any fewer will result in a syntax error.

Usually one or more line breaks are used to break up a tall form expression. This is especially useful when the subexpressions are themselves long stretches of code. The same `:-` colhep expression in wide form is:

```hoon
> :-(11 22)
[11 22]
```

This is the preferred way to write an expression on a single line. The rune itself is followed by a set of parentheses, and the subexpressions inside are separated by a single space. Any more spacing than that results in a syntax error.

Nearly all rune expressions can be written in either form, but there are exceptions. `|%` [barcen](/hoon/rune/bar#barcen) and `|_` [barcab](/hoon/rune/bar#barcab) expressions, for example, can only be written in tall form. (Those are a bit too complicated to fit comfortably on one line anyway.)

### Nesting Runes <a href="#nesting-runes" id="nesting-runes"></a>

Since runes take a fixed number of children, one can visualize how Hoon expressions are built by thinking of each rune being followed by a series of boxes to be filled: one for each of its children. Let us illustrate this with the `:-` [colhep](/hoon/rune/col#colhep) rune.

![](https://media.urbit.org/docs/hoon-syntax/cell1.png)

Here we have drawn the `:-` colhep rune followed by a box for each of its two children. We can fill these boxes with either a value or an additional rune. The following figure corresponds to the Hoon expression `:- 2 3`.

![](https://media.urbit.org/docs/hoon-syntax/cell2.png)

This, of course, evaluates to the cell `[2 3]`.

The next figure corresponds to the Hoon expression `:- :- 2 3 4`.

![](https://media.urbit.org/docs/hoon-syntax/cell3.png)

This evaluates to `[[2 3] 4]`, and we can think of the second `:-` colhep as being “nested” inside of the first `:-` colhep.

What Hoon expression does the following figure correspond to, and what does it evaluate to?

![](https://media.urbit.org/docs/hoon-syntax/cell4.png)

This represents the Hoon expression `:- 2 :- 3 4`, and evaluates to `[2 [3 4]]`. (If you input this into Dojo it will print as `[2 3 4]`, which we'll consider later.)

Thinking in terms of such “LEGO brick” diagrams can be a helpful learning and debugging tactic.

## Preserving Values with Faces <a href="#preserving-values-with-faces" id="preserving-values-with-faces"></a>

A Hoon expression is evaluated against a particular subject, which includes Hoon definitions and the standard library, as well as any user-specified values which have been made available. Unlike many procedural programming languages, a Hoon expression only knows what it has been told explicitly. This means that as soon as we calculate a value, it returns and falls back into the ether.

```hoon
%-  sub  [5 1]
```

Right now, we don't have a way of preserving values for subsequent use in a more complicated Hoon expression.

We are going to store the value as a variable, or in Hoon, “pin a face to the subject”. Hoon faces aren't exactly like variables in other programming languages, but for now we can treat them that way, with the caveat that they are only accessible to daughter or sister expressions.

When we used [`+add`](/hoon/stdlib/1a#add) or [`+sub`](/hoon/stdlib/1a#sub) previously, we wanted an immediate answer. There's not much more to say than `5 + 1`. In contrast, pinning a face accepts three daughter expressions: a name (or face), a value, and the rest of the expression.

```hoon
=/  perfect-number  28
%-  add  [perfect-number 10]
```

This yields `38`, but if you attempt to refer to `.perfect-number` again on the next line, the Dojo fails to locate the value.

```hoon
> =/  perfect-number  28
  %-  add  [perfect-number 10]
38

> perfect-number
-find.perfect-number
dojo: hoon expression failed
```

This syntax is a little bit strange in the Dojo because subsequent expressions, although it works quite well in long-form code. The Dojo offers a workaround to retain named values:

```hoon
> =perfect-number 28
> %-  add  [perfect-number 10]
38

> perfect-number
28
```

The difference is that the Dojo “pin” is permanent until deleted:

```hoon
=perfect-number
```

rather than only effective for the daughter expressions of a `=/` tisfas rune. (We also won't be able to use this Dojo-style pin in a regular Hoon program.)

### Exercise: A Large Power of Two <a href="#exercise-a-large-power-of-two" id="exercise-a-large-power-of-two"></a>

Create two numbers named "two" and "twenty", with appropriate values, using the `=/` tisfas rune.

Then use these values to calculate 2²⁰ with `+pow` and `%-` cenhep.

## Containers & Basic Data Structures <a href="#containers-basic-data-structures" id="containers-basic-data-structures"></a>

Atoms are well and fine for relatively simple data, but we already know about cells as pairs of nouns. How else can we think of collections of data?

### Cells <a href="#cells" id="cells"></a>

A cell is formally a pair of two objects, but as long as the second (right-hand) object is a cell, these can be written stacked together:

```hoon
> [1 [2 3]]
[1 2 3]

> [1 [2 [3 4]]]
[1 2 3 4]
```

This convention keeps the notation from getting too cluttered. For now, let's call this a “running cell” because it consists of several cells run together.

Since almost all cells branch rightwards, the pretty-printer (the printing routine that the Dojo uses) prefers to omit `[]` brackets marking the rightmost cells in a running cell. These read to the right; that is, `[1 2 3]` is the same as `[1 [2 3]]`.

### Exercise: Comparing Cells <a href="#exercise-comparing-cells" id="exercise-comparing-cells"></a>

Enter the following cells:

```hoon
[1 2 3]
[1 [2 3]]
[[1 2] 3]
[[1 2 3]]
[1 [2 [3]]]
[[1 2] [3 4]]
[[[1 2] [3 4]] [[5 6] [7 8]]]
```

Note which are the same as each other, and which are not. We'll look at the deeper structure of cells later when we consider trees.

### Lists <a href="#lists" id="lists"></a>

A running cell which terminates in a `~` sig (null) atom is a list.

* What is `~`'s value? Try casting it to another aura.

  `~` is the null value, and here acts as a list terminator.

Lists are ubiquitous in Hoon, and many specialized tools exist to work with them. (For instance, to apply a gate to each value in a list, or to sum up the values in a list, etc.) We'll see more of them in a future lesson.

### Exercise: Making a List from a Null-Terminated Cell <a href="#exercise-making-a-list-from-a-null-terminated-cell" id="exercise-making-a-list-from-a-null-terminated-cell"></a>

You can apply an aura to explicitly designate a null-terminated running cell as a list containing particular types of data. Sometimes you have to clear the aura using a more general aura (like `@`) before the conversion can work.

```hoon
> `(list @ud)`[1 2 3 ~]
~[1 2 3]

> `(list @ux)`[1 2 3 ~]
mint-nice
-need.?(%~ [i=@ux t=it(@ux)])
-have.[@ud @ud @ud %~]
nest-fail
dojo: hoon expression failed

> `(list @)`[1 2 3 ~]
~[1 2 3]

> `(list @ux)``(list @)`[1 2 3 ~]
~[0x1 0x2 0x3]
```

### Text <a href="#text" id="text"></a>

There are two ways to represent text in Urbit: cords (`@t` aura atoms) and tapes (lists of individual characters). Both of these are commonly called [“strings”](https://en.wikipedia.org/wiki/String_%28computer_science%29).

Why represent text? What does that mean? We have to have a way of distinguishing words that mean something to Hoon (like `+list`) from words that mean something to a human or a process (like `'hello world'`).

Right now, all you need to know is that there are (at least) two valid ways to write text:

* `'with single quotes'` as a cord.
* `"with double quotes"` as text.

We will use these incidentally for now and explain their characteristics in a later lesson. Cords and text both use [UTF-8](https://en.wikipedia.org/wiki/UTF-8) representation, but all actual code is [ASCII](https://en.wikipedia.org/wiki/ASCII).

```hoon
> "You can put ½ in quotes, but not elsewhere!"
"You can put ½ in quotes, but not elsewhere!"

> 'You can put ½ in single quotes, too.'
'You can put ½ in single quotes, too.'

> "Some UTF-8: ἄλφα"
"Some UTF-8: ἄλφα"
```

### Exercise: ASCII Values in Text <a href="#exercise-ascii-values-in-text" id="exercise-ascii-values-in-text"></a>

A cord (`@t`) represents text as a sequence of characters. If you know the [ASCII](https://en.wikipedia.org/wiki/ASCII) value for a particular character, you can identify how the text is structured as a number. (This is most easily done using the hexadecimal `@ux` representation due to bit alignment.)

![](https://media.urbit.org/docs/hoon-syntax/ASCII-Table-wide.png)

If you produce a text string as a cord, you can see the internal structure easily in Hoon:

```hoon
> `@ux`'Mars'
0x7372.614d
```

that is, the character codes `0x73` = `'s'`, `0x72` = `'r'`, `0x61` = `'a'`, and `0x4d` = `'M'`. Thus a cord has its first letter as the smallest (least significant, in computer-science parlance) byte.

## Making a Decision <a href="#making-a-decision" id="making-a-decision"></a>

The final rune we introduce in this lesson will allow us to select between two different Hoon expressions, like picking a fork in a road. Any computational process requires the ability to distinguish options. For this, we first require a basis for discrimination: truthness.

Essentially, we have to be able to decide whether or not some value or expression evaluates as `%.y` *true* (in which case we will do one thing) or `%.n` *false* (in which case we do another). At this point, our basic expressions are always mathematical; later on we will check for existence, for equality of two values, etc.

* [`+gth`](/hoon/stdlib/1a#gth) (greater than `>`)
* [`+lth`](/hoon/stdlib/1a#lth) (less than `<`)
* [`+gte`](/hoon/stdlib/1a#gte) (greater than or equal to `≥`)
* [`+lte`](/hoon/stdlib/1a#lte) (less than or equal to `≤`)

If we supply these with a pair of numbers to a `%-` cenhep call, we can see if the expression is considered `%.y` true or `%.n` false.

```hoon
> %-  gth  [5 6]
%.n
> %-  lth  [7 6]
%.n
> %-  gte  [7 6]
%.y
> %-  lte  [7 7]
%.y
```

Given a test expression like those above, we can use the `?:` wutcol rune to decide between the two possible alternatives. `?:` wutcol accepts three children: a true/false statement, an expression for the `%.y` true case, and an expression for the `%.n` false case.

[Piecewise mathematical functions](https://en.wikipedia.org/wiki/Piecewise) require precisely this functionality. For instance, the Heaviside function is a piecewise mathematical function which is equal to zero for inputs less than zero and one for inputs greater than or equal to zero.

$$
H(x)
====

\begin{cases} 1, & x > 0 \\\ 0, & x \le 0 \end{cases}
$$

*However*, we don't yet know how to represent a negative value! All of the decimal values we have used thus far are unsigned (non-negative) values, `@ud`. For now, the easiest solution is to just translate the Heaviside function so it activates at a different value:

$$
H\_{10}(x)
==========

\begin{cases} 1, & x > 10 \\\ 0, & x \le 10 \end{cases}
$$

Thus equipped, we can evaluate the Heaviside function for particular values of *n*:

```hoon
=/  x  10
?:  %-  gte  [x 10]
  1
0
```

We don't know yet how to store this capability for future use on as-yet-unknown values of $$x$$, but we'll see how to do so in a future lesson.

Carefully map how the runes in that statement relate to each other, and notice how the taller structure makes it relatively easier to read and understand what's going on.

### Exercise: “Absolute” Value (Around Ten) <a href="#exercise-absolute-value-around-ten" id="exercise-absolute-value-around-ten"></a>

Implement a version of the absolute value function, $$|x|$$, similar to the Heaviside implementation above. (Translate it to 10 as well since we still can't deal with negative numbers; call this $$|x|\_{10}$$.)

$$
|x|\_{10}
=========

\begin{cases} x-10, & x > 10 \\\ 10-x & 0 \le x \le 10 \end{cases}
$$

Test it on a few values like 8, 9, 10, 11, and 12.


# 2. Azimuth (Urbit ID)

Overview of Urbit ID (Azimuth) hierarchical address space, including galaxy, star, planet, moon, and comet identity types, their naming conventions, and sponsor relationships.

*This module introduces how Urbit ID is structured and provides practice in converting and working with `@p` Urbit IDs / Azimuth points. It may be considered optional and skipped if you are speedrunning Hoon School.*

## A Public-Key Infrastructure <a href="#a-public-key-infrastructure" id="a-public-key-infrastructure"></a>

What is the purpose of a [public-key infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure)? Essentially a PKI defines a protocol for asymmetrically revealing a public key (which anyone can use to check that a message came from where it says it came) and retaining a private key, used by the owner as a cryptographically secure tool for signing electronic transactions. Azimuth functions as a PKI so that Urbit ID points can be uniquely controlled, transferred, and used to work with instances of Urbit OS (ships).

Urbit ID (also called "Azimuth") provides persistent and stable futureproof identity to its users through a hierarchical address space. Any particular Urbit ID plays a particular role in the overall Urbit system which is determined by its point number and classified into ranks.

### The Urbit Address Space <a href="#the-urbit-address-space" id="the-urbit-address-space"></a>

Each Urbit ID point is a 128-bit address. Urbit is structured with a hierarchy of addressable points, and bands of smaller values (preceded by many zeroes) have more “weight” in the system and broker access for higher-addressed points.

* **Galaxies** represent the “governing council” of Urbit, primarily concerned with peer discovery and packet routing as well as network protocol governance. Galaxies allocate star address space.
* **Stars** provide peer discovery services, handle distribution of software updates, and allocate planet address space.
* **Planets** are the primary single-user identities.
* **Moons** are intended to represent devices and associated accounts for the owning planet, but are currently only rarely used. Each planet has 2³² moons available to it.
* **Comets** are zero-reputation instances, in principle spammers or bots. Comets require a star sponsor to access the network, but once online they are persistent. They are also free to spin up.

In total there are 2¹²⁸ addressable points, of which the vast majority are available as unclaimed “comet space.”

#### Naming

Urbit uses a system of mnemonic syllables to uniquely identify each address point. These mnemonic names are colloqually called Urbit IDs, but Urbit ID proper is the larger ID system including the Azimuth and Ecliptic smart contracts. You'll often hear developers refer to these names as their Hoon type `@p`, pronounced "pat-pee". The `@p` schema emerges out of a set of 256 suffixes (such as “zod”) and 256 prefixes (such as “lit”). They were selected to be memorable and pronounceable, but not inherently meaningful.

| Number | Prefix | Suffix |
| -----: | :----: | :----: |
|      0 |   doz  |   zod  |
|      1 |   mar  |   nec  |
|      2 |   bin  |   bud  |
|      3 |   wan  |   wes  |
|      4 |   sam  |   sev  |
|      … |    …   |    …   |
|    254 |   mip  |   nev  |
|    255 |   fip  |   fes  |

Many point numbers may be determined from the prefix and suffix alone, but planet names are obfuscated, meaning that they are scrambled so that the sponsor is not readily apparent to a peer.

#### Galaxy

The Galaxies span the first 2⁸ addresses of Azimuth. There are 255 (`0xff` - 1) associated stars; counting the galaxy yields 256 points (not counting moons). Galaxy names are suffix-only.

|             | First Address | Last Address |
| ----------- | ------------- | ------------ |
| Decimal     | `0`           | `255`        |
| Hexadecimal | `0x0`         | `0xff`       |
| `@p`        | \~zod         | \~fes        |

As galaxies have no sponsors, they instead have an IP address determined by `gal.urbit.org` at port `13337` plus the galaxy number.

At the current time, galaxies play the role of network peer discovery, but at some future time this will fall to the stars instead.

#### Star

Peer discovery, the primary role of stars besides planet allocation, is an important step in responsibly controlling network traffic. You need someone trustworthy to sponsor your membership on the network. An address that can’t find a sponsor may reasonably be considered to be a bot or a spammer.

The Stars span the remaining addresses to 2¹⁶. There are thus 65,536 - 256 = 65,280 stars. Star names have prefix and suffix. They share the suffix with their sponsoring galaxy.

|             | First Address | Last Address |
| ----------- | ------------- | ------------ |
| Decimal     | `256`         | `65.535`     |
| Hexadecimal | `0x100`       | `0xffff`     |
| `@p`        | \~marzod      | \~fipfes     |

A star's sponsor can be calculated as modulo 2⁸. The first star of \~zod is `0x100` \~marzod. The last star of \~zod is `0xffff` - `0xff` = `0xff00` \~fipzod. The last star (of \~fes) is `0xffff` \~fipfes.

#### Planet

The Planets span the remaining addresses to 2³². There are thus 4,294,967,296 - 65,536 = 4,294,901,760 planets. Planet names occur in pairs separated by a single hyphen. A planet's name is obfuscated so it is not immediately apparent who its sponsor is.

|             | First Address   | Last Address    |
| ----------- | --------------- | --------------- |
| Decimal     | `65.536`        | `4.294.967.295` |
| Hexadecimal | `0x1.0000`      | `0xffff.ffff`   |
| `@p`        | \~dapnep-ropmyl | \~dostec-risfen |

A planet's sponsor can be calculated as modulo 2¹⁶.

Galaxy planets occupy points beginning with `0x1.0000` \~dapnep-ronmyl (for \~zod); \~zod's last galaxy planet is `0xffff.ffff` - `0xffff` = `0xffff.0000` \~lodnyt-ranrud. The last galaxy planet (of \~fes) is `0xffff.ffff` - `0xffff` + `0x00ff` = `0xffff.00ff` \~hidwyt-mogbud.

Star planets span the remaining space. The first star planet (of \~marzod) is `0x1.000` + `0x100` = `0x1.0100` \~wicdev-wisryt. The last star planet (of \~fipfes) is `0xffff.ffff` \~dostec-risfen. Remember that star planet recur module 2¹⁶.

#### Moon

The Moons occupy the block to 2⁶⁴, with 2³² moons for each planet. Moon names have more than two blocks (three or four) separated by single hyphens.

|             | First Address          | Last Address                  |
| ----------- | ---------------------- | ----------------------------- |
| Decimal     | `4.294.967.296`        | `18.446.744.073.709.551.615`  |
| Hexadecimal | `0x1.0000.0000`        | `0xffff.ffff.ffff.ffff`       |
| `@p`        | \~doznec-dozzod-dozzod | \~fipfes-fipfes-dostec-risfen |

Moons recur modulo 2³² from their sponsor. Thus dividing a moon's address by 2³² and taking the remainder yields the address of the sponsor.

Any moon that begins with the prefix "\~dopzod-dozzod-doz..." is a galaxy moon, but not every galaxy moon begins with that prefix. The first galaxy moon of \~zod is 0x1.0000.0000 \~doznec-dozzod-dozzod; the last is `0xffff.ffff.ffff.ffff` - `0xffff.ffff` = `0xffff.ffff.0000.0000` \~fipfes-fipfes-dozzod-dozzod.

Any moon that begins with the prefix "\~dopzod-dozzod-..." is a star moon (other than galaxy moons), but not every star moon begins with that prefix. The first star moon of \~marzod is `0x1.0000.0000.0100` \~doznec-dozzod-dozzod-marzod; the last is `0xffff.ffff.ffff.ffff` - `0xffff.ffff` + `0x100` = `0xffff.ffff.0000.0100` \~fipfes-fipfes-dozzod-marzod.

Any moon from "\~dopzod-...-..." onwards is a planet moon.

#### Comet

The Comets occupy the upper portion of the Urbit address space. There are approximately 3.4×10³⁸ comets, a fantastically large number. Comet names occur in blocks of five to eight syllable pairs, separated by a double hyphen at the fourth.

|             | First Address                         | Last Address                                               |
| ----------- | ------------------------------------- | ---------------------------------------------------------- |
| Decimal     | `18.446.744.073.709.551.616`          | `340.282.366.920.938.463.463.374.607.431.768.211.456`      |
| Hexadecimal | `0x1.0000.0000.0000.0000`             | `0xffff.ffff.ffff.ffff.ffff.ffff.ffff.ffff`                |
| @p          | \~doznec--dozzod-dozzod-dozzod-dozzod | \~fipfes-fipfes-fipfes-fipfes--fipfes-fipfes-fipfes-fipfes |

A comet is sponsored by a star. Currently star sponsors are determined randomly from a list supplied to `u3_dawn_come` in `pkg/urbit/vere/dawn.c` from a [jamfile](/hoon/stdlib/2p#jam) provided by urbit.org at `https://bootstrap.urbit.org/comet-stars.jam`.

Comets cannot be breached or rekeyed: possession of the comet is *ipso facto* attestation of ownership.

## Calculating with Addresses <a href="#calculating-with-addresses" id="calculating-with-addresses"></a>

### Sponsors <a href="#sponsors" id="sponsors"></a>

Each point other than a galaxy has a sponsor. To determine the sponsor of any point, use `+sein:title`:

```hoon
%-(sein:title [our now ~marzod])
```

where \~marzod is the point in question; or more succinctly:

```hoon
(sein:title our now ~marzod)
```

(This previews the irregular syntax of `%-` cenhep; it is equivalent to `%- sein:title [our now ~marzod]`.)

### Exercise: Finding neighbors <a href="#exercise-finding-neighbors" id="exercise-finding-neighbors"></a>

A neighbor of a point is a point which occupies the point immediately above or below that point's `@ud` number.

For instance, the `@ud` of \~sampel-palnet may be found by:

```hoon
> `@ud`~sampel-palnet
1.624.961.343
```

The previous neighbor of \~sampel-palnet is thus:

```hoon
> %-(sub [1.624.961.343 1])
1.624.961.342

> `@p`1.624.961.342
~datwyn-lavrud
```

As an exercise, find the next neighbor of \~sampel-palnet.

### Exercise: Finding the sponsor of a neighbor <a href="#exercise-finding-the-sponsor-of-a-neighbor" id="exercise-finding-the-sponsor-of-a-neighbor"></a>

The sponsor of \~sampel-palnet may be found by:

```hoon
> (sein:title our now ~sampel-palnet)
~talpur
```

The sponsor of the previous neighbor of \~sampel-palnet is thus:

```hoon
> %-(sub [1.624.961.343 1])
1.624.961.342

> `@p`1.624.961.342
~datwyn-lavrud

> (sein:title our now ~datwyn-lavrud)
~talnep
```

Try finding the sponsor of the next neighbor of \~sampel-palnet.

### Exercise: Finding the child of a point <a href="#exercise-finding-the-child-of-a-point" id="exercise-finding-the-child-of-a-point"></a>

A point has many children, but the first moon of a planet is located at that point plus 2³², which is `4.294.967.296`.

The first moon of \~sampel-palnet is:

```hoon
> `@p`%-(add [~sampel-palnet 4.294.967.296])
~doznec-sampel-palnet
```

What are the first moon children of \~sampel-palnet's neighbors?

What is the first planet of the star \~sampel? (Check the above text to determine the offset.)


# 3. Gates (Functions)

Learn to create and use gates (functions) in Hoon, including deferred computations, type specification, file organization, generators, and irregular syntactic sugar forms.

*This module will teach you how to produce deferred computations for later use, like functions in other languages.*

## A Spoonful of Sugar <a href="#a-spoonful-of-sugar" id="a-spoonful-of-sugar"></a>

Until this point in Hoon School, we have rigorously adhered to the regular syntax of runes so that you could get used to using them. In fact, the only two irregular forms we used were these:

Cell definition `[a b]` which represents the `:-` [colhep](/hoon/rune/col#colhep) rune, `:- a b`.

That is, these expressions are all the same for Hoon:

```hoon
> [1 2]
[1 2]

> :-  1  2
[1 2]

> :-
1
2
[1 2]
```

Aura application `` `@ux`500 `` which represents a double `^-` [kethep](/hoon/rune/ket#kethep), like `^- @ux ^- @ 500`.

These are equivalent in Hoon:

```hoon
> ^-  @p  ^-  @  255
~fes

> `@p`255
~fes
```

(Why two `^-`s? We have to clear the type information in general to be able to apply new type information.)

Hoon developers often employ irregular forms, sometimes called “sugar syntax”. Besides the `:-` colhep and `^-` kethep forms, we will commonly use a new form for `%-` [cenhep](/hoon/rune/cen#cenhep) “function calls”:

```hoon
> %-  add  [1 2]
3

> (add 1 2)
3
```

You should get used to reading and interpreting these forms. We will start to use them actively during this lesson. You can find other irregular forms in the [irregular forms reference](/hoon/irregular).

### Exercise: Converting Between Forms <a href="#exercise-converting-between-forms" id="exercise-converting-between-forms"></a>

Convert each of the following irregular forms into the correct regular runic syntax.

1. `(add 1 2)`
2. `` `@ub`16 ``
3. `[%lorem %ipsum]`
4. `[%lorem %ipsum %dolor]` (can do two ways)

Convert each of the following regular forms into the correct irregular syntax.

1. `:- %lemon %jello`
2. `^- @p ^- @ 256`
3. `%- pow :- 2 16`

## Deferring Computations <a href="#deferring-computations" id="deferring-computations"></a>

So far, every time we have calculated something, we have had to build it from scratch in Dojo. This is completely untenable for nontrivial calculations, and clearly the Urbit OS itself is built on persistent code structures defining the behavior.

```hoon
::  Confirm whether a value is greater than one.
=/  a  5
?:  (gth a 1)
  'yes'
'no'
```

This has no flexibility: if we want to change `.a` we have to rewrite the whole thing every time!

(Note also our introduction of the `::` [colcol](/hoon/rune/col#colcol) digraph in the above code block. This marks anything following it as a *comment*, meaning that it is meant for the developer and reader, and ignored by the computer.)

Hoon uses gates as deferred computations. What this means is that we can build a Hoon expression now and use it at need later on, perhaps many times. More than that, we can also use it on different data values. A gate is the Hoon analogue of a [function or subroutine](https://en.wikipedia.org/wiki/Subroutine) in other programming languages.

The word "function" is used in various ways, but let's start by talking about them in the [mathematical sense](https://en.wikipedia.org/wiki/Function_\(mathematics\)). Roughly put, a function takes one or more arguments (i.e., input values) and returns a value. The return value depends solely on the argument(s), and nothing else. For example, we can understand multiplication as a function: it takes two numbers and returns another number. It doesn't matter where you ask, when you ask, or what kind of hat you're wearing when you ask. If you pass the same two numbers (e.g., `3` and `4`), you get the same answer returned every time (`12`).

That output value depends solely upon input value(s) is an important property of functions. This property is called [referential transparency](https://en.wikipedia.org/wiki/Referential_transparency), and it's one of the key ingredients to building a secure Urbit stack.

Functions are implemented in Hoon with a special kind of core called a "gate". In this lesson you'll learn what a gate is and how a gate represents a function. (We *won't* talk about what a core is quite yet.) Along the way you'll build some example gates of your own.

### Building a Gate <a href="#building-a-gate" id="building-a-gate"></a>

Syntactically, a gate is a `|=` [bartis](/hoon/rune/bar#bartis) rune with two children: a [spec](/hoon/stdlib/4o#spec) (specification of input) and a [hoon](/hoon/stdlib/4o#hoon) (body). Think of just replacing the `=/` [tisfas](/hoon/rune/tis#tisfas) with the `|=` bartis:

```hoon
::  Confirm whether a value is greater than one.
|=  a=@ud
?:  (gth a 1)
  'yes'
'no'
```

Compare this to other programming languages, if you know any:

* Does it have a name?
* Does it have a return value?

Beyond those, what is the purpose of each line?

The [`$spec`](https://docs.urbit.org/hoon/stdlib/4o#spec) gives the type as a mold and attaches a face to it for use in the gate.

The Hoon body expression evaluates and yields a result, ultimately sent back to the call site. Frequently it is wise to explicitly require a particular type for the return value using the `^-` [kethep](/hoon/rune/ket#kethep) rune:

```hoon
::  Confirm whether a value is greater than one.
|=  a=@ud
^-  @t
?:  (gth a 1)
  'yes'
'no'
```

The input value, what is included in the `$spec`, is sometimes called the argument or parameter in mathematics and other programming languages. Hoon prefers to call this input value the "sample" for reasons that will become apparent later on, but you won't confuse other Hoon developers if you call it the argument or input.

Note as well that the backbone of the program runs straight down the left-hand margin. This makes it easier to read the essential mainline logic of the program.

Gates enforce the type of incoming and outgoing values. In other words, a `$spec` is a kind of type which is fixing the possible noun inputs. (The lesson on types which follows this one will go into greater detail.)

Gates can take multiple arguments as a cell:

```hoon
::  Return which of two numbers is larger.
|=  [a=@ud b=@ud]
?:  (gth a b)
  a
b
```

You can also call them different ways with raw `%` [cen](/hoon/rune/cen) runes:

```hoon
%-  max  [100 200]
%+  max  100  200
```

### Creating Your Own Gate <a href="#creating-your-own-gate" id="creating-your-own-gate"></a>

You can type the above Hoon code snippets directly into Dojo, but there's no way to actually use them yet! The Dojo recognizes the expression as valid Hoon code, but can't actually apply it to an input sample yet.

```hoon
> |=  [a=@ud b=@ud]
  ?:  (gth a b)
    a
  b
< 1.tfm
  [ [a=@ud b=@ud]
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
```

We need to attach a name (also called a "face" in Hoon's internals) to the expression. Then we'll be able to use it directly. Somewhat confusingly, there are three common ways to do this:

1. Attach the face directly in Dojo. (This is a good quick solution, and we'll use it when teaching and testing code, but it doesn't work inside of code files.)
2. Save the gate as a generator file and call it using the name of the file. (We'll do this in the next section of this lesson.)
3. Attach the face as an arm in a core. (We don't know what those are yet, so we'll set them aside for a couple of lessons.)

To name a gate in Dojo (or any expression resulting in a value, which is *every* expression), you can use the Dojo-specific syntax `=name value`:

```hoon
> =inc |=  [a=@]
       (add 1 a)

> (inc 1)
2

> (inc 12)
13

> (inc 5)
6
```

Notice that there is *one* space ("ace") after the `=name` term and then regular *n*-spaces ("gap"s) thereafter. We could also do this in one line using wide form:

```hoon
> =inc |=(a=@ (add 1 a))

> (inc 123)
124
```

To reiterate: we typically use the `|=` [bartis](/hoon/rune/bar#bartis) rune to create a gate. In the expression above the `|=` is immediately followed by a set of parentheses containing two subexpressions: `a=@` and `(add 1 a)`. The first defines the gate's sample (input value type), and the second defines the gate's product (output value).

In the example gate above, `+inc`, the sample is defined by `a=@`. This means that the sample is defined as an atom `@` meaning that the gate will take as input anything of that type (so, not a cell). The sample is given the face `a`. With a face it's easier to refer to the sample value in later code.

The second subexpression after the `|=` bartis rune is used to build the gate's body, where all the computations go. In `+inc`, the product is defined by `(add 1 a)`. There's not much to it: it returns the value of `a+1`!

### Exercise: Double a Value <a href="#exercise-double-a-value" id="exercise-double-a-value"></a>

Produce a gate which accepts any `@ud` unsigned integer value and doubles it. Call it "double".

```hoon
> =double |=(a=@ud (mul a 2))

> (double 5)
10
```

### Exercise: Convert Between Auras <a href="#exercise-convert-between-auras" id="exercise-convert-between-auras"></a>

Produce a gate which accepts any `@ud` unsigned integer value and converts it to the `@p` equivalent. Call the gate "myship".

Produce a gate which accepts any `@ud` unsigned integer value and calculates the next neighbor (the `@p` of the number plus one). Call it "myneighbor".

Produce a gate which accepts a `@p` ship name and produces the `@ux` unsigned hexadecimal integer value of the ship. Call it "mynumber".

### Output Values <a href="#output-values" id="output-values"></a>

How can we control what kind of value a function returns in Hoon? Many programming languages (such as C, Java, and TypeScript) are *extremely* concerned about this specification. Others, like Python and MATLAB, are *laissez-faire*. Hoon tends to be strict, but leaves some discretion over *how* strict to you, the developer.

Remember `^-` [kethep](/hoon/rune/ket#kethep)? We will use `^-` as a "fence", a way of making sure only data matching the appropriate structure get passed on.

```hoon
::  Confirm whether a value is greater than one by return 1 (if no) or 0 (if yes).
|=  a=@ud
^-  @ud
?:  (gth a 1)
  1
0
```

**This is the correct way to define a gate.** Frequent annotation of type with `^-` fences is *essential* to producing good Hoon code. From this point forward in Hoon School, we will hew to this standard.

In technical language, we describe Hoon as a "statically typed" language. This means that it enforces type constraints on all values very aggressively. If you are used to a dynamic language like Python or Ruby, this will seem very restrictive at first. The flip side is that once your code compiles correctly, you will often find that it is very much along the way towards being a working correct product.

## Coordinating Files <a href="#coordinating-files" id="coordinating-files"></a>

In pragmatic terms, an Urbit ship is what results when you successfully boot a new ship. If you are in the host OS, what you see is an apparently-empty folder:

```sh
$ ls zod
$
```

(For this lesson in particular take pains to distinguish the host OS prompt `$` from the Urbit Dojo prompt `>` . You should look into particular system setup instructions for macOS and Linux hosts.)

Contrast that apparently empty folder with what the `+ls %` command shows you from inside of your Urbit (at the Dojo prompt):

```hoon
> +ls %
app/ desk/bill gen/ lib/ mar/ sur/ sys/ ted/
```

Urbit organizes its internal view of data and files as "desks", which are associated collections of code and data. These are not visible to the host operating system unless you explicitly mount them, and changes on one side are not made clear to the other until you “commit” them. (Think of Dropbox, except that you have to explicitly synchronize to see changes somewhere else.)

Inside of your ship, you can mount a particular desk to the host operating system:

```hoon
> |mount %base
```

Now check what happens outside of your ship:

```sh
$ ls zod
base/
$ ls zod/base
app/  desk.bill gen/ lib/ mar/ sur/ sys/ ted/
```

If we make a change in the folder on Earth, the contents will only update on Mars if we explicitly tell the two systems to coordinate.

On Earth:

```sh
$ cp zod/base/desk.bill zod/base/desk.txt
```

On Mars:

```hoon
> |commit %base
+ /~zod/base/2/desk/txt
```

You can verify the contents of the copied files are the same using the [`+cat`](/user-manual/os/dojo-tools#cat) command:

```hoon
> +cat %/desk/bill

> +cat %/desk/txt
```

(Dojo does know what a "bill" file is, so it displays the contents slightly formatted. They are actually identical.)

We will use this [`|commit`](/user-manual/os/dojo-tools#commit) pattern to store persistent code as files, editing on Earth and then synchronizing to Mars.

## Building Code <a href="#building-code" id="building-code"></a>

The missing piece to really tie all of this together is the ability to store a gate and use it at a later time, not just in the same long Dojo session. Enter the generator.

A generator is a simple program which can be called from the Dojo. It is a gate, so it takes some input as sample and produces some result. Naked generators are the simplest generators possible, having access only to information passed to them directly in their sample.

In this section, we will compose our first generator.

### The Gate <a href="#the-gate" id="the-gate"></a>

```hoon
::  Square a number.
|=  a=@ud
^-  @ud
%+  mul
  a
a
```

(Any time you write code to use later, you should include some comments to explain what the code does and perhaps how it does that.)

### The Process <a href="#the-process" id="the-process"></a>

1. Open a text editor.
2. Copy the gate above into the text editor. (Double-check that two-space gaps are still gaps; some text editors chew them up into single-space aces.)
3. Save the gate as `square.hoon` in the `/base/gen` folder of your fakeship.
4. In the Dojo, `|commit %base`. *You should see a message indicating that the file has been loaded.*
5. Run the generator with `+square 5`.

Any generator can be run the same way, beginning with the `+` lus character and followed by the name of a file in the `/base/gen` directory.

### Hoon Source and Special Characters <a href="#hoon-source-and-special-characters" id="hoon-source-and-special-characters"></a>

Hoon source files are composed almost entirely of the printable ASCII characters. Hoon does not accept any other characters in source files except for [UTF-8](https://en.wikipedia.org/wiki/UTF-8) in quoted strings. Hard tab characters are illegal; use two spaces instead.

```hoon
> "You can put ½ in quotes, but not elsewhere!"
"You can put ½ in quotes, but not elsewhere!"

> 'You can put ½ in single quotes, too.'
'You can put ½ in single quotes, too.'

> "Some UTF-8: ἄλφα"
"Some UTF-8: ἄλφα"
```

**Note:** Urbit requires Unix-style line endings for Hoon files. You might need to manually change the line endings in your IDE from Windows-style `CRLF` to Unix-style `LF`. In many editors the current setting will be shown in a status bar at the bottom.

### Exercise: Triangular Function <a href="#exercise-triangular-function" id="exercise-triangular-function"></a>

Implement the triangular function as a gate and save it as a generator `tri.hoon`.

![](https://media.urbit.org/docs/hoon-school/plot-triangular.png)

### Coding Piecemeal <a href="#coding-piecemeal" id="coding-piecemeal"></a>

If you need to test code without completing it, you can stub out as-yet-undefined arms with the `!!` [zapzap](/hoon/rune/zap#zapzap) crash rune. `!!` is the only rune which has no children, and it's helpful when you need to put in something to satisfy Hoon syntax but aren't ready to flesh out the program yet.

### Building Code Generally <a href="#building-code-generally" id="building-code-generally"></a>

A generator gives us on-demand access to code, but it is helpful to load and use code from files while we work in the Dojo.

A conventional library import with `/+` [faslus](/hoon/rune/fas#faslus) will work in a generator or another file, but won't work in Dojo, so you can't use `/+` faslus interactively.

Instead, you need to use the [`-build-file`](/user-manual/os/dojo-tools#build-file) thread to load the code. Most commonly, you will do this with library code when you need a particular core's functionality.

`-build-file` accepts a file path and returns the built operational code, to which you can then attach a `+face`. For instance:

```hoon
> =ntw -build-file %/lib/number-to-words/hoon

> one-hundred:numbers:ntw
100

> (to-words:eng-us:ntw 19)
[~ "nineteen"]
```

There are also a number of other import runes which make library, structure, and mark code available to you. Right now, the only one you need to worry about is `/+` faslus.

For simplicity, everything we do will take place on the `%base` desk for now. We will learn how to create a library in a subsequent lesson.

### Exercise: Loading a Library <a href="#exercise-loading-a-library" id="exercise-loading-a-library"></a>

In a generator, load the library at `/lib/number-to-words.hoon` using the `/+` faslus rune. (This must take place at the very top of your file.)

Use this to produce a gate which accepts an unsigned decimal integer and returns the text interpretation of its increment.


# 4. Molds (Types)

Guide to Hoon's type system, covering atoms with auras, molds for data structures, type-checking and type inference, core variance, nesting relationships, and type safety.

*This module will introduce the Hoon type system and illustrate how type checking and type inference work.*

## The Hoon Type System <a href="#the-hoon-type-system" id="the-hoon-type-system"></a>

Programming languages use data types to distinguish different kinds of data and associated rules. For instance, what does it mean to add 3 to the letter A? Depending on your programming language, you could see `A3`, `D`, or an error.

Like most modern high-level programming languages, Hoon has a type system. Because Hoon is a functional programming language, its type system differs somewhat from those of non-functional languages. In this lesson we'll introduce Hoon's type system and point out some of its distinctive features. Certain advanced topics (e.g. type polymorphism) won't be addressed until a later chapter.

A type is ordinarily understood to be a set of values. Examples: the set of all atoms is a type, the set of all cells is a type, and so on.

Type systems provide type safety, in part by making sure functions produce values of the correct type. When you write a function whose product is intended to be an atom, it would be nice to know that the function is guaranteed to produce an atom. Hoon's type system provides such guarantees with *type checking* and *type inference*.

A "type" is really a rule for interpretation. But for our Hoonish purposes, it's rather too broad a notion and we need to clarify some different kinds of things we could refer to as “type”. It is instructive for learners to distinguish three kinds of types in Hoon:

1. **Auras:** atoms with type metadata. (e.g. `0` as a `@t` is `''`)
2. **Molds:** typed nouns. Think of cells, lists, and sets.
3. **Marks:** file types. Compare to conventional files distinguished by extension and definite internal structure.

To employ a chemical metaphor, an atom is an atom; a cell is a molecule; a mold is an molecule definition, a template or structural representation; a mark is like a protein, a more complex transformation rule. **All of these are molds, or Hoon types. We are simply separating them by complexity as you learn.**

You have seen and worked with the trivial atoms and cells. We will leave marks until a later discussion of Gall agents or the Clay filesystem, which use marks to type file data. For now, we focus on molds.

This lesson will talk about atoms, cells, then molds in a general sense. We allude to several topics which will be explored in Data Structures.

## Atoms and Auras <a href="#atoms-and-auras" id="atoms-and-auras"></a>

In the most straightforward sense, atoms simply are unsigned integers. But they can also be interpreted as representing signed integers, ASCII symbols, floating-point values, dates, binary numbers, hexadecimal numbers, and more. Every atom is, in and of itself, just an unsigned integer; but Hoon keeps track of type information about each atom, and this bit of metadata tells Hoon how to interpret the atom in question.

The piece of type information that determines how Hoon interprets an atom is called an aura. The set of all atoms is indicated with the symbol `@`. An aura is indicated with `@` followed by some letters, e.g., `@ud` for unsigned decimal. Accordingly, the Hoon type system does more than track sets of values. It also tracks certain other relevant metadata about how those values are to be interpreted.

How is aura information generated so that it can be tracked? One way involves **type inference**. In certain cases Hoon's type system can infer the type of an expression using syntactic clues. The most straightforward case of type inference is for a [literal](https://en.wikipedia.org/wiki/Literal_%28computer_programming%29) expression of data, such as `0x1000` for `@ux`. Hoon recognizes the aura literal syntax and infers that the data in question is an atom with the aura associated with that syntax.

To see the inferred type of a literal expression in the Dojo, use the `?` operator. (This operator isn't part of the Hoon programming language; it's a Dojo-only tool.)

The `?` Dojo operator shows both the product and the inferred type of an expression. Let's try `?` on `15`:

```
> 15
15

> ? 15
  @ud
15
```

`@ud` is the inferred type of `15` (and of course `15` is the product). The `@` is for “atom” and the `ud` is for “unsigned decimal”. The letters after the `@` indicate the “aura” of the atom.

One important role played by the type system is to make sure that the output of an expression is of the intended data type. If the output is of the wrong type then the programmer did something wrong. How does Hoon know what the intended data type is? The programmer must specify this explicitly by using a "cast". To cast for an unsigned decimal atom, you can use the `^-` kethep rune along with the `@ud` from above.

What exactly does the `^-` [kethep](/hoon/rune/ket#kethep) rune do? It compares the inferred type of some expression with the desired cast type. If the expression's inferred type "nests" under the desired type, then the product of the expression is returned.

Let's try one in the Dojo.

```
> ^-(@ud 15)
15
```

Because `@ud` is the inferred type of `15`, the cast succeeds. Notice that the `^-` kethep expression never does anything to modify the underlying noun of the second subexpression. It's used simply to mandate a type-check on that expression. This check occurs at compile-time (when the expression is compiled to Nock).

What if the inferred type doesn't fit under the cast type? You will see a "nest-fail" crash at compile-time:

```
> ^-(@ud [13 14])
nest-fail
[crash message]
```

Why "nest-fail"? The inferred type of `[13 14]` doesn't nest under the cast type `@ud`. It's a cell, not an atom. But if we use the symbol for nouns, `*`, then the cast succeeds:

```
> ^-(* [13 14])
[13 14]
```

A cell of atoms is a noun, so the inferred type of `[13 14]` nests under `*`. Every product of a Hoon expression nests under `*` because every product is a noun.

### What Auras are There? <a href="#what-auras-are-there" id="what-auras-are-there"></a>

Hoon has a wide (but not extensible) variety of atom literal syntaxes. Each literal syntax indicates to the Hoon type checker which predefined aura is intended. Hoon can also pretty-print any aura literal it can parse. Because atoms make great path nodes and paths make great URLs, all regular atom literal syntaxes use only URL-safe characters. The pretty-printer is convenient when you are used to it, but may surprise you occasionally as a learner.

Here's a non-exhaustive list of auras, along with examples of corresponding literal syntax:

| Aura   | Meaning                                                           | Example Literal Syntax       |
| ------ | ----------------------------------------------------------------- | ---------------------------- |
| `@d`   | date                                                              | no literal                   |
| `@da`  | absolute date                                                     | `~2018.5.14..22.31.46..1435` |
| `@dr`  | relative date (ie, timespan)                                      | `~h5.m30.s12`                |
| `@p`   | phonemic base (ship name)                                         | `~sampel-palnet`             |
| `@r`   | [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) floating-point |                              |
| `@rd`  | double precision (64 bits)                                        | `.~6.02214085774e23`         |
| `@rh`  | half precision (16 bits)                                          | `.~~3.14`                    |
| `@rq`  | quad precision (128 bits)                                         | `.~~~6.02214085774e23`       |
| `@rs`  | single precision (32 bits)                                        | `.6.022141e23`               |
| `@s`   | signed integer, sign bit low                                      | no literal                   |
| `@sb`  | signed binary                                                     | `--0b11.1000`                |
| `@sd`  | signed decimal                                                    | `--1.000.056`                |
| `@sv`  | signed base32                                                     | `-0v1df64.49beg`             |
| `@sw`  | signed base64                                                     | `--0wbnC.8haTg`              |
| `@sx`  | signed hexadecimal                                                | `-0x5f5.e138`                |
| `@t`   | UTF-8 text (`$cord`)                                              | `'howdy'`                    |
| `@ta`  | ASCII text (subset) (`$knot`)                                     | `~.howdy`                    |
| `@tas` | ASCII text symbol (subset) (`$term`)                              | `%howdy`                     |
| `@u`   | unsigned integer                                                  | no literal                   |
| `@ub`  | unsigned binary                                                   | `0b11.1000`                  |
| `@ud`  | unsigned decimal                                                  | `1.000.056`                  |
| `@uv`  | unsigned base32                                                   | `0v1df64.49beg`              |
| `@uw`  | unsigned base64                                                   | `0wbnC.8haTg`                |
| `@ux`  | unsigned hexadecimal                                              | `0x5f5.e138`                 |

Some of these auras nest under others. For example, `@u` is for all unsigned auras. But there are other, more specific auras; `@ub` for unsigned binary numbers, `@ux` for unsigned hexadecimal numbers, etc. (For a more complete list of auras, see [Auras](/hoon/auras).)

Note that `$knot` and `$term` values each use a URL-safe subset of ASCII which excludes characters like spaces.

### Aura Inference in Hoon <a href="#aura-inference-in-hoon" id="aura-inference-in-hoon"></a>

Let's work a few more examples in the Dojo using the `?` operator. We'll focus on just the unsigned auras for now:

```
> 15
15

> ? 15
  @ud
15

> 0x15
0x15

> ? 0x15
  @ux
0x15
```

When you enter just `15`, the Hoon type checker infers from the syntax that its aura is `@ud` because you typed an unsigned integer in decimal notation. Hence, when you use `?` to check the aura, you get `@ud`.

And when you enter `0x15` the type checker infers that its aura is `@ux`, because you used `0x` before the number to indicate the unsigned hexadecimal literal syntax. In both cases, Hoon pretty-prints the appropriate literal syntax by using inferred type information from the input expression; the Dojo isn't (just) echoing what you enter.

More generally: for each atom expression in Hoon, you can use the literal syntax of an aura to force Hoon to interpret the atom as having that aura type. For example, when you type `~sampel-palnet` Hoon will interpret it as an atom with aura `@p` and treat it accordingly.

Here's another example of type inference at work:

```
> (add 15 15)
30

> ? (add 15 15)
  @
30

> (add 0x15 15)
36

> ? (add 0x15 15)
  @
36
```

The [`+add`](/hoon/stdlib/1a#add) function in the Hoon standard library operates on all atoms, regardless of aura, and returns atoms with no aura specified. Hoon isn't able to infer anything more specific than `@` for the product of `+add`. This is by design, however. Notice that when you `+add` a decimal and a hexadecimal above, the correct answer is returned (pretty-printed as a decimal). This works for all of the unsigned auras:

```
> (add 100 0b101)
105

> (add 100 0xf)
115

> (add 0b1101 0x11)
30
```

The reason these add up correctly is that unsigned auras all map directly to the 'correct' atom underneath. For example, `16`, `0b1.0000`, and `0x10` are all the exact same atom, just with different literal syntax. (This doesn't hold for signed versions of the auras!)

## Cells <a href="#cells" id="cells"></a>

Let's move on to consider cells. For now we'll limit ourselves to simple cell types made up of various atom types.

### Generic Cells <a href="#generic-cells" id="generic-cells"></a>

The `^` ("ket") symbol is used to indicate the type for cells (i.e., the set of all cells). We can use it for casting as we did with atom auras, like `@ux` and `@t`:

```
> ^-(^ [12 13])
[12 13]

> ^-(^ [[12 13] 14])
[[12 13] 14]

> ^-(^ [[12 13] [14 15 16]])
[[12 13] [14 15 16]]

> ^-(^ 123)
nest-fail

> ^-(^ 0x10)
nest-fail
```

If the expression to be evaluated produces a cell, the cast succeeds; if the expression evaluates produces an atom, the cast fails with a nest-fail crash.

The downside of using `^` for casts is that Hoon will infer only that the product of the expression is a cell of nouns; `^` tells the compiler nothing about the contents of the cell.

```
> ? ^-(^ [12 13])
  [* *]
[12 13]

> ? ^-(^ [[12 13] 14])
  [* *]
[[12 13] 14]

> ? ^-(^ [[12 13] [14 15 16]])
  [* *]
[[12 13] [14 15 16]]
```

When we use the `?` operator to see the type inferred by Hoon for the expression, in all three of the above cases the same thing is returned: `[* *]`. The `*` ("tar") symbol indicates the type for any noun, and the square brackets indicate a cell. Every cell in Hoon is a cell of nouns; remember that cells are defined as pairs of nouns.

Yet the cell `[[12 13] [14 15 16]]` is a bit more complex than the cell `[12 13]`. Can we use the type system to distinguish them? Yes.

### Getting More Specific <a href="#getting-more-specific" id="getting-more-specific"></a>

What if you want to cast for a particular kind of cell? You can use square brackets when casting for a specific cell type. For example, if you want to cast for a cell in which the head and the tail must each be an atom, then simply cast using `[@ @]`:

```
> ^-([@ @] [12 13])
[12 13]

> ? ^-([@ @] [12 13])
  [@ @]
[12 13]

> ^-([@ @] 12)
nest-fail

> ^-([@ @] [[12 13] 14])
nest-fail
```

The `[@ @]` cast accepts any expression that evaluates to a cell with exactly two atoms, and crashes with a [nest-fail](/hoon/hoon-errors#nest-fail) error for any expression that evaluates to something different. The expression `12` doesn't evaluate to a cell; and while the expression `[[12 13] 14]` does evaluate to a cell, the left-hand side isn't an atom, but is instead another cell.

You can get even more specific about the kind of cell you want by using atom auras:

```
> ^-([@ud @ux] [12 0x10])
[12 0x10]

> ^-([@ub @ux] [0b11 0x10])
[0b11 0x10]

> ? ^-([@ub @ux] [0b11 0x10])
  [@ub @ux]
[0b11 0x10]

> ^-([@ub @ux] [12 13])
nest-fail
```

You are also free to embed more square brackets to indicate cells within cells:

```
> ^-([[@ud @sb] @ux] [[12 --0b1101] 0xdead.beef])
[[12 --0b1101] 0xdead.beef]

> ? ^-([[@ud @sb] @ux] [[12 --0b1101] 0xdead.beef])
  [[@ud @sb] @ux]
[[12 --0b1101] 0xdead.beef]

> ^-([[@ @] @] [12 13])
nest-fail
```

You can also be highly specific with certain parts of the type structure, leaving other parts more general. Keep in mind that when you do this, Hoon's type system will infer a general type from the general part of the cast. Type information may be thrown away:

```
> ^-([^ @ux] [[12 --0b1101] 0xdead.beef])
[[12 26] 0xdead.beef]

> ? ^-([^ @ux] [[12 --0b1101] 0xdead.beef])
  [[* *] @ux]
[[12 26] 0xdead.beef]

> ^-(* [[12 --0b1101] 0xdead.beef])
[[12 26] 3.735.928.559]

> ? ^-(* [[12 --0b1101] 0xdead.beef])
  *
[[12 26] 3.735.928.559]
```

Because every piece of Hoon data is a noun, everything nests under `*`. When you cast to `*` you can see the raw noun with cells as brackets and atoms as unsigned integers.

## Molds <a href="#molds" id="molds"></a>

A mold is a template or rule for identifying actual type structures. **Molds are actually gates, meaning that they operate on a value to coerce it to a particular structure.** Technically, a mold is a function from a noun to a noun. What this means is that we can use a mold to coerce any noun into a typed value; if this gate fails, then the mold crashes.

```
> (^ [1 2])
[1 2]

> (@ [1 2])
dojo: hoon expression failed

> `@`[1 2]
mint-nice
-need.@
-have.[@ud @ud]
nest-fail
dojo: hoon expression failed
```

We commonly need to do one of two things with a mold:

1. Validate the shape of a noun. (Or, "clam" the noun in Hoon developer parlance.)

```
> (@ux 0x1000)
0x1000

> (@ux [1 2])
dojo: hoon expression failed
```

2. Produce an example value (bunt).

We often use bunts (default value of a type) to clam; for example, `@ud` implicitly uses the `@ud` default value (`0`) as the type specimen which the computation must match.

To *actually* get the bunt value, use the `^*` [kettar](/hoon/rune/ket#kettar) rune, almost always used in its irregular form `*`

```
> ^*  @ud
0

> ^*  @da
~2000.1.1

> *@da
~2000.1.1

> *[@ud @ux @ub]
[0 0x0 0b0]
```

One more way to validate against type is to use an example instead of the extracted mold. This uses the `^+` [ketlus](/hoon/rune/ket#ketlus) rune similarly to how we used `^-` [kethep](/hoon/rune/ket#kethep) previously:

```hoon
^+(1.000 100)
```

(This is what `^-` is actually doing: `^-(p q)` reduces to `^+(^*(p) q)`. Many runes we use actually reduce to other rune forms, and have been introduced for ease of use.)

We can use more complex structures for molds though, including built-in types like lists and tapes. (A `$tape` represents text.)

```hoon
`(list @)`[104 101 108 108 111 32 77 97 114 115 33 ~]
`tape``(list @)`[104 101 108 108 111 32 77 97 114 115 33 ~]

`(list @)`[144 57 195 46 200 165 186 88 118 99 ~]
`(list @p)``(list @)`[144 57 195 46 200 165 186 88 118 99 ~]
```

(Sometimes you see a `%bad-text` when using `$tape`s, which means that you've tried to convert a number into text which isn't text. More on `$tape`s in the lesson on trees.)

Why does this mold conversion fail? What do we need to do in order to make it succeed?

```hoon
`(list @ux)`[1 2 3 ~]
```

We can have more complex molds as well:

```hoon
::  [[from-ship to-ship] points]
[[@p @p] @ud]
```

Most of the time, we will define such complex types using specific runes and “mold builder” tools. Thus a `+list` mold needs an associated type `(list @)` to correctly denote the data type.

### Identifying Molds <a href="#identifying-molds" id="identifying-molds"></a>

Besides `?` (which is a Dojo-specific tool), the programmatic way to figure out which mold the Hoon compiler thinks something is to use the `!>` [zapgar](/hoon/rune/zap#zapgar) rune.

```hoon
> !>(0xace2.bead)
[#t/@ux q=2.900.541.101]
```

For reasons which will be elaborated in the lesson on trees, this is often employed as the so-called “type spear” `-:!>`:

```
> -:!>(0xace2.bead)
#t/@ux
```

### Type Unions <a href="#type-unions" id="type-unions"></a>

The `$?` [bucwut](/hoon/rune/buc#bucwut) rune forms a type union. Most commonly these are used with types having different structures, such as an atom and a cell.

For instance, if you wanted a gate to accept an atom of an unsigned aura type, but no other type, you could define a type union thus:

```hoon
$?  [@ud @ux @ub ~]
```

and use it in a gate:

```hoon
|=  [n=$?(@ud @ux @ub)]
(add n 1)
```

```
> (foo 4)  
5  
> (foo 0x5)  
6  
> (foo 0b110)  
7  
> (foo ~zod)  
-need.?(@ub @ud @ux)  
-have.@p  
nest-fail  
dojo: hoon expression failed
```

Unfortunately, type unions of atoms are not helpful in filtering over produced values (with `^-` kethep), as they default to the type of the last value in the union. So the type union `$?(@ (list @))` distinguishes an atom and a list, but `(list $?(@ud @sd))` does not successfully produce a list distinguishing both types.

The irregular form of `$?` looks like this:

```hoon
?(@ud @ux @ub)
```

Type unions are mainly helpful when you need to match something that can have multiple options. We will use them extensively with `@tas` terms, such as `?(%red %green %blue)` which would only admit one of those three tags.


# 5. Cores

Introduction to cores, Hoon's fundamental data structure that separates computation from data. Covers traps for recursion, core creation with arms and samples.

*This module will introduce the key Hoon data structure known as the **core**, as well as ramifications.*

The Hoon subject is a noun. One way to look at this noun is to denote each fragment of is as either a computation or data. By strictly separating these two kinds of things, we derive the data structure known within Hoon as a core.

Cores are the most important data structure in Hoon. They allow you to solve many coding problems by identifying a pattern and supplying a proper data structure apt to the challenge. You have already started using cores with `|=` [bartis](/hoon/rune/bar#bartis) gate construction and use.

This lesson will introduce another core to solve a specific use case, then continue with a general discussion of cores. Getting cores straight will be key to understanding why Hoon has the structure and internal logic it does.

## Repeating Yourself Using a Trap <a href="#repeating-yourself-using-a-trap" id="repeating-yourself-using-a-trap"></a>

Computers were built and designed to carry out tasks which were too dainty and temperamental for humans to repeat consistently, or too prodigiously numerous for humans to ever complete. At this point, you know how to build code that can make a decision between two branches, two different Hoon expressions. Computers can decide between alternatives, but they also need to carry out a task until some condition is met. (We can think of it as a recipe step, like “crack five eggs into a bowl”. Until that process is complete, we as humans continue to carry out the equivalent action again and again until the process has been completed.)

In programming, we call this behavior a “loop”. A loop describes the situation in which we set up some condition, and repeat a process over and over until something we do meets that condition. *Most* of the time, this means counting once for each item in a collection, like a list.

Hoon effects the concept of a loop using recursion, return to a particular point in an expression (presumably with some different values). One way to do this is using the `|-` [barhep](/hoon/rune/bar#barhep) rune, which creates a structure called a trap. (Think of the “trap” in the bottom of your sink.) It means a point to which you can return again, perhaps with some key values (like a counter) changed. Then you can repeat the calculation inside the trap again. This continues until some single value, some noun, results, thereby handing a value back out of the expression. (Remember that every Hoon expression results in a value.)

This program adds 1+2+3+4+5 and returns the sum:

```hoon
=/  counter  1
=/  sum  0
|-
?:  (gth counter 5)
  sum
%=  $
  counter  (add counter 1)
  sum      (add sum counter)
==
```

(The last two lines happen simultaneously, so make sure to refer to the *current* version of any variables.)

Let's unroll it:

1. `counter = 1sum = 0`
2. `(gth counter 5) = %.ncounter ← (add counter 1) = 2sum ← (add sum counter) = 0 + 1 = 1`
3. `(gth counter 5) = %.ncounter ← (add counter 1) = 3sum ← (add sum counter) = 1 + 2 = 3`
4. `(gth counter 5) = %.ncounter ← (add counter 1) = 4sum ← (add sum counter) = 3 + 3 = 6`
5. `(gth counter 5) = %.ncounter ← (add counter 1) = 5sum ← (add sum counter) = 6 + 4 = 10`
6. `(gth counter 5) = %.ncounter ← (add counter 1) = 6sum ← (add sum counter) = 10 + 5 = 15`
7. `(gth counter 5) = %.y`

And thus `+sum` yields the final value of `15`.

It is frequently helpful, when constructing these, to be able to output the values at each step of the process. Use the `~&` [sigpam](/hoon/rune/sig#sigpam) rune to create output without changing any values:

```hoon
=/  counter  1
=/  sum  0
|-
~&  "counter:"
~&  counter
~&  "sum:"
~&  sum
?:  (gth counter 5)
  sum
%=  $
  counter  (add counter 1)
  sum      (add sum counter)
==
```

You can do even better using *interpolation*:

```hoon
=/  counter  1
=/  sum  0
|-
~&  "counter: {<counter>}"
~&  "sum: {<sum>}"
?:  (gth counter 5)
  sum
%=  $
  counter  (add counter 1)
  sum      (add sum counter)
==
```

### Exercise: Calculate a Factorial <a href="#exercise-calculate-a-factorial" id="exercise-calculate-a-factorial"></a>

Let's calculate a [factorial](https://mathworld.wolfram.com/Factorial.html). The factorial of a number $$n$$ is $$n \times (n-1) \times \ldots \times 2 \times 1$$. We will introduce a couple of new bits of syntax and a new gate ([`+dec`](/hoon/stdlib/1a#dec)). Make this into a generator `factorial.hoon`:

```hoon
|=  n=@ud
|-
~&  n
?:  =(n 1)
  1
%+  mul
  n
%=  $
  n  (dec n)
==
```

We are using the `=` irregular syntax for the `.=` [dottis](/hoon/rune/dot#dottis) rune, which tests for the equality of two expressions.

```hoon
> +factorial 5
120
```

Let's visualize the operation of this gate using pseudocode (fake code that's explanatory but may not be operational). Here's basically what's happening when `+factorial` receives the value `5`:

```hoon
(factorial 5)
(mul 5 (factorial 4))
(mul 5 (mul 4 (factorial 3)))
(mul 5 (mul 4 (mul 3 (factorial 2))))
(mul 5 (mul 4 (mul 3 (mul 2 (factorial 1)))))
(mul 5 (mul 4 (mul 3 (mul 2 1))))
(mul 5 (mul 4 (mul 3 2)))
(mul 5 (mul 4 6))
(mul 5 24)
120
```

We're “floating” gate calls until we reach the final iteration of such calls that only produces a value. The `mul n` component of the gate leaves `mul 5` waiting for the final series of terms to be operated upon. The `%=($ n (dec n)))` component expands the expression outwards, as illustrated by `(factorial 4)`. This continues until the expression is not expanded further, at which point the operations work backwards, successively feeding values into the `+mul` functions behind them.

The pyramid-shaped illustration approximates what's happening on the "call stack", a memory structure that tracks the instructions of the program. In this code, every time a parent gate calls another gate, the gate being called is "pushed" to the top of the stack in the form of a frame. This process continues until a value is produced instead of a function, completing the stack.

Why do we return the result of `factorial.hoon` at 1 instead of 0?

### Exercise: Tracking Expression Structure <a href="#exercise-tracking-expression-structure" id="exercise-tracking-expression-structure"></a>

As we write more complicated programs, it is helpful to learn to read the runes by identifying which daughter expressions attach to which runes, e.g.:

```hoon
=/
  n
  15
  |-
    ~&
      n
      ?:
        =(n 1)      ::  .=  n  1
        1
      %+
        mul
        n
        %=
          $
          n
          (dec n)   ::  %-  dec  n
        ==
```

Recall that the `::` digraph tells the compiler to ignore the rest of the text on the line. Such text is referred to as a "comment" because, instead of performing a computation, it exists to explain things to human readers of the source code. Here, we have also explicitly marked the expansion of the irregular forms.

We will revert to the irregular form more and more. If you would like to see exactly how an expression is structured, you can use the `!,` [zapcom](/hoon/rune/zap#zapcom) rune. `!,` zapcom produces an annotated *abstract syntax tree* (AST) which labels every value and expands any irregular syntax into the regular runic form.

```hoon
> !,  *hoon  (add 5 6)
[%cncl p=[%wing p=~[%add]] q=[i=[%sand p=%ud q=5] t=[i=[%sand p=%ud q=6] t=~]]]
```

```
> !,  *hoon
 |=  n=@ud
 |-  
 ~&  n  
 ?:  =(n 1)  
   1
 %+  mul  
   n
 %=  $  
   n  (dec n)  
 ==  
[ %brts
  p=[%bcts p=term=%n q=[%base p=[%atom p=~.ud]]]
    q
  [ %brhp
      p
    [ %sgpm
      p=0
      q=[%wing p=~[%n]]
        r
      [ %wtcl
        p=[%dtts p=[%wing p=~[%n]] q=[%sand p=%ud q=1]]
        q=[%sand p=%ud q=1]
          r
        [ %cnls
          p=[%wing p=~[%mul]]
          q=[%wing p=~[%n]]
          r=[%cnts p=~[%$] q=[i=[p=~[%n] q=[%cncl p=[%wing p=~[%dec]] q=[i=[%wing p=~[%n]] t=~]]] t=~]]
        ]
      ]
    ]
  ]
]
```

(*There's a lot going on in there.* Focus on the four-letter runic identifiers: `%sgpm` for `~&` [sigpam](/hoon/rune/sig#sigpam), for instance.)

### Exercise: Calculate a sequence of numbers <a href="#exercise-calculate-a-sequence-of-numbers" id="exercise-calculate-a-sequence-of-numbers"></a>

Produce a gate (generator) which accepts a `@ud` value and calculates the series where the $$i^\text{th}$$ term in the series is given by the equation

$$
n\_{i} = i^{2}
\textrm{,}
$$

that is, the first numbers are 0, 1, 4, 9, 16, 25, etc.

For this exercise, you do not need to store these values in a list. Calculate each one but only return the final value.

### Exercise: Output each letter in a `$tape` <a href="#exercise-output-each-letter-in-a-tape" id="exercise-output-each-letter-in-a-tape"></a>

Produce a gate (generator) which accepts a tape value and returns a `(list @ud)` containing the ASCII value of each character. Use a `|-` [barhep](/hoon/rune/bar#barhep) trap. The previous code simply modified a value by addition. You can generalize this to other arithmetic processes, like multiplication, but you can also grow a data structure like a list.

For example, given the `$tape` `"hello"`, the generator should return the list `[104 101 108 108 111 ~]`. (A list is structurally a null-terminated tuple, or rightwards-branching cell ending in `~` or `0`.) We can equivalently write `~[104 101 108 108 111]` which is a special syntax reducing to the same thing.

Two tools that may help:

* You can retrieve the *n*th element in a `$tape` using the [`+snag`](/hoon/stdlib/2b#snag) gate, e.g. ``(snag 3 `(list @ud)`~[1 2 3 4 5])`` yields `4` (so `+snag` is zero-indexed; it counts from zero).
* You can join an element to a list using the [`+snoc`](/hoon/stdlib/2b#snoc) gate, e.g. ``(snoc `(list @ud)`~[1 2 3] 4)`` yields `~[1 2 3 4]`.

```hoon
|=  [input=tape]
=/  counter  0
=/  results  *(list @ud)
|-
?:  =(counter (lent input))
  results
=/  ascii  `@ud`(snag counter input)
%=  $
  counter  (add counter 1)
  results  (snoc results ascii)
==
```

## Cores <a href="#cores" id="cores"></a>

So far we have introduced and worked with a few key structures:

1. Nouns (raw data/code)
2. Molds (types)
3. Gates (functions)
4. Traps (loops)

Some of them are *data*, like raw values: `0x1234.5678.abcd` and `[5 6 7]`. Others are *code*, programs that do something. What unifies all of these under the hood?

A core is a cell pairing operations to data. Formally, we'll say a core is a cell \[battery payload], where battery describes the things that can be done (the operations) and payload describes the data on which those operations rely. (For many English speakers, the word “battery” evokes a [voltaic pile](https://en.wikipedia.org/wiki/Voltaic_pile) more than a bank of guns, but the artillery metaphor is a better mnemonic for \[battery payload].)

**Cores are the most important structural concept for you to grasp in Hoon.** Everything nontrivial is a core. Some of the runes you have used already produce cores, like the gate. That is, a gate marries a battery (the operating code) to the payload (the input values AND the subject or operating context).

Urbit adopts an innovative programming paradigm called subject-oriented programming. By and large, Hoon (and Nock) is a functional programming language in that running a piece of code twice will always yield the same result, and because runs cause a program to explicitly compose various subexpressions in a somewhat mathematical way.

Hoon (and Nock) very carefully bounds the known context of any part of the program as the subject. Basically, the subject is the noun against which any arbitrary Hoon code is evaluated.

For instance, when we first composed generators, we made what are called “naked generators”: that is, they do not have access to any information outside of the base subject (Arvo, Hoon, and `%zuse`) and their sample (arguments). Other generators (such as `%say` generators, described below) can have more contextual information, including random number generators and optional arguments, passed to them to form part of their subject.

Cores have two kinds of values attached: arms and "legs", both called limbs. Arms describe known labeled addresses (with `++` [luslus](/hoon/rune/lus#luslus) or `+$` [lusbuc](/hoon/rune/lus#lusbuc)) which carry out computations. Legs are limbs which store data (with e.g. `=/` [tisfas](/hoon/rune/tis#tisfas)).

### Arms <a href="#arms" id="arms"></a>

So legs are for data and arms are for computations. But what *specifically* is an arm, and how is it used for computation? Let's begin with a preliminary explanation that we'll refine later.

An arm is some expression of Hoon encoded as a noun. (By 'encoded as a noun' we literally mean: 'compiled to a Nock formula'. But you don't need to know anything about Nock to understand Hoon.) You virtually never need to treat an arm as raw data, even though technically you can, it's just a noun like any other. You almost always want to think of an arm simply as a way of running some Hoon code.

Every expression of Hoon is evaluated relative to a subject. An arm is a Hoon expression to be evaluated against the core subject (i.e. its parent core is its subject).

#### Arms for Gates

Within a core, we label arms as Hoon expressions (frequently `|=` bartis gates) using the `++` [luslus](/hoon/rune/lus#luslus) digraph. (`++` isn't formally a rune because it doesn't actually change the structure of a Hoon expression, it simply marks a name for an expression or value. The `--` [hephep](/hoon/rune/terminators#hephep) limiter digraph is used because `|%` [barcen](/hoon/rune/bar#barcen) can have any number of arms attached. Like `++`, it is not formally a rune.)

```hoon
|%
++  add-one
  |=  a=@ud
  ^-  @ud
  (add a 1)
++  sub-one
  |=  a=@ud
  ^-  @ud
  (sub a 1)
--
```

Give the name "adder" to the above, and use it thus:

```
> (add-one:adder 5)
6

> (sub-one:adder 5)
4
```

Notice here that we read the arm resolution from right-to-left. This isn't the only way to address an arm, but it's the most common one.

### Exercise: Produce a Gate Arm <a href="#exercise-produce-a-gate-arm" id="exercise-produce-a-gate-arm"></a>

Compose a core which contains arms for multiplying a value by two and for dividing a value by two.

#### Arms for Types

We can define custom types for a core using `+$` [lusbuc](/hoon/rune/lus#lusbuc) digraphs. We won't do much with these yet but they will come in handy for custom types later on.

This core defines a set of types intended to work with playing cards:

```hoon
|%
+$  suit  ?(%hearts %spades %clubs %diamonds)
+$  rank  ?(1 2 3 4 5 6 7 8 9 10 11 12 13)
+$  card  [sut=suit val=rank]
+$  deck  (list card)
--
```

#### Cores in Generators

When we write generators, we can include helpful tools as arms either before the main code (with `=>` [tisgar](/hoon/rune/tis#tisgar)) or after the main code (with `=<` [tisgal](/hoon/rune/tis#tisgal)):

```hoon
|=  n=@ud
=<
(add-one n)
|%
++  add-one
  |=  a=@ud
  ^-  @ud
  (add a 1)
--
```

A library (a file in `/lib`) is typically structured as a `|%` [barcen](/hoon/rune/bar#barcen) core.

### Legs <a href="#legs" id="legs"></a>

A "leg" is a data value. They tend to be trivial but useful ways to pin constants. `=/` tisfas values are legs, for instance.

```
> =/  a  1
  (add a 1)
2
```

Under the hood, legs and arms are distinguished by the Nock instructions used in each case. A leg is evaluated by Nock 0, while an arm is evaluated by Nock 9.

### Recalculating a Limb <a href="#recalculating-a-limb" id="recalculating-a-limb"></a>

Arms and legs are both "limbs". Either one can be replaced in a given subject. This turns out to be very powerful, and permits Hoon to implement gates (functions) in a mathematically rigorous way, among other applications.

Often a leg of the subject is produced with its value unchanged. But there is a way to produce a modified version of the leg as well. To do so, we use the `%=` [centis](/hoon/rune/cen#centis) rune:

```hoon
%=  subject-limb
  leg-1  new-leg-1
  leg-2  new-leg-2
  ...
==
```

`%=` centis is frequently used in its irregular form, particularly if the expression within it fits on a single line. The irregular form prepends the arm (often `$`) to brackets (e.g. `$(...)`). In its irregular form, the above would be:

```hoon
subject-limb(leg-1 new-leg-1, leg-2 new-leg-2, ...)
```

In the first example, we saw the expression

```hoon
%=  $
  counter  (add counter 1)
  sum      (add sum counter)
==
```

which can equivalently be expressed as

```hoon
$(counter (add counter 1), sum (add sum counter))
```

This statement means that we recalculate the `$` buc arm of the current subject with the indicated changes. But what is `$` buc? `$` buc is the "default arm" for many core structures, including `|=` [bartis](/hoon/rune/bar#bartis) gate cores and `|-` [barhep](/hoon/rune/bar#barhep) trap cores.

### What is a Gate? <a href="#what-is-a-gate" id="what-is-a-gate"></a>

A core is a cell of \[battery payload].

A gate is a core with two distinctive properties:

1. The battery of a gate contains an arm which has the special name `$` buc. The `$` buc arm contains the instructions for the function in question.
2. The payload of a gate consists of a cell of \[sample context].
   1. The sample is the part of the payload that stores the "argument" (i.e., input value) of the function call.
   2. The **context** contains all other data that is needed for computing the `$` buc arm of the gate correctly. As a tree, a gate looks like the following:

```
[$ [sample context]]

       gate
      /    \
     $      .
           / \
     sample   context
```

Like all arms, `$` buc is computed with its parent core as the subject. When `$` buc is computed, the resulting value is called the “product” of the gate. No other data is used to calculate the product other than the data in the gate itself.

We will always call the values supplied to the gate the “sample” since we will later discover that this technical meaning (\[battery \[sample context]]) holds throughout more advanced cores.

### Exercise: Another Way to Calculate a Factorial <a href="#exercise-another-way-to-calculate-a-factorial" id="exercise-another-way-to-calculate-a-factorial"></a>

Let's revisit our factorial code from above:

```hoon
|=  n=@ud
|-
?:  =(n 1)
  1
%+  mul
  n
%=  $
  n  (dec n)
==
```

We can write this code in several ways using the `%=` [centis](/hoon/rune/cen#centis) plus `$` buc structure.

For instance, we can eliminate the trap by recursing straight back to the gate:

```hoon
|=  n=@ud
?:  =(n 1)
  1
%+  mul
  n
%=  $
  n  (dec n)
==
```

This can be collapsed into a shorter equivalent form by employing the irregular form of `%=` centis:

```hoon
|=  n=@ud
?:  =(n 1)
  1
(mul n $(n (dec n)))
```

(Sugar syntax like `$()` does not affect code efficiency, merely visual layout.)

#### The `$` Buc Arm

The `$` arm of a gate encodes the instructions for the Hoon function in question.

```
> =inc |=(a=@ (add 1 a))

> (inc 5)
6
```

The pretty-printer represents the `$` buc arm of `+inc` as `1.yop`. To see the actual noun of the `$` arm, enter `+2:inc` into the Dojo:

```
> +2:inc
[8 [9 36 0 4.095] 9 2 10 [6 [7 [0 3] 1 1] 0 14] 0 2]
```

This is un-computed Nock. You don't need to understand any of this, except that code and data are homoiconic: they are in a sense the same for Urbit programs.

It's worth pointing out that the arm named `$` can be used like any other name. We can compute `$` directly with `$:inc` in the Dojo:

```
> $:inc
1
```

This result may seem a bit strange. We didn't call `+inc` or in any other way pass it a number. Yet using `$` buc to evaluate `+inc`'s arm seems to work... sort of, anyway. Why is it giving us `1` as the return value? We can answer this question after we understand gate samples a little better.

#### The Sample

The sample of a gate is the address reserved for storing the argument(s) to the Hoon function. Although we don't know about addressing yet, you saw above that `+2` referred to the battery. The sample is always at the head of the gate's tail, `+6`. (We'll look at addressing in more depth in [the next module](/build-on-urbit/hoon-school/g-trees).)

Let's look at the gate for `+inc` again, paying particular attention to its sample:

```
> inc
< 1.mgz
  [ a=@
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.lcv 14.tdo 54.dnu 77.mau 236.dqo 51.njr 139.hzy 33.uof 1.pnw %138>
  ]
>
```

We see `a=@`. This may not be totally clear, but at least the `@` should make a little sense. This is the pretty-printer's way of indicating an atom with the face `a`. Let's take a closer look:

```
> +6:inc
a=0
```

We see now that the sample of `+inc` is the value `0`, and has `a` as a face. This is a placeholder value for the function argument. If you evaluate the `$` buc arm of `+inc` without passing it an argument the placeholder value is used for the computation, and the return value will thus be `0+1`:

```
> $:inc
1
```

The placeholder value, as you saw in the previous module, is sometimes called the bunt value. The bunt value is determined by the input type; for `@` atoms the bunt value is typically `0`.

The face value of `a` comes from the way we defined the gate above: `|=(a=@ (add 1 a))`. This was so we can use `a` to refer to the sample to generate the product with `(add 1 a)`.

#### The Context

The context of a gate contains other data that may be necessary for the `$` buc arm to evaluate correctly. The context is always located at the tail of the tail of the gate, i.e., `+7` of the gate. There is no requirement for the context to have any particular arrangement, though often it does.

Let's look at the context of `+inc`:

```
> +7:inc
[ [ our=~nec
    now=~2024.5.8..17.14.52..ef1e
      eny
    0v304.vhjvs.406g0.bn6ph.ggd02.buadd.2lot0.va6q0.fiqb1.a96gj.9jmb2.6kk07.5d75s.thpbg.9idrt.vmg9j.e748l.fea0l.7ckcf.ieesj.7q6lr
  ]
  <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
]
```

This is the default Dojo subject from before we put `+inc` into the subject. The `|=` [bartis](/hoon/rune/bar#bartis) expression defines the context as whatever the subject is. This guarantees that the context has all the information it needs to have for the `$` buc arm to work correctly.

#### Gates Define Functions of the Sample

The value of a function's output depends solely upon the input value. This is one of the features that make functions desirable in many programming contexts. It's worth going over how Hoon function calls implement this feature.

In Hoon, one can use `(gate arg)` syntax to make a function call. For example:

```
> (inc 234)
235
```

The name of the gate is `+inc`. How is the `$` buc arm of inc evaluated? When a function call occurs, a copy of the `+inc` gate is created, but with one modification: the sample is replaced with the function argument. Then the `$` buc arm is computed against this modified version of the `+inc` gate.

Remember that the default (or “bunt”) value of the sample of inc is `0`. In the function call above, a copy of the `+inc` gate is made but with a sample value of `234`. When `$` buc is computed against this modified core, the product is `235`.

Notice that neither the arm nor the context is modified before the arm is evaluated. That means that the only part of the gate that changes before the arm evaluation is the sample. Hence, we may understand each gate as defining a function whose argument is the sample. If you call a gate with the same sample, you'll get the same value returned to you every time.

Let's unbind inc to keep the subject tidy:

```
> =inc

> inc
-find.inc
```

#### Modifying the Context of a Gate

It is possible to modify the context of a gate when you make a function call; or, to be more precise, it's possible to call a *mutant copy* of the gate in which the context is modified. To illustrate this let's use another example gate. Let's write a gate which uses a value from the context to generate the product. Bind face `b` to the value 10:

```
> =b 10

> b
10
```

Now let's write a gate called `ten` that adds `.b` to the input value:

```
> =ten |=(a=@ (add a b))

> (ten 10)
20

> (ten 20)
30

> (ten 25)
35
```

We can unbind `.b` from the Dojo subject, and `.ten` works just as well because it's using a copy of `.b` stored its context:

```
> =b

> (ten 15)
25

> (ten 35)
45

> b.+14.ten
10
```

We can use `ten(b 25)` to produce a variant of `.ten`. Calling this mutant version of ten causes a different value to be returned than we'd get with a normal `.ten` call:

```
> (ten(b 25) 10)
35

> (ten(b 1) 25)
26

> (ten(b 75) 100)
175
```

Before finishing the lesson let's unbind `.ten`:

```
> =ten
```

### Recursion <a href="#recursion" id="recursion"></a>

"Recursion" refers to a return to the same logical point in a program again and again. It's a common pattern for solving certain problems in most programming languages, and Hoon is no exception.

In the following code, the `|-` [barhep](/hoon/rune/bar#barhep) trap serves as the point of recursion, and the return to that point (with changes) is indicated by the `%=` centis. All this code does is count to the given number, then return that number.

```hoon
|=  n=@ud
=/  index  0
|-
?:  =(index n)
  index
%=($ index +(index))
```

We are using the `+` irregular syntax for the `.+` [dotlus](/hoon/rune/dot#dotlus) rune, which increments a value (adds one).

In a formal sense, we have to make sure that there is always a base case, a way of actually ending the recursion. If there isn't, we end up with an [infinite loop](https://en.wikipedia.org/wiki/Infinite_loop)! Some children's songs like [“Yon Yonson”](https://en.wikipedia.org/wiki/Yon_Yonson) or [“The Song That Never Ends”](https://en.wikipedia.org/wiki/The_Song_That_Never_Ends) rely on such recursive humor.

> This is the song that never ends\
> Yes, it goes on and on, my friends\
> Some people started singing it not knowing what it was\
> And they′ll continue singing it forever just because—
>
> This is the song that never ends\
> . . .

You need to make sure when you compose a trap that it has a base case which returns a noun. The following trap results in an infinite loop:

```hoon
=/  index  1
|-
?:  (lth index 1)  ~
$(index +(index))
```

If you find yourself caught in such a loop, press `Ctrl+C` to stop execution.

Recursion can be set up different ways. A full treatment requires thinking about [algorithmic complexity and efficiency](https://en.wikipedia.org/wiki/Big_O_notation), but we can highlight some good rules of thumb here.

#### Tutorial: The Fibonacci Sequence

For instance, let's talk about calculating the [Fibonacci sequence](https://en.wikipedia.org/wiki/Fibonacci_sequence), which is a sequence of numbers wherein each is formed by adding the two previous numbers together. Thus 1, 1, 1+1→2, 1+2→3, 2+3→5, and so forth. We may write the $$n^\text{th}$$ Fibonacci number in a generic way as:

$$
F\_n = F\_{n-1} + F\_{n-2}
$$

and verify that our program correctly produces the sequence of numbers 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ….

Compose a Fibonacci sequence program which produces a list of the appropriate values.

We can elide some details of working with `+list`s until the next lesson; simply recall that they are a way of storing multiple values in a cell of cells of cells….

The most naïve version of this calculation simply calculates all previous numbers in the sequence every time they are needed.

```hoon
|=  n=@ud
^-  @ud
?:  =(n 1)  1
?:  =(n 2)  1
(add $(n (dec n)) $(n (dec (dec n))))
```

We can use *two* recursion points for `%=` [centis](/hoon/rune/cen#centis). The first calculate $$F$$ for $$n-1$$; the second calculate $$F$$ for $$n-2$$. These are then added together. If we diagram what's happening, we can see that each additional number costs as much as the previous numbers:

```hoon
(fibonacci 5)
(add (fibonacci 4) (fibonacci 3))
(add (add (fibonacci 3) (fibonacci 2)) (add (fibonacci 2) (fibonacci 1)))
(add (add (add (fibonacci 2) (fibonacci 1)) (fibonacci 2)) (add (fibonacci 2) (fibonacci 1)))
(add (add (add 1 1) 1) (add 1 1))
5
```

```hoon
(fibonacci 6)
(add (fibonacci 5) (fibonacci 4))
...
(add (add (add (add (fibonacci 2) (fibonacci 1)) (fibonacci 2)) (add (fibonacci 2) (fibonacci 1))) (add (add (fibonacci 2) (fibonacci 1)) (fibonacci 2)))
(add (add (add (add 1 1) 1) (add 1 1)) (add (add 1 1) 1))
8
```

This fully recursive version of the Fibonacci calculation is very wasteful because it keeps no intermediate results.

An improved version stores each value in the sequence as an element in a list so that it can be used rather than re-calculated. We use the [`+snoc`](/hoon/stdlib/2b#snoc) gate to append a noun to a `+list`.

```hoon
|=  n=@ud
=/  index  0
=/  p  0
=/  q  1
=/  r  *(list @ud)
|-  ^-  (list @ud)
?:  =(index n)  r
~&  >  [index p q r]
%=  $
  index  +(index)
  p      q
  q      (add p q)
  r      (snoc r q)
==
```

(As in an earlier code example, `(add index 1)` can be replaced by the Nock increment rune, `.+` [dotlus](/hoon/rune/dot#dotlus).)

This version is a little more complicated to compare using a diagram because of the trap, but yields something like this:

```hoon
(fibonacci 5)
~[1]
~[1 1]
~[1 1 2]
~[1 1 2 3]
~[1 1 2 3 5]
```

The program can be improved somewhat again by appending to the head of the cell (rather than using `+snoc`). This builds a list in a backwards order, so we apply the [`+flop`](/hoon/stdlib/2b#flop) gate to flip the order of the list before we return it.

```hoon
|=  n=@ud
%-  flop
=/  index  0
=/  p  0
=/  q  1
=/  r  *(list @ud)
|-  ^-  (list @ud)
?:  =(index n)  r
%=  $
  index  +(index)
  p      q
  q      (add p q)
  r      [q r]
==
```

Why are we building the list backwards instead of just producing the list in the order we want it in the first place? Because with lists, adding an element to the end is a computationally expensive operation that gets more expensive the longer the list is, due to the fact that you need to traverse to the end of the tree. Adding an element to the front, however, is cheap. In Big-O notation, adding to the end of a list is *O*(*n*) while adding to the front is *O*(1).

Here's our diagram:

```hoon
(fibonacci 5)
~[1]
~[1 1]
~[2 1 1]
~[3 2 1 1]
~[5 3 2 1 1]
~[1 1 2 3 5]
```

Finally (and then we'll move along) here's a very efficient implementation, which starts with a `0` but builds the list entirely from cells, then appends the `~` `0` at the end:

```hoon
|=  n=@ud
^-  (list @ud)
=/  f0  *@ud
=/  f1=@ud  1
:-  0
|-  ^-  (list @ud)
?:  =(n 0)
  ~
[f1 $(f0 f1, f1 (add f0 f1), n (dec n))]
```

Produce a diagram of how this last implementation yields a Fibonacci sequence for *F*₅, `(fibonacci 5)`.

#### Tutorial: Tail-Call Optimization of the Factorial Gate

The last factorial gate we produced looked like this:

```hoon
|=  n=@ud
?:  =(n 1)
  1
(mul n $(n (dec n)))
```

This example isn't a very efficient use of computing resources. The pyramid-shaped illustration from up above approximates what's happening on the "call stack", a memory structure that tracks the instructions of the program. In our example code, every time a parent gate calls another gate, the gate being called is "pushed" to the top of the stack in the form of a frame. This process continues until a value is produced instead of a function, completing the stack.

```
                  Push order      Pop order
(fifth frame)         ^               |
(fourth frame)        |               |
(third frame)         |               |
(second frame)        |               |
(first frame)         |               V
```

Once this stack of frames is completed, frames "pop" off the stack starting at the top. When a frame is popped, it executes the contained gate and passes produced data to the frame below it. This process continues until the stack is empty, giving us the gate's output.

When a program's final expression uses the stack in this way, it's considered to be **not tail-recursive**. This usually happens when the last line of executable code calls more than one gate, our example code's `(mul n $(n (dec n)))` being such a case. That's because such an expression needs to hold each iteration of `$(n (dec n)` in memory so that it can know what to run against the `+mul` function every time.

To reiterate: if you have to manipulate the result of a recursion as the last expression of your gate, as we did in our example, the function is not tail-recursive, and therefore not very efficient with memory. A problem arises when we try to recurse more times than we have space on the stack. This will result in our computation failing and producing a stack overflow. If we tried to find the factorial of `5.000.000`, for example, we would almost certainly run out of stack space.

But the Hoon compiler, like most compilers, is smart enough to notice when the last statement of a parent can reuse the same frame instead of needing to add new ones onto the stack. If we write our code properly, we can use a single frame that simply has its values replaced with each recursion.

Change the order of the aspects of the call in such a way that the compiler can produce a more [tail-recursive](https://en.wikipedia.org/wiki/Tail_call) program.

With a bit of refactoring, we can write a version of our factorial gate that is tail-recursive and can take advantage of this feature:

```hoon
|=  n=@ud
=/  t=@ud  1
|-
^-  @ud
?:  =(n 1)  t
$(n (dec n), t (mul t n))
```

The above code should look familiar. We are still building a gate that takes one argument a `@ud` unsigned decimal integer `.n`. The `|-` here is used to create a new gate with one arm `$` and immediately call it. As before, think of `|-` as the recursion point.

We then evaluate `.n` to see if it is 1. If it is, we return the value of `.t`. In case that `.n` is anything other than 1, we perform our recursion:

```hoon
$(n (dec n), t (mul t n))
```

All we are doing here is recursing our new gate and modifying the values of `.n` and `.t`. `.t` is used as an accumulator variable that we use to keep a running total for the factorial computation.

Let's use more of our pseudo-Hoon to illustrate how the stack is working in this example for the factorial of 5.

```hoon
(factorial 5)
(|- 5 1)
(|- 4 5)
(|- 3 20)
(|- 2 60)
(|- 1 120)
120
```

We simply multiply `.t` and `.n` to produce the new value of `.t`, and then decrement `.n` before repeating. Since this `$` call is the final and solitary thing that is run in the default case and since we are doing all computation before the call, this version is properly tail-recursive. We don't need to do anything to the result of the recursion except recurse it again. That means that each iteration can be replaced instead of held in memory.

#### Tutorial: The Ackermann Function

The [Ackermann function](https://en.wikipedia.org/wiki/Ackermann_function) is one of the earliest examples of a function that is both totally computabl, meaning that it can be solved, and not primitively recursive, meaning it can not be rewritten in an iterative fashion.

$$
\begin{array}{lcl}
\operatorname{A}(0, n) & = & n + 1 \\\\
\operatorname{A}(m+1, 0) & = & \operatorname{A}(m, 1) \\\\
\operatorname{A}(m+1, n+1) & = & \operatorname{A}(m, \operatorname{A}(m+1, n))
\end{array}
$$

Compose a gate that computes the Ackermann function.

```hoon
|=  [m=@ n=@]
^-  @
?:  =(m 0)  +(n)
?:  =(n 0)  $(m (dec m), n 1)
$(m (dec m), n $(n (dec n)))
```

This gate accepts two arguments of `@` atom type and yields an atom.

There are three cases to consider:

1. If `.m` is zero, return the increment of `.n`.
2. If `.n` is zero, decrement `.m`, set `.n` to 1 and recurse.
3. Else, decrement `.m` and set `.n` to be the value of the Ackermann function with `.n` and the decrement of `.n` as arguments.

The Ackermann function is not terribly useful in and of itself, but it has an interesting history in mathematics. When running this function the value grows rapidly even for very small input. The value of computing this where `.m` is `4` and `.n` is `2` is an integer with 19,729 digits.

Calculate some of the $$m$$/$$n$$ pairs given in [the table](https://en.wikipedia.org/wiki/Ackermann_function#Table_of_values).

### Exercise: The Sudan Function <a href="#exercise-the-sudan-function" id="exercise-the-sudan-function"></a>

The [Sudan function](https://en.wikipedia.org/wiki/Sudan_function) is related to the Ackermann function.

$$
\begin{array}{lll}
F\_0 (x, y) & = x+y \\\\
F\_{n+1} (x, 0) & = x & \text{if } n \ge 0 \\\\
F\_{n+1} (x, y+1) & = F\_n (F\_{n+1} (x, y), F\_{n+1} (x, y) + y + 1) & \text{if } n\ge 0
\end{array}
$$

Implement the Sudan function as a gate.


# 6. Trees and Addressing

Understanding Urbit's binary tree data representation, tree addressing systems, data location and evaluation, the \`+list\` mold, and standard library operations for working with tree-structured data.

*Every noun in Urbit is an atom or a cell. This module will elaborate how we can use this fact to locate data and evaluate code in a given expression. It will also discuss the important `+list` mold builder and a number of standard library operations.*

## Trees <a href="#trees" id="trees"></a>

Every noun in Urbit is a either an atom or a cell. Since a cell has only two elements, a head and a tail, we can derive that everything is representable as a [*binary tree*](https://en.wikipedia.org/wiki/Binary_tree). We can draw this layout naturally:

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree.png)

A binary tree has a single base node, and each node of the tree may have up to two child nodes (but it need not have any). A node without children is a “leaf”. You can think of a noun as a binary tree whose leaves are atoms, i.e., unsigned integers. All non-leaf nodes are cells. An atom is a trivial tree of just one node; e.g., `17`.

For instance, if we produce a cell in the Dojo

```
> =a [[[8 9] [10 11]] [[12 13] [14 15]]]
```

it can be represented as a tree with the contents

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-bottom-row.png)

We will use the convention in these graphics that black-text-on-white-circle represents an address, and that green-text-on-black-circle represents the content at that address. So another way to represent the same data would be this:

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-bottom-row-full.png)

When we input the above cell representation into the Dojo, the pretty-printer hides the rightwards-branching square brackets.

```
> [[[8 9] [10 11]] [[12 13] [14 15]]]
[[[8 9] 10 11] [12 13] 14 15]
```

We can refer to any data stored anywhere in this tree. The numbers in the labeled diagram above are the *numerical addresses* of the tree, and may be extended indefinitely downwards into ever-deeper tree representations.

Most of any possible tree will be unoccupied for any actual data structure. For instance, `+list`s (and thus `$tape`s) are collections of values which occupy the tails of cells, leading to a rightwards-branching tree representation. (Although this may seem extravagant, it has effectively no bearing on efficiency in and of itself; that's a function of the algorithms working with the data.)

### Exercise: Map Nouns to Tree Diagrams <a href="#exercise-map-nouns-to-tree-diagrams" id="exercise-map-nouns-to-tree-diagrams"></a>

Consider each of the following nouns. Which tree diagram do they correspond to? (This is a matching exercise.)

| Noun               | Tree Diagram                                                                          |
| ------------------ | ------------------------------------------------------------------------------------- |
| 1. `[[[1 2] 3] 4]` | A. ![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-exercise-1.png) |
| 2. `[[1 2] 3 4]`   | B. ![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-exercise-2.png) |
| 3. `[1 2 3 4]`     | C. ![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-exercise-3.png) |

### Exercise: Produce a List of Numbers <a href="#exercise-produce-a-list-of-numbers" id="exercise-produce-a-list-of-numbers"></a>

Produce a generator called `/gen/list.hoon` which accepts a single `@ud` number *n* as input and produces a list of numbers from `1` up to (but not including) *n*. For example, if the user provides the number `5`, the program will produce: `~[1 2 3 4]`.

```hoon
|=  end=@
=/  count=@  1
|-
^-  (list @)
?:  =(end count)
  ~
:-  count
$(count (add 1 count))
```

In the Dojo:

```
> +list 5
~[1 2 3 4]

> +list 10
~[1 2 3 4 5 6 7 8 9]

> +list 1
~
```

OK, we've seen these runes before. This time we want to focus on the list, the thing that's being built here.

This program works by having each iteration of the list create a cell. In each of these cells, the head (the cell's first position) is filled with the current-iteration value of `.count`. The tail of the cell, its second position, is filled with *the product of a new iteration of our code* that starts at `|-`. This iteration will itself create another cell, the head of which will be filled by the incremented value of `.count`, and the tail of which will start another iteration. This process continues until `?:` branches to `~` (pronounced "null"). When that happens, it terminates the list and the expression ends. A built-out list of nested cells can be visualized like this:

```
[1 [2 [3 [4 ~]]]]

  .
 / \
1   .
   / \
  2   .
     / \
    3   .
       / \
      4   ~
```

### Tuples as Trees <a href="#tuples-as-trees" id="tuples-as-trees"></a>

What we've been calling a running cell would more conventionally be named a "tuple", so we'll switch to that syntax now that the idea is more familiar. Basically it's a cell series which doesn't necessarily end in `~`.

Given the cell `[1 2 3 4 ~]` (or equivalently `~[1 2 3 4]`, an irregular form for a null-terminated tuple or list), what tree address does each value occupy?

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-1234.png)

At this point, you should start to be able to work this out in your head, at least for the first few rows. The `+` operator can be used to return the limb of the subject at a given numeric address. If there is no such limb, the result is a crash.

```
> =data ~[1 2 3 4]

> +1:data
[1 2 3 4 ~]

> +2:data
1

> +3:data
[2 3 4 ~]

> +4:data
dojo: hoon expression failed

> +6:data
2

> +7:data
[3 4 ~]

> +14:data
3

> +15:data
[4 ~]

> +30:data
4

> +31:data
~
```

### Lists as Trees <a href="#lists-as-trees" id="lists-as-trees"></a>

We have used `+list`s incidentally. A `+list` is an ordered arrangement of elements ending in a `~`. Most lists have the same kind of content in every element (for instance, a `(list @rs)`, a list of numbers with a fractional part), but some lists have many kinds of things within them. Some lists are even empty.

```
> `(list @)`['a' %b 100 ~]
~[97 98 100]
```

(Notice that all values are converted to the specified aura, in this case the empty aura.)

A `+list` is built with the `+list` mold. **A `+list` is actually a "mold builder", a gate that produces a gate. This is a common design pattern in Hoon. (Remember that a `$mold` is a type and can be used as an enforcer: it attempts to convert any data it receives into the given structure, and crashes if it fails to do so.)** Lists are commonly written with a shorthand `~[]`:

```
> `(list)`~['a' %b 100]
~[97 98 100]
```

```
> `(list (list @ud))`~[~[1 2 3] ~[4 5 6]]
~[~[1 2 3] ~[4 5 6]]
```

True `+list`s have `i` and `t` faces which allow the head and tail of the data to be quickly and conveniently accessed; the "head" is the first element while the "tail" is everything else. **If something has the same&#x20;*****structure*****&#x20;as a `+list` but hasn't been explicitly labeled as such, then Hoon won't always recognize it as a `+list`. In such cases, you'll need to explicitly mark it as such:**

```
> [3 4 5 ~]
[3 4 5 ~]

> `(list @ud)`[3 4 5 ~]
~[3 4 5]

> -:!>([3 4 5 ~])
#t/[@ud @ud @ud %~]

> -:!>(`(list @ud)`[3 4 5 ~])
#t/it(@ud)
```

A null-terminated tuple is almost the same thing as a list. (That is, to Hoon all lists are null-terminated tuples, but not all null-terminated tuples are lists. This gets rather involved in subtleties, but you should cast a value as `(list @)` or another type as appropriate whenever you need a `+list`. See also [`+limo`](/hoon/stdlib/2b#limo) which explicitly marks a null-terminated tuple as a `+list`.)

## Addressing Limbs <a href="#addressing-limbs" id="addressing-limbs"></a>

Everything in Urbit is a binary tree. And all code in Urbit is also represented as data. One corollary of these facts is that we can access any arbitrary part of an expression, gate, core, whatever, via addressing (assuming proper permissions, of course). (In fact, we can even hot-swap parts of cores, which is how [wet gates](/build-on-urbit/hoon-school/r-metals#wet-gates) work.)

There are three different ways to access values:

1. [Numeric addressing](#numeric-addressing) is useful when you know the address, rather like knowing a house's street address directly.
2. [Positional addressing](#positional-addressing-lark-notation) is helpful when you don't want to figure out the room number, but you know how to navigate to the value. This is like knowing the directions somewhere even if you don't know the house number.
3. [Wing addressing](#wings) is a way of attaching a name to the address so that you can access it directly.

### Numeric Addressing <a href="#numeric-addressing" id="numeric-addressing"></a>

We have already seen numeric addressing used to refer to parts of a binary tree.

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree.png)

Since a node is *either* an atom (value) *or* a cell (fork), you never have to decide if the contents of a node is a direct value or a tree: it just happens.

### Exercise: Tapes for Text <a href="#exercise-tapes-for-text" id="exercise-tapes-for-text"></a>

A tape is one way of representing a text message in Hoon. It is written with double quotes:

```hoon
"I am the very model of a modern Major-General"
```

A `$tape` is actually a `(list @tD)`, a binary tree of single characters which only branches rightwards and ends in a `~`:

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-tape.png)

What are the addresses of each letter in the tree for the Gilbert & Sullivan quote above? Can you see the pattern?

### Positional Addressing (Lark Notation) <a href="#positional-addressing-lark-notation" id="positional-addressing-lark-notation"></a>

Much like relative directions, one can also state “left, left, right, left” or similar to locate a particular node in the tree. These are written using `-` (left) and `+` (right) alternating with `<` (left) and `>` (right).

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-lark.png)

Lark notation can locate a position in a tree of any size. However, it is most commonly used to grab the head or tail of a cell, e.g. in the "type spear" (on which [more later](/build-on-urbit/hoon-school/m-typecheck)):

```hoon
-:!>('hello Mars')
```

Lark notation is not preferred in modern Hoon for more than one or two elements deep, but it can be helpful when working interactively with a complicated data structure like a JSON data object.

When lark expressions resolve to the part of the subject containing an arm, they don't evaluate the arm. They simply return the indicated noun fragment of the subject, as if it were a leg.

### Exercise: Address the Fruit Tree <a href="#exercise-address-the-fruit-tree" id="exercise-address-the-fruit-tree"></a>

Produce the numeric and lark-notated equivalent addresses for each of the following nodes in the binary fruit tree:

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-fruit.png)

* 🍇
* 🍌
* 🍉
* 🍏
* 🍋
* 🍑
* 🍊
* 🍍
* 🍒

There is a solution at the bottom of the page.

### Exercise: Lark Notation <a href="#exercise-lark-notation" id="exercise-lark-notation"></a>

Use a lark expression to obtain the value 6 in the following noun represented by a binary tree:

```
          .
         / \
        /   \
       /     \
      .       .
     / \     / \
    /   .   10  .
   /   / \     / \
  .   8   9   11  .
 / \             / \
5   .           12  13
   / \
  6   7
```

Use a lark expression to obtain the value `9` in the following noun: `[[[5 6 7] 8 9] 10 11 12 13]`.

Solutions to these exercises may be found at the bottom of this lesson.

## Wings <a href="#wings" id="wings"></a>

One can also identify a resource by a label, called a wing. A wing represents a depth-first search into the current subject (context). A wing is a limb resolution path into the subject. A wing expression indicates the path as a series of limb expressions separated by the `.` character. E.g.,

```hoon
inner-limb.outer-limb.limb
```

You can read this as `.inner-limb` in `.outer-limb` in `.limb`, etc. Notice that these read left-to-right!

A wing is a resolution path pointing to a limb. It's a search path, like an index to a particular labeled part of the subject.

Here are some examples:

```
> c.b:[[4 a=5] b=[c=14 15]]
14

> b.b:[b=[a=1 b=2 c=3] a=11]
2

> a.b:[b=[a=1 b=2 c=3] a=11]
1

> c.b:[b=[a=1 b=2 c=3] a=11]
3

> a:[b=[a=1 b=2 c=3] a=11]
11

> b.a:[b=[a=1 b=2 c=3] a=11]
-find.b.a

> g.s:[s=[c=[d=12 e='hello'] g=[h=0xff i=0b11]] r='howdy']
[h=0xff i=0b11]

> c.s:[s=[c=[d=12 e='hello'] g=[h=0xff i=0b11]] r='howdy']
[d=12 e='hello']

> e.c.s:[s=[c=[d=12 e='hello'] g=[h=0xff i=0b11]] r='howdy']
'hello'

> +3:[s=[c=[d=12 e='hello'] g=[h=0xff i=0b11]] r='howdy']
r='howdy'

> r.+3:[s=[c=[d=12 e='hello'] g=[h=0xff i=0b11]] r='howdy']
'howdy'
```

To locate a value in a named tuple data structure:

```
> =data [a=[aa=[aaa=[1 2] bbb=[3 4]] bb=[5 6]] b=[7 8]]

> -:aaa.aa.a.data
1
```

A wing is a limb resolution path into the subject. This definition includes as a trivial case a path of just one limb. Thus, all limbs are wings, and all limb expressions are wing expressions.

We mention this because it is convenient to refer to all limbs and non-trivial wings as simply “wings”.

### Names and Faces <a href="#names-and-faces" id="names-and-faces"></a>

A name can resolve either an arm or a leg of the subject. Recall that arms are for computations and legs are for data. When a name resolves to an arm, the relevant computation is run and the product of the computation is produced. When a limb name resolves to a leg, the value of that leg is produced.

Hoon doesn't have variables quite like other programming languages do. And instead of variable names, it has "faces". If you're coming in from antoher programming langauge, you can think of faces as variable names and faced values as named variables and you'll usually be okay. Faces play various roles in Hoon, but usually they're just labels for legs in the subject.

A face is a limb expression that consists of a series of alphanumeric characters. A face may contain a combination of lowercase letters, numbers, and `-`s. Faces must begin with a letter. Some example faces are `b`, `c3`, `var`, `this-is-kebab-case123`.

There are various ways to assign (or "pin") a face to a limb of the subject, but for now we'll just use the simplest method: `face=value`. Hereafter in the code, the expression `face` would be equivalent to the product of whatever code we gave as the `value`. This works because Hoon registers the given `face` as metadata about where the `value` is stored in the subject, so that when the `face` is invoked later, its data is produced through wing resolution. This means we can reference any computed result (be that code or data) in the subject by the `face` we assigned to it, which gives us some nice flexibility around code style.

Now we have several ways to access values:

```
> b=5
b=5

> [b=5 cat=6]
[b=5 cat=6]

> -:[b=5 cat=6]
b=5

> b:[b=5 cat=6]
5

> b2:[[4 b2=5] [cat=6 d=[14 15]]]
5

> d:[[4 b2=5] [cat=6 d=[14 15]]]
[14 15]
```

To be clear, `b=5` is equivalent in value to `5`, and `[[4 b2=5] [cat=6 d=[14 15]]]` is equivalent in value to `[[4 5] 6 14 15]`. The faces are not part of the underlying noun, they're stored as metadata about address values in the subject.

```
> (add b=5 1)
6
```

If you reference `face` and it isn't in the subject, you'll get a `-find.face` error:

```
> a:[b=12 c=14]
-find.a
[crash message]
```

You can even give faces to faces:

```
> b:[b=c=123 d=456]
c=123
```

### Duplicate Faces <a href="#duplicate-faces" id="duplicate-faces"></a>

There is no restriction against using the same face name for multiple limbs of the subject. This is one way in which faces aren't like ordinary variables:

```
> [[4 b=5] [b=6 b=[14 15]]]
[[4 b=5] b=6 b=[14 15]]

> b:[[4 b=5] [b=6 b=[14 15]]]
5
```

Why does this return `5` rather than `6` or `[14 15]`? When a face is evaluated on a subject, a head-first binary tree search occurs starting at address `1` of the subject. If there is no matching face for address *n* of the subject, first the head of *n* is searched and then *n*'s tail. The complete search path for `[[4 b=5] [b=6 b=[14 15]]]` is:

1. `[[4 b=5] [b=6 b=[14 15]]]`
2. `[4 b=5]`
3. `4`
4. `b=5`
5. `[b=6 b=[14 15]]`
6. `b=6`
7. `b=[14 15]`

There are matches at steps 4, 6, and 7 of the total search path, but the search ends when the first match is found at step 4.

The children of legs bearing names aren't included in the search path. For example, the search path of `[[4 a=5] b=[c=14 15]]` is:

1. `[[4 a=5] b=[c=14 15]]`
2. `[4 a=5]`
3. `4`
4. `a=5`
5. `b=[c=14 15]`

Neither of the legs `c=14` or `15` is checked. Accordingly, a search for `.c` of `[[4 a=5] b=[c=14 15]]` fails:

```
> c:[[4 b=5] [b=6 b=[c=14 15]]]
-find.c [crash message]
```

In any programming paradigm, good names are valuable and collisions (repetitions, e.g. a list named `+list`) are likely. There is no restriction against using the same face name for multiple limbs of the subject. This is one way in which faces aren't like ordinary variables. If multiple values match a particular face, we need a way to distinguish them. In other words, there are cases when you don't want the limb of the first matching face. You can ‘skip’ the first match by prepending `^` to the face. Upon discovery of the first match at address *n*, the search skips *n* (as well as its children) and continues the search elsewhere:

```
> ^b:[[4 b=5] [b=6 b=[14 15]]]
6
```

Recall that the search path for this noun is:

1. `[[4 b=5] [b=6 b=[14 15]]]`
2. `[4 b=5]`
3. `4`
4. `b=5`
5. `[b=6 b=[14 15]]`
6. `b=6`
7. `b=[14 15]`

The second match in the search path is step 6, `b=6`, so the value at that leg is produced. You can stack `^` characters to skip more than one matching face:

```
> a:[[[a=1 a=2] a=3] a=4]
1

> ^a:[[[a=1 a=2] a=3] a=4]
2

> ^^a:[[[a=1 a=2] a=3] a=4]
3

> ^^^a:[[[a=1 a=2] a=3] a=4]
4
```

When a face is skipped at some address *n*, neither the head nor the tail of *n* is searched:

```
> b:[b=[a=1 b=2 c=3] a=11]
[a=1 b=2 c=3]

> ^b:[b=[a=1 b=2 c=3] a=11]
-find.^b
```

The first `.b`, `b=[a=1 b=2 c=3]`, is skipped; so the entire head of the subject is skipped. The tail has no `.b`; so `^b` doesn't resolve to a limb when the subject is `[b=[a=1 b=2 c=3] a=11]`.

How do you get to that `b=2`? And how do you get to the `.c` in `[[4 a=5] b=[c=14 15]]`? In each case you should use a wing.

We say that the inner face has been "shadowed" when an outer face obscures it.

If you run into `^$`, don't go looking for a nonexistent `^$` ketbuc rune: it's referencing the outer `$` buc arm. `^$` is one way of setting up a `%=` [centis](/hoon/rune/cen#centis) loop/recursion of multiple cores with a `|-` [barhep](/hoon/rune/bar#barhep) trap nested inside of a `|=` [bartis](/hoon/rune/bar#bartis) gate, for instance.

Solution #1 in the [Rhonda Numbers](/hoon/examples/rhonda) tutorial in the Hoon Workbook illustrates using `^` ket to skip `$` buc matches.

### Limb Resolution Operators <a href="#limb-resolution-operators" id="limb-resolution-operators"></a>

There are two symbols we use to search for a face or limb:

* `.` dot resolves the wing path into the current subject.
* `:` col resolves the wing path with the right-hand-side as the subject.

Logically, `a:b` is two operations, while `a.b` is one operation. The compiler is smart about `:` col wing resolutions and reduces it to a regular lookup, though.

### What `%=` Does <a href="#what-does" id="what-does"></a>

Now we're equipped to go back and examine the syntax of the `%=` [centis](/hoon/rune/cen#centis) rune we have been using for recursion: it *resolves a wing with changes*, which in this particular case means that it takes the `$` (default) arm of the trap core, applies certain changes, and re-evaluates the expression.

```hoon
|=  n=@ud
|-
~&  n
?:  =(n 1)
  n
%+  mul
  n
$(n (dec n))
```

The `$()` syntax is the commonly-used irregular form of the `%=` [centis](/hoon/rune/cen#centis) rune.

Now, we noted that `$` buc is the default arm for the trap. It turns out that `$` is also the default arm for some other structures, like the gate! That means we can cut out the trap, in the factorial example, and write something more compact like this:

```hoon
|=  n=@ud
?:  =(n 1)
  1
(mul n $(n (dec n)))
```

It's far more common to just use a trap, but you will see `$` buc used to manipulate a core in many in-depth code instances.

### Expanding the Runes <a href="#expanding-the-runes" id="expanding-the-runes"></a>

`|=` [bartis](/hoon/rune/bar#bartis) produces a gate. It actually expands to

```hoon
=|  a=spec
|%  ++  $  b=hoon
--
```

where `=|` [tisbar](/hoon/rune/tis#tisbar) means to add its sample to the current subject with the given face.

Similarly, `|-` [barhep](/hoon/rune/bar#barhep) produces a core with one arm `$`. How could you write that in terms of `|%` and `++`?

### Example: Number to Digits <a href="#example-number-to-digits" id="example-number-to-digits"></a>

Compose a generator which accepts a number as `@ud` unsigned decimal and returns a list of its digits.

One verbose Hoon program

```hoon
!:
|=  [n=@ud]
=/  values  *(list @ud)
|-  ^-  (list @ud)
?:  (lte n 0)  values
%=  $
  n       (div n 10)
  values  (weld ~[(mod n 10)] values)
==
```

Save this as a file `/gen/num2dig.hoon`, `|commit %base`, and run it:

```
> +num2dig 1.000
~[1 0 0 0]

> +num2dig 123.456.789
~[1 2 3 4 5 6 7 8 9]
```

A more idiomatic solution would use the `^` ket infix to compose a cell and build the list from the head first. (This saves a call to [`+weld`](/hoon/stdlib/2b#weld).)

```hoon
!:
|=  [n=@ud]
=/  values  *(list @ud)
|-  ^-  (list @ud)
?:  (lte n 0)  values
%=  $
  n       (div n 10)
  values  (mod n 10)^values
==
```

A further tweak maps to `@t` ASCII characters instead of the digits.

```hoon
!:
|=  [n=@ud]
=/  values  *(list @t)
|-  ^-  (list @t)
?:  (lte n 0)  values
%=  $
  n       (div n 10)
  values  (@t (add 48 (mod n 10)))^values
==
```

(Notice that we apply `@t` as a mold gate rather than using the tic notation. This is because `^` ket is a rare case where the order of evaluation of operators would cause the intuitive writing to fail.)

Extend the above generator so that it accepts a cell of type and value (a `$vase` as produced by the `!>` [zapgar](/hoon/rune/zap#zapgar) rune). Use the type to determine which number base the digit string should be constructed from; e.g. `+num2dig !>(0xdead.beef)` should yield `~['d' 'e' 'a' 'd' 'b' 'e' 'e' 'f']`.

### Exercise: Resolving Wings <a href="#exercise-resolving-wings" id="exercise-resolving-wings"></a>

Enter the following into dojo:

```hoon
=a [[[b=%bweh a=%.y c=8] b="no" c="false"] 9]
```

Test your knowledge from this lesson by evaluating the following expressions and then checking your answer in the Dojo or see the solutions below.

1. `b:a(a [b=%skrt a="four"])`
2. `^b:a(a [b=%skrt a="four"])`
3. `^^b:a(a [b=%skrt a="four"])`
4. `b.a:a(a [b=%skrt a="four"])`
5. `a.a:a(a [b=%skrt a="four"])`
6. `+.a:a(a [b=%skrt a="four"])`
7. `a:+.a:a(a [b=%skrt a="four"])`
8. `a(a a)`
9. `b:-<.a(a a)`
10. How many times does the atom `9` appear in `a(a a(a a))`?

The answers are at the bottom of the page.

## List operations <a href="#list-operations" id="list-operations"></a>

Once you have your data in the form of a `+list`, there are a lot of tools available to manipulate and analyze the data:

The [`+flop`](/hoon/stdlib/2b#flop) function reverses the order of the elements (exclusive of the `~`):

```
> (flop ~[1 2 3 4 5])
~[5 4 3 2 1]
```

### Exercise: `+flop` Yourself

Without using `+flop`, write a gate that takes a `(list @)` and returns it in reverse order. There is a solution at the bottom of the page.

The [`+sort`](/hoon/stdlib/2b#sort) function uses a `+list` and a comparison function (like [`+lth`](/hoon/stdlib/1a#lth)) to order things:

```
> (sort ~[1 3 5 2 4] lth)
~[1 2 3 4 5]
```

The [`+snag`](/hoon/stdlib/2b#snag) function takes an index and a `+list` to grab out a particular element (note that it starts counting at zero):

```
> (snag 0 `(list @)`~[11 22 33 44])
11

> (snag 1 `(list @)`~[11 22 33 44])
22

> (snag 3 `(list @)`~[11 22 33 44])
44

> (snag 3 "Hello!")
'l'

> (snag 1 "Hello!")
'e'

> (snag 5 "Hello!")
'!'
```

The [`+weld`](/hoon/stdlib/2b#weld) function takes two lists of the same type and concatenates them:

```
> (weld ~[1 2 3] ~[4 5 6])
~[1 2 3 4 5 6]

> (weld "Happy " "Birthday!")
"Happy Birthday!"
```

### Exercise: `+weld` Yourself

Without using weld, write a gate that takes a `[(list @) (list @)]` of which the product is the concatenation of these two lists. There is a solution at the bottom of the page.

There are a couple of sometimes-useful `+list` builders:

The [`+gulf`](/hoon/stdlib/2b#gulf) function spans between two numeric values (inclusive of both):

```
> (gulf 5 10)  
~[5 6 7 8 9 10]
```

The [`+reap`](/hoon/stdlib/2b#reap) function repeats a value many times in a `+list`:

```
> (reap 5 0x0)
~[0x0 0x0 0x0 0x0 0x0]

> (reap 8 'a')
<|a a a a a a a a|>

> `tape`(reap 8 'a')
"aaaaaaaa"

> (reap 5 (gulf 5 10))
~[~[5 6 7 8 9 10] ~[5 6 7 8 9 10] ~[5 6 7 8 9 10] ~[5 6 7 8 9 10] ~[5 6 7 8 9 10]]
```

The [`+roll`](/hoon/stdlib/2b#roll) function takes a list and a gate, and accumulates a value of the list items using that gate. For example, if you want to add or multiply all the items in a list of atoms, you would use roll:

```
> (roll `(list @)`~[11 22 33 44 55] add)
165

> (roll `(list @)`~[11 22 33 44 55] mul)
19.326.120
```

Once you have a `+list` (including a tape), there are a lot of manipulation tools you can use to extract data from it or modify it:

* The [`+lent`](/hoon/stdlib/2b#lent) function takes `[a=(list)]` and gets the number of elements (length) of the list
* The [`+find`](/hoon/stdlib/2b#find) function takes `[nedl=(list) hstk=(list)]` and locates a sublist (`.nedl`, needle) in the list (`.hstk`, haystack)
* The [`+snap`](/hoon/stdlib/2b#snap) function takes `[a=(list) b=@ c=*]` and replaces the element at an index in the list (zero-indexed) with something else
* The [`+scag`](/hoon/stdlib/2b#scag) function takes `[a=@ b=(list)]` and produces the first *a* elements from the front of the list
* The [`+slag`](/hoon/stdlib/2b#slag) function takes `[a=@ b=(list)]` and produces all elements of the list including and after the element at index *a*

There are a few more that you should pick up eventually, but these are enough to get you started.

Using what we know to date, most operations that we would do on a collection of data require a trap.

### Exercise: Evaluating Expressions <a href="#exercise-evaluating-expressions" id="exercise-evaluating-expressions"></a>

Without entering these expressions into the Dojo, what are the products of the following expressions?

```hoon
(lent ~[1 2 3 4 5])
(lent ~[~[1 2] ~[1 2 3] ~[2 3 4]])
(lent ~[1 2 (weld ~[1 2 3] ~[4 5 6])])
```

### Exercise: Welding Nouns <a href="#exercise-welding-nouns" id="exercise-welding-nouns"></a>

First, bind these faces.

```
=b ~['moon' 'planet' 'star' 'galaxy']
=c ~[1 2 3]
```

Determine whether the following Dojo expressions are valid, and if so, what they evaluate to.

```
> (weld b b)

> (weld b c)

> (lent (weld b c))

> (add (lent b) (lent c))
```

### Exercise: Palindrome <a href="#exercise-palindrome" id="exercise-palindrome"></a>

Write a gate that takes in a list `.a` and returns `%.y` if `.a` is a palindrome and `%.n` otherwise. You may use the [`+flop`](/hoon/stdlib/2b#flop) function.

## Solutions to Exercises <a href="#solutions-to-exercises" id="solutions-to-exercises"></a>

### Fruit Tree <a href="#fruit-tree" id="fruit-tree"></a>

* 🍇 `9` or `-<+`
* 🍌 `11` or `->+`
* 🍉 `12` or `+<-`
* 🍏 `16` or `-<-<`
* 🍋 `27` or `+<+>`
* 🍊 `30` or `+>+<`
* 🍑 `42` or `->->-`
* 🍒 `62` or `+>+>-`
* 🍍 `87` or `->->+>`

### Resolving Lark Expressions <a href="#resolving-lark-expressions" id="resolving-lark-expressions"></a>

```
> =b [[[5 6 7] 8 9] 10 11 12 13]

> -<+<:b
6
```

### Resolving Wing Expressions <a href="#resolving-wing-expressions" id="resolving-wing-expressions"></a>

1. `%bweh`
2. `"no"`
3. Error: `ford: %slim failed:`
4. `%skrt`
5. `"four"`
6. `a="four"` - Note that this is different from the above!
7. `"four"`
8. `[[[b=%bweh a=[[[b=%bweh a=%.y c=8] b="no" c="false"] 9] c=8] b="no" c="false"] 9]`
9. `%bweh`
10. `9` appears 3 times:

```
> a(a a(a a))
[[[ b=%bweh a [[[b=%bweh a=[[[b=%bweh a=%.y c=8] b="no" c="false"] 9] c=8] b="no" c="false"] 9] c=8] b="no" c="false"] 9]
```

### Roll-Your-Own-`+flop` <a href="#roll-your-own-flop" id="roll-your-own-flop"></a>

```hoon
::  /gen/flop.hoon
::
|=  a=(list @)
=|  b=(list @)
|-  ^-  (list @)
?~  a  b
$(b [i.a b], a t.a)
```

### Roll-Your-Own-`+weld` <a href="#roll-your-own-weld" id="roll-your-own-weld"></a>

```hoon
::  /gen/weld.hoon
::
|=  [a=(list @) b=(list @)]
|-  ^-  (list @)
?~  a  b
[i.a $(a t.a)]
```

### `+lent` expressions <a href="#lent-expressions" id="lent-expressions"></a>

Running each one in the Dojo:

```
> (lent ~[1 2 3 4 5])
5

> (lent ~[~[1 2] ~[1 2 3] ~[2 3 4]])
3

> (lent ~[1 2 (weld ~[1 2 3] ~[4 5 6])])
3
```

### `+weld` expressions <a href="#weld-expressions" id="weld-expressions"></a>

Running each one in the Dojo:

```
> (weld b b)
<|moon planet star galaxy moon planet star galaxy|>
```

The below will not run because `+weld` expects the elements of both lists to be of the same type, in this case a `(list @t)` or `(list @ud)`.

```
> (weld b c)
```

This also fails for the same reason, but it is important to note that in some languages that are more lazily evaluated, such an expression would still work since it would only look at the length of `.b` and `.c` and not worry about what the elements were. In that case, it would return `7`.

```
> (lent (weld b c))
```

We see here the correct way to find the sum of the length of two lists of unknown type.

```
> (add (lent b) (lent c))
7
```

### Palindrome <a href="#palindrome" id="palindrome"></a>

```hoon
::  palindrome.hoon
::
|=  a=(list)
=(a (flop a))
```


# 7. Libraries

Guide to creating, importing, and using libraries in Hoon programming. Covers library imports with the /+ faslus rune, helper cores, composition runes, and best practices for code organization.

*Libraries allow you to import and share processing code. This module will discuss how libraries can be produced, imported, and used.*

## Importing a Library <a href="#importing-a-library" id="importing-a-library"></a>

If you have only built generators, you will soon or later become frustrated with the apparent requirement that you manually reproduce helper cores and arms every time you need them in a different generator. Libraries are cores stored in `/lib` which provide access to arms and legs (operations and data). While the Hoon standard library is directly available in the regular subject, many other elements of functionality have been introduced by software authors.

### Building Code Generally <a href="#building-code-generally" id="building-code-generally"></a>

A generator gives us on-demand access to code, but it is helpful to load and use code from files while we work in the Dojo.

A conventional library import with `/+` [faslus](/hoon/rune/fas#faslus) will work in a generator or another file, but won't work in Dojo, so you can't use `/+` faslus interactively. The first line of many generators will include an import line like this:

```hoon
/+  number-to-words
```

Subsequent invocations of the core require you to refer to it by name:

**/gen/n2w\.hoon**

```hoon
/+  number-to-words
|=  n=@ud
(to-words:eng-us:number-to-words n)
```

Since `/` fas runes don't work in the Dojo, you need to instead use the [`-build-file`](/user-manual/os/dojo-tools#build-file) thread to load the code. Most commonly, you will do this with library code when you need a particular gate's functionality for interactive coding.

`-build-file` accepts a file path and returns the built operational code. For instance:

```hoon
> =ntw -build-file %/lib/number-to-words/hoon

> one-hundred:numbers:ntw  
100

> (to-words:eng-us:ntw 19)
[~ "nineteen"]
```

There are also a number of other import runes which make library, structure, and mark code available to you. For now, the only one you need to worry about is `/+` [faslus](/hoon/rune/fas#faslus).

For simplicity, everything we do will take place on the `%base` desk for now. We will learn how to create a library in a subsequent lesson.

### Exercise: Loading a Library <a href="#exercise-loading-a-library" id="exercise-loading-a-library"></a>

In a generator, load `/lib/number-to-words.hoon` using the `/+` [tislus](/hoon/rune/tis#tislus) rune. (This must take place at the very top of your file.)

Use this to produce a gate which accepts an unsigned decimal integer and returns the text interpretation of its increment.

## Helper Cores <a href="#helper-cores" id="helper-cores"></a>

Another common design pattern besides creating a library is to sequester core-specific behavior in a helper core, which sits next to the interface operations. Two runes are used to compose expressions together so that the subject has everything it needs to carry out the desired calculations.

* `=>` [tisgar](/hoon/rune/tis#tisgar) composes two expressions so that the first is included in the second's subject (and thus can see it).
* `=<` [tisgal](/hoon/rune/tis#tisgal) inverts the order of composition, allowing heavier helper cores to be composed after the core's logic but still be available for use.

Watch for these being used in generators and libraries over the next few modules.

### Exercise: A Playing Card Library <a href="#exercise-a-playing-card-library" id="exercise-a-playing-card-library"></a>

In this exercise, we examine a library that can be used to represent a deck of 52 playing cards. The core below builds such a library, and can be accessed by programs. You should recognize most of the things this program does aside from the `+shuffle-deck` arm which uses a [door](/build-on-urbit/hoon-school/k-doors) to produce [randomness](/build-on-urbit/hoon-school/o-subject). This is fairly idiomatic Hoon and it relies a lot on the convention that heavier code should be lower in the expression. This means that instead of `?:` [wutcol](/hoon/rune/wut#wutcol) you may see `?.` [wutdot](/hoon/rune/wut#wutdot), which inverts the order of the true/false arms, as well as other new constructions.

```hoon
|%
+$  suit  ?(%hearts %spades %clubs %diamonds)
+$  darc  [sut=suit val=@ud]  :: see below about naming
+$  deck  (list darc)
++  make-deck
  ^-  deck
  =/  mydeck  *deck
  =/  i  1
  |-
  ?:  (gth i 4)
    mydeck
  =/  j  2
  |-
  ?.  (lte j 13)
    ^$(i +(i))
  %=  $
    j       +(j)
    mydeck  [[(num-to-suit i) j] mydeck]
  ==
++  num-to-suit
  |=  val=@ud
  ^-  suit
  ?+  val  !!
    %1  %hearts
    %2  %spades
    %3  %clubs
    %4  %diamonds
  ==
++  shuffle-deck
  |=  [unshuffled=deck entropy=@]
  ^-  deck
  =/  shuffled  *deck
  =/  random  ~(. og entropy)
  =/  remaining  (lent unshuffled)
  |-
  ?:  =(remaining 1)
    :_  shuffled
    (snag 0 unshuffled)
  =^  index  random  (rads:random remaining)
  %=  $
    shuffled      [(snag index unshuffled) shuffled]
    remaining     (dec remaining)
    unshuffled    (oust [index 1] unshuffled)
  ==
++  draw
  |=  [n=@ud d=deck]
  ^-  [hand=deck rest=deck]
  :-  (scag n d)
  (slag n d)
--
```

The `|%` [barcen](/hoon/rune/bar#barcen) core created at the top of the file contains the entire library's code, and is closed by `--` [hephep](/hoon/rune/terminators#hephep) on the last line.

To create three types we're going to need, we use `+$` [lusbuc](/hoon/rune/lus#lusbuc), which is an arm used to define a type.

* `+$ suit ?(%hearts %spades %clubs %diamonds)` defines `$suit`, which can be either `%hearts`, `%spades`, `%clubs`, or `%diamonds`. It's a type union created by the irregular form of `$?` [bucwut](/hoon/rune/buc#bucwut).
* `+$ darc [sut=suit val=@ud]` defines `$darc`, which is a pair of `$suit` and a `@ud`. By pairing a suit and a number, it represents a particular playing card, such as “nine of hearts”. Why do we call it `$darc` and not `$card`? Because `$card` already has a meaning in Gall, the Arvo app module, where one would likely to use this (or any) library. It's worthwhile to avoid any confusion over names.
* `+$ deck (list darc)` is simply a list of `$darc`.

One way to get a feel for how a library works is to skim the `++` [luslus](/hoon/rune/lus#luslus) arm-names before diving into any specific arm. In this library, the arms are `+make-deck`, `+num-to-suit`, `+shuffle-deck`, and `+draw`. These names should be very clear, with the exception of `+num-to-suit` (although you could hazard a guess at what it does). Let's take a closer look at it first:

```hoon
++  num-to-suit
  |=  val=@ud
  ^-  suit
  ?+  val  !!
    %1  %hearts
    %2  %spades
    %3  %clubs
    %4  %diamonds
  ==
```

`+num-to-suit` defines a gate which takes a single `@ud` unsigned decimal integer and produces a `$suit`. The `?+` [wutlus](/hoon/rune/wut#wutlus) rune creates a structure to switch against a value with a default in case there are no matches. (Here the default is to crash with `!!` [zapzap](/hoon/rune/zap#zapzap).) We then have options 1–4 which each resulting in a different suit.

```hoon
++  make-deck
  ^-  deck
  =/  mydeck  *deck
  =/  i  1
  |-
  ?:  (gth i 4)
    mydeck
  =/  j  2
  |-
  ?.  (lte j 14)
    ^$(i +(i))
  %=  $
    j       +(j)
    mydeck  [[(num-to-suit i) j] mydeck]
  ==
```

`+make-deck` assembles a deck of 52 cards by cycling through every possible suit and number and combining them. It uses `+num-to-suit` and a couple of loops to go through the counters. It has an interesting `^$` loop skip where when `.j` is greater than 14 it jumps instead to the outer loop, incrementing `.i`.

`?.` [wutdot](/hoon/rune/wut#wutdot) may be an unfamiliar rune; it is simply the inverted version of `?:` [wutcol](/hoon/rune/wut#wutcol), so the first branch is actually the if-false branch and the second is the if-true branch. This is done to keep the “heaviest” branch at the bottom, which makes for more idiomatic and readable Hoon code.

```hoon
++  draw
  |=  [n=@ud d=deck]
  ^-  [hand=deck rest=deck]
  :-  (scag n d)
  (slag n d)
```

`+draw` takes two arguments: `.n`, an unsigned integer, and `.d`, a `$deck`. The gate will produce a cell of two `$deck`s using [`+scag`](/hoon/stdlib/2b#scag) and [`+slag`](/hoon/stdlib/2b#slag). [`+scag`](/hoon/stdlib/2b#scag) is a standard library gate produces the first `.n` elements from a list, while [`+slag`](/hoon/stdlib/2b#slag) is a standard library gate that produces the remaining elements of a list starting after the `.n`th element. So we use `+scag` to produce the drawn hand of `.n` cards in the head of the cell as `.hand`, and `+slag` to produce the remaining deck in the tail of the cell as `.rest`.

```hoon
++  shuffle-deck
  |=  [unshuffled=deck entropy=@]
  ^-  deck
  =/  shuffled  *deck
  =/  random  ~(. og entropy)
  =/  remaining  (lent unshuffled)
  |-
  ?:  =(remaining 1)
    :_  shuffled
    (snag 0 unshuffled)
  =^  index  random  (rads:random remaining)
  %=  $
    shuffled      [(snag index unshuffled) shuffled]
    remaining     (dec remaining)
    unshuffled    (oust [index 1] unshuffled)
  ==
```

Finally we come to `+shuffle-deck`. This gate takes two arguments: a `$deck`, and a `@` as a bit of "entropy" to seed the [`+og`](/hoon/stdlib/3d#og) random-number core. It will produce a `$deck`.

We add a bunted `$deck`, then encounter a very interesting statement that you haven't run into yet. This is the irregular form of `%~` [censig](/hoon/rune/cen#censig), which “evaluates an arm in a door.” For our purposes now, you can see it as a way of creating a random-value arm that we'll use later on with `+rads:random`.

With `=/ remaining (lent unshuffled)`, we get the length of the unshuffled deck with [`+lent`](/hoon/stdlib/2b#lent).

`?: =(remaining 1)` checks if we have only one card remaining. If that's true, we produce a cell of `.shuffled` and the one card left in `.unshuffled`. We use the `:_` [colcab](/hoon/rune/col#colcab) rune here, so that the “heavier” expression is at the bottom.

If the above conditional evaluates to `%.n` false, we need to do a little work. `=^` [tisket](/hoon/rune/tis#tisket) is a rune that pins the head of a pair and changes a leg in the subject with the tail. It's useful for interacting with the [`+og`](/hoon/stdlib/3d#og) core arms, as many of them produce a pair of a random numbers and the next state of the core. We're going to put the random number in the subject with the face `.index` and change `.random` to be the next core.

With that completed, we use `%=` [centis](/hoon/rune/cen#centis) to call `$` buc to recurse back up to `|-` [barhep](/hoon/rune/bar#barhep) with a few changes:

* `.shuffled` gets the `$darc` from `.unshuffled` at `.index` added to the front of it.
* `.remaining` gets decremented. Why are we using a counter here instead of just checking the length of `.unshuffled` on each loop? [`+lent`](/hoon/stdlib/2b#lent) traverses the entire list every time it's called so maintaining a counter in this fashion is much faster.
* `.unshuffled` becomes the result of using [`+oust`](/hoon/stdlib/2b#oust) to remove 1 `$darc` at `.index` on `.unshuffled`.

This is a very naive shuffling algorithm. We leave the implementation of a better shuffling algorithm as an exercise for the reader.

### Exercise: Using the Playing Card Library <a href="#exercise-using-the-playing-card-library" id="exercise-using-the-playing-card-library"></a>

Unfortunately `/` [fas](/hoon/rune/fas) runes don't work in the Dojo right now, so we need to build code using the [`-build-file`](/user-manual/os/dojo-tools#build-file) thread if we want to use the library directly.

Import the `/lib/playing-cards.hoon` library and use it to shuffle and show a deck and a random hand of five cards.

We first import the library:

```hoon
=playing-cards -build-file /===/lib/playing-cards/hoon
```

We then invoke it using the "entropy" or system randomness. (This is an unpredictable value we will use when we want a process to be random. We will discuss it in detail when we talk about [subject-oriented programming](/build-on-urbit/hoon-school/o-subject).)

```hoon
> =deck (shuffle-deck:playing-cards make-deck:playing-cards eny)

> deck
~[
  [sut=%spades val=12]
  [sut=%spades val=8]
  [sut=%hearts val=5]
  [sut=%clubs val=2]
  [sut=%diamonds val=10]
  ...
  [sut=%spades val=2]
  [sut=%hearts val=6]
  [sut=%hearts val=12]
]
```

Draw a hand of five cards from the deck:

```hoon
> (draw:playing-cards 5 deck)
[   hand
  ~[
    [sut=%spades val=12]
    [sut=%spades val=8]
    [sut=%hearts val=5]
    [sut=%clubs val=2]
    [sut=%diamonds val=10]
  ]
    rest
  ~[
    [sut=%hearts val=2]
    [sut=%clubs val=7]
    [sut=%clubs val=9]
    [sut=%diamonds val=6]
    [sut=%diamonds val=8]
    ...
    [sut=%spades val=2]
    [sut=%hearts val=6]
    [sut=%hearts val=12]
  ]
]
```

Of course, since the deck was shuffled once, any time we draw from the same deck we will get the same hand. But if we replace the deck with the `.rest` remaining, then we can continue to draw new hands.

## Desks <a href="#desks" id="desks"></a>

A desk organizes a collection of files, including generators, libraries, agents, and system code, into one coherent bundle. A desk is similar to a file drive in a conventional computer, or a Git branch. Desks are supported by the Clay vane in Arvo, the Urbit OS.

At this point, you've likely only worked on the `%base` desk. You can see data about any particular desk using the [`+vats`](/user-manual/os/dojo-tools#vats) generator:

```hoon
> +vats %base
%base
  /sys/kelvin:          [%zuse 408]
  base hash ends in:    hih5c
  %cz hash ends in:     hih5c
  app status:           running
  pending updates:      ~

> +vats %base, =verb %.y
%base
  /sys/kelvin:     [%zuse 408]
  base hash:       0v2.vhcjk.rj42q.e3la7.1679q.u2qs2.35vnn.9n1jm.mj66h.kgpe5.hih5c
  %cz hash:        0v2.vhcjk.rj42q.e3la7.1679q.u2qs2.35vnn.9n1jm.mj66h.kgpe5.hih5c
  app status:      running
  force on:        ~
  force off:       ~
  publishing ship: ~
  updates:         remote
  source ship:     ~marnec-dozzod-marzod
  source desk:     %kids
  source aeon:     43
  kids desk:       %kids
  pending updates: ~
```

You'll see a slightly different configuration on the particular ship you are running.

### Aside: Filesystems <a href="#aside-filesystems" id="aside-filesystems"></a>

A filesystem is responsible for providing access to blobs of data somewhere on a disk drive. If you have worked with Windows or macOS, you have become accustomed to using a file browser to view and interact with files. Mobile devices tend to obscure the nature of files more, in favor of just providing an end-user interface for working with or viewing the data. To use files effectively, you need to know a few things:

1. How to identify the data.
2. How to locate the data.
3. How to read or interpret the data.

Files are identified by a *file name*, which is typically a short descriptor like `Waterfall Visit 5.jpg` (if produced by a human) or `DSC_54694.jpg` (if produced by a machine).

Files are located using the "path" or "file path". Colloquially, this is what we mean when we ask which folder or directory a file is located in. It's an address that users and programs can use to uniquely locate a particular file, even if that file has the same name as another file.

An Earth filesystem and path orients itself around some key metaphor:

* Windows machines organize the world by drive, e.g. `C:\`.
* Unix machines (like macOS and Linux) organize the world from `/`, the root directory.

**Absolute paths** are like street addresses, or latitude and longitude. They let you unambiguously locate a file or folder. **Relative paths** are more like informal (but correct) instructions: “It's on the right just three houses past the church.” They are often shorter but require the user to know the starting point.

Once you have located a particular file, you need to load the data. Conventionally, file "extensions" indicate what kind of file you are dealing with: `.jpg`, `.png`, and `.gif` are image files, for instance; `.txt`, `.docx`, and `.pdf` are different kinds of documents; and `.mp3` and `.ogg` are audio files. Simply changing the extension on the file doesn't change the underlying data, but it can either elicit a stern warning from the OS or confuse it, depending on the OS. Normally you have to open the file in an appropriate program and save it as a new type if such a conversion is possible.

### File Data in Urbit <a href="#file-data-in-urbit" id="file-data-in-urbit"></a>

On Mars, we treat a filesystem as a way of organizing arbitrary access to blocks of persistent data. There are some concessions to Earth-style filesystems, but Clay (Urbit's filesystem) organizes everything with respect to a desk, a discrete collection of static data on a particular ship. Of course, like everything else in Hoon, a desk is a tree as well.

So far everything we have done has taken place on the `%base` desk. You have by this point become proficient at synchronizing Earthling data (Unix data) and Martian data (Urbit data), using [`|mount`](/user-manual/os/dojo-tools#mount) and [`|commit`](/user-manual/os/dojo-tools#commit), and every time you've done this with `%base` that has been recorded in the update report the Dojo makes to you.

```hoon
> |commit %base
>=
+ /~zod/base/2/gen/demo/hoon
```

This message says that a file `demo.hoon` was added to the Urbit filesystem at the path in `/gen`. What is the rest of it, though, the first three components? We call this the [`$beak`](/urbit-os/kernel/clay/data-types#beak). The beak lets Clay globally identify any resource on any ship at any point in time. A beak has three components:

1. The **ship**, here \~zod. (You can find this out on any ship using `.our`.)
2. The **desk**, here `%base`.
3. A **revision number** or **timestamp**, here `2`. (The current system time is available as `.now`.) Clay tracks the history of each file, so older versions can be accessed by their revision number. (This is uncommon to need to do today.)

The beak is commonly constructed with the `/` fas prefix and `=` tis signs for the three components:

```hoon
> /===
[~.~zod ~.base ~.~2022.6.14..18.13.35..ccaf ~]
```

Any one of those can be replaced as necessary:

```hoon
> /=sandbox=
[~.~zod %sandbox ~.~2022.6.14..18.14.49..a3da ~]
```

You'll also sometimes see `%` cen stand in for the whole including the “current” desk. The current desk is a Dojo concept, since for Clay we can access any desk at any time (with permission).

```hoon
> %
[~.~zod ~.base ~.~2022.6.14..18.15.10..698c ~]
```

### Paths and Files <a href="#paths-and-files" id="paths-and-files"></a>

A `$path` is a `(list @ta)`, a list of text identifiers. The first three are always the beak and the last one conventionally refers to the mark by which the file is represented.

For instance, the [`+cat`](/user-manual/os/dojo-tools#cat) generator displays the contents of any path, e.g.

```hoon
> +cat /===/gen/ls/hoon
/~zod/base/~2022.6.14..18.16.53..2102/gen/ls/hoon
::  LiSt directory subnodes
::
::::  /hoon/ls/gen
  ::
/?    310
/+    show-dir
::
::::
  ::
~&  %
:-  %say
|=  [^ [arg=path ~] vane=?(%g %c)]
=+  lon=.^(arch (cat 3 vane %y) arg)
tang+[?~(dir.lon leaf+"~" (show-dir vane arg dir.lon))]~
```

If no data are located at the given path, `+cat` simply shows `~` null:

```hoon
> +cat /=landscape=/gen/ls/hoon
~ /~zod/landscape/~2022.6.14..18.17.16..07ff/gen/ls/hoon
```

Every desk has a standard directory structure:

* `/app` for agents
* `/gen` for generators
* `/lib` for library and helper files
* `/mar` for marks
* `/sur` for shared structures
* `/ted` for threads

To run a generator from a different desk in Dojo, you need to prefix the desk name to the generator; to run `/=landscape=/gen/tally/hoon`, you would say:

```hoon
> +landscape!tally

tallied your activity score! find the results below.
to show non-anonymized resource identifiers, +tally |
counted from groups and channels that you are hosting.
groups are listed with their member count.
channels are listed with activity from the past week:
  - amount of top-level content
  - amount of unique authors

the date is ~2022.6.14..18.19.30..8c94
you are in 0 group(s):

you are hosting 0 group(s):
```

### Marks <a href="#marks" id="marks"></a>

Marks play the role of file extensions, with an important upgrade: they are actually molds and define conversion paths.

We won't write them in Hoon School, but you will encounter them when you begin writing apps. They are used more broadly than merely as file types, because they act as smart molds to ingest and yield data structures such as JSON and HTML from Hoon data structures.

In brief, each mark has a `+grab` arm to convert from other types to it; a `+grow` arm to convert it to other types; and a `+grad` arm for some standard operations across marks. You can explore the marks in `/mar`.

## Other Ford Runes <a href="#other-ford-runes" id="other-ford-runes"></a>

The `+ford` arm of Clay builds Hoon code. It provides [a number of runes](/hoon/rune/fas) which allow fine-grained control over building and importing files. These must be in the specific order at the top of any file. (They also don't work in Dojo; see [`-build-file`](/user-manual/os/dojo-tools#build-file) for a workaround.) The runes include:

* `/-` [fashep](/hoon/rune/fas#fashep) imports a structure file from `/sur`. Structure files are a way to share common data structures (across agents, for instance).
* `/+` [faslus](/hoon/rune/fas#faslus) imports a library file from `/lib`.

Both `/-` fashep and `/+` faslus allow you to import by affecting the name of the exposed core:

1. With the default name:

```hoon
/+  apple
```

2. With no name:

```hoon
/-  *orange
```

3. With a new name:

```hoon
/+  pomme=apple
```

Exposing namespaces with `*` is useful when importing libraries with unwieldy names, but otherwise should be avoided as it can shadow names in your current subject.

* `/=` [fastis](/hoon/rune/fas#fastis) builds a user-specified path and wraps it with a given face.
* `/*` [fastar](/hoon/rune/fas#fastar) imports the contents of a file, applies a mark to convert it, and wraps it with a given face.


# 8. Testing Code

Testing Hoon code, including unit testing frameworks, fences for runtime assertions, debugging strategies, and best practices for building reliable and maintainable software.

*This module will discuss how we can have confidence that a program does what it claims to do, using unit testing and debugging strategies. It may be considered optional and skipped if you are speedrunning Hoon School.*

> Code courageously.
>
> If you avoid changing a section of code for fear of awakening the demons therein, you are living in fear. If you stay in the comfortable confines of the small section of the code you wrote or know well, you will never write legendary code. All code was written by humans and can be mastered by humans.
>
> It's natural to feel fear of code; however, you must act as though you are able to master and change any part of it. To code courageously is to walk into any abyss, bring light, and make it right.
>
> **\~wicdev-wisryt -** [***Precepts*****, urbit.org**](https://urbit.org/blog/precepts)

When you produce software, how much confidence do you have that it does what you think it does? Bugs in code are common, but judicious testing can manifest failures so that the bugs can be identified and corrected. We can classify a testing regimen for Urbit code into a couple of layers: fences and unit tests.

### Fences <a href="#fences" id="fences"></a>

"Fences" are barriers employed to block program execution if the state isn’t adequate to the intended task. Typically, these are implemented with `assert` or similar enforcement. In Hoon, this means `?>` [wutgar](/hoon/rune/wut#wutgar), `?<` [wutgal](/hoon/rune/wut#wutgal), and `?~` [wutsig](/hoon/rune/wut#wutsig), or judicious use of `^-` [kethep](/hoon/rune/ket#kethep) and `^+` [ketlus](/hoon/rune/ket#ketlus). For conditions that must succeed, the failure branch in Hoon should be `!!`, which crashes the program.

### Unit Tests <a href="#unit-tests" id="unit-tests"></a>

> Unit tests are so called because they exercise the functionality of the code by interrogating individual functions and methods. Functions and methods can often be considered the atomic units of software because they are indivisible. However, what is considered to be the smallest code unit is subjective. The body of a function can be long are short, and shorter functions are arguably more unit-like than long ones.
>
> **Katy Huff -** [***Python Testing and Continuous Integration*****, Software Carpentry**](https://mq-software-carpentry.github.io/python-testing/05-units/)

In many languages, unit tests refer to functions, often prefixed "test", that specify (and enforce) the expected behavior of a given function. Unit tests typically contain setup, assertions, and tear-down. In academic terms, they’re a grading script.

In Hoon, the `/tests` directory contains the relevant tests for the testing framework to grab and utilize. These can be invoked with the [`-test`](/user-manual/os/dojo-tools#test) thread:

```
> -test /=landscape=/tests ~  
built   /tests/lib/pull-hook-virt/hoon  
built   /tests/lib/versioning/hoon  
>   test-supported: took 1047µs  
OK      /lib/versioning/test-supported  
>   test-read-version: took 28317µs  
OK      /lib/versioning/test-read-version  
>   test-is-root: took 28786µs  
OK      /lib/versioning/test-is-root  
>   test-current-version: took 507µs  
OK      /lib/versioning/test-current-version  
>   test-append-version: took 4804µs  
OK      /lib/versioning/test-append-version  
>   test-mule-scry-bad-time: took 8437µs  
OK      /lib/pull-hook-virt/test-mule-scry-bad-time  
>   test-mule-scry-bad-ship: took 8279µs  
OK      /lib/pull-hook-virt/test-mule-scry-bad-ship  
>   test-kick-mule: took 4614µs  
OK      /lib/pull-hook-virt/test-kick-mule  
ok=%.y    
```

(Depending on when you built your fakeship, particular tests may or may not be present. You can download them from [the Urbit repo](https://github.com/urbit/urbit) and add them manually if you like. Regarding the example above (`%landscape` desk), the tests are likely missing, so download them from [here](https://github.com/urbit/urbit/tree/master/pkg/landscape) if you want to run them.)

Hoon unit tests come in two categories:

1. `+expect-eq` (equality of two values)
2. `+expect-fail` (failure/crash)

Let's look at a practical example first, then dissect these.

### Exercise: Testing a Library <a href="#exercise-testing-a-library" id="exercise-testing-a-library"></a>

Consider an absolute value arm `+absolute` for `@rs` values. The unit tests for `+absolute` should accomplish a few things:

* Verify correct behavior for positive numeric input.
* Verify correct behavior for negative numeric input.
* For the purpose of demonstrating `+expect-fail`, verify an exception is raised on input of zero. (Properly speaking Hoon doesn't have exceptions because Nock is crash-only; tools like `+unit` are a way of dealing with failed computations.)

(You may also think we would need to verify `+absolute` calls only succeed if the input is a `@rs`, but arvo already handles this for us, as a hoon file will not build if a gate call contains an argument that does not match the sample type. So even if you wanted to add an `+expect-fail` test for it, your test file would not build.)

By convention any testing suite has the import line `/+ *test` at the top.

**/tests/lib/absolute.hoon**

```hoon
/+  *test, *absolute
|%
++  test-absolute
  ;:  weld
  %+  expect-eq
    !>  .1
    !>  (absolute .-1)
  %+  expect-eq
    !>  .1
    !>  (absolute .1)
  %-  expect-fail
    |.  (absolute .0)
  ==
--
```

Note that at this point we don’t care what the function looks like, only how it behaves.

**/lib/absolute.hoon**

```hoon
|%
++  absolute
  |=  a=@rs
  ?:  (gth a .0)  a
  (sub:rs .0 a)
--
```

Use the tests to determine what is wrong with this library code and correct it.

The dcSpark blog post [“Writing Robust Hoon — A Guide To Urbit Unit Testing”](https://medium.com/dcspark/writing-robust-hoon-a-guide-to-urbit-unit-testing-82b2631fe20a) covers some more good ideas about testing Hoon code.

### `/lib/test.hoon` <a href="#libtesthoon" id="libtesthoon"></a>

In `/lib/test.hoon` we find a core with a few gates: `+expect`, `+expect-eq`, and `+expect-fail`, among others.

`+expect-eq` checks whether two vases are equal and pretty-prints the result of that test. It is our workhorse. The source for `+expect-eq` is:

{% code title="/lib/test.hoon" %}

```hoon
++  expect-eq
  |=  [expected=vase actual=vase]
  ^-  tang
  ::
  =|  result=tang
  ::
  =?  result  !=(q.expected q.actual)
    %+  weld  result
    ^-  tang
    :~  [%palm [": " ~ ~ ~] [leaf+"expected" (sell expected) ~]]
        [%palm [": " ~ ~ ~] [leaf+"actual  " (sell actual) ~]]
    ==
  ::
  =?  result  !(~(nest ut p.actual) | p.expected)
    %+  weld  result
    ^-  tang
    :~  :+  %palm  [": " ~ ~ ~]
        :~  [%leaf "failed to nest"]
            (~(dunk ut p.actual) %actual)
            (~(dunk ut p.expected) %expected)
    ==  ==
  result
```

{% endcode %}

Test code deals in vases, which are produced by `!>` [zapgar](/hoon/rune/zap#zapgar) as a cell of the type of a value and the value.

`+expect-fail` by contrast take a `|.` [bardot](/hoon/rune/bar#bardot) trap (a trap that has the `$` buc arm but hasn't been called yet) and verifies that the code within fails.

{% code title="/lib/test.hoon" %}

```hoon
++  expect-fail
  |=  a=(trap)
  ^-  tang
  =/  b  (mule a)
  ?-  -.b
    %|  ~
    %&  ['expected failure - succeeded' ~]
  ==
```

{% endcode %}

```
> (expect-fail:test |.(!!))
~

> (expect-fail:test |.((sub 0 1)))
~

> (expect-fail:test |.((sub 1 1)))
~[[%leaf p="expected failure - succeeded"]]
```

(Recall that `~` null is `%.y` true.)

## Producing Error Messages <a href="#producing-error-messages" id="producing-error-messages"></a>

Formal error messages in Urbit are built of tanks.

* A `$tank` is a structure for printing data.
  * `$leaf` is for printing a single noun.
  * `$palm` is for printing backstep-indented lists.
  * `$rose` is for printing rows of data.
* A `$tang` is a `(list tank)`.

As your code evaluates, the Arvo runtime maintains a stack trace, or list of the evaluations and expressions that got the program to its notional point of computation. When the code fails, any error hints currently on the stack are dumped to the terminal for you to see what has gone wrong.

The `~_` [sigcab](/hoon/rune/sig#sigcab) rune, described as a “user-formatted tracing printf”, can include an error message for you, requiring you to explicitly build the `$tank`. ("printf" is a reference to [C's I/O library](https://en.wikipedia.org/wiki/Printf_format_string).)

The `~|` [sigbar](/hoon/rune/sig#sigbar) rune, a “tracing printf”, can include an error message from a simple `@t` cord. What this means is that these print to the stack trace if something fails, so you can use either rune to contribute to the error description:

```hoon
|=  a=@ud
~_  leaf+"This code failed"
!!
```

The `!:` [zapcol](/hoon/rune/zap#zapcol) rune turns on line-by-line stack tracing, which is extremely helpful when debugging programs. Drop it in on the first Hoon line (after `/` [fas](/hoon/rune/fas) imports) of a generator or library while developing.

```
> (sub 0 1)
subtract-underflow
dojo: hoon expression failed

> !:((sub 0 1))
/~zod/base/~2022.6.14..20.47.19..3b7a:<[1 4].[1 13]>
subtract-underflow
dojo: hoon expression failed
```

When you compose your own library cores, include error messages for likely failure modes.

## Test-Driven Development <a href="#test-driven-development" id="test-driven-development"></a>

*In extremis*, rigorous unit testing yields test-driven development (TDD). Test-driven development refers to the practice of fully specifying desired function behavior before composing the function itself. The advantage of this approach is that it forces you to clarify ahead of time what you expect, rather than making it up on the fly.

For instance, one could publish a set of tests which characterize the behavior of a Roman numeral translation library sufficiently that when such a library is provided it is immediately demonstrable.

{% code title="/tests/lib/roman.hoon" %}

```hoon
/+  *test, *roman
|%
++  test-output-one
  =/  src  "i"
  =/  trg  1
  ;:  weld
  %+  expect-eq
    !>  trg
    !>  (from-roman src)
  %+  expect-eq
    !>  trg
    !>  (from-roman (cuss src))
  ==
++  test-output-two
  =/  src  "ii"
  =/  trg  2
  ;:  weld
  %+  expect-eq
    !>  trg
    !>  (from-roman src)
  %+  expect-eq
    !>  trg
    !>  (from-roman (cuss src))
  ==
:: and so forth
++  test-input-one
  =/  trg  "i"
  =/  src  1
  ;:  weld
  %+  expect-eq
    !>  trg
    !>  (to-roman src)
  ==
++  test-input-two
  =/  trg  "ii"
  =/  src  2
  ;:  weld
  %+  expect-eq
    !>  trg
    !>  (to-roman src)
  ==
:: and so forth
--
```

{% endcode %}

By composing the unit tests ahead of time, you exercise a discipline of thinking carefully through details of the interface and implementation before you write a single line of implementation code.

## Debugging Common Errors <a href="#debugging-common-errors" id="debugging-common-errors"></a>

Let’s enumerate the errors you are likely to have encountered by this point:

### nest-fail <a href="#nest-fail" id="nest-fail"></a>

A [`nest-fail`](/hoon/hoon-errors#nest-fail) may be the most common. Likely you are using an atom or a cell where the other is expected.

```
> (add 'a' 'b')
195

> (add "a" "b")
-need.@
-have.[i=@tD t=""]
nest-fail
dojo: hoon expression failed
```

### mint-nice <a href="#mint-nice" id="mint-nice"></a>

The `mint-nice` error arises from typechecking:

```
> ^-(tape ~[78 97 114 110 105 97])
mint-nice  
-need.?(%~ [i=@tD t=""])  
-have.[@ud @ud @ud @ud @ud @ud %~]  
nest-fail  
dojo: hoon expression failed
```

Conversion without casting via auras fails because the atom types (auras) don't nest without explicit downcasting to `@`.

```
> `(list @ud)`~[0x0 0x1 0x2]
mint-nice
-need.?(%~ [i=@ud t=it(@ud)])
-have.[@ux @ux @ux %~]
nest-fail
dojo: hoon expression failed

> `(list @ud)``(list @)`~[0x0 0x1 0x2]
~[0 1 2]
```

### fish-loop <a href="#fish-loop" id="fish-loop"></a>

A `fish-loop` arises when using a recursive mold definition like list. (The relevant mnemonic is that `+fish` goes fishing for the type of an expression.) Alas, this fails today:

```
> ?=((list @) ~[1 2 3 4])
[%test ~[[%.y p=2]]]
fish-loop
```

### generator-build-fail <a href="#generator-build-fail" id="generator-build-fail"></a>

A `generator-build-fail` most commonly results from composing code with mismatched runes (and thus the wrong children including hanging expected-but-empty slots).

Also check if you are using Windows-style line endings, as Unix-style line endings should be employed throughout Urbit.

### Misusing the `$` buc Arm <a href="#misusing-the-buc-arm" id="misusing-the-buc-arm"></a>

Another common mistake is to attempt to use the default `$` buc arm in something that doesn't have it. This typically happens for one of two reasons:

`$.+2` means that `%-` [cenhep](/hoon/rune/cen#cenhep) or equivalent function call cannot locate a battery. This can occur when you try to use a non-gate as a gate. In particular, if you mask the name of a mold (such as list), then a subsequent expression that requires the mold will experience this problem.

```
> =/  list  ~[1 2 3]
 =/  a  ~[4 5 6]
 `(list @ud)`a
-find.$.+2
```

Similarly, `-find.$` means the compiler is looking for a `$` buc arm in something that *is* a core but doesn't have the `$` buc arm present.

```
> *tape
""
> (tape)
""
> *(tape)
-find.$
```

### Debugging Strategies <a href="#debugging-strategies" id="debugging-strategies"></a>

What are some strategies for debugging?

* **Debugging stack.** Use the `!:` [zapcol](/hoon/rune/zap#zapcol) rune to turn on the debugging stack, `!.` [zapdot](/hoon/rune/zap#zapdot) to turn it off again. (Most of the time you just pop this on at the top of a generator and leave it there.)
* **"printf" debugging.** If your code will compile and run, employ `~&` [sigpam](/hoon/rune/sig#sigpam) frequently to make sure that your code is doing what you think it’s doing.
* **Typecast.** Include `^` [ket](/hoon/rune/ket) casts frequently throughout your code. Entire categories of error can be excluded by satisfying the Hoon typechecker.
* **The only wolf in Alaska.** Essentially a bisection search, you split your code into smaller modules and run each part until you know where the bug arose (where the wolf howled). Then you keep fencing it in tighter and tighter until you know where it arose. You can stub out arms with `!!` [zapzap](/hoon/rune/zap#zapzap).
* **Build it again.** Remove all of the complicated code from your program and add it in one line at a time. For instance, replace a complicated function with either a `~&` sigpam and `!!` zapzap, or return a known static hard-coded value instead. That way as you reintroduce lines of code or parts of expressions you can narrow down what went wrong and why.
* **Run without networking**. If you run the Urbit executable with `-L`, you cut off external networking. This is helpful if you want to mess with a *copy* of an actual ship without producing remote effects. That is, if other parts of Ames don’t know what you’re doing, then you can delete that copy (COPY!) of your pier and continue with the original. This is an alternative to using fakeships which is occasionally helpful in debugging userspace apps in Gall. You can also develop using a moon if you want to.

### Further reading

* [Hoon Errors](/hoon/hoon-errors)


# 9. Text Processing I

Introduction to text representation in Hoon, covering cords, tapes, knots, terms, text manipulation tools, and %say generators for command-line argument processing.

*This module will discuss how text is represented in Hoon, discuss tools for producing and manipulating text, and introduce the `%say` generator, a new generator type. We don't deal with formatted text (`$tank`s) or parsers here, deferring that discussion. Formatted text and text parsing are covered\_* [*in a later module*](/build-on-urbit/hoon-school/p-stdlib-io)*.*

## Text in Hoon <a href="#text-in-hoon" id="text-in-hoon"></a>

We've incidentally used `'messages written as cords'` and `"as tapes"`, but aside from taking a brief look at how lists (and thus tapes) work with tree addressing, we haven't discussed why these differ or how text works more broadly.

There are four basic ways to represent text in Urbit:

* `@t`, a `$cord`, which is an atom (single value)
* `@ta`, a `$knot`, used for URL-safe path elements, which is an atom (single value)
* `@tas`, a `$term`, used primarily for constants, which is an atom (single value)
* `$tape`, which is a `(list @t)` (many values)

This is more ways than many languages support: most languages simply store text directly as a character array, or list of characters in memory. Colloquially, we would only call `$cord`s and `$tape`s [strings](https://en.wikipedia.org/wiki/String_\(computer_science\)), however.

What are the applications of each?

### `@t` `$cord` <a href="#t-cord" id="t-cord"></a>

What is a written character? Essentially it is a representation of human semantic content (not sound strictly). (Note that we don't refer to *alphabets*, which prescribe a particular relationship of sound to symbol: there are ideographic and logographic scripts, syllabaries, and other representations. Thus, *characters* not *letters*.) Characters can be combined, particularly in ideographic languages like Mandarin Chinese.

One way to handle text is to assign a code value to each letter, then represent these as subsequent values in memory. (Think, for instance, of [Morse code](https://en.wikipedia.org/wiki/Morse_code).) On all modern computers, the numeric values used for each letter are given by the [ASCII](https://en.wikipedia.org/wiki/ASCII) standard, which defines 128 unique characters (2⁷ = 128).

```
65  83  67  73  73
A   S   C   I   I
```

A cord simply shunts these values together in one-byte-wide slots and represents them as an integer.

```
> 'this is a cord'
'this is a cord'

> `@`'this is a cord'
2.037.307.443.564.446.887.986.503.990.470.772
```

It's very helpful to use the `@ux` aura if you are trying to see the internal structure of a `$cord`. Since the ASCII values align at the 8-bit wide characters, you can see each character delineated by a hexadecimal pair.

```
> `@ux`'HELLO'
0x4f.4c4c.4548

> `@ub`'HELLO'
0b100.1111.0100.1100.0100.1100.0100.0101.0100.1000
```

You can think of this a couple of different ways. One way is to simple think of them as chained together, with the first letter in the rightmost position. Another is to think of them as values multipled by a “place value”:

| Letter | ASCII | Place | “Place Value”                                        |
| ------ | ----- | ----- | ---------------------------------------------------- |
| `H`    | 0x48  | 0     | 2⁰ = 1 → 0x48                                        |
| `E`    | 0x45  | 1     | 2⁸ = 256 = 0x100 → 0x4500                            |
| `L`    | 0x4c  | 2     | 2¹⁶ = 65.536 = 0x1.0000 → 0x4c.0000                  |
| `L`    | 0x4c  | 3     | 2²⁴ = 16.777.216 = 0x100.0000 → 0x4c00.0000          |
| `O`    | 0x4f  | 4     | 2³² = 4.294.967.296 = 0x1.0000.0000 → 0x4f.0000.0000 |

This way, each value slots in after the preceding value.

Special characters (non-ASCII, beyond the standard keyboard, basically) are represented using a more complex numbering convention. [Unicode](https://en.wikipedia.org/wiki/Unicode) defines a standard specification for *code points* or numbers assigned to characters, and a few specific bitwise *encodings* (such as the ubiquitous UTF-8). Urbit uses UTF-8 for `@t` values (thus both `$cord` and `$tape`).

### `(list @t)` `$tape` <a href="#list-t-tape" id="list-t-tape"></a>

There are some tools to work with atom `$cord`s of text, but most of the time it is more convenient to unpack the atom into a tape. A `$tape` splits out the individual characters from a `$cord` into a `+list` of character values.

![](https://media.urbit.org/docs/userspace/hoon-school/binary-tree-tape.png)

We've hinted a bit at the structure of `+list`s before; for now the main thing you need to know is that they are cells which end in a `~` sig. So rather than have all of the text values stored sequentially in a single atom, they are stored sequentially in a rightwards-branching binary tree of cells.

A tape is a list of `@tD` atoms (i.e., characters). (The upper-case character at the end of the aura hints that the `@t` values are D→3 so 2³=8 bits wide.)

```
> "this is a tape"
"this is a tape"

> `(list @)`"this is a tape"
~[116 104 105 115 32 105 115 32 97 32 116 97 112 101]
```

Since a tape is a `(list @tD)`, all of the `+list` tools we have seen before work on them.

### `@ta` `$knot` <a href="#ta-knot" id="ta-knot"></a>

If we restrict the character set to certain ASCII characters instead of UTF-8, we can use this restricted representation for system labels as well (such as URLs, file system paths, permissions). `@ta` `$knot`s and `@tas` `$term`s both fill this role for Hoon.

```
> `@ta`'hello'
~.hello
```

Every valid `@ta` is a valid `@t`, but `@ta` does not permit spaces or a number of other characters. (See `+sane`, discussed below.)

### `@tas` `$term` <a href="#tas-term" id="tas-term"></a>

A further tweak of the ASCII-only concept, the `@tas` `$term` permits only “text constants”, values that are first and foremost only *themselves*.

> \[`@tas` permits only] a restricted text atom for Hoon constants. The only characters permitted are lowercase ASCII letters, `-`, and `0-9`, the latter two of which cannot be the first character. The syntax for `@tas` is the text itself, always preceded by `%`. The empty `@tas` has a special syntax, `$`.

`$term`s are rarely used for message-like text, but they are used all the time for internal labels in code. They differ from regular text in a couple of key ways that can confuse you until you're used to them.

For instance, a `@tas` value is also a mold, and the value will *only* match its own mold, so they are commonly used with [type unions](/build-on-urbit/hoon-school/n-logic) to filter for acceptable values.

```
> ^-  @tas  %5
mint-nice
-need.@tas
-have.%5
nest-fail
dojo: hoon expression failed

> ^-  ?(%5)  %5
%5

> (?(%5) %5)
%5
```

For instance, imagine creating a function to ensure that only a certain [classical element](https://en.wikipedia.org/wiki/Classical_element) can pass through a gate. (This gate is superfluous given how molds work, but it shows off a point.)

```hoon
|=  input=@t
=<
(validate-element input)
|%
+$  element  ?(%earth %air %fire %water)
++  validate-element
  |=  incoming=@t
  %-  element  incoming
--
```

(See how that `=<` [tisgal](/hoon/rune/tis#tisgal) works with the helper core?)

## Text Operations <a href="#text-operations" id="text-operations"></a>

Text-based data commonly needs to be *produced*, *manipulated*, or *analyzed* (including parsing).

### Producing Text <a href="#producing-text" id="producing-text"></a>

String interpolation puts the result of an expression directly into a `$tape`:

```
> "{<(add 5 6)>} is the answer."
"11 is the answer."
```

The [`+weld`](/hoon/stdlib/2b#weld) function can be used to glue two `$tape`s together:

```
> (weld "Hello" "Mars!")
"HelloMars!"
```

```hoon
|=  [t1=tape t2=tape]
^-  tape
(weld t1 t2)
```

### Manipulating Text <a href="#manipulating-text" id="manipulating-text"></a>

If you have text but you need to change part of it or alter its form, you can use standard library `+list` operators like [`+flop`](/hoon/stdlib/2b#flop) as well as `$tape`-specific arms.

Applicable `+list` operations, some of which you've seen before, include:

The [`+flop`](/hoon/stdlib/2b#flop) function takes a list and returns it in reverse order:

```
> (flop "Hello!")
"!olleH"

> (flop (flop "Hello!"))
"Hello!"
```

The [`+sort`](/hoon/stdlib/2b#sort) function uses the [quicksort algorithm](https://en.wikipedia.org/wiki/Quicksort) to sort a list. It takes a `+list` to sort and a gate that serves as a comparator. For example, if you want to sort the list `~[37 62 49 921 123]` from least to greatest, you would pass that list along with the [`+lth`](/hoon/stdlib/1a#lth) gate (for “less than”):

```
> (sort ~[37 62 49 921 123] lth)
~[37 49 62 123 921]
```

To sort the list from greatest to least, use the `+gth` gate ("greater than") as the basis of comparison instead:

```
> (sort ~[37 62 49 921 123] gth)
~[921 123 62 49 37]
```

You can sort letters this way as well:

```
> (sort ~['a' 'f' 'e' 'k' 'j'] lth)
<|a e f j k|>
```

The function passed to sort must produce a flag, i.e., `?`.

The [`+weld`](/hoon/stdlib/2b#weld) function takes two lists of the same type and concatenates them:

```
> (weld "Happy " "Birthday!")
"Happy Birthday!"
```

It does not inject a separator character like a space.

The [`+snag`](/hoon/stdlib/2b#snag) function takes an atom *n* and a list, and returns the *n*th item of the list, where 0 is the first item:

```
> (snag 3 "Hello!")
'l'

> (snag 1 "Hello!")
'e'

> (snag 5 "Hello!")
'!'
```

**Exercise: `+snag` Yourself**

Without using `+snag`, write a gate that returns the *n*th item of a list. There is a solution at the bottom of the page.

The [`+oust`](/hoon/stdlib/2b#oust) function takes a pair of atoms `[a=@ b=@]` and a `+list`, and returns the list with `.b` items removed, starting at item `.a`:

```
> (oust [0 1] `(list @)`~[11 22 33 44])
~[22 33 44]

> (oust [0 2] `(list @)`~[11 22 33 44])
~[33 44]

> (oust [1 2] `(list @)`~[11 22 33 44])
~[11 44]

> (oust [2 2] "Hello!")
"Heo!"
```

The [`+lent`](/hoon/stdlib/2b#lent) function takes a list and returns the number of items in it:

```
> (lent ~[11 22 33 44])
4

> (lent "Hello!")
6
```

**Exercise: Count the Number of Characters in Text**

There is a built-in `+lent` function that counts the number of characters in a `$tape`. Build your own `$tape`-length character counting function without using `+lent`.

You may find the `?~` [wutsig](/hoon/rune/wut#wutsig) rune to be helpful. It tells you whether a value is `~` or not. (How would you do this with a regular `?:` [wutcol](/hoon/rune/wut#wutcol)?)

The foregoing are `+list` operations. The following, in contrast, are `$tape`-specific operations:

The [`+crip`](/hoon/stdlib/4b#crip) function converts a `$tape` to a `$cord` (`$tape`→`$cord`).

```
> (crip "Mars")
'Mars'
```

The [`+trip`](/hoon/stdlib/4b#trip) function converts a `$cord` to a `$tape` (`$cord`→`$tape`).

```
> (trip 'Earth')
"Earth"
```

The [`+cass`](/hoon/stdlib/4b#cass) function: convert upper-case text to lower-case (`$tape`→`$tape`)

```
> (cass "Hello Mars")
"hello mars"
```

The [`+cuss`](/hoon/stdlib/4b#cuss) function: convert lower-case text to upper-case (`$tape`→`$tape`)

```
> (cuss "Hello Mars")
"HELLO MARS"
```

### Analyzing Text <a href="#analyzing-text" id="analyzing-text"></a>

Given a string of text, what can you do with it?

1. Search
2. Tokenize
3. Convert into data

#### Search

The [`+find`](/hoon/stdlib/2b#find) function takes `[nedl=(list) hstk=(list)]` and locates a sublist (`.nedl`, needle) in the list (`.hstk`, haystack). (`+find` starts counting from zero.)

```
> (find "brillig" "'Twas brillig and the slithy toves")
[~ 6]
```

`+find` returns a `+unit`, which right now means that we need to distinguish between nothing found (`~` null) and zero `[~ 0]`. `+unit`s are discussed in more detail in [a later lesson](/build-on-urbit/hoon-school/l-struct).

#### Tokenize/Parse

To "tokenize" text is to break it into pieces according to some rule. For instance, to count words one needs to break at some delimiter.

```
"the sky above the port was the color of television tuned to a dead channel"
 1   2   3     4   5    6   7   8     9  10         11    12 13 14  15
```

Hoon has a sophisticated parser built into it that [we'll use later](/build-on-urbit/hoon-school/p-stdlib-io). There are a lot of rules to deciding what is and isn't a rune, and how the various parts of an expression relate to each other. We don't need that level of power to work with basic text operations, so we'll instead use basic `+list` tools whenever we need to extract or break text apart for now.

## Exercise: Break Text at a Space <a href="#exercise-break-text-at-a-space" id="exercise-break-text-at-a-space"></a>

Hoon has a very powerful text parsing engine, built to compile Hoon itself. However, it tends to be quite obscure to new learners. We can build a simple one using `+list` tools.

Compose a gate which parses a long `$tape` into smaller `$tape`s by splitting the text at single spaces. For example, given a `$tape`

```hoon
"the sky above the port was the color of television tuned to a dead channel"
```

the gate should yield

```hoon
~["the" "sky" "above" "the" ...]
```

To complete this, you'll need[`+scag`](/hoon/stdlib/2b#scag) and[`+slag`](/hoon/stdlib/2b#slag) (who sound like\
villainous henchmen from a children's cartoon).

```hoon
|=  ex=tape
=/  index  0  
=/  result  *(list tape)  
|-  ^-  (list tape)  
?:  =(index (lent ex))  
  (weld result ~[`tape`(scag index ex)])
?:  =((snag index ex) ' ')  
  $(index 0, ex `tape`(slag +(index) ex), result (weld result ~[`tape`(scag index ex)]))    
$(index +(index))
```

#### Convert

If you have a Hoon value and you want to convert it into text as such, use [`+scot`](/hoon/stdlib/4m#scot) and [`+scow`](/hoon/stdlib/4m#scow). These call for a value of type `$dime`, which means the `@tas` equivalent of a regular aura. These are labeled as returning `$cord`s (`@t`s) but in practice seem to return `$knot`s (`@ta`s).

The [`+scot`](/hoon/stdlib/4m#scot) function renders a `$dime` as a `$cord` (`$dime`→`$cord`); the user must include any necessary aura transformation.

```
> `@t`(scot %ud 54.321)
'54.321'

> `@t`(scot %ux 54.000)  
'0xd2f0'
```

```
> (scot %p ~sampel-palnet)
~.~sampel-palnet

> `@t`(scot %p ~sampel-palnet)
'~sampel-palnet'
```

The [`+scow`](/hoon/stdlib/4m#scow) function renders a `$dime` as a `$tape` (`$dime`→`$tape`); it is otherwise identical to [`+scot`](/hoon/stdlib/4m#scot).

The [`+sane`](/hoon/stdlib/4b#sane) function checks the validity of a possible text string as a `$knot` or `$term`. The usage of `+sane` will feel a bit strange to you: it doesn't apply directly to the text you want to check, but it produces a gate that checks for the aura (as `%ta` or `%tas`). (The gate-builder is a fairly common pattern in Hoon that we've started to hint at by using molds.) `+sane` is also not infallible yet.

```
> ((sane %ta) 'ångstrom')
%.n

> ((sane %ta) 'angstrom')
%.y

> ((sane %tas) 'ångstrom')
%.n

> ((sane %tas) 'angstrom')
%.y
```

Why is this sort of check necessary? Two reasons:

1. `@ta` `$knot`s and `@tas` `$term`s have strict rules, such as being ASCII-only.
2. Not every sequence of bits has a conversion to a text representation. That is, ASCII and Unicode have structural rules that limit the possible conversions which can be made. If things don't work, you'll get a `%bad-text` response.

```
> 0x1234.5678.90ab.cdef
0x1234.5678.90ab.cdef
> `@t`0x1234.5678.90ab.cdef
[%bad-text "[39 239 205 171 144 120 86 52 92 49 50 39 0]"]
```

There's a minor bug in Hoon that will let you produce an erroneous `$term` (`@tas`):

```
> `@tas`'hello mars'
%hello mars
```

Since a `@tas` cannot include a space, this is formally incorrect, as `+sane` reveals:

```
> ((sane %tas) 'hello')  
%.y

> ((sane %tas) 'hello mars')
%.n
```

## Exercise: Building Your Own Library <a href="#exercise-building-your-own-library" id="exercise-building-your-own-library"></a>

Let's take some of the code we've built above for processing text and turn them into a library we can use in another generator.

* Take the space-breaking code and the element-counting code gates from above and include them in a `|%` [barcen](/hoon/rune/bar#barcen) core. Save this file as `/lib/text.hoon` in the `%base` desk of your fakeship and commit.
* Produce a generator `/gen/text-user.hoon` which accepts a tape and returns the number of words in the text (separated by spaces). (How would you obtain this from those two operations?)

## Logging <a href="#logging" id="logging"></a>

The most time-honored method of debugging is to simply output relevant values at key points throughout a program in order to make sure they are doing what you think they are doing. To this end, we introduced `~&` [sigpam](/hoon/rune/sig#sigpam) in the last lesson.

The `~&` [sigpam](/hoon/rune/sig#sigpam) rune offers some finer-grained output options than just printing a simple value to the screen. For instance, you can use it with string interpolation to produce detailed error messages.

There are also `>` modifiers which can be included to mark “debugging levels”, really just color-coding the output:

1. No `>`: regular
2. `>`: information
3. `>>`: warning
4. `>>>`: error

(Since all `~&` sigpam output is a side effect of the compiler, it doesn't map to the Unix [stdout/stderr streams](https://en.wikipedia.org/wiki/Standard_streams) separately; it's all stdout.)

You can use these to differentiate messages when debugging or otherwise auditing the behavior of a generator or library. Try these in your own Dojo:

```
> ~&  'Hello Mars!'  ~  
'Hello Mars!'
~  

> ~&  >  'Hello Mars!'  ~  
>   'Hello Mars!'  
~  

> ~&  >>  'Hello Mars!'  ~  
>>  'Hello Mars!'  
~  

> ~&  >>>  'Hello Mars!'  ~  
>>> 'Hello Mars!'  
~
```

## `%say` Generators <a href="#say-generators" id="say-generators"></a>

A naked generator is merely a gate: a core with a `$` arm that Dojo knows to call. However, we can also invoke a generator which is a cell of a metadata tag and a core. The next level-up for our generator skills is the `%say` generator, a cell of `[%say core]` that affords slightly more sophisticated evaluation.

We use `%say` generators when we want to provide something else in Arvo, the Urbit operating system, with metadata about the generator's output. This is useful when a generator is needed to pipe data to another program, a frequent occurrence.

To that end, `%say` generators use `$mark`s to make it clear, to other Arvo computations, exactly what kind of data their output is. A mark is akin to a MIME type on the Arvo level. A `$mark` describes the data in some way, indicating that it's an `%atom`, or that it's a standard such as `%json`, or even that it's an application-specific data structure like `%talk-command`. `$mark`s are not specific to `%say` generators; whenever data moves between programs in Arvo, that data is marked.

So, more formally, a `%say` generator is a cell. The head of that cell is the `%say` tag, and the tail is a `$gate` that produces a `+cask` -- a pair of the output data and the `$mark` describing that data. -- Save this example as `add.hoon` in the `/gen` directory of your `%base` desk:

```hoon
:-  %say
|=  *
:-  %noun
(add 40 2)
```

Run it with:

```
> |commit %base

> +add
42
```

Notice that we used no argument, something that is possible with `%say` generators but impossible with naked generators. We'll explain that in a moment. For now, let's focus on the code that is necessary to make something a `%say` generator.

```hoon
:-  %say
::  ...
```

Recall that the rune `:-` [colhep](/hoon/rune/col#colhep) produces a cell, with the first following expression as its head and the second following expression as its tail.

The expression above creates a cell with `%say` as the head. The tail is the `|= *` expression on the line that follows.

```hoon
|=  *
:-  %noun
(add 40 2)
```

`|= *` constructs a gate that takes a noun. This `$gate` will itself produce a `+cask`, which is cell formed by the prepending `:-`. The head of that `+cask` is `%noun` and the tail is the rest of the program, `(add 40 2)`. The tail of the `+cask` will be our actual data produced by the body of the program: in this case, just adding 40 and 2 together.

A `%say` generator has access to values besides those passed into it and the Hoon standard subject. Namely, a `%say` generator knows about `.our`, `.eny`, and `.now`, as well as the current desk:

* `.our` is our current ship identity.
* `.eny` is entropy, a source of randomness.
* `.now` is the current system timestamp.
* `.bec` is the current path (beak).

These values can be stubbed out with `*` or `^` if they are not needed in a particular generator.

### `%say` generators with arguments <a href="#say-generators-with-arguments" id="say-generators-with-arguments"></a>

We can modify the boilerplate code to allow arguments to be passed into a `%say` generator, but in a way that gives us more power than we would have if we just used a naked generator.

Naked generators are limited because they have no way of accessing data that exists in Arvo, such as the date and time or pieces of fresh entropy. In `%say` generators, however, we can access that kind of subject by identifying them in the gate's sample, which we only specified as `*` in the previous few examples. But we can do more with `%say` generators if we do more with that sample. Any valid sample will follow this 3-tuple scheme:

`[[now=@da eny=@uvJ bec=beak] [list of unnamed arguments] [list of named arguments]]`

This entire structure is a noun, which is why `*` is a valid sample if we wish to not use any of the information here in a generator. But let's look at each of these three elements, piece by piece.

## Exercise: The Magic 8-Ball <a href="#exercise-the-magic-8-ball" id="exercise-the-magic-8-ball"></a>

This Magic 8-Ball generator returns one of a variety of answers in response to a call. In its entirety:

```hoon
!:
:-  %say
|=  [[* eny=@uvJ *] *]
:-  %noun
^-  tape
=/  answers=(list tape)
  :~  "It is certain."
      "It is decidedly so."
      "Without a doubt."
      "Yes - definitely."
      "You may rely on it."
      "As I see it, yes."
      "Most likely."
      "Outlook good."
      "Yes."
      "Signs point to yes."
      "Reply hazy, try again"
      "Ask again later."
      "Better not tell you now."
      "Cannot predict now."
      "Concentrate and ask again."
      "Don't count on it."
      "My reply is no."
      "My sources say no."
      "Outlook not so good."
      "Very doubtful."
  ==
=/  rng  ~(. og eny)
=/  val  (rad:rng (lent answers))
(snag val answers)
```

`~(. og eny)` starts a random number generator with a seed from the current entropy. Right now we don't know quite enough to interpret this line, but we'll revisit the [`+og`](/hoon/stdlib/3d#og) aspect of this `%say` generator in [the lesson on subject-oriented-programming](/build-on-urbit/hoon-school/o-subject). For now, just know that it allows us to produce a random (unpredictable) integer using `+rad:rng`. We slam the `+rad:rng` gate which returns a random number from 0 to *n*-1 inclusive. This gives us a random value from the list of possible answers.

Since this is a `%say` generator, we can run it without arguments:

```
> +magic-8
"Ask again later."
```

If we need to include optional arguments to a generator, we separate them using a `,` com:

```
+cat /===/gen/cat/hoon, =vane %c
```

## Exercise: Using the Playing Card Library <a href="#exercise-using-the-playing-card-library" id="exercise-using-the-playing-card-library"></a>

Recall the playing card library `/lib/playing-cards.hoon` in `/lib`. Let's use it with a `%say` generator.

{% code title="/gen/cards.hoon" %}

```hoon
/+  playing-cards
:-  %say
|=  [[* eny=@uv *] *]
:-  %noun
(shuffle-deck:playing-cards make-deck:playing-cards eny)
```

{% endcode %}

Having already saved the library as `/lib/playing-cards.hoon`, you can import it with the `/+` [faslus](/hoon/rune/fas#faslus) rune. When `cards.hoon` gets built, the Hoon builder will pull in the requested library and also build that. It will also create a dependency so that if `/lib/playing-cards.hoon` changes, this file will also get rebuilt.

Below `/+ playing-cards`, you have the standard `%say` generator boilerplate that allows us to get a bit of entropy from Arvo when the generator is run. Then we feed the entropy and a `$deck` created by `+make-deck` into `+shuffle-deck` to get back a shuffled `$deck`.

## Solutions to Exercises <a href="#solutions-to-exercises" id="solutions-to-exercises"></a>

Roll-Your-Own-`+snag`:

```hoon
::  snag.hoon
::
|=  [a=@ b=(list @)]
?~  b  !!
?:  =(0 a)  i.b
$(a (dec a), b t.b)
```


# 10. Cores and Doors

Advanced core concepts including doors (gate-building gates), doors as parametrized cores with sample data, working with \`+map\`s (key-value stores), and stateful computation patterns.

*Hoon is statically typed, which means (among other things) that auras are subject to strict nesting rules, molds are crash-only, and the whole thing is rather cantankerous about matching types. However, since gate-building arms are possible, Hoon developers frequently employ them as templates to build type-appropriate cores, including gates. This module will start by introducing the concept of gate-building gates; then it will expand our notion of cores to include doors; finally it will introduce a common door, the* [*`+map`*](/hoon/stdlib/2o#map)*, to illustrate how doors work.*

## Gate-Building Gates <a href="#gate-building-gates" id="gate-building-gates"></a>

### Calling Gates <a href="#calling-gates" id="calling-gates"></a>

There are two ways of making a function call in Hoon. First, you can call a gate in the subject by name. For instance, we can produce a gate `+inc` which adds `1` to an input:

```
> =inc |=(a=@ (add 1 a))

> (inc 10)
11

> =inc
```

The second way of making a function call involves an expression that *produces* a gate on demand:

```
> (|=(a=@ (add 1 a)) 123)
124

> (|=(a=@ (mul 2 a)) 123)
246
```

The difference is subtle: the first case has an already-created gate in the subject when we called it, while the latter involves producing a gate that doesn't exist anywhere in the subject, and then calling it.

Are calls to [`+add`](/hoon/stdlib/1a#add) and [`+mul`](/hoon/stdlib/1a#mul) of the Hoon standard library of the first kind, or the second?

```
> (add 12 23)
35

> (mul 12 23)
276
```

They're of the second kind. Neither `+add` nor `+mul` resolves to a gate directly; they're each arms that *produce* gates.

Often the difference doesn't matter much. Either way you can do a function call using the "(gate arg)" syntax.

It's important to learn the difference, however, because for certain use cases you'll want the extra flexibility that comes with having an already produced core in the subject.

### Building Gates <a href="#building-gates" id="building-gates"></a>

Let's make a core with arms that build gates of various kinds. As we did in a previous lesson, we'll use the `|%` [barcen](/hoon/rune/bar#barcen) rune. Copy and paste the following into the Dojo:

```
=c |%
++  inc      |=(a=@ (add 1 a))
++  add-two  |=(a=@ (inc (inc a)))
++  double   |=(a=@ (mul 2 a))
++  triple   |=(a=@ (mul 3 a))
--
```

Let's try out these arms, using them for function calls:

```
> (inc:c 10)
11

> (add-two:c 10)
12

> (double:c 10)
20

> (triple:c 10)
30
```

Notice that each arm in core `.c` is able to call the other arms of `.c`; `+add-two` uses the `+inc` arm to increment a number twice. As a reminder, each arm is evaluated with its parent core as the subject. In the case of `+add-two` the parent core is `.c`, which has `+inc` in it.

#### Mutating a Gate

Let's say you want to modify the default sample of the gate for `+double`. We can infer the default sample by calling `+double` with no argument:

```
> (double:c)
0
```

Given that *a x 2 = 0*, *a* must be 0. (Remember that `a` is the face for the `+double` sample, as defined in the core we bound to the face `c` above.)

Let's say we want to mutate the `+double` gate so that the default sample is `25`. There is only one problem: `+double` isn't a gate!

```
> double.c(a 25)
-tack.a
-find.a
dojo: hoon expression failed
```

It's an arm that produces a gate, and `.a` cannot be found in `+double` until the gate is created. Furthermore, every time the gate is created, it has the default sample, `0`. If you want to mutate the gate produced by `+double`, you'll first have to put a copy of that gate into the subject:

```
> =double-copy double:c

> (double-copy 123)
246
```

Now let's mutate the sample to `25`, and check that it worked with `+6`. (The sample lives at `+6` in a given core tree.)

```
> +6:double-copy(a 25)
a=25
```

Good. Let's call it with no argument and see if it returns double the value of the modified sample.

```
> (double-copy(a 25))
50
```

It does indeed. Unbind the faces `c` and `double-copy`:

```
> =c

> =double-copy
```

Contrast this with the behavior of [`+add`](/hoon/stdlib/1a#add). We can look at the sample of the gate for `+add` with `+6:add`:

```
> +6:add
[a=0 b=0]
```

If you try to mutate the default sample of `+add`, it won't work:

```
> add(a 3)
-tack.a
-find.a
dojo: hoon expression failed
```

As before with `+double`, Hoon can't find an `.a` to modify in a gate that doesn't exist yet.

### Slamming a Gate <a href="#slamming-a-gate" id="slamming-a-gate"></a>

If you check the docs on our now-familiar `%-` [cenhep](/hoon/rune/cen#cenhep), you'll find that it is actually sugar syntax for another rune:

> This rune is for evaluating the `$` arm of a gate, i.e., calling a gate as a function. `.a` is the gate, and `.b` is the desired sample value (i.e., input value) for the gate.
>
> ```hoon
> %~($ a b)
> ```

So all gate calls actually pass back through `%~` [censig](/hoon/rune/cen#censig). What's the difference?

The `%~` [censig](/hoon/rune/cen#censig) rune accepts three children:

* A wing which resolves to an arm in a door.
* The aforesaid door.
* A sample for the door.

Basically, whenever you use `%-` [cenhep](/hoon/rune/cen#cenhep), it actually looks up a wing in a door using `%~` [censig](/hoon/rune/cen#censig), which is a more general type of core than a gate. Whatever that wing resolves to is then provided a sample. The resulting Hoon expression is evaluated and the value is returned.

## Doors <a href="#doors" id="doors"></a>

Doors are another kind of core whose arms evaluate to make gates, as we just discovered. The difference is that a door also has its own sample. A door is the most general case of a function in Hoon. (You could say a "gate-building core" or a "function-building function" to clarify what the intent of most of these are.)

A core is a cell of code and data we call "\[battery payload]". The battery contains a series of arms, and the payload contains all the data necessary to run those arms correctly.

A "door" is a core with a sample. That is, a door is a core whose payload is a cell of sample and context: \[sample context]. A door's overall sample can affect how its gate-building arms work.

```
        door
       /    \
battery      .
            / \
      sample   context
```

It follows from this definition that a gate is a special case of a door. A gate is a door with exactly one arm, named `$` ("buc").

Doors are created with the `|_` [barcab](/hoon/rune/bar#barcab) rune. Doors get used for a few different purposes in the standard library:

* Instrumenting and storing persistent data structures like `+map`s (this module and the next)
* Implementing state machines (the [subject-oriented programming module](/build-on-urbit/hoon-school/o-subject))

One *big* pitfall for thinking about doors is thinking of them as “containing” gates, as if they were more like “objects”. Instead, think of them the same way as you think of gates, just that they can be altered at a higher level.

#### Example: The Quadratic Equation

First, a mathematical example. If we wanted to calculate a quadratic polynomial, *y = a x² + b x + c*, then we need to know two kinds of things: the unknown or variable *x*, AND the parameters *a*, *b*, and *c*. These aren't really the same kind of thing. When we calculate a particular curve *y*(*x*), we assume that the parameters *a*, *b*, and *c* stay constant across evaluations of *x*, and it's inconvenient for us to specify them every single time.

If we were to build this as a gate, we would need to pass in four parameters:

```
> =poly-gate |=  [x=@ud a=@ud b=@ud c=@ud]
(add (add (mul a (mul x x)) (mul b x)) c)
```

Any time we call the gate, we have to provide all four values: one unknown, three parameters. But there's a sense in which we want to separate the three parameters and only call the gate with one `.x` value. One way to accomplish this is to wrap the gate inside of another:

```
> =wrapped-gate |=  [x=@ud]
=/  a  5
=/  b  4
=/  c  3
(poly-gate x a b c)
```

If we built this as a door instead, we could push the parameters out to a different layer of the structure. In this case, the parameters are the sample of the door, while the arm `+quad` builds a gate that corresponds to those parameters and only accepts one unknown variable `.x`. To make a door we use the `|_` [barcab](/hoon/rune/bar#barcab) rune, which we'll discuss later:

```
> =poly |_  [a=@ud b=@ud c=@ud]
++  quad
  |=  x=@ud
  (add (add (mul a (mul x x)) (mul b x)) c)
--
```

This will be used in two steps: a gate-building step then a gate usage step.

We produce a gate from a door's arm using the `%~` [censig](/hoon/rune/cen#censig) rune, almost always used in its irregular form, `~()`. Here we prime the door with `[5 4 3]`, which yields a gate:

```hoon
~(quad poly [5 4 3])
```

By itself, not so much to say. We could pin it into the Dojo, for instance, to use later. Our ultimate goal is to use the built gate on particular data, however:

```
> (~(quad poly [5 4 3]) 2)
31
```

By hand: *5×2² + 4×2 + 3 = 31*, so that's correct.

Doors will enable us to build some very powerful data storage tools by letting us defer parts of a gate calculation to other stages of building and calculating the gate.

#### Example: A Calculator

Let's unpack what's going on more with this next door. Each of the arms in this example door will define a simple gate. Let's bind the door to `.c`. To make a door we use the `|_` [barcab](/hoon/rune/bar#barcab) rune:

```
=c |_  b=@
++  plus  |=(a=@ (add a b))
++  times  |=(a=@ (mul a b))
++  greater  |=(a=@ (gth a b))
--
```

If you type this into the Dojo manually, make sure you attend carefully to the spacing. Feel free to cut and paste the code, if desired.

Before getting into what these arms do, let's digress into how the `|_` [barcab](/hoon/rune/bar#barcab) rune works in general.

`|_` [barcab](/hoon/rune/bar#barcab) works exactly like the `|%` [barcen](/hoon/rune/bar#barcen) rune for making a core, except that it takes one additional daughter expression, the door's sample. Following that are a series of `++` [luslus](/hoon/rune/lus#luslus) runes, each of which defines an arm of the door. Finally, the expression is terminated with a `--` [hephep](/hoon/rune/terminators#hephep) rune.

A door really is, at the bedrock level, the same thing as a core with a sample. Let's ask Dojo to pretty print a simple door.

```
> =a =>  ~  |_  b=@  ++  foo  b  --

> a
<1.zgd [b=@ %~]>
```

Dojo tells us that `.a` is a core with one arm and a payload of `[b=@ %~]`. Since a door's payload is \[sample context], this means that `.b` is the sample and the context is null. (The `=> ~` set the context. We did this to avoid including the standard library that is included in the context by default in Dojo, which would have made the pretty-printed core much more verbose. Try it without `=> ~` as well.)

For the door defined above, `.c`, the sample is defined as a `@` atom and given the face `.b`. The `+plus` arm defines a gate that takes a single atom as its argument `.a` and returns the sum of `.a` and `.b`. The `+times` arm defines a gate that takes a single atom `.a` and returns the product of `.a` and `.b`. The `+greater` arm defines a gate that takes a single atom `.a`, and returns `%.y` if `.a` is greater than `.b`; otherwise it returns `%.n`.

Let's try out the arms of `.c` with ordinary function calls:

```
> (plus:c 10)
10

> (times:c 10)
0

> (greater:c 10)
%.y
```

This works, but the results are not exciting. Passing `10` to the `+plus` gate returns `10`, so it must be that the value of `.b` is `0` (the bunt value of `@`). The products of the other function calls reinforce that assessment. Let's look directly at `+6` of `.c` to see the sample:

```
> +6:c
b=0
```

Having confirmed that `.b` is `0`, let's mutate the `.c` sample and then call its arms:

```
> (plus:c(b 7) 10)
17

> (times:c(b 7) 10)
70

> (greater:c(b 7) 10)
%.y

> (greater:c(b 17) 10)
%.n
```

Doing the same mutation repeatedly can be tedious, so let's bind `.c` to the modified version of the door, where `.b` is `7`:

```
> =c c(b 7)

> (plus:c 10)
17

> (times:c 10)
70

> (greater:c 10)
%.y
```

There's a more direct way of passing arguments for both the door sample and the gate sample simultaneously. We may use the "\~(arm door arg)" syntax. This generates the *arm* product after modifying the *door*'s sample to be *arg*.

```
> (~(plus c 7) 10)
17

> (~(times c 7) 10)
70

> (~(greater c 7) 10)
%.y

> (~(greater c 17) 10)
%.n
```

Readers with some mathematical background may notice that "\~(...)" expressions allow us to [curry](https://en.wikipedia.org/wiki/Currying). For each of the arms above, the "\~(...)" expression is used to create different versions of the same gate:

```
> ~(plus c 7)
< 1.xpd
  [ a=@
    < 3.bnz
      [ b=@
        [our=@p now=@da eny=@uvJ]
        <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
      ]
    >
  ]
>

> b:~(plus c 7)
7

> b:~(plus c 17)
17
```

Thus, you may think of the `.c` door as a function for making functions. Use the "\~(arm c arg)" syntax. *arm* defines which kind of gate is produced (i.e., which arm of the door is used to create the gate), and *arg* defines the value of *b* in that gate, which in turn affects the product value of the gate produced.

The standard library provides [currying functionality](/build-on-urbit/hoon-school/q-func) outside of the context of doors.

#### Creating Doors with a Modified Sample

In the above example we created a door `.c` with sample `b=@` and found that the initial value of `.b` was `0`, the default value of type `@`. We then created new door from `.c` by modifying the value of `.b`. But what if we wish to define a door with a chosen sample value directly? We make use of the `$_` [buccab](/hoon/rune/buc#buccab) rune, whose irregular form is simply `_`. To create the door `.c` with the sample `b=@` set to have the value `7` in the dojo, we would write

```
=c |_  b=_7
++  plus  |=(a=@ (add a b))
++  times  |=(a=@ (mul a b))
++  greater  |=(a=@ (gth a b))
--
```

Here the type of `.b` is inferred to be `@` based on the example value `7`, similar to how we've seen casting done by example. You will learn more about how types are inferred in the [next module](/build-on-urbit/hoon-school/l-struct).

### Exercise: Adding Arms to a Door <a href="#exercise-adding-arms-to-a-door" id="exercise-adding-arms-to-a-door"></a>

Recall the quadratic equation door.

```hoon
|_  [a=@ud b=@ud c=@ud]
++  quad
  |=  x=@ud
  (add (add (mul a (mul x x)) (mul b x)) c)
--
```

1. Add an arm to the door which calculates the linear function *a × x + b*.
2. Add another arm which calculates the derivative of the first quadratic function, *2 × a × x + b*.

## Key-Value Pairs: `+map` as Door <a href="#key-value-pairs-map-as-door" id="key-value-pairs-map-as-door"></a>

In general terms, a [`+map`](/hoon/stdlib/2o#map) is a pattern from a key to a value. You can think of a dictionary, or an index, or a data table. Essentially it scans for a particular key, then returns the data associated with that key (which may be any noun).

For example, a `(map @t @t)` called might "map" car brands to models:

| Key           | Value        |
| ------------- | ------------ |
| `'Mazda'`     | `'RX-8'`     |
| `'Dodge'`     | `'Viper'`    |
| `'Ford'`      | `'Mustang'`  |
| `'Chevrolet'` | `'Chevelle'` |
| `'Porsche'`   | `'Boxster'`  |
| `'Bugatti'`   | `'Type 22'`  |

While `+map` is the name of the mold or type of the value, the door which affords `+map`-related functionality is named [`+by`](/hoon/stdlib/2i#by). (This felicitously affords us a way to read `+map` operations in an English-friendly phrasing, e.g. "(\~(get by cars) 'Ford')".)

In Urbit, all values are static and never change. (This is why we “overwrite” or replace the values in a limb to change it with `%=` [centis](/hoon/rune/cen#centis).) This means that when we build a `+map`, we often rather awkwardly replace it with its modified value explicitly.

We'll build a color `+map`, from a `@tas` of a [color's name](https://en.wikipedia.org/wiki/List_of_Crayola_crayon_colors) to its HTML hexadecimal representation as a `@ux` hex value.

We can produce a `+map` from a list of key-value cells using the [`+malt`](/hoon/stdlib/2l#malt) function. Using `@tas` terms as keys (which is common) requires us to explicitly mark the list as `(list (pair @tas @ux))`:

```
=colors (malt `(list (pair @tas @ux))`~[[%red 0xed.0a3f] [%yellow 0xfb.e870] [%green 0x1.a638] [%blue 0x66ff]])
```

To insert one key-value pair at a time, we use [`+put:by`](/hoon/stdlib/2i#putby). In the Dojo, we need to either pin it into the subject or modify a copy of the map for the rest of the expression using `=/` [tisfas](/hoon/rune/tis#tisfas).

```
=colors (~(put by colors) [%orange 0xff.8833])
=colors (~(put by colors) [%violet 0x83.59a3])
=colors (~(put by colors) [%black 0x0])
```

Note the pattern here: there is a [`+put`](/hoon/stdlib/2i#putby) arm of [`+by`](/hoon/stdlib/2i#by) which builds a gate to modify `.colors` by inserting a value.

What happens if we try to add something that doesn't match the type?

```
=colors (~(put by colors) [%cerulean '#02A4D3'])
```

We'll see a `mull-grow`, a `mull-nice`, and a [`nest-fail`](/hoon/hoon-errors#nest-fail). Essentially these are all flavors of mold-matching errors.

(As an aside, `+put:by` is also how you'd replace an existing key's value.)

The point of a `+map` is to make it easy to retrieve data values given their appropriate key. Use [+get:by](/hoon/stdlib/2i#getby):

```
> (~(get by colors) %orange)
[~ 0xff.8833]
```

What is that cell? Wasn't the value stored as `0xff.8833`? Well, one fundamental problem that a [`+map`](/hoon/stdlib/2o#map) needs to solve is to allow us to distinguish an *empty* result (or failure to locate a value) from a *zero* result (or an answer that's actually zero). To this end, the [`+unit`](/hoon/stdlib/1c#unit) was introduced, a type union of a `~` (for no result) and `[~ item]` (for when a result exists).

(As an exercise, what does `[~ ~]` mean when returned from a `+map`?)

You'll find that `+unit`s are common enough that they have their own syntax and set of operational functions. We'll look at them more in [the next module](/build-on-urbit/hoon-school/l-struct).

```
> (~(get by colors) %brown)
~
```

The arm ([+got:by](/hoon/stdlib/2i#gotby) returns the key's value without the `+unit` wrapper, but crashes on failure to locate. We recommend just using `+get:by` and extracting the tail of the resulting cell after confirming it isn't null with `?~` [wutsig](/hoon/rune/wut#wutsig). See also [`+gut:by`](/hoon/stdlib/2i#gutby) which allows a default in case of failure to locate.)

You can check whether a key is present using [`+has:by`](/hoon/stdlib/2i#hasby):

```
> (~(has by colors) %teal)
%.n

> (~(has by colors) %green)
%.y
```

You can get a list of all keys with [`+key:by`](/hoon/stdlib/2i#keyby):

```
> ~(key by colors)
{%black %red %blue %violet %green %yellow %orange}
```

You can apply a gate to each value using [`+run:by`](/hoon/stdlib/2i#runby). For instance, these gates will break the color hexadecimal value into red, green, and blue components:

```
> =red |=(a=@ux ^-(@ux (cut 2 [4 2] a)))

> =green |=(a=@ux ^-(@ux (cut 2 [2 2] a)))

> =blue |=(a=@ux ^-(@ux (cut 2 [0 2] a)))

> (~(run by colors) blue)
{ [p=%black q=0x0]  
 [p=%red q=0x3f]  
 [p=%blue q=0xff]  
 [p=%violet q=0xa3]  
 [p=%green q=0x38]  
 [p=%yellow q=0x70]  
 [p=%orange q=0x33]  
}
```

### Exercise: Display Cards <a href="#exercise-display-cards" id="exercise-display-cards"></a>

Recall the `/lib/playing-cards.hoon` library. Use a map to pretty-print the `$darc`s as Unicode card symbols.

The map type should be `(map darc @t)`. We'll use [`+malt`](/hoon/stdlib/2l#malt) to build it and associate the fancy (if tiny) [Unicode playing card symbols](https://en.wikipedia.org/wiki/Playing_cards_in_Unicode).

Add the following arms to the library core:

<details>

<summary>+pp-card and +card-table arms</summary>

```hoon
++  pp-card
  |=  c=darc
  (~(got by card-table) c)
++  card-table
  %-  malt
  ^-  (list [darc @t])
  :~  :-  [sut=%clubs val=1]  '🃑'
      :-  [sut=%clubs val=2]  '🃒'
      :-  [sut=%clubs val=3]  '🃓'
      :-  [sut=%clubs val=4]  '🃔'
      :-  [sut=%clubs val=5]  '🃕'
      :-  [sut=%clubs val=6]  '🃖'
      :-  [sut=%clubs val=7]  '🃗'
      :-  [sut=%clubs val=8]  '🃘'
      :-  [sut=%clubs val=9]  '🃙'
      :-  [sut=%clubs val=10]  '🃚'
      :-  [sut=%clubs val=11]  '🃛'
      :-  [sut=%clubs val=12]  '🃝'
      :-  [sut=%clubs val=13]  '🃞'
      :-  [sut=%diamonds val=1]  '🃁'
      :-  [sut=%diamonds val=2]  '🃂'
      :-  [sut=%diamonds val=3]  '🃃'
      :-  [sut=%diamonds val=4]  '🃄'
      :-  [sut=%diamonds val=5]  '🃅'
      :-  [sut=%diamonds val=6]  '🃆'
      :-  [sut=%diamonds val=7]  '🃇'
      :-  [sut=%diamonds val=8]  '🃈'
      :-  [sut=%diamonds val=9]  '🃉'
      :-  [sut=%diamonds val=10]  '🃊'
      :-  [sut=%diamonds val=11]  '🃋'
      :-  [sut=%diamonds val=12]  '🃍'
      :-  [sut=%diamonds val=13]  '🃎'
      :-  [sut=%hearts val=1]  '🂱'
      :-  [sut=%hearts val=2]  '🂲'
      :-  [sut=%hearts val=3]  '🂳'
      :-  [sut=%hearts val=4]  '🂴'
      :-  [sut=%hearts val=5]  '🂵'
      :-  [sut=%hearts val=6]  '🂶'
      :-  [sut=%hearts val=7]  '🂷'
      :-  [sut=%hearts val=8]  '🂸'
      :-  [sut=%hearts val=9]  '🂹'
      :-  [sut=%hearts val=10]  '🂺'
      :-  [sut=%hearts val=11]  '🂻'
      :-  [sut=%hearts val=12]  '🂽'
      :-  [sut=%hearts val=13]  '🂾'
      :-  [sut=%spades val=1]  '🂡'
      :-  [sut=%spades val=2]  '🂢'
      :-  [sut=%spades val=3]  '🂣'
      :-  [sut=%spades val=4]  '🂤'
      :-  [sut=%spades val=5]  '🂥'
      :-  [sut=%spades val=6]  '🂦'
      :-  [sut=%spades val=7]  '🂧'
      :-  [sut=%spades val=8]  '🂨'
      :-  [sut=%spades val=9]  '🂩'
      :-  [sut=%spades val=10]  '🂪'
      :-  [sut=%spades val=11]  '🂫'
      :-  [sut=%spades val=12]  '🂭'
      :-  [sut=%spades val=13]  '🂮'
  ==
```

</details>

Import the library in Dojo (or use `/+` [faslus](/hoon/rune/fas#faslus) in a generator) and build a deck:

```
> =playing-cards -build-file /===/lib/playing-cards/hoon

> =deck (shuffle-deck:playing-cards make-deck:playing-cards eny)
> deck
~[
  [sut=%spades val=12]
  [sut=%spades val=8]
  [sut=%hearts val=5]
  [sut=%clubs val=2]
  [sut=%diamonds val=10]
  ...
  [sut=%spades val=2]
  [sut=%hearts val=6]
  [sut=%hearts val=12]
]
```

Finally, render each card in the hand to a `@t`:

```
> =new-deck (draw:playing-cards 5 deck)

> =/  index  0
  =/  hand  *(list @t)
  |-
  ?:  =(index (lent hand:new-deck))
    hand
  $(index +(index), hand (snoc hand (pp-card:playing-cards (snag index hand:new-deck))))
<|🂭 🂨 🂵 🃒 🃊|>
```

#### Tutorial: Caesar Cipher

The Caesar cipher is a shift cipher ([that was indeed used anciently](https://en.wikipedia.org/wiki/Caesar_cipher)) wherein each letter in a message is encrypted by replacing it with one shifted some number of positions down the alphabet. For example, with a “right-shift” of 1, "a" would become "b", "j" would become "k", and "z" would wrap around back to "a".

Consider the message below, and the cipher that results when we Caesar-shift the message to the right by 1.

```
Plaintext message:   "do not give way to anger"
Right-shifted cipher: "ep opu hjwf xbz up bohfs"
```

Below is a generator that performs a Caesar cipher on a tape. This example isn't the most compact implementation of such a cipher in Hoon, but it demonstrates important principles that more laconic code would not. Save it as `/gen/caesar.hoon` on your `%base` desk.

<details>

<summary>/gen/caesar.hoon</summary>

```hoon
!:
|=  [msg=tape steps=@ud]
=<
=.  msg  (cass msg)
:-  (shift msg steps)
    (unshift msg steps)
::
|%
++  alpha  "abcdefghijklmnopqrstuvwxyz"
::  Shift a message to the right.
::
++  shift
  |=  [message=tape steps=@ud]
  ^-  tape
  (operate message (encoder steps))
::  Shift a message to the left.
::
++  unshift
  |=  [message=tape steps=@ud]
  ^-  tape
  (operate message (decoder steps))
::  Rotate forwards into encryption.
::
++  encoder
  |=  [steps=@ud]
  ^-  (map @t @t)
  =/  value-tape=tape  (rotation alpha steps)
  (space-adder alpha value-tape)
::  Rotate backwards out of encryption.
::
++  decoder
  |=  [steps=@ud]
  ^-  (map @t @t)
  =/  value-tape=tape  (rotation alpha steps)
  (space-adder value-tape alpha)
::  Apply the map of decrypted->encrypted letters to the message.
::
++  operate
  |=  [message=tape shift-map=(map @t @t)]
  ^-  tape
  %+  turn  message
  |=  a=@t
  (~(got by shift-map) a)
::  Handle spaces in the message.
::
++  space-adder
  |=  [key-position=tape value-result=tape]
  ^-  (map @t @t)
  (~(put by (map-maker key-position value-result)) ' ' ' ')
::  Produce a map from each letter to its encrypted value.
::
++  map-maker
  |=  [key-position=tape value-result=tape]
  ^-  (map @t @t)
  =|  chart=(map @t @t)
  ?.  =((lent key-position) (lent value-result))
    ~|  %uneven-lengths  !!
  |-
  ?:  |(?=(~ key-position) ?=(~ value-result))
    chart
  $(chart (~(put by chart) i.key-position i.value-result), key-position t.key-position, value-result t.value-result)
::  Cycle an alphabet around, e.g. from
::  'ABCDEFGHIJKLMNOPQRSTUVWXYZ' to 'BCDEFGHIJKLMNOPQRSTUVWXYZA'
::
++  rotation
  |=  [my-alphabet=tape my-steps=@ud]
  =/  length=@ud  (lent my-alphabet)
  =+  (trim (mod my-steps length) my-alphabet)
  (weld q p)
--
```

</details>

This generator takes two arguments: a tape, which is your plaintext message, and an unsigned integer, which is the shift-value of the cipher. It produces a cell of two `$tape`s: one that has been shifted right by the value, and another that has been shifted left. It also converts any uppercase input into lowercase.

Try it out in the Dojo:

```
> +caesar ["abcdef" 1]
["bcdefg" "zabcde"]

> +caesar ["test" 2]
["vguv" "rcqr"]

> +caesar ["test" 26]
["test" "test"]

> +caesar ["test" 28]
["vguv" "rcqr"]

> +caesar ["test" 104]
["test" "test"]

> +caesar ["tESt" 2]
["vguv" "rcqr"]

> +caesar ["test!" 2]
nest-fail
```

**Examining the Code**

Let's examine our caesar.hoon code piece by piece. We won't necessarily go in written order; instead, we'll cover code in the intuitive order of the program. For each chunk that we cover, try to read and understand the code itself before reading the explanation.

There are a few runes in this which we haven't seen yet; we will deal with them incidentally in the commentary.

```hoon
!:
|=  [msg=tape steps=@ud]
=<
```

The `!:` [zapcol](/hoon/rune/zap#zapcol) in the first line of the above code enables a full stack trace in the event of an error.

`|= [msg=tape steps=@ud]` creates a gate that takes a cell. The head of this cell is a `$tape`, which is a string type that's a list of `$cord`s. Tapes are represented as text surrounded by double-quotes, such as this: `"a tape"`. We give this input tape the face `msg`. The tail of our cell is a `@ud` -- an unsigned decimal atom -- that we give the face `steps`.

`=<` [zapgal](/hoon/rune/tis#tisgal) is the rune that evaluates its first child expression with respect to its second child expression as the subject. In this case, we evaluate the expressions in the code chunk below against the core declared later, which allows us reference the core's contained arms before they are defined. Without `=<`, we would need to put the code chunk below at the bottom of our program. In Hoon, as previously stated, we always want to keep the longer code towards the bottom of our programs - `=<` helps us do that.

```hoon
=.  msg  (cass msg)
:-  (shift msg steps)
    (unshift msg steps)
```

`=. msg (cass msg)` changes the input string `.msg` to lowercases. `=.` [tisdot](/hoon/rune/tis#tisdot) changes the leg of the subject to something else. In our case, the leg to be changed is `.msg`, and the thing to replace it is `(cass msg)`. `+cass` is a gate in `hoon.hoon` that converts uppercase letters to lowercase.

`:- (shift msg steps)` and `(unshift msg steps)` simply composes a cell of a right-shifted cipher and a left-shifted cipher of our original message. We will see how this is done using the core described below, but this is the final output of our generator. We have indented the lower line, which is not strictly good Hoon style but makes the intent clearer.

```hoon
|%
```

`|%` [barcen](/hoon/rune/bar#barcen) creates a core, the second child of `=<` [tisgal](/hoon/rune/tis#tisgal). Everything after `|%` is part of that second child `+core`, and will be used as the subject of the first child of `=<`, described above. The various parts, or arms, of the `+core` are denoted by `++` [luslus](/hoon/rune/lus#luslus) beneath it, for instance:

```hoon
++  rotation
  |=  [my-alphabet=tape my-steps=@ud]
  =/  length=@ud  (lent my-alphabet)
  =+  (trim (mod my-steps length) my-alphabet)
  (weld q p)
```

The `+rotation` arm takes takes a specified number of characters off of a tape and puts them on the end of the tape. We're going to use this to create our shifted alphabet, based on the number of `.my-steps` given as an argument to our gate.

`|= [my-alphabet=tape my-steps=@ud]` creates a gate that takes two arguments: `.my-alphabet`, a `$tape`, and `.my-steps`, a `@ud`.

`=/ length=@ud (lent my-alphabet)` stores the length of `.my-alphabet` to make the following code a little clearer.

The [`+trim`](/hoon/stdlib/4b#trim) gate from the standard library splits a tape into two parts at a specified position. So `=+ (trim (mod my-steps length) my-alphabet)` splits the tape `.my-alphabet` into two parts, `.p` and `.q`, which are now directly available in the subject. We call the modulus operation `+mod` to make sure that the point at which we split our `$tape` is a valid point inside of `.my-alphabet` even if `.my-steps` is greater than `.length`, the length of `.my-alphabet`. Try trim in the dojo:

```
> (trim 2 "abcdefg")
[p="ab" q="cdefg"]

> (trim 4 "yourbeard")
[p="your" q="beard"]
```

The expression `(weld q p)` uses [`+weld`](/hoon/stdlib/2b#weld), which combines two strings into one. Remember that `+trim` has given us a split version of `.my-alphabet` with `.p` being the front half that was split off of `.my-alphabet` and `.q` being the back half. Here we are welding the two parts back together, but in reverse order: the second part `.q` is welded to the front, and the first part `.p` is welded to the back.

```hoon
++  map-maker
  |=  [key-position=tape value-result=tape]
  ^-  (map @t @t)
  =|  chart=(map @t @t)
  ?.  =((lent key-position) (lent value-result))
    ~|  %uneven-lengths  !!
  |-
  ?:  |(?=(~ key-position) ?=(~ value-result))
    chart
  $(chart (~(put by chart) i.key-position i.value-result), key-position t.key-position, value-result t.value-result)
```

The `+map-maker` arm, as the name implies, takes two tapes and creates a [`+map`](/hoon/stdlib/2o#map) out of them. A `+map` is a type equivalent to a dictionary in other languages: it's a data structure that associates a key with a value. If, for example, we wanted to have an association between `'a'` to `'1'` and `'b'` to `'2'`, we could use a `+map`.

`|= [a=tape b=tape]` builds a gate that takes two tapes, `.a` and `.b`, as its sample.

`^- (map @t @t)` casts the gate to a `+map` with a `$cord` (or `@t`) key and a `$cord` value.

You might wonder, if our gate in this arm takes `$tape`s, why then are we producing a map of `$cord` keys and values?

As we discussed earlier, a tape is a list of `$cord`s. In this case what we are going to do is map a single element of a `$tape` (either our alphabet or shifted-alphabet) to an element of a different `$tape` (either our shifted-alphabet or our alphabet). This pair will therefore be a pair of `$cord`s. When we go to use this `+map` to convert our incoming `.msg`, we will take each element (`$cord`) of our `.msg` `$tape`, use it as a key when accessing our `+map` and get the corresponding value from that position in the `+map`. This is how we're going to encode or decode our `.msg` `$tape`.

`=| chart=(map @t @t)` adds a noun to the subject with the default value of the `(map @t @t)` type, and gives that noun the face `chart`.

`?. =((lent key-position) (lent value-result))` checks if the two `$tape`s are the same length. If not, the program crashes with an error message of `%uneven-lengths`, using `~| %uneven-lengths !!`.

If the two `$tape`s are of the same length, we continue on to create a trap. `|-` [barhep](/hoon/rune/bar#barhep) creates a trap, a gate with no arguments that is called immediately.

`?: |(?=(~ key-position) ?=(~ value-result))` checks if either `$tape` is empty. If this is true, the `+map-maker` arm is finished and can return `.chart`, the [map](/hoon/stdlib/2o#map) that we have been creating.

If the above test finds that the `$tape`s are not empty, we trigger a recursion that constructs our `+map`: `$(chart (~(put by chart) i.a i.b), a t.a, b t.b)`. This code recursively adds an entry in our `+map` where the head of the `$tape` `.a` maps to the value of the head of `$tape` `.b` with `~(put by chart)`, our calling of the [`+put:by`](/hoon/stdlib/2i#putby) arm of the [`+by`](/hoon/stdlib/2i#by) map engine core (note that "\~(\<wing> \<door> \<sample>)" is a shorthand for "%\~ \<wing> \<door> \<sample>" (see the `%~` [censig](/hoon/rune/cen#censig) documentation for more information). The recursion also "consumes" those heads with every iteration by changing `.a` and `.b` to their tails using `a t.a, b t.b`.

We have three related arms to look at next, `+decoder`, `+encoder`, and `+space-adder`. `+space-adder` is required for the other two, so we'll look at it first.

```hoon
++  space-adder
  |=  [key-position=tape value-result=tape]
  ^-  (map @t @t)
  (~(put by (map-maker key-position value-result)) ' ' ' ')
```

`|= [key-position=tape value-result=tape]` creates a gate that takes two `$tape`s.

We use the [`+put`](/hoon/stdlib/2i#putby) arm of the [`+by`](/hoon/stdlib/2i#by) core on the next line, giving it a [`+map`](/hoon/stdlib/2o#map) produced by the `+map-maker` arm that we created before as its sample. This adds an entry to the map where the single space character ("ace") simply maps to itself. This is done to simplify the handling of spaces in tapes we want to encode, since we don't want to shift them.

```hoon
++  encoder
  |=  [steps=@ud]
  ^-  (map @t @t)
  =/  value-tape=tape  (rotation alpha steps)
  (space-adder alpha value-tape)
++  decoder
  |=  [steps=@ud]
  ^-  (map @t @t)
  =/  key-tape=tape  (rotation alpha steps)
  (space-adder key-tape alpha)
```

`+encoder` and `+decoder` utilize the `+rotation` and `+space-adder` arms. These gates are essentially identical, with the arguments passed to `+space-adder` reversed. They simplify the two common transactions you want to do in this program: producing `+map`s that we can use to encode and decode messages.

In both cases, we create a gate that accepts a `@ud` named `.steps`. In `+encoder`: `=/ value-tape=tape (rotation alpha steps)` creates a `.value-tape` noun by calling `+rotation` on `+alpha`. `+alpha` is our arm which contains a `$tape` of the entire alphabet. The `.value-tape` will be the list of values in our [map](/hoon/stdlib/2o#map).

In `+decoder`: `=/ key-tape (rotation alpha steps)` does the same work, but when passed to `+space-adder` it will be the list of keys in our `+map`.

The expressions `(space-adder alpha value-tape)`, for `+encoder`, and `(space-adder key-tape alpha)`, for `+decoder`, both produce a `+map` that has the first argument as the keys and the second as the values.

If our two inputs to `+space-adder` were `"abcdefghijklmnopqrstuvwxyz"` and `"bcdefghijklmnopqrstuvwxyza"`, we would get a `+map` where `'a'` maps to `'b'`, `'b'` to `'c'` and so on. By doing this we can produce a `+map` that gives us a translation between the alphabet and our shifted alphabet, or vice versa.

Still with us? Good. We are finally about to use all the stuff that we've walked through.

```hoon
++  shift
  |=  [message=tape shift-steps=@ud]
  ^-  tape
  (operate message (encoder shift-steps))
++  unshift
  |=  [message=tape shift-steps=@ud]
  ^-  tape
  (operate message (decoder shift-steps))
```

Both `+shift` and `+unshift` take two arguments: our `.message`, the `$tape` that we want to manipulate; and our `.shift-steps`, the number of positions of the alphabet by which we want to shift our message.

`+shift` is for encoding, and `+unshift` is for decoding. Thus, `+shift` calls the `+operate` arm with `(operate message (encoder shift-steps))`, and `+unshift` makes that call with `(operate message (decoder shift-steps))`. These both produce the final output of the core, to be called in the form of `(shift msg steps)` and `(unshift msg steps)` in the cell being created at the beginning of our code.

```hoon
++  operate
  |=  [message=tape shift-map=(map @t @t)]
  ^-  tape
  %+  turn  message
  |=  a=@t
  (~(got by shift-map) a)
```

`+operate` produces a `$tape`. The `%+` [cenlus](/hoon/rune/cen#cenlus) rune allows us to pull an arm with a pair sample. The arm we are going to pull is [`+turn`](/hoon/stdlib/2b#turn). This arm takes two arguments, a list and a gate to apply to each element of the `+list`.

In this case, the `$gate` we are applying to our `.message` uses the [`+got`](/hoon/stdlib/2i#gotby) arm of the [`+by`](/hoon/stdlib/2i#by) door with our `.shift-map` as the sample (which is either the standard alphabet for keys, and the shifted alphabet for values, or the other way, depending on whether we are encoding or decoding) to look up each `$cord` in our `.message`, one by one and replace it with the value from our `+map` (either the encoded or decoded version).

Let's give our arm Caesar's famous statement (translated into English!) and get our left-cipher and right-cipher.

```
> +caesar ["i came i saw i conquered" 4]
["m geqi m wea m gsruyivih" "e ywia e ows e ykjmqanaz"]
```

Now, to decode, we can put either of our ciphers in with the appropriate key and look for the legible result.

```
> +caesar ["m geqi m wea m gsruyivih" 4]
["q kium q aie q kwvycmzml" "i came i saw i conquered"]

> +caesar ["e ywia e ows e ykjmqanaz" 4]
["i came i saw i conquered" "a usew a kso a ugfimwjwv"]
```

**Further Exercise**

1. Take the example generator and modify it to add a second layer of shifts.
2. Extend the example generator to allow for use of characters other than a-z. Make it shift the new characters independently of the alpha characters, such that punctuation is only encoded as other punctuation marks.
3. Build a gate that can take a Caesar shifted `$tape` and produce all possible unshifted `$tape`s.
4. Modify the example generator into a `%say` generator.

## A Bit More on Cores <a href="#a-bit-more-on-cores" id="a-bit-more-on-cores"></a>

The `|^` [barket](/hoon/rune/bar#barket) rune is an example of what we can call a "convenience rune", similar to the idea of sugar syntax (irregular syntax to make writing certain things out in a more expressive manner). `|^` [barket](/hoon/rune/bar#barket) produces a core with *at least* a `$` arm and computes it immediately, called a "cork". (So a cork is like a trap in the regard of computing immediately, but it has more arms than just `$`.)

This code calculates the volume of a cylinder, *A=πr²h*.

```
=volume-of-cylinder |^
(mul:rs (area-of-circle .2.0) height)
++  area-of-circle
  |=  r=@rs
  (mul:rs pi r)
++  pi  .3.1415926
++  height  .10.0
--
```

Since all of the values either have to be pinned ahead of time or made available as arms, a `|^` [barket](/hoon/rune/bar#barket) would probably be used inside of a gate. Of course, since it is a core with a `$` arm, one could also use it recursively to calculate values like the factorial.

If you read the docs, you'll find that a `|-` [barhep](/hoon/rune/bar#barhep) rune “produces a trap (a core with one arm `$`) and evaluates it.” So a trap actually evaluates to a `|%` [barcen](/hoon/rune/bar#barcen) core with an arm `$`:

```hoon
:: count to five
=/  index  1
|-
?:  =(index 5)  index
$(index +(index))
```

actually translates to

```hoon
:: count to five
=/  index  1
=<  $
|%
++  $
  ?:  =(index 5)  index
  %=  $
    index  +(index)
  ==
--
```

You can also create a trap for later use with the `|.` [bardot](/hoon/rune/bar#bardot) rune. It's quite similar, but without the `=<($...` part then it doesn't get evaluated immediately.

```
> =forty-two |.(42)
> $:forty-two
42
> (forty-two)
42
```

What is a gate? It is a door with only one arm `$`, and whenever you invoke it then that default arm's expression is referred to and evaluated.

A "gate" and a "trap" are actually very similar: a gate simply has a sample (and can actively change when evaluated or via a `%=` [centis](/hoon/rune/cen#centis)), whereas a trap does not (and can *only* be passively changed via something like `%=` centis).

#### Example: Hoon Workbook

Other examples demonstrating [`+map`](/hoon/stdlib/2o#map) are available in the [Hoon Workbook](/hoon/examples), such as Solution #2 in the [Rhonda Numbers](/hoon/examples/rhonda) tutorial.


# 11. Data Structures

Guide to Hoon data structures including trees, sets, jars, and jugs. Covers mold builders, type system handling, compiler type checking, and practical manipulation operations.

*This module will introduce you to several useful data structures built on the door, then discuss how the compiler handles types and the sample.*

## Key Data Structures and Molds <a href="#key-data-structures-and-molds" id="key-data-structures-and-molds"></a>

Hoon's [`+map`s](/hoon/stdlib/2o#map) are a versatile way to store and access data, but they are far from the only useful pattern. `+map`s were documented in [the previous module](/build-on-urbit/hoon-school/k-doors).

### `+tree` <a href="#tree" id="tree"></a>

We use [`+tree`](/hoon/stdlib/1c#tree) to make a binary `+tree` data structure in Hoon, e.g., `(tree @)` for a binary `+tree` of atoms.

There are two kinds of `+tree` in Hoon:

1. The null `+tree` `~`.
2. A non-null `+tree` which is a cell with three parts.
   1. The node value.
   2. The left child of the node.
   3. The right child of the node.

Each child is itself a `+tree`. The node value has the face `n`, the left child has the face `l`, and the right child has the face `r`. The following diagram provides an illustration of a `(tree @)` (without the faces):

```
          12
        /    \
      8       14
    /   \    /   \
   4     ~  ~     16
 /  \            /  \
~    ~          ~    ~
```

Hoon supports `+tree`s of any type that can be constructed in Hoon, e.g.: `(tree @)`, `(tree ^)`, `(tree [@ ?])`, etc. Let's construct the `+tree` in the diagram above in the dojo, casting it accordingly:

```
> `(tree @)`[12 [8 [4 ~ ~] ~] [14 ~ [16 ~ ~]]]
{4 8 12 14 16}
```

Notice that we don't have to insert the faces manually; by casting the noun above to a `(tree @)` Hoon inserts the faces for us. Let's put this noun in the Dojo subject with the face "b" and pull out the `+tree` at the left child of the `12` node:

```
> =b `(tree @)`[12 [8 [4 ~ ~] ~] [14 ~ [16 ~ ~]]]

> b
{4 8 12 14 16}

> l.b
-find.l.b
find-fork
```

This didn't work because we haven't first proved to Hoon that `.b` is a non-null `+tree`. A null `+tree` has no `.l` in it, after all. Let's try again, using `?~` [wutsig](/hoon/rune/wut#wutsig) to prove that `.b` isn't null. We can also look at `.r` and `.n`:

```
> ?~(b ~ l.b)
{4 8}

> ?~(b ~ r.b)
{14 16}

> ?~(b ~ n.b)
12
```

#### Find and Replace

Here's a program that finds and replaces certain atoms in a `(tree @)`.

```hoon
|=  [nedl=@ hay=(tree @) new=@]
^-  (tree @)
?~  hay  ~
:+  ?:  =(n.hay nedl)
      new
    n.hay
  $(hay l.hay)
$(hay r.hay)
```

The `.nedl` is the atom to be replaced, `.hay` is the `+tree`, and `.new` is the new atom with which to replace `.nedl`. Save this as `findreplacetree.hoon` and run in the dojo:

```
> b
{4 8 12 14 16}

> +findreplacetree [8 b 94]
{4 94 12 14 16}

> +findreplacetree [14 b 94]
{4 8 12 94 16}
```

### `+set` <a href="#set" id="set"></a>

A [`+set`](/hoon/stdlib/2o#set) is rather like a list except that each entry can only be represented once. As with a [`+map`](/hoon/stdlib/2o#map), a `+set` is typically associated with a particular type, such as `(set @ud)` for a collection of decimal values. (`+set`s also don't have an order, so they're basically a bag of unique values.)

`+set` operations are provided by [`+in`](/hoon/stdlib/2h#in). Most names are similar to the `+map` operations in [`+by`](/hoon/stdlib/2i#by) operations when applicable.

[`+silt`](/hoon/stdlib/2l#silt) produces a `+set` from a `+list`:

```
=primes (silt ~[2 3 5 7 11 13])
```

[+put:in](/hoon/stdlib/2h#putin) adds a value to a `+set` (and no-ops when the value is already present):

```
=primes (~(put in primes) 17)
=primes (~(put in primes) 13)
```

[+del:in](/hoon/stdlib/2h#delin) removes a value from a `+set`:

```
=primes (~(put in primes) 18)
=primes (~(del in primes) 18)
```

[+has:in](/hoon/stdlib/2h#hasin) checks for existence:

```
> (~(has in primes) 15)
%.n

> (~(has in primes) 17)
%.y
```

[+tap:in](/hoon/stdlib/2h#tapin) yields a `+list` of the values:

```
> ~(tap in primes)
~[3 2 7 5 11 13 17]

> (sort ~(tap in primes) lth)
~[2 3 5 7 11 13 17]
```

[+run:in](/hoon/stdlib/2h#runin) applies a function across all values:

```
> (~(run in primes) dec)
{10 6 12 1 2 16 4}
```

#### Example: Cartesian Product

Here's a program that takes two [`+set`s](/hoon/stdlib/2o#set) of atoms and returns the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) of those sets. A Cartesian product of two sets `.a` and `.b` is a set of all the cells whose head is a member of `.a` and whose tail is a member of `.b`.

```hoon
|=  [a=(set @) b=(set @)]
=/  c=(list @)  ~(tap in a)
=/  d=(list @)  ~(tap in b)
=|  acc=(set [@ @])
|-  ^-  (set [@ @])
?~  c  acc
%=  $
  c  t.c
  acc  |-  ?~  d  acc
       %=  $
         d  t.d
         acc  (~(put in acc) [i.c i.d])
       ==
==
```

Save this as `cartesian.hoon` in your `%base` desk and run in the dojo:

```
> =c `(set @)`(sy ~[1 2 3])

> c
{1 2 3}

> =d `(set @)`(sy ~[4 5 6])

> d
{5 6 4}

> +cartesian [c d]
{[2 6] [1 6] [3 6] [1 4] [1 5] [2 4] [3 5] [3 4] [2 5]}
```

### `+unit` Redux (and `$vase`) <a href="#unit-redux-and-vase" id="unit-redux-and-vase"></a>

We encountered the [`+unit`](/hoon/stdlib/1c#unit) briefly as a tool for distinguishing null results from actual zeroes: using a `+unit` allows you to specify something that may not be there. For this reason, `+unit`s are commonly used for operations that sometimes fail, such as search functions, database lookups, remote data requests, etc.

You can build a `+unit` using the tic special notation or [`+some`](/hoon/stdlib/2a#some):

```
> `%mars
[~ %mars]

> (some %mars)
[~ u=%mars]
```

While [+got:by](/hoon/stdlib/2i#gotby) is one way to get a value back without wrapping it in a `+unit`, it's better practice to use the [`+unit` logic](/hoon/stdlib/2a) gates to manipulate gates to work correctly with `+unit`s.

For example, use [`+need`](/hoon/stdlib/2a#need) to unwrap a `+unit`, or crash if the `+unit` is `~` null.

```
> =colors (malt `(list (pair @tas @ux))`~[[%red 0xed.0a3f] [%yellow 0xfb.e870] [%green 0x1.a638] [%blue 0x66ff]])

> (~(get by colors) %yellow)
[~ q=0xfb.e870]

> (need (~(get by colors) %yellow))
0xfb.e870

> (~(get by colors) %teal)
~

> (need (~(get by colors) %teal))
dojo: hoon expression failed
```

Rather than unwrap a [`+unit`](/hoon/stdlib/1c#unit), one can modify gates to work with `+unit`s directly even if they're not natively set up that way. For instance, one cannot decrement a `+unit` because [`+dec`](/hoon/stdlib/1a#dec) doesn't accept a `+unit`. [`+bind`](/hoon/stdlib/2a#bind) can bind a non-`+unit` function, another gate-building gate!.

```
> (bind ((unit @ud) [~ 2]) dec)  
[~ 1]

> (bind (~(get by colors) %orange) red)  
[~ 0xff]
```

(There are several others tools listed [in the standard library's `+unit` logic reference](/hoon/stdlib/2a) which may be potentially useful to you.)

A vase is a pair of type and value, such as that returned by `!>` [zapgar](/hoon/rune/zap#zapgar). A `$vase` is useful when transmitting data in a way that may lose its type information.

### Containers of Containers <a href="#containers-of-containers" id="containers-of-containers"></a>

[`+map`s](/hoon/stdlib/2o#map) and [`+set`s](/hoon/stdlib/2o#set) are frequently used in the standard library and in the extended ecosystem. There are other common patterns which recur often enough that they have their own names:

[`+jar`](/hoon/stdlib/2o#jar) is a mold for a `+map` of `+list`s. `+jar` uses the [`+ja`](/hoon/stdlib/2j#ja) core. (Mnemonic: jars hold solid ordered things, like a list.)

[`+jug`](/hoon/stdlib/2o#jug) is a mold for a `+map` of `+set`s. `+jug` uses the [`+ju`](/hoon/stdlib/2j#ju) core. (Mnemonic: jugs hold liquids, evoking the unordered nature of a [`+set`](/hoon/stdlib/2o#set).)

[`+mip`](/hoon/mip) is a mold for a map of maps. `+mip` lives in the `%landscape` desk in `/lib/mip.hoon`. Affordances are still few but a short example follows:

```
> =mip -build-file /=landscape=/lib/mip/hoon

> =my-map-warm (malt `(list (pair @tas @ux))`~[[%red 0xed.0a3f] [%yellow 0xfb.e870]])

> =my-map-cool (malt `(list (pair @tas @ux))`~[[%green 0x1.a638] [%blue 0x66ff]])

> =my-mip *(mip:mip @tas (map @tas @ux))

> =my-mip (~(put bi:mip my-mip) %cool %blue 0x66ff)

> =my-mip (~(put bi:mip my-mip) %cool %green 0x1.a638)

> =my-mip (~(put bi:mip my-mip) %warm %red 0xed.0a3f)

> =my-mip (~(put bi:mip my-mip) %warm %yellow 0xfb.e870)

> my-mip
[ n=[p=%warm q=[n=[p=%yellow q=0xfb.e870] l=[n=[p=%red q=0xed.0a3f] l=~ r=~] r=~]]
  l=[n=[p=%cool q=[n=[p=%green q=0x1.a638] l=[n=[p=%blue q=0x66ff] l=~ r=~] r=~]] l=~ r=~]
  r=~
]

> (~(got bi:mip my-mip) %cool %green)
0x1.a638

> ~(tap bi:mip my-mip) ~[ [x=%warm y=%yellow v=0xfb.e870] [x=%warm
y=%red v=0xed.0a3f] [x=%cool y=%green v=0x1.a638] [x=%cool y=%blue
v=0x66ff] ]
```

Note that `+mip`s are unjetted and quite slow, but serve as a proof of concept.

[`+mop`](/hoon/zuse/2m#mop) ordered maps are discussed in [the App School guides](/build-on-urbit/app-school).

## Molds and Samples <a href="#molds-and-samples" id="molds-and-samples"></a>

### Modifying Gate Behavior <a href="#modifying-gate-behavior" id="modifying-gate-behavior"></a>

Sometimes you need to modify parts of a core (like a gate) on-the-fly to get the desired behavior. For instance, if you are using [`+roll`](/hoon/stdlib/2b#roll) to calculate the multiplicative product of the elements of a list, this “just works”:

```
> (roll `(list @ud)`~[10 12 14 16 18] mul)  
483.840
```

In contrast, if you do the same thing to a list of numbers with a fractional part (`@rs` floating-point values), the naïve operation will fail:

```
> (roll `(list @rs)`~[.10 .12 .14 .16 .18] mul:rs)  
.0
```

Why is this? Let's peek inside the gates and see. Since we know a core is a cell of \[battery payload], let's take a look at the payload:

```
> +:mul
[[a=1 b=1] <33.uof 1.pnw %138>]

> +:mul:rs
[[a=.0 b=.0] <21.ezj [r=?(%d %n %u %z) <51.njr 139.oyl 33.uof 1.pnw %138>]>]
```

The correct behavior for [`+mul:rs`](/hoon/stdlib/3b#mulrs) is really to multiply starting from one, not zero, so that [`+roll`](/hoon/stdlib/2b#roll) doesn't wipe out the entire product.

### Custom Samples <a href="#custom-samples" id="custom-samples"></a>

In an earlier exercise we created a door with sample `[a=@ud b=@ud c=@ud]`. If we investigated, we would find that the initial value of each is `0`, the bunt value of `@ud`.

```
> +6:poly
[a=0 b=0 c=0]
```

What if we wish to define a door with a chosen sample value directly? We can make use of the `$_` [buccab](/hoon/rune/buc#buccab) rune, whose irregular form is simply `_` ("cab"). To create the door `poly` with the sample set to have certain values in the Dojo, we would write

```
> =poly |_  [a=_5 b=_4 c=_3]
++  quad
  |=  x=@ud
  (add (add (mul a (mul x x)) (mul b x)) c)
--

> (quad:poly 2)  
31
```

For our earlier example with [`+roll`](/hoon/stdlib/2b#roll), if we wanted to set the default sample to have a different value than the bunt of the type, we could use `_`:

```
> =mmul |=([a=_.1 b=_.1] (mul:rs a b))

> (roll `(list @rs)`~[.10 .12 .14 .16 .18] mmul)
.483840
```

### Named Tuples <a href="#named-tuples" id="named-tuples"></a>

A named tuple is a structured collection of values with faces. The `$:` [buccol](/hoon/rune/buc#buccol) rune forms a named tuple. We use these implicitly in an irregular form when we specify the sample of a gate, as `|=([a=@ b=@] (add a b))` expands to a `$:` [buccol](/hoon/rune/buc#buccol) expression for `[a=@ b=@]`. Otherwise, we only need these if we are building a special type like a vector (e.g. with two components like an *x* and a *y*).

### Structure Mode <a href="#structure-mode" id="structure-mode"></a>

Most Hoon expressions evaluate normally (that's what “normal” means), what we'll call "noun mode" (or "normal mode"). However, sample definitions and `+$` [lusbuc](/hoon/rune/lus#lusbuc) mold specification arms evaluate in what is called "structure mode". (You may occasionally see this the older term “spec mode”.) Structure mode expressions use a similar syntax to regular Hoon expressions but create structure definitions instead.

For instance, in eval mode if you use the irregular form `p=1` this is an irregular form of the `^=` [kettis](/hoon/rune/ket#kettis) rune. This is one way to define a variable using a `=+` [tislus](/hoon/rune/tis#tislus); these are equivalent statements:

```
> =+(hello=1 hello)
1

> =+(^=(hello 1) hello)
1
```

(Normally we have preferred `=/` [tisfas](/hoon/rune/tis#tisfas) in the Hoon School docs, but that is just for consistency.)

In a sample definition, such as in a gate, the statement is evaluated in structure mode; these are equivalent statements:

```hoon
|=(hello=@ hello)

|=($=(hello @) hello)
```

There are several other subtle cases where normal mode and structure mode diverge, but most of the time structure mode is invisible to you. The [`$` buc runes](/hoon/rune/buc) are typically invoked in structure mode.


# 12. Type Checking

Detailed examination of Hoon's type checking system, covering type casting, type inference, compile-time verification, mold application, and debugging nest-fail errors in complex type hierarchies.

*In this module we'll cover how the Hoon compiler infers type, as well as various cases in which a type check is performed.*

## Type Casting <a href="#type-casting" id="type-casting"></a>

Casting is used to explain to the Hoon compiler exactly what it is we mean with a given data structure. As you get in the habit of casting your data structures, it will not only help anyone reading your code, but it will help you in hunting down bugs in your code.

[`+list`](/hoon/stdlib/1c#list) is a mold builder that is used to produce a mold, i.e. a `+list` of a particular type (like `(list @)` for a list of atoms). A `+list` can be thought of as an ordered arrangement of zero or more elements terminated by a `~` (null). There is a difference to Hoon, however, between something explicitly tagged as a `+list` of some kind and a null-terminated tuple.

```
> -:!>(~[1 2 3])
#t/[@ud @ud @ud %~]

> -:!>(`(list @)`~[1 2 3])
#t/it(@)
```

The former is inflexible and doesn't have the "i" and "t" faces that a `+list` presents. By marking the type explicitly as a `(list @)` for the compiler, we achieve some stronger guarantees that many of the `+list` operators require.

However, we still don't get the faces for free:

```
> =a `(list @)`~[1 2 3]

> i.a
-find.i.a
find-fork
dojo: hoon expression failed
```

What's going on? Formally, a `+list` can be either null or non-null. When the `+list` contains only `~` and no items, it's the null `+list`. Most `+list`s are, however, non-null `+list`s, which have items preceding the `~`. Non-null lists, called `+lest`s, are cells in which the head is the first list item, and the tail is the rest of the list. The tail is itself a `+list`, and if such a `+list` is also non-null, the head of this sublist is the second item in the greater list, and so on. To illustrate, let's look at a list `[1 2 3 4 ~]` with the cell-delineating brackets left in:

```hoon
[1 [2 [3 [4 ~]]]]
```

It's easy to see where the heads are and where the nesting tails are. The head of the above list is the atom `1` and the tail is the list `[2 [3 [4 ~]]]`, (or `[2 3 4 ~]`). Recall that whenever cell brackets are omitted so that visually there appears to be more than two child nouns, it is implicitly understood that the right-most nouns constitute a cell.

You can construct `+list`s of any type. `(list @)` indicates a list of atoms, `(list ^)` indicates a list of cells, `(list [@ ?])` indicates a list of cells whose head is an atom and whose tail is a flag, etc.

```
> `(list @)`~
~

> `(list @)`[1 2 3 4 5 ~]
~[1 2 3 4 5]

> `(list @)`[1 [2 [3 [4 [5 ~]]]]]
~[1 2 3 4 5]

> `(list @)`~[1 2 3 4 5]
~[1 2 3 4 5]
```

Notice how the last Dojo command has a different construction, with the `~` in front of the bracketed items. This is just another way of writing the same thing; `~[1 2]` is semantically identical to `[1 2 ~]`.

Back to our earlier example:

```
> =a `(list @)`~[1 2 3]

> i.a
-find.i.a
find-fork
dojo: hoon expression failed
```

Any time we see a "find-fork" error, it means that the type checker considers the value to be underspecified. In this case, it can't guarantee that `.i.a` exists because although `.a` is a `+list`, it's not known to be a non-null `+lest`. If we enforce that constraint, then suddenly we can use the faces:

```
> ?:  ?=(~ a)  !!  i.a
1
```

It's important to note that performing tests like this will actually transform a `+list` into a `+lest`, a non-null list. Because `+lest` is a different type than `+list`, performing such tests can come back to bite you later in non-obvious ways when you try to use some standard library functions meant for `+list`s.

### Casting Nouns (`^` ket Runes) <a href="#casting-nouns-ket-runes" id="casting-nouns-ket-runes"></a>

As the Hoon compiler compiles your Hoon code, it does a type check on certain expressions to make sure they are guaranteed to produce a value of the correct type. If it cannot be proved that the output value is correctly typed, the compile will fail with a [`nest-fail`](/hoon/hoon-errors#nest-fail) crash. In order to figure out what type of value is produced by a given expression, the compiler uses type inference on that code.

Let's enumerate the most common cases where a type check is called for in Hoon.

The most obvious case is when there is a casting `^` [ket](/hoon/rune/ket) rune in your code. These runes don't directly have any effect on the compiled result of your code; they simply indicate that a type check should be performed on a piece of code at compile-time.

#### `^-` kethep Cast with a Type

You've already seen one rune that calls for a type check: `^-` [kethep](/hoon/rune/ket#kethep):

```
> ^-(@ 12)
12

> ^-(@ 123)
123

> ^-(@ [12 14])
nest-fail

> ^-(^ [12 14])
[12 14]

> ^-(* [12 14])
[12 14]

> ^-(* 12)
12

> ^-([@ *] [12 [23 43]])
[12 [23 43]]

> ^-([@ *] [[12 23] 43])
nest-fail
```

#### `^+` ketlus Cast with an Example Value

The rune `^+` [ketlus](/hoon/rune/ket#ketlus) is like `^-` [kethep](/hoon/rune/ket#kethep), except that instead of using a type name for the cast, it uses an example value of the type in question. E.g.:

```
> ^+(7 12)
12

> ^+(7 123)
123

> ^+(7 [12 14])
nest-fail
```

The `^+` [ketlus](/hoon/rune/ket#ketlus) rune takes two subexpressions. The first subexpression is evaluated and its type is inferred. The second subexpression is evaluated and its inferred type is compared against the type of the first. If the type of the second provably nests under the type of the first, the result of the `^+` ketlus expression is just the value of its second subexpression. Otherwise, the code fails to compile.

This rune is useful for casting when you already have a noun, or an expression producing a noun, whose type you may not know or be able to construct easily. If you want your output value to be of the same type, you can use `^+` ketlus.

More examples:

```
> ^+([12 13] [123 456])
[123 456]

> ^+([12 13] [123 [12 14]])
nest-fail

> ^+([12 [1 2]] [123 [12 14]])
[123 12 14]
```

### Arm Checks <a href="#arm-checks" id="arm-checks"></a>

Whenever an arm is evaluated in Hoon it expects to have some version of its parent core as the subject. Specifically, a type check is performed to see whether the arm subject is of the appropriate type. We see this in action whenever a gate or a multi-arm door is called.

A gate is a one-armed core with a sample. When it is called, its `$` arm is evaluated with (a mutated copy of) the gate as the subject. The only part of the core that might change is the payload, including the sample. Of course, we want the sample to be able to change. The sample is where the argument(s) of the function call are placed. For example, when we call [`+add`](/hoon/stdlib/1a#add) the `$` arm expects two atoms for the sample, i.e., the two numbers to be added. When the type check occurs, the payload must be of the appropriate type. If it isn't, the result is a [`nest-fail`](/hoon/hoon-errors#nest-fail) crash.

```
> (add 22 33)
55

> (add [10 22] [10 33])
nest-fail

> (|=(a=@ [a a]) 15)
[15 15]

> (|=(a=@ [a a]) 22)
[22 22]

> (|=(a=@ [a a]) [22 22])
nest-fail
```

We'll talk in more detail about the various kinds of type-checking that can occur at arm evaluation [when we discuss type polymorphism](/build-on-urbit/hoon-school/r-metals).

This isn't a comprehensive list of the type checks in Hoon: for instance, some other runes that include a type check are `=.` [tisdot](/hoon/rune/tis#tisdot) and `%_` [cencab](/hoon/rune/cen#cencab).

## Type Inference <a href="#type-inference" id="type-inference"></a>

Hoon infers the type of any given expression. How does this inference work? Hoon has available various tools for inferring the type of any given expression: literal syntax, cast expressions, gate sample definitions, conditional expressions, and more.

### Literals <a href="#literals" id="literals"></a>

[Literals](https://en.wikipedia.org/wiki/Literal_%28computer_programming%29) are expressions that represent fixed values as they *literally* exist in the source code. Atom and cell literals are supported in Hoon, and every supported aura has an unambiguous representation that allows the parser to directly infer the type from the form. Here are a few examples of auras and associated literal formats:

| Type         | Literal               |
| ------------ | --------------------- |
| `@ud`        | `123`, `1.000`        |
| `@ux`        | `0x1234`, `0x12.3456` |
| `@ub`        | `0b1011.1110`         |
| `[@ud @ud]`  | `[12 14]`             |
| `[@ux @t ?]` | `[0x1f 'hello' %.y]`  |

### Casts <a href="#casts" id="casts"></a>

Casting with `^` [ket](/hoon/rune/ket) runes also shape how Hoon understands an expression type, as outlined above. The inferred type of a cast expression is just the type being cast for. It can be inferred that, if the cast didn't result in a [`nest-fail`](/hoon/hoon-errors#nest-fail), the value produced must be of the cast type. Here are some examples of cast expressions with the inferred output type on the right:

| Type        | Cast                            |
| ----------- | ------------------------------- |
| `@ud`       | `^-(@ud 123)`                   |
| `@`         | `^-(@ 123)`                     |
| `^`         | `^-(^ [12 14])`                 |
| `[@ @]`     | `^-([@ @] [12 14])`             |
| `*`         | `^-(* [12 14])`                 |
| `@ud`       | `^+(7 123)`                     |
| `[@ud @ud]` | `^+([7 8] [12 14])`             |
| `[@ud @ud]` | `^+([44 55] [12 14])`           |
| `[@ux @ub]` | `^+([0x1b 0b11] [0x123 0b101])` |

You can also use the irregular `` ` `` syntax for casting in the same way as `^-` [kethep](/hoon/rune/ket#kethep); e.g., `` `@`123 `` for `^-(@ 123)`.

Since casts can throw away type information, if the cast type is more general, then the more specific type information is lost. Consider the literal `[12 14]`. The inferred type of this expression is `[@ @]`, i.e., a cell of two atoms. If we cast over `[12 14]` with `^-(^ [12 14])` then the inferred type is just `^`, the set of all cells. The information about what kind of cell it is has been thrown away. If we cast over `[12 14]` with `^-(* [12 14])` then the inferred type is `*`, the set of all nouns. All interesting type information is thrown away on the latter cast.

It's important to remember to include a cast rune with each gate and trap expression. That way it's clear what the inferred product type will be for calls to that core.

### (Dry) Gate Sample Definitions <a href="#dry-gate-sample-definitions" id="dry-gate-sample-definitions"></a>

By now you've used the `|=` [bartis](/hoon/rune/bar#bartis) rune to define several gates. This rune is used to produce a "dry gate", which has different type-checking and type-inference properties than a "wet gate" does. We won't explain the distinction until [a later module](/build-on-urbit/hoon-school/r-metals); for now, just keep in mind that we're only dealing with one kind of gate (albeit the more common kind).

The first subexpression after the `|=` defines the sample type. Any faces used in this definition have the type declared for it in this definition. Consider an addition generator `/gen/sum.hoon`:

```hoon
|=  [a=@ b=@]
^-  @
?:  =(b 0)
  a
$(a +(a), b (dec b))
```

We run it in the Dojo using a cell to pass the two arguments:

```
> +sum [12 14]
26

> +sum 22
nest-fail
-need.[a=@ b=@]
-have.@ud
```

If you try to call this gate with the wrong kind of argument, you get a [`nest-fail`](/hoon/hoon-errors#nest-fail). If the call succeeds, then the argument takes on the type of the sample definition: `[a=@ b=@]`. Accordingly, the inferred type of `.a` is `@`, and the inferred type of `.b` is `@`. In this case some type information has been thrown away; the inferred type of `[12 14]` is `[@ud @ud]`, but the addition program takes all atoms, regardless of aura.

### Inferring Type (`?` wut Runes) <a href="#inferring-type-wut-runes" id="inferring-type-wut-runes"></a>

#### Using Conditionals for Inference by Branch

You have learned about a few conditional runes (e.g., `?:` [wutcol](/hoon/rune/wut#wutcol) and `?.` [wutdot](/hoon/rune/wut#wutdot)), but other runes of the `?` family are used for branch-specialized type inference. The `?@` [wutpat](/hoon/rune/wut#wutpat), `?^` [wutket](/hoon/rune/wut#wutket), and `?~` [wutsig](/hoon/rune/wut#wutsig) conditionals each take three subexpressions, which play the same basic role as the corresponding subexpressions of `?:` wutcol. The first is the test condition, which evaluates to a flag `?`. If the test condition is true, the second subexpression is evaluated; otherwise the third. These second and third subexpressions are the ‘branches’ of the conditional.

There is also a `?=` [wuttis](/hoon/rune/wut#wuttis) rune for pattern-matching expressions by type, returning `%.y` for a match and `%.n` otherwise.

**`?=` wuttis Non-recursive Type Match Test**

The `?=` [wuttis](/hoon/rune/wut#wuttis) rune takes two subexpressions. The first subexpression should be a type. The second subexpression is evaluated and the resulting value is compared to the first type. If the value is an instance of the type, `%.y` is produced. Otherwise, `%.n`. Examples:

```
> ?=(@ 12)
%.y

> ?=(@ [12 14])
%.n

> ?=(^ [12 14])
%.y

> ?=(^ 12)
%.n

> ?=([@ @] [12 14])
%.y

> ?=([@ @] [[12 12] 14])
%.n
```

`?=` [wuttis](/hoon/rune/wut#wuttis) expressions ignore aura information:

```
> ?=(@ud 0x12)
%.y

> ?=(@ux 'hello')
%.y
```

We haven't talked much about types that are made with a type constructor yet. We'll discuss these more soon, but it's worth pointing out that every `+list` type qualifies as such, and hence should not be used with `?=`:

```
> ?=((list @) ~[1 2 3 4])
fish-loop
```

Using these non-basic constructed types with the `?=` [wuttis](/hoon/rune/wut#wuttis) rune results in a `fish-loop` error.

The `?=` [wuttis](/hoon/rune/wut#wuttis) rune is particularly useful when used with the `?:` [wutcol](/hoon/rune/wut#wutcol) rune, because in these cases Hoon uses the result of the `?=` wuttis evaluation to infer type information. To see how this works lets use `=/` [tisfas](/hoon/rune/tis#tisfas) to define a face, `b`, as a generic noun:

```
> =/(b=* 12 b)
12
```

The inferred type of the final `.b` is just `*`, because that's how `.b` was defined earlier. We can see this by using `?` in the Dojo to see the product type:

```
> ? =/(b=* 12 b)
  *
12
```

(Remember that `?` isn't part of Hoon, it's a Dojo-specific instruction.)

Let's replace that last `.b` with a `?:` [wutcol](/hoon/rune/wut#wutcol) expression whose condition subexpression is a `?=` [wuttis](/hoon/rune/wut#wuttis) test. If `.b` is a `@`, it'll produce `[& b]`; otherwise `[| b]`:

```
> =/(b=* 12 ?:(?=(@ b) [& b] [| b]))
[%.y 12]
```

You can't see it here, but the inferred type of `.b` in `[& b]` is `@`. That subexpression is only evaluated if `?=(@ b)` evaluates as true; hence, Hoon can safely infer that `.b` must be an atom in that subexpression. Let's set `.b` to a different initial value but leave everything else the same:

```
> =/(b=* [12 14] ?:(?=(@ b) [& b] [| b]))
[%.n 12 14]
```

You can't see it here either, but the inferred type of `.b` in `[| b]` is `^`. That subexpression is only evaluated if `?=(@ b)` evaluates as false, so `.b` can't be an atom there. It follows that it must be a cell.

**The Type Spear**

What if you want to see the inferred type of `.b` for yourself for each conditional branch? One way to do this is with the "type spear". The `!>` [zapgar](/hoon/rune/zap#zapgar) rune takes one subexpression and constructs a cell from it. The subexpression is evaluated and becomes the tail of the product cell, with a `q` face attached. The head of the product cell is the inferred type of the subexpression.

```
> !>(15)
[#t/@ud q=15]

> !>([12 14])
[#t/[@ud @ud] q=[12 14]]

> !>((add 22 55))
[#t/@ q=77]
```

The `#t/` is the pretty-printer's way of indicating a type.

To get just the inferred type of a expression, we only want the head of the `!>` product, `-`. Thus we should use our mighty weapon, the type spear, `-:!>`.

```
> -:!>(15)
#t/@ud

> -:!>([12 14])
#t/[@ud @ud]

> -:!>((add 22 55))
#t/@
```

Now let's try using `?=` [wuttis](/hoon/rune/wut#wuttis) with `?:` [wutcol](/hoon/rune/wut#wutcol) again. But this time we'll replace `[& b]` with `[& -:!>(b)]` and `[| b]` with `[| -:!>(b)]`. With `.b` as `12`:

```
> =/(b=* 12 ?:(?=(@ b) [& -:!>(b)] [| -:!>(b)]))
[%.y #t/@]
```

… and with `.b` as `[12 14]`:

```
> =/(b=* [12 14] ?:(?=(@ b) [& -:!>(b)] [| -:!>(b)]))
[%.n #t/[* *]]
```

In both cases, `.b` is defined initially as a generic noun, `*`. But when using `?:` with `?=(@ b)` as the test condition, `.b` is inferred to be an atom, `@`, when the condition is true; otherwise `.b` is inferred to be a cell, `^` (identical to `[* *]`).

**`mint-vain`**

Expressions of the form `?:(?=(a b) c d)` should only be used when the previously inferred type of `.b` isn't specific enough to determine whether it nests under `.a`. This kind of expression is only to be used when `?=` can reveal new type information about `.b`, not to confirm information Hoon already has.

For example, if you have a wing expression (e.g., `.b`) that is already known to be an atom, `@`, and you use `?=(@ b)` to test whether `.b` is an atom, you'll get a [`mint-vain`](/hoon/hoon-errors#mint-vain-and-mint-lost) crash. The same thing happens if `.b` is initially defined to be a cell `^`:

```
> =/(b=@ 12 ?:(?=(@ b) [& b] [| b]))
mint-vain

> =/(b=^ [12 14] ?:(?=(@ b) [& b] [| b]))
mint-vain
```

In the first case it's already known that `.b` is an atom. In the second case it's already known that `.b` isn't an atom. Either way, the check is superfluous and thus one of the `?:` wutcol branches will never be taken. The `mint-vain` crash indicates that it's provably the case one of the branches will never be taken.

#### `?@` wutpat Atom Match Tests

The `?@` [wutpat](/hoon/rune/wut#wutpat) rune takes three subexpressions. The first is evaluated, and if its value is an instance of `@`, the second subexpression is evaluated. Otherwise, the third subexpression is evaluated.

```
> =/(b=* 12 ?@(b %atom %cell))
%atom

> =/(b=* [12 14] ?@(b %atom %cell))
%cell
```

If the second `?@` [wutpat](/hoon/rune/wut#wutpat) subexpression is evaluated, Hoon correctly infers that `.b` is an atom. if the third subexpression is evaluated, Hoon correctly infers that `.b` is a cell.

```
> =/(b=* 12 ?@(b [%atom -:!>(b)] [%cell -:!>(b)]))
[%atom #t/@]

> =/(b=* [12 14] ?@(b [%atom -:!>(b)] [%cell -:!>(b)]))
[%cell #t/[* *]]
```

If the inferred type of the first `?@` [wutpat](/hoon/rune/wut#wutpat) subexpression nests under `@` then one of the conditional branches provably never runs. Attempting to evaluate the expression results in a [`mint-vain`](/hoon/hoon-errors#mint-vain-and-mint-lost):

```
> ?@(12 %an-atom %not-an-atom)
mint-vain

> ?@([12 14] %an-atom %not-an-atom)
mint-vain

> =/(b=@ 12 ?@(b %an-atom %not-an-atom))
mint-vain

> =/(b=^ [12 14] ?@(b %an-atom %not-an-atom))
mint-vain
```

`?@` [wutpat](/hoon/rune/wut#wutpat) should only be used when it allows for Hoon to infer new type information; it shouldn't be used to confirm type information Hoon already knows.

#### `?^` wutket Cell Match Tests

The `?^` [wutket](/hoon/rune/wut#wutket) rune is just like `?@` [wutpat](/hoon/rune/wut#wutpat) except it tests for a cell match instead of for an atom match. The first subexpression is evaluated, and if the resulting value is an instance of `^` the second subexpression is evaluated. Otherwise, the third is run.

```
> =/(b=* 12 ?^(b %cell %atom))
%atom

> =/(b=* [12 14] ?^(b %cell %atom))
%cell
```

Again, if the second subexpression is evaluated Hoon infers that `.b` is a cell; if the third, Hoon infers that `.b` is an atom. If one of the conditional branches is provably never evaluated, the expression crashes with a [`mint-vain`](/hoon/hoon-errors#mint-vain-and-mint-lost):

```
> =/(b=@ 12 ?^(b %cell %atom))
mint-vain

> =/(b=^ 12 ?^(b %cell %atom))
nest-fail
```

#### Tutorial: Leaf Counting

Nouns can be understood as binary trees in which each 'leaf' of the tree is an atom. Let's look at a program that takes a noun and returns the number of leaves in it, i.e., the number of atoms.

```hoon
|=  a=*
^-  @
?@  a
  1
(add $(a -.a) $(a +.a))
```

Save this as `/gen/leafcount.hoon` in your fakeship's pier and run it from the Dojo:

```
> +leafcount 12
1

> +leafcount [12 14]
2

> +leafcount [12 [63 [829 12] 23] 13]
6
```

This program is pretty simple. If the noun `.a` is an atom, then it's a tree of one leaf; return `1`. Otherwise, the number of leaves in `.a` is the sum of the leaves in the head, `-.a`, and the tail, `+.a`.

We have been careful to use `-.a` and `+.a` only on a branch for which `.a` is proved to be a cell -- then it's safe to treat `.a` as having a head and a tail.

#### Tutorial: Cell Counting

Here's a program that counts the number of cells in a noun:

```hoon
|=  a=*
=|  c=@
|-  ^-  @
?@  a
  c
%=  $
  c  $(c +(c), a -.a)
  a  +.a
==
```

Save this as `/gen/cellcount.hoon` and run it from the Dojo:

```
> +cellcount 12
0

> +cellcount [12 14]
1

> +cellcount [12 14 15]
2

> +cellcount [[12 [14 15]] 15]
3

> +cellcount [[12 [14 15]] [15 14]]
4

> +cellcount [[12 [14 15]] [15 [14 22]]]
5
```

This code is a little more tricky. The basic idea, however, is simple. We have a counter value, `.c`, whose initial value is `0` (`=|` [tisbar](/hoon/rune/tis#tisbar) pins the bunt of the value with the given face). We trace through the noun `.a`, adding `1` to `.c` every time we come across a cell. For any part of the noun that is just an atom, `.c` is returned unchanged.

What makes this program is little harder to follow is that it recurses within a recursion call. The first recursion expression on line 6 makes changes to two face values: `.c`, the counter, and `.a`, the input noun. The new value for `.c` defined in the line `$(c +(c), a -.a)` is another recursion call (this time in irregular syntax). The new value for `.c` is to be the result of running the same function on the the head of `.a`, `-.a`, and with `1` added to `.c`. We add `1` because we know that `.a` must be a cell. Otherwise, we're asking for the number of cells in the rest of `-.a`.

Once that new value for `.c` is computed from the head of `.a`, we're ready to check the tail of `.a`, `+.a`. We've already got everything we want from `-.a`, so we throw that away and replace `.a` with `+.a`.

### Lists <a href="#lists" id="lists"></a>

You learned about `+list`s earlier in the chapter, but we left out a little bit of information about the way Hoon understands `+list` types.

A non-null list is a cell. If `.b` is a non-null list then the head of `.b` is the first item of `.b` *with an `i` face on it*. The tail of `.b` is the rest of the list. The 'rest of the list' is itself another `+list` *with a `t` face on it*. We can (and should) use these `i` and `t` faces in `+list` functions.

To illustrate: let's say that `.b` is the list of the atoms `11`, `22`, and `33`. Let's construct this in stages:

```hoon
[i=11 t=[rest-of-list-b]]

[i=11 t=[i=22 t=[rest-of-list-b]]]

[i=11 t=[i=22 t=[i=33 t=~]]]
```

(There are lists of every type. Lists of `@ud`, `@ux`, `@` in general, `^`, `[^ [@ @]]`, etc. We can even have `+list`s of `+list`s of `@`, `^`, or `?`, etc.)

#### Tutorial: List Spanning Values

Here's a program that takes atoms `.a` and `.b` and returns a `+list` of all atoms from `.a` to `.b`:

```hoon
|=  [a=@ b=@]
^-  (list @)
?:  (gth a b)
  ~
[i=a t=$(a +(a))]
```

This program is very simple. It takes two `@` as input, `.a` and `.b`, and returns a `(list @)`, i.e., a `+list` of `@`. If `.a` is greater than `.b` the list is finished: return the null `+list` `~`. Otherwise, return a non-null list: a pair in which the head is `.a` with an `i` face on it, and in which the tail is another `+list` with the `t` face on it. This embedded `+list` is the product of a recursion call: add `1` to `.a` and run the function again.

Save this code as `/gen/gulf.hoon` and run it from the Dojo:

```
> +gulf [1 10]
~[1 2 3 4 5 6 7 8 9 10]

> +gulf [10 20]
~[10 11 12 13 14 15 16 17 18 19 20]

> +gulf [20 10]
~
```

Where are all the `i`s and `t`s??? For the sake of neatness the Hoon pretty-printer doesn't show the `i` and `t` faces of `+list`s, just the items.

In fact, we could have left out the `i` and `t` faces in the program itself:

```hoon
|=  [a=@ b=@]
^-  (list @)
?:  (gth a b)
  ~
[a $(a +(a))]
```

Because there is a cast to a `(list @)` on line 2, Hoon will silently include `i` and `t` faces for the appropriate places of the noun. Remember that faces are recorded in the type information of the noun in question, not as part of the noun itself.

We called this program `gulf.hoon` because it replicates the [gulf](/hoon/stdlib/2b#gulf) function in the Hoon standard library:

```
> (gulf 1 10)
~[1 2 3 4 5 6 7 8 9 10]

> (gulf 10 20)
~[10 11 12 13 14 15 16 17 18 19 20]
```

#### `?~` wutsig Null Match Test

The `?~` [wutsig](/hoon/rune/wut#wutsig) rune is a lot like `?@` [wutpat](/hoon/rune/wut#wutpat) and `?^` [wutket](/hoon/rune/wut#wutket). It takes three subexpressions, the first of which is evaluated to see whether the result is `~` null. If so, the second subexpression is evaluated. Otherwise, the third one is evaluated.

```
> =/(b=* ~ ?~(b %null %not-null))
%null

> =/(b=* [12 13] ?~(b %null %not-null))
%not-null
```

The inferred type of `.b` must not already be known to be null or non-null; otherwise, the expression will crash with a [`mint-vain`](/hoon/hoon-errors#mint-vain-and-mint-lost):

```
> =/(b=~ ~ ?~(b %null %not-null))
mint-vain

> =/(b=^ [10 12] ?~(b %null %not-null))
mint-vain

> ?~(~ %null %not-null)
mint-vain
```

Hoon will infer that `.b` either is or isn't null based on which `?~` branch is evaluated after the test.

**Using `?~` wutsig With Lists**

`?~` [wutsig](/hoon/rune/wut#wutsig) is especially useful for working with `+list`s. Is a list null, or not? You probably want to do different things based on the answer to that question. Above, we used a pattern of `?:` [wutcol](/hoon/rune/wut#wutcol) and `?=` [wuttis](/hoon/rune/wut#wuttis) to answer the question, but `?~` wutsig will let us know in one step. Here's a program using `?~` wutsig to calculate the number of items in a `+list` of atoms:

```hoon
|=  a=(list @)
=|  c=@
|-  ^-  @
?~  a
  c
$(c +(c), a t.a)
```

This function takes a `+list` of `@` and returns a `@`. It uses `.c` as a counter value, initially set at `0` on line 2. If `.a` is `~` (i.e., a null `+list`) then the computation is finished; return `.c`. Otherwise `.a` must be a non-null list, in which case there is a recursion to the `|-` [barhep](/hoon/rune/bar#barhep) on line 3, but with `.c` incremented, and with the head of the list `.a` thrown away.

It's important to note that if `.a` is a `+list`, you can only use `i.a` and `t.a` after Hoon has inferred that `.a` is non-null. A null `+list` has no `.i` or `.t` in it! You'll often use `?~` to distinguish the two kinds of list (null and non-null). If you use `i.a` or `t.a` without showing that `.a` is non-null you'll get a "find-fork" crash.

A non-null `+list` is called a `+lest`.

Save the above code as `/gen/lent.hoon` and run it from the Dojo:

```
> +lent ~[11 22 33]
3

> +lent ~[11 22 33 44 55 77]
6

> +lent ~[0xff 0b11 'howdy' %hello]
4
```

#### Tutorial: Converting a Noun to a List of its Leaves

Here's a program that takes a noun and returns a `+list` of its 'leaves' (atoms) in order of their appearance:

```hoon
|=  a=*
=/  lis=(list @)  ~
|-  ^-  (list @)
?@  a
  [i=a t=lis]
$(lis $(a +.a), a -.a)
```

The input noun is `.a`. The `+list` of atoms to be output is `.lis`, which is given an initial value of `~`. If `.a` is just an atom, return a non-null list whose head is `.a` and whose tail is `.lis`. Otherwise, the somewhat complicated recursion `$(lis $(a +.a), a -.a)` is evaluated, in effect looping back to the `|-` with modifications made to `.lis` and `.a`.

The modification to `.lis` in line 6 is to `$(a +.a)`. The latter is a recursion to `|-` but with `.a` replaced by its tail. This evaluates to the `+list` of `@` in the tail of `.a`. So `.lis` becomes the `+list` of atoms in the tail of `.a`, and `.a` becomes the head of `.a`, `-.a`.

Save the above code as `/gen/listleaf.hoon` and run it from the Dojo:

```
> +listleaf [[[[12 13] [33 22] 12] 11] 33]
~[12 13 33 22 12 11 33]
```

### Other Kinds of Type Inference <a href="#other-kinds-of-type-inference" id="other-kinds-of-type-inference"></a>

So far you've learned about four kinds of type inference:

1. Literals
2. Explicit casts
3. Gate sample definitions
4. Branch specialization using runes in the `?` family

There are several other ways that Hoon infers type. Any rune expression that evaluates to a `?` flag, e.g., `.=` [dottis](/hoon/rune/dot#dottis), will be inferred from accordingly. The `.+` [dotlus](/hoon/rune/dot#dotlus) rune always evaluates to a `@`, and Hoon knows that too. The cell constructor runes, `:-` [colhep](/hoon/rune/col#colhep), `:+` [collus](/hoon/rune/col#collus), `:^` [colket](/hoon/rune/col#colket), and `:*` [coltar](/hoon/rune/col#coltar) are all known to produce cells.

More subtly, the `=+` [tislus](/hoon/rune/tis#tislus), `=/` [tisfas](/hoon/rune/tis#tisfas), and `=|` [tisbar](/hoon/rune/tis#tisbar) runes modify the subject by pinning values to the head. Hoon infers from this that the subject has a new type: a cell whose head is the type of the pinned value and whose tail is the type of the (old) subject.

In general, anything that modifies the subject modifies the type of the subject. Type inference can work in subtle ways for various expressions. However, we have covered enough that it should be relatively clear how to anticipate how type inference works for the vast majority of ordinary use cases.

## Auras as 'Soft' Types <a href="#auras-as-soft-types" id="auras-as-soft-types"></a>

It's important to understand that Hoon's type system doesn't enforce auras as strictly as it does other types. Auras are 'soft' type information. To see how this works, we'll take you through the process of converting the aura of an atom to another aura.

Hoon makes an effort to enforce that the correct aura is produced by an expression:

```
> ^-(@ud 0x10)
nest-fail

> ^-(@ud 0b10)
nest-fail

> ^-(@ux 100)
nest-fail
```

But there are ways around this. First, you can cast to a more general aura, as long as the current aura nests under the cast aura. E.g., `@ub` to `@u`, `@ux` to `@u`, `@u` to `@`, etc. By doing this you're essentially telling Hoon to throw away some aura information:

```
> ^-(@u 0x10)
16

> ? ^-(@u 0x10)
  @u
16

> ^-(@u 0b10)
2

> ? ^-(@u 0b10)
  @u
2
```

In fact, you can cast any atom all the way to the most general case `@`:

```
> ^-(@ 0x10)
16

> ? ^-(@ 0x10)
  @
16

> ^-(@ 0b10)
2

> ? ^-(@ 0b10)
  @
2
```

Anything of the general aura `@` can, in turn, be cast to more specific auras. We can show this by embedding a cast expression inside another cast:

```
> ^-(@ud ^-(@ 0x10))
16

> ^-(@ub ^-(@ 0x10))
0b1.0000

> ^-(@ux ^-(@ 10))
0xa
```

Hoon uses the outermost cast to infer the type:

```
> ? ^-(@ub ^-(@ 0x10))
  @ub
0b1.0000
```

As you can see, an atom with one aura can be converted to another aura. For a convenient shorthand, you can do this conversion with irregular cast syntax, e.g. `` `@ud` ``, rather than using the `^-` [kethep](/hoon/rune/ket#kethep) rune twice:

```
> `@ud`0x10
16

> `@ub`0x10
0b1.0000

> `@ux`10
0xa
```

This is what we mean when we call auras 'soft' types. The above examples show that the programmer can get around the type system for auras by casting up to `@` and then back down to the specific aura, say `@ub`; or by casting with `` `@ub` `` for short.

**Note**: there is currently a type system issue that causes some of these functions to fail when passed a `+list` `.b` after some type inference has been performed on `.b`. For an illustration of the bug, let's set `.b` to be a `(list @)` of `~[11 22 33 44]` in the Dojo:

```
> =b `(list @)`~[11 22 33 44]

> b
~[11 22 33 44]
```

Now let's use `?~` [wutsig](/hoon/rune/wut#wutsig) to prove that `.b` isn't null, and then try to snag it:

```
> ?~(b ~ (snag 0 b))
nest-fail
```

The problem is that [`+snag`](/hoon/stdlib/2b#snag) is expecting a raw `+list`, not a list that is known to be non-null.

You can cast `.b` back to `(list)` to work around this:

```
> ?~(b ~ (snag 0 `(list)`b))
11
```

### Pattern Matching and Assertions <a href="#pattern-matching-and-assertions" id="pattern-matching-and-assertions"></a>

To summarize, as values get passed around and checked at various points, the Hoon compiler tracks what the possible data structure or mold looks like. The following runes are particularly helpful when inducing the compiler to infer what it needs to know:

* `?~` [wutsig](/hoon/rune/wut#wutsig) asserts non-null.
* `?^` [wutket](/hoon/rune/wut#wutket) asserts cell.
* `?@` [wutpat](/hoon/rune/wut#wutpat) asserts atom.
* `?=` [wuttis](/hoon/rune/wut#wuttis) tests for a pattern match in type.

There are two additional assertions which can be used with the type system:

* `?>` [wutgar](/hoon/rune/wut#wutgar) is a positive assertion (`%.y` or crash).
* `?<` [wutgal](/hoon/rune/wut#wutgal) is a negative assertion (`%.n` or crash).

If you are running into `find-fork` errors in more complicated data structures (like marks or `$json`s), consider using these assertions to guide the typechecker.


# 13. Conditional Logic

Guide to conditional logic in Hoon, covering loobean (boolean) values, logical operators (AND, OR, NOT), conditional branching, switch statements, and pattern matching.

*Although you've been using various of the `?`* [*wut*](/hoon/rune/wut) *runes for a while now, let's wrap up some loose ends. This module will cover the nature of loobean logic and the rest of the `?` wut runes.*

## Loobean Logic <a href="#loobean-logic" id="loobean-logic"></a>

Throughout Hoon School, you have been using `%.y` and `%.n`, often implicitly, every time you have asked a question like `?: =(5 4)`. The `=()` expression returns a loobean, a member of the type union `?(%.y %.n)`. (There is a proper aura `@f` but unfortunately it can't be used outside of the compiler.) These can also be written as `&` (`%.y`, true) and `|` (`%.n`, false), which is common in older code but should be avoided for clarity in your own compositions.

What are the actual values of these, *sans* formatting?

```
> `@`%.y
0

> `@`%.n
1
```

## Noun Equality <a href="#noun-equality" id="noun-equality"></a>

The most fundamental comparison in Hoon is provided by `.=` [dottis](/hoon/rune/dot#dottis), a test for equality of two nouns using Nock 5. This is almost always used in its irregular form of `=` tis.

```
> =(0 0)
%.y

> =('a' 'b')
%.n
```

Since Nock is unaware of the Hoon metadata type system, only bare atoms in the nouns are compared. If you need to compare include type information, create vases with `!>` [zapgar](/hoon/rune/zap#zapgar).

```
> =('a' 97)
%.y

> =(!>('a') !>(97))
%.n
```

## Making Choices <a href="#making-choices" id="making-choices"></a>

You are familiar in everyday life with making choices on the basis of a decision expression. For instance, you can compare two prices for similar products and select the cheaper one for purchase.

Essentially, we have to be able to decide whether or not some value or expression evaluates as `%.y` true (in which case we will do one thing) or `%.n` false (in which case we do another). Some basic expressions are mathematical, but we also check for existence, for equality of two values, etc.

* [`+gth`](/hoon/stdlib/1a#gth) (greater than `>`)
* [`+lth`](/hoon/stdlib/1a#lth) (less than `<`)
* [`+gte`](/hoon/stdlib/1a#gte) (greater than or equal to `≥`)
* [`+lte`](/hoon/stdlib/1a#lte) (less than or equal to `≤`)
* `.=` [dottis](/hoon/rune/dot#dottis), irregularly `=()` (check for equality)

The key conditional decision-making rune is `?:` [wutcol](/hoon/rune/wut#wutcol), which lets you branch between an `expression-if-true` and an "expression-if-false". `?.` [wutdot](/hoon/rune/wut#wutdot) inverts the order of `?:`. Good Hoon style prescribes that the heavier branch of a logical expression should be lower in the file.

There are also two long-form decision-making runes, which we will call [*switch statements*](https://en.wikipedia.org/wiki/Switch_statement) by analogy with languages like C.

`?-` [wuthep](/hoon/rune/wut#wuthep) lets you choose between several possibilities, as with a type union. Every case must be handled and no case can be unreachable.

Since `@tas` terms are constants first, and not `@tas` unless marked as such, `?-` [wuthep](/hoon/rune/wut#wuthep) switches over term unions can make it look like the expression is branching on the value. It's actually branching on the *type*. These are almost exclusively used with term type unions.

```hoon
|=  p=?(%1 %2 %3)
?-  p
  %1  1
  %2  2
  %3  3
==
```

`?+` [wutlus](/hoon/rune/wut#wutlus) is similar to `?-` but allows a default value in case no branch is taken. Otherwise these are similar to `?-` [wuthep](/hoon/rune/wut#wuthep) switch statements.

```hoon
|=  p=?(%0 %1 %2 %3 %4)
?+  p  0
  %1  1
  %2  2
  %3  3
==
```

## Logical Operators <a href="#logical-operators" id="logical-operators"></a>

Mathematical logic allows the collocation of propositions to determine other propositions. In computer science, we use this functionality to determine which part of an expression is evaluated. We can combine logical statements pairwise:

`?&` [wutpam](/hoon/rune/wut#wutpam), irregularly `&()`, is a logical `AND` (i.e. *p* ∧ *q*) over loobean values, e.g. both terms must be true.

| `AND` | `%.y` | `%.n` |
| ----- | ----- | ----- |
| `%.y` | `%.y` | `%.n` |
| `%.n` | `%.n` | `%.n` |

```
> =/  a  5
  &((gth a 4) (lth a 7))
%.y
```

`?|` [wutbar](/hoon/rune/wut#wutbar), irregularly `|()`, is a logical `OR` (i.e. *p* ∨ *q*) over loobean values, e.g. either term may be true.

| `OR`  | `%.y` | `%.n` |
| ----- | ----- | ----- |
| `%.y` | `%.y` | `%.y` |
| `%.n` | `%.y` | `%.n` |

\\

```
> =/  a  5
  |((gth a 4) (lth a 7))
%.y
```

`?!` [wutzap](/hoon/rune/wut#wutzap), irregularly `!`, is a logical `NOT` (i.e. ¬*p*). Sometimes it can be difficult to parse code including `!` because it operates without parentheses.

|       | `NOT` |
| ----- | ----- |
| `%.y` | `%.n` |
| `%.n` | `%.y` |

\\

```
> !%.y
%.n

> !%.n
%.y
```

From these primitive operators, you can build other logical statements at need.

### Exercise: Design an `XOR` Function <a href="#exercise-design-an-xor-function" id="exercise-design-an-xor-function"></a>

The logical operation `XOR` yields true if one but not both operands are true.

| `XOR` | `%.y` | `%.n` |
| ----- | ----- | ----- |
| `%.y` | `%.n` | `%.y` |
| `%.n` | `%.y` | `%.n` |

Implement `XOR` as a gate in Hoon.

### Exercise: Design a `NAND` Function <a href="#exercise-design-a-nand-function" id="exercise-design-a-nand-function"></a>

The logical operation `NAND` produces false if both operands are true.

| `NAND` | `%.y` | `%.n` |
| ------ | ----- | ----- |
| `%.y`  | `%.n` | `%.y` |
| `%.n`  | `%.y` | `%.y` |

Implement `NAND` as a gate in Hoon.

### Exercise: Design a `NOR` Function <a href="#exercise-design-a-nor-function" id="exercise-design-a-nor-function"></a>

The logical operation `NOR` produces true if both operands are false.

| `NOR` | `%.y` | `%.n` |
| ----- | ----- | ----- |
| `%.y` | `%.n` | `%.n` |
| `%.n` | `%.n` | `%.y` |

Implement `NOR` as a gate in Hoon.

### Exercise: Implement a Piecewise Boxcar Function <a href="#exercise-implement-a-piecewise-boxcar-function" id="exercise-implement-a-piecewise-boxcar-function"></a>

The boxcar function is a piecewise mathematical function which is equal to zero for inputs less than zero and one for inputs greater than or equal to zero. We implemented the similar Heaviside function [previously](/build-on-urbit/hoon-school/b-syntax) using the `?:` [wutcol](/hoon/rune/wut#wutcol) rune.

Compose a gate which implements the boxcar function,

$$
\text{boxcar}(x)
:=
\left(
\begin{matrix}
1, & 10 \leq x < 20 \\\\
0, & \text{otherwise} \\\\
\end{matrix}
\right)
$$

Use Hoon logical operators to compress the logic into a single statement using at least one `AND` or `OR` operation.


# 14. Subject-Oriented Programming

Guide to subject-oriented programming in Hoon, covering how expressions are evaluated within subjects, core composition, state management, deferred computations, and scrying local data.

*This module discusses how Urbit's subject-oriented programming paradigm structures how cores and values are used and maintain state, as well as how deferred computations and remote value lookups ("scrying") are handled. This module does not cover core genericity and variance, which will be explained in* [*a later module*](/build-on-urbit/hoon-school/r-metals)*.*

## The Subject <a href="#the-subject" id="the-subject"></a>

The Urbit operating system hews to a conceptual model wherein each expression takes place in a certain context (the subject). While sharing a lot of practicality with other programming paradigms and platforms, Urbit's model is mathematically well-defined and unambiguously specified. Every expression of Hoon is evaluated relative to its subject, a piece of data that represents the environment, or the context, of an expression.

Subject-oriented programming means that every expression is evaluated with respect to some subject. Every arm of a core is evaluated with its parent core as the subject.

You have also seen how wings work as search paths to identify nouns in the subject, and you have learned three ways to access values by address: numeric addressing, lark notation, and wing search expressions.

Generally speaking, the following rune families allow you to do certain things to the subject:

* `|` [bar](/hoon/rune/bar) runes create cores, i.e. largely self-contained expressions.
* `^` [ket](/hoon/rune/ket) runes transform cores, i.e. change core properties.
* `%` [cen](/hoon/rune/cen) runes pull arms in cores.
* `=` [tis](/hoon/rune/tis) runes modify the subject by introducing or replacing values.

Different kinds of cores can expose or conceal functionality (such as their sample) based on their variance model. We don't need to be concerned about that yet, but if you are building certain kinds of library code or intend to build code expressions directly, you'll need to read [that module](/build-on-urbit/hoon-school/r-metals) as well.

### Accessing the Subject <a href="#accessing-the-subject" id="accessing-the-subject"></a>

Usually the subject of a Hoon expression isn't shown explicitly. In fact, only when using `:`/`.` wing lookup expressions have we made the subject explicit.

An arm is always evaluated with its parent core as its subject. We've briefly mentioned that one can use helper cores (e.g. for generators) by composing the cores side-by-side using `=<` [tisgal](/hoon/rune/tis#tisgal) and `=>` [tisgar](/hoon/rune/tis#tisgar). This way we can make sure that the arms fall within each other's subject horizon.

Why must an arm have its parent core as the subject, when it's computed? As stated previously, the payload of a core contains all the data needed for computing the arms of that core. Arms can only access data in the subject. By requiring that the parent core be the subject we guarantee that each arm has the appropriate data available to it. The tail of its subject contains the payload and thus all the values therein. The head of the subject is the battery, which allows for making reference to sibling arms of that same core.

In the Dojo, if you use `+1` by itself, you can see the current subject.

```
> +1
[ [ our=~zod
    now=~2024.5.7..21.47.30..818c
      eny
    0vb6.cve93.67frc.2gtoj.jfl3i.odojg.urrce.o53d3.44h4o.sf3o5.va2mh.ra1ec.jrkej.u512k.l4lin.f003v.li030.l2e6t.ah7ge.6t5cg.epuil
  ]
  <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
]
```

The `.` operator does the same thing: it always refers to the current subject.

If `.` is the subject, then `..arm` is the subject of a given arm (the second `.` dot being the wing resolution operator). You can check the details of the parent core using something like `..add`. This trick is used when producing agents that have highly nested operations (search `..` in the `/app` directory), or when composing [jets](/build-on-urbit/runtime/jetting), for instance.

Another use case for the `..arm` syntax is when there is a core in the subject without a face bound to it; i.e., the core might be nameless. In that case you can use an arm name in that core to refer to the whole core.

```
> ..add
<33.uof 1.pnw %138>
```

#### Tutorial: The Core Structure of `hoon.hoon`

Let's take a deeper look at how cores can be combined with `=>` [tisgar](/hoon/rune/tis#tisgar) to build up larger structures. `=> p=hoon q=hoon` yields the product of `.q` with the product of `.p` taken as the subject; i.e. it composes Hoon statements, like cores.

We use this to set the context of cores. Recall that the payload of a gate is a cell of \[sample context]. For example:

```
> =foo =>([1 2] |=(@ 15))

> +3:foo
[0 1 2]
```

Here we have created a gate with `[1 2]` as its context that takes in a `@` and returns `15`. `+3:foo` shows the payload of the core to be `[0 [1 2]]`. Here `0` is the default value of `@` and is the sample, while `[1 2]` is the context that was given to `.foo`.

`=>` [tisgar](/hoon/rune/tis#tisgar) (and its reversed version `=<` [tisgal](/hoon/rune/tis#tisgal)) are used extensively to put cores into the context of other cores.

```hoon
=>
|%
++  foo
  |=  a=@
  (mul a 2)
--
|%
++  bar
  |=  a=@
  (mul (foo a) 2)
--
```

At the level of arms, `+foo` is in the subject of `+bar`, and so `+bar` is able to call `+foo`. On the other hand, `+bar` is not in the subject of `+foo`, so `+foo` cannot call `+bar` - you will get a `-find.bar` error.

At the level of cores, the `=>` sets the context of the core containing `+bar` to be the core containing `+foo`. Recall that arms are evaluated with the parent core as the subject. Thus `+bar` is evaluated with the core containing it as the subject, which has the core containing `+foo` in its context. This is why `+foo` is in the scope of `+bar` but not vice versa.

Let's look inside `/sys/hoon.hoon`, where the standard library is located, to see how this can be used.

The first core listed here has just one arm.

```hoon
=>  %138  =>
|%
++  hoon-version  +
--
```

This is reflected in the subject of `hoon-version`.

```
> ..hoon-version
<1.pnw %138>
```

After several lines that we'll ignore for pedagogical purposes, we see

```hoon
|%
::  #  %math
::    unsigned arithmetic
+|  %math
++  add
  ~/  %add
  ::  unsigned addition
  ::
  ::  a: augend
  ::  b: addend
  |=  [a=@ b=@]
  ::  sum
  ^-  @
  ?:  =(0 a)  b
  $(a (dec a), b +(b))
::
++  dec
```

and so on, down to

```hoon
++  unit
  |$  [item]
  ::    maybe
  ::
  ::  mold generator: either `~` or `[~ u=a]` where `a` is the
  ::  type that was passed in.
  ::
  $@(~ [~ u=item])
--
```

This core contains the arms in [sections 1a–1c of the standard library documentation](/hoon/stdlib/1a). If you count them, there are 33 arms in the core from `+add` down to `+unit`. We again can see this fact reflected in the Dojo by looking at the subject of `+add`.

```
> ..add
<33.uof 1.pnw %138>
```

Here we see that core containing `hoon-version` is in the subject of the section 1 core.

Next, [section 2](/hoon/stdlib/2a) starts:

```hoon
=>
::                                                      ::
::::  2: layer two                                      ::
```

...

```hoon
|%
::                                                      ::
::::  2a: unit logic                                    ::
  ::                                                    ::
  ::    biff, bind, bond, both, clap, drop,             ::
  ::    fall, flit, lift, mate, need, some              ::
  ::
++  biff                                                ::  apply
  |*  {a/(unit) b/$-(* (unit))}
  ?~  a  ~
  (b u.a)
```

If you counted the arms in this core by hand, you'll come up with 139 arms. This is also reflected in the dojo:

```
> ..biff
<139.oyl 33.uof 1.pnw %138>
```

and we also see the section 1 core and the core containing `hoon-version` in the subject.

We can also confirm that [`+add`](/hoon/stdlib/1a#add) is in the subject of [`+biff`](/hoon/stdlib/2a#biff)

```
> add:biff
<1.otf [[a=@ b=@] <33.uof 1.pnw %138>]>
```

and that `+biff` is not in the subject of `+add`.

```
> biff:add
-find.biff
```

Lastly, let's check the subject of the last arm in `hoon.hoon` (as of May 2024):

```
> ..pi-tell
<77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
```

This confirms for us, then, that `hoon.hoon` consists of six nested cores, with one inside the payload of the next, with the `hoon-version` core most deeply nested.

### Exercise: Explore `hoon.hoon` <a href="#exercise-explore-hoonhoon" id="exercise-explore-hoonhoon"></a>

Pick a couple of arms in `hoon.hoon` and check to make sure that they are only referenced in its parent core or core(s) that have the parent core put in its context via the `=>` or `=<` runes.

### Axes of the Subject <a href="#axes-of-the-subject" id="axes-of-the-subject"></a>

The core Arvo subject exposes several axes (plural of `$axis` which is the tree address) in the subject. You've encountered these before:

`.our` is the ship's identity.

```
> -<..
our=~nec
```

`.now` is 128-bit timestamp sourced from the wall clock time, Linux's `gettimeofday()`.

```
> ->-..
now=~2022.6.22..20.41.18..82f4
```

`.eny` is 512 bits of entropy as `@uvJ`, sourced from a [CSPRNG](https://en.wikipedia.org/wiki/Cryptographically-secure_pseudorandom_number_generator) and hash-iterated using [`+shax`](/hoon/stdlib/3d#shax). (`.eny` is shared between vanes during an event, so there are currently limits on how much it should be relied on until the Urbit kernel is security-hardened, but it is unique within each Gall agent activation.)

```
> ->+..
eny
0vmr.qobqc.fd9f0.h5hf4.dkurh.b4s37.lt4qf.2k505.j3sir.cnshk.ldpm0.jeppc.ti7gs.vtpru.u09sm.0imu0.cgdln.fvoqc.mt41e.3iga5.qpct7
```

## State and Applications <a href="#state-and-applications" id="state-and-applications"></a>

Default Hoon expressions are stateless. This means that they don't really make reference to any other transactions or events in the system. They don't preserve the results of previous calculations beyond their own transient existence.

However, clearly regular applications, such as Gall agents, are stateful, meaning that they modify their own subject regularly.

There are several ways to manage state. One approach, including `%=` [centis](/hoon/rune/cen#centis), directly modifies the subject using a rune. Another method is to use the other runes to compose or sequence changes together (e.g. as a pipe of gates). By and large the `=` [tis](/hoon/rune/tis) runes are responsible for modifying the subject, and the `;` [mic](/hoon/rune/mic) runes permit chaining deferred computations together.

To act in a stateful manner, a core must mutate itself and then pin the mutated copy in its place. Most of the time this is handled by Arvo's Gall vane, by the Dojo, or another system service, but we need to explicit modify and manage state for cores as we work within these kinds of applications.

We will use `%say` generators as a bridge concept. We will produce some short applications that maintain state while carrying out a calculation; they still result in a single return value, but gesture at the big-picture approach to maintaining state in persistent agents.

[As you may recall](/build-on-urbit/hoon-school/j-stdlib-text), a `%say` generator is like a naked generator except rather than being simply a gate, it is a cell of `%say` (as a tag) and a gate. This gate can receive more information as gate arguments as part of its sample, such as a timestamp `.now`, some entropy `.eny`, and a file system beak `.bec`. These allow us to think about how a core can modify and maintain state. Although a `%say` generator, like all generators, ultimately simply terminates, a Gall agent will be a persistent core with state that can continue to be used.

Here are a couple of new runes for modifying the subject and chaining computations together, aside from `%=` [centis](/hoon/rune/cen#centis) which you've already seen:

* `=.` [tisdot](/hoon/rune/tis#tisdot) is used to change a leg in the subject.
* `=~` [tissig](/hoon/rune/tis#tissig) composes many expressions together serially.

#### Tutorial: Bank Account

In this section, we will write a door that can act as a bank account with the ability to withdraw, deposit, and check the account's balance. This door replaces the sample of the door with the new values as each transaction proceeds.

```hoon
:-  %say
|=  *
:-  %noun
=<  =~  new-account
      (deposit 100)
      (deposit 100)
      (withdraw 50)
      balance
    ==
|%
++  new-account
  |_  balance=@ud
  ++  deposit
    |=  amount=@ud
    +>.$(balance (add balance amount))
  ++  withdraw
    |=  amount=@ud
    +>.$(balance (sub balance amount))
  --
--
```

We start with the three boilerplate lines we have in every `%say` generator:

```hoon
:-  %say
|=  *
:-  %noun
```

In the above code chunk, we're creating a cell. The head of this cell is `%say`. The tail is a gate (`|= *`) that produces another cell (`:- %noun`) with a head of the mark of a the kind of data we are going to produce, a `%noun`; the tail of the second cell is the rest of the program.

```hoon
=<  =~  new-account
      (deposit 100)
      (deposit 100)
      (withdraw 50)
      balance
    ==
```

In this code above, we're going to compose two runes using `=<`, which has inverted arguments. We use this rune to keep the heaviest twig to the bottom of the code.

The `=~` [tissig](/hoon/rune/tis#tissig) rune composes multiple expressions together; we use it here to make the code more readable. We take `+new-account` and use that as the subject for the call to `+deposit`. `+deposit` and `+withdraw` both produce a new version of the door that's used in subsequent calls, which is why we are able to chain them in this fashion. The final reference is to `.balance`, which is the account balance contained in the core that we examine below.

```hoon
|%
++  new-account
  |_  balance=@ud
  ++  deposit
    |=  amount=@ud
    +>.$(balance (add balance amount))
  ++  withdraw
    |=  amount=@ud
    +>.$(balance (sub balance amount))
  --
--
```

We've chosen here to wrap our door in its own core to emulate the style of programming that is used when creating libraries. `+new-account` is the name of our door. A door is a core with one or more arms that has a sample. Here, our door has a sample of one `@ud` with the face `balance` and two arms `+deposit` and `+withdraw`.

Each of these arms produces a gate which takes a `@ud` argument. Each of these gates has a similar bit of code inside:

```hoon
+>.$(balance (add balance amount))
```

`+>` is a kind of wing syntax, lark notation. This particular wing construction looks for the tail of the tail (the third element) in `$`, the subject of the gate we are in. The `+withdraw` and `+deposit` arms create gates with the entire `+new-account` door as the context in their cores' \[battery sample context], in the "tail of the tail" slot. We change `.balance` to be the result of adding `.balance` and `.amount` and produce the door as the result. `+withdraw` functions the same way only doing subtraction instead of addition.

It's important to notice that the sample, `.balance`, is stored as part of the door rather than existing outside of it.

### Exercise: Bank Account <a href="#exercise-bank-account" id="exercise-bank-account"></a>

Modify the `%say` generator above to accept a `@ud` unsigned decimal dollar amount and a `?(%deposit %withdraw)` term and returns the result of only that operation on the starting balance of the bank account. (Note that this will only work once on the door, and the state will not persist between generator calls.)

### Deferred Computations <a href="#deferred-computations" id="deferred-computations"></a>

"Deferred computation" means that parts of the subject have changes that may be underdetermined at first. These must be calculated later using the appropriate runes as new or asynchronous information becomes available.

For instance, a network service call may take a while or may fail. How should the calculation deal with these outcomes? In addition, the successful result of the network data is unpredictable in content (but should not be unpredictable in format!).

We have some more tools available for managing deferred or chained computations, in addition to `=~` [tissig](/hoon/rune/tis#tissig) and `=*` [tistar](/hoon/rune/tis#tistar):

* `=^` [tisket](/hoon/rune/tis#tisket) is used to change a leg in the tail of the subject then evaluate against it. This is commonly used for events that need to be ordered in their resolution e.g. with a `%=` [centis](/hoon/rune/cen#centis). (Used in Gall agents frequently.)
* `=*` [tistar](/hoon/rune/tis#tistar) defers an expression (rather like a macro).
* `;<` [micgal](/hoon/rune/mic#micgal) sequences two computations, particularly for an asynchronous event like a remote system call. (Used in threads.)
* `;~` [micsig](/hoon/rune/mic#micsig) produces a pipeline, a way of piping the output of one gate into another in a chain. (This is particularly helpful when parsing text.)

### `+og` Randomness <a href="#og-randomness" id="og-randomness"></a>

A *random number generator* provides a stream of calculable but unpredictable values from some *distribution*. In [a later lesson](/build-on-urbit/hoon-school/s-math), we explain how random numbers can be generated from entropy; for now, let's see what's necessary to use such a random-number generator.

An RNG emits a sequence of values given a starting *seed*. For instance, a very simple RNG could emit digits of the number *π* given a seed which is the number of digits to start from.

* Seed 1: 1, 4, 1, 5, 9, 2, 6, 5, 3, 5
* Seed 3: 1, 5, 9, 2, 6, 5, 3, 5, 8, 9
* Seed 100: 8, 2, 1, 4, 8, 0, 8, 6, 5, 1

Every time you start this “random” number generator with a given seed, it will reproduce the same sequence of numbers.

While RNGs don't work like our *π*-based example, a given seed will reliably produce the same result every time it is run.

The basic RNG core in Hoon is [`+og`](/hoon/stdlib/3d#og). `+og` is a door whose sample is its seed. We need to use `.eny` to seed it non-deterministically, but we can also pin the state using `=^` [tisket](/hoon/rune/tis#tisket). [`+rads:rng`](/hoon/stdlib/3d#radsog) produces a cell of a random whole number in a given range and a new modified core to continue the random sequence.

```
> =+  rng=~(. og eny)
  [-:(rads:rng 100) -:(rads:rng 100)]
[60 60]
```

Since the `.rng` starts from the same seed value every single time, both of the numbers will always be the same. What we have to do is pin the updated version of the RNG (the tail of `+rads:og`'s return cell) to the subject using `=^` [tisket](/hoon/rune/tis#tisket), e.g.,

```
> =/  rng  ~(. og eny)
  =^  r1  rng  (rads:rng 100)
  =^  r2  rng  (rads:rng 100)
  [r1 r2]
[21 47]
```

#### Tutorial: Magic 8-Ball

The Magic 8-Ball returns one of a variety of answers in response to a call. In its entirety:

<details>

<summary>Magic 8-Ball code</summary>

```hoon
!:
:-  %say
|=  [[* eny=@uvJ *] *]
:-  %noun
^-  tape
=/  answers=(list tape)
  :~  "It is certain."
      "It is decidedly so."
      "Without a doubt."
      "Yes - definitely."
      "You may rely on it."
      "As I see it, yes."
      "Most likely."
      "Outlook good."
      "Yes."
      "Signs point to yes."
      "Reply hazy, try again"
      "Ask again later."
      "Better not tell you now."
      "Cannot predict now."
      "Concentrate and ask again."
      "Don't count on it."
      "My reply is no."
      "My sources say no."
      "Outlook not so good."
      "Very doubtful."
  ==
=/  rng  ~(. og eny)
=/  val  (rad:rng (lent answers))
(snag val answers)
```

</details>

Zoom in on these lines:

```hoon
=/  rng  ~(. og eny)
=/  val  (rad:rng (lent answers))
```

The `~(. og eny)` expression starts a random number generator with a seed from the current entropy. A [random number generator](https://en.wikipedia.org/wiki/Random_number_generation) is a stateful mathematical function that produces an unpredictable result (unless you know the algorithm AND the starting value, or seed). Here we pull the subject of [`+og`](/hoon/stdlib/3d#og), the randomness core in Hoon, to start the RNG. An RNG like `+og` maintains its own state, but we will find that we have to preserve state changes to continue to produce novel random numbers.

We slam the `+rad:rng` gate which returns a random number from 0 to *n*-1 inclusive. This gives us a random value from the list of possible answers.

```
> +magic-8
"Ask again later."
```

## Tutorial: Dice Roll <a href="#tutorial-dice-roll" id="tutorial-dice-roll"></a>

Let's look at an example that uses all three parts. Save the code below in a file called `dice.hoon` in the `/gen` directory of your `%base` desk.

```hoon
:-  %say
|=  [[now=@da eny=@uvJ bec=beak] [n=@ud ~] [bet=@ud ~]]
:-  %noun
[(~(rad og eny) n) bet]
```

This is a very simple dice program with an optional betting functionality. In the code, our sample specifies faces on all of the Arvo data, meaning that we can easily access them. We also require the argument `[n=@ud ~]`, and allow the *optional* argument `[bet=@ud ~]`.

We can run this generator like so:

```
> +dice 6, =bet 2
[4 2]

> +dice 6
[5 0]

> +dice 6
[2 0]

> +dice 6, =bet 200
[0 200]

> +dice
nest-fail
```

We get a different value from the same generator between runs, something that isn't possible with a naked generator. Another novelty is the ability to choose to not use the second argument.

## Scrying (In Brief) <a href="#scrying-in-brief" id="scrying-in-brief"></a>

A "peek" or a scry is a request to Arvo to tell you something about the state of part of the Urbit OS. Scries are used to determine the state of an agent or a vane. The `.^` [dotket](/hoon/rune/dot#dotket) rune sends the scry request to a particular vane with a certain "care" or type of scry. The request is then routed to a particular path in that vane. Scries are discused in detail in [App School](/build-on-urbit/app-school/10-scry). We will only briefly introduce them here as we can use them later to find out about Arvo's system state, such as file contents and agent state.

### `%c` Clay <a href="#c-clay" id="c-clay"></a>

The Clay filesystem stores nouns persistently at hierarchical path addresses. These nouns can be accessed using marks, which are rules for structuring the data. We call the nouns “files” and the path addresses “folders”.

If we want to retrieve the contents of a file or folder, we can directly ask Clay for the data using a scry with an appropriate [`$care`](/urbit-os/kernel/clay/data-types#care).

For instance, the `%x` care to the `%c` Clay vane returns the noun at a given address as a `@` atom.

```
> .^(@ %cx /===/gen/hood/hi/hoon)
3.548.750.706.400.251.607.252.023.288.575.526.190.856.734.474.077.821.289.791.377.301.707.878.697.553.411.219.689.905.949.957.893.633.811.025.757.107.990.477.902.858.170.125.439.223.250.551.937.540.468.638.902.955.378.837.954.792.031.592.462.617.422.136.386.332.469.076.584.061.249.923.938.374.214.925.312.954.606.277.212.923.859.309.330.556.730.410.200.952.056.760.727.611.447.500.996.168.035.027.753.417.869.213.425.113.257.514.474.700.810.203.348.784.547.006.707.150.406.298.809.062.567.217.447.347.357.039.994.339.342.906
```

There are tools like `/lib/pretty-file/hoon` which will render this legible to you by using formatted text `$tank`s:

```
> =pretty-file -build-file %/lib/pretty-file/hoon

> (pretty-file .^(noun %cx /===/gen/hood/hi/hoon))
~[
  [%leaf p="::  Helm: send message to an urbit"]
  [%leaf p="::"]
  [%leaf p="::::  /hoon/hi/hood/gen"]
  [%leaf p="  ::"]
  [%leaf p="/?    310"]
  [%leaf p=":-  %say"]
  [%leaf p="|=([^ [who=ship mez=$@(~ [a=tape ~])] ~] helm-send-hi+[who ?~(mez ~ `a.mez)])"]
]
```

Similarly, you can request the contents at a particular directory path:

```
> .^(arch %cy /===/gen/hood)
[ fil=~
    dir
  { [p=~.resume q=~]
    [p=~.install q=~]
    [p=~.pass q=~]
    [p=~.doze q=~]
    ...
    [p=~.mount q=~]
  }
]
```

There are many more options with Clay than just accessing file and folder data. For instance, we can also scry all of the desks on our current ship with the `%d` care of `%c` Clay:

```
> .^((set desk) %cd /=//=)
{%base %landscape %webterm %kids}
```

Other vanes have their own scry interfaces, which are well-documented in [the Arvo docs](/urbit-os/kernel/arvo).


# 15. Text Processing II

Advanced text processing in Hoon, covering text conversions, formatted text with $tanks, %ask generators, I/O operations, and practical tools for web data, JSON, and output formatting.

*This module will elaborate on text representation in Hoon, including formatted text and `%ask` generators. It may be considered optional and skipped if you are speedrunning Hoon School.*

## Text Conversions <a href="#text-conversions" id="text-conversions"></a>

We frequently need to convert from text to data, and between different text-based representations. Let's examine some specific arms:

* How do we convert text into all lower-case?
  * [`+cass`](/hoon/stdlib/4b#cass)
* How do we turn a `$cord` into a tape?
  * [`+trip`](/hoon/stdlib/4b#trip)
* How can we make a list of a null-terminated tuple?
  * [`+le:nl`](/hoon/stdlib/2m#lenl)
* How can we evaluate Nock expressions?
  * [`+mink`](/hoon/stdlib/4n#mink)

(If you see a `|*` [bartar](/hoon/rune/bar#bartar) rune in the code, it's similar to a `|=` [bartis](/hoon/rune/bar#bartis), but produces what's called a ["wet gate"](/build-on-urbit/hoon-school/r-metals).)

The `+html` core of the standard libary contains some additional important tools for working with web-based data, such as [MIME types](https://en.wikipedia.org/wiki/Media_type) and [JSON strings](https://en.wikipedia.org/wiki/JSON).

To convert a `@ux` hexadecimal value to a `$cord`:

```hoon
> (en:base16:mimes:html [3 0x12.3456])  
'123456'
```

To convert a `$cord` to a `@ux` hexadecimal value:

```hoon
> `@ux`q.+>:(de:base16:mimes:html '123456')
0x12.3456
```

There are tools for working with Bitcoin wallet base-58 values, JSON strings, XML strings, and more.

```hoon
> (en-urlt:html "https://hello.me")
"https%3A%2F%2Fhello.me"
```

## Formatted Text <a href="#formatted-text" id="formatted-text"></a>

Hoon produces messages at the Dojo (or otherwise) using an internal formatted text system, called `$tank`s. A `$tank` is a formatted print tree. Error messages and the like are built of `$tank`s. `$tank`s are defined in `hoon.hoon`:

```hoon
::  $tank: formatted print tree
::
::    just a cord, or
::    %leaf: just a tape
::    %palm: backstep list
::           flat-mid, open, flat-open, flat-close
::    %rose: flat list
::           flat-mid, open, close
::
+$  tank
  $~  leaf/~
  $@  cord
  $%  [%leaf p=tape]
      [%palm p=(qual tape tape tape tape) q=(list tank)]
      [%rose p=(trel tape tape tape) q=(list tank)]
  ==
+$ tang (list tank) :: bottom-first error
```

The [`+ram:re`](/hoon/stdlib/4c#ramre) arm is used to convert these to actual formatted output as a tape, e.g.

```hoon
> ~(ram re leaf+"foo")
"foo"
> ~(ram re [%palm ["|" "(" "!" ")"] leaf+"foo" leaf+"bar" leaf+"baz" ~])
"(!foo|bar|baz)"
> ~(ram re [%rose [" " "[" "]"] leaf+"foo" leaf+"bar" leaf+"baz" ~])
"[foo bar baz]"
```

Many generators build sophisticated output using `$tank`s and the short-format cell builder `+`, e.g. in `/gen/azimuth-block/hoon`:

```hoon
[leaf+(scow %ud block)]~
```

which is equivalent to

```hoon
~[[%leaf (scow %ud block)]]
```

`$tank`s are the primary output mechanism for more advanced generators. Even if you don't end up writing them much, you will encounter them as you delve into the Urbit codebase.

#### Tutorial: Deep Dive into `ls.hoon`

The [`+ls`](/user-manual/os/dojo-tools#ls) generator shows the contents at a particular path in Clay:

```hoon
> +cat /===/gen/ls/hoon
/~nec/base/~2022.6.22..17.25.54..1034/gen/ls/hoon
::  LiSt directory subnodes
::
::::  /hoon/ls/gen
  ::
/?    310
/+    show-dir
::
::::
  ::
~&  %
:-  %say
|=  [^ [arg=path ~] vane=?(%g %c)]
=+  lon=.^(arch (cat 3 vane %y) arg)
tang+[?~(dir.lon leaf+"~" (show-dir vane arg dir.lon))]~
```

Let's go line by line:

```hoon
/?    310
/+    show-dir
```

The first line `/?` faswut represents now-future functionality which will allow the version number of the kernel to be pinned. It is currently non-functioning but you will see it in many Urbit-shipped files.

Then the `show-dir` library is imported.

```hoon
~&  %
```

A separator `%` is printed.

```hoon
:-  %say
```

A `%say` generator is a cell with a metadata tag `%say` as the head and the gate as the tail.

```hoon
|=  [^ [arg=path ~] vane=?(%g %c)]
```

This generator requires a path argument in its sample and optionally accepts a vane tag (`%g` Gall or `%c` Clay). Most of the time, [`+cat`](/user-manual/os/dojo-tools#cat) is used with Clay, so `%c` as the last entry in the type union serves as the bunt value.

```hoon
=+  lon=.^(arch (cat 3 vane %y) arg)
```

We saw `.^` [dotket](/hoon/rune/dot#dotket) for the first time in [the previous module](/build-on-urbit/hoon-school/o-subject), where we learned that it performs a "peek" or scry into the state of an Arvo vane. Most of the time this functionality is used to ask `%c` Clay or `%g` Gall for information about a path, desk, agent, etc. In this case, `(cat 3 %c %y)` is a fancy way of collocating the two `@tas` terms into `%cy`, a Clay file or directory lookup. The type of this lookup is `$arch`, and the location of the file or directory is given by `.arg` from the sample.

```hoon
tang+[?~(dir.lon leaf+"~" (show-dir vane arg dir.lon))]~
```

The result of the lookup on the previous line is adapted into a formatted text block with a head of `%tang` and different results depending on whether the request was `~` null or not.

#### Tutorial: Deep Dive into `cat.hoon`

For instance, how does [`+cat`](/user-manual/os/dojo-tools#cat) work? Let's look at the structure of `/gen/cat/hoon`:

<details>

<summary>/gen/cat.hoon</summary>

```hoon
::  ConCATenate file listings
::
::::  /hoon/cat/gen
  ::
/?    310
/+    pretty-file, show-dir
::
::::
  ::
:-  %say
|=  [^ [arg=(list path)] vane=?(%g %c)]
=-  tang+(flop `tang`(zing -))
%+  turn  arg
|=  pax=path
^-  tang
=+  ark=.^(arch (cat 3 vane %y) pax)
?^  fil.ark
  ?:  =(%sched -:(flop pax))
    [>.^((map @da cord) (cat 3 vane %x) pax)<]~
  [leaf+(spud pax) (pretty-file .^(noun (cat 3 vane %x) pax))]
?-     dir.ark                                          ::  handle ambiguity
    ~
  [rose+[" " `~]^~[leaf+"~" (smyt pax)]]~
::
    [[@t ~] ~ ~]
  $(pax (welp pax /[p.n.dir.ark]))
::
    *
  =-  [palm+[": " ``~]^-]~
  :~  rose+[" " `~]^~[leaf+"*" (smyt pax)]
      `tank`(show-dir vane pax dir.ark)
  ==
==
```

</details>

What is the top-level structure of the generator? (A cell of `%say` and the gate, what Dojo recognizes as a `%say` generator.)

Some points of interest include:

* `/?` faswut pins the expected Arvo kelvin version; right now it doesn't do anything.
* `.^` [dotket](/hoon/rune/dot#dotket) loads a value from Arvo (called a "scry").
* [`+smyt`](/hoon/stdlib/4m#smyt) pretty-prints a path.
* `=-` [tishep](/hoon/rune/tis#tishep) combines a faced noun with the subject, inverted relative to `=+` [tislus](/hoon/rune/tis#tislus) or `=/` [tisfas](/hoon/rune/tis#tisfas).

You can see how much of the generator is concerned with formatting the content of the file into a formatted text `$tank` by prepending `%rose` tags and so forth.

Work line-by-line through the file and clarify parts that are muddy to you at first glance.

### Producing Error Messages <a href="#producing-error-messages" id="producing-error-messages"></a>

Formal error messages in Urbit are built of tanks. “A `$tang` is a list of `$tank`s, and a `$tank` is a structure for printing data. There are three types of `$tank`: `$leaf`, `$palm`, and `$rose`. A `$leaf` is for printing a single noun, a `$rose` is for printing rows of data, and a `$palm` is for printing backstep-indented lists.”

One way to include an error message in your code is the `~_` [sigcab](/hoon/rune/sig#sigcab) rune, described as a “user-formatted tracing printf”, or the `~|` [sigbar](/hoon/rune/sig#sigbar) rune, a “tracing printf”. What this means is that these print to the stack trace if something fails, so you can use either rune to contribute to the error description:

```hoon
|=  a=@ud
~_  leaf+"This code failed"
!!
```

When you compose your own library functions, consider including error messages for likely failure points.

## `%ask` Generators <a href="#ask-generators" id="ask-generators"></a>

Previously, we introduced the concept of a `%say` generator to produce a more versatile form of standalone single computation than a simple naked generator (gate) allowed. Another elaboration, the `%ask` generator, takes things further.

We use an `%ask` generator when we want to create an interactive program that prompts for inputs as it runs, rather than expecting arguments to be passed in at the time of initiation.

This section will briefly walk through an `%ask` generator to give you a taste of how they work. The [CLI app guide](/build-on-urbit/userspace/cli-tutorial) walks through the libraries necessary for working with `%ask` generators in greater detail. We also recommend reading [\~wicdev-wisryt's “Input and Output in Hoon”](https://urbit.org/blog/io-in-hoon) for an extended consideration of relevant input/output issues.

**Tutorial: `%ask` Generator**

The code below is an `%ask` generator that checks if the user inputs `"blue"` when prompted [per a classic Monty Python scene](https://www.youtube.com/watch?v=L0vlQHxJTp0). Save it as `/gen/axe.hoon` in your `%base` desk.

```hoon
/-  sole
/+  generators
=,  [sole generators]
:-  %ask
|=  *
^-  (sole-result (cask tang))
%+  print    leaf+"What is your favorite color?"
%+  prompt   [%& %prompt "color: "]
|=  t=tape
%+  produce  %tang
?:  =(t "blue")
  :~  leaf+"Oh. Thank you very much."
      leaf+"Right. Off you go then."
  ==
:~  leaf+"Aaaaagh!"
    leaf+"Into the Gorge of Eternal Peril with you!"
==
```

Run the generator from the Dojo:

```hoon
> +axe

What is your favorite color?
: color:
```

Something new has happened. Instead of simply returning something, your Dojo's prompt changed from "\~your-urbit:dojo>" to "\~your-urbit:dojo: color:", and now expects additional input. Let's give it an answer:

```hoon
: color: red
Into the Gorge of Eternal Peril with you!
Aaaaagh!
```

Let's go over what exactly is happening in this code.

```hoon
/-  sole
/+  generators
=,  [sole generators]
```

Here we bring in some of the types we are going to need from `/sur/sole` and gates we will use from `/lib/generators`. We use some special runes for this.

* `/-` [fashep](/hoon/rune/fas#fashep) is a Ford rune used to import types from `/sur`.
* `/+` [faslus](/hoon/rune/fas#faslus) is a Ford rune used to import libraries from `/lib`.
* `=,` [tiscom](/hoon/rune/tis#tiscol) is a rune that allows us to expose a namespace. We do this to avoid having to write `sole-result:sole` instead of `sole-result` or `print:generators` instead of `+print`.

```hoon
:-  %ask
|=  *
```

This code might be familiar. Just as with their `%say` cousins, `%ask` generators need to produce a `+cell`, the head of which specifies what kind of generator we are running.

With `|= *`, we create a gate and ignore the standard arguments we are given, because we're not using them.

```hoon
^-  (sole-result (cask tang))
```

`%ask` generators need to have the second half of the cell be a gate that produces a `$sole-result`, one that in this case contains a `+cask` of `$tang`. We use the `^-` [kethep](/hoon/rune/ket#kethep) rune to constrain the generator's output to such a `$sole-result`.

A `+cask` is a pair of a mark name and a noun. We previously described a `$mark` as a kind of complicated mold; here we add that a `$mark` can be thought of as an Arvo-level [MIME](https://en.wikipedia.org/wiki/MIME) type for data.

A `$tang` is a list of `$tank`, and a `$tank` is a structure for printing data, as described above. There are three types of `$tank`: `$leaf`, `$palm`, and `$rose`. A `$leaf` is for printing a single noun, a `$rose` is for printing rows of data, and a `$palm` is for printing backstep-indented lists.

```hoon
%+  print    leaf+"What is your favorite color?"
%+  prompt   [%& %prompt "color: "]
|=  t=tape
%+  produce  %tang
```

Because we imported generators, we can access its contained gates, three of which we use in `axe.hoon`: `+print`, `+prompt`, and `+produce`.

**`+print` is used for printing a `$tank` to the console.**

In our example, `%+` [cenlus](/hoon/rune/cen#cenlus) is used to call the gate `+print`, with two arguments. The first argument is a `$tank` to print. The `+` here is syntactic sugar for `[%leaf "What is your favorite color?"]` that just makes it easier to write. The second argument is the output of the call to `+prompt`.

**`+prompt` is used to construct a prompt for the user to provide input.**

The first argument is a tuple. The second argument is a gate that returns the output of a call to `+produce`. Most `%ask` generators will want to use the `+prompt` gate.

The first element of the `+prompt` tuple/sample is a flag that indicates whether what the user typed should be echoed out to them or hidden. `%&` will produce echoed output and `%|` will hide the output (for use in passwords or other secret text).

The second element of the `+prompt` sample is intended to be information for use in creating autocomplete options for the prompt. This functionality is not yet implemented.

The third element of the `+prompt` sample is the tape that we would like to use to prompt the user. In the case of our example, we use `"color: "`.

**`+produce` is used to construct the output of the generator.**

In our example, we produce a `$tang`.

```hoon
|=  t=tape
```

Our gate here takes a `$tape` that was produced by `+prompt`. If we needed another type of data we could use `+parse` to obtain it.

The rest of this generator should be intelligible to those with Hoon knowledge at this point.

One quirk that you should be aware of, though, is that `$tang` prints in reverse order from how it is created. The reason for this is that `$tang` was originally created to display stack trace information, which should be produced in reverse order. This leads to an annoyance: we either have to specify our messages backwards or construct them in the order we want and then [`+flop`](/hoon/stdlib/2b#flop) the `+list`.


# 16. Functional Programming

Functional programming concepts in Hoon, including gate manipulation, currying and binding samples, changing arity, function composition with cork and corl, and working with lists.

*This module will discuss some gates-that-work-on-gates and other assorted operators that are commonly recognized as functional programming tools.*

Given a gate, you can manipulate it to accept a different number of values than its sample formally requires, or otherwise modify its behavior. These techniques mirror some of the common tasks used in other [functional programming languages](https://en.wikipedia.org/wiki/Functional_programming) like Haskell, Clojure, and OCaml.

Functional programming, as a paradigm, tends to prefer rather mathematical expressions with explicit modification of function behavior. It works as a formal system of symbolic expressions manipulated according to given rules and properties. FP was derived from the [lambda calculus](https://en.wikipedia.org/wiki/Lambda_calculus), a cousin of combinator calculi like Nock. (See also [APL](https://en.wikipedia.org/wiki/APL_%28programming_language%29).)

## Changing Arity <a href="#changing-arity" id="changing-arity"></a>

If a gate accepts only two values in its sample, for instance, you can chain together multiple calls automatically using the `;:` [miccol](/hoon/rune/mic#miccol) rune.

```
> (add 3 (add 4 5))
12

> :(add 3 4 5)
12

> (mul 3 (mul 4 5))
60

> :(mul 3 4 5)
60
```

This is called changing the [*arity*](https://en.wikipedia.org/wiki/Arity) of the gate. (Does this work on [`+mul:rs`](/hoon/stdlib/3b#mulrs)?)

## Binding the Sample <a href="#binding-the-sample" id="binding-the-sample"></a>

["Currying"](https://en.wikipedia.org/wiki/Currying) describes taking a function of multiple arguments and reducing it to a set of functions that each take only one argument. "Binding", an allied process, is used to set the value of some of those arguments permanently.

If you have a gate which accepts multiple values in the sample, you can fix one of these. To fix the head of the sample (the first argument), use [`+cury`](/hoon/stdlib/2n#cury); to bind the tail, use [`+curr`](/hoon/stdlib/2n#curr).

Consider calculating *a x² + b x + c*, a situation we earlier resolved using a door. We can resolve the situation differently using currying:

```
> =full |=([x=@ud a=@ud b=@ud c=@ud] (add (add (mul (mul x x) a) (mul x b)) c))

> (full 5 4 3 2)
117

> =one (curr full [4 3 2])  

> (one 5)  
117
```

One can also [`+cork`](/hoon/stdlib/2n#cork) a gate, or arrange it such that it applies to the result of the next gate. This pairs well with `;:` [miccol](/hoon/rune/mic#miccol). (There is also [`+corl`](/hoon/stdlib/2n#corl), which composes backwards rather than forwards.) This example decrements a value then converts it to `@ux` by corking two gates:

```
> ((cork dec @ux) 20)  
0x13
```

### Exercise: Bind Gate Arguments <a href="#exercise-bind-gate-arguments" id="exercise-bind-gate-arguments"></a>

Create a gate `+inc` which increments a value in one step, analogous to [`+dec`](/hoon/stdlib/1a#dec).

### Exercise: Chain Gate Values <a href="#exercise-chain-gate-values" id="exercise-chain-gate-values"></a>

Write an expression which yields the parent galaxy of a planet's sponsoring star by composing two gates.

## Working Across `+list`s <a href="#working-across-lists" id="working-across-lists"></a>

The [`+turn`](/hoon/stdlib/2b#turn) function takes a list and a gate, and returns a list of the products of applying each item of the input list to the gate. For example, to add 1 to each item in a list of atoms:

```
> (turn `(list @)`~[11 22 33 44] |=(a=@ +(a)))
~[12 23 34 45]
```

Or to double each item in a list of atoms:

```
> (turn `(list @)`~[11 22 33 44] |=(a=@ (mul 2 a)))
~[22 44 66 88]
```

`+turn` is Hoon's version of Haskell's `+map`.

We can rewrite the Caesar cipher program using turn:

```hoon
|=  [a=@ b=tape]
^-  tape
?:  (gth a 25)
  $(a (sub a 26))
%+  turn  b
|=  c=@tD
?:  &((gte c 'A') (lte c 'Z'))
  =.  c  (add c a)
  ?.  (gth c 'Z')  c
  (sub c 26)
?:  &((gte c 'a') (lte c 'z'))
  =.  c  (add c a)
  ?.  (gth c 'z')  c
  (sub c 26)
c
```

[`+roll`](/hoon/stdlib/2b#roll) and [`+reel`](/hoon/stdlib/2b#reel) are used to left-fold and right-fold a list, respectively. To fold a list is similar to [`+turn`](/hoon/stdlib/2b#turn), except that instead of yielding a `+list` with the values having had each applied, `+roll` and `+reel` produce an accumulated value.

```
> (roll `(list @)`[1 2 3 4 5 ~] add)
15

> (reel `(list @)`[1 2 3 4 5 ~] mul)
120
```

### Exercise: Calculate a Factorial <a href="#exercise-calculate-a-factorial" id="exercise-calculate-a-factorial"></a>

Use `+reel` to produce a gate which calculates the factorial of a number.

## Classic Operations <a href="#classic-operations" id="classic-operations"></a>

Functional programmers frequently rely on three design patterns to produce operations on collections of data:

1. Map. The Map operation describes applying a function to each item of a set or iterable object, resulting in the same final number of items transformed. In Hoon terms, we would say slamming a gate on each member of a `+list` or `+set`. The standard library arms that accomplish this include [`+turn`](/hoon/stdlib/2b#turn) for a list, [`+run:in`](/hoon/stdlib/2h#repin) for a [`+set`](/hoon/stdlib/2o#set), and [`+run:by`](/hoon/stdlib/2i#runby) for a [`+map`](/hoon/stdlib/2o#map).
2. Reduce. The Reduce operation applies a function as a sequence of pairwise operations to each item, resulting in one summary value. The standard library arms that accomplish this are [`+roll`](/hoon/stdlib/2b#roll) and [`+reel`](/hoon/stdlib/2b#reel) for a list, [`+rep:in`](/hoon/stdlib/2h#repin) for a [`+set`](/hoon/stdlib/2o#set), and [`+rep:by`](/hoon/stdlib/2i#repby) for a [`+map`](/hoon/stdlib/2o#map).
3. Filter. The Filter operation applies a true/false function to each member of a collection, resulting in some number of items equal to or fewer than the size of the original set. In Hoon, the library arms that carry this out include [`+skim`](/hoon/stdlib/2b#skim), [`+skid`](/hoon/stdlib/2b#skid), [`+murn`](/hoon/stdlib/2b#murn) for a list, and [`+rib:by`](/hoon/stdlib/2i#ribby) for a [`+map`](/hoon/stdlib/2o#map).


# 17. Text Processing III

Guide to text parsing in Hoon, covering parser fundamentals, rule building, parsing techniques with scanning and RegEx-like operations, and building complex parsers.

*This module covers text parsing. It may be considered optional and skipped if you are speedrunning Hoon School.*

We need to build a tool to accept a tape containing some characters, then turn it into something else, something computational.

For instance, a calculator could accept an input like `3+4` and return `7`. A command-line interface may look for a program to evaluate (like Bash and `ls`). A search bar may apply logic to the query (like Google and `-` for `NOT`).

The basic problem all parsers face is this:

1. You need to accept a character string.
2. You need to ingest one or more characters and decide what they “mean”, including storing the result of this meaning.
3. You need to loop back to #1 again and again until you are out of characters.

## The Hoon Parser <a href="#the-hoon-parser" id="the-hoon-parser"></a>

We could build a simple parser out of a trap and [`+snag`](/hoon/stdlib/2b#snag), but it would be brittle and difficult to extend. The Hoon parser is very sophisticated, since it has to take a file of ASCII characters (and some UTF-8 strings) and turn it via an AST into Nock code. What makes parsing challenging is that we have to wade directly into a sea of new types and processes. To wit:

* A `$tape` is the string to be parsed.
* A `$hair` is the position in the text the parser is at, as a cell of line & column, `[p=@ud q=@ud]`.
* A `$nail` is parser input, a cell of `$hair` and `$tape`.
* An `$edge` is parser output, a pair of a `$hair` and a `+unit` containing a pair of the result and a `$nail`. (There are some subtleties around failure-to-parse here that we'll defer a moment.)
* A `$rule` is a parser, a gate which applies a `$nail` to yield an `$edge`.

Basically, one uses a `$rule` on `[hair tape]` to yield an `$edge`.

A substantial swath of the standard library is built around parsing for various scenarios, and there's a lot to know to effectively use these tools. **If you can parse arbitrary input using Hoon after this lesson, you're in fantastic shape for building things later.** It's worth spending extra effort to understand how these programs work.

There is a [full guide on parsing](/hoon/parsing) which goes into more detail than this quick overview.

## Scanning Through a `$tape` <a href="#scanning-through-a-tape" id="scanning-through-a-tape"></a>

[`+scan`](/hoon/stdlib/4g#scan) parses a `$tape` or crashes, simple enough. It will be our workhorse. All we really need to know in order to use it is how to build a `$rule`.

Here we will preview using [`+shim`](/hoon/stdlib/4f#shim) to match characters with in a given range, here lower-case. If you change the character range, e.g. putting `' '` in the `+shim` will span from ASCII `32`, `' '` to ASCII `122`, `'z'`.

```hoon
> `(list)`(scan "after" (star (shim 'a' 'z')))  
~[97 102 116 101 114]  

> `(list)`(scan "after the" (star (shim 'a' 'z')))
{1 6}  
syntax error  
dojo: hoon expression failed
```

## `$rule` Building <a href="#rule-building" id="rule-building"></a>

The `$rule`-building system is vast and often requires various components together to achieve the desired effect.

### `$rule`s to parse fixed strings <a href="#rules-to-parse-fixed-strings" id="rules-to-parse-fixed-strings"></a>

[`+just`](/hoon/stdlib/4f#just) takes in a single `$char` and produces a `$rule` that attempts to match that `$char` to the first character in the `$tape` of the input `$nail`.

```hoon
> ((just 'a') [[1 1] "abc"])
[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
```

[`+jest`](/hoon/stdlib/4f#jest) matches a `$cord`. It takes an input `$cord` and produces a `$rule` that attempts to match that `$cord` against the beginning of the input.

```hoon
> ((jest 'abc') [[1 1] "abc"])
[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]

> ((jest 'abc') [[1 1] "abcabc"])
[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q="abc"]]]]

> ((jest 'abc') [[1 1] "abcdef"])
[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q="def"]]]]
```

(Keep an eye on the structure of the return `$edge` there.)

[`+shim`](/hoon/stdlib/4f#shim) parses characters within a given range. It takes in two atoms and returns a `$rule`.

```hoon
> ((shim 'a' 'z') [[1 1] "abc"])
[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
```

[`+next`](/hoon/stdlib/4f#next) is a simple `$rule` that takes in the next character and returns it as the parsing result.

```hoon
> (next [[1 1] "abc"])
[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
```

### `$rule`s to parse flexible strings <a href="#rules-to-parse-flexible-strings" id="rules-to-parse-flexible-strings"></a>

So far we can only parse one character at a time, which isn't much better than just using [`+snag`](/hoon/stdlib/2b#snag) in a trap.

```hoon
> (scan "a" (shim 'a' 'z'))  
'a'  

> (scan "ab" (shim 'a' 'z'))  
{1 2}  
syntax error  
dojo: hoon expression failed
```

How do we parse multiple characters in order to break things up sensibly?

[`+star`](/hoon/stdlib/4f#star) will match a multi-character list of values.

```hoon
> (scan "a" (just 'a'))
'a'

> (scan "aaaaa" (just 'a'))
! {1 2}
! 'syntax-error'
! exit

> (scan "aaaaa" (star (just 'a')))
"aaaaa"
```

[`+plug`](/hoon/stdlib/4e#plug) takes the `$nail` in the `$edge` produced by one rule and passes it to the next `$rule`, forming a cell of the results as it proceeds.

```hoon
> (scan "starship" ;~(plug (jest 'star') (jest 'ship')))
['star' 'ship']
```

[`+pose`](/hoon/stdlib/4e#pose) tries each `$rule` you hand it successively until it finds one that works.

```hoon
> (scan "a" ;~(pose (just 'a') (just 'b')))
'a'

> (scan "b" ;~(pose (just 'a') (just 'b')))
'b'

> (;~(pose (just 'a') (just 'b')) [1 1] "ab")
[p=[p=1 q=2] q=[~ u=[p='a' q=[p=[p=1 q=2] q=[i='b' t=""]]]]]
```

[`+glue`](/hoon/stdlib/4e#glue) parses a delimiter (a `$rule`) in between each `$rule` and forms a cell of the results of each non-delimiter `$rule`. Delimiters representing each symbol used in Hoon are named according to their aural ASCII pronunciation. Sets of characters can also be used as delimiters, such as `+prn` for printable characters ([more here](/hoon/stdlib/4i)).

```hoon
> (scan "a b" ;~((glue ace) (just 'a') (just 'b')))  
['a' 'b']

> (scan "a,b" ;~((glue com) (just 'a') (just 'b')))
['a' 'b']

> (scan "a,b,a" ;~((glue com) (just 'a') (just 'b')))
{1 4}
syntax error

> (scan "a,b,a" ;~((glue com) (just 'a') (just 'b') (just 'a')))
['a' 'b' 'a']
```

The `;~` [micsig](/hoon/rune/mic#micsig) will create `;~(combinator (list rule))` to use multiple `$rule`s.

```hoon
> (scan "after the" ;~((glue ace) (star (shim 'a' 'z')) (star (shim 'a' 'z'))))  
[[i='a' t=<|f t e r|>] [i='t' t=<|h e|>]

> (;~(pose (just 'a') (just 'b')) [1 1] "ab")  
[p=[p=1 q=2] q=[~ u=[p='a' q=[p=[p=1 q=2] q=[i='b' t=""]]]]]
```

At this point we have two problems: we are just getting raw `@t` atoms back, and we can't iteratively process arbitrarily long strings. [`+cook`](/hoon/stdlib/4f#cook) will help us with the first of these:

[`+cook`](/hoon/stdlib/4f#cook) will take a `$rule` and a gate to apply to the successful parse.

```hoon
> ((cook ,@ud (just 'a')) [[1 1] "abc"])
[p=[p=1 q=2] q=[~ u=[p=97 q=[p=[p=1 q=2] q="bc"]]]]

> ((cook ,@tas (just 'a')) [[1 1] "abc"])
[p=[p=1 q=2] q=[~ u=[p=%a q=[p=[p=1 q=2] q="bc"]]]]

> ((cook |=(a=@ +(a)) (just 'a')) [[1 1] "abc"])
[p=[p=1 q=2] q=[~ u=[p=98 q=[p=[p=1 q=2] q="bc"]]]]

> ((cook |=(a=@ `@t`+(a)) (just 'a')) [[1 1] "abc"])
[p=[p=1 q=2] q=[~ u=[p='b' q=[p=[p=1 q=2] q="bc"]]]]
```

However, to parse iteratively, we need to use the [`+knee`](/hoon/stdlib/4f#knee) function, which takes a noun as the bunt of the type the `$rule` produces, and produces a `$rule` that recurses properly. (You'll probably want to treat this as a recipe for now and just copy it when necessary.)

```hoon
|-(;~(plug prn ;~(pose (knee *tape |.(^$)) (easy ~))))
```

There is an example of a calculator [in the parsing guide](/hoon/parsing#recursive-parsers) that's worth a read at this point. It uses [`+knee`](/hoon/stdlib/4f#knee) to scan in a set of numbers at a time.

### Example: Parse a String of Numbers <a href="#example-parse-a-string-of-numbers" id="example-parse-a-string-of-numbers"></a>

A simple [`+shim`](/hoon/stdlib/4f#shim)-based parser:

```hoon
> (scan "1234567890" (star (shim '0' '9')))  
[i='1' t=<|2 3 4 5 6 7 8 9 0|>]
```

A refined [`+cook`](/hoon/stdlib/4f#cook)/[`+cury`](/hoon/stdlib/2n#cury)/[`+jest`](/hoon/stdlib/4f#jest) parser:

```hoon
> ((cook (cury slaw %ud) (jest '1')) [[1 1] "123"])  
[p=[p=1 q=2] q=[~ u=[p=[~ 1] q=[p=[p=1 q=2] q="23"]]]]  

> ((cook (cury slaw %ud) (jest '12')) [[1 1] "123"])
[p=[p=1 q=3] q=[~ u=[p=[~ 12] q=[p=[p=1 q=3] q="3"]]]]
```

### Example: Hoon Workbook <a href="#example-hoon-workbook" id="example-hoon-workbook"></a>

More examples demonstrating parser usage are available in the [Hoon Workbook](/hoon/examples), such as the [Roman Numeral](/hoon/examples/roman) tutorial.


# 18. Generic and Variant Cores

Advanced core theory. Genericity with wet gates and cores, type polymorphism, dry vs wet core behavior, parametric polymorphism, and compile-time vs. runtime type-checking.

*This module introduces how cores can be extended for different behavioral patterns. It may be considered optional and skipped if you are speedrunning Hoon School.*

Cores can expose and operate with many different assumptions about their inputs and structure. "\[battery payload]" describes the top-level structure of a core, but within that we already know other requirements can be enforced, like "\[battery \[sample context]]" for a gate, or no sample for a trap. Cores can also expose and operate on their input values with different relationships. This lesson is concerned with examining [*genericity*](https://en.wikipedia.org/wiki/Generic_programming) including certain kinds of [parametric polymorphism](https://en.wikipedia.org/wiki/Parametric_polymorphism), which allows flexibility in type, and [*variance*](https://en.wikipedia.org/wiki/Covariance_and_contravariance_%28computer_science%29), which allows cores to use different sets of rules as they evaluate.

If cores never changed, we wouldn't need polymorphism. Of course, nouns are immutable and never change, but we use them as templates to construct new nouns around.

Suppose we take a core, a cell of \[battery payload], and replace payload with a different noun. Then, we invoke an arm from the battery.

Is this legal? Does it make sense? Every function call in Hoon does this, so we'd better make it work well.

The full core stores *both* payload types: the type that describes the payload currently in the core, and the type that the core was compiled with.

In the [Bertrand Meyer tradition of type theory](https://en.wikipedia.org/wiki/Object-Oriented_Software_Construction), there are two forms of polymorphism: *variance* and *genericity*. In Hoon this choice is per core: a core can be either `%wet` or `%dry`. Dry polymorphism relies on variance; wet polymorphism relies on genericity.

This lesson discusses both genericity and variance for core management. These two sections may be read separately or in either order, and all of this content is not a requirement for working extensively with Gall agents. If you're just starting off, wet gates (genericity) make the most sense to have in your toolkit now.

## Genericity <a href="#genericity" id="genericity"></a>

Polymorphism is a programming concept that allows a piece of code to use different types at different times. It's a common technique in most languages to make code that can be reused for many different situations, and Hoon is no exception.

### Dry Cores <a href="#dry-cores" id="dry-cores"></a>

A dry gate is the kind of gate that you're already familiar with: a one-armed core with a sample. A wet gate is also a one-armed core with a sample, but there is a difference in how types are handled. With a dry gate, when you pass in an argument and the code gets compiled, the type system will try to cast to the type specified by the gate; if you pass something that does not fit in the specified type, for example a `$cord` instead of a cell you will get a [`nest-fail`](/hoon/hoon-errors#nest-fail) error.

A core's payload can change from its original value. In fact, this happens in the typical function call: the default sample is replaced with an input value. How can we ensure that the core's arms are able to run correctly, that the payload type is still appropriate despite whatever changes it has undergone?

There is a type check for each arm of a dry core, intended to verify that the arm's parent core has a payload of the correct type.

When the `$` arm of a dry gate is evaluated it takes its parent core (the dry gate itself) as the subject, often with a modified sample value. But any change in sample type should be conservative; the modified sample value must be of the same type as the default sample value (or possibly a subtype). When the `$` arm is evaluated it should have a subject of a type it knows how to use.

### Wet Gates <a href="#wet-gates" id="wet-gates"></a>

When you pass arguments to a wet gate, their types are preserved and type analysis is done at the definition site of the gate rather than at the call site. In other words, for a wet gate, we ask: “Suppose this core was actually *compiled* using the modified payload instead of the one it was originally built with? Would the Nock formula we generated for the original template actually work for the modified payload?” Basically, wet gates allow you to hot-swap code at runtime and see if it “just works”; they defer the actual substitution in the sample. Wet gates are rather like [macros](https://en.wikipedia.org/wiki/Macro_%28computer_science%29) in this sense.

Consider a function like [`+turn`](/hoon/stdlib/2b#turn) which transforms each element of a list. To use `+turn`, we install a list and a transformation function in a generic core. The type of the list we produce depends on the type of the list and the type of the transformation function. But the Nock formulas for transforming each element of the list will work on any function and any list, so long as the function's argument is the list item.

A wet gate is defined by a `|*` [bartar](/hoon/rune/bar#bartar) rune rather than a `|=` [bartis](/hoon/rune/bar#bartis). More generally, cores that contain wet arms **must** be defined using `|@` [barpat](/hoon/rune/bar#barpat) instead of `|%` [barcen](/hoon/rune/bar#barcen) (`|*` expands to a `|@` core with `$` arm). There is also `|$` [barbuc](/hoon/rune/bar#barbuc) which defines the wet gate mold builder (remember, we like gates that build gates).

In a nutshell, compare these two gates:

```
> =dry |=([a=* b=*] [b a])

> =wet |*([a=* b=*] [b a])

> (dry %cat %dog)
[6.778.724 7.627.107]

> (wet %cat %dog)
[%dog %cat]
```

The dry gate does not preserve the type of `.a` and `.b`, but downcasts it to `*`; the wet gate does preserve the input types. It is good practice to include a cast in all gates, even wet gates. But in many cases the desired output type depends on the input type. How can we cast appropriately? Often we can cast by example, using the input values themselves (using `^+` [ketlus](/hoon/rune/ket#ketlus)).

Wet gates are therefore used when incoming type information is not well known and needs to be preserved. This includes parsing, building, and structuring arbitrary nouns. (If you are familiar with them, you can think of C++'s templates and operator overloading, and Haskell's typeclasses.) Wet gates are very powerful; they're enough rope to hang yourself with. Don't use them unless you have a specific reason to do so. (If you see "mull-\*" errors then something has gone wrong with using wet gates.)

### Exercise: The Trapezoid Rule <a href="#exercise-the-trapezoid-rule" id="exercise-the-trapezoid-rule"></a>

The [trapezoid rule](https://en.wikipedia.org/wiki/Trapezoidal_rule) solves a definite integral. It approximates the area under the curve by a trapezoid or (commonly) a series of trapezoids. The rule requires a function as one of the inputs, i.e. it applies *for a specific function*. We will use wet gates to accomplish this without stripping type information of the input gate core.

![](https://media.urbit.org/docs/hoon-school/Integration_num_trapezes_notation.png)

$$
\int\_a^b f(x) , dx \approx \sum\_{k=1}^N \frac{f(x\_{k-1}) + f(x\_k)}{2} \Delta x\_k = \tfrac{\Delta x}{2}\left(f(x\_0) + 2f(x\_1)+2f(x\_2)+ 2f(x\_3)+2f(x\_4)+\cdots+2f(x\_{N-1}) + f(x\_N)\right)
$$

Produce a trapezoid-rule integrator which accepts a wet gate (as a function of a single variable) and a list of *x* values, and yields the integral as a `@rs` floating-point value. (If you are not yet familiar with these, you may wish to skip ahead to the next lesson.)

```hoon
++  trapezint
  |*  [a=(list @rs) b=gate]
  =/  n  (lent a)
  =/  k  1
  =/  sum  .0
  |-  ^-  @rs
  ?:  =(+(k) n)  (add:rs sum (b (snag k a)))
  ?:  =(k 1)
    $(k +(k), sum (add:rs sum (b (snag k a))))
  $(k +(k), sum (mul:rs .2 (add:rs sum (b (snag k a)))))
```

The meat of this gate is concerned with correctly implementing the mathematical equation. In particular, wetness is required because `.b` can be *any* gate (although it should only be a gate with one argument, lest the whole thing fail with a "mull-grow" error). If you attempt to create the equivalent dry gate (`|=` [bartis](/hoon/rune/bar#bartis)), Hoon fails to build it with a [`nest-fail`](/hoon/hoon-errors#nest-fail) due to the loss of type information from the gate `.b`.

#### Tutorial: `+need`

Wet gates and wet cores are used in Hoon when type information isn't well-characterized ahead of time, as when constructing [`+map`s](/hoon/stdlib/2o#map) or [`+sets`](/hoon/stdlib/2o#set). For instance, almost all of the arms in [`+by`](/hoon/stdlib/2i#by) and [`+in`](/hoon/stdlib/2h#in), as well as most `+list` tools, are wet gates.

Let's take a look at a particular wet gate from the Hoon standard library, [`+need`](/hoon/stdlib/2a#need). `+need` works with a [unit](/hoon/stdlib/1c#unit) to produce the value of a successful `+unit` call, or crash on `~`. (As this code is already defined in your `/sys/hoon.hoon`, you do not need to define it in the Dojo to use it.)

```hoon
++  need                                                ::  demand
  |*  a=(unit)
  ?~  a  ~>(%mean.'need' !!)
  u.a
```

Line by line:

```hoon
|*  a=(unit)
```

This declares a wet gate which accepts a `+unit`.

```hoon
?~  a  ~>(%mean.'need' !!)
```

If `.a` is empty, `~`, then the `+unit` cannot be unwrapped. Crash with `!!` [zapzap](/hoon/rune/zap#zapzap), but use `~>` [siggar](/hoon/rune/sig#siggar) to hint to the runtime interpreter how to handle the crash.

```hoon
u.a
```

This returns the value in the `+unit` since we now know it exists.

`+need` is wet because we don't want to lose type information when we extract from the `+unit`.

### Parametric Polymorphism <a href="#parametric-polymorphism" id="parametric-polymorphism"></a>

We encountered `|$` [barbuc](/hoon/rune/bar#barbuc) above as a wet gate that is a mold builder rune which takes in a list of molds and produces a new mold. Here we take another look at this rune as an implementation of *parametric polymorphism* in Hoon.

For example, we have `+list`s, [`+tree`s](/hoon/stdlib/1c#tree), and [`+set`s](/hoon/stdlib/2o#set) in Hoon, which are each defined in `/sys/hoon.hoon` as wet gate mold builders. Take a moment to see for yourself. Each `++` arm is followed by `|$` and a list of labels for input types inside square brackets (e.g. `[item]`). After that subexpression comes another that defines a type that is parametrically polymorphic with respect to the input values. For example, here is the definition of `+list` from `/sys/hoon.hoon`:

```hoon
++  list
  |$  [item]
  ::    null-terminated list
  ::
  ::  mold generator: produces a mold of a null-terminated list of the
  ::  homogeneous type {a}.
  ::
  $@(~ [i=item t=(list item)])
```

The `|$` [barbuc](/hoon/rune/bar#barbuc) rune is especially useful for defining containers of various kinds. Indeed, `+list`s, `+tree`s, and `+set`s are all examples of containers that accept subtypes. You can have a `(list @)`, a `(list ^)`, a `(list *)`, a `(tree @)`, a `(tree ^)`, a `(tree *)`, etc. The same holds for `+set`.

One nice thing about containers defined by `|$` is that they nest in the expected way. Intuitively a `(list @)` should nest under `(list *)`, because `@` nests under `*`. And so it does:

```
> =a `(list @)`~[11 22 33]

> ^-((list *) a)
~[11 22 33]
```

Conversely, a `(list *)` should not nest under `(list @)`, because `*` does not nest under `@`:

```
> =b `(list *)`~[11 22 33]

> ^-((list @) b)
nest-fail
```

### Drying Out a Gate <a href="#drying-out-a-gate" id="drying-out-a-gate"></a>

Some functional tools like [`+cury`](/hoon/stdlib/2n#cury) don't work with wet gates. It is, however, possible to “dry out“ a wet gate using [`+bake`](/hoon/stdlib/2b#bake):

```
> ((curr reel add) `(list @)`[1 2 3 4 ~])
mull-grow
-find.i.a

> ((curr (bake reel ,[(list @) _add]) add) `(list @)`[1 2 3 4 ~])
10
```

Typically it's better to find another way to express your problem than to `+bake` a wet gate, however. As we said before, wet gates are powerful and for that reason not apt for every purpose.

## Variance <a href="#variance" id="variance"></a>

Dry polymorphism works by substituting cores. Typically, one core is used as the interface definition, then replaced with another core which does something useful.

For core *b* to nest within core *a*, the batteries of *a* and *b* must have the same tree shape, and the product of each *b* arm must nest within the product of the *a* arm. Wet arms (described above) are not compatible unless the Hoon expression is exactly the same. But for dry cores we also apply a payload test that depends on the rules of variance.

There are four kinds of cores: `%gold`, `%iron`, `%zinc`, and `%lead`. You are able to use core-variance rules to create programs which take other programs as arguments. Which particular rules depends on which kind of core your program needs to complete.

Before we embark on the following discussion, we want you to know that [variance](https://en.wikipedia.org/wiki/Covariance_and_contravariance_%28computer_science%29) is a bright-line idea, much like cores themselves, which once you “get” illuminates you further about Hoon-nature. For the most part, though, you don't need to worry about core variance much unless you are writing kernel code, since it impinges on how cores evaluate with other cores as inputs. Don't sweat it if it takes a while for core variance to click for you. (If you want to dig into resources, check out Meyer type theory. The rules should make sense if you think about them intuitively and don't get hung up on terminology.) You should read up on the [Liskov substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle) if you want to dive deeper. [Vadzim Vysotski](https://vadzimv.dev/2019/10/01/generic-programming-part-1-introduction.html) and [Jamie Kyle](https://medium.com/@thejameskyle/type-systems-covariance-contravariance-bivariance-and-invariance-explained-35f43d1110f8) explain the theory of type system variance accessibly, while [Eric Lippert](https://archive.ph/QmiqB) provides a more technical description. There are many wrinkles that particular languages, such as object-oriented programming languages, introduce which we can elide here.

Briefly, computer scientist Eric Lippert [clarifies](https://stackoverflow.com/questions/37467882/why-does-c-sharp-use-contravariance-not-covariance-in-input-parameters-with-de) that “variance is a fact about the preservation of an assignment compatibility relationship across a transformation of types.” What trips learners up about variance is that **variance rules apply to the input and output of a core, not directly to the core itself**. A core has a *variance property*, but that property doesn't manifest until cores are used together with each other.

Variance describes the four possible relationships that type rules are able to have to each other. Hoon imaginatively designates these by metals. Briefly:

1. **Covariance (`%zinc`)** means that specific types nest inside of generic types: it's like claiming that a core that produces a `%plant` can produce a `%tree`, a subcategory of `%plant`. Covariance is useful for flexibility in return values.
2. **Contravariance (`%iron`)** means that generic types are expected to nest inside of specific types: it's like claiming that a core that can accept a `%tree` can accept a `%plant`, the supercategory of `%tree`. (Contravariance seems counterintuitive for many developers when they encounter it for the first time.) Contravariance is useful for flexibility in input values (samples).
3. **Bivariance (`%lead`)** means that we can allow both covariant and contravariant behavior. While bivariance is included for completeness (including a worked example below), it is not commonly used and only a few examples exist in the standard library for building shared data structure cores.
4. **Invariance (`%gold`)** means that types must mutually nest compatibly: a core that accepts or produces a `%tree` can only accept or produce a `%tree`. This is the default behavior of cores, so it's the strongest model you have imprinted on. Cores which allow variance are changing that behavior.

A `%gold` core can be cast or converted to any metal, and any metal can be cast or converted to `%lead`.

### `%zinc` Covariance <a href="#zinc-covariance" id="zinc-covariance"></a>

Covariance means that specific types nest inside of generic types: `%tree` nests inside of `%plant`. Covariant data types are sources, or read-only values.

A zinc core *z* has a read-only sample (payload head, `+6.z`) and an opaque context (payload tail, `+7.z`). ("Opaque" here means that the faces and arms are not exported into the namespace, and that the values of faces and arms can't be written to. The object in question can be replaced by something else without breaking type safety.) A core *y* which nests within it must be a gold or zinc core, such that `+6.y` nests within `+6.z`. Hence, **covariant**.

You can read from the sample of a `%zinc` core, but not change it:

```
> =mycore ^&(|=(a=@ 1))

> a.mycore
0

> mycore(a 22)
-tack.a
-find.a
ford: %slim failed:
ford: %ride failed to compute type:
```

Informally, a function fits an interface if the function has a more specific result and/or a less specific argument than the interface.

The `^&` [ketpam](/hoon/rune/ket#ketpam) rune converts a core to a `%zinc` covariant core.

### `%iron` Contravariance <a href="#iron-contravariance" id="iron-contravariance"></a>

Contravariance means that generic types nest inside of specific types. Contravariant data types are sinks, or write-only values.

An `%iron` core *i* has a write-only sample (payload head, `+6.i`) and an opaque context (payload tail, `+7.i`). A core *j* which nests within it must be a `%gold` or `%iron` core, such that `+6.i` nests within `+6.j`. Hence, **contravariant**.

If type *x* nests within type *xx*, and type *y* nests within type *yy*, then a core accepting a *yy* and producing an *x* nests within an iron core accepting a *y* and producing an *xx*.

Informally, a function fits an interface if the function has a more specific result and/or a less specific argument than the interface.

For instance, the archetypal Gall agents in `/sys/lull.hoon` are composed using iron gates since they will be used as examples for building actual agent cores. The [`+rs`](/hoon/stdlib/3b#rs) and sister gates in `/sys/hoon.hoon` are built using iron doors with specified rounding behavior so when you actually use the core (like [+add:rs](/hoon/stdlib/3b#addrs)) the core you are using has been built as an example.

The `|~` [barsig](/hoon/rune/bar#barsig) rune produces an iron gate. The `^|` [ketbar](/hoon/rune/ket#ketbar) rune converts a `%gold` invariant core to an iron core.

### `%lead` Bivariance <a href="#lead-bivariance" id="lead-bivariance"></a>

Bivariance means that both covariance and contravariance apply. Bivariant data types have an opaque payload that can neither be read or written to.

A lead core *l* has an opaque payload which can be neither read nor written to. There is no constraint on the payload of a core *m* which nests within it. Hence, **bivariant**.

If type *x* nests within type *xx*, a lead core producing an *x* nests within a lead core producing an *xx*.

Bivariant data types are neither readable nor writeable, but have no constraints on nesting. These are commonly used for `/mar` marks and `/sur` structure files. They are useful as examples which produce types.

Informally, a more specific generator can be used as a less specific generator.

For instance, several archetypal cores in `/sys/lull.hoon` which define operational data structures for Arvo are composed using lead gates.

The `|?` [barwut](/hoon/rune/bar#barwut) rune produces a lead trap. The `^?` [ketwut](/hoon/rune/ket#ketwut) rune converts any core to a `%lead` bivariant core.

### `%gold` Invariance <a href="#gold-invariance" id="gold-invariance"></a>

Invariance means that type nesting is disallowed. Invariant data types have a read-write payload.

A `%gold` core *g* has a read-write payload; another core *h* that nests within it (i.e., can be substituted for it) must be a `%gold` core whose payload is mutually compatible (`+3.g` nests in `+3.h`, `+3.h` nests in `+3.g`). Hence, **invariant**.

By default, cores are `%gold` invariant cores.

### Illustrations <a href="#illustrations" id="illustrations"></a>

#### Tutorial: `%gold` Invariant Polymorphism

Usually it makes sense to cast for a `%gold` core type when you're treating a core as a state machine. The check ensures that the payload, which includes the relevant state, doesn't vary in type.

Let's look at simpler examples here, using the `^+` [ketlus](/hoon/rune/ket#ketlus) rune:

```
> ^+(|=(^ 15) |=(^ 16))
< 1.jcu
  [ [* *]
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
>

> ^+(|=(^ 15) |=([@ @] 16))
mint-nice
-need.@
-have.*
nest-fail

> ^+(|=(^ 15) |=(* 16))
mint-nice
-need.[* *]
-have.*
nest-fail
```

The first cast goes through because the right-hand gold core has the same sample type as the left-hand gold core. The sample types mutually nest. The second cast fails because the right-hand sample type is more specific than the left-hand sample type. (Not all cells, `^`, are pairs of atoms, `[@ @]`.) And the third cast fails because the right-hand sample type is broader than the left-hand sample type. (Not all nouns, `*`, are cells, `^`.)

Two more examples:

```
> ^+(=>([1 2] |=(@ 15)) =>([123 456] |=(@ 16)))
<1.xqz [@ @ud @ud]>

> ^+(=>([1 2] |=(@ 15)) =>([123 456 789] |=(@ 16)))
nest-fail
```

In these examples, the `=>` rune is used to give each core a simple context. The context of the left-hand core in each case is a pair of atoms, `[@ @]`. The first cast goes through because the right-hand core also has a pair of atoms as its context. The second cast fails because the right-hand core has the wrong type of context: three atoms, `[@ @ @]`.

#### Tutorial: `%iron` Contravariant Polymorphism

`%iron` gates are particularly useful when you want to pass gates (having various payload types) to other gates. We can illustrate this use with a very simple example. Save the following as `/gen/gatepass.hoon` in your `%base` desk:

```hoon
|=  a=_^|(|=(@ 15))
^-  @
=/  b=@  (a 10)
(add b 20)
```

This generator is rather simple except for the first line. The sample is defined as an `%iron` gate and gives it the face `a`. The function as a whole is for taking some gate as input, calling it by passing it the value `10`, adding `20` to it, and returning the result. Let's try it out in the Dojo:

```
> +gatepass |=(a=@ +(a))
31

> +gatepass |=(a=@ (add 3 a))
33

> +gatepass |=(a=@ (mul 3 a))
50
```

But we still haven't fully explained the first line of the code. What does `_^|(|=(@ 15))` mean? The inside portion is clear enough: `|=(@ 15)` produces a normal (i.e., `%gold`) gate that takes an atom and returns `15`. The `^|` [ketbar](/hoon/rune/ket#ketbar) rune is used to turn `%gold` gates to `%iron`. (Reverse alchemy!) And the `_` character turns that `%iron` gate value into a structure, i.e. a type. So the whole subexpression means, roughly: “the same type as an iron gate whose sample is an atom, `@`, and whose product is another atom, `@`”. The context isn't checked at all. This is good, because that allows us to accept gates defined and produced in drastically different environments. Let's try passing a gate with a different context:

```
> +gatepass =>([22 33] |=(a=@ +(a)))
31
```

It still works. You can't do that with a gold core sample!

There's a simpler way to define an iron sample. Revise the first line of `/gen/gatepass.hoon` to the following:

```hoon
|=  a=$-(@ @)
^-  @
=/  b=@  (a 10)
(add b 20)
```

If you test it, you'll find that the generator behaves the same as it did before the edits. The `$-` [buchep](/hoon/rune/buc#buchep) rune is used to create an `%iron` gate structure, i.e., an `%iron` gate type. The first expression defines the desired sample type, and the second subexpression defines the gate's desired output type.

The sample type of an `%iron` gate is contravariant. This means that, when doing a cast with some `%iron` gate, the desired gate must have either the same sample type or a superset.

Why is this a useful nesting rule for passing gates? Let's say you're writing a function *F* that takes as input some gate *G*. Let's also say you want *G* to be able to take as input any **mammal**. The code of *F* is going to pass arbitrary **mammals** to *G*, so that *G* needs to know how to handle all **mammals** correctly. You can't pass *F* a gate that only takes **dogs** as input, because *F* might call it with a **cat**. But *F* can accept a gate that takes all **animals** as input, because a gate that can handle any **animal** can handle **any mammal**.

`%iron` cores are designed precisely with this purpose in mind. The reason that the sample is write-only is that we want to be able to assume, within function *F*, that the sample of *G* is a **mammal**. But that might not be true when *G* is first passed into *F*; the default value of *G* could be another **animal**, say, a **lizard**. So we restrict looking into the sample of *G* by making the sample write-only. The illusion is maintained and type safety secured.

Let's illustrate `%iron` core nesting properties:

```
> ^+(^|(|=(^ 15)) |=(^ 16))
< 1|jcu
  [ [* *]
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
>

> ^+(^|(|=(^ 15)) |=([@ @] 16))
mint-nice
-need.@
-have.*
nest-fail

> ^+(^|(|=(^ 15)) |=(* 16))
< 1|jcu
  [ [* *]
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
>
```

(As before, we use the `^|` [ketbar](/hoon/rune/ket#ketbar) rune to turn `%gold` gates to `%iron`.)

The first cast goes through because the two gates have the same sample type. The second cast fails because the right-hand gate has a more specific sample type than the left-hand gate does. If you're casting for a gate that accepts any cell, `^`, it's because we want to be able to pass any cell to it. A gate that is only designed for pairs of atoms, `[@ @]`, can't handle all such cases, naturally. The third cast goes through because the right-hand gate sample type is broader than the left-hand gate sample type. A gate that can take any noun as its sample, `*`, works just fine if we choose only to pass it cells, `^`.

We mentioned previously that an `%iron` core has a write-only sample and an opaque context. Let's prove it.

Let's define a trivial gate with a context of `[g=22 h=44 .]`, convert it to `%iron` with `^|`, and bind it to `.iron-gate` in the dojo:

```
> =iron-gate ^|  =>([g=22 h=44 .] |=(a=@ (add a g)))

> (iron-gate 10)
32

> (iron-gate 11)
33
```

Not a complicated function, but it serves our purposes. Normally (i.e., with `%gold` cores) we can look at a context value *p* of some gate *q* with a wing expression: `p.q`. Not so with the iron gate:

```
> g.iron-gate
-find.g.iron-gate
```

And usually we can look at the sample value using the face given in the gate definition. Not in this case:

```
> a.iron-gate
-find.a.iron-gate
```

If you really want to look at the sample you can check `+6` of `.iron-gate`:

```
> +6.iron-gate
0
```

… and if you really want to look at the head of the context (i.e., where *g* is located, `+14`) you can:

```
> +14.iron-gate
22
```

… but in both cases all the relevant type information has been thrown away:

```
> -:!>(+6.iron-gate)
#t/*

> -:!>(+14.iron-gate)
#t/*
```

#### Tutorial: `%zinc` Covariant Polymorphism

As with `%iron` cores, the context of `%zinc` cores is opaque; they cannot be written-to or read-from. The sample of a `%zinc` core is read-only. That means, among other things, that `%zinc` cores cannot be used for function calls. Function calls in Hoon involve a change to the sample (the default sample is replaced with the argument value), which is disallowed as type-unsafe for `%zinc` cores.

We can illustrate the casting properties of `%zinc` cores with a few examples. The `^&` [ketpam](/hoon/rune/ket#ketpam) rune is used to convert `%gold` cores to `%zinc`:

```
> ^+(^&(|=(^ 15)) |=(^ 16))
< 1&jcu
  [ [* *]
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
>

> ^+(^&(|=(^ 15)) |=([@ @] 16))
< 1&jcu
  [ [* *]
    [our=@p now=@da eny=@uvJ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
>

> ^+(^&(|=(^ 15)) |=(* 16))
mint-nice
-need.[* *]
-have.*
nest-fail
```

The first two casts succeed because the right-hand core sample type is either the same or a subset of the left-hand core sample type. The last one fails because the right-hand sample type is a superset.

Even though you can't function call a `%zinc` core, the arms of a `%zinc` core can be computed and the sample can be read. Let's test this with a `%zinc` gate of our own:

```
> =zinc-gate ^&  |=(a=_22 (add 10 a))

> (zinc-gate 12)
payload-block

> a.zinc-gate
22

> $.zinc-gate
32
```

#### Tutorial: `%lead` Bivariant Polymorphism

`%lead` cores have more permissive nesting rules than either `%iron` or `%zinc` cores. There is no restriction on which payload types nest. That means, among other things, that the payload type of a `%lead` core is both covariant and contravariant ( ‘bivariant’).

In order to preserve type safety when working with `%lead` cores, a severe restriction is needed. The whole payload of a `%lead` core is opaque; the payload can neither be written-to or read-from. For this reason, as was the case with `%zinc` cores, `%lead` cores cannot be called as functions.

The arms of a `%lead` core can still be evaluated, however. We can use the `^?` rune to convert a `%gold`, `%iron`, or `%zinc` core to lead:

```
> =lead-gate ^?  |=(a=_22 (add 10 a))

> $.lead-gate
32
```

But don't try to read the sample:

```
> a.lead-gate
-find.a.lead-gate
```

#### Tutorial: `%lead` Bivariant Polymorphism

Calculate the Fibonacci series using `%lead` and `%iron` cores.

This program produces a list populated by the first ten elements of the `+fib` arm. It consists of five arms; in brief:

* `+fib` is a trap (core with no sample and default arm `$` buc).
* `+stream` is a mold builder that produces a trap, a function with no argument. This trap can yield a value or a `~`.
* `+stream-type` is a wet gate that produces the type of items stored in `+stream`.
* `+to-list` is a wet gate that converts a `+stream` to a list.
* `+take` is a wet gate that takes a `+stream` and an atom and yields a modified subject (!) and another trap of `+stream`'s type.

<details>

<summary>/gen/fib.hoon</summary>

```hoon
=<  (to-list (take fib 10))
|%
++  stream
  |*  of=mold
  $_  ^?   |.
  ^-  $@(~ [item=of more=^$])
  ~
++  stream-type
  |*  s=(stream)
  $_  =>  (s)
  ?~  .  !!
  item
++  to-list
  |*  s=(stream)
  %-  flop
  =|  r=(list (stream-type s))
  |-  ^+  r
  =+  (s)
  ?~  -  r
  %=  $
    r  [item r]
    s  more
  ==
++  take
  |*  [s=(stream) n=@]
  =|  i=@
  ^+  s
  |.
  ?:  =(i n)  ~
  =+  (s)
  ?~  -  ~
  :-  item
  %=  ..$
    i  +(i)
    s  more
  ==
++  fib
  ^-  (stream @ud)
  =+  [p=0 q=1]
  |.  :-  q
  %=  .
    p  q
    q  (add p q)
  ==
--
```

</details>

Let's examine each arm in detail.

**`+stream`**

```hoon
++  stream
  |*  of=mold
  $_  ^?  |.
  ^-  $@(~ [item=of more=^$])
  ~
```

`+stream` is a mold-builder. It's a wet gate that takes one argument, `.of`, which is a mold, and produces a `%lead` trap: a function with no sample and an arm `$` buc, with opaque payload.

`$_` [buccab](/hoon/rune/buc#buccab) is a rune that produces a type from an example; `^?` [ketwut](/hoon/rune/ket#ketwut) converts (casts) a core to lead; `|.` [bardot](/hoon/rune/bar#bardot) forms the trap. So to follow this sequence we read it backwards: we create a trap, convert it to a lead trap (making its payload inaccessible), and then use that lead trap as an example from which to produce a type.

With the line `^- $@(~ [item=of more=^$])`, the output of the trap will be cast into a new type. `$@` [bucpat](/hoon/rune/buc#bucpat) is the rune to describe a data structure that can either be an atom or a cell. The first part describes the atom, which here is going to be `~`. The second part describes a cell, which we define to have the head of type `$of` with the face `item`, and a tail with a face of `more`. The expression `^$` is not a rune (no children), but rather a reference to the enclosing wet gate, so the tail of this cell will be of the same type produced by this wet gate.

The final `~` here is used as the type produced when initially calling this wet gate. This is valid because it nests within the type we defined on the previous line.

Now you can see that a `+stream` is either `~` or a pair of a value of some type and a `+stream`. This type represents an infinite series.

**`+stream-type`**

```hoon
++  stream-type
  |*  s=(stream)
  $_  =>  (s)
  ?~  .  !!
  item
```

`+stream-type` is a wet gate that produces the type of items stored in the `+stream` arm. The `(stream)` syntax is a shortcut for `(stream *)`; a stream of some type.

Calling a `+stream`, which is a trap, will either produce `.item` and `.more` or it will produce `~`. If it does produce `~`, the `+stream` is empty and we can't find what type it is, so we simply crash with `!!` [zapzap](/hoon/rune/zap#zapzap).

**`+take`**

```hoon
++  take
  |*  [s=(stream) n=@]
  =|  i=@
  ^+  s
  |.
  ?:  =(i n)  ~
  =+  (s)
  ?~  -  ~
  :-  item
  %=  ..$
    i  +(i)
    s  more
  ==
```

The `+take` arm is another wet gate. This time it takes a `+stream` `.s` and an atom `.n`. We add an atom to the subject and then make sure that the trap we are creating is going to be of the same type as `.s`, the `+stream` we passed in.

If `.i` and `.n` are equal, the trap will produce `~`. If not, `.s` is called and has its result put on the front of the subject. If its value is `~`, then the trap again produces `~`. Otherwise the trap produces a cell of `.item`, the first part of the value of `.s`, and a new trap that increments `.i`, and sets `.s` to be the `.more` trap which produces the next value of the `+stream`. The result here is a `+stream` that will only ever produce `.n` items, even if the stream otherwise would have been infinite.

**`+to-list`**

```hoon
++  to-list
  |*  s=(stream)
  %-  flop
  =|  r=(list (stream-type s))
  |-  ^+  r
  =+  (s)
  ?~  -  r
  %=  $
    r  [item r]
    s  more
  ==
```

The `+to-list` arm is a wet gate that takes `.s`, a `+stream`, only here it will, as you may expect, produce a list. The rest of this wet gate is straightforward but we can examine it quickly anyway. As is the proper style, this list that is produced will be reversed, so [`+flop`](/hoon/stdlib/2b#flop) is used to put it in the order it is in the stream. Recall that adding to the front of a list is cheap, while adding to the back is expensive.

`.r` is added to the subject as an empty list of whatever type is produced by `.s`. A new trap is formed and called, and it will produce the same type as `.r`. Then `.s` is called and has its value added to the subject. If the result is `~`, the trap produces `.r`. Otherwise, we want to call the trap again, adding `.item` to the front of `.r` and changing `.s` to `.more`. Now the utility of `+take` should be clear. We don't want to feed `+to-list` an infinite stream as it would never terminate.

**`+fib`**

```hoon
++  fib
  ^-  (stream @ud)
  =+  [p=0 q=1]
  |.  :-  q
  %=  .
    p  q
    q  (add p q)
  ==
```

The final arm in our core is `+fib`, which is a `+stream` of `@ud` and therefore is a `%lead` core. Its subject contains `.p` and `.q`, which will not be accessible outside of this trap, but because of the `%=` [centis](/hoon/rune/cen#centis) will be retained in their modified form in the product trap. The product of the trap is a pair (`:-` [colhep](/hoon/rune/col#colhep)) of a `@ud` and the trap that will produce the next `@ud` in the Fibonacci series.

```hoon
=<  (to-list (take fib 10))
```

Finally, the first line of our program will take the first 10 elements of `.fib` and produce them as a list.

```unknown
~[1 1 2 3 5 8 13 21 34 55]
```

This example is a bit overkill for simply calculating the Fibonacci series, but it illustrates how you could use `%lead` cores. Instead of `+fib`, you can supply any infinite sequence and `+stream` will correctly handle it.

### Exercise: `%lead` Bivariant Polymorphism <a href="#exercise-lead-bivariant-polymorphism" id="exercise-lead-bivariant-polymorphism"></a>

Produce a `%say` generator that yields another self-referential sequence, like the [Lucas numbers](https://en.wikipedia.org/wiki/Lucas_number) or the [Thue–Morse sequence](https://en.wikipedia.org/wiki/Thue%E2%80%93Morse_sequence).


# 19. Mathematics

Guide to mathematical operations in Hoon beyond unsigned integers, covering floating-point arithmetic, signed integers, modular arithmetic, bitwise operations, and IEEE 754.

*This module introduces how non-`@ud` mathematics are instrumented in Hoon. It may be considered optional and skipped if you are speedrunning Hoon School.*

All of the math we've done until this point relied on unsigned integers: there was no negative value possible, and there were no numbers with a fractional part. How can we work with mathematics that require more than just bare unsigned integers?

`@u` unsigned integers (whether `@ud` decimal, `@ux` hexadecimal, etc.) simply count upwards by binary place value from zero. However, if we apply a different interpretive rule to the resulting value, we can treat the integer (in memory) *as if* it corresponded to a different real value, such as a [negative number](https://en.wikipedia.org/wiki/Integer) or a [number with a fractional part](https://en.wikipedia.org/wiki/Rational_number). Auras make this straightforward to explore:

```
> `@ud`1.000.000
1.000.000

> `@ux`1.000.000
0xf.4240

> `@ub`1.000.000
0b1111.0100.0010.0100.0000

> `@sd`1.000.000
--500.000

> `@rs`1.000.000
.1.401298e-39

> `@rh`1.000.000
.~~3.125

> `@t`1.000.000
'@B\0f'
```

How can we actually treat other modes of interpreting numbers as mathematical quantities correctly? That's the subject of this lesson.

(Ultimately, we are using a concept called [Gödel numbering](https://en.wikipedia.org/wiki/G%C3%B6del_numbering) to justify mapping some data to a particular representation as a unique integer.)

## Floating-Point Mathematics <a href="#floating-point-mathematics" id="floating-point-mathematics"></a>

A number with a fractional part is called a “floating-point number” in computer science. This derives from its solution to the problem of representing the part less than one.

Consider for a moment how you would represent a regular decimal fraction if you only had integers available. You would probably adopt one of three strategies:

1. [**Rational numbers**](https://en.wikipedia.org/wiki/Fraction). Track whole-number ratios like fractions. Thus $$1.25 = \frac{5}{4}$$, thence the pair `(5, 4)`. Two numbers have to be tracked: the numerator and the denominator.
2. [**Fixed-point**](https://en.wikipedia.org/wiki/Fixed-point_arithmetic). Track the value in smaller fixed units (such as thousandths). By defining the base unit to be $$\frac{1}{1000}$$ , $$1.25$$ may be written $$1250$$. One number needs to be tracked: the value in terms of the scale. (This is equivalent to rational numbers with only a fixed denominator allowed.)
3. [**Floating-point**](https://en.wikipedia.org/wiki/Floating-point_arithmetic). Track the value at adjustable scale. In this case, one needs to represent $$1.25$$ as something like $$125 \times 10^{-2}$$. Two numbers have to be tracked: the significand ($$125$$) and the exponent ($$-2$$).

Most systems use floating-point mathematics to solve this problem. For instance, single-precision floating-point mathematics designate one bit for the sign, eight bits for the exponent (which has 127 subtracted from it), and twenty-three bits for the significand.

![](https://media.urbit.org/docs/hoon-school/Float_example.png)

This number, `0b11.1110.0010.0000.0000.0000.0000.0000`, is converted to decimal as $$(-1)^0 \times 2^{(124 - 127)} \times 1.25 = 2^{-3} \times 1.25 = 0.15625$$.

(If you want to explore the bitwise representation of values, [this tool](https://evanw.github.io/float-toy/) allows you to tweak values directly and see the results.)

### Hoon Operations <a href="#hoon-operations" id="hoon-operations"></a>

Hoon utilizes the [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) implementation of floating-point math for four bitwidth representations.

| Aura  | Meaning                                 | Example   |
| ----- | --------------------------------------- | --------- |
| `@r`  | Floating-point value                    |           |
| `@rh` | Half-precision 16-bit mathematics       | `.~~4.5`  |
| `@rs` | Single-precision 32-bit mathematics     | `.4.5`    |
| `@rd` | Double-precision 64-bit mathematics     | `.~4.5`   |
| `@rq` | Quadruple-precision 128-bit mathematics | `.~~~4.5` |

There are also a few molds which can represent the separate values of the FP representation. These are used internally but mostly don't appear in userspace code.

As the arms for the four `@r` auras are identical within their appropriate core, we will use [`@rs` single-precision floating-point mathematics](/hoon/stdlib/3b#rs) to demonstrate all operations.

#### Conversion to and from other auras

Any `@ud` unsigned decimal integer can be directly cast as a `@rs`.

```
> `@ud`.1
1.065.353.216
```

However, as you can see here, the conversion is not “correct” for the perceived values. Examining the `@ux` hexadecimal and `@ub` binary representation shows why:

```
> `@ux`.1
0x3f80.0000

> `@ub`.1
0b11.1111.1000.0000.0000.0000.0000.0000
```

If you refer back to the 32-bit floating-point example above, you'll see why: to represent one exactly, we have to use $$1.0 = (-1)^0 \times 2^{{127 - 127}} \times 1$$ and thus `0b11.1111.1000.0000.0000.0000.0000.0000`.

So to carry out this conversion from `@ud` to `@rs` correctly, we should use the [`+sun:rs`](/hoon/stdlib/3b#sunrs) arm.

```
> (sun:rs 1)
.1
```

To go the other way requires us to use an algorithm for converting an arbitrary number with a fractional part back into `@ud` unsigned integers. The [`+fl`](/hoon/stdlib/3b#fl) named tuple representation serves this purpose, and uses the [Dragon4 algorithm](https://dl.acm.org/doi/10.1145/93548.93559) to accomplish the conversion:

```
> (drg:rs .1)
[%d s=%.y e=--0 a=1]

> (drg:rs .3.1415926535)
[%d s=%.y e=-7 a=31.415.927]

> (drg:rs .1000)
[%d s=%.y e=--3 a=1]
```

It's up to you to decide how to handle this result, however! Perhaps a better option for many cases is to round the answer to a `@s` integer with [`+toi:rs`](/hoon/stdlib/3b#toirs):

```
> (toi:rs .3.1415926535)
[~ --3]
```

(`@s` signed integer math is discussed below.)

### Floating-point specific operations <a href="#floating-point-specific-operations" id="floating-point-specific-operations"></a>

As with aura conversion, the standard mathematical operators don't work for `@rs`:

```
> (add .1 1)
1.065.353.217

> `@rs`(add .1 1)
.1.0000001
```

The [`+rs`](/hoon/stdlib/3b#rs) core defines a set of `@rs`-affiliated operations which should be used instead:

```
> (add:rs .1 .1)
.2
```

This includes:

* [`+add:rs`](/hoon/stdlib/3b#addrs), addition.
* [`+sub:rs`](/hoon/stdlib/3b#subrs), subtraction.
* [`+mul:rs`](/hoon/stdlib/3b#mulrs), multiplication.
* [`+div:rs`](/hoon/stdlib/3b#divrs), division.
* [`+gth:rs`](/hoon/stdlib/3b#gthrs), greater than.
* [`+gte:rs`](/hoon/stdlib/3b#gters), greater than or equal to.
* [`+lth:rs`](/hoon/stdlib/3b#lthrs), less than.
* [`+lte:rs`](/hoon/stdlib/3b#lters), less than or equal to.
* [`+equ:rs`](/hoon/stdlib/3b#equrs), check equality (but not nearness!).
* [`+sqt:rs`](/hoon/stdlib/3b#sqtrs), square root.

### Exercise: `+is-close` <a href="#exercise-is-close" id="exercise-is-close"></a>

The [`+equ:rs`](/hoon/stdlib/3b#equrs) arm checks for complete equality of two values. The downside of this arm is that it doesn't find very close values:

```
> (equ:rs .1 .1)
%.y

> (equ:rs .1 .0.9999999)
%.n
```

Produce an arm which check for two values to be close to each other by an absolute amount. It should accept three values: `a`, `b`, and `atol`. It should return the result of the following comparison:

$$
|a-b| \leq \texttt{atol}
$$

#### Tutorial: Length Converter

Write a generator to take a `@tas` input measurement unit of length, a `@rs` value, and a `@tas` output unit to which we will convert the input measurement. For instance, this generator could convert a number of imperial feet to metric decameters.

**`/gen/convert-length.hoon`**

```hoon
|=  [fr-meas=@tas num=@rs to-meas=@tas]
=<
^-  @rs
?.  (check fr-meas to-meas)
  ~|("Invalid Measures" !!)
(output (meters fr-meas num) to-meas)
::
|%
+$  allowed  ?(%inch %foot %yard %furlong %chain %link %rod %fathom %shackle %cable %nautical-mile %hand %span %cubit %ell %bolt %league %megalithic-yard %smoot %barleycorn %poppy-seed %atto %femto %pico %nano %micro %milli %centi %deci %meter %deca %hecto %kilo %mega %giga %tera %peta %exa)
::
++  check
  |=  [fr-meas=@tas to-meas=@tas]
  &(?=(allowed fr-meas) ?=(allowed to-meas))
::
++  meters
  |=  [in=@tas value=@rs]
  =/  factor-one
    (~(got by convert-to-map) in)
  (mul:rs value factor-one)
::
++  output
  |=  [in=@rs out=@tas]
  ?:  =(out %meter)
    in
  (div:rs in (~(got by convert-to-map) out))
::
++  convert-to-map
  ^-  (map @tas @rs)
  %-  malt
  ^-  (list [@tas @rs])
  :~  :-  %atto             .1e-18
      :-  %femto            .1e-15
      :-  %pico             .1e-12
      :-  %nano             .1e-8
      :-  %micro            .1e-6
      :-  %milli            .1e-3
      :-  %poppy-seed       .2.212e-2
      :-  %barleycorn       .8.47e-2
      :-  %centi            .1e-2
      :-  %inch             .2.54e-2
      :-  %deci             .1e-1
      :-  %hand             .1.016e-1
      :-  %link             .2.012e-1
      :-  %span             .2.228e-1
      :-  %foot             .3.048e-1
      :-  %cubit            .4.472e-1
      :-  %megalithic-yard  .8.291e-1
      :-  %yard             .9.145e-1
      :-  %ell              .1.143
      :-  %smoot            .1.7
      :-  %fathom           .1.83
      :-  %rod              .5.03
      :-  %deca             .1e1
      :-  %chain            .2.012e1
      :-  %shackle          .2.743e1
      :-  %bolt             .3.658e1
      :-  %hecto            .1e2
      :-  %cable            .1.8532e2
      :-  %furlong          .2.0117e2
      :-  %kilo             .1e3
      :-  %mile             .1.609e3
      :-  %nautical-mile    .1.850e3
      :-  %league           .4.830e3
      :-  %mega             .1e6
      :-  %giga             .1e8
      :-  %tera             .1e12
      :-  %peta             .1e15
      :-  %exa              .1e18
      :-  %meter            .1
    ==
  --
```

This program shows several interesting aspects, which we've covered before but highlight here:

* Meters form the standard unit of length.
* `~|` [sigbar](/hoon/rune/sig#sigbar) produces an error message in case of a bad input.
* `+$` [lusbuc](/hoon/rune/lus#lusbuc) is a type constructor arm, here for a type union over units of length.

### Exercise: Measurement Converter <a href="#exercise-measurement-converter" id="exercise-measurement-converter"></a>

1. Add to this generator the ability to convert some other measurement (volume, mass, force, or another of your choosing).
2. Add an argument to the cell required by the gate that indicates whether the measurements are distance or your new measurement.
3. Enforce strictly that the `fr-meas` and `to-meas` values are either lengths or your new type.
4. Create a new map of conversion values to handle your new measurement conversion method.
5. Convert the functionality into a library.

### `+rs` as a Door <a href="#rs-as-a-door" id="rs-as-a-door"></a>

What is `+rs`? It's a door with 21 arms:

```
> rs
<21|ezj [r=?(%d %n %u %z) <51.njr 139.oyl 33.uof 1.pnw %138>]>
```

The battery of this core, pretty-printed as `21|ezj`, has 21 arms that define functions specifically for `@rs` atoms. One of these arms is named `+add`; it's a different `+add` from the standard one we've been using for vanilla atoms, and thus the one we used above. When you invoke [`+add:rs`](/hoon/stdlib/3b#addrs) instead of just `+add` in a function call, (1) the `.rs` door is produced, and then (2) the name search for `+add` resolves to the special `+add` arm in `.rs`. This produces the gate for adding `@rs` atoms:

```
> add:rs
<1.uka [[a=@rs b=@rs] <21.ezj [r=?(%d %n %u %z) <51.njr 139.oyl 33.uof 1.pnw %138>]>]>
```

What about the sample of the `.rs` door? The pretty-printer shows `r=?(%d %n %u %z)`. The [`+rs`](/hoon/stdlib/3b#rs) sample can take one of four values: `%d`, `%n`, `%u`, and `%z`. These argument values represent four options for how to round `@rs` numbers:

* `%d` rounds down.
* `%n` rounds to the nearest value.
* `%u` rounds up.
* `%z` rounds to zero.

The default value is `%z`, round to zero. When we invoke `+add:rs` to call the addition function, there is no way to modify the `+rs` door sample, so the default rounding option is used. How do we change it? We use the `~( )` notation: `~(arm door arg)`.

Let's evaluate the `+add` arm of `+rs`, also modifying the door sample to `%u` for 'round up':

```
> ~(add rs %u)
<1.uka [[a=@rs b=@rs] <21.ezj [r=?(%d %n %u %z) <51.njr 139.oyl 33.uof 1.pnw %138>]>]>
```

This is the gate produced by `+add`, and you can see that its sample is a pair of `@rs` atoms. But if you look in the context you'll see the [rs](/hoon/stdlib/3b#rs) door. Let's look in the sample of that core to make sure that it changed to `%u`. We'll use the wing `+6.+7` to look at the sample of the gate's context:

```
> +6.+7:~(add rs %u)
r=%u
```

It did indeed change. We also see that the door sample uses the face `.r`, so let's use that instead of the unwieldy `+6.+7`:

```
> r:~(add rs %u)
%u
```

We can do the same thing for rounding down, `%d`:

```
> r:~(add rs %d)
%d
```

Let's see the rounding differences in action. Because `~(add rs %u)` produces a gate, we can call it like we would any other gate:

```
> (~(add rs %u) .3.14159265 .1.11111111)
.4.252704

> (~(add rs %d) .3.14159265 .1.11111111)
.4.2527037
```

This difference between rounding up and rounding down might seem strange at first. There is a difference of 0.0000003 between the two answers. Why does this gap exist? Single-precision floats are 32-bit and there's only so many distinctions that can be made in floats before you run out of bits.

Just as there is a door for `@rs` functions, there is a Hoon standard library door for `@rd` functions (double-precision 64-bit floats), another for `@rq` functions (quad-precision 128-bit floats), and one more for `@rh` functions (half-precision 16-bit floats).

## Signed Integer Mathematics <a href="#signed-integer-mathematics" id="signed-integer-mathematics"></a>

Similar to floating-point representations, [signed integer](https://en.wikipedia.org/wiki/Signed_number_representations) representations use an internal bitwise convention to indicate whether a number should be treated as having a negative sign in front of the magnitude or not. There are several ways to represent signed integers:

1. [**Sign-magnitude**](https://en.wikipedia.org/wiki/Signed_number_representations#Sign%E2%80%93magnitude). Use the first bit in a fixed-bit-width representation to indicate whether the whole should be multiplied by $$-1$$, e.g. `0010.1011` for $$43\_{10}$$ and `1010.1011` for $$-43\_{10}$$. (This is similar to the floating-point solution.)
2. [**One's complement**](https://en.wikipedia.org/wiki/Ones'_complement). Use the bitwise `NOT` operation to represent the value, e.g. `0010.1011` for $$43\_{10}$$ and `1101.0100` for $$-43\_{10}$$. This has the advantage that arithmetic operations are trivial, e.g. $$43\_{10} - 41\_{10}$$ = `0010.1011` + `1101.0110` = `1.0000.0001`, end-around carry the overflow to yield `0000.0010` = 2. (This is commonly used in hardware.)
3. [**Offset binary**](https://en.wikipedia.org/wiki/Offset_binary). This represents a number normally in binary *except* that it counts from a point other than zero, like `-256`.
4. [**ZigZag**](https://developers.google.com/protocol-buffers/docs/encoding?hl=en#signed-ints). Positive signed integers correspond to even atoms of twice their absolute value, and negative signed integers correspond to odd atoms of twice their absolute value minus one.

There are tradeoffs in compactness of representation and efficiency of mathematical operations.

### Hoon Operations <a href="#hoon-operations" id="hoon-operations"></a>

`@u`-aura atoms are *unsigned* values, but there is a complete set of *signed* auras in the `@s` series. ZigZag was chosen for Hoon's signed integer representation because it represents negative values with small absolute magnitude as short binary terms.

| Aura  | Meaning            | Example                   |
| ----- | ------------------ | ------------------------- |
| `@s`  | signed integer     |                           |
| `@sb` | signed binary      | `--0b11.1000` (positive)  |
|       |                    | `-0b11.1000` (negative)   |
| `@sd` | signed decimal     | `--1.000.056` (positive)  |
|       |                    | `-1.000.056` (negative)   |
| `@sx` | signed hexadecimal | `--0x5f5.e138` (positive) |
|       |                    | `-0x5f5.e138` (negative)  |

The [`+si`](/hoon/stdlib/3a#si) core supports signed-integer operations correctly. However, unlike the `@r` operations, `@s` operations have different names (likely to avoid accidental mental overloading).

To produce a signed integer from an unsigned value, use [`+new:si`](/hoon/stdlib/3a#newsi) with a sign flag, or simply use [`+sun:si`](/hoon/stdlib/3a#sunsi)

```
> (new:si & 2)
--2

> (new:si | 2)
-2

> `@sd`(sun:si 5)
--5
```

To recover an unsigned integer from a signed integer, use [`+old:si`](/hoon/stdlib/3a#oldsi), which returns the magnitude and the sign.

```
> (old:si --5)
[%.y 5]

> (old:si -5)
[%.n 5]
```

* [`+sum:si`](/hoon/stdlib/3a#sumsi), addition.
* [`+dif:si`](/hoon/stdlib/3a#difsi), subtraction.
* [`+pro:si`](/hoon/stdlib/3a#prosi), multiplication.
* [`+fra:si`](/hoon/stdlib/3a#frasi), division.
* [`+rem:si`](/hoon/stdlib/3a#remsi), modulus (remainder after division), b modulo a as `@s`.
* [`+abs:si`](/hoon/stdlib/3a#abssi), absolute value.
* [`+cmp:si`](/hoon/stdlib/3a#synsi), test for greater value (as index, `>` → `--1`, `<` → `-1`, `=` → `--0`).

To convert a floating-point value from number (atom) to text, use[`+scow`](/hoon/stdlib/4m#scow) or[+r-co:co](/hoon/stdlib/4k#r-coco) with[`+rlys`](/hoon/stdlib/3b#rlys) (and friends):

```
> (scow %rs .3.14159)
".3.14159"

> `tape`(r-co:co (rlys .3.14159))
"3.14159"
```

### Beyond Arithmetic <a href="#beyond-arithmetic" id="beyond-arithmetic"></a>

The Hoon standard library at the current time omits many [transcendental functions](https://en.wikipedia.org/wiki/Transcendental_function), such as the trigonometric functions. It is useful to implement pure-Hoon versions of these, although they are not as efficient as jetted mathematical code would be.

Produce a version of `+factorial` which can operate on `@rs` inputs correctly.

Produce an exponentiation function `+pow-n` which operates on integer `@rs` only.

```hoon
++  pow-n
  ::  restricted power, based on integers only
  |=  [x=@rs n=@rs]
  ^-  @rs
  ?:  =(n .0)  .1
  =/  p  x
  |-  ^-  @rs
  ?:  (lth:rs n .2)  p
  $(n (sub:rs n .1), p (mul:rs p x))
```

Using both of the above, produce the `+sine` function, defined by

$$
\sin(x)
\= \sum\_{n=0}^\infty \frac{(-1)^n}{(2n+1)!} x^{2n+1}
\= x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \cdots
$$

```hoon
++  sine
  ::  sin x = x - x^3/3! + x^5/5! - x^7/7! + x^9/9! - ...
  |=  x=@rs
  ^-  @rs
  =/  rtol  .1e-5
  =/  p   .0
  =/  po  .-1
  =/  i   .0
  |-  ^-  @rs
  ?:  (lth:rs (absolute (sub:rs po p)) rtol)  p
  =/  ii  (add:rs (mul:rs .2 i) .1)
  =/  term  (mul:rs (pow-n .-1 i) (div:rs (pow-n x ii) (factorial ii)))
  $(i (add:rs i .1), p (add:rs p term), po p)
```

Implement `+cosine`.

$$
\cos(x)
\= \sum\_{n=0}^\infty \frac{(-1)^n}{(2n)!} x^{2n}
\= 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \frac{x^6}{6!} + \cdots
$$

Implement `+tangent`.

$$
\tan(x) = \frac{\sin(x)}{\cos(x)}
$$

As a stretch exercise, look up definitions for [exp (e^x)](https://en.wikipedia.org/wiki/Exponentiation#The_exponential_function) and [natural logarithm](https://en.wikipedia.org/wiki/Natural_logarithm), and implement these. You can implement a general-purpose exponentiation function using the formula

$$
x^n = \exp(n \\, \text{ln} \\, x)
$$

(We will use these in subsequent examples.)

### Exercise: Calculate the Fibonacci Sequence <a href="#exercise-calculate-the-fibonacci-sequence" id="exercise-calculate-the-fibonacci-sequence"></a>

The Binet expression gives the $$n^\text{th}$$ Fibonacci number.

$$
F\_n = \frac{\varphi^n - (-\varphi)^{-n}}{\sqrt 5}
\= \frac{\varphi^n - (-\varphi)^{-n}}{2 \varphi - 1}
$$

Implement this analytical formula for the Fibonacci series as a gate.

## Date & Time Mathematics <a href="#date-time-mathematics" id="date-time-mathematics"></a>

Date and time calculations are challenging for a number of reasons: What is the correct granularity for an integer to represent? What value should represent the starting value? How should time zones and leap seconds be handled?

One particularly complicating factor is that there is no [Year Zero](https://en.wikipedia.org/wiki/Year_zero); 1 B.C. is immediately followed by A.D. 1. (The date systems used in astronomy [differ](https://en.wikipedia.org/wiki/Julian_day#cite_note-7) from standard time in this regard, for instance.)

In computing, absolute dates are calculated with respect to some base value; we refer to this as the "epoch". Unix/Linux systems count time forward from Thursday 1 January 1970 00:00:00 UT, for instance. Windows systems count in 10⁻⁷ s intervals from 00:00:00 1 January 1601. The Urbit epoch is `~292277024401-.1.1`, or 1 January 292,277,024,401 B.C.; since values are unsigned integers, no date before that time can be represented.

Time values, often referred to as "timestamps", are commonly represented by the [UTC](https://www.timeanddate.com/time/aboututc.html) value. Time representations are complicated by offset such as timezones, regular adjustments like daylight savings time, and irregular adjustments like leap seconds. (Read [Dave Taubler's excellent overview](https://levelup.gitconnected.com/why-is-programming-with-dates-so-hard-7477b4aeff4c) of the challenges involved with calculating dates for further considerations, as well as [Martin Thoma's “What Every Developer Should Know About Time” (PDF)](https://zenodo.org/record/1443533/files/2018-10-06-what-developers-should-know-about-time.pdf).)

### Hoon Operations <a href="#hoon-operations" id="hoon-operations"></a>

A timestamp can be separated into the time portion, which is the relative offset within a given day, and the date portion, which represents the absolute day.

There are two molds to represent time in Hoon: the `@d` aura, with `@da` for a full timestamp and `@dr` for an offset; and the [`$date`](/hoon/stdlib/2q#date) / [`$tarp`](/hoon/stdlib/2q#tarp) structure:

| Aura  | Meaning                    | Example                   |
| ----- | -------------------------- | ------------------------- |
| `@da` | Absolute date              | `~2022.1.1`               |
|       |                            | `~2022.1.1..1.1.1..0000`  |
| `@dr` | Relative date (difference) | `~h5.m30.s12`             |
|       |                            | `~d1000.h5.m30.s12..beef` |

```hoon
+$  date  [[a=? y=@ud] m=@ud t=tarp]
+$  tarp  [d=@ud h=@ud m=@ud s=@ud f=(list @ux)]
```

`.now` returns the `@da` of the current timestamp (in UTC).

To go from a `@da` to a `$tarp`, use [`+yell`](/hoon/stdlib/3c#yell):

```
> *tarp
[d=0 h=0 m=0 s=0 f=~]

> (yell now)
[d=106.751.991.821.625 h=22 m=58 s=10 f=~[0x44ff]]

> `tarp`(yell ~2014.6.6..21.09.15..0a16)
[d=106.751.991.820.172 h=21 m=9 s=15 f=~[0xa16]]

> (yell ~d20)
[d=20 h=0 m=0 s=0 f=~]
```

To go from a `@da` to a `$date`, use [`+yore`](/hoon/stdlib/3c#yore):

```
> (yore ~2014.6.6..21.09.15..0a16)
[[a=%.y y=2.014] m=6 t=[d=6 h=21 m=9 s=15 f=~[0xa16]]]

> (yore now)
[[a=%.y y=2.022] m=5 t=[d=24 h=16 m=20 s=57 f=~[0xbaec]]]
```

To go from a `$date` to a `@da`, use [`+year`](/hoon/stdlib/3c#year):

```
> (year [[a=%.y y=2.014] m=8 t=[d=4 h=20 m=4 s=57 f=~[0xd940]]])
~2014.8.4..20.04.57..d940

> (year (yore now))
~2022.5.24..16.24.16..d184
```

To go from a `$tarp` to a `@da`, use[`+yule`](/hoon/stdlib/3c#yule):

```
> (yule (yell now))
0x8000000d312b148891f0000000000000

> `@da`(yule (yell now))
~2022.5.24..16.25.48..c915

> `@da`(yule [d=106.751.991.823.081 h=16 m=26 s=14 f=~[0xf727]])
~2022.5.24..16.26.14..f727
```

The Urbit date system correctly compensates for the lack of Year Zero:

```
> ~0.1.1
~1-.1.1

> ~1-.1.1
~1-.1.1
```

The [`+yo`](/hoon/stdlib/3c#yo) core contains constants useful for calculating time, but in general you should not hand-roll time or timezone calculations.

### Tutorial: Julian Day <a href="#tutorial-julian-day" id="tutorial-julian-day"></a>

Astronomers use the [Julian day](https://en.wikipedia.org/wiki/Julian_day) to uniquely denote days. (This is not to be confused with the Julian calendar.) The following core demonstrates conversion to and from Julian days using signed integer (`@sd`) and date (`@da`) mathematics.

<details>

<summary>Julian days conversion core</summary>

```hoon
|%
++  ju
  |%
  ++  to
    |=  =@da  ^-  @sd
    =,  si
    =/  date  (yore da)
    =/  y=@sd  (sun y.date)
    =/  m=@sd  (sun m.date)
    =/  d=@sd  (sun d.t.date)
    ;:  sum
      (fra (pro --1.461 :(sum y --4.800 (fra (sum m -14) --12))) --4)
      (fra (pro --367 :(sum m -2 (pro -12 (fra (sum m -14) --12)))) --12)
      (fra (pro -3 (fra :(sum y --4.900 (fra (sum m -14) --12)) --100)) --4)
      d
      -32.075
    ==
  ++  from
    |=  =@sd  ^-  @da
      =,  si
      :: f = J + 1401 + (((4 × J + 274277) ÷ 146097) × 3) ÷ 4 - 38
      =/  f  ;:  sum 
               sd
               --1.401
               (fra (pro (fra (sum (pro --4 sd) --274.277) --146.097) --3) --4)
               -38
             ==
      :: e = 4 × f + 3
      =/  e  (sum (pro --4 f) --3)
      :: g = mod(e, 1461) ÷ 4
      =/  g  (fra (mod e --1.461) --4)
      :: h = 5 × g + 2
      =/  h  (sum (pro --5 g) --2)
      :: D = (mod(h, 153)) ÷ 5 + 1
      =/  dy  (sum (fra (mod h --153) --5) --1)
      :: M = mod(h ÷ 153 + 2, 12) + 1
      =/  mn  (sum (mod (sum (fra h --153) --2) --12) --1)
      :: Y = (e ÷ p) - y + (n + m - M) ÷ n
      =/  yr  (sum (dif (fra e --1.461) --4.716) (fra (sum --12 (dif --2 mn)) --12))
      =/  dy=@ud  (div dy 2)
      =/  mn=@ud  (div mn 2)
      =/  yr=@ud  (div yr 2)
      (year [[a=(gth yr --0) yr] mn [dy 0 0 0 ~]])
--
```

</details>

## Unusual Bases <a href="#unusual-bases" id="unusual-bases"></a>

### Phonetic Base <a href="#phonetic-base" id="phonetic-base"></a>

The `@q` aura is similar to `@p` except for two details: it doesn't obfuscate names (as planets do) and it can be used for any size of atom without adjust its width to fill the same size. Prefixes and suffixes are in the same order as `@p`, however. Thus:

```
> `@q`0
.~zod

> `@q`256
.~marzod

> `@q`65.536
.~nec-dozzod

> `@q`4.294.967.296
.~nec-dozzod-dozzod

> `@q`(pow 2 128)
.~nec-dozzod-dozzod-dozzod-dozzod-dozzod-dozzod-dozzod-dozzod
```

`@q` auras can be used as sequential mnemonic markers for values.

The [`+po`](/hoon/stdlib/4a#po) core contains tools for directly parsing `@q` atoms.

### Base-32 and Base-64 <a href="#base-32-and-base-64" id="base-32-and-base-64"></a>

The base-32 representation uses the characters "0123456789abcdefghijklmnopqrstuv" to represent values. The digits are separated into collections of five characters separated by `.` dot.

```
> `@uv`0
0v0

> `@uv`100
0v34

> `@uv`1.000.000
0vugi0

> `@uv`1.000.000.000.000
0vt3a.aa400
```

The base-64 representation uses the characters `0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-~` to represent values. The digits are separated into collections of five characters separated by `.` dot.

```
> `@uw`0
0w0

> `@uw`100
0w1A

> `@uw`1.000.000
0w3Q90

> `@uw`1.000.000.000
0wXCIE0

> `@uw`1.000.000.000.000
0wez.kFh00
```

## Randomness <a href="#randomness" id="randomness"></a>

### Entropy <a href="#entropy" id="entropy"></a>

You previously saw entropy introduced when we discussed stateful random number generation. Let's dig into what's actually going on with entropy.

It is not straightforward for a computer, a deterministic machine, to produce an unpredictable sequence. We can either use a source of true randomness (such as the third significant digit of chip temperature or another [hardware source](https://en.wikipedia.org/wiki/Hardware_random_number_generator)) or a source of artificial randomness (such as a sequence of numbers the user cannot predict).

For instance, consider the sequence *3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3*. If you recognize the pattern as the constant *π*, you can predict the first few digits, but almost certainly not more than that. The sequence is deterministic (as it is derived from a well-characterized mathematical process) but unpredictable (as you cannot *a priori* guess what the next digit will be).

Computers often mix both deterministic processes (called “pseudorandom number generators”) with random inputs, such as the current timestamp, to produce high-quality random numbers for use in games, modeling, cryptography, and so forth. The Urbit entropy value `.eny` is derived from the underlying host OS's `/dev/urandom` device, which uses sources like keystroke typing latency to produce random bits.

### Random Numbers <a href="#random-numbers" id="random-numbers"></a>

Given a source of entropy to seed a random number generator, one can then use the [`+og`](/hoon/stdlib/3d#og) door to produce various kinds of random numbers. The basic operations of `+og` are described in [the lesson on subject-oriented programming](/build-on-urbit/hoon-school/o-subject).

### Exercise: Implement a random-number generator from scratch <a href="#exercise-implement-a-random-number-generator-from-scratch" id="exercise-implement-a-random-number-generator-from-scratch"></a>

* Produce a random stream of bits using the linear congruential random number generator.

The linear congruential random number generator produces a stream of random bits with a repetition period of $$2^{31}$$. Numericist John Cook [explains how LCGs work](https://www.johndcook.com/blog/2017/07/05/simple-random-number-generator/):

> The linear congruential generator used here starts with an arbitrary seed, then at each step produces a new number by multiplying the previous number by a constant and taking the remainder by $$2^{31} - 1$$.

**`/gen/lcg.hoon`**

```hoon
|=  n=@ud                 :: n is the number of bits to return
=/  z  20.220.524         :: z is the seed
=/  a  742.938.285        :: a is the multiplier
=/  e  31                 :: e is the exponent
=/  m  (sub (pow 2 e) 1)  :: modulus
=/  index  0
=/  accum  *@ub
|-  ^-  @ub
?:  =(index n)  accum
%=  $
  index  +(index)
  z      (mod (mul a z) m)
  accum  (cat 5 z accum)
==
```

Can you verify that `1`s constitute about half of the values in this bit stream, as Cook illustrates in Python?

### Exercise: Produce uniformly-distributed random numbers <a href="#exercise-produce-uniformly-distributed-random-numbers" id="exercise-produce-uniformly-distributed-random-numbers"></a>

Using entropy as the source, produce uniform random numbers: that is, numbers in the range \[0, 1] with equal likelihood to machine precision.

We use the LCG defined above, then chop out 23-bit slices using [`+rip`](/hoon/stdlib/2c#rip) to produce each number, manually compositing the result into a valid floating-point number in the range \[0, 1]. (We avoid producing special sequences like [`NaN`](https://en.wikipedia.org/wiki/NaN).)

<details>

<summary>/gen/uniform.hoon</summary>

```hoon
!:
=<
|=  n=@ud  :: n is the number of values to return
^-  (list @rs)
=/  values  (rip 5 (~(lcg gen 20.220.524) n))
=/  mask-clear           0b111.1111.1111.1111.1111.1111
=/  mask-fill   0b11.1111.0000.0000.0000.0000.0000.0000
=/  clears  (turn values |=(a=@rs (dis mask-clear a)))
(turn clears |=(a=@ (sub:rs (mul:rs .2 (con mask-fill a)) .1.0)))
|%
++  gen
  |_  [z=@ud]
  ++  lcg
    |=  n=@ud                 :: n is the number of bits to return
    =/  a  742.938.285        :: a is the multiplier
    =/  e  31                 :: e is the exponent
    =/  m  (sub (pow 2 e) 1)  :: modulus
    =/  index  0
    =/  accum  *@ub
    |-  ^-  @ub
    ?:  =(index n)  accum
    %=  $
      index  +(index)
      z      (mod (mul a z) m)
      accum  (cat 5 z accum)
    ==
  --
--
```

</details>

Convert the above to a `%say` generator that can optionally accept a seed; if no seed is provided, use `.eny`.

Produce a higher-quality Mersenne Twister uniform RNG, such as [per this method](https://xilinx.github.io/Vitis_Libraries/quantitative_finance/2022.1/guide_L1/RNGs/RNG.html).

### Exercise: Produce normally-distributed random numbers <a href="#exercise-produce-normally-distributed-random-numbers" id="exercise-produce-normally-distributed-random-numbers"></a>

Produce a normally-distributed random number generator using the uniform RNG described above.

The normal distribution, or bell curve, describes the randomness of measurement. The mean, or average value, is at zero, while points fall farther and farther away with increasingly less likelihood.

![](https://storage.googleapis.com/media.urbit.org/docs/hoon-school/Standard_deviation_diagram.png)

One way to get from a uniform random number to a normal random number is [to use the uniform random number as the *cumulative distribution function* (CDF)](https://www.omscs-notes.com/simulation/generating-uniform-random-numbers/), an index into “how far” the value is along the normal curve.

![](https://storage.googleapis.com/media.urbit.org/docs/hoon-school/Normal_Distribution_CDF.png)

This is an approximation which is accurate to one decimal place:

$$
Z = \frac{U^{0.135} - (1-U)^{0.135}}{0.1975}
$$

where sgn is the signum or sign function.

To calculate an arbitrary power of a floating-point number, we require a few transcendental functions, in particular the natural logarithm and exponentiation of base $$e$$. The following helper core contains relatively inefficient but clear implementations of standard numerical methods.

<details>

<summary>/gen/normal.hoon</summary>

```hoon
!:
=<
|=  n=@ud  :: n is the number of values to return
^-  (list @rs)
=/  values  (rip 5 (~(lcg gen 20.220.524) n))
=/  mask-clear           0b111.1111.1111.1111.1111.1111
=/  mask-fill   0b11.1111.0000.0000.0000.0000.0000.0000
=/  clears    (turn values |=(a=@rs (dis mask-clear a)))
=/  uniforms  (turn clears |=(a=@ (sub:rs (mul:rs .2 (con mask-fill a)) .1.0)))
(turn uniforms normal)
|%
++  factorial
  :: integer factorial, not gamma function
  |=  x=@rs
  ^-  @rs
  =/  t=@rs  .1
  |-  ^-  @rs
  ?:  |(=(x .1) (lth x .1))  t
  $(x (sub:rs x .1), t (mul:rs t x))
++  absrs
  |=  x=@rs  ^-  @rs
  ?:  (gth:rs x .0)
    x
  (sub:rs .0 x)
++  exp
  |=  x=@rs
  ^-  @rs
  =/  rtol  .1e-5
  =/  p   .1
  =/  po  .-1
  =/  i   .1
  |-  ^-  @rs
  ?:  (lth:rs (absrs (sub:rs po p)) rtol)  p
  $(i (add:rs i .1), p (add:rs p (div:rs (pow-n x i) (factorial i))), po p)
++  pow-n
  ::  restricted power, based on integers only
  |=  [x=@rs n=@rs]
  ^-  @rs
  ?:  =(n .0)  .1
  =/  p  x
  |-  ^-  @rs
  ?:  (lth:rs n .2)  p
  $(n (sub:rs n .1), p (mul:rs p x))
++  ln
  ::  natural logarithm, z > 0
  |=  z=@rs
  ^-  @rs
  =/  rtol  .1e-5
  =/  p   .0
  =/  po  .-1
  =/  i   .0
  |-  ^-  @rs
  ?:  (lth:rs (absrs (sub:rs po p)) rtol)
    (mul:rs (div:rs (mul:rs .2 (sub:rs z .1)) (add:rs z .1)) p)
  =/  term1  (div:rs .1 (add:rs .1 (mul:rs .2 i)))
  =/  term2  (mul:rs (sub:rs z .1) (sub:rs z .1))
  =/  term3  (mul:rs (add:rs z .1) (add:rs z .1))
  =/  term  (mul:rs term1 (pow-n (div:rs term2 term3) i))
  $(i (add:rs i .1), p (add:rs p term), po p)
++  powrs
  ::  general power, based on logarithms
  ::  x^n = exp(n ln x)
  |=  [x=@rs n=@rs]
  (exp (mul:rs n (ln x)))
++  normal
  |=  u=@rs
  (div:rs (sub:rs (powrs u .0.135) (powrs (sub:rs .1 u) .0.135)) .0.1975)
++  gen
  |_  [z=@ud]
  ++  lcg
    |=  n=@ud                 :: n is the number of bits to return
    =/  a  742.938.285        :: a is the multiplier
    =/  e  31                 :: e is the exponent
    =/  m  (sub (pow 2 e) 1)  :: modulus
    =/  index  0
    =/  accum  *@ub
    |-  ^-  @ub
    ?:  =(index n)  accum
    %=  $
      index  +(index)
      z      (mod (mul a z) m)
      accum  (cat 5 z accum)
    ==
  --
--
```

</details>

### Exercise: Upgrade the normal RNG <a href="#exercise-upgrade-the-normal-rng" id="exercise-upgrade-the-normal-rng"></a>

A more complicated formula uses several constants to improve the accuracy significantly:

$$
Z = \text{sgn}\left(U-\frac{1}{2}\right) \left( t - \frac{c\_{0}+c\_{1} t+c\_{2} t^{2}}{1+d\_{1} t+d\_{2} t^{2} + d\_{3} t^{3}} \right)
$$

where

* sgn is the signum or sign function;
* $$t$$ is $$\sqrt{-\ln\[\min(U, 1-U)^2]}$$; and
* the constants are:
  * $$c\_0 = 2.515517$$
  * $$c\_1 = 0.802853$$
  * $$c\_2 = 0.010328$$
  * $$d\_1 = 1.532788$$
  * $$d\_2 = 0.189268$$
  * $$d\_3 = 0.001308$$

Implement this formula in Hoon to produce normally-distributed random numbers.

How would you implement other random number generators?

## Hashing <a href="#hashing" id="hashing"></a>

A [hash function](https://en.wikipedia.org/wiki/Hash_function) is a tool which can take any input data and produce a fixed-length value that corresponds to it. Hashes can be used for many purposes:

1. **Encryption**. A [cryptographic hash function](https://en.wikipedia.org/wiki/Cryptographic_hash_function) leans into the one-way nature of a hash calculation to produce a fast, practically-irreversible hash of a message. They are foundational to modern cryptography.
2. **Attestation or preregistration**. If you wish to demonstrate that you produced a particular message at a later time (including a hypothesis or prediction), or that you solved a particular problem, hashing the text of the solution and posting the hash publicly allows you to verifiably timestamp your work.
3. **Integrity verification**. By comparing the hash of data to its expected hash, you can verify that two copies of data are equivalent (such as a downloaded executable file). The [MD5](https://en.wikipedia.org/wiki/MD5) hash algorithm is frequently used for this purpose as [`md5sum`](https://en.wikipedia.org/wiki/Md5sum).
4. **Data lookup**. [Hash tables](https://en.wikipedia.org/wiki/Hash_table) are one way to implement a key→value mapping, such as the functionality offered by Hoon's [`+map`](/hoon/stdlib/2o#map).

Theoretically, since the number of fixed-length hashes are finite, an infinite number of possible programs can yield any given hash. This is called a ["hash collision"](https://en.wikipedia.org/wiki/Hash_collision), but for many practical purposes such a collision is extremely unlikely.

### Hoon Operations <a href="#hoon-operations" id="hoon-operations"></a>

The Hoon standard library supports fast insecure hashing with [`+mug`](/hoon/stdlib/2e#mug), which accepts any noun and produces an atom of the hash.

```
> `@ux`(mug 1)
0x715c.2a60

> `@ux`(mug 2)
0x718b.9468

> `@ux`(mug 3)
0x72a8.ef1a

> `@ux`(mug 1.000.000)
0x5145.9d7d

> `@ux`(mug .)
0x6c91.8422
```

`+mug` operates on the raw form of the noun however, without Hoon-specific metadata like aura:

```
> (mug 0x5)
721.923.263

> (mug 5)
721.923.263
```

Hoon also includes [SHA-256 and SHA-512](https://en.wikipedia.org/wiki/SHA-2) [tooling](/hoon/stdlib/3d). ([`+og`](/hoon/stdlib/3d#og), the random number generator, is based on SHA-256 hashing.)

[`+shax`](/hoon/stdlib/3d#shax) produces a hashed atom of 256 bits from any atom.

```
> `@ux`(shax 1)
0x9a45.8577.3ce2.ccd7.a585.c331.d60a.60d1.e3b7.d28c.bb2e.de3b.c554.4534.2f12.f54b

> `@ux`(shax 2)
0x86d9.5764.98ea.764b.4924.3efe.b05d.f625.0104.38c6.a55d.5b57.8de4.ff00.c9b4.c1db

> `@ux`(shax 3)
0xc529.ffad.9a5a.b611.62b1.1d61.6b63.9e00.586b.a846.746a.197d.4daf.78b9.08ed.4f08

> `@ux`(shax 1.000.000)
0x84a4.929b.1d69.708e.d4b7.0fb8.ca97.cc85.c4a6.1aae.4596.f753.d0d2.6357.e7b9.eb0f
```

[`+shaz`](/hoon/stdlib/3d#shaz) produces a hashed atom of 512 bits from any atom.

```
> (shaz 1)
3.031.947.054.025.992.811.210.838.487.475.158.569.967.793.095.050.169.760.709.406.427.393.828.309.497.273.121.275.530.382.185.415.047.474.588.395.933.812.689.047.905.034.106.140.802.678.745.778.695.328.891

> `@ux`(shaz 1)
0x39e3.d936.c6e3.1eaa.c08f.cfcf.e7bb.4434.60c6.1c0b.d5b7.4408.c8bc.c35a.6b8d.6f57.00bd.cdde.aa4b.466a.e65f.8fb6.7f67.ca62.dc34.149e.1d44.d213.ddfb.c136.68b6.547b

> `@ux`(shaz 2)
0xcadc.698f.ca01.cf29.35f7.6027.8554.b4e6.1f35.4539.75a5.bb45.3890.0315.9bc8.485b.7018.dd81.52d9.cc23.b6e9.dd91.b107.380b.9d14.ddbf.9cc0.37ee.53a8.57b6.c948.b8fa

> `@ux`(shaz 3)
0x4ba.a6ba.4a01.12e6.248b.5e89.9389.4786.aced.1a59.136b.78c6.7076.eb90.2221.d7a5.453a.56d1.446d.17d1.33cd.b468.f798.eb6b.dcee.f071.7040.7a2f.aa94.df7d.81f5.5be4

> `@ux`(shaz 1.000.000)
0x4c13.ef8b.09cf.6e59.05c4.f203.71a4.9cec.3432.ba26.0174.f964.48f1.5475.b2dd.2c59.98c2.017c.9c03.cbea.9d5f.591b.ff23.bbff.b0ae.9c67.a4a9.dd8d.748a.8e14.c006.cbcc
```

### Exercise: Produce a secure password tool <a href="#exercise-produce-a-secure-password-tool" id="exercise-produce-a-secure-password-tool"></a>

Produce a basic secure password tool. It should accept a password, salt it (add a predetermined value to the password), and hash it. *That* hash is then compared to a reference hash to determine whether or not the password is correct.


# App School I

Guide to building Gall agents on Urbit.

This guide will walk through everything you need to know to write your own Gall agents.

App School I is suitable for anyone with an intermediate knowledge of Hoon. If you've worked through [Hoon School](/build-on-urbit/hoon-school), you should have everything you need.

## What are Gall agents? <a href="#what-are-gall-agents" id="what-are-gall-agents"></a>

Gall is one of the ten vanes (kernel modules) of Arvo, Urbit's operating system. Gall's purpose is to manage userspace applications called "agents".

An agent is a piece of software that is primarily focused on maintaining and distributing a piece of state with a defined structure. It exposes an interface that lets programs read, subscribe to, and manipulate the state. Every event happens in an atomic transaction, so the state is never inconsistent. Since the state is permanent, when the agent is upgraded with a change to the structure of the state, the developer provides a migration function from the old state type to the new state type.

It's not too far off to think of an agent as simply a database with developer-defined logic. But an agent is significantly less constrained than a database. Databases are usually tightly constrained in one or more ways because they need to provide certain guarantees (like atomicity) or optimizations (like indexes). Arvo is a [single-level store](/urbit-os/kernel/arvo#single-level-store), so atomicity comes for free. Many applications don't use databases because they need relational indices; rather, they use them for their guarantees around persistence. Some do need the indices, though, and it's not hard to imagine an agent which provides a SQL-like interface.

On the other hand, an agent is also a lot like what many systems call a "service". An agent is permanent and addressable -- a running program can talk to an agent just by naming it. An agent can perform [IO](https://urbit.org/blog/io-in-hoon), unlike most databases. This is a critical part of an agent: it performs IO along the same transaction boundaries as changes to its state, so if an effect happens, you know that the associated state change has happened.

But the best way to think about an agent is as a state machine. Like a state machine, any input could happen at any time, and it must react coherently to that input. Output (effects) and the next state of the machine are a pure function of the previous state and the input event.

## Table of Contents <a href="#table-of-contents" id="table-of-contents"></a>

#### Lessons

1. [Arvo](/build-on-urbit/app-school/1-arvo) - This lesson provides an overview of the Arvo operating system, and some other useful background information.
2. [The Agent Core](/build-on-urbit/app-school/2-agent) - This lesson goes over the basic structure of a Gall agent.
3. [Imports and Aliases](/build-on-urbit/app-school/3-imports-and-aliases) - This lesson covers some useful libraries, concepts and boilerplate commonly used when writing Gall agents.
4. [Lifecycle](/build-on-urbit/app-school/4-lifecycle) - This lesson introduces the state management arms of an agent.
5. [Cards](/build-on-urbit/app-school/5-cards) - This lesson covers `$card`s - the structure used to pass messages to other agents and vanes.
6. [Pokes](/build-on-urbit/app-school/6-pokes) - This lesson covers sending and receiving one-off messages called "pokes" between agents.
7. [Structures and Marks](/build-on-urbit/app-school/7-sur-and-marks) - This lesson talks about importing type defintions, and writing mark files.
8. [Subscriptions](/build-on-urbit/app-school/8-subscriptions) - This lesson goes through the mechanics of subscriptions - both inbound and outbound.
9. [Vanes](/build-on-urbit/app-school/9-vanes) - This lesson explains how to interact with vanes (kernel modules) from an agent.
10. [Scries](/build-on-urbit/app-school/10-scry) - This lesson gives an overview of scrying Gall agents, and how scry endpoints are defined in agents.
11. [Failure](/build-on-urbit/app-school/11-fail) - This lesson covers how Gall handles certain errors and crashes, as well as the concept of a helper core.
12. [Next Steps](/build-on-urbit/app-school/12-next-steps) - App School I is now complete - here are some things you can look at next.

#### Appendix

* [Types](/build-on-urbit/app-school/types) - A reference for a few of the types used in App School.


# 1. Arvo

Introduction to Arvo, Urbit's operating system and kernel.

This document is a prologue to App School I. If you've worked though [Hoon School](/build-on-urbit/hoon-school) (or have otherwise learned the basics of Hoon), you'll likely be familiar with generators, but not with all the other parts of the Arvo operating system or the way it fits together. We'll go over the basic details here so you're better oriented to learn Gall agent development. We'll not go into the internal workings of the kernel much, but just what is necessary to understand it from the perspective of userspace.

## Arvo and its Vanes <a href="#arvo-and-its-vanes" id="arvo-and-its-vanes"></a>

[Arvo](/urbit-os/kernel/arvo) is the Urbit OS and kernel which is written in Hoon, compiled to Nock, and executed by the runtime environment and virtual machine Vere. Arvo has ten kernel modules called "vanes":

1. [Ames](/urbit-os/kernel/ames)
2. [Behn](/urbit-os/kernel/behn)
3. [Clay](/urbit-os/kernel/clay)
4. [Dill](/urbit-os/kernel/dill)
5. [Eyre](/urbit-os/kernel/eyre)
6. [Gall](/urbit-os/kernel/gall)
7. [Iris](/urbit-os/kernel/iris)
8. [Jael](/urbit-os/kernel/jael)
9. [Khan](/urbit-os/kernel/khan)
10. [Lick](/urbit-os/kernel/lick)

The Arvo kernel itself has its own small codebase in `/sys/arvo.hoon` which primarily implements the [transition function](/urbit-os/kernel/arvo#operating-function) `(State, Event) -> (State, Effects)` for events injected by the runtime. It also handles inter-vane messaging, the [scry](/urbit-os/kernel/arvo/scry) system, and a couple of other things. Most of the heavy lifting is done by the vanes themselves - Arvo itself typically just routes events to the relevant vanes.

Each vane has its own state. Gall's state contains the agents it's managing, Clay's state contains all the desks and their files, Jael's state contains all its PKI data, etc. All the vanes and their states live in Arvo's state, so Arvo's state ultimately contains the entire OS and its data.

Here's a brief summary of each of the vanes:

* **Ames**: This is both the name of Urbit's networking protocol, as well as the vane that handles communications over it. All inter-ship communications are done with Ames, but you'd not typically deal with it directly in a Gall agent because Gall itself handles it for you.
* **Behn**: A simple timer vane. Behn lets your Gall agent set timers which go off at the time specified and notify your agent.
* **Clay**: Filesystem vane. Clay is a revision-controlled, typed filesystem with a built-in build system. Your agent's source code lives in Clay. Your agent's source code and relevant files are automatically built and loaded upon installation, so your Gall agent itself would not need to interact with Clay unless you specifically wanted to read and write files.
* **Dill**: Terminal driver vane. You would not typically interact with Dill directly; printing debug messages to the terminal is usually done with hinting runes and functions rather than tasks to Dill, and CLI apps are mediated by a sub-module of the `%hood` system agent called `%drum`. CLI apps will not be touched on in this guide, but there's a separate [CLI Apps](/build-on-urbit/userspace/cli-tutorial) guide which covers them if you're interested.
* **Eyre**: HTTP server vane. App web front-ends are served via Eyre. It's possible to handle HTTP requests directly in a Gall agent (see the [Eyre Guide](/urbit-os/kernel/eyre/guide) for details), but usually you'd just serve a front-end [glob](/build-on-urbit/userspace/dist/glob) via the `%docket` agent, so you'd not typically have your agent deal with Eyre directly.
* **Gall**: App management vane; this is where your app's backend will live.
* **Iris**: HTTP client vane. If you want your agent to query external web APIs and the like, it's done via Iris. Oftentimes web API interactions are spun out into [threads](/urbit-os/base/threads/basics/fundamentals) to avoid complicating the Gall agent itself, so a Gall agent would not necessary deal with Iris directly, even if it made use of external APIs.
* **Jael**: Networking key infrastructure vane. Jael keeps track of PKI data for your ship and other ships on the network. Jael's data is most heavily used by Ames, and since Gall handles Ames communications for you, you'd not typically deal with Jael directly unless your were specifically writing something that made use of its data.
* **Khan**: Control plane vane. The main purpose of Khan is for external applications to be able to run threads via a Unix socket and receive their results. Khan's external interface is still experimental, but it's also good for running threads internally.
* **Lick**: Inter-process communication (IPC) vane. Lick manages IPC ports, and the communication between Urbit applications and POSIX applications via these ports. Other vanes and applications ask Lick to open an IPC port, notify it when something is connected or disconnected, and transfer data between itself and the Unix application.

## Userspace <a href="#userspace" id="userspace"></a>

Gall agents live in "userspace" as opposed to "kernelspace". Kernelspace is Arvo and its vanes. Userspace is primarily Gall agents, generators, threads, front-ends, and all of their related files in Clay. The distinction looks something like this:

![](https://media.urbit.org/guides/core/app-school/kernelspace-userspace-diagram-v1.svg)

By and large, Gall *is* the userspace vane - the majority of userspace is either Gall agents, or things used by Gall agents. Apart from the agents themselves, there's also:

* **Generators**: These are basically scripts. You'll likely already be familiar with these from Hoon School. Aside from learning exercises, their main use is to make interacting with Gall agents from the Dojo easier. Rather than having to manually craft `%poke`s to agents, generators can take a simpler input, reformat it into what the agent actually expects, and poke the agent with it. When you do something like `:dojo|wipe` in the dojo, you're actually running the `/gen/dojo/wipe.hoon` generator and poking the `%dojo` agent with its output.
* **Threads**: While generators are for strictly synchronous operations, threads make it easy to implement sequences of asynchronous operations. Threads are managed by the `%spider` agent or the Khan vane. They can be used as mere scripts like generators, but their main purpose is for performing complex IO. For example, suppose you need to query some external web API, then with the data in its response you make another API call, and then another, before finally having the data you need. If one of the API calls fails, your Gall agent is potentially left in a strange intermediary state. Instead, you can put all the IO logic in a separate thread which is completely atomic. That way the Gall agent only has to deal with the two conditions: success or failure. Writing threads is covered in a [separate guide](/urbit-os/base/threads/basics/fundamentals), which you might like to work through after completing App School I.
* **Front-end**: Web UIs. It's possible for Gall agents to handle HTTP requests directly and dynamically produce responses, but it's also possible to have a static [glob](/build-on-urbit/userspace/dist/glob) of HTML, CSS, Javascript, images, etc, which are served to the client like an ordinary web app. Such front-end files are typically managed by the `%docket` agent which serves them via Eyre. The [software distribution guide](/build-on-urbit/userspace/dist/software-distribution) covers this in detail, and you might like to work through it after completing App School I.

## The Filesystem <a href="#the-filesystem" id="the-filesystem"></a>

On an ordinary OS, you have persistent disk storage and volatile memory. An application is launched by reading an executable file on disk, loading it into memory and running it. The application will maybe read some more files from disk, deserialize them into data structures in memory, perform some computations and manipulate the data, then serialize the new data and write it back to disk. This process is necessary because persistent storage is too slow to operate on directly and the fast memory is wiped when it loses power. The result is that all non-ephemeral data is ultimately stored as files in the filesystem on disk. Arvo on the other hand is completely different.

Arvo has no concept of volatile memory - its whole state is assumed to be persistent. This means it's unnecessary for a Gall agent to write its data to the filesystem or read it in from the filesystem - an agent can just modify its state in situ and leave it there. The urbit runtime writes events to disk and backs up Arvo's state on the host OS to ensure data integrity but Arvo itself isn't concerned with such details.

The result of this total persistence is that the filesystem—Clay—does not have the same fundamental role as on an ordinary OS. In Arvo, very little of its data is actually stored in Clay. The vast majority is just in the state of Gall agents and vanes. For example, none of the chat messages, notebooks, etc, in the Tlon app exist in Clay - they're all in the state of the `%channels` agent. For the most part, Clay just stores source code.

Clay has a few unique features—it's a typed filesystem, with all file types defined in mark files. It's revision controlled, in a similar way to Git. It also has a built-in build system (formerly a separate vane called Ford, but was merged with Clay in 2020 to make atomicity of upgrades easier). We'll look at some of these features in more detail later in the guide.

## Desk Anatomy <a href="#desk-anatomy" id="desk-anatomy"></a>

The fundamental unit in Clay is a desk. Desks are kind of like git repositories. By default, new urbits come with the following desks included: `%base`, `%landscape`, `%groups` and `%webterm`.

* `%base` - This desk contains the kernel as well as some core agents and utilities.
* `%landscape` - This desk contains agents and utilities for managing apps, and the home screen that displays other app tiles.
* `%groups` - This desk contains everything for the Groups app.
* `%webterm` - This desk is for the web Dojo app.

You'll typically also have a `%kids` desk, which is just a copy of `%base` from upstream that sponsored ships (moons in the case of a planet, planets in the case of a star) sync their `%base` desk from. Any third-party apps you've installed will also have their own desks.

Desks are typically assumed to store their files according to the following directory structure:

```
desk
├── app
├── gen
├── lib
├── mar
├── sur
├── sys
├── ted
└── tests
```

* `/app`: Gall agents.
* `/gen`: Generators.
* `/lib`: Libraries - these are imported with the `/+` Ford rune.
* `/mar`: mark files, which are filetype definitions.
* `/sur`: Structures - these typically contain type definitions and structures, and would be imported with the `/-` Ford rune.
* `/sys`: Kernel files and standard library. Only the `%base` desk has this directory, it's omitted entirely in all other desks.
* `/ted`: Threads.
* `/tests`: Unit tests, to be run by the `%test` thread. This is often omitted in distributed desks.

This directory hierarchy is not strictly enforced, but most tools expect things to be in their right place. Any of these folders can be omitted if they'd otherwise be empty.

As mentioned, the `%base` desk alone includes a `/sys` directory containing the kernel and standard libraries. It looks like this:

```
sys
├── arvo.hoon
├── hoon.hoon
├── lull.hoon
├── vane
│   ├── ames.hoon
│   ├── behn.hoon
│   ├── clay.hoon
│   ├── dill.hoon
│   ├── eyre.hoon
│   ├── gall.hoon
│   ├── iris.hoon
│   ├── jael.hoon
│   ├── khan.hoon
│   └── lick.hoon
└── zuse.hoon
```

* `/arvo.hoon`: Source code for Arvo itself.
* `/hoon.hoon`: Hoon standard library and compiler.
* `/lull.hoon`: Mostly structures and type definitions for interacting with vanes.
* `/vane`: This directory contains the source code for each of the vanes.
* `/zuse.hoon`: This is an extra utility library. It mostly contains cryptographic functions and functions for dealing with web data like JSON.

The chain of dependency for the core kernel files is as follows: `hoon.hoon` is the subject of `arvo.hoon`, which is the subject of `lull.hoon`, which is the subject of `zuse.hoon`. This nested structure is the subject shared by each of the Arvo vanes. (For more information, see the [Filesystem Hierarchy](/urbit-os/kernel/clay/filesystem) documentation.)

In addition to the directories discussed, there's a handful of special files a desk might contain. All of them live in the root of the desk, and all are optional in the general case, except for `sys.kelvin`, which is mandatory.

* `sys.kelvin`: Specifies the kernel version with which the desk is compatible.
* `desk.bill`: Specifies Gall agents to be auto-started upon desk installation.
* `desk.ship`: If the desk is being republished, the original publisher can be specified here.
* `desk.docket-0`: Configures the front-end, tile, and other metadata for desks which include a home screen app.

Each desk must be self-contained; it must include all the marks, libraries, threads, etc, that it needs. The one exception is the kernel and standard libraries from the `%base` desk. Agents, threads and generators in other desks all have these libraries available to them in their subject.

## APIs <a href="#apis" id="apis"></a>

You should now have a general idea of the different parts of Arvo, but how does a Gall agent interact with these things?

There are two basic ways of interacting with other parts of the system: by scrying into them, and by passing them messages and receiving messages in response. There are also two basic things to interact with: vanes, and other agents.

### Scries

The scry system allows you to access the state of other agents and vanes in a read-only fashion. Scries can be performed from any context with the dotket (`.^`) rune. Each vane has "scry endpoints" which define what you can read, and these are comprehensively documented in the Scry Reference of each vane's section of the [Arvo documentation](/urbit-os/kernel/arvo). Agents define scry endpoints in the `+on-peek` arm of their agent core. Scries can only be done on the local ship; it is not yet possible to perform scries over the network (but this functionality is planned for the future). There is a separate [guide to scries](/urbit-os/kernel/arvo/scry) which you might like to read through for more details.

### Messages:

* Vanes: Each vane has a number of tasks it can be passed and gifts it can respond with in its respective section of `/sys/lull.hoon`. These might do all manner of things, depending on the vane. For example, Iris might fetch an external HTTP resource for you, Clay might read or build a specified file, etc. The tasks and gifts of each vane are comprehensively documented in the API Reference of each vane's section of the [Arvo documentation](/urbit-os/kernel/arvo).
* Agents: These can be `%poke`d with some data, which is a request to perform a single action. They can also be `%watch`ed, which means to subscribe for updates. We'll discuss these in detail later in the guide.

Here's a simplified diagram of the ways an agent can interact with other parts of the system:

![](https://media.urbit.org/guides/core/app-school/api-diagram.svg)

Things like `+on-poke` are arms of the agent core. Don't worry about their meaning for now, we'll discuss them in detail later in the guide.

Inter-agent messaging can occur over the network, so you can interact with agents on other ships as well as local ones. You can only talk to local vanes, but some vanes like Clay are able to make requests to other ships on your behalf. Note this summary is simplified - vanes don't just talk in tasks and gifts in all cases. For example, requests from HTTP clients through Eyre (the webserver vane) behave more like those from agents than vanes, and a couple of other vanes also have some different behaviours. Agent interactions are also a little more complicated, and we'll discuss that later, but the basic patterns described here cover the majority of cases.

## Environment Setup <a href="#environment-setup" id="environment-setup"></a>

Before proceeding with App School, you'll need to have an appropriate text editor installed and configured, and know how to work with a fake ship for development. Best practices are described in the [environment setup guide](/build-on-urbit/environment). Example agents and other code throughout this guide will just be committed to the `%base` desk of a fake ship, but it's a good idea to have a read through that guide for when you begin work on your own apps.


# 2. The Agent Core

Guide to Gall agent structure and core concepts.

In this lesson we'll look at the basic type and structure of a Gall agent.

A Gall agent is a door with exactly ten arms. Each arm is responsible for handling certain kinds of events that Gall feeds in to the agent. A door is just a core with a sample - it's made with the [barcab](/hoon/rune/bar#barcab) rune (`|_`) instead of the usual [barcen](/hoon/rune/bar#barcen) rune (`|%`).

## The ten arms <a href="#the-ten-arms" id="the-ten-arms"></a>

We'll discuss each of the arms in detail later. For now, here's a quick summary. The arms of an agent can be be roughly grouped by purpose:

### State management <a href="#state-management" id="state-management"></a>

These arms are primarily for initializing and upgrading an agent.

1. `+on-init`: Handles starting an agent for the first time.
2. `+on-save`: Handles exporting an agent's state, typically as part of the upgrade process but also when suspending, uninstalling and debugging.
3. `+on-load`: Handles loading a previously exported agent state, typically as part of the upgrade process but also when resuming or reinstalling an agent.

### Request handlers <a href="#request-handlers" id="request-handlers"></a>

These arms handle requests initiated by outside entities, e.g. other agents, HTTP requests from the frontend, etc.

4. `+on-poke`: Handles one-off requests, actions, etc.
5. `+on-watch`: Handles subscription requests from other entities.
6. `+on-leave`: Handles unsubscribe notifications from other, previously subscribed entities.

### Response handlers <a href="#response-handlers" id="response-handlers"></a>

These two arms handle responses to requests our agent previously initiated.

7. `+on-agent`: Handles request acknowledgements and subscription updates from other agents.
8. `+on-arvo`: Handles responses from vanes.

### Scry handler <a href="#scry-handler" id="scry-handler"></a>

9. `+on-peek`: Handles local read-only requests.

### Failure handler <a href="#failure-handler" id="failure-handler"></a>

10. `+on-fail`: Handles certain kinds of crash reports from Gall.

## Bowl <a href="#bowl" id="bowl"></a>

The sample of a Gall agent door is always a `$bowl:gall`. Every time an event triggers the agent, Gall populates the bowl with things like the current date-time, fresh entropy, subscription information, which ship the request came from, etc, so that all the arms of the agent have access to that data. For the exact structure and contents of the bowl, have a read through [its entry in the Gall vane types documentation](/urbit-os/kernel/gall/data-types#bowl).

One important thing to note is that the bowl is only repopulated when there's a new Arvo event. If a local agent or web client were to send multiple messages to your agent at the same time, these would all arrive in the same event. This means if your agent depended on a unique date-time or entropy to process each message, you could run into problems if your agent doesn't account for this possibility.

## State <a href="#state" id="state"></a>

If you've worked through [Hoon School](/build-on-urbit/hoon-school), you may recall that a core is a cell of \[battery payload]. The battery is the core itself compiled to Nock, and the payload is the subject which it operates on.

For an agent, the payload will at least contain the bowl, the usual Hoon and `/sys/zuse.hoon` standard library functions, and the **state** of the agent. For example, if your agent were for an address book app, it might keep a `+map` of ships to address book entries. It might add entries, delete entries, and modify entries. This address book `+map` would be part of the state stored in the payload.

## Transition function <a href="#transition-function" id="transition-function"></a>

If you recall from the prologue, the whole Arvo operating system works on the basis of a simple transition function `(event, oldState) -> (effects, newState)`. Gall agents also function the same way. Eight of an agent's ten arms produce `(quip card _this)`, a cell of:

* **Head**: A list of effects called `$card`s (which we'll discuss later).
* **Tail**: A new agent core, possibly with a modified payload.

It goes something like this:

1. An event is routed to Gall.
2. Gall calls the appropriate arm of the agent, depending on the kind of event.
3. That arm processes the event, returning a list `$card`s to be sent off, and the agent core itself with a modified state in the payload.
4. Gall sends the `$card`s off and saves the modified agent core.
5. Rinse and repeat.

## Virtualization <a href="#virtualization" id="virtualization"></a>

When a crash occurs in the kernel, the system usually aborts the computation and discards the event as though it never happened. Gall on the other hand virtualizes all its agents, so this doesn't happen. Instead, when a crash occurs in an agent, Gall intercepts the crash and takes appropriate action depending on the kind of event that caused it. For example, if a poke from another ship caused a crash in the `+on-poke` arm, Gall will respond to the poke with a "nack", a negative acknowledgement, telling the original ship the poke was rejected.

What this means is that you can intentionally design your agent to crash in cases it can't handle. For example, if a poke comes in with an unexpected `$mark`, it crashes. If a permission check fails, it crashes. This is quite different to most programs written in procedural languages, which must handle all exceptions to avoid crashing.

## Example <a href="#example" id="example"></a>

Here's about the simplest valid Gall agent:

```hoon
|_  =bowl:gall
++  on-init   `..on-init
++  on-save   !>(~)
++  on-load   |=(vase `..on-init)
++  on-poke   |=(cage !!)
++  on-watch  |=(path !!)
++  on-leave  |=(path `..on-init)
++  on-peek   |=(path ~)
++  on-agent  |=([wire sign:agent:gall] !!)
++  on-arvo   |=([wire sign-arvo] !!)
++  on-fail   |=([term tang] `..on-init)
--
```

This is just a dummy agent that does absolutely nothing - it has no state and rejects all messages by crashing. Typically we'd cast this to an `+agent:gall`, but in this instance we won't so it's easier to examine its structure in the dojo. We'll get to what each of the arms do later. For now, we'll just consider a few particular points.

Firstly, note its structure - it's a door (created with `|_`) with a sample of `$bowl:gall` and the ten arms described earlier. The `=bowl:gall` syntax simply means `bowl=bowl:gall` (see [`$=` irregular syntax](/hoon/irregular#buctis)).

Secondly, you'll notice some of the arms return:

```hoon
`..on-init
```

A backtick at the beginning is an irregular syntax meaning "prepend with null", so for example, in the dojo:

```
> `50
[~ 50]
```

The next part has `..on-init`, which means "the subject of the `+on-init` arm". The subject of the `+on-init` arm is our whole agent. In the [transition function](#transition-function) section we mentioned that most arms return a list of effects called `$card`s and a new agent core. Since an empty list is `~`, we've created a cell that fits that description.

Let's examine our agent. In the Dojo of a fake ship, mount the `%base` desk with `|mount %base`. On the Unix side, navigate to `/path/to/fake/ship/base`, and save the above agent in the `/app` directory as `skeleton.hoon`. Back in the Dojo, commit the file to the desk with `|commit %base`.

For the moment we won't install our `%skeleton` agent. Instead, we'll use the `-build-file` thread to build it and save it in the dojo's subject so we can have a look. Run the following in the dojo:

```
> =skeleton -build-file %/app/skeleton/hoon
```

Now, let's have a look:

```
> skeleton
< 10.fxw
  [   bowl
    [ [our=@p src=@p dap=@tas sap=/]
      [ wex=nlr([p=[wire=/ ship=@p term=@tas] q=[acked=?(%.y %.n) path=/]])
        sup=nlr([p=it(/) q=[p=@p q=/]])
        sky=nlr([p=/ q=nlr([key=@ud val=[p=@da q=?([%.y p=[p=@tas q=*]] [%.n p=@uvI])]])])
      ]
      act=@ud
      eny=@uvJ
      now=@da
      byk=[p=@p q=@tas r=?([%da p=@da] [%tas p=@tas] [%ud p=@ud] [%uv p=@uv])]
    ]
    <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
  ]
>
```

The Dojo pretty-prints cores with a format of `number-of-arms.hash`. You can see the head of `%skeleton` is `10.fxw` - that's the battery of the core, our 10-arm agent. If we try printing the head of `%skeleton` we'll see it's a whole lot of compiled Nock:

```
> -.skeleton
[ [ 11
    [ 1.953.460.339
      1
      [ 7.368.801
        7.957.707.045.546.060.659
        1.852.796.776
        0
      ]
      [7 15]
      7
      34
    ]
...(truncated for brevity)...
```

The battery's not too important, it's not something we'd ever touch in practice. Instead, let's have a look at the core's payload by printing the tail of `%skeleton`. We'll see its head is the `$bowl:gall` sample we specified, and then the tail is just all the usual standard library functions:

```
> +.skeleton
[   bowl
  [ [our=~zod src=~zod dap=%$ sap=/]
    [wex=~ sup=~ sky=~]
    act=0
    eny=0v0
    now=~2000.1.1
    byk=[p=~zod q=%$ r=[%uv p=0v0]]
  ]
  <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
]
```

Currently `%skeleton` has no state, but of course in practice you'd want to store some actual data. We'll add `foo=42` as our state with the `=+` rune at the beginning of our agent:

```hoon
=+  foo=42
|_  =bowl:gall
++  on-init   `..on-init
++  on-save   !>(~)
++  on-load   |=(vase `..on-init)
++  on-poke   |=(cage !!)
++  on-watch  |=(path !!)
++  on-leave  |=(path `..on-init)
++  on-peek   |=(path ~)
++  on-agent  |=([wire sign:agent:gall] !!)
++  on-arvo   |=([wire sign-arvo] !!)
++  on-fail   |=([term tang] `..on-init)
--
```

Save the modified `skeleton.hoon` in `/app` on the `%base` desk like before, and run `|commit %base` again in the dojo. Then, rebuild it with the same `-build-file` command as before:

```
> =skeleton -build-file %/app/skeleton/hoon
```

If we again examine our agent core's payload by looking at the tail of `%skeleton`, we'll see `foo=42` is now included:

```
> +.skeleton
[   bowl
  [ [our=~zod src=~zod dap=%$ sap=/]
    [wex=~ sup=~ sky=~]
    act=0
    eny=0v0
    now=~2000.1.1
    byk=[p=~zod q=%$ r=[%uv p=0v0]]
  ]
  foo=42
  <15.eah 40.ihi 14.tdo 54.xjm 77.vsv 236.zqw 51.njr 139.oyl 33.uof 1.pnw %138>
]
```

## Summary <a href="#summary" id="summary"></a>

* A Gall agent is a door with exactly ten specific arms and a sample of `$bowl:gall`.
* Each of the ten arms handle different kinds of events - Gall calls the appropriate arm for the kind of event it receives.
* The ten arms fit roughly into five categories:
  * State management.
  * Request handlers.
  * Response handlers.
  * Scry handler.
  * Failure handler.
* The state of an agent (the data it's storing) lives in the core's payload.
* Most arms produce a list of effects called `$card`s, and a new agent core with a modified state in its payload.

## Exercises <a href="#exercises" id="exercises"></a>

* Run through the [example](#example) yourself on a fake ship if you've not done so already.
* Have a look at the [`$bowl` entry in the Gall data types documentation](/urbit-os/kernel/gall/data-types#bowl) if you've not done so already.


# 3. Imports and Aliases

Importing libraries, virtual arms, core composition, and boilerplate patterns.

In the last lesson we looked at the most basic aspects of a Gall agent's structure. Before we get into the different agent arms in detail, there's some boilerplate to cover that makes life easier when writing Gall agents.

## Useful libraries <a href="#useful-libraries" id="useful-libraries"></a>

There are a couple of libraries that you'll very likely use in every agent you write. These are [`/lib/default-agent.hoon`](#default-agent) and [`/lib/dbug.hoon`](#dbug). In brief, `/lib/default-agent.hoon` provides simple default behaviours for each agent arm, and `/lib/dbug.hoon` lets you inspect the state and bowl of an agent from the dojo, for debugging purposes. Every example agent we look at from here on out will make use of both libraries.

Let's look at each in more detail:

### `/lib/default-agent.hoon` <a href="#default-agent" id="default-agent"></a>

The default-agent library contains a basic agent with sane default behaviours for each arm. In some cases it just crashes and prints an error message to the terminal, and in others it succeeds but does nothing. It has two primary uses:

* For any agent arms you don't need, you can just have them call the matching function in `/lib/default-agent.hoon`, rather than having to manually handle events on those arms.
* A common pattern in an agent is to switch on the input of an arm with [wutlus](/hoon/rune/wut#wutlus) (`?+`) runes or maybe [wutcol](/hoon/rune/wut#wutcol) (`?:`) runes. For any unexpected input, you can just pass it to the relevant arm of default-agent rather than handling it manually.

The default-agent library lives in `/lib/default-agent/hoon` of the `%base` desk, and you would typically include a copy in any new desk you created. It's imported at the beginning of an agent with the [faslus](/hoon/rune/fas#faslus) (`/+`) rune.

The library is a wet gate which takes two arguments: an `+agent` and a `$help`. The first is your agent core itself, and the second is a `?`. If `$help` is `%.y` (equivalently, `%&`), it will crash in all cases. If `$help` is `%.n` (equivalently, `%|`), it will use its defaults. You would almost always have `$help` as `%.n`.

The wet gate returns an `+agent:gall` door with a sample of `$bowl:gall` - a typical agent core. Usually you would define an alias for it in a virtual arm ([explained below](#virtual-arms)) so it's simple to call.

### `/lib/dbug.hoon` <a href="#dbug" id="dbug"></a>

The dbug library lets you inspect the state and `$bowl` of your agent from the dojo. It includes an `+agent:dbug` function which wraps your whole `+agent:gall` door, adding its extra debugging functionality while transparently passing events to your agent for handling like usual.

To use it, you just import `/lib/dbug.hoon` with a [faslus](/hoon/rune/fas#faslus) (`/+`) rune at the beginning, then add the following line directly before the door of your agent:

```hoon
%-  agent:dbug
```

With that done, you can poke your agent with the `+dbug` generator from the Dojo and it will pretty-print its state, like:

```
> :your-agent +dbug
```

The generator also has a few useful optional arguments:

* `%bowl`: Print the agent's bowl.
* `[%state 'some code']`: Evaluate some code with the agent's state as its subject and print the result. The most common case is `[%state %some-face]`, which will print the contents of the wing with the given face.
* `[%incoming ...]`: Print details of the matching incoming subscription, one of:
  * `[%incoming %ship ~some-ship]`
  * `[%incoming %path /part/of/path]`
  * `[%incoming %wire /part/of/wire]`
* `[%outgoing ...]`: Print details of the matching outgoing subscription, one of:
  * `[%outgoing %ship ~some-ship]`
  * `[%outgoing %path /part/of/path]`
  * `[%outgoing %wire /part/of/wire]`
  * `[outgoing %term %agent-name]`

By default it will retrieve your agent's state by using its `+on-save` arm, but if your app implements a scry endpoint with a path of `/x/dbug/state`, it will use that instead.

We haven't yet covered some of the concepts described here, so don't worry if you don't fully understand `/lib/dbug.hoon`'s functionality - you can refer back here later.

## Virtual arms <a href="#virtual-arms" id="virtual-arms"></a>

An agent core must have exactly ten arms. However, there's a special kind of "virtual arm" that can be added without actually increasing the core's arm count, since it really just adds code to the other arms in the core. A virtual arm is created with the [lustar](/hoon/rune/lus#lustar) (`+*`) rune, and its purpose is to define "deferred expressions". It takes a list of pairs of names and Hoon expressions. When compiled, the deferred expressions defined in the virtual arm are implicitly inserted at the beginning of every other arm of the core, so they all have access to them. Each time a name in a `+*` is called, the associated Hoon is evaluated in its place, similar to lazy evaluation except it is re-evaluated whenever needed. See the `=*` [tistar](/hoon/rune/tis#tistar) reference for more information on deferred expressions.

A virtual arm in an agent often looks something like this:

```hoon
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
```

`.this` and `.def` are the deferred expressions, and next to each one is the Hoon expression it evaluates whenever called. Notice that unlike most things that take *n* arguments, a virtual arm is not terminated with a `==`. You can define as many aliases as you like. The two in this example are conventional ones you'd use in most agents you write. Their purposes are:

```hoon
this  .
```

Rather than having to return `+on-init` like we did in the last lesson, instead our arms can just refer to `.this` whenever modifying or returning the agent core.

```
%=  this
  foo  bar
==
```

```hoon
def  ~(. (default-agent this %.n) bowl)
```

This sets up the default-agent library we [described above](#default-agent), so you can easily call its arms like `+on-poke:def`, `+on-agent:def`, etc.

## Additional cores <a href="#additional-cores" id="additional-cores"></a>

While Gall expects a single ten-arm agent core, it's possible to include additional cores by composing them into the subject of the agent core itself. The contents of these cores will then be available to arms of the agent core.

Usually to compose cores in this way, you'd have to do something like insert `=>` [tisgar](/hoon/rune/tis#tisgar) runes in between them. However, Clay's build system implicitly composes everything in a file by wrapping it in a `=~` [tissig](/hoon/rune/tis#tissig) expression, which means you can just butt separate cores up against one another and they'll all still get composed.

You can add as many extra cores as you'd like before the agent core, but typically you'd just add one containing type definitions for the agent's state, as well as any other useful structures. We'll look at the state in more detail in the next lesson.

## Example <a href="#example" id="example"></a>

Here's the `/app/skeleton.hoon` dummy agent from the previous lesson, modified with the concepts discussed here:

<details>

<summary>/app/skeleton (modified)</summary>

```hoon
/+  default-agent, dbug
|%
+$  card  card:agent:gall
--
%-  agent:dbug
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
++  on-init
  ^-  (quip card _this)
  `this
++  on-save   on-save:def
++  on-load   on-load:def
++  on-poke   on-poke:def
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

The first line uses the faslus (`/+`) Ford rune to import `/lib/default-agent.hoon` and `/lib/dbug.hoon`, building them and loading them into the subject of our agent so they're available for use. You can read more about Ford runes in the [fas section of the rune documentation](/hoon/rune/fas).

Next, we've added an extra core. Notice how it's not explicitly composed, since Clay's build system will do that for us. In this case we've just added a single `$card` arm, which makes it simpler to reference the `$card:agent:gall` type.

After that core, we call `+agent:dbug` with our whole agent core as its argument. This allows us to use the `/lib/dbug.hoon` features described earlier.

Inside our agent door, we've added an extra virtual arm and defined a couple deferred expressions:

```hoon
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
```

In most of the arms, you see we've been able to replace the dummy code with simple calls to the corresponding arms of default-agent, which we set up as a deferred expression named `def` in the virtual arm. We've also replaced the old `..on-init` with our deferred expression named `.this` in the `+on-init` arm as an example, it makes things a bit simpler.

You can save the code above in `/app/skeleton.hoon` of your `%base` desk like before and `|commit %base` in the dojo. Additionally, you can start the agent so we can try out `+dbug`. To start it, run the following in the dojo:

```
> |rein %base [& %skeleton]
```

For details of using the `|rein` generator, see the [Dojo Tools](/user-manual/os/dojo-tools#rein) documentation.

Now our agent should be running, so let's try out `+dbug`. In the dojo, let's try poking our agent with the `+dbug` generator:

```
>   ~
> :skeleton +dbug
>=
```

It just printed out `~`. Our dummy `%skeleton` agent doesn't have any state defined, so it's printing out null as a result. Let's try printing the `$bowl` instead:

```
>   [ [our=~zod src=~zod dap=%skeleton sap=/gall/dojo]
  [wex=~ sup=~ sky=~]
  act=5
    eny
  0v209.tg795.bc2e8.uja0d.11eq9.qp3b3.mlttd.gmf09.q7ro3.6unfh.16jiu.m9lh9.6jlt8.4f847.f0qfh.up08t.3h4l2.qm39h.r3qdd.k1r11.bja8l
  now=~2024.5.9..13.28.24..e20e
  byk=[p=~zod q=%base r=[%da p=~2024.5.9..12.02.22..f99b]]
]
> :skeleton +dbug %bowl
>=
```

We'll use `+dbug` more throughout the guide, but hopefully you should now have an idea of its basic usage.

## Summary <a href="#summary" id="summary"></a>

The key takeaways are:

* Libraries are imported with `/+`.
* `/lib/default-agent.hoon` is a library that provides default behaviors for Gall agent arms.
* `/lib/dbug.hoon` is a library that lets you inspect the state and `$bowl` of an agent from the dojo, with the `+dbug` generator.
* Convenient deferred expressions for Hoon expressions can be defined in a virtual arm with the [lustar](/hoon/rune/lus#lustar) (`+*`) rune.
* `this` is a conventional deferred expression name for the agent core itself.
* `def` is a conventional deferred expression name for accessing arms in the `/lib/default-agent.hoon` library.
* Extra cores can be composed into the subject of the agent core. The composition is done implicitly by the build system. Typically we'd include one extra core that defines types for our agent's state and maybe other useful types as well.

## Exercises <a href="#exercises" id="exercises"></a>

* Run through the [example](#example) yourself on a fake ship if you've not done so already.
* Have a read through the [Ford rune documentation](/hoon/rune/fas) for details about importing libraries, structures and other things.
* Try the `+dbug` generator out on some other agents, like `:azimuth +dbug`, `:spider +dbug`, `:hood +dbug`, etc, and try some of its options [described above](#dbug).
* Have a quick look over the source of the `/lib/default-agent.hoon` library, located at `/lib/default-agent.hoon` in the `%base` desk. We've not yet covered what the different arms do but it's still useful to get a general idea, and you'll likely want to refer back to it later.


# 4. Lifecycle

Agent lifecycle management: state initialization, import and export, versioned state types, and upgrade procedures.

In the last lesson we looked at a couple of useful things used as boilerplate in most agents. Now we're going to get into the guts of how agents work, and start looking at what the agent arms do. The first thing we'll look at is the agent's state, and the three arms for managing it: `+on-init`, `+on-save`, and `+on-load`. These arms handle what we call an agent's "lifecycle".

## Lifecycle <a href="#lifecycle" id="lifecycle"></a>

An agent's lifecycle starts when it's first installed. At this point, the agent's `+on-init` arm is called. This is the *only* time `+on-init` is ever called - its purpose is just to initialize the agent. The `+on-init` arm might be very simple and just set an initial value for the state, or even do nothing at all and return the agent core exactly as-is. It may also be more complicated, and perform some [scries](/urbit-os/kernel/arvo/scry) to obtain extra data or check that another agent is also installed. It might send off some `$card`s to other agents or vanes to do things like load data in to the `%settings` agent, bind an Eyre endpoint, or anything else. It all depends on the needs of your particular application. If `+on-init` fails for whatever reason, the agent installation will fail and be aborted.

Once initialized, an agent will just go on doing its thing - processing events, updating its state, producing effects, etc. At some point, you'll likely want to push an update for your agent. Maybe it's a bug fix, maybe you want to add extra features. Whatever the reason, you need to change the source code of your agent, so you commit a modified version of the file to Clay. When the commit completes, Gall updates the app as follows:

* The agent's `+on-save` arm is called, which packs the agent's state in a `$vase` and exports it.
* The new version of the `+agent` is built and loaded into Gall.
* The previously exported `$vase` is passed to the `+on-load` arm of the newly built agent. The `+on-load` arm will process it, convert it to the new version of the state if necessary, and load it back into the state of the agent.

A `$vase` is just a cell of \[type-of-the-noun the-noun]. Most data an agent sends or receives will be encapsulated in a vase. A `$vase` is made with the [zapgar](/hoon/rune/zap#zapgar) (`!>`) rune like `!>(some-data)`, and unpacked with the [zapgal](/hoon/rune/zap#zapgal) (`!<`) rune like `!<(type-to-extract vase)`. Have a read through the [`$vase` section of the type reference for details](/build-on-urbit/app-school/types#vase).

We'll look at the three arms described here in a little more detail, but first we need to touch on the state itself.

## Versioned state type <a href="#versioned-state-type" id="versioned-state-type"></a>

In the previous lesson we introduced the idea of composing additional cores into the subject of the agent core. Here we'll look at using such a core to define the type of the agent's state. In principle, we could make it as simple as this:

```hoon
|%
+$  my-state-type  @ud
--
```

However, when you update your agent as described in the [lifecycle](#lifecycle) section, you may want to change the type of the state itself. This means `+on-load` might find different versions of the state in the `$vase` it receives, and it might not be able to distinguish between them.

For example, if you were creating an agent for a To-Do task management app, your tasks might initially have a `?(%todo %done)` union to specify whether they're complete or not. Something like:

```hoon
(map task=@t status=?(%todo %done))
```

At some point, you might want to add a third status to represent "in progress", which might involve changing `.status` like:

```hoon
(map title=@t status=?(%todo %done %work))
```

The conventional way to keep this managable and reliably differentiate possible state types is to have "versioned states". The first version of the state would typically be called `$state-0`, and its head would be tagged with `%0`. Then, when you change the state's type in an update, you'd add a new structure called `$state-1` and tag its head with `%1`. The next would then be `$state-2`, and so on.

In addition to each of those individual state versions, you'd also define a structure called `$versioned-state`, which just contains a union of all the possible states. This way, the `$vase` `+on-load` receives can be unpacked to a `$versioned-state` type, and then a [wuthep](/hoon/rune/wut#wuthep) (`?-`) expression can switch on the head (`%0`, `%1`, `%2`, etc) and process each one appropriately.

For example, your state definition core might initially look like:

```hoon
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 tasks=(map title=@t status=?(%todo %done))]
--
```

When you later update your agent with a new state version, you'd change it to:

```hoon
|%
+$  versioned-state
  $%  state-0
      state-1
  ==
+$  state-0  [%0 tasks=(map title=@t status=?(%todo %done))]
+$  state-1  [%1 tasks=(map title=@t status=?(%todo %done %work))]
--
```

Another reason for versioning the state type is that there may be cases where the state type doesn't change, but you still want to apply special transition logic for an old state during upgrade. For example, you may need to reprocess the data for a new feature or to fix a bug.

## Adding the state <a href="#adding-the-state" id="adding-the-state"></a>

Along with a core defining the type of the state, we also need to actually add it to the subject of the core. The conventional way to do this is by adding the following immediately before the agent core itself:

```hoon
=|  state-0
=*  state  -
```

The first line bunts (produces the default value) of the state type we defined in the previous core, and adds it to the head of the subject *without a face*. The next line uses [tistar](/hoon/rune/tis#tistar) to give it the name of "state". You might wonder why we don't just give it a face when we bunt it and skip the tistar part. If we did that, we'd have to refer to `.tasks` as `.tasks.state`. With tistar, we can just reference `.tasks` while also being able to reference the whole `.state` when necessary.

Note that adding the state like this only happens when the agent is built. From then on the arms of our agent will just modify it.

## State management arms <a href="#state-management-arms" id="state-management-arms"></a>

We've described the basic lifecycle process and the purpose of each state management arm. Now let's look at each arm in detail:

### `+on-init` <a href="#on-init" id="on-init"></a>

This arm takes no argument, and produces a `(quip card _this)`. It's called exactly once, when the agent is first installed. Its purpose is to initialize the agent.

`(quip a b)` is equivalent to `[(list a) b]`, see the [types reference](/build-on-urbit/app-school/types#quip) for details.

A `$card` is a message to another agent or vane. We'll discuss `$card`s in detail later.

`.this` is our agent core, which we give the `.this` alias in the virtual arm described in the previous lesson. The underscore at the beginning is the irregular syntax for the [buccab](/hoon/rune/buc#buccab) (`$_`) rune. Buccab is like an inverted bunt - instead of producing the default value of a type, instead it produces the type of some value. So `_this` means "the type of `.this`" - the type of our agent core.

Recall that in the last lesson, we said that most arms return a cell of \[effects new-agent-core]. That's exactly what `(quip card _this)` is.

### `+on-save` <a href="#on-save" id="on-save"></a>

This arm takes no argument, and produces a `$vase`. Its purpose is to export the state of an agent - the state is packed into the `$vase` it produces. The main time it's called is when an agent is upgraded. When that happens, the agent's state is exported with `+on-save`, the new version of the agent is compiled and loaded, and then the state is imported back into the new version of the agent via the [`+on-load`](#on-load) arm.

As well as the agent upgrade process, `+on-save` is also used when an agent is suspended or an app is uninstalled, so that the state can be restored when it's resumed or reinstalled.

The state is packed in a `$vase` with the [zapgar](/hoon/rune/zap#zapgar) (`!>`) rune, like `!>(state)`.

### `+on-load` <a href="#on-load" id="on-load"></a>

This arm takes a `$vase` and produces a `(quip card _this)`. Its purpose is to import a state previously exported with [`+on-save`](#on-save). Typically you'd have used a [versioned state](#versioned-state-type) as described above, so this arm would test which state version the imported data has, convert data from an old version to the new version if necessary, and load it into the `.state` wing of the subject.

The `$vase` would be unpacked with a [zapgal](/hoon/rune/zap#zapgal) (`!<`) rune, and then typically you'd test its version with a [wuthep](/hoon/rune/wut#wuthep) (`?-`) expression.

## Example <a href="#example" id="example"></a>

Here's a new agent to demonstrate the concepts we've discussed here:

<details>

<summary>Example agent</summary>

```hoon
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 val=@ud]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this(val 42)
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke   on-poke:def
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

Let's break it down and have a look at the new parts we've added. First, the state core:

```hoon
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 val=@ud]
+$  card  card:agent:gall
--
```

In `$state-0` we've defined the structure of our state, which is just a `@ud`. We've tagged the head with a `%0` constant representing the version number, so `+on-load` can easily test the state version. In `$versioned-state` we've created a union and just added our `$state-0` type. We've added an extra `$card` arm as well, just so we can use `$card` as a type, rather than the unwieldy `$card:agent:gall`.

After that core, we have the usual `+agent:dbug` call, and then we have this:

```hoon
=|  state-0
=*  state  -
```

We've just bunted the `$state-0` type, which will produce `[%0 val=0]`, pinning it to the head of the subject. Then, we've use [tistar](/hoon/rune/tis#tistar) (`=*`) to give it a name of `.state`.

Inside our agent core, we have `+on-init`:

```hoon
++  on-init
  ^-  (quip card _this)
  `this(val 42)
```

The `a(b c)` syntax is the irregular form of the [centis](/hoon/rune/cen#centis) (`%=`) rune. You'll likely be familiar with this from recursive functions, where you'll typically call the buc arm of a trap like `$(a b, c d, ...)`. It's the same concept here - we're saying `.this` (our agent core) with `.val` replaced by `42`. Since `+on-init` is only called when the agent is first installed, we're just initializing the state.

Next we have `+on-save`:

```hoon
++  on-save
  ^-  vase
  !>(state)
```

This exports our agent's state, and is called during upgrades, suspensions, etc. We're having it pack the `.state` value in a `$vase`.

Finally, we have `+on-load`:

```hoon
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
```

It takes in the old state in a `$vase`, then unpacks it to the `$versioned-state` type we defined earlier. We test its head for the version, and load it back into the state of our agent if it matches. This test is a bit redundant at this stage since we only have one state version, but you'll soon see the purpose of it.

You can save it as `/app/lifecycle.hoon` in the `%base` desk and `|commit %base`. Then, run `|rein %base [& %lifecycle]` to start it.

Let's try inspecting our state with `+dbug`:

```
>   [%0 val=42]
> :lifecycle +dbug
>=
```

`+dbug` can also dig into the state with the `%state` argument, printing the value of the specified face:

```
>   42
> :lifecycle +dbug [%state %val]
>=
```

Next, we're going to modify our agent and change the structure of the state so we can test out the upgrade process. Here's a modified version, which you can again save in `/app/lifecycle.hoon` and `|commit %base`:

<details>

<summary>/app/lifecycle.hoon</summary>

```hoon
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
      state-1
  ==
+$  state-0  [%0 val=@ud]
+$  state-1  [%1 val=[@ud @ud]]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-1
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this(val [27 32])
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %1  `this(state old)
    %0  `this(state 1+[val.old val.old])
  ==
::
++  on-poke   on-poke:def
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

As soon as you `|commit` it, Gall will immediately export the existing state with `+on-save`, build the new version of the agent, then import the state back in with `+on-load`.

In the state definition core, you'll see we've added a new state version with a different structure:

```hoon
+$  versioned-state
  $%  state-0
      state-1
  ==
+$  state-0  [%0 val=@ud]
+$  state-1  [%1 val=[@ud @ud]]
+$  card  card:agent:gall
--
```

We've also changed the part that adds the state, so it uses the new version instead:

```hoon
=|  state-1
=*  state  -
```

In `+on-init`, we've updated it to initialize the state with a value that fits the new type we've defined:

```hoon
++  on-init
  ^-  (quip card _this)
  `this(val [27 32])
```

`+on-init` won't be called in this case, but if someone were to directly install this new version of the agent, it would be, so we still need to update it.

`+on-save` has been left unchanged, but `+on-load` has been updated like so:

```hoon
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %1  `this(state old)
    %0  `this(state 1+[val.old val.old])
  ==
```

We've updated the `?-` expression with a new case that handles our new state type, and for the old state type we've added a function that converts it to the new type - in this case by duplicating `.val` and changing the head-tag from `%0` to `%1`. This is an extremely simple state type transition function - it would likely be more complicated for an agent with real functionality.

Note: the `a+b` syntax (as in `1+[val.old val.old]`) forms a cell of the constant `%a` and the noun `.b`. The constant may either be an integer or a `@tas`. For example:

```
> foo+'bar'
[%foo 'bar']

> 42+'bar'
[%42 'bar']
```

Let's now use `+dbug` to confirm our state has successfully been updated to the new type:

```
>   [%1 val=[42 42]]
> :lifecycle +dbug
>=
```

## Summary <a href="#summary" id="summary"></a>

* The app lifecycle rougly consists of initialization, state export, upgrade, state import and state version transition.
* This is managed by three arms: `+on-init`, `+on-save` and `+on-load`.
* `+on-init` initializes the agent and is called when it's first installed.
* `+on-save` exports the agent's state and is called during upgrade or when an app is suspended.
* `+on-load` imports an agent's state and is called during upgrade or when an app is unsuspended. It also handles converting data from old state versions to new state versions.
* The type of an agent's state is typically defined in a separate core.
* The state type is typically versioned, with a new type definition for each version of the state.
* The state is initially added by bunting the state type and then naming it `state` with the tistar (`=*`) rune, so its contents can be referenced directly.
* A `$vase` is a cell of \[type-of-the-noun the-noun].
* `(quip a b)` is the same as `[(list a) b]`, and is the \[effects new-agent-core] pair returned by many arms of an agent core.

## Exercises <a href="#exercises" id="exercises"></a>

* Run through the [example](#example) yourself on a fake ship if you've not done so already.
* Have a look at the [`$vase` entry in the type reference](/build-on-urbit/app-school/types#vase).
* Have a look at the [`+quip` entry in the type reference](/build-on-urbit/app-school/types#quip).
* Try modifying the second version of the agent in the [example](#example) section, adding a third state version. Include functions in the wuthep expression in `+on-load` to convert old versions to your new state type.


# 5. Cards

Gall agent effects, requests, responses, and subscription management.

As we previously discussed, most arms of an agent core produce a cell of \[effects new-agent-core], and the type we use for this is typically `(quip card _this)`. We've covered `_this`, but we haven't yet looked at `$card` effects in detail. That's what we'll do here. In explaining `$card`s we'll touch on some concepts relating to the mechanics of pokes, subscriptions and other things we've not yet covered. Don't worry if you don't understand how it all fits together yet, we just want to give you a basic idea of `$card`s so we can then dig into how they work in practice.

## `$card` type <a href="#card-type" id="card-type"></a>

The `$card:agent:gall` type (henceforth just `$card`) has a slightly complex structure, so we'll walk through it step-by-step.

`/sys/lull.hoon` defines a `$card` like so:

```hoon
+$  card  (wind note gift)
```

A `+wind` is defined in `/sys/arvo.hoon` as:

```hoon
++  wind
  |$  [a b]
  $%  [%pass p=wire q=a]
      [%slip p=a]
      [%give p=b]
  ==
```

Gall will not accept a `%slip`, so we can ignore that. A `$card`, then, is one of:

```hoon
[%pass wire note]
[%give gift]
```

We'll consider each separately.

## `%pass` <a href="#pass" id="pass"></a>

```hoon
[%pass wire note]
```

The purpose of a `%pass` card is to send some kind of one-off request, action, task, or what have you, to another agent or vane. A `%pass` card is a request your agent *initiates*. This is in contrast to a [`%give`](#give) card, which is sent in *response* to another agent or vane.

The type of the first field in a `%pass` card is a `$wire`. A `$wire` is just a `(list @ta)`, with a syntax of `/foo/bar/baz`. When you `%pass` something to an agent or vane, the response will come back on the `$wire` you specify here. Your agent can then check the `$wire` and maybe do different things depending on its content. The [`$wire`](/build-on-urbit/app-school/types#wire) type is covered in the [types reference](/build-on-urbit/app-school/types). We'll show how `$wire`s are practically used later on.

The type of the next field is a `$note:agent:gall` (henceforth just `$note`), which `lull.hoon` defines as:

```hoon
+$  note
  $%  [%agent [=ship name=term] =task]
      [%arvo note-arvo]
      [%pyre =tang]
  ::
      [%grow =spur =page]
      [%tomb =case =spur]
      [%cull =case =spur]
  ::
      [%tend =coop =path =page]
      [%germ =coop]
      [%snip =coop]
  ::
      [%keen secret=? spar:ames]
  ==
```

* An `%agent` `$note` is a request to another Gall agent, either local or on a remote ship. The `$ship` and `$name` fields are just the target ship and agent name. The task is the request itself, we'll discuss it separately [below](#task).
* An `%arvo` `$note` is a request to a vane. We'll discuss such requests [below](#note-arvo).
* A `%pyre` `$note` is used to abort an event. It's mostly used internally by `%kiln` (a submodule of `%hood`), it's unlikely you'd use it in your own agent. The `$tang` contains an error message.
* The other notes are for [remote scry](/build-on-urbit/userspace/remote-scry).

### task <a href="#task" id="task"></a>

A `$task:agent:gall` (henceforth just task) is defined in `lull.hoon` as:

```hoon
+$  task
  $%  [%watch =path]
      [%watch-as =mark =path]
      [%leave ~]
      [%poke =cage]
      [%poke-as =mark =cage]
  ==
```

Note a few of these include a `$path` field. The `$path` type is exactly the same as a `$wire` (another `(list @ta)`) with a syntax of `/foo/bar/baz`. The reason for the `$wire`/`$path` distinction is just to indicate their separate purposes. While a `$wire` is for *responses*, a `$path` is for *requests*. The [`$path`](/build-on-urbit/app-school/types#path) type is also covered in the [types reference](/build-on-urbit/app-school/types).

The kinds of tasks can be divided into two categories:

#### Subscriptions

`%watch`, `%watch-as` and `%leave` all pertain to subscriptions.

* `%watch`: A request to subscribe to the specified `$path`. Once subscribed, your agent will receive any updates the other agent sends out on that `$path`. You can subscribe more than once to the same `$path`, but each subscription must have a separate `$wire` specified at the beginning of the [`%pass` card](#pass).
* `%watch-as`: This is the same as `%watch`, except Gall will convert updates to the given `$mark` before delivering them to your agent.
* `%leave`: Unsubscribe. The subscription to cancel is determined by the `$wire` at the beginning of the [`$pass` card](#pass) rather than the subscription `$path`, so its argument is just `~`.

**Examples**

![](https://media.urbit.org/guides/core/app-school/sub-cards.svg)

#### Pokes

Pokes are requests, actions, or just some data which you send to another agent. Unlike subscriptions, these are just one-off messages.

A `%poke` contains a `$cage` of some data. A `$cage` is a cell of `[mark vase]`. The `$mark` is just a `@tas` like `%foo`, and corresponds to a mark file in the `/mar` directory. We'll cover `$mark`s in greater detail later. The `$vase` contains the actual data you're sending.

The `%poke-as` task is the same as `%poke` except Gall will convert the `$mark` in the `$cage` to the `$mark` you specify before sending it off.

**Examples**

![](https://media.urbit.org/guides/core/app-school/poke-cards.svg)

### `$note-arvo` <a href="#note-arvo" id="note-arvo"></a>

A `$note-arvo` is defined in `lull.hoon` like so:

```hoon
+$  note-arvo
  $~  [%b %wake ~]
  $%  [%a task:ames]
      [%b task:behn]
      [%c task:clay]
      [%d task:dill]
      [%e task:eyre]
      [%g task:gall]
      [%i task:iris]
      [%j task:jael]
      [%k task:khan]
      [%l task:lick]
      [%$ %whiz ~]
      [@tas %meta vase]
  ==
```

The letter at the beginning corresponds to the vane - `%b` for Behn, `%c` for Clay, etc. After the vane letter comes the task. Each vane has an API with a set of tasks that it will accept, and are defined in each vane's section of `lull.hoon`. Each vane's tasks are documented on the API Reference page of its section in the [Arvo documentation](/urbit-os/kernel/arvo).

#### Examples

![](https://media.urbit.org/guides/core/app-school/arvo-cards.svg)

## `%give` <a href="#give" id="give"></a>

```hoon
[%give gift]
```

The purpose of a `%give` card is to respond to a request made by another agent or vane. More specifically, it's either for acknowledging a request, or for sending out updates to subscribers. This is in contrast to a [`%pass`](#give) card, which is essentially unsolicited.

A `%give` card contains a `$gift:agent:gall` (henceforth just gift), which is defined in `lull.hoon` as:

```hoon
+$  gift
  $%  [%fact paths=(list path) =cage]
      [%kick paths=(list path) ship=(unit ship)]
      [%watch-ack p=(unit tang)]
      [%poke-ack p=(unit tang)]
  ==
```

These can be divided into two categories:

### Acknowledgements <a href="#acknowledgements" id="acknowledgements"></a>

`%watch-ack` is sent in response to a `%watch` or `%watch-as` request, and `%poke-ack` is sent in response to a `%poke` or `%poke-as` request. If the `(unit tang)` is null, it's an ack - a positive acknowledgement. If the `(unit tang)` is non-null, it's a nack - a negative acknowledgement, and the `$tang` contains an error message. Gall automatically sends a nack with a stack trace if your agent crashes while processing the request, and automatically sends an ack if it does not. Therefore, you would not explicitly produce a `%watch-ack` or `%poke-ack` gift.

#### Examples

![](https://media.urbit.org/guides/core/app-school/ack-cards.svg)

### Subscriptions <a href="#subscriptions" id="subscriptions"></a>

`%fact` and `%kick` are both sent out to existing subscribers - entities that have previously `%watch`ed a path on your ship.

A `%kick` gift takes a list of subscription `$path`s and a `(unit ship)`, which is the ship to kick from those paths. If the `+unit` is null, all subscribers are kicked from the specified paths. Note that sometimes Gall can produce `%kick` gifts without your agent explicitly sending a card, due to networking conditions.

`%fact`s are how updates are sent out to subscribers. The `paths` field is a list of subscription paths - all subscribers of the specified `$path`s will receive the `%fact`. The `$cage` is the data itself - a cell of a `$mark` and a `$vase`.

#### Examples

![](https://media.urbit.org/guides/core/app-school/gift-cards.svg)

## Summary <a href="#summary" id="summary"></a>

Here's a diagram that summarizes the different kinds of `$card`s:

![](https://media.urbit.org/guides/core/app-school/card-diagram.svg)

## Exercises <a href="#exercises" id="exercises"></a>

* Have a read of the [`$wire`](/build-on-urbit/app-school/types#wire) and [`$path`](/build-on-urbit/app-school/types#path) entries in the type reference.


# 6. Pokes

Handling one-off messages (pokes) in Gall agents.

In this lesson we'll look at sending and receiving one-off messages called `%poke`s. We'll look at the `+on-poke` agent arm which handles incoming pokes. We'll also introduce the `+on-agent` arm, and look at the one kind of response it can take - a `%poke-ack`.

## Receiving a poke <a href="#receiving-a-poke" id="receiving-a-poke"></a>

Whenever something tries to poke your agent, Gall calls your agent's `+on-poke` arm and give it the `$cage` from the poke as its sample. The `+on-poke` arm will produce a `(quip card _this)`. Here's how it would typically begin:

```hoon
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ...
```

The sample of the gate is usually specified as a cell of `$mark` and `$vase` rather than just `$cage`, simply because it's easier to work with.

Typically, you'd first test the `$mark` with something like a [wutlus](/hoon/rune/wut#wutlus) `?+` expression, passing unexpected `$mark`s to default-agent, which just crashes. We'll look at custom `$mark`s in a subsequent lesson, but the basic pattern looks like:

```hoon
?+  mark  (on-poke:def mark vase)
  %noun            ...
  %something-else  ...
  ...
==
```

After testing the `$mark`, you'd usually extract the `$vase` to the expected type, and then apply whatever logic you need. For example:

```hoon
=/  action  !<(some-type vase)
?-  -.action
  %foo  ...
  %bar  ...
  ...
==
```

Your agent will then produce a list of `$card`s to be sent off and a new, modified state, as appropriate. We'll go into subscriptions in the next lesson, but just to give you an idea of a typical pattern: An agent for a chat app might take new messages as pokes, add them to the list of messages in its state, and send out the new messages to subscribed chat participants as gifts.

As discussed in the previous lesson, Gall will automatically send a `%poke-ack` gift back to wherever the poke came from. The `%poke-ack` will be a nack if your agent crashed while processing the poke, and an ack otherwise. If it's a nack, the `$tang` in the `%poke-ack` will contain a stack trace of the crash.

As a result, you do not need to explicitly send a `%poke-ack`. Instead, you would design your agent to handle only what you expect and crash in all other cases. You can crash by passing the `$cage` to default-agent, or just with a `!!`. In the latter case, if you want to add an error message to the stack trace, you can do so like:

```hoon
~|  "some error message"
!!
```

This will produce a trace that looks something like:

```
/sys/vane/gall/hoon:<[1.372 9].[1.372 37]>
/app/pokeme/hoon:<[31 3].[43 5]>
/app/pokeme/hoon:<[32 3].[43 5]>
/app/pokeme/hoon:<[34 5].[42 7]>
/app/pokeme/hoon:<[35 5].[42 7]>
/app/pokeme/hoon:<[38 7].[41 27]>
/app/pokeme/hoon:<[39 9].[40 11]>
"some error message"
/app/pokeme/hoon:<[40 9].[40 11]>
```

Note that the `$tang` in the nack is just for debugging purposes, you should not try to pass actual data by encoding it in the nack `$tang`.

## Sending a poke <a href="#sending-a-poke" id="sending-a-poke"></a>

An agent can send pokes to other agents by producing [`%poke` `$card`s](/build-on-urbit/app-school/5-cards#pokes). Any agent arm apart from `+on-peek` and `+on-save` can produce such `$card`s. The arms would typically produce the `(quip card _this)` like so:

```hoon
:_  this
:~  [%pass /some/wire %agent [~target-ship %target-agent] %poke %some-mark !>('some data')]
==
```

The [colcab](/hoon/rune/col#colcab) (`:_`) rune makes an inverted cell, it's just `:-` but with the head and tail swapped. We use colcab to produce the `(quip card _this)` because the list of cards is "heavier" here than the new agent core expression (`.this`), so it makes it more readable.

### Receiving the `%poke-ack` <a href="#receiving-the-poke-ack" id="receiving-the-poke-ack"></a>

The pokes will be processed by their targets [as described in the previous section](#receiving-a-poke), and they'll `%give` back a `%poke-ack` on the `$wire` you specified (`/some/wire` in the previous example). When Gall gets the `%poke-ack` back, it will call the `+on-agent` arm of your agent, with the `$wire` it came in on and the `%poke-ack` itself in a `sign:agent:gall`. Your `+on-agent` arm would therefore begin like so:

```hoon
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ...
```

A `$sign:agent:gall` (henceforth just `$sign`) is defined in `/sys/lull.hoon` as:

```hoon
+$  sign
  $%  [%poke-ack p=(unit tang)]
      [%watch-ack p=(unit tang)]
      [%fact =cage]
      [%kick ~]
  ==
```

It's basically the same as a [gift](/build-on-urbit/app-school/5-cards#give), but incoming instead of outgoing.

The simplest way to handle a `%poke-ack` by passing it to default-agent's `+on-agent` arm, which will just print an error message to the terminal if it's a nack, and otherwise do nothing. Sometimes you'll want your agent to do something different depending on whether the poke failed or succeeded (and therefore whether it's a nack or an ack).

You should always route on wire before sign, never sign before wire. You might do something like:

```hoon
?+  wire  (on-agent:def wire sign)
  [%some %wire ~]  ...
  ...
==
```

After that, you'll need to see what kind of `$sign` it is:

```hoon
?+  -.sign  (on-agent:def wire sign)
  %poke-ack  ...
  ...
```

Then, you can tell whether it's an ack or a nack by testing whether the `(unit tang)` in the `%poke-ack` is null:

```hoon
?~  p.sign
  ...(what to do if the poke succeeded)...
...(what to do if the poke failed)...
```

Finally, you can produce the `(quip card _this)`.

## Example <a href="#example" id="example"></a>

We're going to look at a couple of agents to demonstrate both sending and receiving pokes. Here's the first, an agent that receives pokes:

<details>

<summary>/app/pokeme.hoon</summary>

```hoon
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 val=@ud]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %noun
    =/  action  !<(?(%inc %dec) vase)
    ?-    action
      %inc  `this(val +(val))
    ::
        %dec
      ?:  =(0 val)
        ~|  "Can't decrement - already zero!"
        !!
      `this(val (dec val))
    ==
  ==
::
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

This is a very simple agent that just has `.val`, a number, in its state. It will take pokes that either increment or decrement `.val`. Here's its `+on-poke` arm:

```hoon
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %noun
    =/  action  !<(?(%inc %dec) vase)
    ?-    action
        %inc  `this(val +(val))
        %dec
      ?:  =(0 val)
        ~|  "Can't decrement - already zero!"
        !!
      `this(val (dec val))
    ==
  ==
```

It only expects pokes with a `%noun` mark, and passes all others to `+on-poke:def`, which just crashes. For `%noun` pokes, it expects to receive either `%inc` or `%dec` in the `$vase`. If it's `%inc`, it produces a new `.this` with `.val` incremented. If it's `%dec`, it produces `.this` with `.val` decremented, or crashes if `.val` is already zero.

Let's try it out. Save the agent above as `/app/pokeme.hoon` in the `%base` desk and `|commit %base`. Then, start it up with `|rein %base [& %pokeme]`. We can check its initial state with `+dbug`:

```
>   0
> :pokeme +dbug [%state %val]
>=
```

Next, we'll try poking it. The Dojo lets you poke agents with the following syntax:

```
:agent-name &some-mark ['some' 'noun']
```

If the `$mark` part is omitted, it'll just default to `%noun`. Since our agent only takes a `%noun` mark, we can skip that. The rest will be packed in a `$vase` by the Dojo and delivered as a poke, so we can do:

```
> :pokeme %inc
>=
```

If we now look at the state with `+dbug`, we'll see the poke was successful and it's been incremented:

```
>   1
> :pokeme +dbug [%state %val]
>=
```

Let's try decrement:

```
> :pokeme %dec
>=
>   0
> :pokeme +dbug [%state %val]
>=
```

As you can see, it's back at zero. If we try again, we'll see it fails, and the Dojo will print the `$tang` in the `%poke-ack` nack:

```
> :pokeme %dec
/sys/vane/gall/hoon:<[1.828 9].[1.828 37]>
/app/pokeme/hoon:<[35 3].[48 5]>
/app/pokeme/hoon:<[36 3].[48 5]>
/app/pokeme/hoon:<[38 5].[47 7]>
/app/pokeme/hoon:<[39 5].[47 7]>
/app/pokeme/hoon:<[43 7].[46 27]>
/app/pokeme/hoon:<[44 9].[45 11]>
"Can't decrement - already zero!"
/app/pokeme/hoon:<[45 9].[45 11]>
dojo: app poke failed
```

Here's a second agent. It takes a poke of `%inc` or `%dec` like before, but rather than updating its own state, it sends two pokes to `%pokeme`, so `%pokeme`'s state will be incremented or decremented by two.

<details>

<summary>/app/pokeit.hoon</summary>

```hoon
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 ~]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %noun
    =/  action  !<(?(%inc %dec) vase)
    ?-    action
        %inc
      :_  this
      :~  [%pass /inc %agent [our.bowl %pokeme] %poke %noun !>(%inc)]
          [%pass /inc %agent [our.bowl %pokeme] %poke %noun !>(%inc)]
      ==
    ::
        %dec
      :_  this
      :~  [%pass /dec %agent [our.bowl %pokeme] %poke %noun !>(%dec)]
          [%pass /dec %agent [our.bowl %pokeme] %poke %noun !>(%dec)]
      ==
    ==
  ==
::
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
::
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:def wire sign)
      [%inc ~]
    ?.  ?=(%poke-ack -.sign)
      (on-agent:def wire sign)
    ?~  p.sign
      %-  (slog '%pokeit: Increment poke succeeded!' ~)
      `this
    %-  (slog '%pokeit: Increment poke failed!' ~)
    `this
  ::
      [%dec ~]
    ?.  ?=(%poke-ack -.sign)
      (on-agent:def wire sign)
    ?~  p.sign
      %-  (slog '%pokeit: Decrement poke succeeded!' ~)
      `this
    %-  (slog '%pokeit: Decrement poke failed!' ~)
    `this
  ==
::
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

Here's the `+on-poke` arm:

```hoon
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %noun
    =/  action  !<(?(%inc %dec) vase)
    ?-    action
        %inc
      :_  this
      :~  [%pass /inc %agent [our.bowl %pokeme] %poke %noun !>(%inc)]
          [%pass /inc %agent [our.bowl %pokeme] %poke %noun !>(%inc)]
      ==
        %dec
      :_  this
      :~  [%pass /dec %agent [our.bowl %pokeme] %poke %noun !>(%dec)]
          [%pass /dec %agent [our.bowl %pokeme] %poke %noun !>(%dec)]
      ==
    ==
  ==
```

It's similar to `%pokeme`, except it sends two `%poke` `$card`s to `%pokeme` for each case, rather than modifying its own state. The `%inc` pokes specify a `$wire` of `/inc`, and the `%dec` pokes specify a `$wire` of `/dec`, so we can differentiate the responses. It also has the following `+on-agent`:

```hoon
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent wire sign)
      [%inc ~]
    ?.  ?=(%poke-ack -.sign)
      (on-agent wire sign)
    ?~  p.sign
      %-  (slog '%pokeit: Increment poke succeeded!' ~)
      `this
    %-  (slog '%pokeit: Increment poke failed!' ~)
    `this
  ::
      [%dec ~]
    ?.  ?=(%poke-ack -.sign)
      (on-agent wire sign)
    ?~  p.sign
      %-  (slog '%pokeit: Decrement poke succeeded!' ~)
      `this
    %-  (slog '%pokeit: Decrement poke failed!' ~)
    `this
  ==
```

`+on-agent` tests the `$wire`, checks if it's a `%poke-ack`, and then prints to the terminal whether it succeeded or failed.

Save this agent to `/app/pokeit.hoon` on the `%base` desk, `|commit %base`, and start it with `|rein %base [& %pokeme] [& %pokeit]`.

Let's try it out:

```
%pokeit: Increment poke succeeded!
%pokeit: Increment poke succeeded!
> :pokeit %inc
>=
```

`%pokeit` has received positive `%poke-ack`s, which means both pokes succeeded. It could tell they were increments because the `%poke-ack`s came back on the `/inc` wire we specified. We can check the state of `%pokeme` to confirm:

```
>   2
> :pokeme +dbug [%state %val]
>=
```

Let's try decrementing `%pokeme` so `.val` is 1, and then try a `%dec` via `%pokeit`:

```
> :pokeme %dec
>=
%pokeit: Decrement poke succeeded!
%pokeit: Decrement poke failed!
> :pokeit %dec
>=
```

The `+on-agent` arm of `%pokeit` has received one ack and one nack. The first took `.val` to zero, and the second crashed trying to decrement below zero.

## Summary <a href="#summary" id="summary"></a>

* Incoming pokes go to the `+on-poke` arm of an agent.
* The `+on-poke` arm takes a `$cage` and produces an `(quip card _this)`.
* Gall will automatically return a `%poke-ack` to the poke's source, with a stack trace in the `(unit tang)` if your agent crashed while processing the poke.
* Outgoing pokes can be sent by including `%poke` `%pass` `$card`s in the `+quip` produced by most agent arms.
* `%poke-ack`s in response to pokes you've sent will come in to the `+on-agent` arm in a `$sign`, on the `$wire` you specified in the original `%poke` `$card`.
* You can poke agents from the Dojo with a syntax of `:agent &mark ['some' 'noun']`.

## Exercises <a href="#exercises" id="exercises"></a>

* Run through the [example](#example) yourself on a fake ship if you've not done so already.
* Have a look at the `+on-agent` arm of `/lib/default-agent.hoon` to see how default-agent handles incoming `$sign`s.
* Try modifying the `%pokeme` agent with another action of your choice (in addition to `%inc` and `%dec`).
* Try modifying the `%pokeit` agent to send your new type of poke to `%pokeme`, and handle the `%poke-ack` it gets back.


# 7. Structures and Marks

Type definitions and data conversion for Gall agents.

Before we get into subscription mechanics, there's three things we need to touch on that are very commonly used in Gall agents. The first is defining an agent's types in a `/sur` structure file, the second is mark files, and the third is permissions. Note the example code presented in this lesson will not yet build a fully functioning Gall agent, we'll get to that in the next lesson.

## `/sur` <a href="#sur" id="sur"></a>

In the [previous lesson on pokes](/build-on-urbit/app-school/6-pokes), we used a very simple union in the `$vase` for incoming pokes:

```hoon
=/  action  !<(?(%inc %dec) vase)
```

A real Gall agent is likely to have a more complicated API. The most common approach is to define a head-tagged union of all possible poke types the agent will accept, and another for all possible updates it might send out to subscribers. Rather than defining these types in the agent itself, you would typically define them in a separate core saved in the `/sur` directory of the desk. The `/sur` directory is the canonical location for userspace type definitions.

With this approach, your agent can simply import the structures file and make use of its types. Additionally, if someone else wants to write an agent that interfaces with yours, they can include your structure file in their own desk to interact with your agent's API in a type-safe way.

#### Example

Let's look at a practical example. If we were creating a simple To-Do app, our agent might accept a few possible `$action`s as pokes: Adding a new task, deleting a task, toggling a task's "done" status, and renaming an existing task. It might also be able to send `$update`s out to subscribers when these events occur. If our agent were named `%todo`, it might have the following structure in `/sur/todo.hoon`:

<details>

<summary>/sur/todo.hoon</summary>

```hoon
|%
+$  id  @
+$  name  @t
+$  task  [=name done=?]
+$  tasks  (map id task)
+$  action
  $%  [%add =name]
      [%del =id]
      [%toggle =id]
      [%rename =id =name]
  ==
+$  update
  $%  [%add =id =name]
      [%del =id]
      [%toggle =id]
      [%rename =id =name]
      [%initial =tasks]
  ==
--
```

</details>

Our `%todo` agent could then import this structure file with a [fashep ford rune](/hoon/rune/fas#fashep) (`/-`) at the beginning of the agent like so:

```hoon
/-  todo
```

The agent's state could be defined like:

```hoon
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 =tasks:todo]
+$  card  card:agent:gall
--
```

Then, in its `+on-poke` arm, it could handle these actions in the following manner:

```hoon
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  |^
  ?>  =(src.bowl our.bowl)
  ?+    mark  (on-poke:def mark vase)
      %noun
    =^  cards  state
      (handle-poke !<(action:todo vase))
    [cards this]
  ==
  ::
  ++  handle-poke
    |=  =action:todo
    ^-  (quip card _state)
    ?-    -.action
        %add
      :_  state(tasks (~(put by tasks) now.bowl [name.action %.n]))
      :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`[%add now.bowl name.action])
          ==
      ==
    ::
        %del
      :_  state(tasks (~(del by tasks) id.action))
      :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`action)
          ==
      ==
    ::
        %toggle
      :_  %=  state
            tasks  %+  ~(jab by tasks)
                     id.action
                   |=(=task:todo task(done !done.task))
          ==
      :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`action)
          ==
      ==
    ::
        %rename
      :_  %=  state
            tasks  %+  ~(jab by tasks)
                     id.action
                   |=(=task:todo task(name name.action))
          ==
      :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`action)
          ==
      ==
    ==
  --
```

Let's break this down a bit. Firstly, our `+on-poke` arm includes a [barket](/hoon/rune/bar#barket) (`|^`) rune. Barket creates a core with a `$` arm that's computed immediately. We extract the `$vase` to the `$action:todo` type and immediately pass it to the `+handle-poke` arm of the core created with the barket. This `+handle-poke` arm tests what kind of `$action` it's received by checking its head. It then updates the state, and also sends an update to subscribers, as appropriate. Don't worry too much about the `%give` `$card` for now - we'll cover subscriptions in the next lesson.

Notice that the `+handle-poke` arm produces a `(quip card _state)` rather than `(quip card _this)`. The call to `+handle-poke` is also part of the following expression:

```hoon
=^  cards  state
  (handle-poke !<(action:todo vase))
[cards this]
```

The [tisket](/hoon/rune/tis#tisket) (`=^`) expression takes two arguments: A new named noun to pin to the subject (`.cards` in this case), and an existing wing of the subject to modify (`.state` in this case). Since `+handle-poke` produces `(quip card _state)`, we're saving the `$card`s it produces to `.cards` and replacing the existing `.state` with its new one. Finally, we produce `[cards this]`, where `.this` will now contain the modified `.state`. The `[cards this]` is a `(quip card _this)`, which our `+on-poke` arm is expected to produce.

This might seem a little convoluted, but it's a common pattern we do for two reasons. Firstly, it's not ideal to be passing around the entire `.this` agent core - it's much tidier just passing around the `.state`, until you actually want to return it to Gall. Secondly, It's much easier to read when the poke handling logic is separated into its own arm. This is a fairly simple example but if your agent is more complex, handling multiple marks and containing additional logic before it gets to the actual contents of the `$vase`, structuring things this way can be useful.

You can of course structure your `+on-poke` arm differently than we've done here - we're just demonstrating a typical pattern.

## Mark files <a href="#mark-files" id="mark-files"></a>

So far we've just used a `%noun` mark for pokes - we haven't really delved into what such marks represent, or considered writing custom ones.

Formally, marks are file types in the Clay filesystem. They correspond to mark files in the `/mar` directory of a desk. The `%noun` mark, for example, corresponds to the `/mar/noun.hoon` file. Mark files define the actual hoon data type for the file (e.g. a `*` noun for the `%noun` mark), but they also specify some extra things:

* Methods for converting between the mark in question and other marks.
* Revision control functions like patching, diffing, merging, etc.

Aside from their use by Clay for storing files in the filesystem, they're also used extensively for exchanging data with the outside world, and for exchanging data between Gall agents. When data comes in from a remote ship, destined for a particular Gall agent, it will be validated by the file in `/mar` that corresponds to its mark before being delivered to the agent. If the remote data has no corresponding mark file in `/mar` or it fails validation, it will crash before it touches the agent.

A mark file is a door with exactly three arms. The door's sample is the data type the mark will handle. For example, the sample of the `%noun` mark is just `non=*`, since it handles any noun. The three arms are as follows:

* `+grab`: Methods for converting *to* our mark *from* other marks.
* `+grow`: Methods for converting *from* our mark *to* other marks.
* `+grad`: Revision control functions.

In the context of Gall agents, you'll likely just use marks for sending and receiving data, and not for actually storing files in Clay. Therefore, it's unlikely you'll need to write custom revision control functions in the `+grad` arm. Instead, you can simply delegate `+grad` functions to another mark - typically `%noun`. If you want to learn more about writing such `+grad` functions, you can refer to the [marks Guide](/urbit-os/kernel/clay/marks) in the Clay vane documentation, which is much more comprehensive, but it's not necessary for our purposes here.

#### Example

Here's a very simple mark file for the `$action` structure we created in the [previous section](#sur):

```hoon
/-  todo
|_  =action:todo
++  grab
  |%
  ++  noun  action:todo
  --
++  grow
  |%
  ++  noun  action
  --
++  grad  %noun
--
```

We've imported the `/sur/todo.hoon` structure library from the previous section, and we've defined the sample of the door as `=action:todo`, since that's what it will handle. Now let's consider the arms:

### `+grab`

This handles conversion methods *to* our mark. It contains a core with arm names corresponding to other marks. In this case, it can only convert from a `%noun` mark, so that's the core's only arm. The `+noun` arm simply calls the `$action` structure from our structure library. This is called "clamming" or "molding" - when some noun comes in, it gets called like `(action:todo [some-noun])` - producing data of the `$action` type if it nests, and crashing otherwise.

### `+grow`

This handles conversion methods *from* our mark. Like `+grab`, it contains a core with arm names corresponding to other marks. Here we've also only added an arm for a `%noun` mark. In this case, `$action` data will come in as the sample of our door, and the `+noun` arm simply returns it, since it's already a `$noun` (as everything is in Hoon).

### `+grad`

This is the revision control arm, and as you can see we've simply delegated it to the `%noun` mark.

This mark file could be saved as `/mar/todo/action.hoon`, and then the `+on-poke` arm in the previous example could test for it instead of `%noun` like so:

```hoon
++  on-poke
  |=  [=mark =vase]
  |^  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %todo-action
    ...
```

Note how `%todo-action` will be resolved to `/mar/todo/action.hoon` - the hyphen will be interpreted as `/` if there's not already a `/mar/todo-action.hoon`.

This simple mark file isn't all that useful. Typically, you'd add `+json` arms to `+grow` and `+grab`, which allow your data to be converted to and from JSON, and therefore allow your agent to communicate with a web front-end. Front-ends, JSON, and Eyre's APIs which facilitate such communications will be covered in the separate [full-stack walkthrough](/build-on-urbit/app-school-full-stack), which you might like to work through after completing this guide. For now though, it's still useful to use marks and understand how they work.

One further note on marks: while data from remote ships must have a matching mark file in `/mar`, it's possible to exchange data between local agents with "fake" marks - ones that don't exist in `/mar`. Your `+on-poke` arm could, for example, use a made-up mark like `%foobar` for actions initiated locally. This is because marks come into play only at validation boundaries, none of which are crossed when doing local agent-to-agent communications.

## Permissions <a href="#permissions" id="permissions"></a>

In example agents so far, we haven't bothered to check where events such as pokes are actually coming from - our example agents would accept data from anywhere, including random foreign ships. We'll now have a look at how to handle such permission checks.

Back in [lesson 2](/build-on-urbit/app-school/2-agent#bowl) we discussed the [`$bowl`](/urbit-os/kernel/gall/data-types#bowl). The `$bowl` includes a couple of useful fields: `.our` and `.src`. The `.our` field just contains the `@p` of the local ship. The `.src` field contains the `@p` of the ship from which the event originated, and is updated for every new event.

When messages come in over Ames from other ships on the network, they're [encrypted](/urbit-os/kernel/ames/cryptography) with our ship's public keys and signed by the ship which sent them. The Ames vane decrypts and verifies the messages using keys in the Jael vane, which are obtained from the [Azimuth Ethereum contract](/urbit-id/azimuth-eth) and [Layer 2 data](/urbit-id/l2) where Urbit ID ownership and keys are recorded. This means the originating `@p` of all messages are cryptographically validated before being passed on to Gall, so the `@p` specified in the `.src` field of the `$bowl` can be trusted to be correct, which makes checking permissions very simple.

You're free to use whatever logic you want for this, but the most common way is to use [wutgar](/hoon/rune/wut#wutgar) (`?>`) and [wutgal](/hoon/rune/wut#wutgal) (`?<`) runes, which are respectively True and False assertions that crash if they don't evaluate to the expected truth value. To only allow messages from the local ship, you can just do the following in the relevant agent arm:

```hoon
?>  =(src.bowl our.bowl)
```

If we want to only allow messages from a particular set of ships, we could, for example, have a `(set @p)` in our agent's state called `allowed`. Then, we can use the `+has:in` set function to check:

```hoon
?>  (~(has in allowed) src.bowl)
```

There are many ways to handle permissions, it all depends on your particular use case.

## Summary <a href="#summary" id="summary"></a>

Type definitions:

* An agent's type definitions live in the `/sur` directory of a desk.
* The `/sur` file is a core, typically containing a number of lusbuc (`+$`) arms.
* `/sur` files are imported with the fashep (`/-`) Ford rune at the beginning of a file.
* Agent API types, for pokes and updates to subscribers, are commonly defined as head-tagged unions such as `[%foo bar=baz]`.

### Mark files

* Mark files live in the `/mar` directory of a desk.
* A mark like `%foo` corresponds to a file in `/mar` like `/mar/foo.hoon`
* Marks are file types in Clay, but are also used for passing data between agents as well as for external data generally.
* A mark file is a door with a sample of the data type it handles and exactly three arms: `+grab`, `+grow` and `+grad`.
* `+grab` and `+grow` each contain a core with arm names corresponding to other marks.
* `+grab` and `+grow` define functions for converting to and from our mark, respectively.
* `+grad` defines revision control functions for Clay, but you'd typically just delegate such functions to the `%noun` mark.
* Incoming data from remote ships will have their marks validated by the corresponding mark file in `/mar`.
* Messages passed between agents on a local ship don't necessarily need mark files in `/mar`.
* Mark files are most commonly used for converting an agent's native types to JSON, in order to interact with a web front-end.

### Permissions

* The source of incoming messages from remote ships are cryptographically validated by Ames and provided to Gall, which then populates the `.src` field of the `$bowl` with the `@p`.
* Permissions are most commonly enforced with wutgar (`?>`) and wutgal (`?<`) assertions in the relevant agent arms.
* Messages can be restricted to the local ship with `?> =(src.bowl our.bowl)` or to its moons as well with `?> |(=(our src):bowl (moon:title our.bowl src.bowl))`.
* There are many other ways to handle permissions, it just depends on the needs of the particular agent.

## Exercises <a href="#exercises" id="exercises"></a>

* Have a quick look at the `=^` [tisket documentation](/hoon/rune/tis#tisket).
* Try writing a mark file for the `$update:todo` type, in a similar fashion to the `$action:todo` one in the [mark file section](#mark-files). You can compare yours to the one we'll use in the next lesson.


# 8. Subscriptions

Gall agent subscription management.

In this lesson we're going to look at subscriptions. Subscriptions are probably the most complicated part of writing agents, so there's a fair bit to cover. Before we get into the nitty-gritty details, we'll give a brief overview of Gall's subscription mechanics.

The basic unit of subscriptions is the `$path`. An agent will typically define a number of subscription paths in its `+on-watch` arm, and other agents (local or remote) can subscribe to those paths. The agent will then send out updates called `%fact`s on one or more of its paths, and *all* subscribers of those paths will receive them. An agent cannot send out updates to specific subscribers, it can only target its paths. An agent can kick subscribers from its paths, and subscribers can unsubscribe from any paths.

The subscription paths an agent defines can be simple and fixed like `/foo/bar/baz`. They can also be dynamic, containing data of a particular atom aura encoded in certain elements of the path. These paths can therefore be as simple or complex as you need for your particular application.

Note it's not strictly necessary to define subscription paths explicitly. As long as the arm doesn't crash, the subscription will succeed. In practice, however, it's nearly always appropriate to define them explicitly and crash on unrecognized paths.

For a deeper explanation of subscription mechanics in Arvo, you can refer to Arvo's [subscriptions](/urbit-os/kernel/arvo/subscriptions) section.

## Incoming subscriptions <a href="#incoming-subscriptions" id="incoming-subscriptions"></a>

Subscription requests from other entities arrive in your agent's `+on-watch` arm. The `+on-watch` arm takes the `$path` to which they're subscribing, and produces a `(quip card _this)`:

```hoon
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ...
```

Your agent's subscription paths would be defined in this arm, typically in a wutlus (`?+`) expression or similar:

```hoon
?+    path  (on-watch:def path)
    [%updates ~]
  ::  ...
  ::  ...
    [%blah %blah ~]
  ::  ...
  ::  ...
    [%foo @ ~]
  =/  when=@da  (slav %da i.t.path)
  ::  ...
  ::  ...
    [%bar %baz *]
  ?+     t.t.path
       (on-watch:def path)
      ~
    ::  ...
      [%abc %def ~]
    ::  ...
      [%blah ~]
    ::  ...
  ==
==
```

Subscription paths can be simple and fixed like the first two examples above: `/updates` and `/blah/blah`. They can also contain "wildcard" elements, with an atom of a particular aura encoded in an element of the `$path`, as in the `[%foo @ ~]` example. The type pattern matcher is quite limited, so we just specify such variable elements as `@`, and then decode them with something like `(slav %da i.t.path)` (for a `@da`), as in the example. The incoming `$path` in this example would look like `/foo/~2021.11.14..13.30.39..6b17`. For more information on decoding atoms in strings, see the [strings guide](/hoon/strings#decoding-from-text).

In the last case of `[%bar %baz *]`, we're allowing a variable number of elements in the path. First we check it's `/bar/baz/...something...`, and then we check what the "something" is in another wutlus expression and handle it appropriately. In this case, it could be `/bar/baz`, `/bar/baz/abc/def`, or `/bar/baz/blah`. You could of course also have "wildcard" elements here too, so there's not really a limit to the complexity of your subscription paths, or the data that might be encoded therein.

Permissions can be checked as described in the previous lesson, comparing the source `@p` of the request in `src.bowl` to `our.bowl` or any other logic you find appropriate.

If a permission check fails, the path is not valid, or any other reason you want to reject the subscription request, your agent can simply crash. The behavior here is the same as with `+on-poke` - Gall will send a `%watch-ack` card in response, which is either an ack (positive acknowledgement) or a nack (negative acknowledgement). The `(unit tang)` in the `%watch-ack` will be null if processing succeeded, and non-null if it crashed, with a stack trace in the `$tang`. Like with `%poke-ack`s, you don't need to explicitly send a `%watch-ack` - Gall will do it automatically.

As well as sending a `%watch-ack`, Gall will also record the subscription in the `.sup` field of the `$bowl`, if it succeeded. Then, when you send updates out to subscribers of the `$path` in question, the new subscriber will begin receiving them as well.

Updates to subscribers would usually be sent from other arms, but there's one special case for `+on-watch` which is very useful. Normally updates can only be sent to all subscribers of a particular path - you can't target a specific subscriber. There's one exception to this: In `+on-watch`, when there's a new subscription, you can send a `%fact` back with an empty `(list path)`, and it'll only go to the new subscriber. This is most useful when you want to give the subscriber some initial state, which you otherwise couldn't do without sending it to everyone. It might look something like this:

```hoon
:_  this
:~  [%give %fact ~ %todo-update !>(`$update:todo`initial+tasks)]
==
```

## Sending updates to subscribers <a href="#sending-updates-to-subscribers" id="sending-updates-to-subscribers"></a>

Once your agent has subscribers, it's easy to send them out updates. All you need to do is produce `$card`s with `%fact`s in them:

```hoon
:_  this
:~  [%give %fact ~[/some/path /another/path] %some-mark !>('some data')]
    [%give %fact ~[/some/path] %some-mark !>('more data')]
    ....
==
```

The `(list path)` in the `%fact` specifies which subscription `$path`s the `%fact` should be sent on. All subscribers of all `$path`s specified will receive the `%fact`. Any agent arm which produces a `(quip card _this)` can send `%fact`s to subscribers. Most often they will be produced in the `+on-poke` arm, since new data will often be added in `$poke`s.

## Kicking subscribers <a href="#kicking-subscribers" id="kicking-subscribers"></a>

To kick a subscriber, you just send a `%kick` `$card`:

```hoon
[%give %kick ~[/some/path] `~sampel-palnet]
```

The `(list path)` specifies which subscription `$path`s the ship should be kicked from, and the `(unit ship)` specifies which ship to kick. The `(unit ship)` can also be null, like so:

```hoon
[%give %kick ~[/some/path] ~]
```

In this case, all subscribers to the specified `$path`s will be kicked.

Note that `%kick`s are not exclusively sent by the agent itself - Gall itself can also kick subscribers under certain network conditions. Because of this, `%kick`s are not assumed to be intentional, and the usual behavior is for a kicked agent to try and resubscribe. Therefore, if you want to disallow a particular subscriber, your agent's `+on-watch` arm should reject further subscription requests from them - your agent should not just `%kick` them and call it a day.

## Outgoing subscriptions <a href="#outgoing-subscriptions" id="outgoing-subscriptions"></a>

Now that we've covered incoming subscriptions, we'll look at the other side of it: Subscribing to other agents. This is done by `%pass`ing the target agent a `%watch` task in a `$card`:

```hoon
[%pass /some/wire %agent [~some-ship %some-agent] %watch /some/path]
```

If your agent's subscription request is successful, updates will come in to your agent's `+on-agent` arm on the `$wire` specified (`/some/wire` in this example). The `$wire` can be anything you like; its purpose is for your agent to figure out which subscription the updates came from. The `[ship term]` pair specifies the ship and agent you're trying to subscribe to, and the final `$path` (`/some/path` in this example) is the path you want to subscribe to: a `$path` the target agent has defined in its `+on-watch` arm.

Gall will deliver the `$card` to the target agent and call that agent's `+on-watch` arm, which will process the request [as described above](#incoming-subscription-requests), accept or reject it, and send back either a positive or negative `%watch-ack`. The `%watch-ack` will come back in to your agent's `+on-agent` arm in a `$sign`, along with the `$wire` you specified (`/some/wire` in this example). Recall in the lesson on pokes, the `+on-agent` arm starts with:

```hoon
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ::  ...
```

The `$sign` will be of the following format:

```hoon
[%watch-ack p=(unit tang)]
```

How you want to handle the `%watch-ack` really depends on the particular agent. In the simplest case, you can just pass it to the `+on-agent` arm of default-agent, which will just accept it and do nothing apart from printing the error in the `%watch-ack` `$tang` if it's a nack. You shouldn't have your agent crash on a `%watch-ack` - even if it's a nack your agent should process it successfully. If you wanted to apply some additional logic on receipt of the `%watch-ack`, you'd typically first test the `$wire`, then test whether it's a `%watch-ack`, then test whether it's an ack or a nack and do whatever's appropriate:

```hoon
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:def wire sign)
      [%expected %wire ~]
    ?+    -.sign  (on-agent:def wire sign)
        %watch-ack
      ?~  p.sign
        ::  ...(do something if ack)...
      ::  ...(do something if nack)...
  ::  ...
```

The `+on-agent` arm produces a `(quip card _this)`, so you can produce new `$card`s and update your agent's state, as appropriate.

One further thing to note with subscriptions is that you can subscribe multiple times to the same `$path` on the same ship and agent, as long as the `$wire` is unique. If the ship, agent, `$path` and `$wire` are all the same as an existing subscription, Gall will not allow the request to be sent, and instead fail with an error message fed into the `+on-fail` arm of your agent.

## Receiving updates <a href="#receiving-updates" id="receiving-updates"></a>

Assuming the `%watch` succeeded, your agent will now begin receiving any `%fact`s the other agent publishes on the `$path` to which you've subscribed. These `%fact`s will also come in to your agent's `+on-agent` arm in a `$sign`, just like the initial `%watch-ack`. The `%fact` `$sign` will have the following format:

```hoon
[%fact =cage]
```

You would typically handle such `%fact`s in the following manner: Test the `$wire`, test whether the `$sign` is a `%fact`, test the `$mark` in the `$cage`, extract the data from the `$vase` in the `$cage`, and apply your logic. For example:

```hoon
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:def wire sign)
      [%expected %wire ~]
    ?+    -.sign  (on-agent:def wire sign)
        %fact
      ?+    p.cage.sign  (on-agent:def wire sign)
          %expected-mark
        =/  foo  !<(expected-type q.cage.sign)
        ::  ...
  ::  ...
```

Note that Gall will not allow `$sign`s to come into `+on-agent` unsolicited, so you don't necessarily need to include permission logic in this arm.

The `+on-agent` arm produces a `(quip card _this)`, so you can produce new `$card`s and update your agent's state, as appropriate.

## Getting kicked <a href="#getting-kicked" id="getting-kicked"></a>

For whatever reason, the agent you're `%watch`ing might want to kick your agent from a `$path` to which it's subscribed, ending your subscription and ceasing to send your agent `%fact`s. To do this, it will send your agent a `%kick` card [as described above](#kicking-subscribers). The `%kick` will come in to your agent's `+on-agent` arm in a `$sign`, like `%watch-ack`s and `%fact`s do. The `%kick` `$sign` will have the following format:

```hoon
[%kick ~]
```

Since the `%kick` itself contains no information, you'll need to consider the `$wire` it comes in on to know what it pertains to. As explained previously, `%kick`s aren't always intentional - sometimes Gall will kick subscribers due to network issues. Your `+on-agent` arm therefore has no way to know whether the other agent actually intended to kick it. This means *your agent should almost always try to resubscribe if it gets kicked*. Then, if the resubscribe `%watch` request is rejected with a negative `%watch-ack`, you can conclude that it was intentional and give up. The logic would look something like this:

```hoon
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:def wire sign)
      [%some %wire ~]
    ?+    -.sign  (on-agent:def wire sign)
        %kick
      :_  this
      :~  [%pass /some/wire %agent [src.bowl dap.bowl] %watch /some/path]
      ==
  ::  ...
```

## Leaving a subscription <a href="#leaving-a-subscription" id="leaving-a-subscription"></a>

Eventually you may wish to unsubscribe from a `$path` in another agent and stop receiving updates. This is done by `%pass`ing a `%leave` task to the agent in question:

```hoon
[%pass /some/wire %agent [~some-ship %some-agent] %leave ~]
```

The subscription to be ended is determined by the combination of the `$wire`, ship and agent, so the `%leave` task itself always just has `~` at the end.

## Example <a href="#example" id="example"></a>

Here we're going to give a pretty well fleshed out example. It will demonstrate both inbound and outbound subscriptions, most of the concepts we've discussed here, as well as some from the previous lesson: `/sur` files, mark files, and permission checks.

In previous lessons we've only dealt with things on a local ship; this example will demonstrate messages being sent over the network.

The example will be composed of two separate agents, a publisher called `/app/todo.hoon` and a subscriber called `/app/todo-watcher.hoon`, which will live on separate ships. It will be a very rudimentary To-Do app. To-do tasks will be poked into the publisher and sent out to the subscriber as `%fact`s, which will just print them to the dojo. It will have its types defined in `/sur/todo.hoon`, and it will have a couple of mark files for pokes and updates: `/mar/todo/action.hoon` and `/mar/todo/update.hoon`.

Before we get into trying it out, we'll first walk through the `/sur` file, mark files, and each agent.

### Types and marks <a href="#types-and-marks" id="types-and-marks"></a>

<details>

<summary>/sur/todo.hoon</summary>

```hoon
|%
+$  id  @
+$  name  @t
+$  task  [=name done=?]
+$  tasks  (map id task)
+$  who  @p
+$  friends  (set who)
+$  action
  $%  [%add =name]
      [%del =id]
      [%toggle =id]
      [%rename =id =name]
      [%allow =who]
      [%kick =who]
  ==
+$  update
  $%  [%add =id =name]
      [%del =id]
      [%toggle =id]
      [%rename =id =name]
      [%initial =tasks]
  ==
--
```

</details>

This file defines most of the types for the agents. The list of to-do tasks will be stored in the state of the publisher agent as the `$tasks` type, a `(map id task)`, where a task is a `[=name done=?]`. The set of ships allowed to subscribe will be stored in `.friends`, a `(set @p)`, also in the publisher's state. After that, there are the head-tagged unions of accepted poke `$action`s and `$update`s for subscribers.

**`/mar/todo/action.hoon`**

```hoon
/-  todo
|_  =action:todo
++  grab
  |%
  ++  noun  action:todo
  --
++  grow
  |%
  ++  noun  action
  --
++  grad  %noun
--
```

This is a very simple mark file for the `$action` type.

**`/mar/todo/update.hoon`**

```hoon
/-  todo
|_  =update:todo
++  grab
  |%
  ++  noun  update:todo
  --
++  grow
  |%
  ++  noun  update
  --
++  grad  %noun
--
```

This is a very simple mark file for the `$update` type.

### Publisher <a href="#publisher" id="publisher"></a>

<details>

<summary>/app/todo.hoon</summary>

```hoon
/-  todo
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 =friends:todo =tasks:todo]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  |^
  ?>  =(src.bowl our.bowl)
  ?+    mark  (on-poke:def mark vase)
      %todo-action
    =^  cards  state
      (handle-poke !<(action:todo vase))
    [cards this]
  ==
  ++  handle-poke
    |=  =action:todo
    ^-  (quip card _state)
    ?-    -.action
        %add
      ?:  (~(has by tasks) now.bowl)
        $(now.bowl (add now.bowl ~s0..0001))
      :_  state(tasks (~(put by tasks) now.bowl [name.action %.n]))
      :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`[%add now.bowl name.action])
          ==
      ==
    ::
        %del
      :_  state(tasks (~(del by tasks) id.action))
      :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`action)
          ==
      ==
    ::
        %toggle
      :-  :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`action)
          ==  ==
      %=  state
        tasks  %+  ~(jab by tasks)
                 id.action
               |=(=task:todo task(done !done.task))
      ==
    ::
        %rename
      :-  :~  :*  %give  %fact  ~[/updates]  %todo-update
              !>(`$update:todo`action)
          ==  ==
      %=  state
        tasks  %+  ~(jab by tasks)
                 id.action
               |=(=task:todo task(name name.action))
      ==
        %allow
      `state(friends (~(put in friends) who.action))
    ::
        %kick
      :_  state(friends (~(del in friends) who.action))
      :~  [%give %kick ~[/updates] `who.action]
      ==
    ==
  --
::
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?+    path  (on-watch:def path)
      [%updates ~]
    ?>  (~(has in friends) src.bowl)
    :_  this
    :~  [%give %fact ~ %todo-update !>(`$update:todo`initial+tasks)]
    ==
  ==
::
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

This is the publisher agent, `/app/todo.hoon`. The bulk of its logic is in its `+on-poke` arm, where it handles the various possible actions like `%add`ing a task, `%toggle`ing its "done" state, `%rename`ing a task, and so on. It also has a couple of `$action`s for `%allow`ing and `%kick`ing subscribers.

Most of these cases both update the state of the agent, as well as producing `%fact` cards to send out to subscribers with the new data.

You'll notice it only allows these pokes from the local ship, and enforces this in `+on-poke` with:

```hoon
?>  =(src.bowl our.bowl)
```

Additionally, you might notice the `%add` case in `+handle-poke` begins with the following:

```hoon
?:  (~(has by tasks) now.bowl)
  $(now.bowl (add now.bowl ~s0..0001))
```

Back in lesson two, we mentioned that the bowl is only repopulated when there's a new Arvo event, so simultaneous messages from a local agent or web client would be processed with the same bowl. Since we're using `.now.bowl` for the task ID, this means multiple `%add` actions could collide. To handle this case, we check if there's already an entry in the `.tasks` map with the current date-time, and if there is, we increase the time by a fraction of a second and try again.

Let's now look at `+on-watch`:

```hoon
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?+    path  (on-watch:def path)
      [%updates ~]
    ?>  (~(has in friends) src.bowl)
    :_  this
    :~  [%give %fact ~ %todo-update !>(`$update:todo`initial+tasks)]
    ==
  ==
```

When `+on-watch` gets a subscription request, it checks whether the requesting ship is in the `.friends` set, and crashes if it is not. If they're in `.friends`, it produces a `%fact` card with a null `(list path)`, which means it goes only to the new subscriber. This `%fact` contains the entire `.tasks` map as it currently exists, getting the new subscriber up to date.

### Subscriber <a href="#subscriber" id="subscriber"></a>

#### `/app/todo-watcher.hoon`

<details>

<summary>/app/todo-watcher.hoon</summary>

```hoon
/-  todo
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 ~]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?>  =(src.bowl our.bowl)
  ?+    mark  (on-poke:def mark vase)
      %noun
    =/  action  !<(?([%sub @p] [%unsub @p]) vase)
    ?-    -.action
        %sub
      :_  this
      :~  [%pass /todos %agent [+.action %todo] %watch /updates]
      ==
        %unsub
      :_  this
      :~  [%pass /todos %agent [+.action %todo] %leave ~]
      ==
    ==
  ==
::
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
::
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:def wire sign)
      [%todos ~]
    ?+    -.sign  (on-agent:def wire sign)
        %watch-ack
      ?~  p.sign
        ((slog '%todo-watcher: Subscribe succeeded!' ~) `this)
      ((slog '%todo-watcher: Subscribe failed!' ~) `this)
    ::
        %kick
      %-  (slog '%todo-watcher: Got kick, resubscribing...' ~)
      :_  this
      :~  [%pass /todos %agent [src.bowl %todo] %watch /updates]
      ==
    ::
        %fact
      ?+    p.cage.sign  (on-agent:def wire sign)
          %todo-update
        ~&  !<(update:todo q.cage.sign)
        `this
      ==
    ==
  ==
::
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

This is the subscriber agent. Since it's just for demonstrative purposes, it has no state and just prints the updates it receives. In practice it would keep the `.tasks` map it receives in its own state, and then update it as it receives new `%fact`s.

The `+on-poke` arm is fairly simple - it accepts two pokes, to either `[%sub ~some-ship]` or `[%unsub ~some-ship]`.

The `+on-agent` arm will print whether a subscription request succeeded or failed, as well as printing a message when it gets kicked. When it receives a `%fact` from the publisher agent, it will just print it to the terminal with a `~&` expression.

### Trying it out <a href="#trying-it-out" id="trying-it-out"></a>

We're going to try this between two different ships. The first ship will be the usual fakeship. We'll add both mark files, the `/sur` file, and the `%todo` agent to the `%base` desk of our fakeship, putting them in the following directories:

```
base
├── app
│   └── todo.hoon
├── mar
│   └── todo
│       ├── action.hoon
│       └── update.hoon
└── sur
    └── todo.hoon
```

In \~zod's dojo, we can `|commit %base`, and then start the `%todo` agent:

```
|rein %base [& %todo]
```

Now we need to spin up another fake ship. We'll use \~nut in this example:

```
urbit -F nut
```

Once it's booted, we can `|mount %base` and then add just the `/mar/todo/update.hoon` mark file, the `/sur` file, and the `todo-watcher.hoon` agent like so:

```
base
├── app
│   └── todo-watcher.hoon
├── mar
│   └── todo
│       └── update.hoon
└── sur
    └── todo.hoon
```

On \~nut we can then `|commit %base`, and start the `%todo-watcher` agent:

```
|rein %base [& %todo-watcher]
```

Now, on \~nut, let's try subscribing:

```
> :todo-watcher [%sub ~zod]
>=
%todo-watcher: Subscribe failed!
```

Our `%todo-watcher` agent tried, but received a negative `%watch-ack` from `%todo`, because we haven't yet added \~nut to the `.friends` set of allowed ships. Let's now remedy that on \~zod:

```
> :todo &todo-action [%allow ~nut]
>=
```

Let's also add a couple of to-do tasks, on \~zod:

```
> :todo &todo-action [%add 'foo']
>=
> :todo &todo-action [%add 'bar']
>=
```

If we now check its state with `+dbug`, we'll see they're in the `.tasks` map, and \~nut will also now be in the `.friends` set:

```
>   [ %0
  friends={~nut}
    tasks
  { [ p=170.141.184.505.349.079.206.522.766.950.035.095.552
      q=[name='foo' done=%.n]
    ]
    [ p=170.141.184.505.349.079.278.538.984.166.386.565.120
      q=[name='bar' done=%.n]
    ]
  }
]
> :todo +dbug
>=
```

Let's now try subscribing again on \~nut:

```
> :todo-watcher [%sub ~zod]
>=
%todo-watcher: Subscribe succeeded!
[ %initial
    tasks
  { [ p=170.141.184.505.349.079.206.522.766.950.035.095.552
      q=[name='foo' done=%.n]
    ]
    [ p=170.141.184.505.349.079.278.538.984.166.386.565.120
      q=[name='bar' done=%.n]
    ]
  }
]
```

As you can see, this time it's worked, and we've immediately received the initial `.tasks` map.

Now, let's try adding another task on \~zod:

```
> :todo &todo-action [%add 'baz']
>=
```

On \~nut, we'll see it has received the `%fact` with the new task in it:

```
[ %add
  id=170.141.184.505.349.082.779.030.192.959.445.270.528
  name='baz'
]
```

Let's try toggle its done state on \~zod:

```
> :todo &todo-action [%toggle 170.141.184.505.349.082.779.030.192.959.445.270.528]
>=
```

\~nut will again get the `%fact`:

```
[ %toggle
  id=170.141.184.505.349.082.779.030.192.959.445.270.528
]
```

Recall that incoming subscriptions are stored in `.sup.bowl`, and outgoing subscriptions are stored in `.wex.bowl`. Let's have a look at the incoming subscription on \~zod:

```
>   [ path=/updates
  from=~nut
  duct=~[/gall/sys/req/~nut/todo /ames/bone/~nut/1 //ames]
]
> :todo +dbug [%incoming %ship ~nut]
>=
```

On \~nut, let's look at the outgoing subscription:

```
>   [wire=/todos agnt=[~zod %todo] path=/updates ackd=%.y]
> :todo-watcher +dbug [%outgoing %ship ~zod]
>=
```

Now on \~zod, let's try kicking \~nut and removing it from our `.friends` set:

```
> :todo &todo-action [%kick ~nut]
>=
```

On \~nut, we'll see it got the `%kick`, tried resubscribing automatically, but was rejected because \~nut is no longer in `.friends`:

```
%todo-watcher: Got kick, resubscribing...
%todo-watcher: Subscribe failed!
```

## Summary <a href="#summary" id="summary"></a>

* Incoming subscription requests arrive in an agent's `+on-watch` arm.
* An agent will define various subscription `$path`s in its `+on-watch` arm, which others can subscribe to.
* Gall will automatically produce a negative `%watch-ack` if `+on-watch` crashed, and a positive one if it was successful.
* Incoming subscribers are recorded in the `.sup` field of the `$bowl`.
* `+on-watch` can produce a `%fact` with a null `(list path)` which will go only to the new subscriber.
* Updates are sent to subscribers in `%fact` cards, and contain a `$cage` with a `$mark` and some data in a `$vase`.
* `%fact`s are sent to all subscribers of the paths specified in the `(list path)`.
* A subscriber can be kicked from subscription paths with a `%kick` card specifying the ship in the `(unit ship)`. All subscribers of the specified paths will be kicked if the `(unit ship)` is null.
* An outgoing subscription can be initiated with a `%watch` card.
* The `%watch-ack` will come back in to the subscriber's `+on-agent` arm as a `$sign`, and may be positive or negative, depending on whether the `(unit tang)` is null.
* `%kick`s will also arrive in the subscriber's `+on-agent` arm as a `$sign`. Since kicks may not be intentional, the subscriber should attempt to resubscribe and only give up if the subsequent `%watch-ack` is negative.
* `%fact`s will also arrive in the subscriber's `+on-agent` arm.
* All such `$sign`s that arrive in `+on-agent` will also have a `$wire`.
* The `$wire` for subscription updates to arrive on is specified in the initial `%watch` card.
* A subscriber can unsubscribe by passing a `%leave` card on the original `$wire`.

## Exercises <a href="#exercises" id="exercises"></a>

* Have a look at the [strings guide](/hoon/strings) if you're not already familiar with decoding/encoding atoms in strings.
* Try running through the [example](#example) yourself, if you've not done so already.
* Try modifying `%todo-watcher` to recording the data it receives in its state, rather than simply printing it to the terminal.
* If you'd like, try going back to [lesson 6](/build-on-urbit/app-school/6-pokes) (on pokes) and modifying the agents with an appropriate permission system, and also try running them on separate ships.


# 9. Vanes

Guide to interacting with Arvo vanes (kernel modules) from Gall agents.

In this lesson we're going to look at interacting with vanes (kernel modules). The API for each vane consists of tasks it can take, and gifts it can return. The tasks and gifts for each vane are defined in its section of `/sys/lull.hoon`. Here's the `$task:iris`s and `$gift:iris`s for Iris, the HTTP client vane, as an example:

```hoon
|%
+$  gift
  $%  [%request id=@ud request=request:http]
      [%cancel-request id=@ud]
      [%http-response =client-response]
  ==
+$  task
  $~  [%vega ~]
  $%  $>(%born vane-task)
      $>(%trim vane-task)
      $>(%vega vane-task)
      [%request =request:http =outbound-config]
      [%cancel-request ~]
      [%receive id=@ud =http-event:http]
  ==
```

The API of each vane is documented in its respective section of the [Arvo documentation](/urbit-os/kernel/arvo). Each vane has a detailed API reference and examples of their usage. There are far too many tasks and gifts across the vanes to cover here, so in the [`Example`](#example) section of this document, we'll just look at a single, simple example with a Behn timer. The basic pattern in the example is broadly applicable to the other vanes as well.

## Sending a vane task <a href="#sending-a-vane-task" id="sending-a-vane-task"></a>

A task can be sent to a vane by `%pass`ing it an `%arvo` card. We touched on these in the [Cards](/build-on-urbit/app-school/5-cards) lesson, but we'll briefly recap it here. The type of the card is as follows:

```hoon
[%pass path %arvo note-arvo]
```

The `$path` will just be the `$wire` you want the response to arrive on. The `$note-arvo` is the following union:

```hoon
+$  note-arvo
  $~  [%b %wake ~]
  $%  [%a task:ames]
      [%b task:behn]
      [%c task:clay]
      [%d task:dill]
      [%e task:eyre]
      [%g task:gall]
      [%i task:iris]
      [%j task:jael]
      [%k task:khan]
      [%l task:lick]
      [%$ %whiz ~]
      [@tas %meta vase]
  ==
```

The letter tags just specify which vane it goes to, and then follows the task itself. Here are a couple of examples. The first sends a `%wait` `$task:behn` to Behn, setting a timer to go off one minute in the future. The second sends a `%warp` `$task:clay` to Clay, asking whether `sys.kelvin` exists on the `%base` desk.

```hoon
[%pass /some/wire %arvo %b %wait (add ~m1 now.bowl)]
[%pass /some/wire %arvo %c %warp our.bowl %base ~ %sing %u da+now.bowl /sys/kelvin]
```

## Receiving a vane gift <a href="#receiving-a-vane-gift" id="receiving-a-vane-gift"></a>

Once a task has been sent to a vane, any gifts the vane sends back in response will arrive in the `+on-arvo` arm of your agent. The `+on-arvo` arm exclusively handles such vane gifts. The gifts will arrive in a `$sign-arvo`, along with the `$wire` specified in the original request. The `+on-arvo` arm produces a `(quip card _this)` like usual, so it would look like:

```hoon
++  on-arvo
  |=  [=wire =sign-arvo]
  ^-  (quip card _this)
  ::  ...
```

A `$sign-arvo` is the following structure, defined in `lull.hoon`:

```hoon
+$  sign-arvo
  $%  [%ames gift:ames]
      $:  %behn
          $%  gift:behn
              $>(%wris gift:clay)
              $>(%writ gift:clay)
              $>(%mere gift:clay)
              $>(%unto gift:gall)
          ==
      ==
      [%clay gift:clay]
      [%dill gift:dill]
      [%eyre gift:eyre]
      [%gall gift:gall]
      [%iris gift:iris]
      [%jael gift:jael]
      [%khan gift:khan]
      [%lick gift:lick]
  ==
```

The head of the `$sign-arvo` will be the name of the vane like `%behn`, `%clay`, etc. The tail will be the gift itself. Here are a couple of `$sign-arvo` examples, and the responses to the example tasks in the previous section:

```hoon
[%behn %wake ~]
```

```
[ %clay
  [ %writ
      p
    [ ~
      [ p=[p=%u q=[%da p=~2021.11.17..13.55.00..c195] r=%base]
        q=/sys/kelvin
        r=[p=%flag q=[#t/?(%.y %.n) q=0]]
      ]
    ]
  ]
]
```

The typical pattern is to first test the `$wire` with something like a wutlus (`?+`) expression, and then test the `$sign-arvo`. Since most gifts are head-tagged, you can test both the vane and the gift at the same time like:

```hoon
?+    sign-arvo  (on-arvo:def wire sign-arvo)
    [%behn %wake *]
  ::  ..
::  ...
```

## Example <a href="#example" id="example"></a>

Here's a very simple example that takes a poke of a `@dr` (a relative date-time value) and sends Behn a `%wait` `$task:behn`, setting a timer to go off `@dr` in the future. When the timer goes off, `+on-arvo` will take the `%wake` `$gift:behn` and print "Ding!" to the terminal.

<details>

<summary>/app/ding.hoon</summary>

```hoon
/+  default-agent, dbug
|%
+$  card  card:agent:gall
--
%-  agent:dbug
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
++  on-init  on-init:def
++  on-save  on-save:def
++  on-load  on-load:def
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %noun
    :_  this
    :~  [%pass /timers %arvo %b %wait (add now.bowl !<(@dr vase))]
    ==
  ==
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo
  |=  [=wire =sign-arvo]
  ^-  (quip card _this)
  ?+    wire  (on-arvo:def wire sign-arvo)
      [%timers ~]
    ?+    sign-arvo  (on-arvo:def wire sign-arvo)
        [%behn %wake *]
      ?~  error.sign-arvo
        ((slog 'Ding!' ~) `this)
      (on-arvo:def wire sign-arvo)
    ==
  ==
++  on-fail   on-fail:def
--
```

</details>

Let's examine the `+on-poke` arm:

```hoon
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+    mark  (on-poke:def mark vase)
      %noun
    :_  this
    :~  [%pass /timers %arvo %b %wait (add now.bowl !<(@dr vase))]
    ==
  ==
```

A Behn `%wait` task has the format `[%wait @da]` - the `@da` (an absolute date-time value) is the time the timer should go off. The `$vase` of the poke takes a `@dr`, so we extract it directly into an `+add` expression, producing a date-time `@dr` from now. Behn will receive the `%wait` task and set the timer in Unix. When it fires, Behn will produce a `%wake` `$gift:behn` and deliver it to `+on-arvo`, on the `$wire` we specified (`/timers`). Here's the `+on-arvo` arm:

```hoon
++  on-arvo
  |=  [=wire =sign-arvo]
  ^-  (quip card _this)
  ?+    wire  (on-arvo:def wire sign-arvo)
      [%timers ~]
    ?+    sign-arvo  (on-arvo:def wire sign-arvo)
        [%behn %wake *]
      ?~  error.sign-arvo
        ((slog 'Ding!' ~) `this)
      (on-arvo:def wire sign-arvo)
    ==
  ==
```

We remark that, just like in the case of agent-agent communication, gifts from Arvo are also routed `$wire` before `$sign-arvo`.

First we check the `$wire` is `/timers`, and then we check the `$sign-arvo` is `[%behn %wake *]`, where `*` is any noun. Behn's `%wake` gift has the following format:

```hoon
[%wake error=(unit tang)]
```

The `.error` is null if the timer fired successfully, and contains an error in the `$tang` if it did not. We therefore test whether `.error.sign-arvo` is `~`, and if it is, we print `Ding!` to the terminal. If the `$wire`, `$sign-arvo` or `.error` are something unexpected, we pass it to `%default-agent`, which will just crash and print an error message.

Let's try it out. Save the agent above as `/app/ding.hoon` on the `%base` desk and `|commit %base`. Then, start the agent with `|rein %base [& %ding]`.

Next, in the Dojo let's try poking our agent, setting a timer for five seconds from now:

```
> :ding ~s5
>=
```

After approximately five seconds, we see the timer fired successfully:

```
> Ding!
```

## Summary <a href="#summary" id="summary"></a>

* Each vane has an API composed of tasks it takes and gifts it produces.
* Each vane's tasks and gifts are defined in `lull.hoon`
* Each vane's section of the [Arvo documentation](/urbit-os/kernel/arvo) includes an API reference that explains its tasks and gifts, as well as an Examples section demonstrating their usage.
* Vane tasks can be sent to vanes by `%pass`ing them an `%arvo` `$card`.
* Vane gifts come back to the `+on-arvo` arm of the agent core in a `$sign-arvo`.

## Exercises <a href="#exercises" id="exercises"></a>

* Run through the [example](#example) yourself if you've not done so already.
* Have a look at some vane sections of `lull.hoon` to familiarize yourself with its structure.
* Have a quick look at the API reference sections of a couple of vanes in the [Arvo documentation](/urbit-os/kernel/arvo).


# 10. Scries

Guide to Gall agent scry functionality.

In this lesson we'll look at scrying agents, as well as how agents handle such scries. If you're not at all familiar with performing scries in general, have a read through the [scry guide](/urbit-os/kernel/arvo/scry), as well as the [dotket rune documentation](/hoon/rune/dot#dotket).

## Scrying <a href="#scrying" id="scrying"></a>

A scry is a read-only request to Arvo's global namespace. Vanes and agents define "scry endpoints" which allow data to be requested from their states. The endpoints can process the data in any way before returning it, but they cannot alter the actual state - scries can only read, not modify.

Gall itself defines some special vane-level endpoints [as described in its scry reference](/urbit-os/kernel/gall/scry), but most scries to Gall are routed to particular agents and handled by them instead. Agent scries are what we'll focus on here.

Scries are performed with the [dotket](/hoon/rune/dot#dotket) (`.^`) rune. Here's a summary of their format:

![](https://media.urbit.org/docs/arvo/scry-diagram-v2.svg)

A note on `$care`s: Cares are most carefully implemented by Clay, where they specify submodules and have tightly defined behaviors. For Gall agents, most of these don't have any special behavior, and are just used to indicate the general kind of data produced by the endpoint, with the exception of the `%x` care:

#### `%x`

Gall handles `%x` specially, and expects an extra field at the end of the `$path` that specifies the `$mark` to return. Gall will take the data produced by the specified endpoint and try to convert it to the given mark, crashing if the mark conversion fails. The extra field specifying the mark is not passed through to the agent itself. Here's a couple of examples:

```
> =g -build-file /=groups=/sur/groups/hoon

> .^(groups:g %gx /=groups=/groups/noun)
[   n
  [ p=[p=~zod q=%test-group]
      q
    [ fleet=[n=[p=~zod q=[sects=[n=%admin l={} r={}] joined=~2000.1.1]] l=~ r=~]
        cabals
      ...

> (en:json:html .^(json %gx /=profile=/widgets/json/json))
'{"groups":{"profile":"Profile Header","profile-bio":"Profile Bio","join-button":"\\"Join me\\" button"}}'
```

The majority of Gall agents simply take `%x` `$care`s in their scry endpoints, but in principle it's possible for a Gall agent to define a scry endpoint that takes any one of the `$care`s listed in the diagram above. An agent's scry endpoints are defined in its `+on-peek` arm, which we'll look at next.

## Handling scries <a href="#handling-scries" id="handling-scries"></a>

When a scry is performed on a Gall agent, Gall will strip out some extraneous parts, and deliver it to the agent's `+on-peek` arm as a `$path`. The `$path` will only have two components from the diagram above: The "care" and the "path". For example, a scry of `.^(groups:g %gx /=groups=/groups/noun)` will come into the `+on-peek` arm of `%groups` as `/x/groups`.

The `+on-peek` arm produces a `(unit (unit cage))`. The reason for the double `+unit` is that Arvo interprets `~` to mean the scry path couldn't be resolved, and interprets `[~ ~]` to means it resolved to nothing. In either case the dotket expression which initiated the scry will crash. The `$cage` will contain the actual data to return.

An ordinary `+on-peek` arm, therefore, begins like so:

```hoon
++  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ....
```

Typically, you'd handle the `$path` similarly to `+on-watch`, as we discussed in the lesson on subscriptions. You'd use something like a [wutlus](/hoon/rune/wut#wutlus) `?+` expression to test the value of the `$path`, defining your scry endpoints like so:

```hoon
?+    path  (on-peek:def path)
    [%x %some %path ~]
  ::  ...
    [%x %foo ~]
  ::  ...
    [%x %blah @ ~]
  =/  =ship  (slav %p i.t.t.path)
  ::  ...
::  ...
```

Each endpoint would then compose the `(unit (unit cage))`. The simplest way to format it is like:

```hoon
``noun+!>('some data')
```

If it requires a more complex expression to retrieve or compose the data, you can do something like:

```hoon
:^  ~  ~  %some-mark
!>  ^-  some-type
:+  'foo'
  'bar'
'baz'
```

Previously we discussed custom mark files. Such mark files are most commonly used when the data might be accessed through Eyre's HTTP API, and therefore require JSON conversion methods. We cover such things separately in the [Full-Stack Walkthrough](/build-on-urbit/app-school-full-stack), but note that if that's the case for your agent, you may wish to also have your scry endpoints return data with your custom `$mark` so it can easily be converted to JSON when accessed from the web.

In some cases, typically with scry `$path`s that contain wildcards like the `[%x %blah @ ~]` example above, your agent may not always be able to find the requested data. In such cases, you can just produce a cell of `[~ ~]` for the `(unit (unit cage))`. Keep in mind, however, that this will result in a crash for the dotket expression which initiated the scry. In some cases you may want that, but in other cases you may not, so instead you could wrap the data inside the `$vase` in a `+unit` and have *that* be null instead. It all depends on the needs of your particular application and its clients.

## Example <a href="#example" id="example"></a>

Here's a simple example agent with three scry endpoints:

<details>

<summary>/app/peeker.hoon</summary>

```hoon
/+  default-agent, dbug
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 data=(map @p @t)]
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  `this
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?>  =(src.bowl our.bowl)
  ?+    mark  (on-poke:def mark vase)
      %noun
    `this(data (~(put by data) !<([@p @t] vase)))
  ==
::
++  on-watch  on-watch:def
++  on-leave  on-leave:def
::
++  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ?+    path  (on-peek:def path)
      [%x %all ~]  ``noun+!>(data)
  ::
      [%x %has @ ~]
    =/  who=@p  (slav %p i.t.t.path)
    ``noun+!>(`?`(~(has by data) who))
  ::
      [%x %get @ ~]
    =/  who=@p  (slav %p i.t.t.path)
    =/  maybe-res  (~(get by data) who)
    ?~  maybe-res
      [~ ~]
    ``noun+!>(`@t`u.maybe-res)
  ==
::
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

</details>

The agent's `+on-poke` arm takes a cell of `[@p @t]` and saves it in the agent's state, which contains a `(map @p @t)` called `data`. The `+on-peek` arm is:

```hoon
++  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ?+    path  (on-peek:def path)
      [%x %all ~]  ``noun+!>(data)
  ::
      [%x %has @ ~]
    =/  who=@p  (slav %p i.t.t.path)
    ``noun+!>(`?`(~(has by data) who))
  ::
      [%x %get @ ~]
    =/  who=@p  (slav %p i.t.t.path)
    =/  maybe-res  (~(get by data) who)
    ?~  maybe-res
      [~ ~]
    ``noun+!>(`@t`u.maybe-res)
  ==
```

It defines three scry endpoints, all using a `%x` `$care`: `/x/all`, `/x/has/[ship]`, and `/x/get/[ship]`. The first will simply return the entire `(map @p @t)` in the agent's state. The second will check whether the given ship is in the map and produce a `?`. The third will produce the `@t` for the given `@p` if it exists in the map, or else return `[~ ~]` to indicate the data doesn't exist, producing a crash in the dotket expression.

Let's try it out. Save the agent above as `/app/peeker.hoon` in the `%base` desk, `|commit %base` and start the agent with `|rein %base [& %peeker]`.

First, let's add some data to the map:

```
> :peeker [~zod 'foo']
>=
> :peeker [~nut 'bar']
>=
> :peeker [~wet 'baz']
>=
```

Now if we use `+dbug` to inspect the state, we'll see the data has been added:

```
>   [n=[p=~zod q='foo'] l=[n=[p=~wet q='baz'] l={} r={[p=~nut q='bar']}] r=~]
> :peeker +dbug [%state %data]
>=
```

Next, let's try the `/x/all` scry endpoint:

```
> .^((map @p @t) %gx /=peeker=/all/noun)
[n=[p=~zod q='foo'] l=[n=[p=~wet q='baz'] l={} r={[p=~nut q='bar']}] r=~]
```

The `/x/has/[ship]` endpoint:

```
> .^(? %gx /=peeker=/has/~zod/noun)
%.y
> .^(? %gx /=peeker=/has/~wet/noun)
%.y
> .^(? %gx /=peeker=/has/~nes/noun)
%.n
```

And finally, the `/x/get/[ship]` endpoint:

```
> .^(@t %gx /=peeker=/get/~zod/noun)
'foo'
> .^(@t %gx /=peeker=/get/~wet/noun)
'baz'
```

We'll now try scrying for a ship that doesn't exist in the map.

```
~zod:dojo> .^(@t %gx /=peeker=/get/~nes/noun)
bail: 4

bail: 2
dojo: failed to process input
```

## Summary <a href="#summary" id="summary"></a>

* Scries are read-only requests to vanes or agents which can be done inside any code, during its evaluation.
* Scries are performed with the dotket (`.^`) rune.
* Scries will fail if the scry endpoint does not exist, the requested data does not exist, or the data does not nest in the return type specified.
* Scries can only be performed on the local ship, not on remote ships.
* Gall scries with an agent name in the `/[desk]` field and without an extra empty element at the beginning of the path will be passed to that agent's `+on-peek` arm for handling.
* Gall scries with a `%x` `$care` take a `$mark` at the end of the scry `$path`, telling Gall to convert the data returned by the scry endpoint to the mark specified.
* The `+on-peek` arm takes a `$path` with the `$care` in the head and the `$path` part of the scry in the tail, like `/x/some/path`.
* The `+on-peek` arm produces a `(unit (unit cage))`. The outer `+unit` is null if the scry endpoint does not exist, and the inner `+unit` is null if the data does not exist.

## Exercises <a href="#exercises" id="exercises"></a>

* Have a read through the [scry guide](/urbit-os/kernel/arvo/scry).
* Have a look at Gall's [scry reference](/urbit-os/kernel/gall/scry).
* Have a read through the [dotket rune documentation](/hoon/rune/dot#dotket).
* Run through the [example](#example) yourself if you've not done so already.
* Try adding another scry endpoint to the `%peeker` agent, which uses a [`+wyt:by`](/hoon/stdlib/2i#wytby) map function to produce the number of items in the `.data` map.
* Have a look through the `+on-peek` arms of some other agents on your ship, and try performing some scries to some of the endpoints.


# 11. Failure

Guide to Gall agent error handling through the +on-fail arm and helper core patterns.

In this lesson we'll cover the last agent arm we haven't touched on yet: `+on-fail`. We'll also touch on one last concept, which is the "helper core".

## Failures

When crashes or errors occur in certain cases, Gall passes them to an agent's `+on-fail` arm for handling. This arm is very seldom used, almost all agents leave it for default-agent to handle, which just prints the error message to the terminal. While you're unlikely to use this arm, we'll briefly go over its behavior for completeness.

`+on-fail` takes a `$term` error message and a `$tang`, typically containing a stack trace, and often with additional messages about the error. If it weren't delegated to `+on-fail:def`, it would begin with:

```hoon
++  on-fail
  |=  [=term =tang]
  ^-  (quip card _this)
  ::  ...
```

Gall calls `+on-fail` in four cases:

* When there's a crash in the `+on-arvo` arm.
* When there's a crash in the `+on-agent` arm.
* When there's a crash in the `+on-leave` arm.
* When an agent produces a `%watch` card but the `$wire`, ship, agent and `$path` specified are the same as an existing subscription.

For an `+on-arvo` failure, the `$term` will always be `%arvo-response`, and the `$tang` will contain a stack trace.

For `+on-agent`, the `$term` will be the head of the `$sign` (`%poke-ack`, `%fact`, etc). The `$tang` will contain a stack trace and a message of "closing subscription".

For an `+on-leave` failure, the `$term` will always be `%leave`, and the `$tang` will contain a stack trace.

For a `%watch` failure, the `$term` will be `%watch-not-unique`. The `$tang` will include a message of "subscribe wire not unique", as well as the agent name, the `$wire`, the target ship and the target agent.

How you might handle these cases (if you wanted to manually handle them) depends on the purpose of your particular agent.

### Helper core <a href="#helper-core" id="helper-core"></a>

Back in the lesson on lustar virtual arms, we briefly mentioned a common pattern is to define a deferred expression for a helper core named `hc` like:

```hoon
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
    hc    ~(. +> bowl)
```

The name `do` is also used frequently besides `hc`.

A helper core is a separate core composed into the subject of the agent core, containing useful functions for use by the agent arms. Such a helper core would typically contain functions that would only ever be used internally by the agent - more general functions would usually be included in a separate `/lib` library and imported with a [faslus](/hoon/rune/fas#faslus) (`/+`) rune. Additionally, you might recall that the example agent of the [subscriptions lesson](/build-on-urbit/app-school/8-subscriptions) used a barket (`|^`) rune to create a core in the `+on-poke` arm with a separate `+handle-poke` arm. That approach is typically used when functions will only be used in that one arm. The helper core, on the other hand, is useful when functions will be used by multiple agent arms.

The conventional pattern is to have the helper core *below* the agent core, so the structure of the agent file is like:

```
[imports]
[state types core]
[agent core]
[helper core]
```

Recall that the build system will implicitly compose any discrete expressions. If we simply added the helper core below the agent core, the agent core would be composed into the subject of the helper core, which is the opposite of what we want. Instead, we must inversely compose the two cores with a [tisgal](/hoon/rune/tis#tisgal) (`=<`) rune. We add the tisgal rune directly above the agent core like:

```hoon
::  ...
=<
|_  =bowl:gall
+*  this      .
    def   ~(. (default-agent this %.n) bowl)
    hc    ~(. +> bowl)
++  on-init
::  ...
```

We can then add the helper core below the agent core. The helper core is most typically a door like the agent core, also with the `$bowl` as its sample. This is just so any functions you define in it have ready access to the `$bowl`. It would look like:

```hoon
|_  =bowl:gall
++  some-function  ::  ...
++  another        ::  ...
++  etc            ::  ...
--
```

Back in the lustar virtual arm of the agent core, we give it a deferred expression name of `hc` and call it like so:

```hoon
hc  ~(. +>  bowl)
```

To get to the helper core we composed from within the door, we use a [censig](/hoon/rune/cen#censig) expression to call `+>` of the subject (`.`) with the `$bowl` as its sample. After that, any agent arms can make use of helper core functions by calling them like `(some-function:hc ...)`.

### Summary <a href="#summary" id="summary"></a>

* `+on-fail` is called in certain cases of crashes or failures.
* Crashes in the `+on-agent`, `+on-arvo`, or `+on-watch` arms will trigger a call to `+on-fail`.
* A non-unique `%watch` `$card` will also trigger a call to `+on-fail`.
* `+on-fail` is seldom used - most agents just leave it to `%default-agent` to handle, which just prints the error to the terminal.
* A helper core is an extra core of useful functions, composed into the subject of the agent core.
* Helper cores are typically placed below the agent core, and composed with a tisgal (`=<`) rune.
* The helper core is typically a door with the `$bowl` as a sample.
* The helper core is typically given a name of `hc` or `do` in the lustar virtual arm of the agent core.


# 12. Next Steps

End of App School I.

We've now covered all the arms of a Gall agent, and everything you need to know to start writing your own agent.

The things haven't touched on yet are front-end development and integration, Eyre's HTTP API for communicating with agents from the web, and dealing with JSON data. The [full-stack walkthrough](/build-on-urbit/app-school-full-stack) covers these aspects of Urbit app development, and it also puts into practice many of the concepts we've discussed in this guide, so you might like to work through that next. In addition to that walkthrough, you can refer to the following documents for help writing a web front-end for your app:

* [Eyre's external API reference](/urbit-os/kernel/eyre/external-api-ref) - This explains Eyre's HTTP API, through which a browser or other HTTP client can interact with a Gall agent.
* [Eyre's usage guide](/urbit-os/kernel/eyre/guide) - This walks through examples of using Eyre's HTTP API.
* [JSON guide](/hoon/json-guide) - This walks through the basics of converting Hoon data structures to JSON, for use with a web client. It also covers JSON conversion methods in mark files.
* [Zuse reference](/hoon/zuse) - This contains documentation of all JSON encoding and decoding functions included in the `/sys/zuse.hoon` utility library.
* [The software distribution guide](/build-on-urbit/userspace/dist/software-distribution) - This covers everything you need to know to distribute apps to other ships. It includes details of bundling a web front-end and serving it to the user in the browser.
* [The HTTP API guide](/build-on-urbit/tools/js-libs/http-api-guide) - This is a reference and guide to using the `@urbit/http-api` NPM module.
* [The Sail guide](/hoon/sail) - Sail is a domain-specific language for composing XML structure in Hoon. It can be used to compose front-ends for Urbit apps directly in agents, as an alternative approach to having a separate Javascript app.

In addition to these documents about creating a web-based user interface for your app, there are some other guides you might like to have a look at:

* [Threads guide](/urbit-os/base/threads/basics/fundamentals) - Threads are like transient agents, typically used for handling complex I/O functionality for Gall agents - like interacting with an external HTTP API.
* [The software distribution guide](/build-on-urbit/userspace/dist/software-distribution) - This explains how to set up a desk for distribution, so other people can install your app.


# Appendix: Types

Explanation of types commonly used in Gall agents. Vase, mark, cage, quip, path, wire.

This document explains a few of the types commonly used in Gall agents. In addition to these, the [Data Types](/urbit-os/kernel/gall/data-types) section of the Gall vane documentation is a useful reference. In particular, the whole [`+agent`](/urbit-os/kernel/gall/data-types#agent) subsection, as well as [`$bowl`](/urbit-os/kernel/gall/data-types#bowl), [`$boat`](/urbit-os/kernel/gall/data-types#boat), and [`$bitt`](/urbit-os/kernel/gall/data-types#bitt).

## `$vase` <a href="#vase" id="vase"></a>

Vases are used to encapsulate *dynamically typed* data - they let typed data be moved around in contexts where you can't know the type ahead of time, and therefore can't have a *static* type.

Vases are used extensively - almost all data your agent will send and received is wrapped in a vase.

A `$vase` is just a cell with data in the tail and the type of the data in the head. Its formal definition is:

```hoon
+$  vase  [p=type q=*]
```

Here's what it looks like if we bunt a `$vase` in the dojo:

```
> *vase
[#t/* q=0]
```

There are two simple runes used to create and unpack vases. We'll look at each of these next.

### Create a `$vase` <a href="#create-a-vase" id="create-a-vase"></a>

The [zapgar](/hoon/rune/zap#zapgar) rune (`!>`) takes a single argument of any noun, and wraps it in a vase. For example, in the dojo:

```
> !>([1 2 3])
[#t/[@ud @ud @ud] q=[1 2 3]]

> !>('foo')
[#t/@t q=7.303.014]

> !>([[0xdead 0xb33f] 'foo'])
[#t/[[@ux @ux] @t] q=[[57.005 45.887] 7.303.014]]

> !>(foo='bar')
[#t/foo=@t q=7.496.034]
```

You would typically use `!>` as part of a [`$cage`](#cage) when you're constructing a `$card` like a poke or a `%fact` gift to be sent off.

### Extract data from `$vase` <a href="#extract-data-from-vase" id="extract-data-from-vase"></a>

The [zapgal](/hoon/rune/zap#zapgal) rune (`!<`) takes two arguments: A mold specifying the type to try and extract the data as, and the `$vase` to be extracted.

Let's look at an example in the dojo. First, let's create a `$vase` of `[@t @ux @ud]`:

```
> =myvase !>(['foo' 0xabcd 123])
> myvase
[#t/[@t @ux @ud] q=[7.303.014 43.981 123]]
```

Next, let's try extracting our vase:

```
> !<  [@t @ux @ud]  myvase
['foo' 0xabcd 123]
```

Now let's try asking for a `@p` rather than `@t`:

```
> !<  [@p @ux @ud]  myvase
-need.@p
-have.@t
nest-fail
```

As you can see, it will crash if the type does not nest. Note that rather than using `!<`, you can also just clam the tail of the `$vase` like:

```
> ((trel @t @ux @ud) +.myvase)
[p='foo' q=0xabcd r=123]
```

The only problem is that you can't tell if the auras were wrong:

```
> ((trel @p @ud @ux) +.myvase)
[p=~sibtel-tallyd q=43.981 r=0x7b]
```

You'd typically use `!<` on the data in `$card`s that come in from other ships, agents, etc.

## `$mark` <a href="#mark" id="mark"></a>

The `$mark` type is just a `@tas` like `%foo`, and specifies the Clay filetype of some data. The `$mark` corresponds to a mark file in the `/mar` directory, so a `$mark` of `%foo` corresponds to `/mar/foo/hoon`. Mark files are used for saving data in Clay, validating data sent between agents or over the network, and converting between different data types. For more information about mark files, you can refer to the [marks section of the Clay documentation](/urbit-os/kernel/clay/marks).

## `$cage` <a href="#cage" id="cage"></a>

A `$cage` is a cell of a [`$mark`](#mark) and a [`$vase`](#vase), like `[%foo !>('bar')]`. The data in the `$vase` should match the data type of the specified mark.

Most data an agent sends will be in a `$cage`, and most data it receives will arrive in a `$cage`. The `$mark` may be used to validate or convert the data in the `$vase`, depending on the context.

## `+quip` <a href="#quip" id="quip"></a>

`+quip` is a mold-builder. A `(quip a b)` is equivalent to `[(list a) b]`, it's just a more convenient way to specify it. Most arms of an agent return a `(quip card _this)`, which is a list of effects and a new state.

## `$path` <a href="#path" id="path"></a>

The `$path` type is formally defined as:

```hoon
+$  path  (list knot)
```

A knot is a `@ta` text atom (see the [strings guide](/hoon/strings) for details), so a `$path` is just a list of text. Rather than having to write `[~.foo ~.bar ~.baz ~]` though, it has its own syntax which looks like `/foo/bar/baz`.

A `$path` is similar to a filesystem path in Unix, giving data a location in a nested hierarchy. In Arvo though, they're not only used for files, but are a more general type used for several different purposes. Its elements have no inherent significance, it depends on the context. In a Gall agent, a `$path` is most commonly a subscription path - you might subscribe for updates to `/foo/bar` on another agent, or another agent might subscribe to `/baz` on your agent.

A `$path` might just be a series of fixed `@ta` like `/foo/bar`, but some elements might also be variable and include encoded atoms, or some other datum. For example, you might like to include a date in the path like `/updates/~2021.10.31..07.24.27..db68`. Other agents might create the path by doing something like:

```hoon
/update/(scot %da now.bowl)
```

Then, when you get a subscription request, you might do something like:

```hoon
?+    path  !!
    [%updates @ ~]
  =/  date=@da  (slav %da i.t.path)
  ...(rest of code)...
```

See the [encoding in text](/hoon/strings#encoding-in-text) and [decoding from text](/hoon/strings#decoding-from-text) sections of the Strings guide for more information on dealing with atoms encoded in strings.

Aside from using function calls when constructing a `$path` as demonstrated above, you can also insert text you're previously stored with `=/` or what have you, simply by enclosing them in brackets. For example, in the dojo:

```
> =const ~.bar
> `$path`/foo/[const]/baz
/foo/bar/baz
```

## `$wire` <a href="#wire" id="wire"></a>

The type of a wire is formally defined as:

```hoon
+$  wire  path
```

So, a `$wire` is just a [`$path`](#path), type-wise they're exactly the same. The reason there's a separate `$wire` type is just to differentiate their purpose. A `$wire` is a path for responses to requests an agent initiates. If you subscribe to the `$path` `/some/path` on another agent, you also specify `/some/wire`. Then, when that agent sends out updates to subscribers of `/some/path`, your agent receives them on `/some/wire`.

More formally, `$wire`s are used by Arvo to represent an event cause, and therefore return path, in a call stack called a [`$duct`](/urbit-os/kernel/arvo#duct). Inter-vane communications happen over `$duct`s as `$move`s, and Gall converts the `$card`s produced by agents into such `$move`s behind the scenes. A detailed understanding of this system is not necessary to write Gall agents, but if you're interested it's comprehensively documented in the [Arvo overview](/urbit-os/kernel/arvo) and [move trace tutorial](/urbit-os/kernel/arvo/move-trace).

For agents, the `$wire` is specified in the second argument of a `%pass` `$card`. It's used for anything you can `%pass`, such as `%poke`s, `%watch`es, and `%arvo` notes. For example:

```hoon
[%pass /this/is/wire %agent [~zod %foobar] %watch /this/is/path]
::
[%pass /this/is/wire %agent [~zod %foobar] %poke %foo !>('hello')]
::
[%pass /this/is/wire %arvo %b %wait (add now.bowl ~m1)]
```

The `+on-agent` and `+on-arvo` arms of the agent core include a `$wire` in their respective sample. Responses from agents come in to the former, and responses from vanes come in to the latter.


# App School II (Full-Stack)

Walkthrough for building a full-stack Urbit application with a Gall backend and React frontend, demonstrating how to create complete web applications on Urbit.

In [App School I](/build-on-urbit/app-school) you learned about the structure of a Gall agent. In this course, we'll walk through the writing of a full agent that serves as the backend for a React application which we'll also write.

We'll write a simple journal app, called "Journal". In the browser, users will be able to add plaintext journal entires organized by date. They'll be able to scroll through their entires in ascending date order, with more entries loaded each time the bottom of the list is reached. They'll be able to edit and delete old entries, and search through entries by specifying a date range.

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/entries.png)

The Journal app we'll be writing can be installed from `~pocwet/journal`, and its source code is available [here](https://github.com/urbit/docs-examples/tree/main/journal-app).

This walkthrough doesn't cover every aspect of building the app in full depth. Rather, its purpose is to demonstrate the process of creating a full-stack app, showing how everything fits together, and how concepts you learned in App School I are applied in practice. Consequently, the example app is fairly simple and runs on a local ship only, rather than one with more complex inter-ship networking.

Each section of this walkthrough will list additional resources and learning material at the bottom of the page, which will cover the concepts discussed in a more comprehensive manner.

Here is the basic structure of the app we'll be building:

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/journal-app-diagram.svg)

## Sections <a href="#sections" id="sections"></a>

* [1. Types](/build-on-urbit/app-school-full-stack/1-types) - Creating the `/sur` structure file for our `%journal` agent.
* [2. Agent](/build-on-urbit/app-school-full-stack/2-agent) - Creating the `%journal` agent itself.
* [3. JSON](/build-on-urbit/app-school-full-stack/3-json) - Writing a library to convert between our agent's marks and JSON. This lets our React front-end poke our agent, and our agent send updates back to it.
* [4. Marks](/build-on-urbit/app-school-full-stack/4-marks) - Creating the mark files for the pokes our agent takes and updates it sends out.
* [5. Eyre](/build-on-urbit/app-school-full-stack/5-eyre) - A brief overview of how the webserver vane Eyre works.
* [6. React App Setup](/build-on-urbit/app-school-full-stack/6-react-setup) - Creating a new React app, installing the required packages, and setting up some basic things for our front-end.
* [7. React App Logic](/build-on-urbit/app-school-full-stack/7-app-logic) - Analyzing the core logic of our React app, with particular focus on using methods of the `Urbit()` class from `@urbit/http-api` to communicate with our agent.
* [8. Desk and Glob](/build-on-urbit/app-school-full-stack/8-desk) - Building and "globbing" our front-end, and putting together a desk for distribution.
* [9. Summary](/build-on-urbit/app-school-full-stack/9-final) - Some final comments and additional resources.


# 1. Types

First step in building a full-stack Urbit app. Defining data structures in /sur files including basic types, action commands, update events, and state organization.

The best place to start when building a new agent is its type definitions in its `/sur` structure file. The main things to think through are:

1. What basic types of data does my agent deal with?
2. What actions/commands does my agent need to handle?
3. What updates/events will my agent need to send out to subscribers?
4. What does my agent need to store in its state?

Let's look at each of these questions in turn, and put together our agent's `/sur` file, which we'll call `/sur/journal.hoon`.

## 1. Basic types <a href="#id-1-basic-types" id="id-1-basic-types"></a>

Our journal entries will just be plain text, so a simple `@t` will work fine to store their contents. Entries will be organized by date, so we'll also need to decide a format for that.

One option would be to use a `@da`, and then use the date functions included in the `@urbit/api` NPM package on the front-end to convert them to ordinary Javascript `Date` objects. In this case, to keep it simple, we'll just use the number of milliseconds since the Unix Epoch as an `$atom`, since it's natively supported by the Javascript `Date` object.

The structure for a journal entry can therefore be:

```hoon
+$  id  @
+$  txt  @t
+$  entry  [=id =txt]
```

## 2. Actions <a href="#id-2-actions" id="id-2-actions"></a>

Now that we know what a journal entry looks like, we can think about what kind of actions/commands our agent will handle in its `+on-poke` arm. For our journal app, there are three basic things we might do:

1. Add a new journal entry.
2. Edit an existing journal entry.
3. Delete an existing journal entry.

We can create a tagged union structure for these actions, like so:

```hoon
+$  action
  $%  [%add =id =txt]
      [%edit =id =txt]
      [%del =id]
  ==
```

## 3. Updates <a href="#id-3-updates" id="id-3-updates"></a>

Updates are a little more complicated than our actions. Firstly, our front-end needs to be able to retrieve an initial list of journal entries to display. Once it has that, it also needs to be notified of any changes. For example, if a new entry is added, it needs to know so it can add it to the list it's displaying. If an entry gets deleted, it needs to remove it from the list. Etc.

The simplest approach to the initial entries is just a `(list entry)`. Then, for the subsequent updates, we could send out the `$action`. Since an `$action` is a tagged union, it's simpler to have all updates be a tagged union, so when we get to doing mark conversions we can just switch on the head tag. Therefore, we can define an `$update` structure like so:

```hoon
+$  update
  $%  action
      [%jrnl list=(list entry)]
  ==
```

There's one drawback to this structure. Suppose either an agent on a remote ship or an instance of the front-end client is subscribed for updates, and the network connection is disrupted. In the remote ship case, Gall will only allow so many undelivered messages to accumulate in Ames before it automatically kicks the unresponsive subscriber. In the front-end case, the subscription will also be ended if enough unacknowledged messages accumulate, and additionally the client may sometimes need to establish an entirely new connection with the ship, discarding existing subscriptions. When this happens, the remote ship or web client has no way to know how many (if any) updates they've missed.

The only way to resynchronize their state with ours is to discard their existing state, refetch the entire initial state once again, and then resubscribe for updates. This might be fine if the state of our agent is small, but it becomes a problem if it's very large. For example, if our agent holds tens of thousands of chat messages, having to resend them all every time anyone has connectivity issues is quite inefficient.

One solution to this is to keep an "update log". Each update can be tagged with the time it occurred, and stored in our agent's state, separately to the entries. If an agent or web client needs to resynchronize with our agent, it can just request all updates since the last one it received. Our agent is local-only and doesn't have a huge state so it might not be strictly necessary, but we'll use it to demonstrate the approach.

We can define a logged update like so, where the `@` is the update timestamp in milliseconds since the Unix Epoch:

```hoon
+$  logged  (pair @ action)
+$  update
  %+  pair  @
  $%  action
      [%jrnl list=(list entry)]
      [%logs list=(list logged)]
  ==
```

## 4. State <a href="#id-4-state" id="id-4-state"></a>

We need to store two things in our state: the journal entries and the update log. We could just use a couple of `+map`s like so:

```hoon
+$  journal  (map id txt)
+$  log  (map @ action)
```

Ordinary `+map`s are fine if we just want to access one value at a time, but we want to be able to:

1. Retrieve only some of the journal entries at a time, so we can have "lazy loading" in the front-end, loading more entries each time the user scrolls to the bottom of the list.
2. Retrieve only logged updates newer than a certain time, in the case where the subscription is interrupted due to connectivity issues.
3. Retrieve journal entries between two dates.

Maps are ordered by the hash of their key, so if we convert them to a list they'll come out in seemingly random order. That means we'd have to convert the map to a list, sort the list, and then iterate over it again to pull out the items we want. We could alternatively store things in a list directly, but retrieving or modifying arbitrary items would be less efficient.

To solve this, rather than using a `+map` or a `+list`, we can use an *ordered map*. The mold builder for an ordered map is a `+mop`, and it's included in the [`zuse.hoon`](https://github.com/urbit/urbit/blob/master/pkg/arvo/sys/zuse.hoon#L5284) utility library rather than the standard library.

A `+mop` is defined similarly to a `+map`, but it takes an extra argument in the following manner:

```hoon
((mop key-mold val-mold) comparator-gate)
```

The gate is a binary gate which takes two keys and produces a `?`. The comparator is used to decide how to order the items in the mop. In our case, we'll create a `$journal` and `$log` `+mop` like so:

```hoon
+$  journal  ((mop id txt) gth)
+$  log  ((mop @ action) lth)
```

The entries in `$journal` are arranged in ascending time order using `+gth`, so the right-most item is the newest. The `$log` `+mop` contains the update log, and is arranged in descending time order, so the right-most item is the oldest.

We'll look at how to use ordered maps later when we get to writing the agent itself.

## Conclusion <a href="#conclusion" id="conclusion"></a>

When we put each of these parts together, we have our complete `/sur/journal.hoon` file:

<details>

<summary>/sur/journal.hoon</summary>

```hoon
|%
:: Basic types of the data we're dealing with
::
+$  id  @
+$  txt  @t
+$  entry  [=id =txt]
:: Poke actions
::
+$  action
  $%  [%add =id =txt]
      [%edit =id =txt]
      [%del =id]
  ==
:: Types for updates to subscribers or returned via scries
::
+$  logged  (pair @ action)
+$  update
  %+  pair  @
  $%  action
      [%jrnl list=(list entry)]
      [%logs list=(list logged)]
  ==
:: Types for our agent's state
::
+$  journal  ((mop id txt) gth)
+$  log  ((mop @ action) lth)
--
```

</details>

## Resources <a href="#resources" id="resources"></a>

* [App School I /sur section](/build-on-urbit/app-school/7-sur-and-marks) - This section of App School covers writing a `/sur` structure library for an agent.
* [Ordered map functions in `/sys/zuse.hoon`](https://github.com/urbit/urbit/blob/master/pkg/arvo/sys/zuse.hoon#L5284-L5688) - This section of `zuse.hoon` contains all the functions for working with `+mop`s, and is well commented.


# 2. Agent

Building the Gall agent backend for a full-stack app including imports, state management, poke handling, subscription patterns, and scry endpoints for data retrieval.

Now that we have our agent's types defined and have thought through its behavior, we can write the `%journal` agent itself.

## Imports <a href="#imports" id="imports"></a>

```hoon
/-  *journal
/+  default-agent, dbug, agentio
```

We first import the `/sur/journal.hoon` file we previously created and expose its structures. We import the standard `/lib/default-agent.hoon` and `/lib/dbug.hoon`, and also an additional library called agentio.

Agentio contains a number of convenience functions to make common agent tasks simpler. For example, rather than writing out the full `$card`s when sending `%fact`s to subscribers, we can call `+fact` in agentio with the `$cage` and `$path`s and it will compose them for us. There are many more functions in agentio than we'll use here - you can have a look through the library in [`/base/lib/agentio.hoon`](https://github.com/urbit/urbit/blob/master/pkg/base-dev/lib/agentio.hoon) to see what else it can do.

## State and type core <a href="#state-and-type-core" id="state-and-type-core"></a>

```hoon
|%
+$  versioned-state
    $%  state-0
    ==
+$  state-0  [%0 =journal =log]
+$  card  card:agent:gall
++  j-orm  ((on id txt) gth)
++  log-orm  ((on @ action) lth)
++  unique-time
  |=  [=time =log]
  ^-  @
  =/  unix-ms=@
    (unm:chrono:userlib time)
  |-
  ?.  (has:log-orm log unix-ms)
    unix-ms
  $(time (add unix-ms 1))
--
```

As we discussed in the previous section, our state will contain a `$journal` structure containing all our journal entries, and a `$log` structure containing the update log. These are both *ordered maps*, defined as `((mop id txt) gth)` and `((mop @ action) lth)` respectively. We can therefore define our "versioned state" as `[%0 =journal =log]`, in the usual manner.

We've define `$card` for convenience as usual, and we've also added three more arms. The first two relate to our two ordered maps. If you'll recall, an ordinary `+map` is called with the `+by` door in the standard library, like so:

```hoon
(~(get by foo) %bar)
```

An ordered map uses the `+on` gate in `/sys/zuse.hoon` rather than `+by`, and its invocation is slightly different. It must first be setup in a similar manner to the `+mop` type, by providing it the key/value molds and comparator gates. Once that's done, its individual functions can be called with the `+mop` and arguments, like:

```hoon
(get:((on @ud @ud) gth) foo %bar)
```

This is quite a cumbersome expression to use every time we want to interact with our `+mop`. To make it easier, we can store the `((on @ud @ud) gth)` part in an arm, and then when we need to use it we can just do `(get:arm-name foo %bar)`. In this case, we've done one each of our ordered maps like so:

```hoon
++  j-orm  ((on id txt) gth)
++  log-orm  ((on @ action) lth)
```

The last arm in our state definition core is `+unique-time`. Since we'll use `.now.bowl` to derive the timestamp for updates, we run into an issue if multiple pokes arrive in a single Arvo event. In that case, `.now.bowl` would be the same for each poke, so they'd be given the same key and override each other in the `+mop`. To avoid this, `+unique-time` is just a simple recursive function that will increment the timestamp by one millisecond if the key already exists in the `$log` `+mop`, ensuring all updates get unique timestamps and there are no collisions.

## Agent core setup <a href="#agent-core-setup" id="agent-core-setup"></a>

```hoon
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %|) bowl)
    io    ~(. agentio bowl)
++  on-init  on-init:def
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-vase=vase
  ^-  (quip card _this)
  `this(state !<(versioned-state old-vase))
::
```

Here we setup our agent core and define the three lifecycle arms. Since we only have a single state version at present, these are very simple functions. You'll notice in our `+*` arm, along with the usual `.this` and `.def`, we've also setup the agentio library we imported, giving it the bowl and an alias of `.io`.

## Pokes <a href="#pokes" id="pokes"></a>

```hoon
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  |^
  ?>  (team:title our.bowl src.bowl)
  ?.  ?=(%journal-action mark)  (on-poke:def mark vase)
  =/  now=@  (unique-time now.bowl log)
  =/  act  !<(action vase)
  =.  state  (poke-action act)
  :_  this(log (put:log-orm log now act))
  ~[(fact:io journal-update+!>(`$update`[now act]) ~[/updates])]
  ::
  ++  poke-action
    |=  act=action
    ^-  _state
    ?-    -.act
        %add
      ?<  (has:j-orm journal id.act)
      state(journal (put:j-orm journal id.act txt.act))
    ::
        %edit
      ?>  (has:j-orm journal id.act)
      state(journal (put:j-orm journal id.act txt.act))
    ::
        %del
      ?>  (has:j-orm journal id.act)
      state(journal +:(del:j-orm journal id.act))
    ==
  --
::
```

Here we have our `+on-poke` arm, where we handle `$action`s. Since our `%journal` agent is intended for local use only, we make sure only our ship or our moons may perform actions with:

```hoon
?>  (team:title our.bowl src.bowl)
```

We haven't yet written our mark files, but our mark for `$action`s will be `%journal-action`, so we make sure that's what we've received and if not, call `+on-poke:def` to crash with an error message. We make sure the the timestamps are unique with our `+unique-time` function described earlier, and then we extract the poke's `$vase` to an `$action` structure and call `+poke-action` to handle it. We've made `+on-poke` a door with a separate `+poke-action` arm to make the logic a little simpler, but in principle we could have had it all directly inside the main `+poke-action` gate, or even separated it out into a helper core below.

The logic in `+poke-action` is very simple, with three cases for each of the possible `$action`s:

* `%add` - Add a new journal entry. We check it doesn't already exist with `+has:j-orm`, and then add it to our `$journal` with `+put:j-orm`.
* `%edit` - Edit an existing journal entry. We make sure it *does* exist with `+has:j-orm`, and then override the old entry with the new one using `+put:j-orm` again.
* `%del` - Delete an existing journal entry. We make sure it exists again with `+has:j-orm`, and then use `+del:j-orm` to delete it from our `$journal` `+mop`.

Back in the main part of `+on-poke`, `+poke-action` updates the state with the new `$journal`, then we proceed to:

```hoon
:_  this(log (put:log-orm log now act))
~[(fact:io journal-update+!>(`$update`[now act]) ~[/updates])]
```

We add the timestamp to the action, converting it to a logged update. We add it to the `.log` update log using `+put:log-orm`, and also send the logged update out to subscribers on the `/updates` subscription path. We haven't written our mark files yet, but `%journal-update` is the mark we'll use for `$update`s, so we pack the `$update` in a `$vase` and add the mark to make it a `$cage`. Notice we're using the `+fact` function in agentio (which we aliased as `.io`) rather than manually composing the `%fact`.

## Subscriptions <a href="#subscriptions" id="subscriptions"></a>

```hoon
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?>  (team:title our.bowl src.bowl)
  ?+  path  (on-watch:def path)
    [%updates ~]  `this
  ==
::
```

Our subscription logic is extremely simple - we just have a single `/updates` path, which the front-end or other local agents may subscribe to. All updates get sent out on this path. We enforce local-only with the `+team:title` check.

We could have had our `+on-watch` arm send out some initial state to new subscribers, but for our front-end we'll instead fetch the initial state separately with a scry. This just makes it slightly easier if our front-end needs to resubscribe at some point - it'll already have some state in that case so we don't want it to get sent again.

## Scry Endpoints <a href="#scry-endpoints" id="scry-endpoints"></a>

```hoon
++  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ?>  (team:title our.bowl src.bowl)
  =/  now=@  (unm:chrono:userlib now.bowl)
  ?+    path  (on-peek:def path)
      [%x %entries *]
    ?+    t.t.path  (on-peek:def path)
        [%all ~]
      :^  ~  ~  %journal-update
      !>  ^-  update
      [now %jrnl (tap:j-orm journal)]
    ::
        [%before @ @ ~]
      =/  before=@  (rash i.t.t.t.path dem)
      =/  max=@  (rash i.t.t.t.t.path dem)
      :^  ~  ~  %journal-update
      !>  ^-  update
      [now %jrnl (tab:j-orm journal `before max)]
    ::
        [%between @ @ ~]
      =/  start=@
        =+  (rash i.t.t.t.path dem)
        ?:(=(0 -) - (sub - 1))
      =/  end=@  (add 1 (rash i.t.t.t.t.path dem))
      :^  ~  ~  %journal-update
      !>  ^-  update
      [now %jrnl (tap:j-orm (lot:j-orm journal `end `start))]
    ==
  ::
      [%x %updates *]
    ?+    t.t.path  (on-peek:def path)
        [%all ~]
      :^  ~  ~  %journal-update
      !>  ^-  update
      [now %logs (tap:log-orm log)]
    ::
        [%since @ ~]
      =/  since=@  (rash i.t.t.t.path dem)
      :^  ~  ~  %journal-update
      !>  ^-  update
      [now %logs (tap:log-orm (lot:log-orm log `since ~))]
    ==
  ==
::
```

Here we have our `+on-peek` arm. The scry endpoints we've defined are divided into two parts: querying the update `.log` and retrieving entries from the `$journal`. Each end-point is as follows:

* `/x/entries/all` - Retrieve all entries in the `$journal`. Our front-end will use lazy-loading and only get a few at a time, so it won't use this. It's nice to have it though, in case other agents want to get that data.
* `/x/entries/before/[before]/[max]` - Retrieve at most `/[max]` entries older than the entry on `/[before]` date. This is so our lazy-loading front-end can progressively load more as the user scrolls down the page. The Javascript front-end will format numbers without dot separators, so the path will look like `/x/entries/before/1648051573109/10`. We therefore have to use the [`+dem`](/hoon/stdlib/4i#dem) parsing `$rule` in a [`+rash`](/hoon/stdlib/4g#rash) parser to convert it to an ordinary atom. We then use the `+tap:log-orm` `+mop` function to retrieve the requested range as a list and return it as an `$update` with a `%journal-update` mark.
* `/x/entries/between/[start]/[end]` - Retrieve all journal entries between two dates. This is so our front-end can have a search function, where the user can enter a start and end date and get all the entries in between. The `+lot:j-orm` `+mop` function returns the subset of a `+mop` between the two given keys as a `+mop`, and then we call `+tap:j-orm` to convert it to a list. The `+lot:j-orm function` excludes the start and end values, so we subtract 1 from the start and add 1 to the end to make sure it includes the full range.
* `/x/updates/all` - Retrieve the entire update `.log`. Our front-end won't use this but it might be useful for other agents, so we've included it here.
* `/x/updates/since/[since]` - Retrieve all `$update`s that have happened since the specified timestamp, if any. This is so our front-end (or another agent) can resynchronize its state in the event its subscription is interrupted, without having to fetch everything from scratch again.

We don't use any of the other agent arms, so the remainder have all been passed to default-agent for handling:

```hoon
++  on-leave  on-leave:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

The full agent source can be viewed [here](https://github.com/urbit/docs-examples/blob/main/journal-app/bare-desk/app/journal.hoon).

## Resources <a href="#resources" id="resources"></a>

* [App School I](/build-on-urbit/app-school) - App School I covers all aspects of writing Gall agents in detail.
* [Ordered map functions in `zuse.hoon`](https://github.com/urbit/urbit/blob/master/pkg/arvo/sys/zuse.hoon#L5284-L5688) - This section of `zuse.hoon` contains all the functions for working with `+mop`s, and is well commented.
* [`/lib/agentio.hoon`](https://github.com/urbit/urbit/blob/master/pkg/base-dev/lib/agentio.hoon) - The agentio library in the `%base` desk contains a large number of useful functions which making writing Gall agents easier.


# 3. JSON

Building JSON conversion libraries for full-stack Urbit apps covering +enjs and +dejs functions for encoding/decoding data, mark conversion workflows, and bridging Gall agents with web frontends.

Data sent between our agent and our front-end will all be encoded as JSON. In this section, we'll briefly look at how JSON works in Urbit, and write a library to convert our agent's structures to and from JSON for our front-end.

JSON data comes into Eyre as a string, and Eyre parses it with the [`+de:json:html`](/hoon/zuse/2e_2-3#dejsonhtml) function in [`zuse.hoon`](/hoon/zuse). The hoon type it's parsed to is `$json`, which is defined as:

```hoon
+$  json                    ::  normal json value
  $@  ~                     ::  null
  $%  [%a p=(list json)]    ::  array
      [%b p=?]              ::  boolean
      [%o p=(map @t json)]  ::  object
      [%n p=@ta]            ::  number
      [%s p=@t]             ::  string
  ==                        ::
```

Once Eyre has converted the raw JSON string to a `$json` structure, it will be converted to the mark the web client specified and then delivered to the target agent (unless the mark specified is already `%json`, in which case it will be delivered directly). Outbound facts will go through the same process in reverse - converted from the agent's native mark to `$json`, then encoded in a string by Eyre using [`+en:json:html`](/hoon/zuse/2e_2-3#enjsonhtml) and delivered to the web client. The basic flow for both inbound messages (pokes) and outbound messages (facts and scry results) looks like this:

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/eyre-mark-flow-diagram.svg)

The mark conversion will be done by the corresponding mark file in `/mar` on the agent's desk. In our case it would be `/mar/journal/action.hoon` and `/mar/journal/update.hoon` in the `%journal` desk for our `%journal-action` and `%journal-update` marks, which are for the `$action` and `$update` structures we defined previously.

Mark conversion functions can be included directly in the mark file, or they can be written in a separate library, then imported and called by the mark file. We will do the latter in this case, so before we create the mark files themselves, we'll write a library called `/lib/journal.hoon` with the conversion functions.

## `$json` utilities <a href="#json-utilities" id="json-utilities"></a>

[`zuse.hoon`](/hoon/zuse) contains three main cores for converting to and from `$json`:

* [`+enjs:format`](/hoon/zuse/2d_1-5#enjsformat) - Functions to help encode data structures as `$json`.
* [`+dejs:format`](/hoon/zuse/2d_6#dejsformat) - Functions to decode `$json` to other data structures.
* [`+dejs-soft:format`](/hoon/zuse/2d_7#dejs-softformat) - Mostly the same as `+dejs:format` except the functions produce units which are null if decoding fails, rather than just crashing.

### `+enjs:format` <a href="#enjsformat" id="enjsformat"></a>

This contains ten functions for encoding `$json`. Most of them are for specific hoon data types, such as `+tape:enjs:format`, `+ship:enjs:format`, `+path:enjs:format`, etc. We'll just have a look at the two most general and useful ones: `+frond:enjs:format` and `+pairs:enjs:format`.

#### `+frond`

This function is for forming a JSON object from a single key-value pair. For example:

```
> (frond:enjs:format 'foo' s+'bar')
[%o p={[p='foo' q=[%s p='bar']]}]
```

When stringified by Eyre, this will look like:

```json
{ "foo": "bar" }
```

#### `+pairs`

This is similar to `+frond` and also forms a JSON object, but it takes multiple key-value pairs rather than just one:

```
> (pairs:enjs:format ~[['foo' n+~.123] ['bar' s+'abc'] ['baz' b+&]])
[%o p={[p='bar' q=[%s p='abc']] [p='baz' q=[%b p=%.y]] [p='foo' q=[%n p=~.123]]}]
```

When stringified by Eyre, this will look like:

```json
{
  "foo": 123,
  "baz": true,
  "bar": "abc"
}
```

Notice that we used a knot for the value of `foo` (`n+~.123`). Numbers in JSON can be signed or unsigned and integers or floating point values. The `$json` structure uses a knot so that you can decide whether a particular number should be treated as `@ud`, `@sd`, `@rs`, etc.

### `+dejs:format` <a href="#dejsformat" id="dejsformat"></a>

This core contains many functions for decoding `$json`. We'll touch on some useful families of `+dejs` functions in brief, but because there's so many, in practice you'll need to look through the [`+dejs` reference](/hoon/zuse/2d_6) to find the correct functions for your use case.

#### Number functions

* `+ne` - decode a number to a `@rd`.
* `+ni` - decode a number to a `@ud`.
* `+no` - decode a number to a `@ta`.
* `+nu` - decode a hexadecimal string to a `@ux`.

For example:

```
> (ni:dejs:format n+'123')
123
```

#### String functions

* `+sa` - decode a string to a `$tape`.
* `+sd` - decode a string containing a `@da` aura date value to a `@da`.
* `+se` - decode a string containing the specified aura to that aura.
* `+so` - decode a string to a `@t`.
* `+su` - decode a string by parsing it with the given [parsing rule](/hoon/stdlib/4f).

#### Array functions

`+ar`, `+as`, and `+at` decode a `$json` array to a `+list`, `+set`, and *n*-tuple respectively. These gates take other `+dejs` functions as an argument, producing a new gate that will then take the `$json` array. For example:

```
> ((ar so):dejs:format a+[s+'foo' s+'bar' s+'baz' ~])
<|foo bar baz|>
```

Notice that `+so` is given as the argument to `+ar`. `+so` is a `+dejs` function that decodes a `$json` string to a `$cord`. The gate resulting from `(ar so)` is then called with a `$json` array as its argument, and its product is a `(list @t)` of the elements of the array.

Many `+dejs` functions take other `+dejs` functions as their arguments. A complex nested `$json` decoding function can be built up in this manner.

#### Object functions

* `+of` - decode an object containing a single key-value pair to a head-tagged cell.
* `+ot` - decode an object to a *n*-tuple.
* `+ou` - decode an object to an *n*-tuple, replacing optional missing values with a given value.
* `+oj` - decode an object of arrays to a `+jug`.
* `+om` - decode an object to a `+map`.
* `+op` - decode an object to a `+map`, and also parse the object keys with a [parsing rule](/hoon/stdlib/4f).

For example:

```
> =js %-  need  %-  de:json:html
  '''
  {
    "foo": "hello",
    "baz": true,
    "bar": 123
  }
  '''

> %-  (ot ~[foo+so bar+ni]):dejs:format  js
['hello' 123]
```

## Our types as JSON <a href="#our-types-as-json" id="our-types-as-json"></a>

We need to decide how our `$action` and `$update` types will be represented as JSON in order to write our conversion functions. There are many ways to do this, but in this case we'll do it as follows:

### Actions <a href="#actions" id="actions"></a>

| JSON                                              | Noun                                           |
| ------------------------------------------------- | ---------------------------------------------- |
| `{"add":{"id":1648366311070,"txt":"some text"}}`  | `[%add id=1.648.366.034.844 txt='some text']`  |
| `{"edit":{"id":1648366311070,"txt":"some text"}}` | `[%edit id=1.648.366.034.844 txt='some text']` |
| `{"del":{"id":1648366311070}}`                    | `[%del id=1.648.366.034.844]`                  |

### Updates <a href="#updates" id="updates"></a>

| Noun                                                                                            | JSON                                                                                                 |
| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `[1.648.366.492.459 %add id=1.648.366.034.844 txt='some text']`                                 | `{time:1648366481425,"add":{"id":1648366311070,"txt":"some text"}}`                                  |
| `[1.648.366.492.459 %edit id=1.648.366.034.844 txt='some text']`                                | `{time:1648366481425,"edit":{"id":1648366311070,"txt":"some text"}}`                                 |
| `[1.648.366.492.459 %del id=1.648.366.034.844]`                                                 | `{time:1648366481425,"del":{"id":1648366311070}}`                                                    |
| `[1.648.366.492.459 %jrnl ~[[id=1.648.366.034.844 txt='some text'] ...]`                        | `{time:1648366481425,"entries":[{"id":1648366311070,"txt":"some text"},...]}`                        |
| `[1.648.366.492.459 %logs ~[[1.648.366.492.459 %add id=1.648.366.034.844 txt='some text'] ...]` | `{time:1648366481425,"logs":[{time:1648366481425,"add":{id":1648366311070,"txt":"some text"}},...]}` |

Now let's write our library of encoding/decoding functions.

## `/lib/journal.hoon` <a href="#libjournalhoon" id="libjournalhoon"></a>

```hoon
/-  *journal
|%
```

First, we'll import the `/sur/journal.hoon` structures we previously created. Next, we'll create two arms in our core, `+dejs-action` and `+enjs-update`, to handle incoming poke `$action`s and outgoing facts or scry result `$update`s.

### `$json` to `$action` <a href="#json-to-action" id="json-to-action"></a>

```hoon
++  dejs-action
  =,  dejs:format
  |=  jon=json
  ^-  action
  %.  jon
  %-  of
  :~  [%add (ot ~[id+ni txt+so])]
      [%edit (ot ~[id+ni txt+so])]
      [%del (ot ~[id+ni])]
  ==
```

The first thing we do is use the [`=,` rune](/hoon/rune/tis#tiscom) to expose the `+dejs:format` namespace. This allows us to reference `ot`, `ni`, etc. rather than having to write `ot:dejs:format` every time. Note that you should be careful using `=,` generally as the exposed wings can shadow previous wings if they have the same name.

We then create a gate that takes `$json` and returns a `$action` structure. Since we'll only take one action at a time, we can use the `+of` function, which takes a single key-value pair. `+of` takes a list of all possible `$json` objects it will receive, tagged by key.

For each key, we specify a function to handle its value. Ours will be objects, so we use `+ot` and specify the pairs of the key and `+dejs` function to decode it. We then cast the output to our `$action` structure.

You'll notice the nesting of these `+dejs` functions approximately reflects the nested structure of the `$json` it's decoding.

### `$update` to `$json` <a href="#update-to-json" id="update-to-json"></a>

```hoon
++  enjs-update
  =,  enjs:format
  |=  upd=update
  ^-  json
  |^
  ?+    -.q.upd  (logged upd)
      %jrnl
    %-  pairs
    :~  ['time' (numb p.upd)]
        ['entries' a+(turn list.q.upd entry)]
    ==
  ::
      %logs
    %-  pairs
    :~  ['time' (numb p.upd)]
        ['logs' a+(turn list.q.upd logged)]
    ==
  ==
  ++  entry
    |=  ent=^entry
    ^-  json
    %-  pairs
    :~  ['id' (numb id.ent)]
        ['txt' s+txt.ent]
    ==
  ++  logged
    |=  lgd=^logged
    ^-  json
    ?-    -.q.lgd
        %add
      %-  pairs
      :~  ['time' (numb p.lgd)]
          :-  'add'
          %-  pairs
          :~  ['id' (numb id.q.lgd)]
              ['txt' s+txt.q.lgd]
      ==  ==
        %edit
      %-  pairs
      :~  ['time' (numb p.lgd)]
          :-  'edit'
          %-  pairs
          :~  ['id' (numb id.q.lgd)]
              ['txt' s+txt.q.lgd]
      ==  ==
        %del
      %-  pairs
      :~  ['time' (numb p.lgd)]
          :-  'del'
          (frond 'id' (numb id.q.lgd))
      ==
    ==
  --
--
```

Our `$update` encoding function's a little more complex than our `$action` decoding function, since our `$update` structure is more complex.

Like the previous one, we use `=,` to expose the namespace of `+enjs:format`.

Our gate takes an `$update` and returns a `$json` structure. We use `|^` so we can separate out the encoding functions for individual entries (`+entry`) and individual logged actions (`+logged`).

We first test the head of the `$update`, and if it's `%jrnl` (a list of entries), we `+turn` over the entries and call `+entry` to encode each one. If it's `%logs`, we do the same, but call `+logged` for each item in the list. Otherwise, if it's just a single update, we encode it with `+logged`.

We primarily use `+pairs` to form the object, though sometimes `+frond` if it only contains a single key-value pair. We also use `+numb` to encode numerical values.

You'll notice more of our encoding function is done manually than our previous decoding function. For example, we form arrays by tagging an ordinary `+list` with `%a`, and strings by tagging an ordinary `$cord` with `%s`. This is typical when you write `$json` encoding functions, and is the reason there are far fewer `+enjs` functions than `+dejs` functions.

## Resources <a href="#resources" id="resources"></a>

* [The JSON Guide](/hoon/json-guide) - The stand-alone JSON guide covers JSON encoding/decoding in great detail.
* [The Zuse reference](/hoon/zuse) - The `/sys/zuse.hoon` reference documents all JSON-related functions in detail.
* [`+enjs:format` reference](/hoon/zuse/2d_1-5#enjsformat) - This section of the `zuse.hoon` documentation covers all JSON encoding functions.
* [`+dejs:format` reference](/hoon/zuse/2d_6) - This section of the `zuse.hoon` documentation covers all JSON *decoding* functions.
* [Eyre overview](/urbit-os/kernel/eyre) - This section of the Eyre vane documentation goes over the basic features of the Eyre vane.


# 4. Marks

Creating mark files for full-stack Urbit apps. Conversion methods between native types and JSON, defining +grow and +grab arms for custom type conversion logic.

In this section we'll write the mark files for our agent. We'll need two marks, one for poke `$action`s and one for subscription updates and scry results, both of which are `$update`s. Our `$action` mark will be called `%journal-action` and our `$update` mark will be called `%journal-update`. These will be located at `/mar/journal/action.hoon` and `/mar/journal/update.hoon`.

Note that a mark called `%foo-bar` will first be looked for in `/mar/foo-bar.hoon`, and if it's not there it will be looked for in `/mar/foo/bar.hoon`. That's why we can have a single name like `%journal-action` but have it in `/mar/journal/action.hoon`

## `%journal-action` <a href="#journal-action" id="journal-action"></a>

```hoon
/-  *journal
/+  *journal
|_  act=action
++  grow
  |%
  ++  noun  act
  --
++  grab
  |%
  ++  noun  action
  ++  json  dejs-action
  --
++  grad  %noun
--
```

First we import our `/sur/journal.hoon` structure file and also our `/lib/journal.hoon` library (containing our `$json` conversion functions). The sample of our mark door is just our `$action` structure. The `+grow` arm of a mark core, if you recall, contains methods for converting *from* our mark *to* another mark. Actions only ever come inwards in pokes, so we don't need to worry about converting an `$action` *to* `$json`. The `+grow` arm can therefore just handle the generic `%noun` case, simply returning our mark door's sample without doing anything.

`+grab`, conversely, defines methods for converting *to* our mark *from* another mark. Since `$action`s will come in from the front-end as `$json`, we need to be able to convert `$json` data to our `$action` structure. Our `/lib/journal.hoon` library contains the `+dejs-action` function for performing this conversion, so we can just specify that function for the `%json` case. We'll also define a standard `%noun` method, which will just "clam" the incoming noun with the `$action` mold. Clamming/molding coerces a noun to a type and is done by calling a mold as a function.

Lastly, `+grad` defines revision control methods, but can be delegated to another mark. Since this mark will never be used for actually storing files in Clay, we can just delegate it to the generic `%noun` mark rather than writing a proper set of `+grad` methods.

## `%journal-update` <a href="#journal-update" id="journal-update"></a>

```hoon
/-  *journal
/+  *journal
|_  upd=update
++  grow
  |%
  ++  noun  upd
  ++  json  (enjs-update upd)
  --
++  grab
  |%
  ++  noun  update
  --
++  grad  %noun
--
```

Next we have our `%journal-update` mark file. The sample of our mark door is our `$update` structure. Our `$update`s are always outbound, never inbound, so we only need to define a method for converting our `$update` structure to `$json` in the `+grow` arm, and not the opposite direction in `+grad`. Our `/lib/journal.hoon` library contains the `+enjs-update` function for performing this conversion, so we can call it with the sample `$update` as its argument. We can add `%noun` conversion methods and delegate revision control to the `%noun` mark in the same manner as our `%journal-action` mark above.

## Resources <a href="#resources" id="resources"></a>

* [The Marks section of the Clay documentation](/urbit-os/kernel/clay/marks) - This section of the Clay vane documentation covers mark files comprehensively.
* [The mark file section of the Gall Guide](/build-on-urbit/app-school/7-sur-and-marks#mark-files) - This part of App School goes through the basics of mark files.
* [The JSON Guide](/hoon/json-guide) - This also covers writing mark files to convert to/from JSON.


# 5. Eyre

Using Eyre, the HTTP server vane, in full-stack Urbit development. Covers the channel system for pokes and subscriptions, scry interface for data queries, and authentication mechanisms.

Now that we have our structure file, agent, `$json` conversion library and mark file, our back-end is complete. Before we start writing our front-end, though, we should give a brief overview of how Eyre works.

[Eyre](/urbit-os/kernel/eyre) is the HTTP server vane of Arvo. Eyre has a handful of different subsystems, but the main two are the channel system and the scry interface. These two are what we'll focus on here.

In order to use the channel system or perform scries, a web client must have authenticated with the ship's web login code (e.g. `lidlut-tabwed-pillex-ridrup`) and obtained a session cookie. Our front-end will be served directly from the ship by the `%docket` agent, so we can assume a session cookie was already obtained when the user logged into landscape, and skip over authentication.

## Channels <a href="#channels" id="channels"></a>

Eyre's channel system is the main way to interact with agents from a web client. It provides a JSON interface to the ordinary poke and subscription system for Gall agents.

First, a unique channel ID is generated by the web client (`@urbit/http-api` uses the current Unix time concatenated with a random hex string). The client then sends a poke or subscription request for the channel, and Eyre automatically opens a new channel with that ID. Once open, the client can then connect to the channel and receive any events such as poke acks, watch acks, facts from subscriptions, etc.

The new channel is an SSE ([Server Sent Event](https://html.spec.whatwg.org/#server-sent-events)) stream, and can be handled by an `EventSource` object in Javascript. The `@urbit/http-api` library we'll use abstracts this for us, so we won't need to deal with an `EventSource` object directly. The channel can handle multiple concurrent subscriptions to different agent subscription paths, and different agents can be poked through the one channel. This means a client only needs to open a single channel for all of its interactions with the ship. Each subscription is given a different ID, so they can be individually unsubscribed later.

A channel will timeout after 12 hours of inactivity, and the timeout is reset every time Eyre receives a message of any kind from the client. Additionally, each subscription on the channel may only accumulate 50 unacknowledged facts before it's considered "clogged", in which case the individual clogged subscription will be closed by Eyre after a short delay. All events of any kind which Eyre sends out on the channel must be ack'd by the client. Ack'ing one event will also ack all previous events too. The `@urbit/http-api` library we'll use automatically acks events for us, so we don't need to worry about clogged subscriptions or manually ack'ing events.

Eyre expects a particular JSON object structure for each of these different requests, but the `@urbit/http-api` library we'll use includes functions to send pokes, subscription requests, etc, so we won't need to manually construct the JSON objects in our front-end.

## Scries <a href="#scries" id="scries"></a>

Eyre's scry interface is separate to the channel system. Scries are performed by a simple GET request to a path with a format of `/~/scry/{agent}{path}.{mark}`. The specified `$mark` must include a `+mime` arm in its `+grow` core which converts this datatype to a MIME, which will be included in the HTTP response. If successful, the HTTP response will contain the result with the mark specified. If unsuccessful, an HTTP error will be thrown in response.

The `@urbit/http-api` library we'll use includes a function for performing scries, so we'll not need to manually send GET requests to the ship.

## Resources <a href="#resources" id="resources"></a>

* [The Eyre vane documentation](/urbit-os/kernel/eyre) - This section of the vane docs covers all aspects of Eyre.
* [Eyre External API Reference](/urbit-os/kernel/eyre/external-api-ref) - This section of the Eyre documentation contains reference material for Eyre's external API.
* [The Eyre Guide](/urbit-os/kernel/eyre/guide) - This section of the Eyre documentation walks through using Eyre's external API at a low level (using `curl`).


# 6. React app setup

Setting up React frontend for full-stack Urbit apps using the npm @urbit/create-landscape-app package, configuring @urbit/http-api for ship communication.

Now that we have a basic idea of how Eyre works, we can begin working on our React app front-end.

## Create React app <a href="#create-react-app" id="create-react-app"></a>

Node.js must be installed, and can be downloaded from their [website](https://nodejs.org/en/download). With that installed, we'll have the `npm` package manager available and its utility binaries like `npx` to help set up our project. The first thing we'll do is create a project using the [`create-landscape-app`](https://www.npmjs.com/package/@urbit/create-landscape-app) template with the following command:

```sh
npx @urbit/create-landscape-app
✔ What should we call your application? … journal
✔ What URL do you use to access Urbit? … http://127.0.0.1:8080
```

We can then open our new directory:

```sh
cd journal/ui
```

Its contents should look something like this:

```
ui
├── index.html
├── package.json
├── package-lock.json
├── postcss.config.js
├── tailwind.config.js
├── vite.config.js
└── src
```

## Install dependencies <a href="#install-dependencies" id="install-dependencies"></a>

Inside our React app directory, let's install the NPM packages used by our project:

```sh
npm i
```

This command will install the Urbit interface package (i.e. `@urbit/http-api`) and all the other packages used by our React application. When building from scratch with `create-landscape-app`, this includes a number of useful development libraries that enable automatic refresh on file edits (i.e. `vite` and `@vitejs/plugin-react-refresh`) and simple page styling (i.e. `tailwindcss`). The remainder of this tutorial will focus primarily on how the Urbit interface package is used to communicate with a live ship from within a React application.

## Basic app setup <a href="#basic-app-setup" id="basic-app-setup"></a>

With all the basics now in place, we can begin work on the app itself. For this simple demonstration, we'll be working just with the `/src/app.jsx` file, which contains the rendering logic for our React application. Before we look at the full front-end source for our journal app, let's first review the simpler default code provided by `create-landscape-app` to cover some Urbit API and React basics.

### Urbit API setup <a href="#urbit-api-setup" id="urbit-api-setup"></a>

First, let's open up `/src/app.jsx` and look at the import statements at the top of this file:

```javascript
import React, { useEffect, useState } from 'react';
import Urbit from '@urbit/http-api';
import { scryCharges } from '@urbit/api';
import { AppTile } from './components/AppTile';
```

The first two of these statements are very common in Urbit React applications; the first imports the React library and a few of its important functions (to be covered in a moment) and the second imports the `Urbit()` class, which will be used subsequently to enable browser-to-ship communication.

Next, the code sets up the `Urbit()` API object as a global variable, which allows the browser-to-ship connection to be established *exactly once* when the page is first being loaded:

```javascript
const api = new Urbit('', '', window.desk);
api.ship = window.ship;
```

The first statement creates a new instance of the `Urbit()` class we imported from `@urbit/http-api`, and saves it to the `api` variable. The `Urbit()` class constructor takes three arguments: `url`, `code`, and `desk`, of which only `url` is mandatory.

* `url` is the URL of the ship we want to talk to. Since our React app will be served by the ship, we can just leave it as an empty `''` string and let `Urbit()` use root-relative paths.
* `code` is the web login code for authentication. Since the user will already have logged in, we can also leave it as an empty `''` string.
* `desk` is only necessary if we want to run threads through Eyre. This example doesn't submit any such requests, but the `desk` is set anyway for demonstration purposes.

The second statement sets the ship name in our `Urbit()` instance. Eyre requires the ship name be specified in all requests; if we don't set it, Eyre will reject all the messages we send. Fortunately, `create-landscape-app` handles this detail by automatically initializing the active ship's name to the variable `window.ship`, so we just set `api.ship` to this value.

While not referenced in the `create-landscape-app` default code, the `Urbit()` class has three additional callbacks that can be set: `onOpen()`, `onRetry()`, and `onError()`. These callbacks are fired when the state of our channel connection changes:

* `onOpen()` is called when a connection is established.
* `onRetry()` is called when a channel connection has been interrupted (such as by network issues) and the `Urbit()` object is trying to reconnect. Reconnection will be attempted up to three times: immediately, after 750ms, and after 3000ms.
* `onError()` is called with an `Error` message once all retries have failed, or otherwise when a fatal error occurs.

We'll look at how we can use these callbacks in the next section. Note that it's not mandatory to set these callbacks, but leaving connection problems unhandled is usually a bad idea.

### React app setup <a href="#react-app-setup" id="react-app-setup"></a>

Finally, let's take a quick look at the React rendering logic for our application. React rendering occurs within components, which are defined either as classes (e.g. `class A extends Component { /* ... */ }`) or functions (e.g. `function A() { /* ... */ }`). While recent React versions support both styles, the latter "modern" style is preferred and used by most Urbit React applications.

Our code defines a few components, but we'll just focus on the primary component for this tutorial; this component is defined as a functional component named `App()`:

```javascript
export function App() {
  /* ... */
}
```

As is common for React components, the first thing we'll define in our `App()` component is its state. In React, modifying a component's state causes it to be re-rendered, so state variables should be carefully chosen to constitute all "display-affecting" values. In modern React, component state is defined using the \[`useState()`] hook, which returns a pair of `[stateVariable, setStateVariableFunction]`. Since our default `create-landscape-app` code just displays the list of apps installed on a ship, it only needs to store this list as its state:

```javascript
const [apps, setApps] = useState();
```

With the state established, we now define the code responsible for populating this state. The canonical way to grab data from an external service/system in React is to use the \[`useEffect()`] hook. This function takes two arguments: (1) the callback function for loading the external data and (2) a list of all state variables dependencies, which will cause re-invocations of the first argument when modified. Our app just needs to load the list of apps on our ship (called `charges`) once, so its \[`useEffect()`] invocation is simple:

```javascript
useEffect(() => {
  async function init() {
    const charges = (await api.scry(scryCharges)).initial;
    setApps(charges);
  }

  init();
}, []);
```

The last step is to return the HTML that will be used to render our component in the browser. This HTML must adhere to the syntactic rules of [JSX](https://en.wikipedia.org/wiki/JSX_\(JavaScript\)), which allow for greater flexibility through extensions like embedded JavaScript (contained in curly brace enclosures). Our component renders each app it found when scrying our ship as a tile accompanied by its title and description:

```javascript
return (
  <main className="flex items-center justify-center min-h-screen">
    <div className="max-w-md space-y-6 py-20">
      <h1 className="text-3xl font-bold">Welcome to hut</h1>
      <p>Here&apos;s your urbit&apos;s installed apps:</p>
      {apps && (
        <ul className="space-y-4">
          {Object.entries(apps).map(([desk, app]) => (
            <li key={desk} className="flex items-center space-x-3 text-sm leading-tight">
              <AppTile {...app} />
              <div className="flex-1 text-black">
                <p>
                  <strong>{app.title || desk}</strong>
                </p>
                {app.info && <p>{app.info}</p>}
              </div>
            </li>
          ))}
        </ul>
      )}
    </div>
  </main>
);
```

With this brief primer complete, we'll take a closer look at our journal application's front-end and how it utilizes the Urbit HTTP API in the next section.

## Resources <a href="#resources" id="resources"></a>

* [React Tutorial](https://react.dev/learn/tutorial-tic-tac-toe) - A tutorial walking through the basics of writing a modern React application.
* [HTTP API Guide](/build-on-urbit/tools/js-libs/http-api-guide) - Reference documentation for `@urbit/http-api`.
* [React app source code](https://github.com/urbit/docs-examples/tree/main/journal-app/ui) - The source code for the Journal app UI.
* [`@urbit/http-api` source code](https://github.com/urbit/js-http-api) - The source code for the `@urbit/http-api` NPM package.
* [`useState()`](https://react.dev/reference/react/useState)
* [`useEffect()`](https://react.dev/reference/react/useEffect)


# 7. React app logic

React application logic for full-stack Urbit apps covering state management, ship communication patterns, subscription handling, poke operations, and data synchronization.

Now that we've reviewed the basics of setting up an Urbit React app, we can dive into the more complex logic that drives our [journal app's front-end](https://github.com/urbit/docs-examples/tree/main/journal-app/ui). We'll focus on the app's main component `App()` (defined in [`/src/app.jsx`](https://github.com/urbit/docs-examples/tree/main/journal-app/ui/src/app.jsx)) and how it leverages functions related to ship communications using the `Urbit()` object. For more information on UI components and other helper functions, see the [resources section](#resources).

## State <a href="#state" id="state"></a>

In the previous section, we introduced how React components use \[`useState()`] to declare state variables within components. The main `App()` component in our journal app contains a number of these statements to manage its many constituents and sub-components:

```javascript
// Control/Meta State //
const [subEvent, setSubEvent] = useState({});
const [latestUpdate, setLatestUpdate] = useState(null);
const [status, setStatus] = useState(null);
const [errorCount, setErrorCount] = useState(0);
const [errors, setErrors] = useState(new Map());

// Journal State //
const [entries, setEntries] = useState([]);
const [drafts, setDrafts] = useState({});
const [newDraft, setNewDraft] = useState({});
const [entryToDelete, setEntryToDelete] = useState(null);

// Search State //
const [results, setResults] = useState([]);
const [searchMeta, setSearchMeta] = useState({
  time: null,
  start: null,
  end: null,
});
```

We'll see how these are used subsequently.

## Initialize <a href="#initialize" id="initialize"></a>

After defining its state, the next thing our `App()` component does is define a function called `init()`, which is one of the first functions called during its bootstrapping process:

```javascript
const init = () => {
  getEntries().then(
    (result) => {
      setSubEvent(result);
      setLatestUpdate(result.time);
      subscribe();
    },
    (err) => {
      addError("Connection failed");
      setStatus("err");
    }
  );
};
```

This function just calls `getEntries()` to retrieve the initial list of journal entries; then, if that succeeded, it publishes this update with `setSubEvent()` and `setLatestUpdate()` and then calls `subscribe()` to subscribe for new updates. If the initial entry retrieval failed, we set the connection `status` and save an error message in the `errors` map. We'll look at what we do with errors later.

## Getting entries <a href="#getting-entries" id="getting-entries"></a>

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/entries.png)

The `getEntries()` function scries our `%journal` agent for up to 10 entries before the oldest we currently have. We call this initially and then each time the user scrolls to the bottom of the list.

```javascript
const getEntries = async () => {
  const e = entries;
  const before = e.length === 0 ? Date.now() : e[e.length - 1].id;
  const max = 10;
  const path = `/entries/before/${before}/${max}`;
  return window.urbit.scry({
    app: "journal",
    path: path,
  });
};
```

The scry is done with the `Urbit.scry()` method. This function takes two arguments in an object:

* `app` - the agent to scry.
* `path` - the scry path. Note the `care` is not included - all scries through Eyre are `%x` scries.

The `Urbit.scry()` method only allows JSON results, but note that scries done via direct GET requests allow other marks too.

The `Urbit.scry()` method returns a Promise which will contain an HTTP error message if the scry failed. We handle it with a `.then()` expression back in the function that called it, either [`init()`](#initialize) or `moreEntries()`. If the Promise is successfully evaluated, the results are passed to the [`setSubEvent()`](#updates) function, which appends the new entries to the existing ones via a \[`useEffect()`] hook (more on this [below](#updates)).

## Subscription <a href="#subscription" id="subscription"></a>

A subscription to the `/updates` path of our `%journal` agent is opened with our `subscribe()` function:

```javascript
const subscribe = () => {
  try {
    window.urbit.subscribe({
      app: "journal",
      path: "/updates",
      event: setSubEvent,
      err: () => addError("Subscription rejected"),
      quit: () => addError("Kicked from subscription"),
    });
  } catch {
    addError("Subscription failed");
  }
};
```

We use the `Urbit.subscribe()` method for this, which takes five arguments in an object:

* `app` - the target agent.
* `path` - the `%watch` path we're subscribing to.
* `event` - a function to handle each fact the agent sends out. We call our `setSubEvent()` function to set off a cascade to update the interface; this process is described [below](#updates).
* `err` - a function to call if the subscription request is rejected (nacked). We just display an error in this case.
* `quit` - a function to call if we get kicked from the subscription. We also just display an error in this case.

Note that the `Urbit.subscribe()` method returns a subscription ID number. Since we only have one subscription in our app which we never close, we don't bother to record it. If your app has multiple subscriptions to manage, you may wish to keep track of these IDs in your app's state.

## Updates <a href="#updates" id="updates"></a>

The architecture for updating a React interface based on incoming facts from an `Urbit()` subscription tends to follow a common pattern constituted of three major parts:

1. A \[`useState()`] call that creates an update object field as part of the main component's state:

   ```javascript
   const [subEvent, setSubEvent] = useState({});
   ```
2. An `Urbit.subscribe` call that passes the update object's setter function as its `event` field:

   ```javascript
   window.urbit.subscribe({/* ... */, event: setSubEvent});
   ```
3. A \[`useEffect()`] invocation that triggers off of the update object, which contains the logic for handling subscription updates:

   ```javascript
   useEffect(() => {/* ... */}, [subEvent]);
   ```

The key piece of this architecture is the \[`useEffect()`] trigger, which is called whenever an event comes in on the subscription wire (achieved by including the subscription object `subEvent()` as a re-invocation trigger in \[`useEffect()`]'s second argument). In our application, this hook is also triggered by calls to [`getEntries()`](#getting-entries) and [`getUpdates()`](#error-handling), which will be described in greater detail later.

The trigger code is a bit complex, but in broad brushstrokes it just checks the header of the incoming JSON object (i.e. one of `add`, `edit`, `delete`, or `entries`) and then updates the state appropriately. The object it's receiving is just the `$update` structure converted to JSON by the mark conversion functions we wrote previously.

<details>

<summary>useEffect</summary>

```javascript
useEffect(() => {
  const getDataIndex = (id, data) => {
    let low = 0;
    let high = data.length;
    while (low < high) {
      let mid = (low + high) >>> 1;
      if (data[mid].id > id) low = mid + 1;
      else high = mid;
    }
    return low;
  };

  const isInSearch = (id, time) => (
    searchMeta.time !== null &&
    time >= searchMeta.time &&
    searchMeta.start.getTime() <= id &&
    searchMeta.end.getTime() >= id
  );

  if (subEvent.time !== latestUpdate) {
    if ("entries" in subEvent) {
      // NOTE: `BottomScrollListener` can fire on top of `init`, which can
      // cause entries to be double loaded; we trim duplicates to avoid overlap
      const [existing, incoming] = [entries, subEvent.entries];
      const oldestExistingId = existing.length === 0
        ? Date.now()
        : existing[existing.length - 1].id;
      let newestIncomingInd = getDataIndex(oldestExistingId, incoming);
      newestIncomingInd += newestIncomingInd < incoming.length
        && incoming[newestIncomingInd].id >= oldestExistingId;
      setEntries(existing.concat(incoming.slice(newestIncomingInd)));
    } else if ("add" in subEvent) {
      const { time, add } = subEvent;
      const eInd = getDataIndex(add.id, entries);
      const rInd = getDataIndex(add.id, results);
      const toE = entries.length === 0 || add.id > entries[entries.length - 1].id;
      const toR = isInSearch(add.id, time);
      toE && entries.splice(eInd, 0, add);
      toR && results.splice(rInd, 0, add);
      toE && setEntries([...entries]);
      toR && setResults([...results]);
      setLatestUpdate(time);
    } else if ("edit" in subEvent) {
      const { time, edit } = subEvent;
      const eInd = entries.findIndex((e) => e.id === edit.id);
      const rInd = results.findIndex((e) => e.id === edit.id);
      const toE = eInd !== -1;
      const toR = rInd !== -1 && isInSearch(edit.id, time);
      if (toE) entries[eInd] = edit;
      if (toR) results[rInd] = edit;
      (toE || toR) && delete drafts[edit.id];
      toE && setEntries([...entries]);
      toR && setResults([...results]);
      (toE || toR) && setDrafts({...drafts});
      setLatestUpdate(time);
    } else if ("del" in subEvent) {
      const { time, del } = subEvent;
      const eInd = entries.findIndex((e) => e.id === del.id);
      const rInd = results.findIndex((e) => e.id === del.id);
      const toE = eInd !== -1;
      const toR = isInSearch(del.id, time) && rInd !== -1;
      toE && entries.splice(eInd, 1);
      toR && results.splice(rInd, 1);
      (toE || toR) && delete drafts[del.id];
      toE && setEntries([...entries]);
      toR && setResults([...results]);
      (toE || toR) && setDrafts({...drafts});
      setLatestUpdate(time);
    }
  }
}, [subEvent]);
```

</details>

## Add, edit, delete <a href="#add-edit-delete" id="add-edit-delete"></a>

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/add.png)

When a user writes a new journal entry and hits submit, the `createEntry()` function is called. It uses the `Urbit.poke()` method to poke our `%journal` agent.

```javascript
const createEntry = (id, txt) => {
  window.urbit.poke({
    app: "journal",
    mark: "journal-action",
    json: { add: { id: id, txt: txt } },
    onSuccess: () => setDraft({}),
    onError: () => setError("New entry rejected"),
  });
};
```

The `Urbit.poke()` method takes five arguments:

* `app` is the agent to poke.
* `mark` is the mark of the data we're sending. We specify `"journal-action"`, so Eyre will use the `/mar/journal/action.hoon` mark we created to convert it to a `$action` structure with a `%journal-action` mark before it's delivered to our agent.
* `json` is the actual data we're poking our agent with. In this case it's the JSON form of the `%add` `$action`.
* `onSuccess()` is a callback that fires if we get a positive ack in response. In this case we just clear the draft.
* `onError()` is a callback that fires if we get a negative ack (nack) in response, meaning the poke failed. In this case we just set an error message to be displayed.

Note that `onSuccess()` and `onError()` are optional, but it's usually desirable to handle these cases.

The `deleteEntry()` and `editEntry()` functions are similar to `createEntry()`, but for the `%del` and `%edit` actions rather than `%add`:

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/edit.png)

```javascript
const editEntry = (id, txt) => {
  if (txt === null) {
    delete drafts[id];
    setDrafts({...drafts});
  } else {
    window.urbit.poke({
      app: "journal",
      mark: "journal-action",
      json: { edit: { id: id, txt: txt } },
      onError: () => setError("Edit rejected"),
    });
  }
};
```

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/delete.png)

```javascript
const deleteEntry = (id) => {
  window.urbit.poke({
    app: "journal",
    mark: "journal-action",
    json: { del: { id: id } },
    onError: () => setError("Deletion rejected"),
  });
  setDeleteId(null);
};
```

Note that whether we're adding, editing or deleting entries, we update our state when we receive the update back on the `/updates` subscription, not when we poke our agent.

## Search <a href="#search" id="search"></a>

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/search.png)

When searching for entries between two dates, the `searchEntries()` function is called, which uses the `Urbit.scry()` method to scry for the results in a similar fashion to [`getEntries`](#getting-entries), but using the `/x/entries/between/[start]/[end]` endpoint.

```javascript
const searchEntries = async () => {
  const start = Math.max(inputStart.getTime(), 0);
  const end = Math.max(inputEnd.getTime(), 0);
  window.urbit.scry({
    app: "journal",
    path: `/entries/between/${start}/${end}`,
  }).then(
    (result) => {
      setInputStart(null);
      setInputEnd(null);
      setResults(result.entries);
      setSearchMeta({
        time: result.time,
        start: inputStart,
        end: inputEnd
      });
    },
    (err) => {
      setError("Search failed");
    }
  );
};
```

## Error handling <a href="#error-handling" id="error-handling"></a>

When the channel connection is interrupted, the `Urbit()` object will begin trying to reconnect. On each attempt, it sets the connection `status` to `"try"`, as we specified for the `onRetry()` callback. When this is set, a "reconnecting" message is displayed at the bottom of the screen:

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/reconnecting.png)

If all three reconnection attempts fail, the `onError()` callback is fired and we replace the "reconnecting" message with a "reconnect" button:

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/reconnect.png)

When clicked, the following function is called:

```javascript
const reconnect = () => {
  window.urbit.reset();
  if (latestUpdate === null) {
    init();
  } else {
    getUpdates().then(
      (result) => {
        result.logs.map(setSubEvent);
        subscribe();
      },
      (err) => {
        addError("Connection failed");
        setStatus("err");
      }
    );
  }
};
```

Our `reconnect()` function first calls the `Urbit.reset()` method. This closes the channel connection, wipes event counts and subscriptions, and generates a new channel ID. We could have tried reconnecting without resetting the connection, but we don't know whether the channel still exists. We could time how long the connection has been down and estimate whether it still exists, but it's easier to just start fresh in this case.

Since we've reset the channel, we don't know if we've missed any updates. Rather than having to refresh our whole state, we can use the `getUpdates()` function to get any missing update:

```javascript
const getUpdates = async () => {
  const since = latestUpdate === null ? Date.now() : latestUpdate;
  const path = `/updates/since/${since}`;
  return window.urbit.scry({
    app: "journal",
    path: path,
  });
};
```

This function uses the `Urbit.scry()` method to scry the `/x/updates/since/[since]` path, querying the agent's update `.log` for entries more recent than `latestUpdate`, which is always set to the last logged action we received. The `getUpdates()` function returns a Promise to the `reconnect()` function above which called it. The `reconnect()` function handles it in a `.then()` expression, where the success case passes each update retrieved to the [`setSubEvent()`](#updates) function, updating our state.

Lastly, as well as handling channel connection errors, we also handle errors such as poke nacks or failed scries by printing error messages added to the `error` map by the `setErrorMsg()` function. You could of course handle nacks, kicks, scry failures, etc differently than just printing an error; it depends on the needs of your app.

![](https://media.urbit.org/guides/core/app-school-full-stack-guide/search-failed.png)

## Resources <a href="#resources" id="resources"></a>

* [React Tutorial](https://react.dev/learn/tutorial-tic-tac-toe) - A tutorial walking through the basics of writing a modern React application.
* [HTTP API Guide](/build-on-urbit/tools/js-libs/http-api-guide) - Reference documentation for `@urbit/http-api`.
* [React app source code](https://github.com/urbit/docs-examples/tree/main/journal-app/ui) - The source code for the Journal app UI.
* [`@urbit/http-api` source code](https://github.com/urbit/js-http-api) - The source code for the `@urbit/http-api` NPM package.
* [`useState()`](https://react.dev/reference/react/useState)
* [`useEffect()`](https://react.dev/reference/react/useEffect)


# 8. Desk and glob

Building and deploying full-stack Urbit apps including desk configuration, kelvin versions, docket file for app metadata, glob creation for frontend assets, and publishing for distribution.

With our React app now complete, we can put together the final desk and publish it.

## Config files <a href="#config-files" id="config-files"></a>

So far we've written the following files for the back-end:

```
ourfiles
├── app
│   └── journal.hoon
├── lib
│   └── journal.hoon
├── mar
│   └── journal
│       ├── action.hoon
│       └── update.hoon
└── sur
    └── journal.hoon
```

There's a handful of extra files we need in the root of our desk:

* `desk.bill` - the list of agents that should be started when our app is installed.
* `sys.kelvin` - the kernel version our app is compatible with.
* `desk.docket-0` - configuration of our app tile, front-end glob and other metadata.

We only have one agent to start, so `desk.bill` is very simple:

```
:~  %journal
==
```

Likewise, `sys.kelvin` just contains:

```
[%zuse 408]
```

The `desk.docket-0` file is slightly more complicated:

```
:~
  title+'Journal'
  info+'Dear diary...'
  color+0xd9.b06d
  version+[0 1 0]
  website+'https://urbit.org'
  license+'MIT'
  base+'journal'
  glob-ames+[~zod 0v0]
==
```

The fields are as follows:

* `%title` is the name of the app - this will be displayed on the tile and when people search for the app to install it.
* `%info` is a brief description of the app.
* `%color` - the RGB hex color of the tile.
* `%version` - the version number of the app. The fields represent major, minor and patch version.
* `%website` - a link to a website for the app. This would often be its Github repo.
* `%license` - the license of for the app.
* `%base` - the desk name of the app.
* `%glob-ames` - the ship to retrieve the front-end files from, and the hash of those files. We've put \~zod here but this would be the actual ship distributing the app when it's live on the network. The hash is `0v0` initially, but once we upload the front-end files it will be updated to the hash of those files automatically. Note that it's also possible to distribute front-end files from a separate web server. In that case, you'd use `%glob-http` rather than `%glob-ames`. The [glob documentation](/build-on-urbit/userspace/dist/glob) covers this alternative approach in more detail.

Our files should now look like this:

```
ourfiles
├── app
│   └── journal.hoon
├── desk.bill
├── desk.docket-0
├── lib
│   └── journal.hoon
├── mar
│   └── journal
│       ├── action.hoon
│       └── update.hoon
├── sur
│   └── journal.hoon
└── sys.kelvin
```

## New desk <a href="#new-desk" id="new-desk"></a>

Next, we'll create a new `%journal` desk on our ship by forking an existing one. Once created, we can mount it to the unix filesystem.

In the Dojo of a fake ship:

```
|new-desk %journal
|mount %journal
```

Now we can browse to it in the Unix terminal:

```sh
cd /path/to/zod/journal
```

Currently it just contains some skeleton files, so we need to delete those:

```sh
rm -rI /path/to/zod/journal/*
```

Apart from the kernel and standard library, desks need to be totally self-contained, including all mark files and libraries necessary to build them. For example, since our app contains a number of `.hoon` files, we need the `%hoon` mark, and its dependencies. The easiest way to ensure our desk has everything it needs is to copy in the "dev" versions of the `%base` and `%landscape` desks. To do this, we first clone the Urbit and Landscape git repositories:

```sh
git clone https://github.com/urbit/urbit.git urbit-git
git clone https://github.com/tloncorp/landscape.git landscape-git
```

If we navigate to the `/pkg` directory in the cloned `/urbit` repo:

```sh
cd /path/to/urbit-git/pkg
```

...we can combine the `/base-dev` and Landscape `/desk-dev` desks with the included `symbolic-merge.sh` script:

```sh
./symbolic-merge.sh base-dev journal
./symbolic-merge.sh ../../landscape-git/desk-dev journal
```

Now, we copy the contents of the new `/journal` folder into our empty desk:

```sh
cp -rL journal/* /path/to/zod/journal/
```

Note we've used the `-L` flag to resolve symbolic links.

We can copy across all of our own files too:

```sh
cp -r /path/to/ourfiles/* /path/to/zod/journal/
```

Finally, in the dojo, we can commit the whole lot:

```
|commit %journal
```

## Glob <a href="#glob" id="glob"></a>

The next step is to build our front-end and upload the files to our ship. If you haven't yet downloaded the journal front-end source files, you can grab them from [their repository](https://github.com/urbit/docs-examples). In the folder containing our React app (`/journal-app/ui` relative to the repository base directory), we can run:

```sh
npm run build
```

This will create a `/build` directory containing the compiled front-end files. To upload it to our ship, we need to first install the `%journal` desk. In the dojo:

```
|install our %journal
```

Next, in the browser, we navigate to the `%docket` globulator at `http://localhost:8080/docket/upload` (replacing localhost with the actual host):

![](https://m.tinnus-napbus.xyz/pub/globulator.png)

We select our `%journal` desk, then we hit "Choose file", and select the whole `/build` directory which was created when we build our React app. Finally, we hit "glob!" to upload it.

(For real production apps, you might prefer to use the [`-make-glob` thread](/build-on-urbit/userspace/dist/glob#make-glob) in the `%landscape` desk. This allows you to publish the glob from somewhere other than your ship; if your ship publishing the app is down, anyone can install the desk from a peer and get the frontend glob from this other location.)

If we now return to the homescreen of our ship, we'll see our tile displayed, and we can open our app by clicking on it:

![](https://m.tinnus-napbus.xyz/pub/tiles.png)

## Publishing <a href="#publishing" id="publishing"></a>

The last thing we need to do is publish our app, so other users can install it from our ship. To do that, we just run the following command in the dojo:

```
:treaty|publish %journal
```

## Resources <a href="#resources" id="resources"></a>

* [App publishing/distribution documentation](/build-on-urbit/userspace/dist) - Documentation covering third party desk composition, publishing and distribution.
* [Glob documentation](/build-on-urbit/userspace/dist/glob) - Comprehensive documentation of handling front-end files.
* [Desk publishing guide](/build-on-urbit/userspace/dist/software-distribution) - A step-by-step guide to creating and publishing a desk.
* [-make-glob guide](/build-on-urbit/userspace/dist/glob#make-glob)


# 9. Summary

Summary and conclusion of the full-stack Urbit app tutorial with reference links for all covered topics including types, agents, JSON conversion, marks, Eyre, React frontend, and desk publishing.

That's it! We've built our agent and React front-end, put together a desk and published it. We hope this walkthrough has helped you see how all the pieces for together for building and distributing an app in Urbit.

The reference material for each section of this walkthrough is listed [below](#reference-material), the source code for our app is available [here](https://github.com/urbit/docs-examples/tree/main/journal-app), and it can be installed from `~pocwet/journal`.

In this guide we've built a separate React app for the front-end, but Hoon also has a native domain-specific language for composing HTML structures called Sail. Sail allows you to compose a front-end inside a Gall agent and serve it directly. See the [Sail guide](/hoon/sail) for details.

Along with `@urbit/http-api`, there's also the `@urbit/api` npm package, which contains a large number of helpful functions for dealing with Hoon data types and interacting with a number of agents - particularly those used by the Groups app. Its source code is [available here](https://github.com/urbit/js-http-api).

## Reference material <a href="#reference-material" id="reference-material"></a>

Here is the reference material for each section of this walkthrough.

### Types <a href="#types" id="types"></a>

* [App School `/sur` section](/build-on-urbit/app-school/7-sur-and-marks#sur) - This section of App School covers writing a `/sur` structure library for an agent.
* [Ordered map functions in `zuse.hoon`](https://github.com/urbit/urbit/blob/master/pkg/arvo/sys/zuse.hoon#L5284-L5688) - This section of `zuse.hoon` contains all the functions for working with `+mop`s, and is well commented.

### Agent <a href="#agent" id="agent"></a>

* [App School I](/build-on-urbit/app-school) - App School I covers all aspects of writing Gall agents in detail.
* [Ordered map functions in `zuse.hoon`](https://github.com/urbit/urbit/blob/master/pkg/arvo/sys/zuse.hoon#L5284-L5688) - This section of `zuse.hoon` contains all the functions for working with `+mop`s, and is well commented.
* [`/lib/agentio.hoon`](https://github.com/urbit/urbit/blob/master/pkg/base-dev/lib/agentio.hoon) - The agentio library in the `%base` desk contains a large number of useful functions which making writing Gall agents easier.

### JSON <a href="#json" id="json"></a>

* [The JSON Guide](/hoon/json-guide) - The stand-alone JSON guide covers JSON encoding/decoding in great detail.
* [The Zuse Reference](/hoon/zuse) - The `zuse.hoon` reference documents all JSON-related functions in detail.
* [`+enjs:format` reference](/hoon/zuse/2d_1-5#enjsformat) - This section of the `zuse.hoon` documentation covers all JSON encoding functions.
* [`+dejs:format` reference](/hoon/zuse/2d_6) - This section of the `zuse.hoon` documentation covers all JSON *decoding* functions.
* [Eyre Overview](/urbit-os/kernel/eyre) - This section of the Eyre vane documentation goes over the basic features of the Eyre vane.

### Marks <a href="#marks" id="marks"></a>

* [The Marks section of the Clay documentation](/urbit-os/kernel/clay/marks) - This section of the Clay vane documentation covers mark files comprehensively.
* [The mark file section of the Gall Guide](/build-on-urbit/app-school/7-sur-and-marks#mark-files) - This part of App School goes through the basics of mark files.
* [The JSON Guide](/hoon/json-guide) - This also covers writing mark files to convert to/from JSON.

### Eyre <a href="#eyre" id="eyre"></a>

* [The Eyre vane documentation](/urbit-os/kernel/eyre) - This section of the vane docs covers all aspects of Eyre.
* [Eyre External API Reference](/urbit-os/kernel/eyre/external-api-ref) - This section of the Eyre documentation contains reference material for Eyre's external API.
* [The Eyre Guide](/urbit-os/kernel/eyre/guide) - This section of the Eyre documentation walks through using Eyre's external API at a low level (using `curl`).

### React App Setup and Logic <a href="#react-app-setup-and-logic" id="react-app-setup-and-logic"></a>

* [HTTP API Guide](/build-on-urbit/tools/js-libs/http-api-guide) - Reference documentation for `@urbit/http-api`.
* [React app source code](https://github.com/urbit/docs-examples/tree/main/journal-app/ui) - The source code for the Journal app UI.
* [`@urbit/http-api` source code](https://github.com/urbit/js-http-api) - The source code for the `@urbit/http-api` NPM package.

### Desk and Glob <a href="#desk-and-glob" id="desk-and-glob"></a>

* [App publishing/distribution docs](/build-on-urbit/userspace/dist/software-distribution) - Documentation covering third party desk composition, publishing and distribution.
* [Glob documentation](/build-on-urbit/userspace/dist/glob) - Comprehensive documentation of handling front-end files.
* [Desk publishing guide](/build-on-urbit/userspace/dist/software-distribution) - A step-by-step guide to creating and publishing a desk.


# Core Academy

Course for developers who have finished Hoon School to learn Arvo kernel development, covering each part of Arvo, Zuse, Hoon, and the runtime for core Urbit OS development.

Core Academy prepares [Hoon](/hoon/why-hoon) developers to become [Arvo](/urbit-os/what-is-urbit-os) kernel developers. While only [Hoon School](/build-on-urbit/hoon-school) is strictly necessary to follow this material, readers should be capable of writing fairly complex applications in Hoon. Some knowledge of C and/or Rust is also necessary for some of the runtime material.

This written course is adapted from the teaching notes for the 2023 cohort of Core Academy, its first and to-date only live cohort. The course explores each part of Arvo, Zuse, Hoon, and the runtime to contextualize how Urbit works today and where further development is needed.

*The lesson notes represent a snapshot of Urbit core development at Zuse 412k and Vere 2.12. Some details have changed since that release.*

Video recordings of the 2023 Core Academy are [available to watch here](https://www.youtube.com/playlist?list=PLYGEMSwLguIGgrEGwxu2AAbESpfF_LRKx). The order in which vanes are covered differs between this written material and the recorded lectures, but aside from some Q\&A in the lectures it's all the same information.

## Preliminary material <a href="#preliminary-material" id="preliminary-material"></a>

* [Hoon School, Text Parsing III](/build-on-urbit/hoon-school/q2-parsing)
* [Hoon School, Generic and Variant Cores](/build-on-urbit/hoon-school/r-metals)
* [The Engine Pattern](/hoon/engine-pattern)


# 1. Evaluating Nock

Core Academy lesson on Nock evaluation covering all Nock rules 0-12, binary tree addressing, hints and jetting, scry operations, virtualized Nock with +mock, and the Urbit lifecycle function.

*This lesson explains each Nock rule and how it relates to Hoon expressions. We cover hints (Nock 11), jetting, and how Nock 12 works to effect a scry.*

Urbit is a machine for writing and evaluating Nock code. Much of the characteristic nature of Hoon derives directly from the affordances of Nock. Perhaps the most fundamental design decision undergirding Nock is that Nock orients the world around binary trees rather than linear arrays.

```
Nock 4K

A noun is an atom or a cell. An atom is a natural number. A cell is an ordered pair of nouns.

Reduce by the first matching pattern; variables match any noun.

nock(a)             *a
[a b c]             [a [b c]]

?[a b]              0
?a                  1
+[a b]              +[a b]
+a                  1 + a
=[a a]              0
=[a b]              1

/[1 a]              a
/[2 a b]            a
/[3 a b]            b
/[(a + a) b]        /[2 /[a b]]
/[(a + a + 1) b]    /[3 /[a b]]
/a                  /a

#[1 a b]            a
#[(a + a) b c]      #[a [b /[(a + a + 1) c]] c]
#[(a + a + 1) b c]  #[a [/[(a + a) c] b] c]
#a                  #a

*[a [b c] d]        [*[a b c] *[a d]]

*[a 0 b]            /[b a]
*[a 1 b]            b
*[a 2 b c]          *[*[a b] *[a c]]
*[a 3 b]            ?*[a b]
*[a 4 b]            +*[a b]
*[a 5 b c]          =[*[a b] *[a c]]

*[a 6 b c d]        *[a *[[c d] 0 *[[2 3] 0 *[a 4 4 b]]]]
*[a 7 b c]          *[*[a b] c]
*[a 8 b c]          *[[*[a b] a] c]
*[a 9 b c]          *[*[a c] 2 [0 1] 0 b]
*[a 10 [b c] d]     #[b *[a c] *[a d]]

*[a 11 [b c] d]     *[[*[a c] *[a d]] 0 3]
*[a 11 b c]         *[a c]

*a                  *a
```

(To avoid confusion in Nock versions versus Nock rules, this lesson will refer to Nock rules with a written number, e.g. “Nock Four” refers to `*[a 4 b]` rather than to Nock 4K.)

Many aspects of Hoon's nature and Urbit's operating model are natural consequences of Nock's structure and preferences. Hoon acts as a sort of (very glorified) macro assembler over Nock, providing runes and idioms for Nock patterns and a type system which reduces to Nock. However, it is possible to do things in pure Nock which Hoon restricts; this is one motivation for vase mode.

In this lesson, we will begin our study of the Urbit kernel by looking at the wellsprings of Urbit. Nock has evolved substantially; the earliest public record is Nock 13K. However, as of Nock 4K no further revisions are expected to be necessary and the system is very cold.

* [\~lagrev-nocfep, \~sorreg-namtyv, (2025) "A Documentary History of the Nock Combinator Calculus", *Urbit Systems Technical Journal* vol. 2, iss. 1](https://urbitsystems.tech/article/v02-i01/a-documentary-history-of-the-nock-combinator-calculus)

For clarity, this document will refer to rules with written-out numbers (Nock Four) and versions with Arabic digits (Nock 4K). It will also prefer the shorthand `s` for subject (instead of `a`) and `f`, `g`, etc. for formula (instead of `b`, `c`, etc.).

* [Nock Definition](/nock/definition)

## The Nock Preface <a href="#the-nock-preface" id="the-nock-preface"></a>

The Nock specification needs to lay some interpretive ground before the rules can be defined. These definitions allow us to settle on an unambiguous interpretation of the rules. Importantly, most of the syntax defined here is unique to Nock and does not bear the same meaning in Hoon, with the exception of `[` sel and `]` ser as bounding cells.

```
nock(a)             *a
[a b c]             [a [b c]]

?[a b]              0
?a                  1
+[a b]              +[a b]
+a                  1 + a
=[a a]              0
=[a b]              1

/[1 a]              a
/[2 a b]            a
/[3 a b]            b
/[(a + a) b]        /[2 /[a b]]
/[(a + a + 1) b]    /[3 /[a b]]
/a                  /a

#[1 a b]            a
#[(a + a) b c]      #[a [b /[(a + a + 1) c]] c]
#[(a + a + 1) b c]  #[a [/[(a + a) c] b] c]
#a                  #a

*[a [b c] d]        [*[a b c] *[a d]]
```

### Evaluation and Structure <a href="#evaluation-and-structure" id="evaluation-and-structure"></a>

#### `nock(a)` → `*a`

Nock is an evaluated language, and we write the evaluation of this function as `*`. `*` represents an attempt to systematically apply the rules of Nock in an inside-out fashion to the given noun. `*` runs on a valid `[argument function]` pair; i.e. the `function` is the `formula` and the `argument` is the `subject`.

It is perhaps an oddity of `*` that it “spins forever” instead of merely returning the noun on evaluation.

#### `[a b c]` → `[a [b c]]`

For convenience, the pretty-printer elides rightward-branching tuples.

#### `?[a b]` → `0`

#### `?a` → `1`

This represents the ability to tell a cell from an atom. At this point in the preface we actually don't know yet if `0` is `TRUE` or `FALSE`, merely that they are distinct.

#### `+[a b]` → `+[a b]`

#### `+a` → `1 + a`

The increment operation is not distributed across cell entries. (However, note that if `*` evaluation leads to the operated-upon cell reducing to an atom then this will proceed.)

#### `=[a a]` → `0`

#### `=[a b]` → `1`

Now we can compare equality with the `=` operator that accepts a cell. (No specification of behavior applied to an atom is made here, but Nock is presumptively crash-only.)

#### `/[1 a]` → `a`

#### `/[2 a b]` → `/[2 [a b]]` → a

#### `/[3 a b]` → `/[3 [a b]]` → `b`

These three statements define the basic approach to binary tree addressing: namely that there is a root of the tree at `1`, a head at `2`, and a tail at `3`.

#### `/[(a + a) b]` → `/[2 /[a b]]`

#### `/[(a + a + 1) b]` → `/[3 /[a b]]`

So what do these do? These define the subsequent layers of the tree recursively. Thus the head (`2`) of the cell at `a` has the address 2×`a`, and the tail (`3`) of the cell at `a` has the address 2×`a`+1.

#### `#[1 a b]` → a

#### `#[(a + a) b c]` → `#[a [b /[(a + a + 1) c]] c]`

#### `#[(a + a + 1) b c]` → `#[a [/[(a + a) c] b] c]`

These statements define what it means to replace part of one noun with another noun. “Replace addresss `x` of `z` with `y`.” They require a similar sort of tree definition to the slot addressing.

The last definition, distribution, is reserved for later discussion.

## The Simple Rules: Nock Zero to Five <a href="#the-simple-rules-nock-zero-to-five" id="the-simple-rules-nock-zero-to-five"></a>

Nock is separated into simple rules which span its Turing-complete expressibility and compound rules which give it more terseness. The simple Nock rules span Nock Zero to Nock Five:

```
*[a 0 b]            /[b a]
*[a 1 b]            b
*[a 2 b c]          *[*[a b] *[a c]]
*[a 3 b]            ?*[a b]
*[a 4 b]            +*[a b]
*[a 5 b c]          =[*[a b] *[a c]]
```

A good entry point to the study of these basic Nock rules can be made through their Hoon equivalents. Hoon directly exposes several Nock rules as the [`.` dot](/hoon/rune/dot) runes.

| Nock Rule  | Nock Expression    | Hoon Equivalent | Comment                    |
| ---------- | ------------------ | --------------- | -------------------------- |
| Nock Zero  | `/[b a]`           | (some wings)    | Address at slot            |
| Nock One   | `b`                | (any data)      | Constant                   |
| Nock Two   | `*[*[a b] *[a c]]` | `.*` dottar     | Evaluate                   |
| Nock Three | `?*[a b]`          | `.?` dotwut     | Distinguish cell from atom |
| Nock Four  | `+*[a b]`          | `.+` dotlus     | Increment                  |
| Nock Five  | `=[*[a b] *[a c]]` | `.=` dottis     | Test for equality          |

The following paragraphs summarize these Nock rules. Two runes and one gate will be particularly useful as we analyze Nock:

* We will evaluate Nock statements in Hoon using [`.*` dottar](/hoon/rune/dot#dottar) or Nock Two.
* We can produce the Nock equivalent of a Hoon statement using [`!=` zaptis](/hoon/rune/zap#zaptis), which takes a single `hoon` for its sample.
* We can run a virtualized Nock (with crash data) in [`+mock`](/hoon/stdlib/4n#mock). An optional hint handler can be supplied but we will typically ignore it.

### Nock Zero: Slot <a href="#nock-zero-slot" id="nock-zero-slot"></a>

#### `[s 0 f]` → `/[f s]`

Nock Zero locates a noun at address `f` in subject `s`.

```nock
      .
     / \
    /   \
   /     \
  .       .
 / \     / \
1   2   3   4

[[1 2] [3 4]]

> .*([[1 2] [3 4]] [0 2])
[1 2]

> .*([[1 2] [3 4]] [0 4])
1

> .*([[1 2] [3 4]] [0 8])
dojo: hoon expression failed

> (mock [[[1 2] [3 4]] 0 7] ,~)
[%0 p=4]

> (mock [[[1 2] [3 4]] 0 8] ,~)
[%2 p=~]
```

* Lark notation
* Wing search path
* more on these in `csl02`

It is common to see `[… 0 6]` in Nock expressions generated from Hoon. What is at address 6 that we'd like to grab?

`..arm` is a Nock Zero lookup, for instance.

```hoon
> !=(..add)
[0 2.047]
```

One of the perennial controversies around Nock has arisen from Nock Zero's binary tree nature. From a processor standpoint, arbitrary data lookup does not take advantage of cache behavior. However, Nock bytecode and subject knowledge analysis (SKA) both work to mitigate this situation.

### Nock One: Constant <a href="#nock-one-constant" id="nock-one-constant"></a>

#### `*[s 1 f]` → `f`

Nock One simply yields its formula as a noun.

For instance, a numeric value in Nock can refer to a rule, a slot address, or a numeric value. Nock One would clarify that we intend the latter, a numeric constant.

```hoon
      .
     / \
    /   \
   /     \
  .       .
 / \     / \
1   2   3   4

[[1 2] [3 4]]

> .*([[1 2] [3 4]] [1 7])
7

> .*([[1 2] [3 4]] [1 [7 8 9]])
[7 8 9]
```

### Nock Two: Evaluate <a href="#nock-two-evaluate" id="nock-two-evaluate"></a>

#### `*[s 2 f g]` → `*[*[s f] *[s g]]`

Nock Two effectively modifies the subject against which a second formula is evaluated. Run formula `f` against subject `s` to yield noun `sf`. Run formula `g` against subject `s` to yield noun `sg`. Run formula `sg` against subject `sf`.

Nock Two is useful when constructing a formula explicitly, but most of the time you are actually just fetching a formula from your context instead—thus Nock Nine, when we arrive to it. (Thus we seem to use bare Nock Two in practice less frequently than one may at first expect.)

For now, let's build a simple example using only rules Zero and One.

```hoon
  .
 / \
1   2

[1 2]

> .*([1 2] [2 [0 2] [1 [0 1]]])  
1
```

* First, the formula `.*([1 2] [0 2])` yields the single atom `1`.
* Then the second formula `.*([1 2] [1 [0 1]])` results in `[0 1]`.
* Finally the formula `.*(1 [0 1])` yields `1`.

You can think of Nock Two as a way of storing a subprocedure in a subject, then accessing it for evaluation.

### Nock Three: Distinguish <a href="#nock-three-distinguish" id="nock-three-distinguish"></a>

#### `*[s 3 f]` → `?*[s f]`

Does the formula as applied to the subject resolve to a cell?

```hoon
> .*([[1 2] [3 4]] [3 0 1])
0

> .*([[1 2] [3 4]] [3 0 4])
1
```

I.e., is the noun at slot 1 a cell? (Yes.) Is the noun at slot 4 a cell? (No.)

### Nock Four: Increment <a href="#nock-four-increment" id="nock-four-increment"></a>

#### `*[s 4 f]` → `+*[s f]`

Increment the value. This is the only arithmetic operation.

```hoon
> .*(5 [4 0 1])
6
```

Note that typically Nock Four is paired with a constant lookup for the formula. You could construct other patterns by hand; what is this doing?

```hoon
> (mock [5 4 3 0 1] ,~)
[%0 p=2]
```

### Nock Five: Equality <a href="#nock-five-equality" id="nock-five-equality"></a>

#### `*[s 5 f g]` → `=[*[s f] *[s g]]`

Are the two nouns, as resolved against the subject, identical? (Compare the result in Nock, not Hoon.) This refers to the structure of the noun, which gives you insight into its behavior in Hoon.

```hoon
> .*([[1 2] [1 2]] [5 [0 2] [0 3]])
0

> .*([[1 2] [3 4]] [5 [0 2] [0 3]])
1

> .*([[1 2] [3 4]] [5 [0 5] [4 0 4]])
0
```

### Distribution <a href="#distribution" id="distribution"></a>

#### `*[s [f g] h]` → `[*[s f g] *[s h]]`

While not a Nock rule per se, the distributive behavior of Nock (“implicit cons”) requires some attention before we examine the compound rules.

Formulas have a number corresponding to a Nock rule at their head. However, you can also put a cell at the head of a formula. What does this mean?

```hoon
> .*([[1 2] [3 4]] [[0 3] [4 0 5]])  
[[3 4] 3]
```

It means that each entry is itself a formula. So what this means is that `[0 3]` resolves against `[[1 2] [3 4]]` and `[4 0 5]` resolves against `[[1 2] [3 4]]` before they together yield a cell of the results `[[3 4] 3]`.

In general, we can string many formulas together in a cell to resolve them all sequentially against the same subject.

## The Compound Rules: Nock Six to Eleven <a href="#the-compound-rules-nock-six-to-eleven" id="the-compound-rules-nock-six-to-eleven"></a>

| Nock Rule   | Nock Expression                         | Hoon Equivalent | Comment            |
| ----------- | --------------------------------------- | --------------- | ------------------ |
| Nock Six    | `*[a *[[c d] 0 *[[2 3] 0 *[a 4 4 b]]]]` | `?:` wutcol     | Conditional branch |
| Nock Seven  | `*[*[a b] c]`                           | `=>` tisgar     | Compose            |
| Nock Eight  | `*[[*[a b] a] c]`                       | `=+` tislus     | Extend             |
| Nock Nine   | `*[*[a c] 2 [0 1] 0 b]`                 |                 | Invoke             |
| Nock Ten    | `#[b *[a c] *[a d]]`                    | `%=` centtis    | Replace at address |
| Nock Eleven | `*[[*[a c] *[a d]] 0 3]`, `*[a c]`      | `~` sig runes   | Hint               |

### Nock Six: Conditional Branch <a href="#nock-six-conditional-branch" id="nock-six-conditional-branch"></a>

#### `*[s 6 f g h]` → `*[s *[[g h] 0 *[[2 3] 0 *[s 4 4 f]]]]`

This expression is straightforward to understand, but the logic of its structure is interesting to follow through. What is being said? If formula `f` evaluated against the subject `s` evaluates `TRUE` (`0`), then the result of `g` evaluated against the subject is produced. If it evaluates to `FALSE` (`1`), then the result of `h` evaluated against the subject is produced. Otherwise, crash.

* First, at right we evaluate `*[s 4 4 f]`. This means evaluate `f` against `s` then increment it twice. This selects either `0+2` = `2` or `1+2` = `3`. These are slot addresses into Rule Zero immediately before this expression.
* `[2 3]` is the lookup subject for the `0`. Why? This enforces that we have `2` (`0`) or `3` (`1`), not any other possible answer from `f`.
* Now we use that `2` or `3` as a Rule Zero slot address into `g` or `h`.

The funkiness of the equation is because we have to make sure that the result is `0` or `1`; we must crash on another value or on a cell.

* How would you write `?.` wutdot?
* How would you write `?~` wutsig?
* What do `?^` wutket and `?@` wutpat expand to? (Nock Six of Nock Three)

### Nock Seven: Compose <a href="#nock-seven-compose" id="nock-seven-compose"></a>

#### `*[s 7 f g]` → `*[*[s f] g]`

Composition describes the evaluation of one formula against the subject, then using that result as the subject of the next formula.

```
::  With the subject set to a cell of our and now, return the head.
> !=  =>  [our now]  -
[7 [[0 12] 0 26] 0 2]
```

(Incidentally, although we'll get to this later in `ca02`, what are `our` and `now`?)

### Nock Eight: Extend <a href="#nock-eight-extend" id="nock-eight-extend"></a>

#### `*[s 8 f g]` → `*[[*[s f] s] g]`

Nock Eight allows you to pin a value into the subject. It is very similar to Nock Seven.

The runes `=+` tislus, `=-` tishep, `=/` tisfas, `=;` tismic, and `=|` tisbar are all instances of Nock Eight. You can use the runes like `=+` and `=-` with immediate use of `-` to refer to the value just added to the subject are Nock Eight followed by Nock Zero.

```hoon
> =+  5
  -
5

> !=(=+(5 -))
[8 [1 5] 0 2]
```

Faces are not even necessary in Hoon—you can refer to a value using lark notation `-` after it has been defined. `+>` frequently refers to a helper core in Gall, for instance. (While lark notation is often unsatisfactory, it can be more expressive in simple relational statements like these.)

### Nock Nine: Invoke <a href="#nock-nine-invoke" id="nock-nine-invoke"></a>

#### `*[s 9 a f]` → `*[*[s f] 2 [0 1] 0 a]`

The underlying rationale for Nock Nine is to invoke a closure or compute over an association of code and data. This is used throughout Urbit in invocation—think gate-building gates, on which more in `ca01`. In plain language, produce a noun containing code and data (a core) by evaluating formula `g` against the subject `s` and then evaluate the formula in it at slot `a`.

An arm in any core is accessed through a Nock Nine. For instance, here we have a call to `+dec`, the decrement gate.

```
> !=((dec 1))
[8 [9 2.398 0 2.047] 9 2 10 [6 7 [0 3] 1 1] 0 2]
```

Compare the actual Hoon:

```hoon
++  dec
  ~/  %dec
  ::    unsigned decrement by one.
  |=  a=@
  ~_  leaf+"decrement-underflow"
  ?<  =(0 a)
  =+  b=0
  ::  decremented integer
  |-  ^-  @
  ?:  =(a +(b))  b
  $(b +(b))
```

What's actually being said here? What we see in `+dec` that we don't see in the Nock is a Nock Four increment operator. That's a hint that the Nock code resulting from `(dec 1)` isn't the equivalent of the gate at all. Check the battery of `+dec` to verify this (it's quite long).

No, something else is going on with this function invocation with Nock Nine. Based on what we've seen so far of the Nock rules:

* The subject of the outer Nock Nine call is `[8 [9 2.398 0 2.047]]`. The subject is extended using Nock Eight by retrieving the axis at constant `2.047` within the current subject, then grabbing the noun at slot `2.398` within that noun.
  * Extend the namespace with the lookup of the `battery` of the core at `+2047` in the subject. (`[9 2 …` is a very common idiom in Nock, resulting from invoking the battery of a core, but here it's not at `2` it's more complexly buried in the subject tree.)
* The formula is `[2 10 [6 7 [0 3] 1 1] 0 2]`.
  * Use the battery of that (second Nock Nine) as the subject for evaluating an `if`-`else` Nock Six.
  * That has a constant `1` fired inside of a lookup of the `+3` (`payload`) via Nock Seven (which is the sample).
  * Finally the slot `2` is accessed which refers to the battery in the subject as received at this point.

In general, watch for `[0 2]` to grab the battery and `[0 3]` to grab the payload.

There's another way around this, which will also be illuminating: build it from the ground up.

```hoon
[8 [1 0] 8 [1 6 [5 [0 7] 4 0 6] [0 6] 9 2 [0 2] [4 0 6] 0 7] 9 2 0 1]
```

* Why do these look different from each other? The first one is locating a battery somewhere else in the subject and pulling it in. The second is explicitly constructing the operation in Nock and invoking it at the same point.

In another example, see what happens with `(add 2 2)` in Nock:

```
> !=((add 2 2))
[8 [9 36 0 2.047] 9 2 10 [6 [7 [0 3] 1 2] 7 [0 3] 1 2] 0 2]
```

Two Nock Nines again, one to retrieve the `battery` of `+add` and another to invoke it.

The core nature of Hoon fits hand-in-glove with Nock Nine. `+arm` is always an arm, and a wing that references it *directly* always performs a Nock Nine for the lookup.

(In the Vere runtime, Nock Nine corresponds to `u3v_poke`, in which a `fun` gets a `gate` to evaluate. More in `ca05`.)

### Nock Ten: Replace at Address <a href="#nock-ten-replace-at-address" id="nock-ten-replace-at-address"></a>

#### `*[s 10 [f g] h]` → `#[f *[s g] *[s h]]`

* `%-` centtis resolves to Nock Nine of Nock Ten: it uses Nock Ten to replace the sample of a core, then runs Nock Nine to fire the arm at axis 2.

```hoon
> !=  %-  add  [3 4]
[8 [9 36 0 2.047] 9 2 10 [6 7 [0 3] 1 3 4] 0 2]
```

* [Nock examples implemented by hand](https://github.com/urbit/examples/blob/master/wip/lib/nock.hoon)

### Nock Eleven: Hint <a href="#nock-eleven-hint" id="nock-eleven-hint"></a>

#### `*[s 11 [f g] h]` → `*[[*[s g] *[s h]] 0 3]`

#### `*[s 11 f g]` → `*[s g]`

The fundamental idea of a hint is that it can provide an arbitrary annotation for a computation without changing the result. In practice, this is used to signal to the runtime that it should do something operationally which Nock doesn't know about.

The first kind of hint is called a dynamic hint. The head of the cell is discarded and the tail is calculated against the current subject, then thrown away. (What is an example of this in practice? `~&` sigpam.) Put another way: the value in a computation is available to the runtime to use as a message.

```hoon
> !=(~&('hello' ~))
[ 11
  [ 1.735.355.507
    [1 0]
    8
    [9 2.558 0 127]
    9
    2
    10
    [ 6
      7
      [0 3]
      8
      [9 158 0 127]
      9
      2
      10
      [6 7 [0 3] 1 [1.836.020.833 116 0] 478.560.413.032]
      0
      2
    ]
    0
    2
  ]
  1
  0
]
```

which in Pseudo-Hoon is:

```
> !=(~&('hello' ~))
[ 11
  [ %slog ~
    8
    [9 2.558 %127]
    [9 2 10
    [ 6
      7
      %3
      8
      [9 158 %127]
      [9 2 10
      [6 7 %3 [%atom 't' ~] 'hello'] %2
    ]
    ]
    ]
    %2
  ]
  ~
]
```

The second kind of hint is a static hint and is simply directly specified. The runtime may pick this up, and if it is a registered hint then it may trigger an appropriate side effect. Put otherwise: the value is passed as a key to a hashmap in the interpreter which can do what it needs, then discard the value.

```hoon
> !=(~>(%bout ~))
[11 1.953.853.282 1 0]
```

#### Practical Hints

The most fundamental way to send a hint to the runtime is to simply `~>` siggar a raw hint tag with data. For instance, this is used in the update sequence to `%slog` (`printf`) a formatted text block. (Note that raw hints use a special `.` dot notation.)

```hoon
~>(%slog.0^leaf/"clay: rebuilding {<syd>} after kernel update" .)
```

There are a number of ways of signaling a Nock Eleven hint to the runtime. Several have dedicated runes in the `~` sig family, but all of these convenience runes do ultimately unwrap to raw hints and we'll examine them that way. `~>` siggar lets you directly specify a raw hint. Some of these are not in active use; the important ones for you to know at this point have been marked with a `*`.

* \* `%bout` is used to time calculations.
* \* `%fast` corresponds to jet hinting with `~%` sigcen and `~/` sigfas. (See also \~rivpyl-sidfyl's notes.)
* `%germ` is marked as unused in `nock.c` but corresponds to `~=` sigtis noun duplicate testing.
* `%hand` is mentioned but it's unclear what it does.
* `%hela` is used to unify `%slog` traces from all roads. (See the runtime lessons for details on the road.)
* `%hunk` tracks a failed scry in a stack trace.
* `%live` corresponds to `~$` sigbuc profiling hit counter.
* `%lose` tracks elided frames in a stack trace.
* \* `%mean` corresponds to `~|` sigbar and `~_` sigcab debug printing.
* `%meme` prints memory usage.
* `%memo` corresponds to `~+` siglus caching.
* `%nara` is used to `%slog` the deepest road's trace. (See the runtime lessons for details on the road.)
* \* `%slog` prints a value directly, and corresponds to `~&` sigpam and `~|` sigzap debug printing .
* `%sole` is superseded, but was for global deduplication.
* `%spot` drives stack traces.
* `%xray` prints bytecode.
* Enumerate the raw hints used with `~>` siggar. Provide an example of each.

```
> !=(~>(%slog [0 leaf/"ride: parsing"]))
[ 11
  1.735.355.507
  [1 0]
  [1 1.717.658.988]
  7
  [0 1]
  8
  [1 1 114 105 100 101 58 32 112 97 114 115 105 110 103 0]
  9
  2
  0
  1
]
```

```hoon
> =>  42  ~>  %memo.[42]  ~>  %slog.[3 '42']  42
42
> =>  42  ~>  %memo.[42]  ~>  %slog.[3 '42']  42
42
```

#### Jet-Accelerated Code

One “problem” with Nock is that although it's formally correct and Turing-complete, that doesn't mean that it's computationally efficient for every scenario. How can we solve the problem correctly but quickly in a situation for which this calculation is a bottleneck? We treat the Nock code as a specification of behavior rather than an assertion of method. Jet-accelerated code ("jets" for short) solves our speed problem by allowing the programmer to specify an equivalent algorithm which may (should) be much more efficient than the pure Nock version.

(This is analogous to C's use of the `#pragma` preprocessor command to specify to its compiler how to treat certain code.)

We mark certain blocks of code with hints, which suggest to the runtime (Nock interpreter) that an equivalent algorithm may exist in the runtime and that the jet dashboard should be checked for a comparable registration. Such jet dispatch then runs the equivalent fast code and supplies the resulting noun back to the compiled Nock code as if it arisen directly from the Hoon expression. (The jet dashboard handles jet state in the runtime. We don't need to delve into it to understand jet dispatch now, but see the appendix for more details.)

Jets are hashed and checked before they are compiled, and there is an option to run both the Nock and jet and check that they match. One consequence of jetted code is that you can theoretically have a jet mismatch, wherein the jet produces different code than the Hoon expression. This is considered grounds for removing a jet from the runtime since it destroys the deterministic reproducibility of Urbit state.

Today, jets are compiled and linked directly into the Urbit runtime. They are only supplied for code in `/sys`. There is not a technical reason for this—any core can be jetted—but it is currently a code hygiene practice to keep things manageable and secure.

* [“Writing Jets”](/build-on-urbit/runtime/jetting)
* [\~rovnys-ricfer, \~lagrev-nocfep, “Dev-Week: Core School Preview”](https://www.youtube.com/watch?v=i1GRoY3QDdU)
* [\~ritpub-sipsyl, “Notes on Jets”](https://hackmd.io/@zorp/HyVWew3ch)

### The Fake Rule: Nock Twelve <a href="#the-fake-rule-nock-twelve" id="the-fake-rule-nock-twelve"></a>

| Nock Rule   | Nock Expression | Hoon Equivalent | Comment           |
| ----------- | --------------- | --------------- | ----------------- |
| Nock Twelve | `*[a 12 b]`     | `.^` dotket     | Scry in namespace |

A scry is a pure function mapping from a path to a response. A scry is a referentially transparent way of referring to data. A `.^` dotket scry resolves via an artificial Nock instruction, Nock Twelve. De facto, we are attempting to solve the problem that sometimes information is necessary to complete a calculation which may not be present in the subject. The `12` instruction is picked up by the virtual Nock implementation within Arvo and the appropriate value inserted. The scry pattern is read-only and synchronous.

While a scry request does violate function purity, it is reproducible due to the scry namespace, in which immutable values are bound by convention at fixed endpoints. The scry namespace is defined by a unique path for each resource.

```
/~zod/1/2/c/x/~2023.8.31..12.00.00..0000/base/sys/arvo/hoon
/host/rift/life/vane/request-type/revision-number/desk/file-path/mark
```

When in userspace, we work with a stripped-down version of the full scry namespace, which is supplied by Arvo. Thus scries are a bit mangled in userspace.

![](https://media.urbit.org/docs/arvo/scry-diagram-v3.svg)

The scry interface defined by Arvo and each vane is not intrinsic to the runtime, but is an OS-level convention. (The values `eny` and `now` are not scries, but atoms provided by Arvo—and thither dynamically by the runtime from the system. More on these in `ca02`.)

* [“Additional Guides: Remote Scry”](/build-on-urbit/userspace/remote-scry)

For instance, consider a namespace scry into Clay:

```
> !=(.^(* %cx /===/gen/cat/hoon))
[ 12
  [1 139 1.853.189.998]
  1
  30.819
  1.685.027.454
  1.702.060.386
  161.311.343.539.592.098.933.172.536.971.821.663.896.217.238.133.980.260.425.806.462
  7.234.919
  7.627.107
  1.852.796.776
  0
]
```

Refactored back into Hoon atoms, we can see:

```
[ 12
  [1 139 %noun]
  1
  %cx
  ~.~zod
  ~.base
  ~.~2023.7.31..20.39.12..1abd
  ~.gen
  ~.cat
  ~.hoon
  ~
]
```

### Virtualized Nock (`+mock`) <a href="#virtualized-nock-mock" id="virtualized-nock-mock"></a>

How does scrying actually work? Hoon can run both Nock and Hoon within itself, and in particular has Nock Twelve implemented in the [`+mink`](/hoon/stdlib/4n#mink) emulator. (`+mink` underlies `+mock`, which we've been using.) `+mink` takes a subject and a formula like any Nock interpreter, and also a gate called any time a formula contains Nock Twelve at its head. Userspace code such as Gall agents is run using `+mink` and thus has the ability to fill out nouns using calls into Arvo to construct its nouns.

`+mock` and friends work by coordinating with the runtime to virtualize Nock:

> With the power of `u3`, we nest arbitrary layers of `mock` without any particular performance cost. Moreover, we simply treat Nock proper as a special case of `mock`. (More precisely, the internal VM loop is `+mink` and the error compiler is `+mook`. But we call the whole sandbox system `mock`.)\
> The nice thing about `mock` functions is that (by executing within `u3m_soft_run()`, which as you may recall uses a nested road) they provide both exceptions and the namespace operator - `.^` in Hoon, which becomes operator `11` in `mock`.

For instance, we have been running `.*` dottar at the Dojo prompt. In fact, this does produce Nock code—but then that Nock code is evaluated by whatever the interpreter is at that level. In userspace, this is `+mock`; in kernelspace, the runtime evaluates the Nock code directly (not a metacircular interpreter). (What consequence does this have for scries?)

This works in userspace because of the metacircular interpreter, but would not work in kernelspace:

```hoon
> ;;(@t .*(. !=(.^(* %cx /===/gen/cat/hoon))))
'::  ConCATenate file listings\0a\0a::\0a::::  /hoon/cat/gen\0a  ::\0a/?    310\  
0a/+    pretty-file, show-dir\0a::\0a::::\0a  ::\0a:-  %say\0a|=  [^ [arg=(list  
path)] vane=?(%g %c)]\0a=-  tang+(flop `tang`(zing -))\0a%+  turn  arg\0a|=  pax  
=path\0a^-  tang\0a=+  ark=.^(arch (cat 3 vane %y) pax)\0a?^  fil.ark\0a  ?:  =(  
%sched -:(flop pax))\0a    [>.^((map @da cord) (cat 3 vane %x) pax)<]~\0a  [leaf  
+(spud pax) (pretty-file .^(noun (cat 3 vane %x) pax))]\0a?-     dir.ark          
                                 ::  handle ambiguity\0a    ~\0a  [rose+[" " `~  
]^~[leaf+"~" (smyt pax)]]~\0a::\0a    [[@t ~] ~ ~]\0a  $(pax (welp pax /[p.n.dir  
.ark]))\0a::\0a    *\0a  =-  [palm+[": " ``~]^-]~\0a  :~  rose+[" " `~]^~[leaf+"  
*" (smyt pax)]\0a      `tank`(show-dir vane pax dir.ark)\0a  ==\0a==\0a'
```

## The Urbit Lifecycle Function <a href="#the-urbit-lifecycle-function" id="the-urbit-lifecycle-function"></a>

The core expression in all of Urbit is the lifecycle function.

```
[2 [0 3] 0 2]
```

This is replay. The idea of Urbit is a computer whose entire lifecycle is defined by a single small fixed frozen function. We will revisit this function when we examine Arvo.

You can find it instantiated directly in `/sys/arvo.hoon` in expressions like `[%9 2 %10 [6 %0 3] %0 2]`.

## Exercises <a href="#exercises" id="exercises"></a>

* Write your own Nock interpreter in Hoon.
* Write your own jet, following the [tutorial](/build-on-urbit/runtime/jetting).
* Optional: Compose a Nock addition gate from scratch. ([Solution](https://github.com/urbit/examples/blob/master/wip/lib/nock.hoon))

## Appendix: The Nock Interpreter <a href="#appendix-the-nock-interpreter" id="appendix-the-nock-interpreter"></a>

As hinted at by our jetting discussion above, Nock is not directly evaluated by the runtime. There are three key parts of the Nock interpreter:

1. Bytecode interpreter
2. Jet dashboard
3. Allocator

This appendix summarizes some of the considerations and architecture that the current runtime interpreters (Vere and NockVM, née Sword, née Ares) must make in some respect.

### Nock Bytecode <a href="#nock-bytecode" id="nock-bytecode"></a>

The VM doesn't actually execute Nock directly from its noun form. Instead it produces a Nock-based bytecode. Producing computationally efficient bytecode from Nock is one key to building a practical Urbit runtime, and has been a focus of the NockVM (née Sword, née Ares) project. The details of this in Vere are in `vere/pkg/noun/nock.c`.

```
> ~>(%xray (add 2 3))
{[fask 1023] [kicb 1] snol head swap tail [lilb 2] swap tail [lilb 3] auto musm [ticb 0] halt}
```

```hoon
> ~>(%xray =+(2 [- -]))
{[litb 2] snol head swap head ault halt}
```

The `%xray` hint on a core doesn't show the core, but it shows the formula that invokes the core.

```
> ~>  %xray.[0 %outer]
=|  i=@
|-  ^-  @
~>  %xray.[0 %inner]
?:(=(i ^~((bex 0))) ~ $(i +(i)))
outer: {lit0 snol [libk i:0] snol [ticb 0] halt}
  inner
  {[fabk 6] sam1 [sbin 3] lil0 [sbip 8] copy swap [fabk 6] bump musm [ticb 0] halt}
```

Core Academy lessons will not further explore the Nock bytecode interpreter.

* [`u3n`: Nock Execution](/build-on-urbit/runtime/api#u3n-nock-execution)
* (no docs exist for the Vere bytecode convention)
* [NockVM (née Sword, née Ares), “Codegen Bootstrapping”](https://github.com/urbit/ares/blob/status/docs/codegen-bootstrap.md?plain=1)

### Jet Dashboard <a href="#jet-dashboard" id="jet-dashboard"></a>

> The jet dashboard is the system in the runtime that registers, validates, and runs jets: specific pieces of Nock code reimplemented in C for performance.

The main part we have to be aware of as developers is jet registration and hinting. However, deeper parts of the jet dashboard do affect performance.

The jet dashboard maintains three jet state systems:

1. `cold` state results from the logical execution history of the pier and consists of nouns. `cold` jet state registers jets as they are found. `cold` state ignore restarts.
2. `hot` state is the global jet dashboard and describes the actual set of jets loaded into the pier for the current running process. Calls to `hot` state result from Nock Nine invocations of a core and an axis. `hot` state is thus tied to process restart.
3. `warm` lists dependencies between `cold` and `hot` state. `warm` state can be cleared at any time and is cleared on restart.

The jet dashboard will not be explored in detail in Core Academy.

* [“API Overview by Prefix: `u3j` jets”](/build-on-urbit/runtime/api#u3j-jets)

### Allocator <a href="#allocator" id="allocator"></a>

The allocator is responsible for memory management. The Vere runtime has some peculiarities, such as the loom/road structure, which are not directly tied to the nature of Nock as a computable language.

Persistence means that we can store events and the event log, and that we can produce the state derived from them as a snapshot.

The allocator will be examined more in the runtime lessons on Vere.

* [Vere, “Land of Nouns: `u3` the road model”](/build-on-urbit/runtime/nouns#u3-the-road-model)
* [NockVM (née Sword, née Ares), “New Mars Persistence”](https://github.com/urbit/ares/blob/status/docs/persistence.md)


# 2. Building Hoon

Core Academy lesson on the Hoon compiler covering vase-mode programming with +slap/+slop algebra, the compilation pipeline from text to AST to Nock, type system mechanics, and Ford file building.

*This lesson explains vase-mode Hoon, including the arms `+slap`, `+slop`, `+slot`, and `+slam`. We examine the compiler stack, and how Ford uses vase mode to process file inclusions.*

## Vase-Mode Hoon <a href="#vase-mode-hoon" id="vase-mode-hoon"></a>

A `vase` is a pair `[type noun]` used to describe data of a type unknown at compile time. (For instance, this can result from compiling and running Hoon code.) We use vases throughout the kernel, such as to run userspace agents in Gall or to upgrade Arvo itself.

What do we mean when we talk about `[type noun]`? Nouns are straightforward but what is a `$type`?

### `$type` <a href="#type" id="type"></a>

Hoon uses the word “type” a few different ways casually (mold, mark, \&c.). `$type` is also a core definition of Hoon, a way of specifying a set of nouns, such as all atoms or all cells. One challenge in interpreting a vase in the Dojo is that the prettyprinter obscures these. Take the type spear of a gate like `+add`, for instance:

```hoon
> -:!>(add)
#t/<1.otf [[a=@ b=@] <33.sam 1.pnw %139>]>

> -<:!>(add)
%hold
```

**`/sys/hoon.hoon`**

```hoon
+$  type  $~  %noun                                     ::
          $@  $?  %noun                                 ::  any nouns
                  %void                                 ::  no noun
              ==                                        ::
          $%  [%atom p=term q=(unit @)]                 ::  atom / constant
              [%cell p=type q=type]                     ::  ordered pair
              [%core p=type q=coil]                     ::  object
              [%face p=$@(term tune) q=type]            ::  namespace
              [%fork p=(set type)]                      ::  union
              [%hint p=(pair type note) q=type]         ::  annotation
              [%hold p=type q=hoon]                     ::  lazy evaluation
          ==                                            ::
```

* `%noun` is the superset of all nouns.
* `%void` is the empty noun, but won't occur in practice.
* `%atom` spans the set of all atoms.
* `%cell` contains ordered pairs.

The other types are more complex:

* `%core` is the descriptor type for a core. Besides the `$type`, it uses a `$coil`, which is a tuple of variance information, context, and chapters (limbs).
* `%face` spans the same set as nouns but includes a face.
* `%fork` is a union, or choice over options.
* `%hint` is an annotation for the compiler (Nock Eleven).
* `%hold` types are lazily evaluated, such as a recursive type (like the `+list` mold builder).

  * `%hold` types are why the compiler can have trouble with lists at runtime, such as needing to distinguish a `lest` or the TMI problem with `+snag` \&c.

  * A `%hold` type is a “finite subtype” of an infinite type. Hoon doesn't actually know about these directly, just in that it can be lazy about evaluating recursions.

  * They result from arms in cores because the `hoon` of the arm is played against the core type as the subject type to get the result. This permits wetness and variance, since the core can be modified to have a sample of a different type.

  > One can "evaluate" a hold by asking the compiler to "play" the hoon against the subject type, meaning to infer what type of value would result from running that hoon against a value of the subject type. For a recursive type, this result type refers to the same hold, usually in one or more of the cases of a `%fork`. (\~rovnys-ricfer)

For instance, let's evaluate a value of the type `(list @ud)`.

```hoon
> =a `(list @ud)`~[1 2 3]

> !>(a)
[#t/it(@ud) q=[1 2 3 0]]

> -<:!>(a)
%hold

> -<:!>(?~(a 0 a))
%fork

> ->-:!>(?~(a 0 a))
#t/@ud
[%atom p=%ud q=~]

> ->+<:!>(?~(a 0 a))
l=[#t/[i=@ud t=it(@ud)] l=~ r=~]
```

The prettyprinter really wants to gussy these up for you, so you have to play some tricks to see the value unexpurgated:

```hoon
> ;;($>(%atom type) -:!>(3))
[%atom p=%ud q=~]

> ;;($>(?(%core %hold) type) -:!>(dec))
[ %hold
...
]

> =+  r=~(repo ut ~(repo ut -:!>(dec)))  ?>(?=(%core -.r) r)
[ %core
 #t/[a=@ <33.sam 1.pnw %139>]
   q
 [ p=[p=~ q=%dry r=%gold]
 ...
 ]
]

> ?? 4
 [%atom %ud ~]
4
```

A `%hold` can be resolved by one step using `(~(play ut type) hoon)`, which type-infers what the result of running that `hoon` would be against a subject of that `type`.

```hoon
> =-  ?>(?=(%fork -<) -)  ~(repo ut -:!>(*(list @)))
[%fork p=[#t/%~ l=[#t/[i=@ t=it(@)] l=~ r=~] r=~]]
```

Whenever we talk about `$type`, this is it. This may in some ways feel odd to those coming from other languages, since we can—and often will—talk about the “type of the subject” for instance. That means this head of the vase, or the type used to interpret the associated noun.

### Building Vases <a href="#building-vases" id="building-vases"></a>

The most common way to get a vase is using the `!>` zapgar rune.

```hoon
> !>('Hello Mars')
[#t/@t q=545.182.085.650.269.906.691.400]
```

A vase can also be manually constructed, e.g.

```hoon
> `vase`[`type`[%atom %ud `4] 4]
[#t/%4 q=4]
```

(This is the approach that the Hoon compiler will use internally when it processes structures.)

### Eliminating Vases <a href="#eliminating-vases" id="eliminating-vases"></a>

> An elimination form for a vase is something that converts a vase to a statically typed value.

There are three common ways to eliminate a vase back into a value:

* `!<` zapgal is an unsafe form which accepts a mold and a vase. (It's unsafe because it doesn't guarantee the value has the type, an evil vase.)

```hoon
> !<(@ !>(4))
4

> !<(path !>(/===))
/~zod/base/~2023.8.30..19.57.38..6b94
```

* `;;` micmic is a safer coercion which applies the `spec` to the `hoon` and yields a mold-coerced form (structurally nests).

```hoon
> ;;(@ 4)
4

> ;;(path /===)
/~zod/base/~2023.8.30..19.58.47..d09f
```

* A mold is a straightforward way to eliminate the vase, à la `;;` micmic.

### Vase Algebra <a href="#vase-algebra" id="vase-algebra"></a>

Given a vase that contains an expression of data and its type, how can we use it besides to just eliminate it back to a static value? We will use vases to build Hoon code, so let's get good at combining and manipulating vases. Facility with vase mode is one of [the bright-line ideas](https://www.joelonsoftware.com/2005/12/29/the-perils-of-javaschools-2/) for kernel work, so read through these concepts several times and ask dumb questions.

The [`+slap`/`+slop` vase algebra](https://github.com/urbit/docs.urbit.org/blob/master/guides/additional/vases/README.md#slap'n'slop-vase-algebra) provides a framework for working through Hoon expressions at an abstract level.

#### `+slap`: Envase Hoon

[`+slap`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#slap) runs a `hoon` against a `vase` and produces a `vase` of the result.

```hoon
++  slap
  |=  [vax=vase gen=hoon]  ^-  vase                     ::  untyped vase .*
  =+  gun=(~(mint ut p.vax) %noun gen)
  [p.gun .*(q.vax q.gun)]
```

First this gate compiles the parsed Hoon expression to Nock against the type of a passed subject using `+mint:ut` (on which more later). Then it envases the product type and the actual evaluation of that Nock against the subject (value) passed in.

Since any Nock formula is resolved against a subject, `+slap` is Hoon's answer to raw `.*` Nock Two evaluation. The type of the subject contains the information about the arms, faces, and raw subject axes.

```hoon
> =>  [one=1 two=2 tri=3]
  !=  [one two]
[[0 2] 0 6]
[[0 2] [0 6]]
```

Examples:

```hoon
> (slap !>(3) (ream '.'))
[#t/@ud q=3]

> (slap !>(3) (ream '+(.)'))
[#t/@ q=4]

> (slap !>(3) (ream '[33 44]'))
[#t/[@ud @ud] q=[33 44]]

> (slap !>(3) (ream '%foo'))
[#t/%foo q=7.303.014]

> (slap !>([33 44]) (ream '.'))
[#t/[@ud @ud] q=[33 44]]

> (slap !>([33 44]) (ream '-'))
[#t/@ud q=33]

> (slap !>([33 44]) (ream '+'))
[#t/@ud q=44]

> (slap !>([foo=33 bar=44]) (ream '.'))
[#t/[foo=@ud bar=@ud] q=[33 44]]

> (slap !>([foo=33 bar=44]) (ream 'foo'))
[#t/@ud q=33]

> (slap !>([foo=33 bar=44]) (ream 'bar'))
[#t/@ud q=44]

> (slap !>([foo=33 bar=44]) (ream '+(foo)'))
[#t/@ q=34]
```

#### `+slop`: Combine Vases

[`+slop`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#slop) combines a `cell` of `vase`s into a `vase` of a `cell`.

```hoon
++  slop                                                ::  cons two vases
  |=  [hed=vase tal=vase]
  ^-  vase
  [[%cell p.hed p.tal] [q.hed q.tal]]
```

Examples:

```hoon
> (slop !>(3) !>(4))
[#t/[@ud @ud] q=[3 4]]

> (slop !>(foo=3) !>(4))
[#t/[foo=@ud @ud] q=[3 4]]

> (slop !>(foo=3) !>(bar=4))
[#t/[foo=@ud bar=@ud] q=[3 4]]

> (slop !>(foo=3) !>([bar=4 baz=5]))
[#t/[foo=@ud bar=@ud baz=@ud] q=[3 4 5]]

> (slop !>(foo=%foo) !>([bar=[4 5] baz=%baz]))
[#t/[foo=%foo bar=[@ud @ud] baz=%baz] q=[7.303.014 [4 5] 8.020.322]]
```

#### Vase Algebra Operations

With `+slap` and `+slop`, we have the ability to build higher-level operators. Each value is a `vase` and the fundamental operators are `+slap` and `+slop`.

```
         +slap
        /      \
    +slop       h2
   /      \
 v1      +slop
        /      \
      v2      +slap
             /      \
           v3        h1

(slap (slop v1 (slop v2 (slap v3 h1))) h2)
```

A concrete example in Hoon:

```hoon
=/  v1=vase  !>(%foo)
=/  v2=vase  !>(%bar)
=/  v3=vase  !>(%baz)
=/  h1=hoon  (ream '%qux')
=/  h2=hoon  (ream '[%result .]')
::
%+  slap
  %+  slop  v1
  %+  slop  v2
  (slap v3 h1)
h2

::  result
[#t/[%result %foo %bar %qux] q=[128.009.175.786.866 7.303.014 7.496.034 7.894.385]]
```

Compilation and execution take place using vase mode, including many operators built of `+slap` and `+slop`. For instance, this is how Ford imports library cores when a file is built to code.

#### `+slam`: Slam Gate with Sample

[`+slam`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#slam) accepts a gate as a `vase` and appropriate arguments.

(This is an older implementation that's cleaner to interpet.)

```hoon
++  slam
  |=  [gat=vase arg=vase]
  ^-  vase
  (slap (slop gat arg) !,(*hoon (- +)))
```

Example:

```hoon
> ;;($>(%hold type) -:(slam !>(dec) !>(5)))
[ %hold
  #t/<1.hkg [a=@ <33.sam 1.pnw %139>]>
    q
  [ %sgcb
      p
    [ p=[%rock p=%tas q=1.717.658.988]
        q
      [ %knit
          p
        ~[100 101 99 114 101 109 101 110 116 45 117 110 100 101 114 102 108 111 119]
      ]
    ]
      q
    [ %wtgl
      p=[%dtts p=[%sand p=%ud q=0] q=[%wing p=~[%a]]]
        q
      [ %tsls
        p=[%ktts p=term=%b q=[%sand p=%ud q=0]]
          q
        [ %brhp
            p
          [ %kthp
            p=[%base p=[%atom p=~.]]
              q
            [ %wtcl
              p=[%dtts p=[%wing p=~[%a]] q=[%dtls p=[%wing p=~[%b]]]]
              q=[%wing p=~[%b]]
              r=[%cnts p=~[%$] q=[i=[p=~[%b] q=[%dtls p=[%wing p=~[%b]]]] t=~]]
            ]
          ]
        ]
      ]
    ]
  ]
]
```

* How do we get the result back out? (Elimination mode, such as `!<`.)

#### `+slot`: Retrieve Noun

[`+slot`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#slot) can be used to decompose vases.

```hoon
++  slot                                                ::  got axis in vase
  |=  [axe=@ vax=vase]  ^-  vase
  [(~(peek ut p.vax) %free axe) .*(q.vax [0 axe])]
```

`(slot 2 vase)` and `(slot 3 vase)` decompose the `vase` back into constituent types and values.

#### Tutorial: Finding the Sample

* How can you retrieve the type of the sample from a vase for a gate?

```hoon
> =+  -:!>(+6:add)
  ?>(?=([%cell *] -) -)
p=[%cell #t/a=@ #t/b=@]

> =+  -:!>(+6:|=([p=@tas q=@da r=^] ~))
 ?>(?=([%cell *] -) -)
p=[%cell #t/p=@tas #t/[q=@da r=[* *]]]
```

* How can you examine the faces of the sample?

```hoon
> =+  -:!>(+12:|=([p=@tas q=@da r=^] ~))
  ?>(?=([%face *] -) -)
p=[%face p=%p #t/@tas]
```

Some places to examine vase mode:

* Dojo uses vase mode to evaluate statements.
* \~rovnys-ricfer produces a simplified Gall using vase mode in [this video](https://drive.google.com/file/d/10SaE5doCfdeqc2j945t8GvGvexKIBBZq/view).
* [Hoon, “Vases”](https://github.com/urbit/docs.urbit.org/blob/master/guides/additional/vases/README.md)
* [\~rovnys-ricfer, “Hoon Vases”](https://rovnys.cataphract.us/vases) (older version but has some diagrams that aren't in the docs version)

## The Compiler Stack <a href="#the-compiler-stack" id="the-compiler-stack"></a>

### The Lifecycle of Hoon (`+ride` or Die) <a href="#the-lifecycle-of-hoon-ride-or-die" id="the-lifecycle-of-hoon-ride-or-die"></a>

Hoon code begins life as text, presumably a `cord` `@t` value. If we would like to go from text interpretable as Hoon code to executable Nock, what does that look like?

There are a few different paths Hoon can take to run the gauntlet:

1. A text file containing Hoon code, processed via the `%hoon` mark.
2. A text `cord` containing Hoon code, processed via the `+ream` arm
3. A noun interpretable as a Hoon AST.

We need to go from Hoon code to a Hoon abstract syntax tree (AST), then from the AST to Nock. We can build and describe this process at several layers of granularity. For instance, `+ride` accomplishes this in one step.

[`+ride`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5d/README.md#ride) accepts a pair of `$type` and an atom (really a `cord`) of text containing Hoon code.

```hoon
++  ride
  |=  [typ=type txt=@]
  ^-  (pair type nock)
  ~>  %slog.[0 leaf/"ride: parsing"]
  =/  gen  (ream txt)
  ~>  %slog.[0 leaf/"ride: compiling"]
  ~<  %slog.[0 leaf/"ride: compiled"]
  (~(mint ut typ) %noun gen)
```

Line by line:

* `|= [typ=type txt=@]`
  * Accept a `$type` and a `cord`.
* `^- (pair type nock)`
  * A pair of `$type` and compiled `$nock` result. A `nock` is a cell of numbers interpretable as a Nock formula. Notably, the Nock rules tend to be embedded as constants (such as `%0`), which makes it more readable than conventional Nock.
* `~> %slog.[0 leaf/"ride: parsing"]`
  * Issue a raw hint to output a starting message.
* `=/ gen (ream txt)`
  * `+ream` the `cord`, which compiles the Hoon expression into a Hoon abstract syntax tree (AST).
* `~> %slog.[0 leaf/"ride: compiling"]`
  * Issue a raw hint to output a continuation message.
* `~< %slog.[0 leaf/"ride: compiled"]`
  * Issue a raw hint to the product (so it prints after the completion).
* `(~(mint ut typ) %noun gen)`
  * Do the shovel work of compiling the parsed AST into Nock code, using the type `%noun`.

As another example, [`+make`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5d/README.md#make) shows the process in capsule form for converting a `cord` of Hoon code into executable `$nock` Nock code. `+make` doesn't explicitly require or deal with `$type` like `+ride` does.

```hoon
++  make
  |=  txt=@
  q:(~(mint ut %noun) %noun (ream txt))
```

Given a Hoon expression, convert it to `$nock`.

```hoon
> (make '~[1 2 3]')
[%1 p=[1 2 3 0]]
```

### Text `cord` → AST `$hoon` <a href="#text-cord-ast-hoon" id="text-cord-ast-hoon"></a>

#### `+ream`

To take a `cord` containing Hoon code and transform it to an AST, use [`+ream`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5d/README.md#ream):

```hoon
> (ream '(add 1 2)')
[%cncl p=[%wing p=~[%add]] q=~[[%sand p=%ud q=1] [%sand p=%ud q=2]]]

> (ream '-:!>(5)')
[%tsgl p=[%cnts p=~[[%.y p=2]] q=~] q=[%zpgr p=[%sand p=%ud q=5]]]
```

`+ream` simply wraps `+vast`:

```hoon
++  ream                                                ::  parse cord to hoon
  |=  txt=@
  ^-  hoon
  (rash txt vest)
```

Any irregular annotation is changed into its basic Hoon form. However, at this point no desugaring has taken place; equivalent forms may still have different AST representations as `$hoon`:

```hoon
> (ream '~[1 2 3]')
[%clsg p=~[[%sand p=%ud q=1] [%sand p=%ud q=2] [%sand p=%ud q=3]]]

> (ream '[1 2 3 ~]')
[%cltr p=~[[%sand p=%ud q=1] [%sand p=%ud q=2] [%sand p=%ud q=3] [%bust p=%null]]]
```

You can also clearly see how Hoon supplements "pure" values with metadata to establish context for the values.

#### `+vast`

`+ream` is a wrapper for the main parser `+vast`. While `+vast` spans all of Hoon parsing, it is particularly intended to parse Hoon, unsurprisingly. It starts with an attempt to parse in tall form, which will fall back to wide form as necessary. A `+gay` is a way to ignore `+gap` plural whitespace on either side of the text (file).

```hoon
++  vest
  ~/  %vest
  |=  tub=nail
  ^-  (like hoon)
  %.  tub
  %-  full
  (ifix [gay gay] tall:vast)
```

The `+vang` wrapper lets you turn off debugging info and doccords in parsing.

```hoon
> (rash 'goo' tall:(vang | /))
[%wing p=~[%goo]]

> (rash 'goo' tall:(vang & /))
[%dbug p=[p=/ q=[p=[p=1 q=1] q=[p=1 q=4]]] q=[%wing p=~[%goo]]]
```

#### `+open:ap`

A parsed Hoon AST has not yet been desugared. Many Hoon runes are simply convenience wrappers over a few fundamental runes. (For instance, most (all?) `%` cen runes reduce to `%~` censig.) `+open:ap` unwraps these one layer at a time until fundamental runes are reached in the Hoon AST.

```hoon
> (ream '(add 1 2)')
[%cncl p=[%wing p=~[%add]] q=[i=[%sand p=%ud q=1] t=[i=[%sand p=%ud q=2] t=~]]]

> ~(open ap (ream '(add 1 2)'))
[ %cnsg
  p=~[%$]
  q=[%wing p=~[%add]]
  r=[i=[%sand p=%ud q=1] t=[i=[%sand p=%ud q=2] t=~]]
]

> ~(open ap ~(open ap (ream '(add 1 2)')))
[ %cntr
 p=~[%$]
 q=[%wing p=~[%add]]
   r
 [ i=[p=~[[%.n p=0 q=~] [%.y p=12]] q=[%sand p=%ud q=1]]
   t=[i=[p=~[[%.n p=0 q=~] [%.y p=13]] q=[%sand p=%ud q=2]] t=~]
 ]
]

> ~(open ap ~(open ap ~(open ap (ream '(add 1 2)'))))
[ %tsls
 p=[%wing p=~[%add]]
   q
 [ %cnts
   p=~[%$ [%.y p=2]]
     q
   [   i
     [p=~[[%.n p=0 q=~] [%.y p=12]] q=[%tsgr p=[%$ p=3] q=[%sand p=%ud q=1]]]
       t
     [   i
       [ p=~[[%.n p=0 q=~] [%.y p=13]]
         q=[%tsgr p=[%$ p=3] q=[%sand p=%ud q=2]]
       ]
       t=~
     ]
   ]
 ]
]

> ~(open ap ~(open ap ~(open ap ~(open ap (ream '(add 1 2)')))))
[ %tsgr
 p=[p=[%wing p=~[%add]] q=[%$ p=1]]
   q
 [ %cnts
   p=~[%$ [%.y p=2]]
     q
   [   i
     [p=~[[%.n p=0 q=~] [%.y p=12]] q=[%tsgr p=[%$ p=3] q=[%sand p=%ud q=1]]]
       t
     [   i
       [ p=~[[%.n p=0 q=~] [%.y p=13]]
         q=[%tsgr p=[%$ p=3] q=[%sand p=%ud q=2]]
       ]
       t=~
     ]
   ]
 ]
]
```

* [Hoon, “Constants (Atoms and Strings)”](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/rune/constants/README.md)
* Examine `+open:ap` in `/sys/hoon.hoon` to see how desugaring proceeds.

### AST `$hoon` → Nock `$nock` <a href="#ast-hoon-nock-nock" id="ast-hoon-nock-nock"></a>

All Hoon code is ultimately run by `+mint:ut`, although there can be many paths there. `+mint:ut` parses from a Hoon AST into a pair of the type and the Nock. What is [`+ut`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#ut)? It's the Hoon compiler backend, containing all of the arms necessary to actually process a `$hoon` into a `$nock`.

One simple approach is to look at the 2013 Hoon compiler, which presents a relatively uncluttered version

* [`+mint:ut` (2013)](https://github.com/urbit/archaeology/blob/6b2ce202207b9bb3f4e65fc1ea9a2fb434396dd4/urb/zod/arvo/hoon.hoon#L7698)

In all cases, the parent door `+ut` receives the sample of `$type` when an arm is invoked. The significant arms include:

**`+mull:ut`**

`+mull` is an assertion for wet gates. Each place in the code that calls a wet gate needs the compiler to check in order to ensure that the wet gate when called with that sample would have the same Nock as it would otherwise.

* `sut` is the subject type
* `gol` is a product constraint (result type must nest in `gol`, passed around to support accurate stack traces)
* `dox` is the formal subject type

`+mull` is conceptually equivalent to saying, “compile this expression against the actual subject type `sut`, compile it again against the formal type `dox`, assert that they produce the same Nock”.

In practice, +mull traverses both subject types at the same time, short-circuiting in a couple of scenarios, and simply crashing if the (conceptual) result would be different.

**`+find:ut`**

`+find` is a wing resolution arm. It tries Nock Zero first to see if the wing is a leg, then tries Nock Nine if that fails. It also needs information about the core variance (`%read`, `%rite`, `%both`, `%free`); see [`+slab`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#slab) as well, which also uses `+fond` under the hood.

```hoon
> (slab %read %$ -:!>(add))
%.y

> (slab %read %a -:!>(add))
%.y

> (slab %read %b -:!>(add))
%.y

> (slab %read %c -:!>(add))
%.n
```

**`+nest:ut`**

`+nest` provides a structural test on whether two `$type`s nest properly. It is called via `+nice` so that the `need`/`have` error messages can be presented neatly.

**`+mint:ut`**

`+mint` is quite long, but deserves some attention.

* `+mint` itself takes a pair of `typ` and `hoon`, a Hoon AST.
* `+mint` produces a pair of `type` and `nock`.
* It features a lot of `+nice` (`+nest`) checks.
* Each branch in the main `switch` operates on an AST tag to convert it recursively to `$nock`.

```hoon
> (~(mint ut %noun) %noun (ream '~[1 2 3]'))
[#t/[@ud @ud @ud %~] q=[%1 p=[1 2 3 0]]]
```

This can be evaluated as `$nock` then using `.*` dottar:

```hoon
> .*(. (make '~[1 2 3]'))
[1 2 3 0]
```

`+mint:ut` is never called on its own in the compiler. It's used to generate a Nock formula then to run it against the subject to make it useful, e.g. for an agent running a formula against the standard library plus imports.

```hoon
> (~(mint ut -:!>(.)) %noun ~(open ap (ream '(add 1 2)')))
[ #t/@
    q
  [ %8
    p=[%9 p=36 q=[%0 p=2.047]]
      q
    [ %9
      p=2
        q
      [ %10
        p=[p=6 q=[p=[%7 p=[%0 p=3] q=[%1 p=1]] q=[%7 p=[%0 p=3] q=[%1 p=2]]]]
        q=[%0 p=2]
      ]
    ]
  ]
]

> !=((add 1 2))
[8 [9 36 0 2.047] 9 2 10 [6 [7 [0 3] 1 1] 7 [0 3] 1 2] 0 2]

> ;;($>(?(%hold %core) type) -:(~(mint ut -:!>(.)) %noun ~(open ap (ream '(add 1
2)'))))
[ %hold
 #t/<1.otf [[a=@ b=@] <33.sam 1.pnw %139>]>
   q
 [ %kthp
   p=[%base p=[%atom p=~.]]
     q
   [ %wtcl
     p=[%dtts p=[%sand p=%ud q=0] q=[%wing p=~[%a]]]
     q=[%wing p=~[%b]]
       r
     [ %cnts
       p=~[%$]
         q
       [ i=[p=~[%a] q=[%cncl p=[%wing p=~[%dec]] q=[i=[%wing p=~[%a]] t=~]]]
         t=[i=[p=~[%b] q=[%dtls p=[%wing p=~[%b]]]] t=~]
       ]
     ]
   ]
 ]
]

;;($>(?(%hold %core) type) -:(~(mint ut -:!>(.)) %noun ~(open ap (rash 'add' t
all:(vang | /)))))
```

In the case of a core, `+mint` returns a 3-tuple. The head is the battery of the core; the middle is the default sample, or Nock One of the bunted sample; the tail is `[0 1]`, to take the subject and put it in the `context` of the core.

```hoon
> +:(~(mint ut -:!>(~)) %noun !,(*hoon |=(@ +<)))
q=[%8 p=[%1 p=0] q=[p=[%1 p=[0 6]] q=[%0 p=1]]]
```

The Hoon compiler is not an optimizing compiler. It can recognize and replace some simple expressions, like Nock Seven of two Nock Zeros into a single Nock Zero.

(As an aside, note that Hoon only uses `#` hax in a rune as a placeholder for an experimental rune, or a rune that is involved in the first stage of a multi-stage upgrade process, like swapping two runes and needing a temporary rune to use in the intermediate version of the language.)

Ted points out that the nature of Hoon as an ergonomic harness to generate Nock code is here very apparent. There is no additional runtime system materials like C, C++, or Rust would inject; there is no extra Nock glue. Hoon is a good assembly language (which is really what it's for)—so one future of Hoon is to strip out features and make it more concrete bare-bones then build things on top of it.

* [Hoon 141 Compiler Documentation](https://docs.google.com/document/d/1C0k0AY2vsFu5fNfp0nKiU0GsKThkLMLdHBReqAXNy_w/edit)

Questions:

* If you create a Gall agent with the wrong number of arms (i.e. add an arm), what goes wrong? Where does this error arise in the compilation process?

#### `$nock`

While there are no surprises in `$nock` as a representation of Nock nouns, the use of constants for the rules and the deferral of formulae as subnocks is very readable.

```hoon
+$  nock  $^  [p=nock q=nock]                           ::  autocons
          $%  [%1 p=*]                                  ::  constant
              [%2 p=nock q=nock]                        ::  compose
              [%3 p=nock]                               ::  cell test
              [%4 p=nock]                               ::  increment
              [%5 p=nock q=nock]                        ::  equality test
              [%6 p=nock q=nock r=nock]                 ::  if, then, else
              [%7 p=nock q=nock]                        ::  serial compose
              [%8 p=nock q=nock]                        ::  push onto subject
              [%9 p=@ q=nock]                           ::  select arm and fire
              [%10 p=[p=@ q=nock] q=nock]               ::  edit
              [%11 p=$@(@ [p=@ q=nock]) q=nock]         ::  hint
              [%12 p=nock q=nock]                       ::  grab data from sky
              [%0 p=@]                                  ::  axis select
          ==                                            ::
```

### Files with Imports (`+ford`) <a href="#files-with-imports-ford" id="files-with-imports-ford"></a>

`+ford` is the code builder arm (formerly vane) which handles producing code from a file on a desk, including library imports. Ford uses the `+slap`/`+slop` algebra to produce the subject type and value.

For instance, with `/-` wuthep, Ford builds the `/sur` file then `+slop`s that `vase` with `%zuse` to yield the new build subject, the new `vase`. That combined vase is the subject when Ford `+slap`s the file contents. This illustrates the production of a high-level AST using the `+slap` of `vase` and `hoon` dependent on the `+slop` of two or more `vase`s.

Clay maintains some state to trigger Ford builds, for instance if the agent is rebuilt after a `|commit`. Thus Gall can subscribe to the next revision of any query that can be sent to Clay. As of December 2022, Clay maintains an official state about which agents are supposed to be running from each desk, then eagerly runs the build.

* [\~rovnys-ricfer, “Ford Pinto Spec”](https://gist.github.com/belisarius222/dc2aae8230a22a84d389c19cb613a1d2)
* [\~rovnys-ricfer, “Ford Fusion”](https://urbit.org/blog/ford-fusion)

#### `-build-file`

The `-build-file` and `-build-dependency` threads are the simplest way to invoke `+ford`. Ultimately these resolve through `/lib/strandio`:

**`/lib/strandio.hoon`**

```hoon
::
::  +build-file: build the source file at the specified $beam
::
++  build-file
  |=  [[=ship =desk =case] =spur]
  =*  arg  +<
  =/  m  (strand ,(unit vase))
  ^-  form:m
  ;<  =riot:clay  bind:m
    (warp ship desk ~ %sing %a case spur)
  ?~  riot
    (pure:m ~)
  ?>  =(%vase p.r.u.riot)
  (pure:m (some !<(vase q.r.u.riot)))
```

We will consider marks and tubes in the next lesson.

### Tutorial: Trace a Rune AST <a href="#tutorial-trace-a-rune-ast" id="tutorial-trace-a-rune-ast"></a>

The [`|$` barbuc](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/rune/bar/README.md#-barbuc) rune is used to build a mold builder gate. (As such, it's fairly abstract.) Essentially, it is sugar for a certain use of [`|*` bartar](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/rune/bar/README.md#-bartar):

```hoon
|$  [a b]
body

|*  [a=$~(* $-(* *)) b=$~(* $-(* *))]
^:
body
```

The formal AST for `%brbc` is `[%brbc sample=(lest term) body=spec]`. How is this actually built?

Let's look at the sample first. The sample is a `(lest term)`, or a non-empty list of identifiers. Notice, however, that the actual value passed in (like `[a b]`) is *not* a list at all. Several runes that accept a `list` type (like `:~` colsig) handle adding the terminal `~` in the first parsing step.

The definition of `%brbc` is a little dense but it's doing the same thing, so the sample isn't explicitly a `lest` but becomes one in the first pass of building the rune AST.

In `hoon.hoon`, find the following line:

```
[%brbc sample=(lest term) body=spec]                ::  |$
```

Next, find where it is parsed in `+vast`, the main parsing core. This will tell you how the rune children are parsed.

```hoon
++  expression
  %-  stew
  ^.  stet  ^.  limo
  :~  :-  '|'
        ;~  pfix  bar
          %-  stew
          ^.  stet  ^.  limo                              
          :~  ['$' (rune buc %brbc exqe)]
          ==                                            
        ==
  ==
```

In this case, the rune children are handled using `+exqe`, which automatically turns the sample into a list of names:

```hoon
++  exqe  |.(;~(goop lynx loan))                    ::  list of names then spec                 
```

In this expression, there are three new names:

1. `+goop` is a separated `list` with doccord compatible docs included.
2. `+lynx` is a parser for `a` or `[a b c]` or `a b c ==`.
3. `+loan` parses a `spec`.

So by the time the sample is processed, it has already acquired `list`-type from `+exqe`.

Next, let's look at the direct handling of the result of `+exqe`:

```hoon
[%brbc *]  =-  ?~  -  !!                                                                    
                 :+  %brtr                            
                   [%bccl -]
                 |-
                 ?.  ?=([%gist *] body.gen)
                   [%ktcl body.gen]
                 [%note p.body.gen $(body.gen q.body.gen)]
             %+  turn  `(list term)`sample.gen           
             |=  =term
             ^-  spec
             =/  tar  [%base %noun]
             [%bcts term [%bcsg tar [%bchp tar tar]]]
```

Desugared and annotated:

```hoon
=+  ::  first parse the sample into a lest
    %+  turn  `(list term)`sample.gen
    |=  =term  ^-  spec
    [%bcts term [%bcsg [%base %noun] [%bchp [%base %noun] [%base %noun]]]]
    ::
    ::  if the foregoing turned list is empty, crash
    ?~  -  !!
    ::  otherwise, produce the desugared |*
    :+  %brtr
      ::  this is the cell of the sample in the first arm
      [%bccl -]
    |-
    ?:  ?=([%gist *] body.gen)
      ::  if it's a gist, then include the doccords note
      [%note p.body.gen $(body.gen q.body.gen)]
    ::  otherwise just include the body
    [%ktcl body.gen]
```

Look for the structure:

```hoon
|*  [a=$~(* $-(* *)) b=$~(* $-(* *))]
^:
body
```

#### Exercise

* Carry out the foregoing analysis for another sugar rune, such as `?~`, `:~`, `=.`, or `=/`.

### Evaluating Nock <a href="#evaluating-nock" id="evaluating-nock"></a>

As mentioned in `ca00`, Nock is dispatched to the Nock interpreter in the runtime, where it is evaluated as a mixture of Nock bytecode and runtime jets. Different parts of the system can have different Nock interpreters; for instance, in userspace, the `+mink` metacircular interpreter is used so that crashes can be handled.

### Hoon and Nock Nine <a href="#hoon-and-nock-nine" id="hoon-and-nock-nine"></a>

In a sense, Hoon is an assembly language macro for Nock as machine code. One of the most common patterns in Hoon is how a gate is defined as a core, then invoked into a particular instance by replacing the sample then evaluating the statement. (This happens in gate-building gates but elsewhere as well.)

```
> !=  %-(add [3 4])
[8 [9 36 0 2.047] 9 2 10 [6 7 [0 3] 1 3 4] 0 2]
```

This code takes the `add` core, modifies its sample to be constant `[3 4]`, then fires the `$` arm at axis `2`. Then `[9 36 …]` fires the `add` arm of that core to get the `add` gate.

You can connect how Nock thinks of a double Nock Nine invocation to how Hoon has a `%hold` then resolves it into a `%core`. For instance, let's look inside of `+add`, where we can get the `%hold` and then see the AST (on which more later). Here the `%hold` means that the “actual” type of `+add` has not yet been calculated—although it is a core, it will have to be built with a particular sample for it to result in a `%core`. (What I mean by this is that `%hold` is a lazy evaluation.)

```hoon
> +2:!>(add)
#t/<1.otf [[a=@ b=@] <33.rnj 1.pnw %139>]>

> +4:!>(add)
%hold

> +10:!>(add)
#t/<33.rnj 1.pnw %139>

> +22:!>(add)
%sgfs

> +46:!>(add)
i='add'
> +47:!>(add)
  q
[ %note
    p
  [ %help
      p
    [ cuff=~
        crib
      [ summary='unsigned addition'
        details=~[~[[p=%.y q='a: augend'] [p=%.y q='b: addend']]]
      ]
    ]
  ]
    q
  [ %brts
      p
    [ %bccl
        p
      [ i=[%bcts p=term=%a q=[%base p=[%atom p=~.]]]
        t=[i=[%bcts p=term=%b q=[%base p=[%atom p=~.]]] t=~]
      ]
    ]
      q
    [ %kthp
      p=[%base p=[%atom p=~.]]
        q
      [ %wtcl
        p=[%dtts p=[%sand p=%ud q=0] q=[%wing p=~[%a]]]
        q=[%wing p=~[%b]]
          r
        [ %cnts
          p=~[%$]
            q
          [ i=[p=~[%a] q=[%cncl p=[%wing p=~[%dec]] q=[i=[%wing p=~[%a]] t=~]]]
            t=[i=[p=~[%b] q=[%dtls p=[%wing p=~[%b]]]] t=~]
          ]
        ]
      ]
    ]
  ]
]
```

```hoon
> +20:!>(*add)
%core

> +23:!>(*add)
 q
[ p=[%base p=[%atom p=~.]]
   q
 [ %wtcl
   p=[%dtts p=[%sand p=%ud q=0] q=[%wing p=~[%a]]]
   q=[%wing p=~[%b]]
     r
   [ %cnts
     p=~[%$]
       q
     [ i=[p=~[%a] q=[%cncl p=[%wing p=~[%dec]] q=[i=[%wing p=~[%a]] t=~]]]
       t=[i=[p=~[%b] q=[%dtls p=[%wing p=~[%b]]]] t=~]
     ]
   ]
 ]
]
```

## Exercises <a href="#exercises" id="exercises"></a>

* Exercise: The original `;<` micgal macro was unhygienic; that is, it introduced a spurious `$` each time it was called. This could be worked around using `=* foo $` after a `|-`, but was inconvenient. The original AST expansion of the macro was `[%cnls [%cnhp q [%ktcl p]] r [%brts p s]]:gen`; this was improved to `[%cnls [%cnhp q [%ktcl p]] r [%brts p [%tsgr $+3 s]]]:gen`. Explain how this works.
  * `[%tsgr $+3 s]` is `=>(+ s)`, i.e., the body of the generated gate `s` should not be evaluated against the generated gate but against the payload at axis `+3`; this will be both the sample and context, where the sample is the newly bound value `.p` and the context is the subject against which the `;<` was expressed. Now it works just like `=/`.
* Exercise: Write your own [`+slam`](https://github.com/urbit/docs.urbit.org/blob/master/hoon/reference/stdlib/5c/README.md#slam) using `+slap` and `+slop`.
* Exercise: Implement a rune. The easiest rune to implement is a five-tuple; let's call it `:#` colhax. The parser (`+vast`) and the AST processor (`+open:ap`) need to be modified for this rune to work. Formally this is a language change, but you should be able to just reload `/sys/hoon.hoon` automatically and have it update in place. Alternatively, implement the ;. micdot rune.


# 3. The Core Stack

Core Academy lesson on Urbit's core architecture. Subject-oriented programming, limb resolution mechanics, nested core structures, and the Hoon standard library organization.

*This lesson goes over the nested core structures of `/sys/hoon.hoon`, `/sys/zuse.hoon`, and `/sys/lull.hoon`. We explain how subject search and limb resolution work.*

## Subject-Oriented Programming <a href="#subject-oriented-programming" id="subject-oriented-programming"></a>

### Subject Search and Limb Resolution <a href="#subject-search-and-limb-resolution" id="subject-search-and-limb-resolution"></a>

A face is a label for an axis in a tree. The main use for a face is to label a slot (axis) within a noun of a corresponding type. Without faces, you would have to refer to all data by numeric axis. Faces are a Hoon convention, and Nock knows nothing about labels or faces. These values are stripped out of the Nock result. In fact, it's possible but cumbersome to construct Hoon programs without labels.

From a `$type` perspective, a face results from `=^` kettis (`foo=bar`) modifying the enclosed expression to be wrapped in `[%face %foo original-type]`.

```hoon
> =/  a  b=c=5
 c.b.a
5

> =/  a  b=c=5
 ([%face type] -:!>(a))

> =/  a  b=c=5
 ([%face *] -:!>(a))
[1.701.011.814 98 1.701.011.814 99 1.836.020.833 25.717 0]
```

An arm name is not the same thing as a face. `+add` is the name of an arm in the standard subject. When that arm is fired, the result is the `+add` gate, which is then `+slam`med by swapping out the sample with an argument and firing the `$` arm.

Wings are expressions that compile to an axis. A `$wing` is a `(list limb)`, or basically a `path` to a value in the subject. We can compose wings:

1. Relatively, using lark syntax (`+>`).
2. Absolutely, using numeric syntax (`+6`, `&6`, `|6`).
3. By name, using faces and arm names.

```hoon
+$  limb  $@  term                                      ::  wing element
          $%  [%& p=axis]                               ::  by geometry
              [%| p=@ud q=(unit term)]                  ::  by name
          ==                                            ::
+$  axis  @                                             ::  tree address
+$  wing  (list limb)                                   ::  search path
```

Wings are parsed by `+rope` (`+ven` for lark syntax; `+lus`+`+pam`+`+bar` for numeric syntax). `,` dot wing resolution is conducted by the `+ax:musk` door in the Hoon compiler (invoked by `+open:ap`). This resolves a wing against a sample-supplied subject. Wings resolve by depth first (in other words, from the outermost “closest” match towards the inner cores).

```hoon
> (spec +:(ream '$:(a=@ b=@)'))
[ %bccl
   p
 [ i=[%bcts p=term=%a q=[%base p=[%atom p=~.]]]
   t=[i=[%bcts p=term=%b q=[%base p=[%atom p=~.]]] t=~]
 ]
]

::  Produce a bunt of the given spec.
> ~(example ax (spec +:(ream '$:(a=@ b=@)')))
[p=[%ktts p=term=%a q=[%sand p=%$ q=0]] q=[%ktts p=term=%b q=[%sand p=%$ q=0]]]

::  Produce a normalizing gate (mold) for a given spec, as an AST.
> ~(factory ax (spec +:(ream '$:(a=@ b=@)')))
[ %brcl
    p
  [ %ktsg
      p
...
```

`^` skips a match. In the compiler, this corresponds to a number of skips.

```hoon
> (ream '$')
[%wing p=~[%$]]

> (ream '^$')
[%wing p=~[[%.n p=1 q=[~ %$]]]]

> (ream '^^$')
[%wing p=~[[%.n p=2 q=[~ %$]]]]
```

```hoon
> (spec +:(ream '*spec'))
[%like p=~[%spec] q=~]
```

`.` dot is Hoon-native syntax (not rune sugar) for a wing resolution search path. `:` col is a shorthand for `=>` tisgar, and generally results in a longer Hoon AST than the `.` dot expression would. (The Nock formula may well come out the same.)

```hoon
> =/  a  [b=42]
 !,(*hoon b.a)
[%wing p=~[%b %a]]

> =/  a  [b=42]
 !,(*hoon b:a)
[%tsgl p=[%wing p=~[%b]] q=[%wing p=~[%a]]]

> =/  a  [b=42]
 !=(b:a)
[0 2]

> =/  a  [b=42]
 !=(b.a)
[0 2]
```

Since the subject of a core is the core itself, `..add` resolves to the core containing `+add` (which is Layer 1) and thus exposes mutual visibility between all arms in the core.

We have made some noise in the past about arms and legs. With everything under your belt at this point, you are equipped to really understand the difference:

* A *leg* is a noun accessible in the current subject using a Nock Zero call. Thus a value like `=/ pi .3.1415926` would be a leg.
* An *arm* is a noun which requires a Nock Nine call. Thus `++ pi .3.1415926` would be an arm even though it is an atom *simpliciter*.

When the compiler dereferences a limb, it either finds an arm (in the battery of a core) or a leg (anywhere else). For an arm, it must be computed against the whole core (Nock Nine) or simply retrieved (Nock Zero).

Arms are only pulled by name. If you retrieve them by axis or lark syntax then they are treated as raw nouns. The name of an arm is not a face.

```hoon
> => 
  |%
  ++  $  .+  100
  --
  +2
[4 1 100]

> => 
  |%
  ++  $  .+  100
  --
  $
101
```

### Structure Mode <a href="#structure-mode" id="structure-mode"></a>

Most Hoon is written in value mode, meaning that sugar syntax like `[]` resolves to a `:` col family rune. However, `$spec` values are written in structure mode.

```hoon
> !,(*hoon [a=@ b=@])
[ %cltr
    p
  [ i=[%ktts p=term=%a q=[%base p=[%atom p=~.]]]
    t=[i=[%ktts p=term=%b q=[%base p=[%atom p=~.]]] t=~]
  ]
]

> !,(*hoon $:(a=@ b=@))
[ %ktcl
    p
  [ %bccl
      p
    [ i=[%bcts p=term=%a q=[%base p=[%atom p=~.]]]
      t=[i=[%bcts p=term=%b q=[%base p=[%atom p=~.]]] t=~]
    ]
  ]
]
```

The Hoon parser can be switched from one to the other using a leading `,` com.

```hoon
> !,(*hoon ,[a=@ b=@])
[ %ktcl
    p
  [ %bccl
      p
    [ i=[%bcts p=term=%a q=[%base p=[%atom p=~.]]]
      t=[i=[%bcts p=term=%b q=[%base p=[%atom p=~.]]] t=~]
    ]
  ]
]
```

The root type of a structure mode quantity is a `$spec`:

```hoon
+$  spec                                                ::  structure definition
          $~  [%base %null]                             ::
          $%  [%base p=base]                            ::  base type
              [%dbug p=spot q=spec]                     ::  set debug
              [%gist p=[%help p=help] q=spec]           ::  formal comment
              [%leaf p=term q=@]                        ::  constant atom
              [%like p=wing q=(list wing)]              ::  reference
              [%loop p=term]                            ::  hygienic reference
              [%made p=(pair term (list term)) q=spec]  ::  annotate synthetic
              [%make p=hoon q=(list spec)]              ::  composed spec
              [%name p=term q=spec]                     ::  annotate simple
              [%over p=wing q=spec]                     ::  relative to subject
          ::                                            ::
              [%bcgr p=spec q=spec]                     ::  $>, filter: require
              [%bcbc p=spec q=(map term spec)]          ::  $$, recursion
              [%bcbr p=spec q=hoon]                     ::  $|, verify
              [%bccb p=hoon]                            ::  $_, example
              [%bccl p=[i=spec t=(list spec)]]          ::  $:, tuple
              [%bccn p=[i=spec t=(list spec)]]          ::  $%, head pick
              [%bcdt p=spec q=(map term spec)]          ::  $., read-write core
              [%bcgl p=spec q=spec]                     ::  $<, filter: exclude
              [%bchp p=spec q=spec]                     ::  $-, function core
              [%bckt p=spec q=spec]                     ::  $^, cons pick
              [%bcls p=stud q=spec]                     ::  $+, standard
              [%bcfs p=spec q=(map term spec)]          ::  $/, write-only core
              [%bcmc p=hoon]                            ::  $;, manual
              [%bcpm p=spec q=hoon]                     ::  $&, repair
              [%bcsg p=hoon q=spec]                     ::  $~, default
              [%bctc p=spec q=(map term spec)]          ::  $`, read-only core
              [%bcts p=skin q=spec]                     ::  $=, name
              [%bcpt p=spec q=spec]                     ::  $@, atom pick
              [%bcwt p=[i=spec t=(list spec)]]          ::  $?, full pick
              [%bczp p=spec q=(map term spec)]          ::  $!, opaque core
```

A `spec` produces a mold, thus a `%core` with a `$` arm, rather than e.g. a `%cell`.

```hoon
> -<:!>(,[a=@ b=@])
%core

> -<:!>([a=@ b=@])
%cell

> (,[%const *] [%const 10])
[%const 10]

> $:,[%const *]
[%const 0]

> -:,[%const *]
[ 8
  [[6 [5 [0 12] 1 500.152.823.651] [1 500.152.823.651] 0 0] 0 13]
  8
  [5 [0 14] 0 2]
  0
  6
]

> +6:,[%const *]
[500.152.823.651 0]

> $:@
0

> !<(@ (slam !>(@) !>(5)))
5
```

### Cores <a href="#cores" id="cores"></a>

In the AST, a `%core` consists of a lot of information about the behavior of various components.

```hoon
+$  type  ...
  $:  %core
    $=  p
      $:  p=(unit term)
          q=?(%wet %dry)
          r=?(%gold %iron %lead %zinc))
      ==
    $=  q
      $:  $=  p
              $:  p=(unit term)
                  q=?(%wet %dry)
                  r=?(%gold %iron %lead %zinc))
              ==
          $=  q  type
          $=  r  (pair seminoun (map term tome))
  ==
+$  tome  (pair what (map term hoon))
```

The core variance is repeated because of the dry/wet gate distinction. [Core variance](https://github.com/urbit/docs.urbit.org/tree/master/guides/core/hoon-school/R-metals/README.md) starts to make more sense once you've popped the cover off of cores this way.

> “Suppose this core was actually *compiled* using the modified payload instead of the one it was originally built with? Would the Nock formula we generated for the original template actually work for the modified `payload`?”

What we're saying, in other words, is that if you produced Nock using a wet gate via more than one input, would that Nock end up the same? If so, then for a wet gate it's valid. Wetness is handled at three points in the compiler:

1. `+hemp` dispatches the Nock formula generation slightly differently, turning off `vet` (sample nesting) in `+mint:ut`.
2. `+mint:ut` when it builds `|@` barpat wet doors and `|*` bartar wet gates.
3. `+dext:crop:ar` when `%core` types are handled, enforcing the condition that for `%wet` gates `=(q.r.q.sut q.r.q.ref)`, that the formula results are the same.

Variance matters when comparing structural nesting. For instance, the main Gall agent type should permit checking the type of the door since it will be used as examples for building actual agent cores, but should not be reliant on things like the sample. Thus in `/sys/lull`, `+agent` is marked as `%iron` using `^|` ketbar.

In `/sys/lull`, several shared representations like vane interfaces and `+http` are marked as `%lead` using `^?` ketwut. Bivariance here permits any kind of nesting, useful for examples for types.

In `+deem:nest:ut` we can see how the `%read`/`%rite` permissions are directly set.

```hoon
++  deem
  |=  [mel=vair ram=vair]
  ^-  ?
  ?.  |(=(mel ram) =(%lead mel) =(%gold ram))  |
  ?-  mel
    %lead  &
    %gold  meet
    %iron  dext(sut (peek(sut ref) %rite 2), ref (peek %rite 2))
    %zinc  dext(sut (peek %read 2), ref (peek(sut ref) %read 2))
  ==
```

Likewise in `+peel:ut`:

```hoon
++  peel
  |=  [way=vial met=?(%gold %iron %lead %zinc)]
  ^-  [sam=? con=?]
  ?:  ?=(%gold met)  [& &]
  ?-  way
    %both  [| |]
    %free  [& &]
    %read  [?=(%zinc met) |]
    %rite  [?=(%iron met) |]
  ==
```

Those permissions sets are the ones actually used in core behavior checks. In kernelspace, you are not strictly limited by the core type system—but you will have to manually construct handlers for other wetness/metallic behaviors and extend things to get the behavior you are aiming for.

* [Hoon School, “Generic and Variant Cores”](https://github.com/urbit/docs.urbit.org/tree/master/guides/core/hoon-school/R-metals/README.md)

#### Aside: Constructing Gates

Gates are special `$`-armed instances of doors. It's interesting to see how that particular sausage is made in `+mint:ut`:

```hoon
[%brts *]  :+  %brcb  p.gen
           =-  [~ [[%$ ~ -] ~ ~]]
           (~(put by *(map term hoon)) %$ q.gen)
[%brcb *]  :+  %tsls  [%kttr p.gen]
           :+  %brcn  ~
           %-  ~(run by r.gen)
           |=  =tome
           :-  p.tome
           %-  ~(run by q.tome)
           |=  =hoon
           ?~  q.gen  hoon
           [%tstr [p.i.q.gen ~] q.i.q.gen $(q.gen t.q.gen)]
```

### Arvo-Supplied Values <a href="#arvo-supplied-values" id="arvo-supplied-values"></a>

Arvo values such as `our`, `eny`, and `now` are simply supplied at axes in the subject (rather than being scries). (This is why they must be explicitly provided for in generators.) Compare the following Nock results on a fakeship \~zod.

```
> !=(=(0 ~zod))
[5 [1 0] 1 0]

> !=(=(0 our))
[5 [1 0] 0 12]
```

In the latter, `our` refers to a slot in the subject which needs to be looked up (at `12`) and replaced into the final evaluated noun.

* `our` is at `12`
* `now` is at `26`
* `eny` is at `27`

### Dynamic Dispatch <a href="#dynamic-dispatch" id="dynamic-dispatch"></a>

> Static dispatch (or early binding) happens when I know at compile time which function body will be executed when I call a method. In contrast, dynamic dispatch (or run-time dispatch or virtual method call or late binding) happens when I defer that decision to run time.

The conventional behavior of Urbit's Hoon language is to statically dispatch against limb labels known at compile time. It's somewhat difficult to get around this in userspace; for instance, to retrieve a list of faces in a core and selectively run against those that exist. Why? As we showed a moment ago, a face or an arm name is a compile-time construct that stands in for an axis in the subject.

With a subject and the `slap`/`slop` algebra, we can effect dynamic (runtime) dispatch for an interactive interface via `slam`. For instance, Dojo does this for every input. (Cf. [ll. 530–539 in `/app/dojo.hoon`](https://github.com/urbit/urbit/blob/7a205c4b76e1871235dd6f4f4caa1cb6dc5b8c6c/pkg/arvo/app/dojo.hoon#L530).)

### `-test` Thread <a href="#test-thread" id="test-thread"></a>

* The `/ted/test` thread invokes arms beginning with `test` in the context of the subject provided in the core. How does it do this?

## Kelvin Versioning <a href="#kelvin-versioning" id="kelvin-versioning"></a>

The innermost core of Hoon is the root marker for the language version. Not every part of a system should be subject to kelvin versioning: userspace generally will not be, and even in the `%base` desk many portions will not be subject to kelvin versioning.

> Deep, onion-like layering is essential. A thin layer has no room to grow. A good example of this principle is the difference between Urbit and Lisp machines. Both Nock and Lisp are very simple axiomatic definitions of computing. But practical Lisp systems expand by extending the model, whereas Urbit layers over a frozen axiom system. — [\~sorreg-namtyv, \~ravmel-ropdyl, “Towards a Frozen Operating System”](https://urbit.org/blog/toward-a-frozen-operating-system)

The parts of the system subject to kelvin versioning are:

* Nock, `%4` (liquid helium, 4.15 K).
* Hoon, `%137` (about liquid krypton, 115.8 K). (`%140` in [December 2020](https://groups.google.com/a/urbit.org/g/dev/c/2H28uz4T6xU/m/coJSpg4aAgAJ)).
* Arvo, `%236` (about liquid mercury, 234.4 K). (`%240` in December 2020).
* Lull, `%322`, (about gaseous water, 373.1 K). (`%330` in December 2020).
* Zuse, `%410`, which in a sense represents the most important kelvin for userspace developers since it's what they peg releases against. (`%420` in December 2020.)

What parts are subject to kelvin versioning? Essentially, the things we see as *platform*: as you can see, the language, the event handler, and parts of the standard library.

What results in a kelvin change? Not every release, even a change in a system file, motivates a kelvin decrement. The rule of thumb is that something which changes the specification of the platform burns a kelvin. In practice, although there are many kelvins yet to burn, it is more straightforward to bundle breaking changes together. This is both frugal of platform changes and generous to userspace developers.

Formally, `/?` faswut is used to pin a version number; in practice, it is not enforced at the compiler level.

### Telescoping Kelvins <a href="#telescoping-kelvins" id="telescoping-kelvins"></a>

> The rules of telescoping are simple:
>
> 1. If tool B sits on platform A, either both A and B must be at absolute zero, or B must be warmer than A.
> 2. Whenever the temperature of A (the platform) declines, the temperature of B (the tool) must also decline.
> 3. B must state the version of A it was developed against. A, when loading B, must state its own current version, and the warmest version of itself with which it's backward-compatible. — [\~sorreg-namtyv, \~ravmel-ropdyl, “Towards a Frozen Operating System”](https://urbit.org/blog/toward-a-frozen-operating-system)

Thus if you introduced a tool into kernelspace which relies on Nock alone, you could version it at anything above 4. If it relies on Hoon, then it should be above 139. And preferably a fair bit above—fat onion rings are tastier than paper-thin ones.

## The Structure of Kernelspace <a href="#the-structure-of-kernelspace" id="the-structure-of-kernelspace"></a>

The kernel is constructed of nested cores from the innermost `/sys/hoon` definitions out to `/sys/zuse`. All of userspace runs outside of these cores.

Although Arvo (`ca03`) is the operational core of Urbit, we actually require a boot process (see the boot lesson) building on the definition of Hoon itself. Thus we begin with `hoon.hoon`, `zuse.hoon`, and `lull.hoon` today before proceeding into Arvo proper.

### Core 0 <a href="#core-0" id="core-0"></a>

The first core consists of the Hoon version tag, currently `%139`. Since there are no documentation references to this core, we call it `0`, the “version stub”.

```hoon
=>  %139  =>
::                                                      ::
::::    0: version stub                                 ::
  ::                                                    ::
~%  %k.139  ~  ~                                        ::
|%
++  hoon-version  +
--  =>
```

1. This resolves down to `++ hoon-version %139` in a circuitous way.
2. The `~%` sigcen tag starts a jet registration tree. Unlike other jet registrations we have seen and will see later, this one is the root jet registration, meaning it has no parent and exports no named formulas, although it contains all of `hoon.hoon`.

   Since we refer to a “parent core” and imply a “child core“, we need to clarify something a bit counterintuitive about Urbit's subject-oriented nature. We say that the child core contains the parent core, through its `context`; and we refer to the parent core as the “inner” core, the child being “outer”. In fact, the parent/inner core is a leg in the child/outer core.

   Compare the expansion of `~/` sigfas: `~%(p +7 ~ q)`.

Thus, all things considered, Core 0 `layer-0` is the innermost core of all of Urbit. It appears at the rightmost side when the prettyprinter shows a core:

```hoon
> add
<1.otf [[a=@ b=@] <33.sam 1.pnw %139>]>

> +7:add
<33.sam 1.pnw %139>

> +7:+7:add
%139
```

### Core 1 <a href="#core-1" id="core-1"></a>

The next core contains arithmetic. Since each core can only access limbs present in its `payload` (`+3`), and in particular its `context` (`+7`), each core builds outwards on its predecessors, in this case on a foundation of straightforward integer arithmetic.

```hoon
++  add
++  dec
++  div
++  dvr
++  gte
++  gth
++  lte
++  lth
++  max
++  min
++  mod
++  mul
++  sub
```

The next block are for binary tree calculations:

```hoon
++  cap
++  mas
++  peg
```

Then we have some standard definitions of types and values for mold building and handling types like `unit`s.

```hoon
+$  bite
+$  bloq
++  each
+$  gate
++  list
++  lone
++  lest
+$  mold
++  pair
++  pole
++  qual
++  quip
++  step
++  trap
++  tree
++  trel
++  unit
```

* Several of these, like `trel` and `qual`, are hardly used even in `hoon.hoon` but standardize named faces.
* We particularly draw your attention to `pole`, which is a faceless `list`. This has more recently shown up in contexts where it is helpful to replace supplied faces with your own, as in `+on-peek` `?+` wutlus statements.
* `each` allows you to discriminate between values on type using a flag. (This is useful when returning structures out of a parser for instance, like `(each manx marl)` where `manx` is a structure and `marl` is a `list`.)

```hoon
> ((each @ ^) [%& 6])
[%.y p=6]

> ((each @ ^) [%| [6 7]])
[%.n p=[6 7]]
```

### Core 2 <a href="#core-2" id="core-2"></a>

Many practical tools live in `layer-2`, including functional tools, maps, sets, list operators, and string and formatted text operators.

**Unit Logic**

First up, the `unit` tools. While `unit`s are often just stripped off in userspace, there is a full-featured algebra handling `unit`s. (I'm of the opinion that these are probably underutilized because it can be hard to reason correctly with `unit`s.)

```hoon
++  biff                                                ::  apply
++  bind                                                ::  argue
++  bond                                                ::  replace
++  both                                                ::  all the above
++  clap                                                ::  combine
++  clef                                                ::  compose
++  drop                                                ::  enlist
++  fall                                                ::  default
++  flit                                                ::  make filter
++  hunt                                                ::  first of units
++  lift                                                ::  lift mold (fmap)
++  mate                                                ::  choose
++  need                                                ::  demand
++  some                                                ::  lift (pure)
```

* In particular, check out the definitions of `+biff`, `+bond`, `+flit`, and `+lift`, which apply wet gates and deferred traps.

**List Logic**

```hoon
++  snoc
++  lure
++  fand                                                ::  all indices
++  find                                                ::  first index
++  flop                                                ::  reverse
++  gulf                                                ::  range inclusive
++  homo                                                ::  homogenize
++  join
++  bake
++  lent                                                ::  length
++  levy
++  lien                                                ::  some of
++  limo                                                ::  listify
++  murn                                                ::  maybe transform
++  oust                                                ::  remove
++  reap                                                ::  replicate
++  rear                                                ::  last item of list
++  reel                                                ::  right fold
++  roll                                                ::  left fold
++  scag                                                ::  prefix
++  skid                                                ::  separate
++  skim                                                ::  only
++  skip                                                ::  except
++  slag                                                ::  suffix
++  snag                                                ::  index
++  snip                                                ::  drop tail off list
++  sort                                                ::  quicksort
++  spin                                                ::  stateful turn
++  spun                                                ::  internal spin
++  swag                                                ::  slice
++  turn
++  weld                                                ::  concatenate
++  snap                                               ::  replace item
++  into                                               ::  insert item
++  welp                                                ::  faceless weld
++  zing                                                ::  promote
```

* You are likely familiar with all of these except `+lure`, which is a list builder that's unused in the system.
* Note that `+sort` (quicksort) turns off the stack trace because feedback from such a crash is liable to be a mess.

```hoon
++  sort  !.                                            ::  quicksort
  ~/  %sort
  |*  [a=(list) b=$-([* *] ?)]
  =>  .(a ^.(homo a))
  |-  ^+  a
  ?~  a  ~
  =+  s=(skid t.a |:(c=i.a (b c i.a)))
  %+  weld
    $(a p.s)
  ^+  t.a
  [i.a $(a q.s)]
```

* Optional exercise: Implement [another sorting algorithm](https://en.wikipedia.org/wiki/Sorting_algorithm) besides quicksort in Hoon applicable to `list`s.

**Bitwise Logic**

```hoon
++  bex                                                 ::  binary exponent
++  can                                                 ::  assemble
++  cat                                                 ::  concatenate
++  cut                                                 ::  slice
++  end                                                 ::  tail
++  fil                                                 ::  fill bloqstream
++  lsh                                                 ::  left-shift
++  met                                                 ::  measure
++  rap                                                 ::  assemble variable
++  rep                                                 ::  assemble fixed
++  rev
++  rip                                                 ::  disassemble
++  rsh                                                 ::  right-shift
++  run                                                 ::  +turn into atom
++  rut                                                 ::  +turn into list
++  sew                                                 ::  stitch into
++  swp                                                 ::  naive rev bloq order
++  xeb                                                 ::  binary logarithm
++  fe                                                  ::  modulo bloq
 ++  dif                                               ::  difference
 ++  inv  |=(b=@ (sub (dec out) (sit b)))              ::  inverse
 ++  net  |=  b=@  ^-  @                               ::  flip byte endianness
 ++  out  (bex (bex a))                                ::  mod value
 ++  rol  |=  [b=bloq c=@ d=@]  ^-  @                  ::  roll left
 ++  ror  |=  [b=bloq c=@ d=@]  ^-  @                  ::  roll right
 ++  sum  |=([b=@ c=@] (sit (add b c)))                ::  wrapping add
 ++  sit  |=(b=@ (end a b))                            ::  enforce modulo
++  con                                                 ::  binary or
++  dis                                                 ::  binary and
++  mix                                                 ::  binary xor
++  not  |=  [a=bloq b=@ c=@]                           ::  binary not (sized)
```

* These provide bitwise operators for atoms. `+fe` is barely used but seems like it could be used to organize some of the logic.
* Optional exercise: Produce `+rip` and `+sew`.

**Insecure Hashing**

```hoon
++  muk                                                 ::  standard murmur3
++  mug                                                 ::  mug with murmur3
++  aor
++  dor
++  gor
++  mor
```

* These provide simple hashing and ordering algorithms.

The [Murmur3 algorithm](https://en.wikipedia.org/wiki/MurmurHash) is non-cryptographic hash function. `+muk` implements the 32-bit version. In pseudocode from Wikipedia:

```
algorithm Murmur3_32 is
    // Note: In this version, all arithmetic is performed with unsigned 32-bit integers.
    //       In the case of overflow, the result is reduced modulo 232.
    input: key, len, seed

    c1 ← 0xcc9e2d51
    c2 ← 0x1b873593
    r1 ← 15
    r2 ← 13
    m ← 5
    n ← 0xe6546b64

    hash ← seed

    for each fourByteChunk of key do
        k ← fourByteChunk

        k ← k × c1
        k ← k ROL r1
        k ← k × c2

        hash ← hash XOR k
        hash ← hash ROL r2
        hash ← (hash × m) + n

    with any remainingBytesInKey do
        remainingBytes ← SwapToLittleEndian(remainingBytesInKey)
        // Note: Endian swapping is only necessary on big-endian machines.
        //   The purpose is to place the meaningful digits towards the low end of the value,
        //   so that these digits have the greatest potential to affect the low range digits
        //   in the subsequent multiplication.  Consider that locating the meaningful digits
        //   in the high range would produce a greater effect upon the high digits of the
        //   multiplication, and notably, that such high digits are likely to be discarded
        //   by the modulo arithmetic under overflow.  We don't want that.

        remainingBytes ← remainingBytes × c1
        remainingBytes ← remainingBytes ROL r1
        remainingBytes ← remainingBytes × c2

        hash ← hash XOR remainingBytes

    hash ← hash XOR len

    hash ← hash XOR (hash >> 16)
    hash ← hash × 0x85ebca6b
    hash ← hash XOR (hash >> 13)
    hash ← hash × 0xc2b2ae35
    hash ← hash XOR (hash >> 16)
```

**Unsigned Powers**

```hoon
++  pow                                                 ::  unsigned exponent
++  sqt                                                 ::  unsigned sqrt/rem
```

**Container Logic**

```hoon
++  jar                                                ::  map of lists
++  jug                                                ::  map of sets
++  map
++  qeu
++  set
++  in                                                 ::  set engine
 ++  all                                               ::  logical AND
 ++  any                                               ::  logical OR
 ++  apt                                               ::  check correctness
 ++  bif                                               ::  splits a by b
 ++  del                                               ::  b without any a
 ++  dif                                               ::  difference
 ++  dig                                               ::  axis of a in b
 ++  gas                                               ::  concatenate
 ++  has
 ++  int                                               ::  intersection
 ++  put                                               ::  puts b in a, sorted
 ++  rep                                               ::  reduce to product
 ++  run                                               ::  apply gate to values
 ++  tap                                               ::  convert to list
 ++  uni                                               ::  union
 ++  wyt                                               ::  size of set
++  by                                                 ::  map engine
 ++  all                                               ::  logical AND
 ++  any                                               ::  logical OR
 ++  bif                                               ::  splits a by b
 ++  del                                               ::  delete at key b
 ++  dif                                               ::  difference
 ++  dig                                               ::  axis of b key
 ++  apt                                               ::  check correctness
 ++  gas                                               ::  concatenate
 ++  get                                               ::  grab value by key
 ++  got                                               ::  need value by key
 ++  gut                                               ::  fall value by key
 ++  has                                               ::  key existence check
 ++  int                                               ::  intersection
 ++  jab
 ++  mar                                               ::  add with validation
 ++  put                                               ::  adds key-value pair
 ++  rep                                               ::  reduce to product
 ++  rib                                               ::  transform + product
 ++  run                                               ::  apply gate to values
 ++  rut                                               ::  apply gate to nodes
 ++  tap                                               ::  listify pairs
 ++  uni                                               ::  union, merge
 ++  uno                                               ::  general union
 ++  urn                                               ::  apply gate to nodes
 ++  wyt                                               ::  depth of map
 ++  key                                               ::  set of keys
 ++  val                                               ::  list of vals
++  ja                                                 ::  jar engine
 ++  get                                               ::  gets list by key
 ++  add                                               ::  adds key-list pair
++  ju                                                 ::  jug engine
 ++  del                                               ::  del key-set pair
 ++  gas                                               ::  concatenate
 ++  get                                               ::  gets set by key
 ++  has                                               ::  existence check
 ++  put                                               ::  add key-set pair
++  to                                                 ::  queue engine
 ++  apt                                               ::  check correctness
 ++  bal
 ++  dep                                               ::  max depth of queue
 ++  gas                                               ::  insert list to que
 ++  get                                               ::  head-rest pair
 ++  nip                                               ::  removes root
 ++  nap                                               ::  removes root
 ++  put                                               ::  insert new tail
 ++  tap                                               ::  adds list to end
 ++  top                                               ::  produces head
++  malt                                               ::  map from list
++  molt                                               ::  map from pair list
++  silt                                               ::  set from list
++  ly                                                 ::  list from raw noun
++  my                                                 ::  map from raw noun
++  sy                                                 ::  set from raw noun
++  nl                                                 ::  noun-to-container ops
```

* Jars and jugs seem oddly specific, and are only invoked in a couple of special contexts in the base distribution.
* We don't see `+ly` used often. What are the difference in these constructed lists?

```hoon
> -:!>((ly ~[1 2 3]))
#t/it(?(@ud ?(@ud ?(@ud #!))))

> -:!>((homo ~[1 2 3]))
#t/it(@ud)

> -:!>((limo ~[1 2 3]))
#t/^#3.?([i=@ud t=#3] ^#2.?([i=@ud t=#2] it(@ud)))

> -:!>(`(list @ud)`~[1 2 3])
#t/it(@ud)
```

* `+ly` uses the crash type for an empty list.

```hoon
> (scag 0 ((list @) ~))
~

> (scag 0 (ly ~))
mull-grow
-find.b
find-fork
dojo: hoon expression failed
```

**Serialization**

```hoon
++  cue                                                 ::  unpack
++  jam                                                 ::  pack
++  mat                                                 ::  length-encode
++  rub                                                 ::  length-decode
```

* `+jam` and `+cue` are critically important for noun communication operations.

Here is an annotated version of `+jam`. The basic idea is to produce a serial noun (in order of head/tail):

1. One bit marks cell or atom.
2. Next entry marks bit length of value.
3. Then the actual value.

(`+cue` distinguishes the bit length from the value by unary until the first `0`.)

```hoon
> `@ub`(jam ~)
0b10
::  start at LSB, so `0` for atom, `1` for length, `0` for value (head-trimmed zero)

> `@ub`(jam 1)
0b1100
::  start at LSB, so `0` for atom, `01`

> `@ub`(jam [0 0])
0b10.1001
::  start at LSB, so `01` for cell, then `0` for head atom, length `1`, value `0`, repeat

> `@ub`(jam [0 1])
0b1100.1001

> `@ub`(jam [1 0])
0b1011.0001
```

Functional programming combinators:

```hoon
++  aftr                                                ::  pair after
++  cork                                                ::  compose forward
++  corl                                                ::  compose backwards
++  cury                                                ::  curry left
++  curr                                                ::  curry right
++  fore  |*(a=$-(* *) |*(b=$-(* *) (pair a b)))        ::  pair before
++  head  |*(^ ,:+<-)                                   ::  get head
++  same  |*(* +<)                                      ::  identity
++  succ  |=(@ +(+<))                                   ::  successor
++  tail  |*(^ ,:+<+)                                   ::  get tail
++  test  |=(^ =(+<- +<+))                              ::  equality
++  lead  |*(* |*(* [+>+< +<]))                         ::  put head
++  late  |*(* |*(* [+< +>+<]))                         ::  put tail
```

**Various Type Definitions**

```hoon
++  fn  ::    float, infinity, or NaN
++  dn  ::    decimal float, infinity, or NaN
++  rn  ::    parsed decimal float
+$  axis  @                                             ::  tree address
+$  bean  ?                                             ::  0=&=yes, 1=|=no
+$  flag  ?
+$  char  @t                                            ::  UTF8 byte
+$  cord  @t                                            ::  UTF8, LSB first
+$  byts  [wid=@ud dat=@]                               ::  bytes, MSB first
+$  date  [[a=? y=@ud] m=@ud t=tarp]                    ::  parsed date
+$  knot  @ta                                           ::  ASCII text
+$  noun  *                                             ::  any noun
+$  path  (list knot)                                   ::  like unix path
+$  pith  (list iota)                                   ::  typed urbit path
+$  stud                                                ::  standard name
+$  tang  (list tank)                                   ::  bottom-first error
+$  iota                                                ::  typed path segment
+$  tank
+$  tape  (list @tD)                                    ::  utf8 string as list
+$  tour  (list @c)                                     ::  utf32 clusters
+$  tarp  [d=@ud h=@ud m=@ud s=@ud f=(list @ux)]        ::  parsed time
+$  term  @tas                                          ::  ascii symbol
+$  wain  (list cord)                                   ::  text lines
+$  wall  (list tape)                                   ::  text lines
```

* Floating-point structs
* Paths
* Strings

### Core 3 <a href="#core-3" id="core-3"></a>

```hoon
++  egcd                                                ::  schneier's egcd
++  fo                                                  ::  modulo prime
 ++  dif
 ++  exp
 ++  fra
 ++  inv
 ++  pro
 ++  sit
 ++  sum
++  si                                                  ::  signed integer
 ++  abs  |=(a=@s (add (end 0 a) (rsh 0 a)))           ::  absolute value
 ++  dif  |=  [a=@s b=@s]                              ::  subtraction
 ++  dul  |=  [a=@s b=@]                               ::  modulus
 ++  fra  |=  [a=@s b=@s]                              ::  divide
 ++  new  |=  [a=? b=@]                                ::  [sign value] to @s
 ++  old  |=(a=@s [(syn a) (abs a)])                   ::  [sign value]
 ++  pro  |=  [a=@s b=@s]                              ::  multiplication
 ++  rem  |=([a=@s b=@s] (dif a (pro b (fra a b))))    ::  remainder
 ++  sum  |=  [a=@s b=@s]                              ::  addition
 ++  sun  |=(a=@u (mul 2 a))                           ::  @u to @s
 ++  syn  |=(a=@s =(0 (end 0 a)))                      ::  sign test
 ++  cmp  |=  [a=@s b=@s]                              ::  compare
++  fl                                                  ::  arb. precision fp
   ++  rou
   ++  rau
   ++  add                                             ::  add; exact if e
   ++  sub                                             ::  subtract; exact if e
   ++  mul                                             ::  multiply
   ++  div                                             ::  divide
   ++  sqt                                             ::  square root
   ++  lth                                             ::  less-than
   ++  equ                                             ::  equals
   ++  ibl
   ++  uni
   ++  xpd
   ++  lug
   ++  drg                                             ::  dragon4; get
   ++  toj                                             ::  round to integer
   ++  ned                                             ::  require ?=([%f *] a)
   ++  shf                                             ::  a * 2^b; no rounding
   ++  fli                                             ::  flip sign
   ++  swr  ?+(r r %d %u, %u %d)                       ::  flipped rounding
   ++  prc  ?>((gth p 1) p)                            ::  force >= 2 precision
   ++  den  d                                          ::  denorm+flush+inf exp
   ++  emn  v                                          ::  minimum exponent
   ++  emx  (sum:si emn (sun:si w))                    ::  maximum exponent
   ++  spd  [e=emn a=1]                                ::  smallest denormal
   ++  spn  [e=emn a=(bex (dec prc))]                  ::  smallest normal
   ++  lfn  [e=emx a=(fil 0 prc 1)]                    ::  largest
   ++  lfe  (sum:si emx (sun:si prc))                  ::  2^lfe is > than all
   ++  zer  [e=--0 a=0]
 ++  rou                                               ::  round
 ++  syn                                               ::  get sign
 ++  abs                                               ::  absolute value
 ++  add                                               ::  add
 ++  ead                                               ::  exact add
 ++  sub                                               ::  subtract
 ++  mul                                               ::  multiply
 ++  emu                                               ::  exact multiply
 ++  div                                               ::  divide
 ++  fma                                               ::  fused multiply-add
 ++  sqt                                               ::  square root
 ++  inv                                               ::  inverse
 ++  sun                                               ::  uns integer to float
 ++  san                                               ::  sgn integer to float
 ++  lth                                               ::  less-than
 ++  lte                                               ::  less-equal
 ++  equ                                               ::  equal
 ++  gte                                               ::  greater-equal
 ++  gth                                               ::  greater-than
 ++  drg                                               ::  float to decimal
 ++  grd                                               ::  decimal to float
 ++  toi                                               ::  round to integer @s
 ++  toj                                               ::  round to integer fn
::  provided to ++rd, ++rs, ++rq, and ++rh
::  r=rounding mode: same as in ++fl
++  ff                                                  ::  ieee 754 format fp
 ++  sb  (bex (^add w p))                              ::  sign bit
 ++  me  (dif:si (dif:si --1 b) (sun:si p))            ::  minimum exponent
 ++  pa
 ++  sea                                               ::  @r to fn
 ++  bit  |=  [a=fn]  (bif (rou:pa a))                 ::  fn to @r w+ rounding
 ++  bif                                               ::  fn to @r no rounding
 ++  sig                                               ::  get sign
 ++  exp                                               ::  get exponent
 ++  add                                               ::  add
 ++  sub                                               ::  subtract
 ++  mul                                               ::  multiply
 ++  div                                               ::  divide
 ++  fma                                               ::  fused multiply-add
 ++  sqt                                               ::  square root
 ++  lth                                               ::  less-than
 ++  lte                                               ::  less-equals
 ++  equ                                               ::  equals
 ++  gte                                               ::  greater-equals
 ++  gth                                               ::  greater-than
 ++  sun                                               ::  uns integer to @r
 ++  san                                               ::  signed integer to @r
 ++  toi                                               ::  round to integer
 ++  drg                                               ::  @r to decimal float
 ++  grd                                               ::  decimal float to @r
++  rlyd  |=  a=@rd  ^-  dn  (drg:rd a)                 ::  prep @rd for print
++  rlys  |=  a=@rs  ^-  dn  (drg:rs a)                 ::  prep @rs for print
++  rlyh  |=  a=@rh  ^-  dn  (drg:rh a)                 ::  prep @rh for print
++  rlyq  |=  a=@rq  ^-  dn  (drg:rq a)                 ::  prep @rq for print
++  ryld  |=  a=dn  ^-  @rd  (grd:rd a)                 ::  finish parsing @rd
++  ryls  |=  a=dn  ^-  @rs  (grd:rs a)                 ::  finish parsing @rs
++  rylh  |=  a=dn  ^-  @rh  (grd:rh a)                 ::  finish parsing @rh
++  rylq  |=  a=dn  ^-  @rq  (grd:rq a)                 ::  finish parsing @rq
++  rd                                                  ::  double precision fp
 ++  ma
 ++  sea                                               ::  @rd to fn
 ++  bit                                               ::  fn to @rd
 ++  add  ~/  %add                                     ::  add
 ++  sub  ~/  %sub                                     ::  subtract
 ++  mul  ~/  %mul                                     ::  multiply
 ++  div  ~/  %div                                     ::  divide
 ++  fma  ~/  %fma                                     ::  fused multiply-add
 ++  sqt  ~/  %sqt                                     ::  square root
 ++  lth  ~/  %lth                                     ::  less-than
 ++  lte  ~/  %lte                                     ::  less-equals
 ++  equ  ~/  %equ                                     ::  equals
 ++  gte  ~/  %gte                                     ::  greater-equals
 ++  gth  ~/  %gth                                     ::  greater-than
 ++  sun  |=  [a=@u]  ^-  @rd  (sun:ma a)              ::  uns integer to @rd
 ++  san  |=  [a=@s]  ^-  @rd  (san:ma a)              ::  sgn integer to @rd
 ++  sig  |=  [a=@rd]  ^-  ?  (sig:ma a)               ::  get sign
 ++  exp  |=  [a=@rd]  ^-  @s  (exp:ma a)              ::  get exponent
 ++  toi  |=  [a=@rd]  ^-  (unit @s)  (toi:ma a)       ::  round to integer
 ++  drg  |=  [a=@rd]  ^-  dn  (drg:ma a)              ::  @rd to decimal float
 ++  grd  |=  [a=dn]  ^-  @rd  (grd:ma a)              ::  decimal float to @rd
++  rs                                                  ::  single precision fp
 ++  ma
 ++  sea                                               ::  @rs to fn
 ++  bit                                               ::  fn to @rs
 ++  add  ~/  %add                                     ::  add
 ++  sub  ~/  %sub                                     ::  subtract
 ++  mul  ~/  %mul                                     ::  multiply
 ++  div  ~/  %div                                     ::  divide
 ++  fma  ~/  %fma                                     ::  fused multiply-add
 ++  sqt  ~/  %sqt                                     ::  square root
 ++  lth  ~/  %lth                                     ::  less-than
 ++  lte  ~/  %lte                                     ::  less-equals
 ++  equ  ~/  %equ                                     ::  equals
 ++  gte  ~/  %gte                                     ::  greater-equals
 ++  gth  ~/  %gth                                     ::  greater-than
 ++  sun  |=  [a=@u]  ^-  @rs  (sun:ma a)              ::  uns integer to @rs
 ++  san  |=  [a=@s]  ^-  @rs  (san:ma a)              ::  sgn integer to @rs
 ++  sig  |=  [a=@rs]  ^-  ?  (sig:ma a)               ::  get sign
 ++  exp  |=  [a=@rs]  ^-  @s  (exp:ma a)              ::  get exponent
 ++  toi  |=  [a=@rs]  ^-  (unit @s)  (toi:ma a)       ::  round to integer
 ++  drg  |=  [a=@rs]  ^-  dn  (drg:ma a)              ::  @rs to decimal float
 ++  grd  |=  [a=dn]  ^-  @rs  (grd:ma a)              ::  decimal float to @rs
++  rq                                                  ::  quad precision fp
 ++  ma
 ++  sea                                               ::  @rq to fn
 ++  bit                                               ::  fn to @rq
 ++  add  ~/  %add                                     ::  add
 ++  sub  ~/  %sub                                     ::  subtract
 ++  mul  ~/  %mul                                     ::  multiply
 ++  div  ~/  %div                                     ::  divide
 ++  fma  ~/  %fma                                     ::  fused multiply-add
 ++  sqt  ~/  %sqt                                     ::  square root
 ++  lth  ~/  %lth                                     ::  less-than
 ++  lte  ~/  %lte                                     ::  less-equals
 ++  equ  ~/  %equ                                     ::  equals
 ++  gte  ~/  %gte                                     ::  greater-equals
 ++  gth  ~/  %gth                                     ::  greater-than
 ++  sun  |=  [a=@u]  ^-  @rq  (sun:ma a)              ::  uns integer to @rq
 ++  san  |=  [a=@s]  ^-  @rq  (san:ma a)              ::  sgn integer to @rq
 ++  sig  |=  [a=@rq]  ^-  ?  (sig:ma a)               ::  get sign
 ++  exp  |=  [a=@rq]  ^-  @s  (exp:ma a)              ::  get exponent
 ++  toi  |=  [a=@rq]  ^-  (unit @s)  (toi:ma a)       ::  round to integer
 ++  drg  |=  [a=@rq]  ^-  dn  (drg:ma a)              ::  @rq to decimal float
 ++  grd  |=  [a=dn]  ^-  @rq  (grd:ma a)              ::  decimal float to @rq
++  rh                                                  ::  half precision fp
 ++  ma
 ++  sea                                               ::  @rh to fn
 ++  bit                                               ::  fn to @rh
 ++  add  ~/  %add                                     ::  add
 ++  sub  ~/  %sub                                     ::  subtract
 ++  mul  ~/  %mul                                     ::  multiply
 ++  div  ~/  %div                                     ::  divide
 ++  fma  ~/  %fma                                     ::  fused multiply-add
 ++  sqt  ~/  %sqt                                     ::  square root
 ++  lth  ~/  %lth                                     ::  less-than
 ++  lte  ~/  %lte                                     ::  less-equals
 ++  equ  ~/  %equ                                     ::  equals
 ++  gte  ~/  %gte                                     ::  greater-equals
 ++  gth  ~/  %gth                                     ::  greater-than
 ++  tos                                               ::  @rh to @rs
 ++  fos                                               ::  @rs to @rh
 ++  sun  |=  [a=@u]  ^-  @rh  (sun:ma a)              ::  uns integer to @rh
 ++  san  |=  [a=@s]  ^-  @rh  (san:ma a)              ::  sgn integer to @rh
 ++  sig  |=  [a=@rh]  ^-  ?  (sig:ma a)               ::  get sign
 ++  exp  |=  [a=@rh]  ^-  @s  (exp:ma a)              ::  get exponent
 ++  toi  |=  [a=@rh]  ^-  (unit @s)  (toi:ma a)       ::  round to integer
 ++  drg  |=  [a=@rh]  ^-  dn  (drg:ma a)              ::  @rh to decimal float
 ++  grd  |=  [a=dn]  ^-  @rh  (grd:ma a)              ::  decimal float to @rh
++  year                                                ::  date to @d
++  yore                                                ::  @d to date
++  yell                                                ::  tarp from @d
++  yule                                                ::  time atom
++  yall                                                ::  day / to day of year
++  yawn                                                ::  days since Jesus
++  yelp                                                ::  leap year
++  yo                                                  ::  time constants
 |%  ++  cet  36.524                 ::  (add 24 (mul 100 365))
     ++  day  86.400                 ::  (mul 24 hor)
     ++  era  146.097                ::  (add 1 (mul 4 cet))
     ++  hor  3.600                  ::  (mul 60 mit)
     ++  jes  106.751.991.084.417    ::  (mul 730.692.561 era)
     ++  mit  60
     ++  moh  `(list @ud)`[31 28 31 30 31 30 31 31 30 31 30 31 ~]
     ++  moy  `(list @ud)`[31 29 31 30 31 30 31 31 30 31 30 31 ~]
     ++  qad  126.144.001            ::  (add 1 (mul 4 yer))
     ++  yer  31.536.000             ::  (mul 365 day)
++  shad  |=(ruz=@ (shax (shax ruz)))                   ::  double sha-256
++  shaf                                                ::  half sha-256
++  sham                                                ::  128bit noun hash
++  shas                                                ::  salted hash
++  shax                                                ::  sha-256
++  shay                                                ::  sha-256 with length
++  shaw                                                ::  hash to nbits
++  shaz                                                ::  sha-512
++  shal                                                ::  sha-512 with length
++  shan                                                ::  sha-1 (deprecated)
++  og                                                  ::  shax-powered rng
 ++  rad                                               ::  random in range
 ++  rads                                              ::  random continuation
 ++  raw                                               ::  random bits
 ++  raws                                              ::  random bits
++  sha                                                 ::  correct byte-order
     ++  flin  |=(a=@ (swp 3 a))                       ::  flip input
     ++  flim  |=(byts [wid (rev 3 wid dat)])          ::  flip input w= length
     ++  flip  |=(w=@u (cury (cury rev 3) w))          ::  flip output of size
     ++  meet  |=(a=@ [(met 3 a) a])                   ::  measure input size
 ++  sha-1     (cork meet sha-1l)
 ++  sha-256   :(cork flin shax (flip 32))
 ++  sha-512   :(cork flin shaz (flip 64))
 ++  sha-256l  :(cork flim shay (flip 32))
 ++  sha-512l  :(cork flim shal (flip 64))
 ++  sha-1l
++  un                                                  ::  =(x (wred (wren x)))
 ++  wren                                              ::  conceal structure
 ++  wred                                              ::  restore structure
 ++  xafo  |=([a=@ b=@] +((mod (add (dec b) a) 255)))
 ++  xaro  |=([a=@ b=@] +((mod (add (dec b) (sub 255 (mod a 255))) 255)))
 ++  zaft                                              ::  forward 255-sbox
 ++  zart                                              ::  reverse 255-sbox
 ++  zyft                                              ::  forward 256-sbox
 ++  zyrt                                              ::  reverse 256-sbox
++  ob
 ++  fein
 ++  fynd
 ++  feis
 ++  tail
 ++  fee
 ++  feen
 ++  fe
 ++  fen
 ++  eff
 ++  raku
+$  coin  $~  [%$ %ud 0]                                ::  print format
+$  dime  [p=@ta q=@]                                   ::
+$  edge  [p=hair q=(unit [p=* q=nail])]                ::  parsing output
+$  hair  [p=@ud q=@ud]                                 ::  parsing trace
++  like  |*  a=$-(* *)                                 ::  generic edge
+$  nail  [p=hair q=tape]                               ::  parsing input
+$  pint  [p=[p=@ q=@] q=[p=@ q=@]]                     ::  line+column range
+$  rule  _|:($:nail $:edge)                            ::  parsing rule
+$  spot  [p=path q=pint]                               ::  range in file
+$  tone  $%  [%0 product=*]                            ::  success
+$  toon  $%  [%0 p=*]                                  ::  success
++  wonk  |*  veq=_$:edge                                ::  product from edge
```

### Core 4 <a href="#core-4" id="core-4"></a>

```hoon
++  po                                                  ::  phonetic base
  ++  ins  ~/  %ins                                     ::  parse prefix
  ++  ind  ~/  %ind                                     ::  parse suffix
  ++  tos  ~/  %tos                                     ::  fetch prefix
  ++  tod  ~/  %tod                                     ::  fetch suffix
++  fa                                                  ::  base58check
++  at                                                  ::  basic printing
++  cass                                                ::  lowercase
++  cuss                                                ::  uppercase
++  crip  |=(a=tape `@t`(rap 3 a))                      ::  tape to cord
++  mesc                                                ::  ctrl code escape
++  runt                                                ::  prepend repeatedly
++  sand                                                ::  atom sanity
++  sane                                                ::  atom sanity
++  ruth                                                ::  biblical sanity
++  trim                                                ::  tape split
++  trip                                                ::  cord to tape
++  teff                                                ::  length utf8
++  taft                                                ::  utf8 to utf32
++  tuba                                                ::  utf8 to utf32 tape
++  tufa                                                ::  utf32 to utf8 tape
++  tuft                                                ::  utf32 to utf8 text
++  wack                                                ::  knot escape
++  wick                                                ::  knot unescape
++  woad                                                ::  cord unescape
++  wood                                                ::  cord escape
++  wash                                                ::  render tank at width
++  re
  ++  ram
  ++  win
    ++  rig
    ++  wig
++  show                                                ::  XX deprecated!
  ++  shep
  ++  shop
  ++  shol
++  last  |=  [zyc=hair naz=hair]                       ::  farther trace
++  lust  |=  [weq=char naz=hair]                       ::  detect newline
++  bend                                                ::  conditional comp
++  comp
++  fail  |=(tub=nail [p=p.tub q=~])                    ::  never parse
++  glue                                                ::  add rule
++  less                                                ::  no first and second
++  pfix                                                ::  discard first rule
++  plug                                                ::  first then second
++  pose                                                ::  first or second
++  simu                                                ::  first and second
++  sfix                                                ::  discard second rule
++  bass                                                ::  leftmost base
++  boss                                                ::  rightmost base
++  cold                                                ::  replace w+ constant
++  cook                                                ::  apply gate
++  easy                                                ::  always parse
++  fuss
++  full                                                ::  has to fully parse
++  funk                                                ::  add to tape first
++  here                                                ::  place-based apply
++  inde  |*  sef=rule                                  :: indentation block
++  ifix
++  jest                                                ::  match a cord
++  just                                                ::  XX redundant, jest
++  knee                                                ::  callbacks
++  mask                                                ::  match char in set
++  more                                                ::  separated, *
++  most                                                ::  separated, +
++  next                                                ::  consume a char
++  perk                                                ::  parse cube fork
++  pick                                                ::  rule for ++each
++  plus  |*(fel=rule ;~(plug fel (star fel)))          ::
++  punt  |*([a=rule] ;~(pose (stag ~ a) (easy ~)))     ::
++  sear                                                ::  conditional cook
++  shim                                                ::  match char in range
++  stag                                                ::  add a label
++  stet                                                ::
++  stew                                                ::  switch by first char
++  slug                                                ::
++  star                                                ::  0 or more times
++  stir
++  stun                                                ::  parse several times
++  rash  |*([naf=@ sab=rule] (scan (trip naf) sab))
++  rose  |*  [los=tape sab=rule]
++  rush  |*([naf=@ sab=rule] (rust (trip naf) sab))
++  rust  |*  [los=tape sab=rule]
++  scan  |*  [los=tape sab=rule]
++  ace  (just ' ')                                     ::  spACE
++  bar  (just '|')                                     ::  vertical BAR
++  bas  (just '\\')                                    ::  Back Slash (escaped)
++  buc  (just '$')                                     ::  dollars BUCks
++  cab  (just '_')                                     ::  CABoose
++  cen  (just '%')                                     ::  perCENt
++  col  (just ':')                                     ::  COLon
++  com  (just ',')                                     ::  COMma
++  doq  (just '"')                                     ::  Double Quote
++  dot  (just '.')                                     ::  dot dot dot ...
++  fas  (just '/')                                     ::  Forward Slash
++  gal  (just '<')                                     ::  Greater Left
++  gar  (just '>')                                     ::  Greater Right
++  hax  (just '#')                                     ::  Hash
++  hep  (just '-')                                     ::  HyPhen
++  kel  (just '{')                                     ::  Curly Left
++  ker  (just '}')                                     ::  Curly Right
++  ket  (just '^')                                     ::  CareT
++  lus  (just '+')                                     ::  pLUS
++  mic  (just ';')                                     ::  seMIColon
++  pal  (just '(')                                     ::  Paren Left
++  pam  (just '&')                                     ::  AMPersand pampersand
++  par  (just ')')                                     ::  Paren Right
++  pat  (just '@')                                     ::  AT pat
++  sel  (just '[')                                     ::  Square Left
++  ser  (just ']')                                     ::  Square Right
++  sig  (just '~')                                     ::  SIGnature squiggle
++  soq  (just '\'')                                    ::  Single Quote
++  tar  (just '*')                                     ::  sTAR
++  tic  (just '`')                                     ::  backTiCk
++  tis  (just '=')                                     ::  'tis tis, it is
++  wut  (just '?')                                     ::  wut, what?
++  zap  (just '!')                                     ::  zap! bang! crash!!
++  alf  ;~(pose low hig)                               ::  alphabetic
++  aln  ;~(pose low hig nud)                           ::  alphanumeric
++  alp  ;~(pose low hig nud hep)                       ::  alphanumeric and -
++  bet  ;~(pose (cold 2 hep) (cold 3 lus))             ::  axis syntax - +
++  bin  (bass 2 (most gon but))                        ::  binary to atom
++  but  (cook |=(a=@ (sub a '0')) (shim '0' '1'))      ::  binary digit
++  cit  (cook |=(a=@ (sub a '0')) (shim '0' '7'))      ::  octal digit
++  dem  (bass 10 (most gon dit))                       ::  decimal to atom
++  dit  (cook |=(a=@ (sub a '0')) (shim '0' '9'))      ::  decimal digit
++  dog  ;~(plug dot gay)                               ::  .  number separator
++  dof  ;~(plug hep gay)                               ::  - @q separator
++  doh  ;~(plug ;~(plug hep hep) gay)                  ::  --  phon separator
++  dun  (cold ~ ;~(plug hep hep))                      ::  -- (stop) to ~
++  duz  (cold ~ ;~(plug tis tis))                      ::  == (stet) to ~
++  gah  (mask [`@`10 ' ' ~])                           ::  newline or ace
++  gap  (cold ~ ;~(plug gaq (star ;~(pose vul gah))))  ::  plural space
++  gaq  ;~  pose                                       ::  end of line
++  gaw  (cold ~ (star ;~(pose vul gah)))               ::  classic white
++  gay  ;~(pose gap (easy ~))                          ::
++  gon  ;~(pose ;~(plug bas gay fas) (easy ~))         ::  long numbers \ /
++  gul  ;~(pose (cold 2 gal) (cold 3 gar))             ::  axis syntax < >
++  hex  (bass 16 (most gon hit))                       ::  hex to atom
++  hig  (shim 'A' 'Z')                                 ::  uppercase
++  hit  ;~  pose                                       ::  hex digits
++  iny                                                 :: indentation block
++  low  (shim 'a' 'z')                                 ::  lowercase
++  mes  %+  cook                                       ::  hexbyte
++  nix  (boss 256 (star ;~(pose aln cab)))             ::
++  nud  (shim '0' '9')                                 ::  numeric
++  prn  ;~(less (just `@`127) (shim 32 256))           ::  non-control
++  qat  ;~  pose                                       ::  chars in blockcord
++  qit  ;~  pose                                       ::  chars in a cord
++  qut  ;~  simu  soq                                  ::  cord
++  soz  ;~(plug soq soq soq)                           ::  delimiting '''
++  sym                                                 ::  symbol
++  mixed-case-symbol
++  ven  ;~  (comp |=([a=@ b=@] (peg a b)))             ::  +>- axis syntax
++  vit                                                 ::  base64 digit
++  vul  %+  cold   ~                                   ::  comments
++  ab
  ++  bix  (bass 16 (stun [2 2] six))
  ++  fem  (sear |=(a=@ (cha:fa a)) aln)
  ++  haf  (bass 256 ;~(plug tep tiq (easy ~)))
  ++  hef  %+  sear  |=(a=@ ?:(=(a 0) ~ (some a)))
  ++  hif  (bass 256 ;~(plug tip tiq (easy ~)))
  ++  hof  (bass 0x1.0000 ;~(plug hef (stun [1 3] ;~(pfix hep hif))))
  ++  huf  (bass 0x1.0000 ;~(plug hef (stun [0 3] ;~(pfix hep hif))))
  ++  hyf  (bass 0x1.0000 ;~(plug hif (stun [3 3] ;~(pfix hep hif))))
  ++  pev  (bass 32 ;~(plug sev (stun [0 4] siv)))
  ++  pew  (bass 64 ;~(plug sew (stun [0 4] siw)))
  ++  piv  (bass 32 (stun [5 5] siv))
  ++  piw  (bass 64 (stun [5 5] siw))
  ++  qeb  (bass 2 ;~(plug seb (stun [0 3] sib)))
  ++  qex  (bass 16 ;~(plug sex (stun [0 3] hit)))
  ++  qib  (bass 2 (stun [4 4] sib))
  ++  qix  (bass 16 (stun [4 4] six))
  ++  seb  (cold 1 (just '1'))
  ++  sed  (cook |=(a=@ (sub a '0')) (shim '1' '9'))
  ++  sev  ;~(pose sed sov)
  ++  sew  ;~(pose sed sow)
  ++  sex  ;~(pose sed sox)
  ++  sib  (cook |=(a=@ (sub a '0')) (shim '0' '1'))
  ++  sid  (cook |=(a=@ (sub a '0')) (shim '0' '9'))
  ++  siv  ;~(pose sid sov)
  ++  siw  ;~(pose sid sow)
  ++  six  ;~(pose sid sox)
  ++  sov  (cook |=(a=@ (sub a 87)) (shim 'a' 'v'))
  ++  sow  ;~  pose
  ++  sox  (cook |=(a=@ (sub a 87)) (shim 'a' 'f'))
  ++  ted  (bass 10 ;~(plug sed (stun [0 2] sid)))
  ++  tep  (sear |=(a=@ ?:(=(a 'doz') ~ (ins:po a))) til)
  ++  tip  (sear |=(a=@ (ins:po a)) til)
  ++  tiq  (sear |=(a=@ (ind:po a)) til)
  ++  tid  (bass 10 (stun [3 3] sid))
  ++  til  (boss 256 (stun [3 3] low))
  ++  urs  %+  cook
  ++  urt  %+  cook
  ++  urx  %+  cook
  ++  voy  ;~(pfix bas ;~(pose bas soq bix))
++  ag
  ++  ape  |*(fel=rule ;~(pose (cold `@`0 (just '0')) fel))
  ++  bay  (ape (bass 16 ;~(plug qeb:ab (star ;~(pfix dog qib:ab)))))
  ++  bip  =+  tod=(ape qex:ab)
  ++  dem  (ape (bass 1.000 ;~(plug ted:ab (star ;~(pfix dog tid:ab)))))
  ++  dim  (ape dip)
  ++  dip  (bass 10 ;~(plug sed:ab (star sid:ab)))
  ++  dum  (bass 10 (plus sid:ab))
  ++  fed  %+  cook  fynd:ob
  ++  feq  %+  cook  |=(a=(list @) (rep 4 (flop a)))
  ++  fim  (sear den:fa (bass 58 (plus fem:ab)))
  ++  hex  (ape (bass 0x1.0000 ;~(plug qex:ab (star ;~(pfix dog qix:ab)))))
  ++  lip  =+  tod=(ape ted:ab)
  ++  mot  ;~  pose
  ++  viz  (ape (bass 0x200.0000 ;~(plug pev:ab (star ;~(pfix dog piv:ab)))))
  ++  vum  (bass 32 (plus siv:ab))
  ++  wiz  (ape (bass 0x4000.0000 ;~(plug pew:ab (star ;~(pfix dog piw:ab)))))
++  mu
  ++  zag  [p=(end 4 (add top bot)) q=bot]
  ++  zig  [p=(end 4 (add top (sub 0x1.0000 bot))) q=bot]
  ++  zug  (mix (lsh 4 top) bot)
++  ne
  ++  c  (cut 3 [tig 1] key:fa)
  ++  d  (add tig '0')
  ++  x  ?:((gte tig 10) (add tig 87) d)
  ++  v  ?:((gte tig 10) (add tig 87) d)
  ++  w  ?:(=(tig 63) '~' ?:(=(tig 62) '-' ?:((gte tig 36) (add tig 29) x)))
++  co
      ++  rear  |=(rom=tape rend(rep rom))
      ++  rent  ~+  `@ta`(rap 3 rend)
      ++  rend
      ++  a-co  |=(dat=@ ((d-co 1) dat))
      ++  c-co  (em-co [58 1] |=([? b=@ c=tape] [~(c ne b) c]))
      ++  d-co  |=(min=@ (em-co [10 min] |=([? b=@ c=tape] [~(d ne b) c])))
      ++  r-co
      ++  s-co
      ++  v-co  |=(min=@ (em-co [32 min] |=([? b=@ c=tape] [~(v ne b) c])))
      ++  w-co  |=(min=@ (em-co [64 min] |=([? b=@ c=tape] [~(w ne b) c])))
      ++  x-co  |=(min=@ (em-co [16 min] |=([? b=@ c=tape] [~(x ne b) c])))
      ++  y-co  |=(dat=@ ((d-co 2) dat))
      ++  z-co  |=(dat=@ `tape`['0' 'x' ((x-co 1) dat)])
  ++  em-co
  ++  ed-co
  ++  ox-co
  ++  ro-co
++  so
  ++  bisk
  ++  crub
  ++  nuck
  ++  nusk
  ++  perd
  ++  royl
  ++  royl-rh  (cook rylh ;~(pfix ;~(plug sig sig) (cook royl-cell royl-rn)))
  ++  royl-rq  (cook rylq ;~(pfix ;~(plug sig sig sig) (cook royl-cell royl-rn)))
  ++  royl-rd  (cook ryld ;~(pfix sig (cook royl-cell royl-rn)))
  ++  royl-rs  (cook ryls (cook royl-cell royl-rn))
  ++  royl-rn
  ++  royl-cell
  ++  tash
  ++  twid
  ++  when
  ++  zust
++  scot
++  scow
++  slat  |=(mod=@tas |=(txt=@ta (slaw mod txt)))
++  slav  |=([mod=@tas txt=@ta] (need (slaw mod txt)))
++  slaw
++  slay
++  smyt                                                ::  pretty print path
++  spat  |=(pax=path (crip (spud pax)))                ::  render path to cord
++  spud  |=(pax=path ~(ram re (smyt pax)))             ::  render path to tape
++  stab  |=(zep=@t `path`(rash zep stap))              ::  parse cord to path
++  stap                                                ::  path parser
++  stip                                                ::  typed path parser
  ++  swot  |=(n=nail (;~(pfix fas (more fas spot)) n))
  ++  spot
++  pout
++  pave
++  mack
++  mink  !.
  ++  frag
  ++  edit
++  mock
++  mook
  ++  skip
  ++  rend
++  mole
++  mong
++  mule
++  mure
++  mute
++  slum
++  soft
```

If you encounter a biblical name (`$abel`, `$onan`, etc.) then you're in the prettyprinter.

```hoon
+$  abel  typo                                          ::  original sin: type
+$  alas  (list (pair term hoon))                       ::  alias list
+$  atom  @                                             ::  just an atom
+$  aura  @ta                                           ::  atom format
+$  base                                                ::  base mold
+$  woof  $@(@ [~ p=hoon])                              ::  simple embed
+$  chum  $?  lef=term                                  ::  jet name
+$  coil  $:  p=garb                                    ::  name, wet=dry, vary
+$  garb  (trel (unit term) poly vair)                  ::  core
+$  poly  ?(%wet %dry)                                  ::  polarity
+$  foot  $%  [%dry p=hoon]                             ::  dry arm, geometric
+$  link                                                ::  lexical segment
+$  cuff  (list link)                                   ::  parsed lex segments
+$  crib  [summary=cord details=(list sect)]            ::
+$  help  [=cuff =crib]                                 ::  documentation
+$  limb  $@  term                                      ::  wing element
+$  null  ~                                             ::  null, nil, etc
+$  onyx  (list (pair type foot))                       ::  arm activation
+$  opal                                                ::  limb match
+$  pica  (pair ? cord)                                 ::  & prose, | code
+$  palo  (pair vein opal)                              ::  wing trace, match
+$  pock  (pair axis nock)                              ::  changes
+$  port  (each palo (pair type nock))                  ::  successful match
+$  spec                                                ::  structure definition
+$  tent                                                ::  model builder
+$  tiki                                                ::  test case
+$  skin                                                ::  texture
+$  tome  (pair what (map term hoon))                   ::  core chapter
+$  tope                                                ::  topographic type
++  hoot                                                ::  hoon tools
  +$  beer  $@(char [~ p=hoon])                         ::  simple embed
  +$  mane  $@(@tas [@tas @tas])                        ::  XML name+space
  +$  manx  $~([[%$ ~] ~] [g=marx c=marl])              ::  dynamic XML node
  +$  marl  (list tuna)                                 ::  dynamic XML nodes
  +$  mart  (list [n=mane v=(list beer)])               ::  dynamic XML attrs
  +$  marx  $~([%$ ~] [n=mane a=mart])                  ::  dynamic XML tag
  +$  mare  (each manx marl)                            ::  node or nodes
  +$  maru  (each tuna marl)                            ::  interp or nodes
  +$  tuna                                              ::  maybe interpolation
+$  hoon                                                ::  hoon AST
+$  tyre  (list [p=term q=hoon])                        ::
+$  tyke  (list (unit hoon))                            ::
+$  nock  $^  [p=nock q=nock]                           ::  autocons
+$  note                                                ::  type annotation
+$  type  $~  %noun                                     ::
+$  tony                                                ::  ++tone done right
+$  tine                                                ::  partial noun
+$  tool  $@(term tune)                                 ::  type decoration
+$  tune                                                ::  complex
+$  typo  type                                          ::  old type
+$  vase  [p=type q=*]                                  ::  type-value pair
+$  vise  [p=typo q=*]                                  ::  old vase
+$  vial  ?(%read %rite %both %free)                    ::  co/contra/in/bi
+$  vair  ?(%gold %iron %lead %zinc)                    ::  in/contra/bi/co
+$  vein  (list (unit axis))                            ::  search trace
+$  sect  (list pica)                                   ::  paragraph
+$  whit                                                ::  prefix docs parse
+$  whiz  cord                                          ::  postfix doc parse
+$  what  (unit (pair cord (list sect)))                ::  help slogan/section
+$  wing  (list limb)                                   ::  search path
+$  block
+$  result
+$  thunk
+$  seminoun
+$  stencil
+$  output
+$  doss
+$  moan                                                ::  sample metric
+$  hump
```

### Core 5 <a href="#core-5" id="core-5"></a>

**Parsing and Compiler**

```hoon
++  musk  !.                                            ::  nock with block set
 ++  abet
 ++  araw
 ++  apex
 ++  combine
 ++  complete
 ++  fragment
 ++  mutate
 ++  require
 ++  squash
++  bool  `type`(fork [%atom %f `0] [%atom %f `1] ~)    ::  make loobean
++  cell                                                ::  make %cell type
++  core                                                ::  make %core type
++  hint
++  face                                                ::  make %face type
++  fork                                                ::  make %fork type
++  cove                                                ::  extract [0 *] axis
++  comb                                                ::  combine two formulas
++  cond                                                ::  ?:  compile
++  cons                                                ::  make formula cell
++  fitz                                                ::  odor compatibility
++  flan                                                ::  loobean  &
++  flip                                                ::  loobean negation
++  flor                                                ::  loobean  |
++  hike
 ++  contains
 ++  parent
 ++  sibling
 ++  insert
++  jock
++  look
++  loot
++  ah                                                  ::  tiki engine
 ++  blue
 ++  teal
 ++  tele
 ++  gray
 ++  puce
 ++  wthp  |=  opt=(list (pair spec hoon))
 ++  wtkt  |=([sic=hoon non=hoon] (gray [%wtkt puce (blue sic) (blue non)]))
 ++  wtls  |=  [gen=hoon opt=(list (pair spec hoon))]
 ++  wtpt  |=([sic=hoon non=hoon] (gray [%wtpt puce (blue sic) (blue non)]))
 ++  wtsg  |=([sic=hoon non=hoon] (gray [%wtsg puce (blue sic) (blue non)]))
 ++  wthx  |=(syn=skin (gray [%wthx (tele syn) puce]))
 ++  wtts  |=(mod=spec (gray [%wtts (teal mod) puce]))
++  ax
 ++  autoname
 ++  function
 ++  interface
 ++  home
 ++  clear
 ++  basal
 ++  unfold
 ++  unreel
 ++  descend
 ++  decorate
 ++  pieces
 ++  spore
 ++  example
 ++  factory
 ++  analyze
   ++  basic
   ++  clear
   ++  fetch
   ++  fetch-wing
   ++  choice
   ++  switch
   ++  relative
++  ap                                                  ::  hoon engine
 ++  grip
 ++  name
 ++  feck
 ::  not used at present; see comment at %csng in ++open
::++  hail
 ++  half
 ++  flay
 ++  open
     ++  open-mane
     ++  open-mart
 ++  rake  ~>(%mean.'rake-hoon' (need reek))
 ++  reek
 ++  rusk
++  ut
 ++  clip
 ++  ar  !:
   ++  fish
   ++  gain
   ++  lose
 ++  blow
 ++  bran
 ++  burp
 ++  busk
 ++  buss
 ++  crop
   ++  dext
   ++  sint
 ++  cool
 ++  duck  ^-(tank ~(duck us sut))
 ++  dune  |.(duck)
 ++  dunk
 ++  elbo
 ++  ergo
 ++  endo
 ++  et
   ++  play
   ++  mint
   ++  mull
 ++  epla
 ++  emin
 ++  emul
 ++  felt  !!
 ++  feel                                              ::  detect existence
 ++  fond
       ++  pony                                        ::  raw match
     |%  ++  here  ?:  =(0 p.heg)
         ++  lose  [%| %& p.heg]
         ++  stop  ?~(q.heg here lose)
         ++  twin  |=  [hax=pony yor=pony]
         ++  $
             ++  main
             ++  next
 ++  find
 ++  fund
 ++  fine
 ++  fire
 ++  fish
 ++  fuse
 ++  gain
 ++  hemp
 ++  laze
   ++  chapter
 ++  lose
 ++  chip
 ++  bake
 ++  balk
 ++  mile
 ++  mine
   +$  gol-type
   ++  core-check
   ++  chapters-check
   ++  get-tomes
   ++  get-arms
   ++  arms-check
   ++  get-arm-type
   ++  nice
 ++  mint
   ++  nice
   ++  grow
 ++  moot
 ++  mull
   ++  beth
   ++  nice
   ++  grow
 ++  meet  |=(ref=type &((nest | ref) (nest(sut ref) | sut)))
 ++  miss                                              ::  nonintersection
   ++  dext
   ++  sint
 ++  mite  |=(ref=type |((nest | ref) (nest(sut ref) & sut)))
 ++  nest
   ++  deem
   ++  deep
   ++  dext
   ++  meet  &(dext dext(sut ref, ref sut))
   ++  sint
 ++  peek
 ++  peel
 ++  play
 ++  redo                                              ::  refurbish faces
   ++  dear                                            ::  resolve tool stack
   ++  dext                                            ::  subject traverse
   ++  done                                            ::  complete assembly
   ++  sint                                            ::  reduce by reference
 ++  repo
 ++  rest
 ++  sink
   ++  mup  |=(* (scot %p (mug +<)))
 ++  take
 ++  tack
 ++  tend
 ++  toss
 ++  wrap
++  us                                                  ::  prettyprinter
     +$  cape  [p=(map @ud wine) q=wine]               ::
     +$  wine                                          ::
 ++  dash
 ++  deal  |=(lum=* (dish dole lum))
 ++  dial
   ++  many
   ++  $
 ++  dish  !:
 ++  doge
 ++  dole
 ++  duck  (dial dole)
++  cain  sell                                          ::  $-(vase tank)
++  noah  text                                          ::  $-(vase tape)
++  onan  seer                                          ::  $-(vise vase)
++  levi                                                ::  $-([type type] ?)
++  text                                                ::  tape pretty-print
++  seem  |=(toy=typo `type`toy)                        ::  promote typo
++  seer  |=(vix=vise `vase`vix)                        ::  promote vise
++  sell
++  skol
++  slam                                                ::  slam a gate
++  slab                                                ::  test if contains
++  slap
++  slog                                                ::  deify printf
++  mean                                                ::  crash with trace
++  road
++  slew                                                ::  get axis in vase
++  slim                                                ::  identical to seer?
++  slit                                                ::  type of slam
++  slob                                                ::  superficial arm
++  sloe                                                ::  get arms in core
++  slop                                                ::  cons two vases
++  slot                                                ::  got axis in vase
++  slym                                                ::  slam w+o sample-type
++  sped                                                ::  reconstruct type
++  swat
++  vang
++  vast                                                ::  main parsing core
 ++  gash  %+  cook                                    ::  parse path
 ++  gasp  ;~  pose                                    ::  parse =path= etc.
 ++  glam  ~+((glue ace))
 ++  hasp  ;~  pose                                    ::  path element
 ++  limp  %+  cook
 ++  mota  %+  cook
 ++  docs
   ++  apex
   ++  apse
   ++  leap                                            ::  whitespace w/o docs
   ++  smol
   ++  larg
   ++  rant
   ++  skip                                            ::  non-doccord comment
   ++  null  (cold ~ (star ace))
   ++  text  (pick line code)
   ++  teyt  (pick line ;~(pfix step code))
   ++  line  ;~(less ace (cook crip (star prn)))
   ++  code  ;~(pfix step ;~(less ace (cook crip (star prn))))
   ++  step  ;~(plug ace ace)
   ++  into
   ++  en-link
 ++  clad                                              ::  hoon doccords
 ++  coat                                              ::  spec doccords
 ++  scye                                              ::  with prefix doccords
 ++  seam                                              ::  with doccords
 ++  plex                                              ::  reparse static path
 ++  phax
 ++  posh
 ++  poof                                              ::  path -> (list hoon)
 ++  poon                                              ::  try to replace '='s
 ++  poor
 ++  porc
 ++  rump
 ++  rood
 ++  reed
 ++  stem
   ++  slip  |*(r=rule (stag %hoon r))
   ++  slot  |*(r=rule (sear (soft iota) r))
   ++  spit
 ++  rupl
```

**Sail and XML Parsing**

```hoon
 ++  sail                                              ::  xml template
   ++  apex                                            ::  product hoon
   ++  top-level                                       ::  entry-point
   ++  inline-embed                                    ::  brace interpolation
   ++  script-or-style                                 ::  script or style
   ++  tuna-mode                                       ::  xml node(s) kind
   ++  wide-top                                        ::  wide outer top
   ++  wide-inner-top                                  ::  wide inner top
   ++  wide-attrs                                      ::  wide attributes
   ++  wide-tail                                       ::  wide elements
   ++  wide-elems                                      ::  wide elements
   ++  wide-paren-elems                                ::  wide flow
   ++  drop-top
   ++  join-tops
   ++  wide-quote                                      ::  wide quote
   ++  quote-innards                                   ::  wide+tall flow
   ++  bracketed-elem                                  ::  bracketed element
   ++  wrapped-elems                                   ::  wrapped tuna
   ++  a-mane                                          ::  mane as hoon
   ++  en-class
   ++  tag-head                                        ::  tag head
   ++  tall-top                                        ::  tall top
   ++  tall-attrs                                      ::  tall attributes
   ++  tall-elem                                       ::  tall preface
   ++  hopefully-quote                                 :: prefer "quote" form
   ++  script-style-tail                               ::  unescaped tall tail
   ++  tall-tail                                       ::  tall tail
   ++  tall-kids                                       ::  child elements
   ++  collapse-chars                                  ::  group consec chars
 ++  cram                                              ::  parse unmark
+$  mane  $@(@tas [@tas @tas])                          ::  XML name+space
+$  manx  $~([[%$ ~] ~] [g=marx c=marl])                ::  dynamic XML node
+$  marl  (list manx)                                   ::  XML node list
+$  mars  [t=[n=%$ a=[i=[n=%$ v=tape] t=~]] c=~]        ::  XML cdata
+$  mart  (list [n=mane v=tape])                        ::  XML attributes
+$  marx  $~([%$ ~] [n=mane a=mart])                    ::  dynamic XML tag
```

**Compiler**

```hoon
 ++  scad
 ++  scat
 ++  soil
 ++  sump  (ifix [kel ker] (stag %cltr (most ace wide)))
 ++  norm                                              ::  rune regular form
   ++  structure
   ++  expression
   ++  boog  !:
   ++  bola                                           ::  ++  arms
     ;~  pfix  (jest '++')
   ++  boba                                           ::  +$  arms
     ;~  pfix  (jest '+$')
  ++  lynx
   ++  whap  !:                                        ::  chapter
   ++  glow
       ::  we only support ++ and +$ batch comments right now
   ++  whip                                            ::  chapter declare
   ++  wasp                                            ::  $brcb aliases
   ++  wisp  !:                                        ::  core tail
   ++  toad                                            ::  untrap parser expr
   ++  rune                                            ::  build rune
   ++  runo                                            ::  rune plus
   ++  runq                                            ::  wide or tall if tol
   ++  butt  |*  zor=rule                              ::  closing == if tall
   ++  ulva  |*  zor=rule                              ::  closing -- and tall
   ++  glop  ~+((glue mash))                           ::  separated by space
   ++  gunk  ~+((glue muck))                           ::  separated list
   ++  goop  ~+((glue mush))                           ::  separator list & docs
   ++  hank  (most mush loaf)                          ::  gapped hoons
   ++  hunk  (most mush loan)                          ::  gapped specs
   ++  jump  ;~(pose leap:docs gap)                    ::  gap before docs
   ++  loaf  ?:(tol tall wide)                         ::  hoon
   ++  loll  ?:(tol tall(doc |) wide(doc |))           ::  hoon without docs
   ++  loan  ?:(tol till wyde)                         ::  spec
   ++  lore  (sear |=(=hoon ~(flay ap hoon)) loaf)     ::  skin
   ++  lomp  ;~(plug sym (punt ;~(pfix tis wyde)))     ::  typeable name
   ++  mash  ?:(tol gap ;~(plug com ace))              ::  list separator
   ++  muss  ?:(tol jump ;~(plug com ace))             ::  list w/ doccords
   ++  muck  ?:(tol gap ace)                           ::  general separator
   ++  mush  ?:(tol jump ace)                          ::  separator w/ docs
   ++  teak  %+  knee  *tiki  |.  ~+                   ::  wing or hoon
   ++  rack  (most muss ;~(goop loaf loaf))            ::  list [hoon hoon]
   ++  ruck  (most muss ;~(goop loan loaf))            ::  list [spec hoon]
   ++  rick  (most mash ;~(goop rope loaf))            ::  list [wing hoon]
   ++  expa  |.(loaf)                                  ::  one hoon
   ++  expb  |.(;~(goop loaf loaf))                    ::  two hoons
   ++  expc  |.(;~(goop loaf loaf loaf))               ::  three hoons
   ++  expd  |.(;~(goop loaf loaf loaf loaf))          ::  four hoons
   ++  expe  |.(wisp)                                  ::  core tail
   ++  expf  |.(;~(goop ;~(pfix cen sym) loaf))        ::  %term and hoon
   ++  expg  |.(;~(gunk lomp loll loaf))               ::  term/spec, two hoons
   ++  exph  |.((butt ;~(gunk rope rick)))             ::  wing, [wing hoon]s
   ++  expi  |.((butt ;~(goop loaf hank)))             ::  one or more hoons
   ++  expj  |.(;~(goop lore loaf))                    ::  skin and hoon
  :: ++  expk  |.(;~(gunk loaf ;~(plug loaf (easy ~))))::  list of two hoons
  :: ++  expl  |.(;~(gunk sym loaf loaf))              ::  term, two hoons
   ++  expm  |.((butt ;~(gunk rope loaf rick)))        ::  several [spec hoon]s
   ++  expn  |.  ;~  gunk  rope  loaf                  ::  wing, hoon,
   ++  expo  |.(;~(goop wise loaf loaf))               ::  =;
   ++  expp  |.(;~(goop (butt rick) loaf))             ::  [wing hoon]s, hoon
   ++  expq  |.(;~(goop rope loaf loaf))               ::  wing and two hoons
   ++  expr  |.(;~(goop loaf wisp))                    ::  hoon and core tail
   ++  exps  |.((butt hank))                           ::  closed gapped hoons
   ++  expt  |.(;~(gunk wise rope loaf loaf))          ::  =^
   ++  expu  |.(;~(gunk rope loaf (butt hank)))        ::  wing, hoon, hoons
  :: ++  expv  |.((butt rick))                         ::  just changes
   ++  expw  |.(;~(goop rope loaf loaf loaf))          ::  wing and three hoons
   ++  expx  |.(;~(goop ropa loaf loaf))               ::  wings and two hoons
   ++  expy  |.(loaf(bug &))                           ::  hoon with tracing
   ++  expz  |.(;~(goop loan loaf loaf loaf))          ::  spec and three hoons
   ++  exqa  |.(loan)                                  ::  one spec
   ++  exqb  |.(;~(goop loan loan))                    ::  two specs
   ++  exqc  |.(;~(goop loan loaf))                    ::  spec then hoon
   ++  exqd  |.(;~(goop loaf loan))                    ::  hoon then spec
   ++  exqe  |.(;~(goop lynx loan))                    ::  list of names then spec
   ++  exqs  |.((butt hunk))                           ::  closed gapped specs
   ++  exqg  |.(;~(goop sym loan))                     ::  term and spec
   ::++  exqk  |.(;~(goop loaf ;~(plug loan (easy ~))))::  hoon with one spec
   ++  exqn  |.(;~(gunk loan (stag %cltr (butt hank))))::  autoconsed hoons
   ++  exqr  |.(;~(gunk loan ;~(plug wasp wisp)))      ::  spec/aliases?/tail
   ::++  exqw  |.(;~(goop loaf loan))                  ::  hoon and spec
   ++  exqx  |.(;~(goop loaf loan loan))               ::  hoon, two specs
   ++  exqy  |.(;~(goop loaf loan loan loan))          ::  hoon, three specs
   ++  exqz  |.(;~(goop loaf (butt hunk)))             ::  hoon, n specs
   ++  txhp  |.  %+  cook  |=  [a=tiki b=(list (pair spec hoon))]
   ++  tkkt  |.  %+  cook  |=  [a=tiki b=hoon c=hoon]
   ++  txls  |.  %+  cook  |=  [a=tiki b=hoon c=(list (pair spec hoon))]
   ++  tkvt  |.  %+  cook  |=  [a=tiki b=hoon c=hoon]
   ++  tksg  |.  %+  cook  |=  [a=tiki b=hoon c=hoon]
   ++  txts  |.  %+  cook  |=  [a=spec b=tiki]
   ++  txhx  |.  %+  cook  |=  [a=skin b=tiki]
   ++  hinb  |.(;~(goop bont loaf))                    ::  hint and hoon
   ++  hinc  |.                                        ::  optional =en, hoon
   ++  hind  |.(;~(gunk bonk loaf ;~(goop bonz loaf))) ::  jet hoon "bon"s hoon
   ++  hine  |.(;~(goop bonk loaf))                    ::  jet-hint and hoon
   ++  hinf  |.                                        ::  0-3 >s, two hoons
   ++  hing  |.                                        ::  0-3 >s, three hoons
   ++  bonk                                            ::  jet signature
   ++  hinh  |.                                        ::  1/2 numbers, hoon
   ++  bont  ;~  (bend)                                ::  term, optional hoon
   ++  bony  (cook |=(a=(list) (lent a)) (plus tis))   ::  base 1 =en count
   ++  bonz                                            ::  term-labelled hoons
 ++  lang                                              ::  lung sample
 ++  lung
 ++  long
 ++  lobo  (most ;~(plug com ace) ;~(glam rope wide))
 ++  loon  (most ;~(plug com ace) ;~(glam wide wide))
 ++  lute                                              ::  tall [] noun
 ++  ropa  (most col rope)
 ++  rope                                              ::  wing form
 ++  wise
 ++  tall                                              ::  full tall form
 ++  till                                              ::  mold tall form
 ++  wede                                              ::  wide bulb
 ++  wide                                              ::  full wide form
 ++  wyde                                              ::  mold wide form
 ++  wart
 ++  wert
++  vest
++  vice
++  make                                                ::  compile cord to nock
++  rain                                                ::  parse with % path
++  ream                                                ::  parse cord to hoon
++  reck                                                ::  parse hoon file
++  ride                                                ::  end-to-end compiler
+$  mite  (list @ta)                                    ::  mime type
+$  pass  @                                             ::  public key
+$  ring  @                                             ::  private key
+$  ship  @p                                            ::  network identity
+$  shop  (each ship (list @ta))                        ::  urbit/dns identity
+$  spur  path                                          ::  ship desk case spur
+$  time  @da                                           ::  galactic time
++  pi-heck
++  pi-noon                                             ::  sample trace
++  pi-mope                                             ::  add sample
++  pi-moth                                             ::  count sample
++  pi-mumm                                             ::  print sample
++  pi-tell                                             ::  produce dump
```

### Core 6 <a href="#core-6" id="core-6"></a>

Hoon is the root of the whole system—you cannot parse and build Arvo or anything else without these definitions. As part of `ca01`, you examined how `$hoon` types are built and how the AST is implemented for a basic rune.

Outside of the language-necessary components, the `%lull` core provides kernel-wide structures (essentially, a header file) and the `%zuse` core provides a kernel-appropriate standard library. `%zuse` organizes its cores into what it terms “engines”.

#### `%lull`

**Models**

```hoon
::  +capped-queue: a +qeu with a maximum number of entries
++  capped-queue
::
::  +clock: polymorphic cache type for use with the clock replacement algorithm
++  clock
::
::  +mop, +on: ordered map engine
++  mop
++  on
 ++  all                                                :: apply logical AND on all values
 ++  any                                                :: apply logical OR on all values
 ++  apt                                                :: verify horz/vert orderings
 ++  bap                                                :: convert to list, right-to-left
 ++  del                                                :: delete key if it exists
 ++  dip                                                :: partial inorder traversal
 ++  gas                                                :: put a list of items
 ++  get                                                :: get value at key (unit)
 ++  got                                                :: get value at key (crash)
 ++  has                                                :: check for key existence
 ++  lot                                                :: take subset range
 ++  nip                                                :: remove root (internal)
 ++  pop                                                :: produce head and rest
 ++  pry                                                :: produce head or null
 ++  put                                                :: insert ordered item
 ++  ram                                                :: produce tail or null
 ++  run                                                :: apply gate to all values
 ++  tab                                                :: tabulate subset to max count
 ++  tap                                                :: convert to list, left-to-right
 ++  uni                                                :: unify two ordered maps
 ++  wyt                                                :: measure size
```

* [`+on`](https://github.com/urbit/docs.urbit.org/tree/master/hoon/reference/zuse/2m/README.md) provides the services for `+mop` ordered maps.

```hoon
+$  deco  ?(~ %bl %br %un)                              ::  text decoration
+$  json                                                ::  normal json value
+$  life  @ud                                           ::  ship key revision
+$  rift  @ud                                           ::  ship continuity
+$  mime  (pair mite octs)                              ::  mimetyped data
+$  octs  (pair @ud @)                                  ::  octet-stream
+$  sock  (pair ship ship)                              ::  outgoing [our his]
+$  stub  (list (pair stye (list @c)))                  ::  styled unicode
+$  stye  (pair (set deco) (pair tint tint))            ::  decos/bg/fg
+$  styl  %+  pair  (unit deco)                         ::  cascading style
+$  styx  (list $@(@t (pair styl styx)))                ::  styled text
+$  tint  $@  ?(%r %g %b %c %m %y %k %w %~)             ::  text color
+$  turf  (list @t)                                     ::  domain, tld first
```

* Various types, notably [structured text support](https://github.com/urbit/docs.urbit.org/tree/master/guides/additional/app-workbook/track7/README.md).

```
++  ethereum-types
 ++  address  @ux
 +$  event-id  [block=@ud log=@ud]
 ++  events  (set event-id)
::
++  azimuth-types
 ++  point
 +$  dnses  [pri=@t sec=@t ter=@t]
 ++  diff-azimuth
 ++  diff-point
+$  vane-task
::
++  http  ^?
 +$  header-list
 +$  method
 +$  request
 +$  response-header
 +$  http-event
 ++  get-header
 ++  set-header
 ++  delete-header
 ++  unpack-header
 +$  simple-payload
```

* Common structures: Ethereum, Azimuth, HTTP.

**Networking (Ames)**

After this point, `%lull` defines types and interfaces for interacting with vanes. We will skip lightly over these, but come back to them in the appropriate lessons.

```hoon
::::                    ++ames                            ::  (1a) network
++  ames
```

**Timekeeping (Behn)**

```hoon
::::                    ++behn                            ::  (1b) timekeeping
++  behn
```

**Versioning (Clay)**

```hoon
::::                    ++clay                            ::  (1c) versioning
++  clay
```

**Console (Dill)**

```hoon
::::                    ++dill                            ::  (1d) console
++  dill
```

**HTTP Server (Eyre)**

```hoon
::::                    ++eyre                            ::  (1e) http-server
++  eyre
```

**Extensions (Gall)**

```hoon
::::                    ++gall                            ::  (1g) extensions
++  gall  ^?
```

**HTTP Client (Iris)**

```hoon
::  %iris http-client interface
++  iris
```

**Security (Jael)**

```hoon
::::                    ++jael                          ::  (1h) security
++  jael
  ++  pki
```

**Threads (Khan)**

```hoon
::::                    ++khan                            ::  (1i) threads
++  khan
```

**IPC (Lick)**

```hoon
::::                    ++lick                            ::  (1j) IPC
++  lick
```

**Computation**

Various definitions for cards, strands, and moves.

```hoon
++  rand                                                ::  computation
+$  gift-arvo                                           ::  out result <-$
+$  task-arvo                                           ::  in request ->$
+$  note-arvo                                           ::  out request $->
+$  sign-arvo                                           ::  in result $<-
+$  unix-task                                           ::  input from unix
```

#### `%zuse`

**Cryptography**

One of the most important components of `%zuse` is the crypto library. This supplies modular arithmetic (`+fu`) and several specific algorithms. (In general, signed arithmetic in Urbit uses different names (like `+sum` instead of `+add`) to prevent accidental confusion.) Some significant portions of this include:

```hoon
++  number  ^?
  ++  fu                                                ::  modulo (mul p q)
    ++  dif                                             ::  subtract
    ++  exp                                             ::  exponent
    ++  out                                             ::  garner's formula
    ++  pro                                             ::  multiply
    ++  sum                                             ::  add
    ++  sit                                             ::  represent
  ++  curt                                              ::  curve25519
++  crypto                                              ::  (2b) cryptography
  ++  aes
  ++  keccak                                            ::  (2b7) keccak family
    ++  keccak-224
    ++  keccak-256
    ++  keccak-384
    ++  keccak-512
    ++  keccak
    ++  sha3-224
    ++  sha3-256
    ++  sha3-384
    ++  sha3-512
    ++  sha3
  ++  hmac                                              ::  (2b8) hmac family
    ++  hmac-sha1
    ++  hmac-sha256
    ++  hmac-sha512
    ++  hmac-sha1t
    ++  hmac-sha256t
    ++  hmac-sha512t
    ++  hmac-sha1l
    ++  hmac-sha256l
    ++  hmac-sha512l
    ++  hmac
  ++  secp                                            
    ++  secp256k1
        ++  sign                                        ::  schnorr signature
        ++  verify                                      ::  schnorr verify
  ++  blake
    ++  blake2b
  ++  argon2
  ++  ripemd
    ++  ripemd-160
  ++  pbkdf
    ++  hmac-sha1
    ++  hmac-sha256
    ++  hmac-sha512
    ++  hmac-sha1t
    ++  hmac-sha256t
    ++  hmac-sha512t
    ++  hmac-sha1l
    ++  hmac-sha256l
    ++  hmac-sha512l
    ++  hmac-sha1d
    ++  hmac-sha256d
    ++  hmac-sha512d
    ++  pbkdf
```

Notes on the above:

* [Modular arithmetic](https://en.wikipedia.org/wiki/Modular_arithmetic) is used in public key systems.
* [Curve25519](https://en.wikipedia.org/wiki/Curve25519) is elliptic-curve cryptography.
* [Advanced encryption standard](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) (AES) is a cipher for data encryption. There are a lot of modes available in `+aes`.
* [KECCAK](https://en.wikipedia.org/wiki/SHA-3) is a cryptographic family underlying SHA-3.
* A [Schnorr signature](https://en.wikipedia.org/wiki/Schnorr_signature) is a digital signature scheme.
* [BLAKE](https://en.wikipedia.org/wiki/BLAKE_%28hash_function%29) is a cryptographic hash function like KECCAK.
* [Argon2](https://en.wikipedia.org/wiki/Argon2) is a key derivation function.
* [RIPEMD](https://en.wikipedia.org/wiki/RIPEMD) cryptographic hash functions are used in Bitcoin.
* [Hashed-based message authentication codes](https://en.wikipedia.org/wiki/HMAC) (HMAC) are used in shared-secret key exchange based on the [SHA-2](https://en.wikipedia.org/wiki/SHA-2) cryptographic hash algorithms, of which several are made available. (Variants of these also live in the [password-based key derivation function](https://en.wikipedia.org/wiki/PBKDF2) (PBKDF) arm.)

**Units**

After cryptography, there are a number of library utility functions.

```hoon
++  unity                                               ::  (2c) unit promotion
  ++  drop-list                                         ::  collapse unit list
  ++  drop-map                                          ::  collapse unit map
  ++  drop-pole                                         ::  collapse to tuple
```

* I don't see `unit`s used a lot outside of standard functions, but there are some convenience operators for them:

```hoon
> (drop-list:unity `(list (unit @))`~[`1 `2 `3])
[~ [i=1 t=~[2 3]]]

> (drop-list:unity `(list (unit @))`~[`1 `2 ~])
~
```

**Formatting Text and JSON Reparsing**

```hoon
++  format                                              ::  (2d) common formats
  ++  to-wain                                           ::  cord to line list
  ++  of-wain                                           ::  line list to cord
  ++  of-wall                                           ::  line list to tape
  ++  json-rn                                           ::  json to rn parser
  ++  enjs                                              ::  json encoders
    ++  frond                                           ::  object from k-v pair
    ++  pairs                                           ::  object from k-v list
    ++  tape                                            ::  string from tape
    ++  wall                                            ::  string from wall
    ++  ship                                            ::  string from ship
    ++  numb                                            ::  number from unsigned
    ++  sect                                            ::  s timestamp
    ++  time                                            ::  ms timestamp
    ++  path                                            ::  string from path
    ++  tank                                            ::  tank as string arr
  ++  dejs                                              ::  json reparser
    ++  ar                                              ::  array as list
    ++  as                                              ::  array as set
    ++  at                                              ::  array as tuple
    ++  bo                                              ::  boolean
    ++  bu                                              ::  boolean not
    ++  ci                                              ::  maybe transform
    ++  cu                                              ::  transform
    ++  di                                              ::  millisecond date
    ++  du                                              ::  second date
    ++  mu                                              ::  true unit
    ++  ne                                              ::  number as real
    ++  ni                                              ::  number as integer
    ++  ns                                              ::  number as signed
    ++  no                                              ::  number as cord
    ++  nu                                              ::  parse number as hex
    ++  of                                              ::  object as frond
    ++  ot                                              ::  object as tuple
    ++  ou                                              ::  object of units
    ++  oj                                              ::  object as jug
    ++  om                                              ::  object as map
    ++  op                                              ::  parse keys of map
    ++  pa                                              ::  string as path
    ++  pe                                              ::  prefix
    ++  sa                                              ::  string as tape
    ++  sd                                              ::  string @ud as date
    ++  se                                              ::  string as aura
    ++  so                                              ::  string as cord
    ++  su                                              ::  parse string
    ++  uf                                              ::  unit fall
    ++  un                                              ::  unit need
    ++  ul                                              ::  null
    ++  za                                              ::  full unit pole
    ++  zl                                              ::  collapse unit list
    ++  zp                                              ::  unit tuple
    ++  zm                                              ::  collapse unit map
  ++  klr                                               ::  styx/stub engine
++  cloy
```

* [`+enjs:format`](https://github.com/urbit/docs.urbit.org/tree/master/hoon/reference/zuse/2d_1-5/README.md) supports noun-to-JSON conversions.
* [`+dejs:format`](https://github.com/urbit/docs.urbit.org/tree/master/hoon/reference/zuse/2d_6/README.md) are the reparsers (see `+de:json:html` for the parser). Notably this is where many noun-to-text converters live.
* [`+dejs-soft`](https://github.com/urbit/docs.urbit.org/tree/master/hoon/reference/zuse/2d_7/README.md) offers non-crashing versions of the `+dejs` arms (thus, returning `unit`s).

**Diffs**

Diff tools, using the [Hunt-McIlroy algorithm](https://en.wikipedia.org/wiki/Hunt%E2%80%93Szymanski_algorithm):

```hoon
++  differ                                              ::  (2d) hunt-mcilroy
  ++  berk                                              ::  invert diff patch
  ++  loss                                              ::  longest subsequence
  ++  lurk                                              ::  apply list patch
  ++  lusk                                              ::  lcs to list patch
```

**Web Text (HTML \&c.)**

```hoon
++  html                                                ::  (2e) text encodings
  ++  mimes                                             ::  (2e1) MIME
    ++  as-octs                                         ::  atom to octstream
    ++  as-octt                                         ::  tape to octstream
    ++  en-mite                                         ::  mime type to text
    ++  base16                                          ::  MSB hex strings
    ++  base64                                          ::  flexible le base64 encoding
    ++  en-base58                                       ::  Bitcoin base-58 address, encode
    ++  de-base58                                       ::  Bitcoin base-58 address, decode
```

More JSON, this time the parser:

```hoon
++  html
  ++  json                                              ::  (2e2) JSON
    ++  en                                              ::  encode JSON to tape
    ++  de                                              ::  parse cord to JSON
      ++  abox                                          ::  array
      ++  apex                                          ::  any value
      ++  bool                                          ::  boolean
      ++  esca                                          ::  escaped character
      ++  expo                                          ::  exponent
      ++  frac                                          ::  fraction
      ++  jcha                                          ::  string character
      ++  mayb                                          ::  optional
      ++  numb                                          ::  number
      ++  obje                                          ::  object list
      ++  obox                                          ::  object
      ++  pear                                          ::  key-value
      ++  piec                                          ::  listify
      ++  stri                                          ::  string
      ++  spac                                          ::  whitespace
      ++  unic                                          ::  escaped UTF16
      ++  utfe                                          ::  UTF-8 sequence
      ++  wish                                          ::  with whitespace
      ++  sune                                          ::  cord UTF-8 sanity
      ++  sung                                          ::  char UTF-8 sanity
      ++  teff                                          ::  UTF-8 length
```

* [`+json:html`](https://github.com/urbit/docs.urbit.org/tree/master/hoon/reference/zuse/2e_2-3/README.md) tools are reparsers (see `+dejs:format` for the reparsers).
* Since JSON do not have a single canonical form as text, these parsers provide an opportunity to see how to parse something structurally when whitespace doesn't matter.

```hoon
++  html
  ++  en-xml                                            ::  xml printer
    ++  apex                                            ::  top level
    ++  attr                                            ::  attributes to tape
    ++  escp                                            ::  escape for xml
    ++  many                                            ::  nodelist to tape
    ++  name                                            ::  name to tape
    ++  clot                                            ::  self-closing tags
  ++  de-xml                                            ::  xml parser
    ++  apex                                            ::  top level
    ++  attr                                            ::  attributes
    ++  cdat                                            ::  CDATA section
    ++  chrd                                            ::  character data
    ++  comt                                            ::  comments
    ++  decl                                            ::  ++decl:de-xml:html
    ++  escp                                            ::
    ++  enty                                            ::  entity
    ++  empt                                            ::  self-closing tag
    ++  head                                            ::  opening tag
    ++  many                                            ::  contents
    ++  name                                            ::  tag name
    ++  tail                                            ::  closing tag
    ++  whit                                            ::  whitespace
  ++  en-urlt                                           ::  url encode
  ++  de-urlt                                           ::  url decode
  ++  en-purl                                           ::  print purl
  ++  de-purl                                           ::  url+header parser
  ++  en-turf                                           ::  encode as TLD-last string
  ++  de-turf                                           ::  decode from TLD-last string
  ++  fuel                                              ::  parse urbit fcgi
```

**Wires**

```hoon
++  wired                                               ::  wire formatting
  ++  dray                                              ::  load tuple in path
  ++  raid                                              ::  demand path odors
```

**Identity**

Retrieve your mathematical sponsor, convert a number to a rank, etc.

```hoon
++  title                                               ::  (2j) identity
      ++  sein
      ++  clan                                          ::  ship to rank
      +$  rank  ?(%czar %king %duke %earl %pawn)        ::  ship width class
      ++  name                                          ::  identity
      ++  saxo                                          ::  autocanon
      ++  sein                                          ::  autoboss
  ++  cite                                              ::  render ship
  ++  saxo                                              ::  autocanon
  ++  sein                                              ::  autoboss
  ++  team                                              ::  her
  ++  moon                                              ::  her moon
```

**Millisecond Timing**

Some time-related tools (currently used for timing in Eyre):

```hoon
++  milly                                               ::  (2k) milliseconds
  ++  around                                            ::  relative msec
  ++  about                                             ::  unit relative msec
  ++  mill                                              ::  msec diff
  ++  msec                                              ::  @dr to @ud ms
  ++  mull                                              ::  unit msec diff
++  contain
  ++  by-clock
  ++  to-capped-queue
```

**Userlib**

More userspace stuff. (At this point, `%zuse` is a grab bag of things that people have added over the years, and it's not clear who needs what or if it's even in contemporary use.)

```hoon
++  userlib                                             ::  (2u) non-vane utils
  ++  chrono                                            ::  (2uB) time
    ++  from-unix
    ++  from-unix-ms
    ++  dawn                                            ::  Jan 1 weekday
    ++  daws                                            ::  date weekday
    ++  deal                                            ::  to leap sec time
    ++  lead                                            ::  from leap sec time
    ++  dust                                            ::  print UTC format
    ++  stud                                            ::  parse UTC format
    ++  unm                                             ::  Urbit to Unix ms
    ++  unt                                             ::  Urbit to Unix time
    ++  yu                                              ::  UTC format constants
  ++  space                                             ::  (2uC) file utils
    ++  feel                                            ::  simple file write
    ++  file                                            ::  simple file load
    ++  foal                                            ::  high-level write
    ++  fray                                            ::  high-level delete
    ++  furl                                            ::  unify changes
  ++  unix                                              ::  (2uD) unix line-list
    ++  lune                                            ::  cord by unix line
    ++  nule                                            ::  lines to unix cord
  ++  scanf                                             ::  (2uF) exterpolation
```

* `+chrono:userlib` provides tools to print and parse basic UTC time statements.

```hoon
> (dust:chrono:userlib (yore now))
"Wed, 13 Sep 2023 20:9:38 +0000"

> (stud:chrono:userlib '13 Sep 2023 20:9:38 +0000')
[~ [[a=%.y y=2.023] m=9 t=[d=13 h=20 m=9 s=38 f=~]]]

> (stud:chrono:userlib 'Sep 13, 2023 20:9:38 +0000')
[~ [[a=%.y y=2.023] m=9 t=[d=13 h=20 m=9 s=38 f=~]]
```

(Overheard memo to self: work this into whatever [`/lib/chronos`](https://github.com/sigilante/chronos) becomes.)

* `+space:userlib` is used by the Hood tools like `|mv`.

#### The Compilation Subject

Ford uses `%zuse` (thus the full standard library) as the compilation subject for a `hoon` file. Typically a userspace file will produce a core (or, in the case of some generators, a head tag and a core). That core will contain the standard library in its context because `|` bar runes (the only runes that produce cores) return cores containing the original subject in their payloads.

Ford also allows you to modify the compilation subject by imports. (This is why you have to import files at the top of a `hoon` file, and why you do it in a particular order.) Each import is prepended to the compilation subject, so in general your compilation subject will look like `[lib1 lib2 sur1 sur2 zuse]`.

You can see this process in `+run-prelude:ford` in `/sys/vane/clay`. (`/` fas Ford runes are actually parsed in `+parse-pile` using `+pile-rule`. This is also where `/?` faswut is ignored.) In `ca10` we'll take a deep dive through Clay.

**`/lib/tiny`**

The whole standard library is included in every piece of userspace Hoon, unless you go out of your way to remove it from the subject. This is only rarely a good idea, but you \*can\* build a small working Hoon against a minimalist subject. For instance, this is done for the naïve rollup smart contract code and NockVM (née Sword, née Ares) development using `/lib/tiny`.

```hoon
> =>  tiny  (add 1 1)
2

> =>  tiny  (add:rs 1 1)
-find.rs
dojo: hoon expression failed
```

## Exercises <a href="#exercises" id="exercises"></a>

* Implement a custom aura, `@uo` (octal/byte encoding). At one level, simply implementing an aura requires no overhead. However, the rune must have a unique parsed format for input, and should have a corresponding output. (The rules around this are laxer for more complex nouns like `set`s and `tree`s.) One format which would be compatible with the restrictions on atom syntax as well as not shadow any current atom types is `0o1234.5670` (`89acbdef` are not valid characters in octal). You can model heavily on `@ux` to implement this aura. (A [tutorial](/hoon/aura-guide) is available for a degree–minute–second implementation which you can use as a guide.)
* `+sloe` is intended to receive a `$type` and return a list of the named arms in that type. Modeling on `+sloe`, produce a gate `+beau` which retrieves each face in the sample of a supplied gate argument and produces a list of them.

```hoon
> (sloe -:!>(..add))
~[%sub %gte %mod %min %mul %gth %add %div %lth %dec %dvr %max %lte
  %mas %cap %peg %list %lest %trap %mold %tree %bloq %step %pair
  %gate %bite %each %lone %qual %unit %pole %quip %trel]

> (sloe -:!>(add))
~[%$]
```

Hint:

```hoon
> ([%cell *] +<+<:~(repo ut ~(repo ut -:!>(add))))
[ 1.819.043.171
  [1.701.011.814 97 1.836.020.833 0 0]
  1.701.011.814
  98
  1.836.020.833
  0
  0
]
```


# 4. Arvo I: The Main Sequence

Core Academy lesson on Arvo's event handling architecture covering the main sequence, moves and events, ducts and wires, vane interactions, scry operations, and Arvo's ACID transaction guarantees.

*This lesson covers the structure of a mature Arvo. We'll also look at event handling: wires and ducts, running vanes, vane scries, builds, and kernel upgrades.*

> If you ever find yourself on solid ground, protect that with your life. Learn how it works and how to use it well. Then build a strong structure on top of that. Don't let it decompose into a layer of mud. (*Precepts Discussion*, C1)

A good place to begin our study of [Arvo](https://www.youtube.com/watch?v=QtFPdBUl7XQ) is to considered the design intent to Urbit as a system:

> Arvo is designed to avoid the usual state of complex event networks: event spaghetti. We keep track of every event's cause so that we have a clear causal chain for every computation. At the bottom of every chain is a Unix I/O event, such as a network request, terminal input, file sync, or timer event. We push every step in the path the request takes onto the chain until we get to the terminal cause of the computation. Then we use this causal stack to route results back to the caller.

You are likely familiar with the high-level concept of Arvo as an event handler and the main ratchet of the state machine.

$$
L: \text{History} \rightarrow \text{State}
$$

Before we embark into Arvo, let's define some terms:

* A *move* is a “cause and action”. There is a formal `$move` type that specifies the associated call stack and action.
* An *event* is a completed move. The event results in an updated state, or subject for future computations. A completed event is recorded immutably in the event log history. Arvo itself doesn't know about the event log and history—it is amnesiac in the sense that it only computes against its current state.

> Arvo is a pure function f(logs) of its event log, so formally Arvo is just a function run against an event log. A naive implementation has very bad asymptotics; processing each new event is O(n) in the number of historical events. Choose the function g(state,log) such that f(logs ++ log) = g(f(logs),log). Then, as long as you keep the state in memory, processing each new event is constant in the number of previous events. This still requires O(n) restart from disk, but you can also periodically (and non-blockingly) write a checkpoint of the state to disk, so that restart from disk is only linear in the number of events since the last checkpoint. (*Precepts Discussion* A11)

* [“Arvo Overview”](https://github.com/urbit/docs.urbit.org/tree/master/reference/arvo/overview/README.md)

### Guarantees <a href="#guarantees" id="guarantees"></a>

1. Solid-state. Specifically, Urbit is a *solid-state interpreter*. What is meant by that? If you search "solid-state interpreter", then Urbit is the answer, so it's not much help! “Solid state” refers to Urbit's grounded ability to acquire and specify state, as a unique, auditable, and reproducible basis for computing. An event, once committed to the log, is immutable and permanent, never lost.

   Arvo's solid-statefulness differs from every major operating system in that no required information in the state of the OS is stored in RAM alone. (Incidentally, of course, the runtime uses RAM quite a lot, and a move being processed may be lost with a sudden shutdown.) This is one reason that Urbit is particularly write-heavy on hard drives, a consideration that hosting providers must take into account.
2. Interpreter. Urbit is an “interpreter” in the same sense that the Java Virtual Machine is an interpreter, only Nock serves the role of the Java bytecode. This means that it can receive a noun and update itself *in situ*.

   Furthermore, Hoon is compiled and built continually throughout the lifetime of the process.
3. Atomic. Events

> An interrupted event never happened. The computer is deterministic; an event is a transaction; the event log is a log of successful transactions. In a sense, replaying this log is not Turing complete. The log is an existence proof that every event within it terminates.

4. Consistency. Every update (completed event) leads to a new valid state.
5. Isolation. Events are performed sequentially, meaning that the effects are isolated. (See the “Breadth-First Move Ordering” section below.)
6. Durability. Completed events are permanent and immutable. No events will be reversed. But:

> It is easy to think that "completed transaction will survive permanently" along with "the state of Arvo is pure function of its event log" implies that nothing can ever be deleted. This is not quite true. Clay is our referentially transparent file system, which could naively be thought to mean that since data must be immutable, files cannot be deleted. However, Clay can replace a file with a "tombstone" that causes Clay to crash whenever it is accessed. Referential transparency only guarantees that there won't be new data at a previously accessed location - not that it will still be available.

## Arvo’s Structure <a href="#arvos-structure" id="arvos-structure"></a>

The beating heart of Arvo is a Nock expression, formally `[%2 [%0 3] %0 2]`. This means to evaluate the `battery` formula against the current subject to yield a new `battery`, then to evaluate that second `battery` against the subject produced from evaluating the `payload` against the current subject. In other words, to process the next event.

This is presented explicitly by the arm `+aeon`, for instance, in the `+eden` lifecycle formula generator. (The inline commentary there is worth reading in full; we'll see `+eden` again in [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04).) Elsewhere it results operationally.

In most respects, tho, Arvo is an event processor. Most of the event processing and routing machinery is contained in `/sys/arvo`, with some plumbing on the vane-side as well.

```hoon
+$  move  [=duct =ball]                                     :: CAUSE & ACTION
                                                            ::
+$  duct  (list wire)                                       :: CAUSAL HISTORY
                                                            ::
+$  ball  (wite [vane=term task=maze] maze)                 :: see below
+$  ball                                                    :: DYNAMIC KERNEL
                                                            :: ACTION
    $?  [%hurl [%error-tag stack-trace] wite=pass-or-gift]  :: action failed
        [%pass wire=/vane-name/etc note=[vane=%vane-name task=[%.y p=vase]]]
                                                            :: advance
                                                            :: request
        [%slip note=[vane=%vane-name task=[%.y p=vase]]]    :: lateral;
                                                            :: make a request
                                                            :: as though you're
                                                            :: a different vane
        [%give gift=[%.y vase]                              :: retreat; response
                                                            ::
+$  card  (cask)                                            :: tagged untyped 
                                                            :: event
                                                            ::
++  cask  |$  [a]  (pair mark a)                            :: marked data 
                                                            :: builder
                                                            ::
+$  meta  (pair)                                            :: meta-vase
                                                            ::
+$  maze  (each vase meta)                                  :: vase or meta-vase
```

(There is some entanglement with particular vanes, which undercuts the simplicity of the system. It would be formally nice to refactor `/sys/arvo` into a pure statement of the event log without reference to vanes, which should be largely hypothetical to an event loop aside from routing labels. Cf. [\~wicrum-wicrun on “Urbit is a ball of mud”](https://gist.github.com/wicrum-wicrun/b308b298721ccbe0de9c6c73a6ee4242#urbit-is-a-ball-of-mud). On the other hand, perhaps that loses efficiency. A question for a cooler kelvin.)

* A `$move` is a cause and effect, simply a request to complete some task in a computation on behalf of the causal stack. Think of this as message data and history metadata. A move sends an action to a location along a call stack.
* A `$duct` is a causal history, or the auditable chain of causes that leads to the current computation.
* A `$ball` is an action. This is the part we conventionally think of as a computation: one of a `%hurl` failure, a `%pass` advance, a `%slip` lateral move, or a `%give` return back down the causal chain.
* A `$card` is a tagged untyped event. (Notably, it is not `$card:agent:gall`.) A card is an event of action. Cards can be arbitrarily complicated depending on the vane and message.
* A `+cask` is a marked data builder, commonly used to transmit data over the network (since vases are local only).
* A `+meta` meta-vase is an untyped vase (e.g. a vase of vase). We'll see more of these in their `$maze` form.

If you read up on the runtime, at some point it used to refer to an “Arvo-shaped noun”. That means that the runtime expects to receive a core with certain arms, as this is how it will systematically interact with the noun it hosts.

Arvo defines four standard arms for vanes and the binary runtime to use:

* `+load` is used in kernel upgrades, allowing Arvo to update itself in-place, and is canonically at `+4`. (Formerly a `+come` arm was defined to assist in this procedure, targeting `$type` changes.)
* `+wish` accepts a core and parses it against `%zuse`, which is instrumentation for runtime access, and is canonically at `+10`. (See `ca02`.)
* `+peek` grants read-only access to a vane; this is called a *scry*, and is canonically at `+22`.
* `+poke` accepts `$move`s and processes them; this arm alters Arvo’s state, and is canonically at `+23`.

Each arm possesses the same structure, which means that as the Urbit OS kernel grows and changes the main event dispatcher can remain the same. For instance, when the build vane `%ford` was incorporated into `%clay`, no brain surgery was needed on Arvo to make this possible and legible. Only the affected vanes (and any calls to `%ford`) needed to change.

There's another wrinkle to the “simplicity” of Arvo: there are actually four Arvos in `/sys/arvo`:

1. The larval core, used in building Arvo for the first time. We'll discuss this with the boot process in [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04).
2. The structural interface core, the primary mature core, which is in operation for most of a ship's lifecycle and contains Arvo's state. It primarily calls out to the next two cores.
3. The implementation core, the actual operational core carrying out particular computations and dispatches.
4. The Section 3bE core, containing helper functions for Arvo that are sequestered for security and to maintain the four-arm prototype of Arvo. E.g., how to parse a scry path, how to negotiate versions, just Arvo's library code. Only pure functions, no state machines.

Urbit hews to the principle that “stateless is better than stateful” (*Precepts* A16). Most parts of the system are designed with the intent that they are stateless, or that any state is explicitly sequestered to a particular core. In Arvo, the state is the entire kernel in operation. There are persistent components (the Arvo state) and the ephemeral move worklists.

```hoon
::  persistent arvo state
::
=/  pit=vase  !>(..is)                                  ::
=/  vil=vile  (viol p.pit)                              ::  cached reflexives
=|  $:  lac=_&                                          ::  laconic bit
        eny=@                                           ::  entropy
        our=ship                                        ::  identity
        bud=vase                                        ::  %zuse
        vanes=(map term vane)                           ::  modules
    ==                                                  ::
```

* What is `!>(..is)` doing here? It refers to the code above this point in Arvo as a `vase`.
* `vil` is a cache of specific types—`type`, `duct`, `path`, `vase`. This saves on recompilation.
* `lac` is the verbosity dial (`|verb`).
* `eny` is the entropy.
* `our` is the ship's identity.
* `bud` is the stdlib.
* `vanes` is a list of vanes (more in a moment).

### `+wish` and `urbit eval` <a href="#wish-and-urbit-eval" id="wish-and-urbit-eval"></a>

Is `urbit eval` just `+wish`? Yes, it is—but it's not, in practice, any of the Arvo cores in `/sys/arvo`. In the king (the main process you'd run `urbit eval` through), it's actually an Arvo-shaped wrapper in `/lib/vere` which also sees Azimuth and Ethereum data. This is the outermost layer of the ivory pill (see [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04)).

## Vanes & Move Handling <a href="#vanes-move-handling" id="vanes-move-handling"></a>

### Vanes <a href="#vanes" id="vanes"></a>

Arvo is essentially an event router between heavyweight modules but including a stack discipline. (It's sort of like using a router to bind names to callback functions.) We need to consider two aspects of event processing:

1. Kinds of moves
2. Move mechanics

But first, what is Arvo routing moves between? Arvo contains in its state:

```hoon
vanes=(map term vane)
```

where a `$vane` is `[=vase =worm]`. (On which more later.) Arvo also stores a map from a name to a `vase` and a `worm` cache.

```hoon
++  grow
  |=  way=term
  ?+  way  way
    %a  %ames
    %b  %behn
    %c  %clay
    %d  %dill
    %e  %eyre
    %g  %gall
    %i  %iris
    %j  %jael
    %k  %khan
    %l  %lick
  ==
```

A vane is ultimately a `vase` describing an outer gate which produces a core. The vane's state is that gate that wraps the core that has the right standard arms. Why a `vase`? Because all of this is going to be done explicitly in vase mode.

Eliding a lot of code, here is the skeleton of a vane, in this case Behn:

```hoon
|=  our=ship
=>  |%
    +$  behn-state
      $:  %2
          timers=(tree [key=@da val=(qeu duct)])
          unix-duct=duct
          next-wake=(unit @da)
          drips=drip-manager
      ==
    --
::
=>
=|  behn-state
=*  state  -
|=  [now=@da eny=@uvJ rof=roof]
=*  behn-gate  .
^?
|%
::  +call: handle a +task:behn request
::
++  call
  |=  $:  hen=duct
          dud=(unit goof)
          wrapped-task=(hobo task)
      ==
  ^-  [(list move) _behn-gate]
  * * *
::  +load: migrate an old state to a new behn version
::
++  load
  |=  old=behn-state
  ^+  behn-gate
  * * *
::  +scry: view timer state
::
++  scry
  ^-  roon
  |=  [lyc=gang pov=path car=term bem=beam]
  ^-  (unit (unit cage))
  * * *
::
++  stay  state
::
++  take
  |=  [tea=wire hen=duct dud=(unit goof) hin=sign]
  ^-  [(list move) _behn-gate]
  * * *
```

A vane exposes five standard arms:

* `+stay` produces the state of the vane.
* `+load` migrates the state of the vane.
* `+call` handles an incoming task request.
* `+take` handles a response `sign`.
* `+scry` return the vane's state (as before a state migration).

### Kinds of Moves <a href="#kinds-of-moves" id="kinds-of-moves"></a>

Arvo is an event loop, but until an event completes it is merely a move, a pair of cause and effect. Most of Arvo's work is done dispatching `$card`s to and from vanes, which do the actual computation. From userspace, we are accustomed to seeing `%pass` and `%gift` Gall `$card:agent:gall`s. These are actually nerfed relative to Arvo `$move`s, which contain more information. (They are laundered through `+wind`.)

A `%pass` `move` is analogous to a call:

```hoon
[duct %pass return-path=path vane-name=@tD data=card]
```

Arvo pushes the return path (preceded by the first letter of the vane name) onto the `duct` and sends the given data, a `card`, to the vane we specified. Any response will come along the same `duct` with the `wire` `return-path`.

A `%give` `move` is analogous to a return:

```hoon
[duct %give data=card]
```

Arvo pops the top `wire` off the `duct` and sends the given `card` back to the caller.

Each vane defines a protocol for interacting with other vanes (via Arvo) by defining four types of cards for its own namespace: tasks, gifts, notes, and signs.

> When one vane is `%pass`ed a `card` in its `task` (defined in `zuse`), Arvo activates the `+call` gate with the `card` as its argument. To produce a result, the vane `%give`s one of the `card`s defined in its `gift`. If the vane needs to request something of another vane, it `%pass`es it a `note` `card`. When that other vane returns a result, Arvo activates the `+take` gate of the initial vane with one of the `card`s defined in its `sign`.

In other words, there are only four ways for Arvo and vanes to see a move:

1. as a request seen by the caller, which is a `note`.
2. that same request as seen by the callee, a `task`.
3. the response to that first request as seen by the callee, a `gift`.
4. the response to the first request as seen by the caller, a `sign`.

![](https://media.urbit.org/docs/arvo/cycle.png)

What does a vane call look like? They are specific to each vane. The top-level type for a `note` looks like this:

```hoon
+$  note-arvo                                           ::  out request $->
  $~  [%b %wake ~]
  $%  [%a task:ames]
      [%b task:behn]
      [%c task:clay]
      [%d task:dill]
      [%e task:eyre]
      [%g task:gall]
      [%i task:iris]
      [%j task:jael]
      [%k task:khan]
      [%l task:lick]
      [%$ %whiz ~]
      [@tas %meta vase]
  ==
```

`/sys/vane/behn` defines the following interface in `/sys/lull`:

```hoon
::                                                      ::::
::::                    ++behn                            ::  (1b) timekeeping
  ::                                                    ::::
++  behn  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%doze p=(unit @da)]                            ::  next alarm
        [%wake error=(unit tang)]                       ::  wakeup or failed
        [%meta p=vase]
        [%heck syn=sign-arvo]                           ::  response to %huck
    ==
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  $>(%born vane-task)                             ::  new unix process
        [%rest p=@da]                                   ::  cancel alarm
        [%drip p=vase]                                  ::  give in next event
        [%huck syn=sign-arvo]                           ::  give back
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
        [%wait p=@da]                                   ::  set alarm
        [%wake ~]                                       ::  timer activate
    ==
  --  ::behn
```

From Behn's perspective, it can receive a task or a gift. Anything sent to another vane is a note or a sign. So if you issue a call to Behn from Gall, the lifecycle looks like this:

1. Agent sends vane-specific `card` to Gall as `card:agent:gall`.
2. Gall `%pass`es a `note` to Arvo through the `+poke` arm.
3. Arvo `%pass`es a `task` to Behn through the `+call` arm.
4. When Behn wakes up on the timer, it `%give`s a `%gift` to Arvo through the `+poke` arm.
5. Arvo `%give`s a `sign` to Gall through the `+take` arm.
6. Gall passes a `card:agent:gall` to agent's `+on-arvo` arm.

Besides a `%pass` (forward) or `%gift` (reverse) move, there is also a `%slip` move.

> A `%slip` `move` is a cousin of `%pass`. Any `card` that can be `%pass`ed can also be `%slip`ed, but while a `%pass` says to "push this `wire` onto the `duct` and transfer control to the receiving vane", a `%slip` transfers control to the receiving vane without altering the `duct`. Therefore, a `%give` in response to a `%slip` will go to the caller of the vane that sent the `%slip` rather than the vane that actually sent the `%slip`. `%slip`s are much more rare than `%pass`es and `%give`s. In general, `%slip` and `%pass` `move`s are both referred to as "passes" and it should be clear from the context if one means to refer only to `%pass`es and not `%slip`s or vice versa. Lastly, we note that `%slip` is a code smell and should nearly always be avoided. It can result in unexpected behavior like receiving a gift from a vane you never passed a note to.

In short, you `%slip` without pushing onto the `duct`so that control is given back to the top `wire` in the `duct`.

When is `%slip` preferred? It's not—it's typically code smell. Since it returns not to you but to your caller, it violates layering and breaks abstraction. In practice, it's not used much anymore, but it shows up in a few sensitive places like the subscription lifecycle, `%init` tasks (which only happen once in the lifecycle of each ship), and in a Clay-initiated Arvo upgrade. (Joe argues this is probably legacy functionality, and can likely be cleaned up after breadth-first move ordering is completed.)

The old docs refer to a `%unix` task, but this was always sort of ill-defined. `%lull`'s '[`$unix-task`](https://github.com/urbit/urbit/blob/5ff4e88890ee17413b35d2bd648392e3236ad9c7/pkg/arvo/sys/lull.hoon#L3488) defines a subset of expected tasks, but the way vanes work now, the `$unix-task` predicate is no longer needed so you don't think of it as a `%unix` task anymore. In actuality any arbitrary task can come in via `conn.c`. In practical terms `%aqua` (and likely `%pyro`) use this feature.

#### Exercise

* Produce a minimalist Arvo and vane system. This should look like three cores: an Arvo core which can handle the four basic move types; and two vanes (say `+lorem` and `+ipsum`) with minimalist `gift`/`task` interfaces and `+call`/`+take` arms. You can do this in a `%say` generator, for instance, or a `/lib`—don't worry about setting up a whole agent or modifying the kernel. (We'll build a vane later!)

### Move Mechanics <a href="#move-mechanics" id="move-mechanics"></a>

Arvo basically only knows how to glue things together; in particular, it knows vanes by labels and simple interfaces alone. (See e.g. `+grow`.) Arvo's formal state (thus Urbit's formal state) is always just to be a gate which operates on an event to produce the next state.

```hoon
+$  vane  [=vase =worm]
```

A vane is a `vase` and a `worm`. The `vase` encompasses the `type` and the noun of the vane. Vanes are compiled with `%zuse` as their subject (thus the other inner cores as well).

Vanes operate in vase mode. Any vane call is a `+slap` in vase mode. For instance, vanes can emit cards or metacards, a vase of a card. The `+va` and `+wa` engines assist with operating vanes.

**Aside: Metacards & `;;` micmic**

The vane interface is normally strictly typed, but using a metavase it can punch a hole through the `type` system. (This used to be the only way to do vase reduction before `!<` zapgal was introduced: a vane had to pass to Arvo to get into double vase mode, which Arvo would collapse into one vase mode and hand back.)

Urbit core developers used to be more cautious about molding data. The mold system in particular was a total system prior to the introduction of structure mode in 2018: if the input mismatched, the mold would bunt (so the correct type was always returned). (Joe describes the mold system at this point as “a pile of hacks”.) If you knew the `type`, you could mold a value quickly to make it a static value.

Since molds would bunt on input mismatch, you needed to have a fixed-point assertion that guaranteed validation, thus `;;` micmic. After the introduction of structure mode (“spec mode”) changes, molds crash if they fail (rather than bunting). So the primary reason `;;` micmic still exists is the easy of entering parser mode:

```hoon
> (,[%foo %bar] [%foo %bar])
[%foo %bar]

> ;;([%foo %bar] [%foo %bar])
[%foo %bar]
```

Other than that, it is essentially superfluous in contemporary Hoon.

#### `+va`: Vane Operations

Vanes have a definite interface as described above. `+va` is an engine for interacting with them. So the `+va` core is for actually running vanes. It accepts `$maze`s as input (like the `+call` arm, which advances the vane state). `+va` is basically a workhorse to handle vane transactions and there's not a lot to say about it as a vane.

E.g. `+plow` is the sole arm of an inner core, designed to “operate in time and space” on a vane. It is a gate which attaches a `rook` (meta-namespace) to an Arvo-side interface for a vane and evaluates the vane within that namespace.

Although not in `+va`, a related gate is the `+look` arm in the Section 3bE Arvo core. Each vane receives a `$roof` in its function call which it can call when it needs to scry. `+look` converts into a `+mink`-compatible version, acting as a bridge between `.^` dotket and the scry handler (`+look`) with access to the `roof` that came from the vane. This pattern imposes constraints on interpreters, but if you just gave agents a scry handler then its a gate closed over the entire vane state so one could access anything in any vane state. Thus `.^` dotket gives mediated access so you don't reveal state of the entire system in the subject (untyped permissions-free access). This uses the `+wa` interface (below) but doesn't cache because it's a stateful function, mold and path hoon compiler turns mold into a type and you get an untyped nest-check on the result. (This is where `scry-lost` comes from.) \~mastyr-bottec will ultimately replace this in-Arvo stateful cache with persistent memoization in the runtime.

#### `+wa`: `worm` Cache

All vane calls are fundamentally a `+slap` in vase mode, so formally Arvo runs the compiler all the time. To help with this, there's a `worm` cache to help speed that up. Thus `+wa` is where first-time computations are initiated and then stored in the cache. (The `worm` cache is part of Arvo's state; where did that slip in through the definition we had above?)

The `worm` cache is manual memoization in the dumbest possible way.

```hoon
+$  worm
  $:  ::  +nest, +play, and +mint
      ::
      nes=(set ^)
      pay=(map (pair type hoon) type)
      mit=(map (pair type hoon) (pair type nock))
  ==
```

* `nes` is s set of pairs of types that nest. (I.e., if it's in the set then it nests.)
* `pay` is a map of `type` and `hoon` to `type`.
* `mit` is a map of `type` and `hoon` to `type` and `nock`.

There are no faces so lark notation is used throughout `+wa`.

Each call to a vane falls into one of three categories:

1. Cached: produce `%.y` and `wa-cache`.
2. Not cached and failed `nest-check`: produce a manual `printf`.
3. Not cached and succeeded `nest-check`: produce `%.n` and `wa-cache`.

In Arvo, a lot of pieces have to be done manually. For instance, `+open:wa` is a manual `%~` censig on a door. (Joe points out that `+wa` is currently a messy core because half of the operations are on `vase`s and half are on `maze`s.) You have to operate untyped throughout:

* `+slur` calls `+neat` instead of `+nest`
* `+neat` is either `vase` or `maze`, and calls `+nets` instead of `+nest` because it is untyped, then makes the `nest-check` with `+slum` (an untyped `+slam` working on a raw `nock`).
* E.g. when Arvo runs the `+scry` arm of a vane, it receives back a `vase` of `(unit (unit cage))` so there are multiple layers of nesting. This needs a sort of manual `!<` zapgal, not actually promoting into a type but still operating on it. (Both of these methods are hacks, but this is less dangerous (only true because vanes are trusted), whereas `!<` zapgal is a type hole implicating the whole system.)
* The other time we construct these is when we get raw events (any `$ovum`).

Arvo maintains a `worm` cache for each vane, built when initializing. Ultimately this should be changed to runtime memoization instead. That will allow arbitrary associative memory (and presumably constant-time lookup), but these are currently double-`+mug` balanced treaps.

### Runtime Connexions <a href="#runtime-connexions" id="runtime-connexions"></a>

Two functions are of particular interest in beginning to see how the runtime handles an event: `u3_serf_work`, which applies events and produces effects; and `u3v_poke_sure`, which injects an event and saves the new state if successful.

We'll see more about this in `ca05` when we look at the structure of Vere.

### Breadth-First Move Ordering <a href="#breadth-first-move-ordering" id="breadth-first-move-ordering"></a>

Planned for the future is breadth-first move ordering ([current PR](https://github.com/urbit/urbit/pull/6775)). What does this mean? (Much of this section quotes [\~wicdev-wisryt's #6041 PR](https://github.com/urbit/urbit/pull/6041) description.)

Arvo currently orders moves by depth first. Visually, when evaluating depth-first an event may look like this:

```
["" %unix %belt /d/term/1 ~2022.10.27..06.32.09..30db]
["|" %pass [%dill %g] [[%deal [~zod ~zod] %hood %poke] /] ~[//term/1]]
["||" %give %gall [%unto %poke-ack] i=/dill t=~[//term/1]]
["||" %pass [%gall %g] [[%deal [~zod ~zod] %dojo %poke] /use/hood/0w2.efXKi/out/~zod/dojo/drum/phat/~zod/dojo] ~[/dill //term/1]]
["|||" %give %gall [%unto %poke-ack] i=/gall/use/hood/0w2.efXKi/out/~zod/dojo/drum/phat/~zod/dojo t=~[/dill //term/1]]
["|||" %give %gall [%unto %fact] i=/gall/use/hood/0w2.efXKi/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["||||" %give %gall [%unto %fact] i=/dill t=~[//term/1]]
["|||||" %give %dill %blit i=/gall/use/herm/0w2.efXKi/~zod/view/ t=~[/dill //term/1]]
["|||||" %give %dill %blit i=/gall/use/herm/0w2.efXKi/~zod/view/ t=~[/dill //term/1]]
["|||||" %give %dill %blit i=/gall/use/herm/0w2.efXKi/~zod/view/ t=~[/dill //term/1]]
["|||" %give %gall [%unto %fact] i=/gall/use/hood/0w2.efXKi/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["||||" %give %gall [%unto %fact] i=/dill t=~[//term/1]]
["|||||" %give %dill %blit i=/gall/use/herm/0w2.efXKi/~zod/view/ t=~[/dill //term/1]]
["|||" %give %gall [%unto %fact] i=/gall/use/hood/0w2.efXKi/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["|||" %give %gall [%unto %fact] i=/gall/use/hood/0w2.efXKi/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
```

The breadth-first equivalent looks like this:

```
["" %unix %belt /d/term/1 ~2022.11.2..00.18.15..ce5b]
["1" %pass [%dill %g] [[%deal [~zod ~zod] %hood %poke] /] ~[//term/1]]
["11" %give %gall [%unto %poke-ack] i=/dill t=~[//term/1]]
["12" %pass [%gall %g] [[%deal [~zod ~zod] %dojo %poke] /use/hood/0w2.Rh6DI/out/~zod/dojo/drum/phat/~zod/dojo] ~[/dill //term/1]]
["121" %give %gall [%unto %poke-ack] i=/gall/use/hood/0w2.Rh6DI/out/~zod/dojo/drum/phat/~zod/dojo t=~[/dill //term/1]]
["122" %give %gall [%unto %fact] i=/gall/use/hood/0w2.Rh6DI/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["123" %give %gall [%unto %fact] i=/gall/use/hood/0w2.Rh6DI/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["124" %give %gall [%unto %fact] i=/gall/use/hood/0w2.Rh6DI/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["125" %give %gall [%unto %fact] i=/gall/use/hood/0w2.Rh6DI/out/~zod/dojo/1/drum/phat/~zod/dojo t=~[/dill //term/1]]
["1221" %give %gall [%unto %fact] i=/dill t=~[//term/1]]
["1231" %give %gall [%unto %fact] i=/dill t=~[//term/1]]
["12211" %give %dill %blit i=/gall/use/herm/0w2.Rh6DI/~zod/view/ t=~[/dill //term/1]]
["12213" %give %dill %blit i=/gall/use/herm/0w2.Rh6DI/~zod/view/ t=~[/dill //term/1]]
["12215" %give %dill %blit i=/gall/use/herm/0w2.Rh6DI/~zod/view/ t=~[/dill //term/1]]
["12311" %give %dill %blit i=/gall/use/herm/0w2.Rh6DI/~zod/view/ t=~[/dill //term/1]]
```

These are the same 15 lines, and they're each at the same depth, but they're in a different order. Most code we write is agnostic to this order, because there are many circumstances where this order gets inverted compared to our expectations.

The most obvious difference is that the four `%fact`s from `%dojo` to `%hood` happen one right after the other instead of being mingled with other moves. If some of those intermingled moves invoked dojo (which would be a form of reentrancy) and caused dojo to emit more facts, those facts would be given to hood *before* the %facts which were already on the stack to be sent to hood. If this is textual output, then it will be in reverse order. The breadth-first move order fixes this problem completely by running all the facts that were issued at the same time before processing the moves that those themselves produced.

This leads to a very important principle: breadth-first ordering guarantees that moves will be processed in the order they are emitted. Depth-first ordering constantly violates this, and in the presence of reentrancy this can cause extremely unexpected results.

(The [#6041 PR](https://github.com/urbit/urbit/pull/6041) elaborates on the motivation for the current depth-first ordering.)

Depth-first ordering will be deprecated in favor of breadth-first ordering, which should make the issuance of multiple cards more transparent to reason about.

> he actually emailed me the other day (i haven't communicated with him in years) to exhort me to reject the breadth-first move ordering pr, and we had a little back-and-forth. he still believes you can use moves essentially as function calls, and if you just add enough queues to random places, it'll all work out cleanly, like adding epicycles

#### Exercise

* Follow a move trace. Type `|verb` into Dojo, followed by `+ls %`. The system will respond with a verbose description of how the move was processed. (Since the system will produce more output in response to subsequent events, it's easiest to copy this into a text editor for review.) You can turn `|verb` off again as a toggle. There is also an [annotated move trace](https://github.com/urbit/docs.urbit.org/tree/master/reference/arvo/tutorials/move-trace/README.md) you can read for more perspective.

## OTAs <a href="#otas" id="otas"></a>

How does an over-the-air (OTA) update work? Essentially we transmit a noun to Arvo describing the new state and update (transition) rules. In practice, there are three possibilities when an upgrade command is sent to Arvo as a task using `+what`, picking the Hoon source out of the list there:

1. No-op: empty list or the same version.
2. No Arvo change: just an upgrade of vanes or stdlib (`%lull`/`%zuse`).
3. New Arvo kernel, up to and including a language change.

Any time you need your code to be flexible for future upgrades, you have to be inflexible about a few things—but as few as possible. The Arvo upgrade process is intended to constrain future Arvo as little as possible.

Arvo knows about `/sys` and the relative precedence of `%hoon`/`%arvo`/`%lull`/`%zuse`/vanes so it can rebuild based on the topology of changes. If only `%lull`/`%zuse`/a vane are different, then you only need to build the new cores and provide them as inner cores for the new system. If, on the other hand, `%hoon` or `%arvo` are different, then we need to build the new `/sys/hoon`, compile the new `/sys/arvo`, then gather all persistent and ephemeral state plus the new upgrade state and hand that to the new `%arvo` core. (If Arvo itself changes, then it tries to get into the new world as quickly as possible.)

This is an area where depth-first v. breadth-first move ordering makes a difference. Current DFMO means there is always a worklist to pass to the new Arvo, because there may still be moves to run. This acts as a constraint on new vanes because they may have to handle old-world moves still. BFMO means there will be more flexibility.

Concretely, once you learn a new revision exists, you tell Clay to merge the desks. Clay talks to the publisher over Ames to retrieve the metadata, then requests the individual files using Fine (fee-NAY), the remote scry protocol. (In essence, this allows the subscribers to check out the desk updates to `%base` directly from the serving runtime rather than needing to request to the server's Arvo.) With that data, the commit process starts. This is handled by `+park` in Clay.

`+park` has some state for the pending blob store \&c. Then `+sys-update` is called, which sends the move to Arvo with all source code for the new commit. A blob is a content-addressed store of data. This is one place where `%slip` comes in handy: Clay `%slip`s a `%pork` to itself to trigger the continuation, so once Arvo is done upgrading, the next thing for Arvo to run is Clay with an empty `%pork` task—thus Clay goes right back into the same arm and continues where it left off in the commit process.

One place complications arise is with `$type`, since a change to `$type` impacts vase mode handling.

> Some subtleties regarding types arise when handling OTA updates, since they can potentially alter the type system. Put more concretely, the type of `type` may be updated. In that case, the update is an untyped Nock formula from the perspective of the old kernel, but ordinary typed Hoon code from the perspective of the new kernel. Besides this one detail, the only functionality of the Arvo kernel proper that is untyped are its interactions with the Unix runtime.

Some particular details of an Arvo upgrade will be discussed later, since ultimately, OTA upgrades to Arvo have much in common with the boot process—next week's lesson, [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04), covers a new ship's boot process.

## Homework <a href="#homework" id="homework"></a>

* Annotate a move trace. Produce a move trace, such as from a generator invocation like `|pass [%d %text "foo"]`. Comment line-by-line on what is happening. (Some lines, like repeated `%hood` calls, can be grouped.) You can turn verbose logging off again using `|verb` to toggle.
* Produce a functional minimalist Arvo to boot on Vere. You need to make a gate that produce a core with four arms, each returning a gate with the correct type signature. (`/sys/arvo` can see all of `/sys/hoon`.) This should be built into a new boot pill, `baby.pill`. Phil walks through this whole process in [this video](https://www.youtube.com/watch?v=fOVhCx1a-9A). You could `%slog` input on a vane no-op.


# 5. Arvo II: The Boot Sequence

Core Academy lesson on Arvo's boot sequence covering pill creation and loading, larval and adult phases, vane initialization, userspace bootstrapping, and the complete boot process.

*This lesson covers Arvo's boot sequence: pills, the larval phase, initializing vanes, initializing userspace.*

> The formal state of an Arvo instance is an event history, as a linked list of nouns from first to last. The history starts with a bootstrap sequence that delivers Arvo itself, first as an inscrutable kernel, then as the self-compiling source for that kernel. (Whitepaper)

When Arvo starts for the first time, how does it work? It tells you something of that process; as of Vere 2.12/Arvo 412 K, the boot sequence output looks like this:

Fake ship \~fes:

```hoon
~
urbit 2.12
boot: home is fes
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
boot: downloading pill https://bootstrap.urbit.org/urbit-v2.12.pill
boot: parsing %solid pill
dock: pace (live): configured at fes/.bin/pace
vere: binary copy succeeded
loom: mapped 2048MB
boot: protected loom
live: logical boot
boot: installed 661 jets
---------------- playback starting ----------------
pier: replaying events 1-14
arvo: metamorphosis
clay: kernel updated
clay: rebuilding %base after kernel update
gall: installing %acme
gall: installing %azimuth
gall: installing %dbug
gall: installing %dojo
gall: installing %eth-watcher
gall: installing %hood
drum: link [~fes %dojo]
kiln: boot
gall: installing %herm
gall: installing %lens
gall: installing %ping
gall: installing %spider
gall: installing %talk-ui
Not running %settings-store yet, got %poke
gall: installing %docket
gall: installing %treaty
gall: installing %hark-store
gall: installing %hark-system-hook
gall: installing %settings
gall: installing %settings-store
gall: installing %storage
gall: installing %reel
gall: installing %bait
gall: installing %vitals
gall: installing %growl
docket: fetching %http glob for %talk desk
docket: fetching %http glob for %garden desk
docket: fetching %http glob for %talk desk
docket: fetching %http glob for %garden desk
docket: fetching %http glob for %webterm desk
docket: fetching %http glob for %landscape desk
gall: installing %metadata-store
gall: installing %contact-store
gall: installing %chat-store
gall: installing %graph-store
gall: installing %group-store
%group-store: on-init
gall: installing %invite-store
gall: installing %s3-store
gall: installing %chat-hook
gall: installing %chat-view
gall: installing %clock
gall: installing %contact-hook
gall: installing %contact-pull-hook
gall: installing %contact-push-hook
gall: installing %contact-view
gall: installing %dm-hook
gall: installing %graph-pull-hook
gall: installing %graph-push-hook
gall: installing %group-pull-hook
gall: installing %group-push-hook
gall: installing %group-view
gall: installing %hark-chat-hook
gall: installing %hark-graph-hook
gall: installing %hark-group-hook
gall: installing %hark-invite-hook
gall: installing %invite-hook
gall: installing %invite-view
gall: installing %launch
gall: installing %metadata-hook
gall: installing %metadata-pull-hook
gall: installing %metadata-push-hook
gall: installing %observe-hook
gall: installing %sane
gall: installing %weather
gall: not running %file-server yet, got %poke
%group-store: on-peek on path /y/groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
docket: fetching %http glob for %groups desk
gall: installing %groups
gall: installing %chat
gall: installing %contacts
%contacts: on-init
gall: installing %heap
gall: installing %diary
gall: installing %hark
gall: installing %notify
gall: installing %groups-ui
gall: installing %grouper
%contacts: on-poke with mark %noun
%contacts: on-agent on wire /migrate, %poke-ack
[%agent-giving-on-system-duct %diary %fact]
pier: (14): play: done
---------------- playback complete ----------------
vere: checking version compatibility
loom: image backup complete
lick init mkdir fes/.urb/dev
ames: live on 31592 (localhost only)
conn: listening on fes/.urb/conn.sock
lick: %born failure;
http: web interface live on http://localhost:8080
http: loopback live on http://localhost:12321
pier (25): live
docket: fetching %http glob for %garden desk
ames: metamorphosis
; ~zod is your neighbor
~fes:dojo>
```

Live ship comet:

```
~
urbit 2.12
boot: home is /home/neal/comet-412
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
Downloading pill https://bootstrap.urbit.org/urbit-v2.12.pill
Mining a comet. May take up to an hour.
If you want to boot faster, get an Urbit identity.
Found comet ~mipped-pinlug-loshec-tastun--tabfen-bitwex-norsul-wanzod
boot: verifying keys
Getting sponsor
boot: retrieving galaxy table
boot: retrieving network domains
boot: retrieving keys for sponsor ~wanzod
boot: retrieving keys for sponsor ~zod
boot: parsing %solid pill
pace (live): configured at /home/neal/comet-412/.bin/pace
vere: binary copy succeeded
loom: mapped 2048MB
boot: protected loom
logical boot
boot: installed 661 jets
---------------- playback starting ----------------
pier: replaying events 1-14
arvo: metamorphosis
gall: not running %azimuth yet, got %poke
arvo: kernel updated
clay: rebuilding %base after kernel update
gall: installing %acme
gall: installing %azimuth
gall: installing %dbug
gall: installing %dojo
gall: installing %eth-watcher
gall: installing %hood
link [~mipped-pinlug-loshec-tastun--tabfen-bitwex-norsul-wanzod %dojo]
kiln: boot
gall: installing %herm
gall: installing %lens
gall: nstalling %ping
gall: installing %spider
gall: installing %talk-ui
gall: not running %settings-store yet, got %poke
gall: installing %docket
gall: installing %treaty
gall: installing %hark-store
gall: installing %hark-system-hook
gall: installing %settings
gall: installing %settings-store
gall: installing %storage
gall: installing %reel
gall: installing %bait
gall: installing %vitals
gall: installing %growl
docket: fetching %http glob for %talk desk
docket: fetching %http glob for %garden desk
docket: fetching %http glob for %talk desk
docket: fetching %http glob for %garden desk
docket: fetching %http glob for %webterm desk
docket: fetching %http glob for %landscape desk
gall: installing %metadata-store
gall: installing %contact-store
gall: installing %chat-store
gall: installing %graph-store
gall: installing %group-store
%group-store: on-init
gall: installing %invite-store
gall: installing %s3-store
gall: installing %chat-hook
gall: installing %chat-view
gall: installing %clock
gall: installing %contact-hook
gall: installing %contact-pull-hook
gall: installing %contact-push-hook
gall: installing %contact-view
gall: installing %dm-hook
gall: installing %graph-pull-hook
gall: installing %graph-push-hook
gall: installing %group-pull-hook
gall: installing %group-push-hook
gall: installing %group-view
gall: installing %hark-chat-hook
gall: installing %hark-graph-hook
gall: installing %hark-group-hook
gall: installing %hark-invite-hook
gall: installing %invite-hook
gall: installing %invite-view
gall: installing %launch
gall: installing %metadata-hook
gall: installing %metadata-pull-hook
gall: installing %metadata-push-hook
gall: installing %observe-hook
gall: installing %sane
gall: installing %weather
gall: not running %file-server yet, got %poke
|«play»store: on-peek on path /y/groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
%group-store: on-watch on path /groups
docket: fetching %http glob for %groups desk
gall: installing %groups
gall: installing %chat
gall: installing %contacts
%contacts: on-init
gall: installing %heap
gall: installing %diary
gall: installing %hark
gall: installing %notify
gall: installing %groups-ui
gall: installing %grouper
%contacts: on-poke with mark %noun
%contacts: on-agent on wire /migrate, %poke-ack
[%agent-giving-on-system-duct %diary %fact]
pier: (14): play: done
---------------- playback complete ----------------
vere: checking version compatibility
loom: image backup complete
lick init mkdir /home/neal/comet-412/.urb/dev
ames: live on 52253
conn: listening on /home/neal/comet-412/.urb/conn.sock
lick: %born failure;
http: web interface live on http://localhost:8081
http: loopback live on http://localhost:12322
pier (25): live
docket: fetching %http glob for %garden desk
ames: czar zod.urbit.org: ip .35.247.119.159
ames: metamorphosis
; ~zod is your neighbor
ames: czar at zod.ur
~mipped_wanzod:dojo>
```

At the 10,000' level, we can read the current boot process into a few discrete stages:

1. Runtime startup
2. Boot sequence (pill)
3. Arvo larval phase
4. Arvo main sequence
5. Userspace startup (a lot of that output results from userspace slogs)

In this lesson, we will examine each of these steps.

## Runtime Startup <a href="#runtime-startup" id="runtime-startup"></a>

To start an Urbit ship for the first time, you have to provide a ship name and the corresponding private key. It's easiest to demonstrate this with a moon using the values obtained from `|moon`. After allocating memory, the logical boot process proceeds.

The runtime spawns the king (`king.c`) and indirectly the serf (`serf.c`) processes. These will both run for the lifetime of the Urbit process.

* The serf is the Nock runtime. It tracks the current state of Arvo as a noun, updating the state by poking it with nouns. It informs the king of the new state.
  * Vere provides a standard serf, what was known formerly as the `urbit-worker` process.
  * NockVM (née Sword, née Ares) can be used as a serf in its Nock interpreter capacity, but requires I/O driver support to function this way entirely for Urbit.
* The king manages snapshots of Arvo's state and interfaces with Unix.
  * Vere is the only Urbit king currently.
  * King Haskell was an alternative king process that was dropped for maintenance reasons.

> The serf only ever talks to the king, while the king talks with both the serf and Unix.

When the runtime begins, it drops into `vere/main.c` and checks the command-line options and commands. `main()` has to decide what it needs to do (i.e. the command) and then it accordingly sets global flags. If this is a first-time boot or a restart of a pier, then `main()` starts the king with `u3_king_commence()`. (In general, `main.c` isn't very Urbit-y, it's a fairly orthodox C startup file.)

The first thing the king does is use `vere/dawn.c` to retrieve the state of Ethereum and the claimed ship's identity. If this can be verified, then the sponsor chain is retrieved and preparation for the Arvo bootstrap sequence is made.

```
boot: verifying keys
Getting sponsor
boot: retrieving galaxy table
boot: retrieving network domains
boot: retrieving keys for sponsor ~wanzod
boot: retrieving keys for sponsor ~zod
```

* See how a comet is mined in `vere/dawn.c:u3_dawn_come`.

You can see `u3v_wish` present at several places, demonstrating the Arvo `+wish` evaluation arm.

The boot sequence is set up in `vere/pier.c:u3_pier_boot()`, triggered by the king immediately after the `dawn.c` call. There are some runtime boilerplate issues to resolve, such as creating the `/.urb` folder for the event log and the loom. Snapshots are made and replays are checked, \&c.

Finally, the king hooks up the bootstrap from the supplied pill (`vere/king.c:_king_boot_ivory` → `noun/serial.c:u3s_cue_xeno`) and starts the main event loop (`uv_run`). `uv_run` is actually a loop handler from `libuv`, not a part of Urbit proper. It provides asynchronous I/O, which makes sense since every event in Urbit either comes from or results in a Unix system call.

Regarding [`libuv`](https://github.com/libuv/libuv):

> libuv's name and logo stand for "Unicorn Velociraptor", where:
>
> * U or Unicorn is a reference to universal and multi-platform.
> * V or Velociraptor is a reference to velocity and high-performance.

## Pill I <a href="#pill-i" id="pill-i"></a>

When you boot a ship, you need all the parts of the boot sequence that are not unique, as well as your private keys and up-to-date information about the PKI, and some entropy etc. The runtime provides some of this information. The pill is then the recipe for the bootstrap sequence. The bootstrap sequence is how you get to an Arvo kernel. Once you have an Arvo kernel, you can compute in the normal event timeline.

A big part of the practical complexity is obtaining identity and keys from Azimuth. You need your own keys of course, but you need the public keys of anyone you need to talk to. So you start with the galaxy table (hard-coded) and can build the sponsorship chain by construction. Then you can get the rest from an Ethereum node.

We also want to avoid booting into an invalid state.

The pill contains:

1. A list of Nock events to create an Arvo kernel.
2. A list of Arvo events to follow once the Arvo kernel has been created.
3. A list of userspace events to follow that setup.

There are three main pill types:

* An [**ivory pill**](https://github.com/urbit/urbit/blob/develop/pkg/arvo/gen/pill/ivory.hoon) is a runtime support pill compiled into the binary. It produces just the `%zuse` core for use by Vere's I/O process. (This prevents needing to redefine certain parts of the Hoon stdlib functionality in Vere.)

```hoon
.ivory/pill +ivory %base
```

* A [**brass pill**](https://github.com/urbit/urbit/blob/develop/pkg/arvo/gen/pill/brass.hoon) is a complete bootstrap sequence including the vanes being recompiled against a target `%base` desk (the first argument).

```hoon
.brass/pill +pill/brass %base
```

> A brass pill is a recipe for a complete bootstrap sequence, starting with a bootstrap Hoon compiler as a Nock formula. It compiles a Hoon compiler from source, then uses it to compile everything else in the kernel. (`~master-morzod`)

For instance, the [developer pill](https://github.com/urbit/docs.urbit.org/tree/master/blog/dev-pill/README.md) is produced as a brass pill.

* A [**solid pill**](https://github.com/urbit/urbit/blob/develop/pkg/arvo/gen/pill/solid.hoon) is a kernel developer expedient, which doesn't recompile the vanes the way a brass pill does.
* A **baby pill** is a minimalist pill, like the one you produced for [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04)'s homework. (\~wicdev-wisryt walks through the process of creating a baby pill [here](https://www.youtube.com/watch?v=fOVhCx1a-9A).)

## Bootstrapping <a href="#bootstrapping" id="bootstrapping"></a>

> Before we plug the newborn node into the network, we feed it a series of bootstrap or \`\`larval'' packets that prepare it for adult life as a packet transceiver on the public network. The larval sequence is private, solving the secret delivery problem, and can contain as much code as we like. (Whitepaper)

If Nock is a frozen function from nouns, then we can define an OS. That OS, Arvo, guarantees that the state of your ship is a pure function of the things that have happened to it. The event log is a linked list of events, operated on by what the whitepaper calls a “functional BIOS”: `[2 [0 3] [0 2]]`.

So the formula starts with the first event. What's in the first event? It performs the bootstrap of Arvo itself then loops to take the events off one at a time.

When we boot a ship, the runtime implements this directly using `u3v_boot`, which is given a list and runs the formula from a solid pill.

An `$ovum` is a pair of `wire` (routing data) and `card`. A `$card` is raw event datum, a pair of a `term` tag and an arbitrary noun.

### The First Five Events <a href="#the-first-five-events" id="the-first-five-events"></a>

The `+eden` core supplies the first five events to create the event series that will result in Arvo and its lifecycle function.

* Event One, `+aeon`. Start the event loop.
* Event Two, `+boot`. Bootstrap an `arvo` kernel from source.
* Event Three, `+fate`. Produce the Hoon bootstrap compiler.
* Event Four, `+hoon`. Produce the compiler source.
* Event Five, `+arvo`. Produce the kernel source.

#### Event One: `+aeon`

```hoon
++  aeon
  ^-  *
  =>  *log=[boot=* tale=*]
  !=
  =+  [arvo epic]=.*(tale.log boot.log)
  |-  ^-  *
  ?@  epic  arvo
  %=  $
    epic  +.epic
    arvo  .*([arvo -.epic] [%9 2 %10 [6 %0 3] %0 2])
  ==
```

`+aeon` is the first function run on any ship. The gate on the outer edge of Arvo is retrieved, then all of the events are run in a list processing loop, invoking Arvo in each one. This is Hoon code to just produce raw Nock using `!=` zaptis. The `=>` tisgar asserts that we expect the subject (outside the Nock) to look like the first thing in the event log and the rest of the log. This is the first event of the log. The subject that it expects is the rest of the log (which will be evaluated using `[2 [0 3] [0 2]]`.) So `boot.log` is Event Two and `tale.log` is Events Three through Infinity.

This produces `arvo`, the stateless kernel, and `epic`, the rest of the log. The formula in Event Two can take as many events as it needs from the sequence to construct `arvo`, then the incremental process can continue. If `epic` is an atom `~` sig (the null terminator), then `arvo` is ready. Arvo is the result of calling `arvo` on `-.epic`, the next event.

#### Event Two: `+boot`

```hoon
++  boot
      ^-  *
      =>  *log=[fate=* hoon=@ arvo=@ epic=*]
      !=
      ::
      ::  activate the compiler gate. the product of this formula
      ::  is smaller than the formula. so you might think we should
      ::  save the gate itself rather than the formula producing it.
      ::  but we have to run the formula at runtime, to register jets.
      ::
      ::  as always, we have to use raw nock as we have no type.
      ::  the gate is in fact ++ride.
      ::
      ~>  %slog.[0 leaf+"1-b"]
      =/  compiler-gate  .*(0 fate.log)
      ::
      ::  compile the compiler source, producing (pair span nock).
      ::  the compiler ignores its input so we use a trivial type.
      ::
      ~>  %slog.[0 leaf+"1-c (compiling compiler, wait a few minutes)"]
      =/  compiler-tool
        ~>  %bout
        .*([compiler-gate noun/hoon.log] [%9 2 %10 [6 %0 3] %0 2])
      ::
      ::  switch to the second-generation compiler. we want to be
      ::  able to generate matching reflection nouns even if the
      ::  language changes -- the first-generation formula will
      ::  generate last-generation spans for `!>`, etc.
      ::
      ~>  %slog.[0 leaf+"1-d"]
      =.  compiler-gate  ~>(%bout .*(0 +.compiler-tool))
      ::
      ::  get the span (type) of the kernel core, which is the context
      ::  of the compiler gate. we just compiled the compiler,
      ::  so we know the span (type) of the compiler gate. its
      ::  context is at tree address `+>` (ie, `+7` or Lisp `cddr`).
      ::  we use the compiler again to infer this trivial program.
      ::
      ~>  %slog.[0 leaf+"1-e"]
      =/  kernel-span
        ~>  %bout
        -:.*([compiler-gate -.compiler-tool '+>'] [%9 2 %10 [6 %0 3] %0 2])
      ::
      ::  compile the arvo source against the kernel core.
      ::
      ~>  %slog.[0 leaf+"1-f"]
      =/  kernel-tool
        ~>  %bout
        .*([compiler-gate kernel-span arvo.log] [%9 2 %10 [6 %0 3] %0 2])
      ::
      ::  create the arvo kernel, whose subject is the kernel core.
      ::
      ~>  %slog.[0 leaf+"1-g"]
      ~>  %bout
      [.*(+>.compiler-gate +.kernel-tool) epic.log]
    --
```

The next event noun bootstraps a kernel from its source (`arvo`).

#### Event Three: `+fate`

The next noun (event) is the Hoon bootstrap compiler as source.

Details of this process are supplied in [`/lib/pill`](https://github.com/urbit/urbit/blob/develop/pkg/base-dev/lib/pill.hoon).

#### Event Four: `+hoon`

Next we produce the compiler source.

Details of this process are supplied in [`/lib/pill`](https://github.com/urbit/urbit/blob/develop/pkg/base-dev/lib/pill.hoon).

#### Event Five: `+arvo`

Then we produce the kernel source.

Details of this process are supplied in [`/lib/pill`](https://github.com/urbit/urbit/blob/develop/pkg/base-dev/lib/pill.hoon).

#### `/lib/pill`

* To see how the events are created, let's take a look at the `+brass` arm in `/lib/pill`. This uses vase mode (see e.g. the `swat` which is a delayed `slap` in a trap) to produce the cores and events.

### Lifecycle <a href="#lifecycle" id="lifecycle"></a>

Once you have all of this, you have completed the lifecycle evaluation of the bootstrap sequence and can run the rest of the event log.

Arvo enters the larval phase during the boot sequence and during certain OTAs. An OTA can just change `%lull`/`%zuse`/a vane, which doesn't touch Arvo; or it can changes `/sys/hoon` or `/sys/arvo`, in which case it needs to handle an Arvo upgrade. From the big picture, we build the new Arvo, compile Hoon and Arvo, gather all persistent and ephemeral state plus any new upgrade state, and hand that to the new Arvo (new world).

The larval core is the outermost core in `/sys/arvo`. When you first bootstrap, the core that the runtime talks to is the larval phase. The larval core is designed to accumulate preconditions and then metamorphose into the adult Arvo. It needs the current time, entropy, the identity, and the standard library.

> Symmetry breaking---the event that defines the identity of the com-pu-ter---is exempt from this requirement. Once identity is established, it can't be updated. If you want a new identity, create a new instance. (Whitepaper)

The larval stage was introduced into the boot and upgrade sequence as a way to solve a practical problem in self-reference. If the ship is not known, then what should happen? If you bunt, then everything is \~zod. If you have a `(unit @p)`, then all code using `our` becomes cumbersome. So while identity is injected early into the kernel, it hasn't happened yet. Specifically, it takes place when it acquires identity and entropy and sheds the larval core. This is called “breaking symmetry” because prior to this point every Urbit is identical. (This was not always true, as Joe Bryan notes [in this talk](https://youtu.be/V7XPSvVs5TQ?t=681) at 10:21ff.) The larval stage performs the following steps in order:

* The standard library, `zuse`, is installed.
* Entropy is added
* Identity is added
* Metamorphosis into the next stage of Arvo

```hoon
|%
++  load                                                ::   +4
  |=  hir=heir
  ?:  ?=(%grub -.hir)
    ~>(%mean.'arvo: larval reboot' !!)    :: XX support
  (^load hir)
::
++  peek  _~                                            ::  +22
++  poke                                                ::  +23
  |=  [now=@da ovo=ovum]
  ^-  ^
  ~|  poke/p.card.ovo
  =/  wip
    ~>  %mean.'arvo: bad wisp'
    ;;(wisp card.ovo)
  ::
  =.  ..poke
    ?-    -.wip
      %verb  ..poke(lac ?~(p.wip !lac u.p.wip))
      %wack  ..poke(eny `p.wip)
      %what  ..poke(gub (what gub p.wip))
      %whom  ..poke(who ~|(%whom-once ?>(?=(~ who) `p.wip)))
    ::
      %wyrd  ?.  (sane:wyrd kel.p.wip)
                   ~>(%mean.'wyrd: insane' !!)
             %-  %+  need:wyrd  kel.p.wip
                 ^-  wynn
                 :*  hoon/hoon-version
                     arvo/arvo
                     ?~  lul  ~
                     :-  lull/;;(@ud q:(slap $:u.lul limb/%lull))
                     ?~  zus  ~
                     [zuse/;;(@ud q:(slap $:u.zus limb/%zuse)) ~]
                 ==
             ..poke(ver `p.wip)
    ==
  ::
  ::  upgrade once we've accumulated necessary state
  ::
  ?~  hir=(molt now gub)
    [~ ..poke]
  ~>  %slog.[0 leaf+"arvo: metamorphosis"]
  (load u.hir)
::
++  wish                                                ::  +10
  |=  txt=*
  q:(slap ?~(zus pit $:u.zus) (ream ;;(@t txt)))
--
```

* `+load`/`+4` passes through to the new world Arvo, because the larval stage is a trivial core wrapped around the mature core.
* `+wish`/`+10` evaluates as normal.
* `+peek`/`+22` produces the type of `~`; i.e., every scry blocks.
* `+poke`/`+23` handles the upgrade-related events. Larval events are actually quite simple:

```hoon
+$  wisp
  $%  $>(?(%verb %what) waif)    :: update from files (event from anywhere)
      $>(?(%wack %wyrd) wasp)    :: iterate entropy (event from runtime)
      [%whom p=ship]             :: acquire identity (frozen after boot)
  ==
```

#### `+le`

`+le` is the Arvo event-loop engine. It provides an `+abet`-pattern-driven core for building worklists.

* `+peek` handles any reads into the larval stage block.
* `+poke` is how you compute an event, as normal.
* `+load` is how Arvo transfers state to a future version of itself.
* `+wish` is of course the `urbit eval` wrapper and generates no events.

See also:

* `+what`, the update engine, which handles a kernel update and continuation from the worklist.

#### Metamorphosis

Metamorphosis means producing the parent core, the mature Arvo. This takes place through `+load`, which hands itself forward.

The larval stage upgrade mechanism is just a pass-through or it crashes.

`+molt` takes care of the process in a practical way by gathering the known system information and packaging it for `+load`.

When bootstrapping is done, the runtime strips off the gate to access the real Arvo core. (This is something of a manual trick.) This is how the structural interface Arvo is accessed, through the hardcoded arm addresses.

* [\~master-morzod, “Annotation on the Boot Process”](https://groups.google.com/a/urbit.org/g/dev/c/ESrqJb3Ol54/m/bns0S1QkBAAJ)
* [\~master-morzod, \~lagrev-nocfep “Dev Chat: Joe Bryan on the Boot Sequence”](https://www.youtube.com/watch?v=V7XPSvVs5TQ)

### Main Sequence (Mature Arvo) <a href="#main-sequence-mature-arvo" id="main-sequence-mature-arvo"></a>

Once the outer larval core has been shed, the system is back in the Arvo main sequence ([*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04)).

### Vanes & Userspace <a href="#vanes-userspace" id="vanes-userspace"></a>

`%lull` is compiled against `..part`, the first half of Arvo.

## Shutdown <a href="#shutdown" id="shutdown"></a>

In a general sense, Urbit is only aware of the world while it lives. But of course on a real machine, the ship will be shut down, migrated, and execute on different runtimes.

* What happens when you run `|exit`? Trace out that process and back into the king for graceful shutdown.


# 6. Vere I: u3 and the Serf

Core Academy lesson on Vere runtime architecture covering the u3 noun system, king/serf processes, the Mars/Urth split, event log management, jet dashboard, and the Arvo/Vere interface.

*This lesson covers the Arvo/Vere interface, including Arvo's `+peek` and `+poke` arms. We also examine the Mars/Urth split, the event log, snapshots, and the jet dashboard.*

*We need some C background to fluently read the runtime source. If you are not familiar with C, here are some good resources:*

* [Beginner C resource list](https://stackoverflow.com/questions/562303/the-definitive-c-book-guide-and-list)
* [Brian Kernighan & Dennis Richie, *The C Programming Language*](https://en.wikipedia.org/wiki/The_C_Programming_Language)
* [Ben Klemens, *21st Century C: Tips from the New School*](https://www.amazon.com/dp/1449327141)
* [Lawrence Angrave, \_System Programming](https://github.com/angrave/SystemProgramming/wiki)

## Runtime Structure and Responsibilities <a href="#runtime-structure-and-responsibilities" id="runtime-structure-and-responsibilities"></a>

If the vision of Urbit is to implement `[2 [0 3] 0 2]` as a frozen lifecycle function, then it needs some scaffolding on any real system. Real computers have memories, chipset architectures, operating system conventions, and other affordances (and limitations). Conventionally, an operating system takes explicit care of such niceties, which is one reason why calling Urbit an “OS” has been controversial. The runtime interpreter is designed to take Nock as a specification and produce a practical computer of it.

Today, there are two primary Nock executable runtimes: Vere and NockVM (née Sword, née Ares). (Jaque, the JVM experiment, and King Haskell have fallen by the wayside.)

* Vere is written in C and is the standard runtime for Arvo.
* NockVM is written in Rust and aims to solve some deep theoretical challenges to producing efficient Nock on a contemporary chipset. NockVM is under development by [Zorp](https://zorp.io) and formerly with contributions from [Tlon Corporation](https://tlon.io), and the [Urbit Foundation](https://urbit.org).

We will take Vere as the normative runtime for Core Academy.

As we mentioned last time in the boot sequence lesson, the runtime spawns the king (`king.c`) and indirectly the serf (`serf.c`) processes. These both run for the lifetime of the Urbit process.

There are two competing frames for how to structure the Urbit process: king/serf and urth/mars.

King v. serf separates the Nock and Arvo material from the I/O and event log material. It has the advantage that (per the whitepaper), “The serf only ever talks to the king, while the king talks with both the serf and Unix.”

The king process is in charge of:

* IPC
* Event log
* Unix effects including I/O
* Stateless Nock interpreter

The serf process is the Nock runtime and bears responsible for:

* Nock virtual machine (tracking current state of Arvo as a noun and `+poke`ing it with nouns)
* Bytecode interpretation
* Jet dashboard
* Snapshotting
* Noun allocation for Arvo

The Mars/Urth split reframes the worker process so that it includes the event log with the current serf responsibility (“Mars”), thus enabling online event log management and truncation.

### The Structure of Vere’s Source <a href="#the-structure-of-veres-source" id="the-structure-of-veres-source"></a>

Vere is provided in the [`urbit/vere`](https://github.com/urbit/vere) repo. It is built from the `pkg/` directory and contains the following top-level folders:

```
.
├── c3
├── ent
├── noun
├── ur
├── urcrypt
└── vere
```

* `/c3` contains the types and definitions to enable [the `c3` logical system](https://github.com/urbit/docs.urbit.org/tree/master/reference/runtime/c/README.md).

  `c3` is the set of C conventions which Vere enforces. These include well-specified integer types, tooling for loobeans (instead of booleans), and motes (`#define`s for short Urbit words). “The C3 style uses Hoon style TLV variable names, with a quasi Hungarian syntax.” There are no Urbit-specific requirements for C3, which could otherwise just be a general-purpose C discipline.

  Like aura bitwidth markers, C documents programmer intent but does not generally enforce it. Most of the parts of `c3` are simply lapidary terms for C99 types.

  * Scan the files in `/c3`.
* `/ent` provides entropy for the runtime. Entropy is derived from [`/dev/urandom`](https://en.wikipedia.org/wiki//dev/random), which is a special file that provides pseudorandom numbers derived from system noise. `/dev/urandom` produces [machine randomness as close to true randomness as possible](https://www.2uo.de/myths-about-urandom/#before-linux-48), including seeds like network latency and keystroke latency to seed the cryptographically secure pseudo-random number generator (CSPRNG).
* `/noun` is the gorilla, containing `u3` (the noun library) and the jets. We'll go into it in detail with the system architecture in a moment in Section `u3`.
* `/ur`, is like `/ent` a single-purpose library, in this case for bitstreams and serialization.
* `/urcrypt` is a C library to standardize cryptographic calls across a number of libraries.

  This library is a dependency for both Vere and NockVM, and is in the process of being moved into a standalone repo.
* `/vere` contains the runtime architecture itself, the king and the serf and related tooling, as independent from `u3`.

| file           | purpose                                           |
| -------------- | ------------------------------------------------- |
| `auto.c`       | I/O drivers                                       |
| `benchmarks.c` | performance tests                                 |
| `dawn.c`       | key validation for bootstrapping                  |
| `disk.c`       | database reads and writes for event log           |
| `foil.c`       | file synching                                     |
| `king.c`       | main runtime loop                                 |
| `lord.c`       | manage IPC between king and serf                  |
| `main.c`       | setup and entrypoint for runtime execution        |
| `mars.c`       | Mars event log replay (see Mars/Urth split above) |
| `newt.c`       | noun blob messages                                |
| `pier.c`       | manage pier (files on host OS disk)               |
| `save.c`       | save events to pier                               |
| `serf.c`       | the serf itself                                   |
| `time.c`       | Unix/Urbit time operations                        |
| `vere.h`       | shared Vere-specific `struct`s                    |
| `ward.c`       | lifecycle management for structures               |

#### `u3`

**Nouns**

A noun is either an atom or a cell. However, we have to decide what this implementation looks like in a language like C, that prefers arrays and pointers. `u3` is the noun library, which features Urbit-specific memory operations, tracing and profiling tools, and so forth.

A `u3_noun` is a 32-bit `c3_w` = `uint32_t`. The first bits indicate what kind of value the noun is and thus how to approach it:

| Bit 31 | Bit 30 | Meaning               |
| ------ | ------ | --------------------- |
| `1`    | `1`    | Indirect cell (`pom`) |
| `1`    | `0`    | Indirect atom (`pug`) |
| `0`    | ·—     | Direct atom (`cat`)   |

An indirect noun is a `dog`. For indirect nouns, bits 29–0 are a word pointer into the loom. In addition, `0xffff.ffff` is `u3_none`, which is “not a noun”.

A common pattern is to extract values from a noun into C-typed values, carry out the manipulation, and then wrap them back into the noun. Furthermore, the value from an arbitrary atom may in fact be a bignum, and so [GMP](https://en.wikipedia.org/wiki/GNU_Multiple_Precision_Arithmetic_Library) is used to manage these values.

* Examine `/noun/jets/a/add.c`, in particular `u3qa_add`.

One of the painful parts of working with `u3` is the reference counting system. [Reference counting](https://en.wikipedia.org/wiki/Reference_counting) is an expedient to handle tracking the number of pointers to an object in memory so that the memory can be freed at the appropriate time. Since C doesn't provide reference counting support in the language, we must manually track these and free the value only when the refcount goes to zero. The relevant functions are `u3k` to gain a refcount and `u3z` to lose one.

There are also two different protocols for reference counting, used by different parts of the system:

* `transfer` semantics relinquishes a refcount of any sent values. Most functions behave this way, which means that you don't have to think about de-allocating values if they've been sent elsewhere.
* `retain` semantics hold onto the refcount even if the value is sent elsewhere. The functions which use `retain` semantics tend to inspect or query nouns rather than make or modify nouns.

> The `u3` convention is that, unless otherwise specified, **all functions have transfer semantics** - with the exception of the prefixes: `u3r`, `u3x`, `u3z`, `u3q` and `u3w`. Also, within jet directories `a` through `f` (but not `g`), internal functions retain (for historical reasons).

* Compare `u3ka_add` and `u3qa_add`.

`u3` is designed to make some guarantees for the programmer. It's not Urbit itself, but it's designed to be an implementation platform for Urbit. Thus:

* Every event is logged internally before it enters `u3`.
* A permanent state noun maintains a single reference.
* Any event can be aborted without damaging the permanent state (“solid state”).
* We snapshot the permanent state and can prune logs.

We will discuss the specifics of the memory model next week in `ca06` when we discuss the loom and the road model.

* [“Land of Nouns”](https://github.com/urbit/docs.urbit.org/tree/master/reference/runtime/nouns/README.md); note particularly the section `u3: reference protocols`, labeled ``THIS IS THE MOST CRITICAL SECTION IN THE `u3` DOCUMENTATION.`` Read that if nothing else.

**Library**

The contents of `/noun` constitute the `u3` noun library. Functions are organized by file and prefix into certain namespaces by operation. Because `u3` is a library, we can't cleanly separate it into serf/king components, although certain modules do have close identification with one or the other.

| prefix     | purpose                     | `.h`                                                                             | `.c`                                                                             |
| ---------- | --------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `u3a_`     | allocation                  | [`allocate.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/allocate.h)   | [`allocate.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/allocate.c)   |
| `u3e_`     | persistence                 | [`events.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/events.h)       | [`events.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/events.c)       |
| `u3h_`     | hashtables                  | [`hashtable.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/hashtable.h) | [`hashtable.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/hashtable.c) |
| `u3i_`     | noun construction           | [`imprison.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/imprison.h)   | [`imprison.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/imprison.c)   |
| `u3j_`     | jet control                 | [`jets.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/jets.h)           | [`jets.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/jets.c)           |
| `u3l_`     | logging                     | [`log.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/log.h)             | [`log.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/log.c)             |
| `u3m_`     | system management           | [`manage.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/manage.h)       | [`manage.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/manage.c)       |
| `u3n_`     | nock computation            | [`nock.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/nock.h)           | [`nock.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/nock.c)           |
| `u3o_`     | command-line options        | [`options.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/options.h)     | [`options.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/options.c)     |
| `u3r_`     | noun access (error returns) | [`retrieve.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/retrieve.h)   | [`retrieve.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/retrieve.c)   |
| `u3s_`     | noun serialization          | [`serial.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/serial.h)       | [`serial.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/serial.c)       |
| `u3t_`     | profiling                   | [`trace.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/trace.h)         | [`trace.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/trace.c)         |
| `u3u_`     | urth (memory management)    | [`urth.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/urth.h)           | [`urth.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/urth.c)           |
| `u3v_`     | arvo                        | [`vortex.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/vortex.h)       | [`vortex.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/vortex.c)       |
| `u3x_`     | noun access (error crashes) | [`xtract.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/xtract.h)       | [`xtract.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/xtract.c)       |
| `u3z_`     | memoization                 | [`zave.h`](https://github.com/urbit/vere/blob/develop/pkg/noun/zave.h)           | [`zave.c`](https://github.com/urbit/vere/blob/develop/pkg/noun/zave.c)           |
| `u3k[a-g]` | jets (transfer, C args)     | `jets/k.h`                                                                       | `jets/[a-g]/*.c`                                                                 |
| `u3q[a-g]` | jets (retain, C args)       | `jets/q.h`                                                                       | `jets/[a-g]/*.c`                                                                 |
| `u3w[a-g]` | jets (retain, nock core)    | `jets/w.h`                                                                       | `jets/[a-g]/*.c`                                                                 |

* `u3a` defines memory allocation functions. These are used throughout, but we'll discuss it a bit more when we talk about the king. You will quickly run into reference counting features, like `u3k` (`u3a_gain()`) to gain a refcount and `u3z` (`u3a_lose()`) to lose one.
* `u3e` manages the loom.
* `u3h` provides fast custom hashing for the runtime.
* `u3i` puts a value (expected to be a `c3` type) into a noun. (Look at this one now.)
* `u3l` supports logging.
* `u3m` manages the system: boots `u3`, makes a pier, handles crashes, etc.
* `u3n` implements the Nock bytecode interpreter.
* `u3o` parses the manifold command-line options of Urbit and writes them into globals.
* `u3r` extracts a value from a noun, with a `u3_weak` on failure. (Look at this one now.)
* `u3s` implements noun serialization (`+jam` and `+cue`).
* `u3t` provides tracing for crashes.
* `u3u` offers memory management tooling (deduplication and memory mapping).
* `u3v` supports Arvo interaction.
* `u3x` extracts a value from a noun., with a crash on failure
* `u3z` supports `~+` siglus rune memoization.

If you work much in Vere, you will get used to seeing these. There are basically two broad categories of functions: single-use functions (like starting a pier, `u3m_pier`) and utility functions (like writing a value to a noun, `u3i_word`).

* Return to `/noun/jets/a/add.c` and look at `u3wa_add` and `u3ka_add`.
* [“C Runtime System”](https://github.com/urbit/docs.urbit.org/tree/master/reference/runtime/runtime/README.md)
* [“Land of Nouns”](https://github.com/urbit/docs.urbit.org/tree/master/reference/runtime/nouns/README.md)

## The Serf <a href="#the-serf" id="the-serf"></a>

The serf process is the Nock runtime and bears responsible for:

* Nock virtual machine (tracking current state of Arvo as a noun and `+poke`ing it with nouns)
* Bytecode interpretation
* Jet dashboard

If you examine `/vere/serf.c`, you can get a feel for how it is organized. See e.g. `u3_serf_work` and callees.

### Arvo Noun Management <a href="#arvo-noun-management" id="arvo-noun-management"></a>

* `/vere/vortex.c`, e.g. `u3v_peek`, `u3v_wish`, and `u3v_poke_sure`.

### Nock Bytecode Interpreter (`u3n`) <a href="#nock-bytecode-interpreter-u3n" id="nock-bytecode-interpreter-u3n"></a>

* `/noun/nock.c`, e.g. `u3n_nock_on`, `u3n_slam_on` (calling convention for gates).

The end result of the Hoon compilation process is Nock code as a noun. This noun is evaluated by the runtime, but it is not actually directly run as such. Instead, the runtime builds an efficient bytecode stream and executes that instead to complete the calculation.

The Nock bytecode for any expression can be obtained using the `%xray` raw hint.

```hoon
> ~>  %xray  =+(2 [- +(-)])
{[litb 2] snol head swap head bump ault halt}
[2 3]

> ~>  %xray  =+(2 [(add - -) +(-)])
{[litb 2] snol [fask 4095] [kicb 1] snoc head swap [fabk 6] swap [fabk 6] auto musm [kicb 0] swap head bump ault halt}
[4 3]
```

The Nock bytecode is defined in the `OPCODES` macro in `/noun/nock.c` and evaluated by `_n_burn` in that same folder. The `OPCODES` `#define` uses the [X macro](https://en.wikipedia.org/wiki/X_Macro), which is a bit of C deep lore.

As a consequence of the architecture of Vere today, we see a lot of expensive call overhead. For instance, when you wrap an `%xray` hint around a core, you don't get the core itself—instead you get the formula that invokes the code.

```hoon
> ~>  %xray  (met 3 (jam .))
{[fask 1023] [kicb 3] snol head swap tail [lilb 3] swap tail [fask 1023] [kicb 2] snol head swap tail musm [kicb 1] auto musm [ticb 0] halt}
984.339
```

Since many things are computed in virtual Nock, `+mock`, we have bail/trace/bounded computation at the price of slow virtualization.

One objective of NockVM, subject knowledge analysis, is to improve on Nock bytecode generation. This is being implemented into Vere as well.

### Jet Dashboard (`u3j`) <a href="#jet-dashboard-u3j" id="jet-dashboard-u3j"></a>

As we summarized when first introducing jets in `ca00`, the runtime manages jets, including re-running them when playing back the event log history.

> The jet dashboard is the system in the runtime that registers, validates, and runs jets: specific pieces of Nock code reimplemented in C for performance.

The jet dashboard maintains three jet state systems:

1. `cold` state results from the logical execution history of the pier and consists of nouns. `cold` jet state registers jets as they are found. `cold` state ignore restarts.
2. `hot` state is the global jet dashboard and describes the actual set of jets loaded into the pier for the current running process. Calls to `hot` state result from Nock Nine invocations of a core and an axis. `hot` state is thus tied to process restart.
3. `warm` lists dependencies between `cold` and `hot` state. `warm` state can be cleared at any time and is cleared on restart.

The jet dashboard (`u3j`, `/noun/jets.c`) will not be explored in detail in Core Academy, but we do want to look at a couple of actual jets.

* [“API Overview by Prefix: `u3j` jets”](https://github.com/urbit/docs.urbit.org/tree/master/reference/runtime/api/README.md#u3j-jets)
* [\~timluc-miptev, “Jets in the Urbit Runtime”](https://github.com/timlucmiptev/docs-runtime/blob/master/jets1_current.md)

### Jets <a href="#jets" id="jets"></a>

* Examine `/noun/jets/b/lent.c`, `/noun/jets/b/turn.c`, `/noun/jets/c/turn.c`, `/noun/jets/e/rs.c`, `/noun/jets/e/slaw.c`.

Many Urbit contributors may find jet composition to be their first serious encounter with the runtime. On the bright side, jetting is a fairly constrained and well-understood space. However, it has a complex interface for unpacking calls and nouns, including reference counting requirements.

* `u3w` functions are the main entry point (as identified in `/noun/tree.c`). These unpack and sanity-check the sample, then call either `u3q` or `u3k` variants of the jet. The unpacking axes are hard-coded in `/noun/xtract.h`.
* By convention, `u3q` and `u3w` functions have `transfer` semantics.
* `u3k` functions have `retain` semantics, so they are responsible to `u3z` free their values after the computation completes.
* `u3_none` (`0x7fff.ffff`) is NOT the same as `u3_nul`. A jet that returns `u3_none` punts the value back to the Hoon/Nock version.
* [“Writing Jets”](https://github.com/urbit/docs.urbit.org/tree/master/reference/runtime/jetting/README.md)
* [\~timluc-miptev, “Jets in the Urbit Runtime”](https://github.com/timlucmiptev/docs-runtime/blob/master/jets1_current.md)

### Snapshotting <a href="#snapshotting" id="snapshotting"></a>

We'll cover snapshotting in the next lesson, `ca06`.


# 7. Vere II: The Loom

Core Academy lesson on Vere's memory management covering the loom allocator, noun memory layout, reference counting, structural sharing, the road system, metacircularity, paging, and snapshots.

*In this lesson we explain the memory allocator: the loom, noun memory layout, reference counting, structural sharing, raods, and metacircularity. We'll also see how paging and snapshots work, including page faults, memory protection, the guard page, and demand paging.*

## The Loom <a href="#the-loom" id="the-loom"></a>

Vere's main memory model is called the *loom*. (Presumably this is from the roads shuttling back and forth, and perhaps mixing a metaphor.) A contiguous block of memory, formerly 2GB but now specified by the runtime flag, is allocated for the loom. This is the noun arena, and to work on it we need to use special `u3`-specific allocators (`u3a`).

One standard contemporary memory model afforded by an operating system has a [heap](https://en.wikipedia.org/wiki/Memory_management#HEAP) for manual dynamic memory allocation (C `malloc()`) and a [stack](https://en.wikipedia.org/wiki/Stack-based_memory_allocation) for local variable data (last-in-first-out, C `alloca()` but also implicit). The heap grows from the bottom up and the stack from the top down. (The alternative model today is to use SunOS-style `mmap()` to allow virtual memory paging anywhere in memory.)

```
    0           brk                                          ffff
    |   heap     |                                    stack    |
    |------------#################################+++++++++++++|
    |                                             |            |
    0                                             sp         ffff
```

* `brk` is the [`brk()` System V call](https://utcc.utoronto.ca/~cks/space/blog/unix/SbrkVersusMmap), which marked the limit of the heap arena.
* `sp` is the stack pointer.

`u3` differs from this model in one particular: by permitting the heap and stack to point either way, we can efficiently nest pairs of stack and heap. We call such a pair (and their free memory arena) a *road*. The outermost road is the *surface road*, and inner roads are created in alternating directions when dependent calculations are embarked upon.

When a new inner road is created, it switches direction from the outer road. This puts its heap up against the outer road's stack, and its stack up against the outer road's heap. But when the road is terminated, its stack is freed while its heap becomes part of the outer road's stack.

A conventional heap-low-stack-high road is a north road:

```
    0           rut   hat                                    ffff
    |            |     |                                       |
    |~~~~~~~~~~~~-------##########################+++++++$~~~~~|
    |                                             |      |     |
    0                                            cap    mat  ffff
```

while a reversed heap-high-stack-low road is a south road:

```
    0           mat   cap                                    ffff
    |            |     |                                       |
    |~~~~~~~~~~~~$++++++##########################--------~~~~~|
    |                                             |      |     |
    0                                            hat    rut  ffff
```

* `cap` is the top of the stack.
* `mat` is the bottom of the stack (`ffff` in the example surface road).
* `rut` is the bottom of the heap arena (not `0` because of immutable storage).
* `hat` is the top of the heap arena.
* `~` is deep storage (immutable).
* `-` is durable storage (heap).
* `+` is temporary storage (stack).
* `$` is the allocation frame.
* `#` is free memory.

The motivation for the road model is that you need not update refcounts in senior memory. This diminishes the downsides of reference counting. (The upside of refcounting is deterministic finalization: automatic memory management comes from the properties of the computation itself not from external preemptive events. Refcounting gives eager determinism.)

When we need to process an event or perform any kind of complicated calculation, we process it using an inner road. Because the roads alternate direction, any data from an inner heap that needs to be preserved must be copied back out. The advantages, tho, are:

1. The surface road is left read-only by `u3` and thus clean. Thus when snapshotting, clean pages are kept clean.
2. An inner calculation can be aborted without affecting the surface.
3. The surface is not fragmented because the inner results are copied in when necessary.

Vere ends up in operation with nested roads during a computation:

```
    0                                                        ffff
    |-------                                       ++++++++++++|   surface road
    |~~~~~~~$+++++++++                    ---------~~~~~~~~~~~~|   inner road 1
    |~~~~~~~~~~~~~~~~~---------     +++++$~~~~~~~~~~~~~~~~~~~~~|   inner road 2
    |                          ##&##                           |   free memory

```

The Vere interpreter runs in a road, and you can check if you're on the surface road or in an inner road. (Most of the time you should just assume you're on an inner road.) This distinction matters, such as `c3_assert()`, which produces an exception with stack trace on an inner road, but kills the process on an outer road.

The current road is `u3R`, a global. Within Arvo, a new road is currently begun in the following cases:

* every event
* every read from the namespace (Arvo scry gate into vane doesn't, userspace `.^` dotket does)
* every call to `+mink` (and `+mock`, etc.)

Any work that will be done where you only want to keep one portion of it is a good candidate for a road.

The loom is organized into pages, each 16 KB in size. There is a guard page `&` in the middle of free memory to make sure that the stack and heap do not overwrite each other. The guard page is adjusted in `u3e_ward()` when necessary:

> When a fault is detected in the guard page, the guard page is recentered in the free space of the current road. if the guard page cannot be recentered, then memory exhaustion has occurred.

There is a hope to add both a raw hint to suggest to the system to use a new road and a bump-allocation mode to permit turning off refcounting on inner roads when practical. (Cf. [#6805](https://github.com/urbit/urbit/issues/6805#issuecomment-1754208392).)

#### `u3a` Allocator

To work with `u3` memory, use the `u3a` memory allocation functions:

* `u3a_malloc()`
* `u3a_free()`
* `u3a_realloc()`

You should never call `malloc()` in the loom (but can, of course, in the Vere layer above `u3`).

> Of course, we don't always know how large our atom will be. Therefore, the standard way of building large atoms is to allocate a block of raw space with `u3i_slab_init()`, then chop off the end with `u3i_slab_malt()` (which does the measuring itself) or `u3i_slab_mint()` in case you've measured it yourself.

Keep in mind that atoms do not retain leading zeros.

The reference counters introduced last time, `u3a_gain()`=`u3k()` and `u3a_lose()`=`u3z()`, are also part of `u3a`. However, other than these you typically use `u3a` indirectly through `u3i` and `u3r`/`u3x`.

{% hint style="warning" %}

### Note on allocator documentation

As of the [Vere 4.0 release](https://github.com/urbit/vere/releases/tag/vere-v4.0), the allocator has been completely rewritten to reduce memory usage and decreases snapshot size. Some of the finer details of it's documentation here may thus be out of date.
{% endhint %}

Some details of the allocator [are in flux right now](https://github.com/urbit/urbit/issues/6805#issuecomment-1754208392): “As an experiment, \[\~master-morzod has] rewritten the serf to a) stop allocating events, effects, and IPC messages on the home road, and b) keep the Arvo kernel on an inner road for as long as possible (i.e. until we need to save/pack/meld/\&c.).”

* [“Land of Nouns”, section “`u3`: the road model”](/build-on-urbit/runtime/nouns#u3-the-road-model)
* [“API overview by prefix”, section “`u3a`: allocation functions”](/build-on-urbit/runtime/api#u3a-allocation-functions)

## The King (Urth) <a href="#the-king-urth" id="the-king-urth"></a>

The king process is in charge of:

* IPC
* Event log
* Unix effects including I/O
* Stateless Nock interpreter (the “ghost ship” ivory pill material from `ca05`)

(We discussed the serf/Mars in `ca05`.)

### Event Log & Snapshotting <a href="#event-log-snapshotting" id="event-log-snapshotting"></a>

The event log is the ordered list of all of the Arvo events (completed moves) resulting in the present state of Arvo. Events are handled at two levels: the event log, which is written consistently at each event, and the snapshot of the loom, which allows rapid recovery of the current state.

> In practice, event logs become large and unwieldy over time. Periodically a snapshot of the permanent state is taken, so the entire event log needn't be replayed on reboot. You're still able to rebuild your state down to the last keystroke. This is due to the practice of persistence.\
> Persistence, in the context of storing data in a computer system, means that data is stored in a non-volatile manner and that input must be recorded before the output result is performed. Thus, every event must be written to disk - or must be *persisted* - before the event effects actually take place.

The snapshot of the loom allows the last few events from the event log to be replayed to recover the present state.

In fact, with current usage patterns (\~2023.10.12), there's a problem:

> Every few minutes, the runtime applies a patch to its on-disk snapshot. This pauses the process, so the size is important. In particular, for many large ships (\~nibset-napwyn, \~wicdev-wisryt, \~natnex-ronret), this is around 600MB. If the maximum acceptable pause is 1 second, this requires a disk which can handle 600MB/s of throughput (or maybe twice that, because it writes the patchfile, then applies it?), which is extremely high. ([#6805](https://github.com/urbit/urbit/issues/6805))

#### `noun/error.h`

Before we look at the particulars of the event log and snapshotting code, let's take a brief detour to see how assertion errors are handled.

```c
# define u3_assert(x)                       \
    do {                                    \
      if (!(x)) {                           \
        fflush(stderr);                     \
        fprintf(stderr, "\rAssertion '%s' " \
                "failed in %s:%d\r\n",      \
                #x, __FILE__, __LINE__);    \
        u3m_bail(c3__oops);                 \
        abort();                            \
      }                                     \
    } while(0)
```

An error here then triggers into `u3m_bail()`, the primary crash handler.

* Review `u3m_bail` in `noun/manage.c`.

#### Event Log

What is an event on disk? Urbit maintains an [LMDB](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database) transactional database for key–value pairs, with a `META` table for metadata and an `EVENTS` table for event number–event data pairs, sequentially ordered.

An event is a `u3_fact`, a `struct` including the timestamp and event `ovum`:

```c
/* u3_fact: completed event
*/
  typedef struct _u3_fact {
    c3_d             eve_d;               //  event number
    c3_l             mug_l;               //  kernel mug after
    u3_noun            job;               //  (pair date ovum)
    struct _u3_fact* nex_u;               //  next in queue
  } u3_fact;
```

Event log replay thus refers to retrieving the sequence of events from the pier's database instance and playing back each sequential event.

* Examine `u3_mars_play()` for details of how playback works.
  * `_mars_play_batch()`
  * `_mars_poke_play()`
  * `u3v_poke_raw()` and we're back to a conventional Arvo poke as we saw in `ca05`.

The introduction of epochs will enable finer-grained system recovery when necessary:

> Historically, Vere has stored a single event log and snapshot. To facilitate replay across different binary versions more convenient and less error-prone, an improved design is the "epoch" system.\
> In the epoch system, Vere breaks up the event log into "epoch"s, where an epoch represents a snapshot and some events after that snapshot.\
> An epoch lives in its own folder, named after the first event in that epoch.\
> In addition to storing a snapshot and a log of events, each epoch folder also stores a version file indicating which version of Vere originally ran these events -- this makes replay across different binary versions much easier, especially in the case of a jet mismatch in an old binary.

#### Snapshots

> Replay is how Vere computes the state of a ship's Arvo instance from the event log after a ship reboots. In order to avoid replaying the entire event log, Replay takes a snapshot of the current state of the ship approximately once every ten minutes. Then when a ship reboots, Replay loads the most recent snapshot and replays events from the event log up to the most recent event.

`u3e_save()` saves the loom (snapshots), often called via `u3m_save()`.

#### Demand Paging

> Demand paging refers to the ability to load only needed pages of memory into RAM, leaving other pages on disk, to reduce memory use.

Pages are marked as clean (`PROT_READ`) or dirty (`PROT_READ|PROT_WRITE`) or guard (`PROT_NONE`). The access pattern assumes all pages are accessed from the outside inwards (another advantage of the loom model).

* [`urbit/vere` #402](https://github.com/urbit/vere/pull/402) (merged)
* [`urbit/vere` #410](https://github.com/urbit/vere/pull/410) (merged)

### Unix I/O <a href="#unix-io" id="unix-io"></a>

The king is responsible for the I/O operations of the communicating vanes: Ames, Behn, Clay, Dill, Eyre, Iris, Khan, Lick. (The other two vanes, Gall and Jael, are landlocked and only interact within Urbit.)

We will cover the I/O drivers in a later lesson `ca11` after we have covered the major vanes which need to interface with the host OS.

### IPC <a href="#ipc" id="ipc"></a>

The Urbit runtime has two categories of IPC:

1. King/serf interprocess communication
2. Vane-driven interprocess communication
3. `%khan`/`conn.c`-based sockets
4. `%lick`-based communications

In general, [POSIX IPC](https://www.geeksforgeeks.org/inter-process-communication-ipc/) is “a mechanism that allows processes to communicate with each other and synchronize their actions.” This can be done by sharing memory directly between the processes or by passing messages. Vere does a little of both: the loom is the shared memory arena, and sometimes messages are used.

For instance, in Vere's pier management (mainly `vere/pier.c`), the [lord](https://github.com/urbit/vere/blob/ea3eeee0d5efc198c279f2c916b73fc8df283af6/pkg/vere/lord.c#L313) coordinates the king and the serf through messages. Like Arvo, the king and the serf thus need to have the right API shape to connect to each other. The lord coordinates using `writ` to pass a value from the king to the serf, and `plea` to pass a value from the serf to the king. (These are rather like Arvo passes and gifts, but can be initiated from either side rather than just the pass/give pattern.) Whimsically, this is defined in Hoon inside a C comment in `vere/lord.c`:

```hoon
|%
::  +writ: from king to serf
::
+$  writ
  $%  $:  %live
          $%  [%cram eve=@]
              [%exit cod=@]
              [%save eve=@]
              [%meld ~]
              [%pack ~]
      ==  ==
      [%peek mil=@ sam=*]
      [%play eve=@ lit=(list ?((pair @da ovum) *))]
      [%work mil=@ job=(pair @da ovum)]
  ==
::  +plea: from serf to king
::
+$  plea
  $%  [%live ~]
      [%ripe [pro=%1 hon=@ nok=@] eve=@ mug=@]
      [%slog pri=@ tank]
      [%flog cord]
      $:  %peek
          $%  [%done dat=(unit (cask))]
              [%bail dud=goof]
      ==  ==
      $:  %play
          $%  [%done mug=@]
              [%bail eve=@ mug=@ dud=goof]
      ==  ==
      $:  %work
          $%  [%done eve=@ mug=@ fec=(list ovum)]
              [%swap eve=@ mug=@ job=(pair @da ovum) fec=(list ovum)]
              [%bail lud=(list goof)]
      ==  ==
  ==
--
```

The procedure for IPC needs to establish communications, which is a follow-on from the king starting the serf. (Vere is single-threaded but runs two processes.)

`$writ` from king to serf:

* `%live` is a request to start up the serf.
* `%peek` is a request for data from Arvo in the serf.
* `%play` is a request to the serf to play an event (in an event playback).
* `%work` is a request to the serf to carry out a computation in Arvo.

`$plea` from serf to king:

* `%live` tells if the serf is alive.
* `%ripe` tracks the serf startup state.
* `%slog` is an output request.
* `%flog` is a debug output request.
  * `%peek` is a response to the king with a scry result.
  * `%play` is a response to an event playback.
  * `%work` is a response to an injected `ovum` event.

Other parts of IPC depend on the atom-framing implementations in `vere/newt.c`, another key part of king–serf IPC. `newt.c` produces noun blobs that have a five-byte header and a variable-length payload. The header has a one-byte version tag, typically `0x0`, followed by a four-byte little-endian message byte count. The payload is the `+jam`med noun. This is used by `urbit eval` for `stdin` computed against the ivory pill ghost ship, for instance:

```sh
$ echo "(add 1 41)" | urbit eval
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
eval (run):
42
```

`eval` supports several options for processing Hoon nouns as input to or output from `conn.c`:

* `-j`, `--jam`: output result as a jammed noun
* `-c`, `--cue`: read input as a jammed noun
* `-n`, `--newt`: write output / read input as a newt-encoded jammed noun, when paired with `-j` or `-c` respectively
* `-k`: treat the input as the jammed noun input of a `%fyrd` request to `conn.c`; if the result is a `goof`, pretty-print it to `stderr` instead of returning it

In `vere/newt.c`, see particularly:

* `u3_newt_send()` transmits a jammed noun (using `u3s_jam_xeno()`, for instance) to a task buffer for `libuv`. (Recall that `libuv` is the main event loop driver for the king process.)
* `u3_newt_read()` pulls out the jammed noun from the buffer.

Khan and Lick both use `newt.c`.

* Demonstrate invoking a statement at the CLI with the `urbit` executable.

#### Example: `|meld` Trace

* Let's walk through the lifecycle of a command-line initiated `./zod/.run meld`.
  * `vere/main.c`
  * `noun/urth.c`
* Compare the lifecycle of `|meld`.
  * `/gen/hood/meld`
  * `/lib/helm/kiln`
  * `/lib/hood/kiln`
  * `/sys/clay`
  * `vere/lord.c`
  * `vere/serf.c`

We will discuss Khan and Lick in detail during the next lesson, but here's a quick recap of their functionality.

#### Khan

> Khan is the "control plane" and thread-runner vane. Its main purpose is to allow external applications to run [threads](/urbit-os/base/threads) via a Unix Socket and receive the result.

A [socket](https://en.wikipedia.org/wiki/Unix_domain_socket) is an endpoint for data communications. In the runtime, it is implemented by [`conn.c`](https://github.com/urbit/vere/blob/develop/pkg/vere/io/conn.c), the runtime counterpart to `%khan`.

* [`%khan` Overview](/urbit-os/kernel/khan)

#### Lick

> Lick manages IPC ports, and the communication between Urbit applications and POSIX applications via these ports. Other vanes and applications ask Lick to open an IPC port, notify it when something is connected or disconnected, and transfer data between itself and the Unix application.

* [`%lick` Overview](/urbit-os/kernel/lick)

## Debugging the Runtime <a href="#debugging-the-runtime" id="debugging-the-runtime"></a>

To conclude today's material, we would like to briefly demonstrate several debugging principles with Vere / NockVM.

### `printf` <a href="#printf" id="printf"></a>

`fprintf`-based output should be done using `fprintf()` to `stderr`. Use both and to achieve line feed (move cursor down one line) and carriage return (move it to the left). You can also use `u3l_log` which does not require `\r`, but should not be used in cases where the I/O drivers have not yet been initialized or can no longer be relied upon, e.g. crashing or shutdown.

### `gdb` <a href="#gdb" id="gdb"></a>

> For C, make heavy use of `gdb`. `lldb` is far worse than `gdb` for debugging Urbit, so it's worth developing on a Linux box even if that means `ssh`ing into a server. (\~wicdev-wisryt)

`gdb` works best when you build with debugging symbols.

```sh
bazel build :urbit --compilation_mode=dbg

- OR -

bazel build :urbit --copt=-DU3_CPU_DEBUG

- OR -

bazel build :urbit --copt=-DU3_MEM_DEBUG
```

When using GDB, before attaching to the process you should set the following:

```gdb
set follow-fork-mode child
handle SIGSEGV nostop noprint
```

If you are debugging jets or the serf, then you want to attach to the serf at `urbit-worker`.

```sh
gdb --args ./bazel-bin/pkg/vere/urbit --args zod

- OR -

gdb attach <PID>
```

```gdb
break jet_file:jet_name
```

### Valgrind <a href="#valgrind" id="valgrind"></a>

[Valgrind](https://valgrind.org/) is a memory profiling tool for diagnosing memory usage and memory leaks. As a memory tracking tool, Valgrind uses several times more memory than the native application requires.

The possible expedient to make it viable is:

* Decrease the loom size with `--loom` (e.g. `--loom 29` or something similarly constrained).

[`urbit/urbit` #5161](https://github.com/urbit/urbit/issues/5161) has some discussion about using Valgrind. (The memory leak in question was ultimately resolved by [#5614](https://github.com/urbit/urbit/pull/5614), which presents an instructive insight.)

### Changing the Serf <a href="#changing-the-serf" id="changing-the-serf"></a>

The process call to invoke the serf is hard-coded in the king. To run Sword (née Ares) as a serf, for instance, you need to change the `protocol` entry in `vere/lord.c:u3_lord_init()` to the Sword executable as a full-path string literal.

## Exercises <a href="#exercises" id="exercises"></a>

* Turn on `LORD_TRACE_CUE` and boot a new fake ship.
* Examine the new `/ted/runtime-version` thread which is in the latest release (412 K). How does it work? Trace the types and values back through to see how the values are recovered from the runtime.


# 8. Vanes I: Behn, Dill, Kahn, Lick

Core Academy lesson on Arvo vanes covering the Behn timer vane, Dill terminal driver vane, Khan thread runner vane, Lick IPC vane.

*In this lesson we'll look at Arvo's timer vane, Behn, and Dill, the terminal driver vane. This includes discussion of the Dojo terminal, and the userspace/kernelspace interfaces `%helm` and `%hood`.*

A **vane** is an Arvo kernel module that performs essential system operations. The vanes are:

* Ames, the peer-to-peer networking vane.
* Behn, the timer vane.
* Clay, the filesystem, revision-control and build system vane.
* Dill, the terminal-driver vane.
* Eyre, the HTTP vane.
* Gall, the application vane.
* Iris, the server HTTP vane.
* Jael, the security vane.
* Khan, the control vane.
* Lick, the interprocess communication (IPC) vane.

> As described above, we use Arvo proper to route and control the flow of `move`s. However, Arvo proper is rarely directly responsible for processing the event data that directly causes the desired outcome of a `move`. This event data is contained within a `card`. Instead, Arvo proper passes the `card` off to one of its vanes, which each present an interface to clients for a particular well-defined, stable, and general-purpose piece of functionality.

## Vane Interface <a href="#vane-interface" id="vane-interface"></a>

Arvo is a message dispatcher, which doesn't really know about the vanes except via their existence in a `van=(map term vane)` in Arvo's `$soul`.

Formally, a vane must be a “vane-shaped noun”—an interface presenting the arms:

```hoon
|%
::  +call: handle a +task request
++  call
::  +load: migrate an old state to a new vane version
++  load
::  +scry: view vane state at a particular /path
++  scry
::  +stay: extract state before reload
++  stay
::  +take: handle $response sign
++  take
--
```

* `+load` and `+stay` are necessary to update the vane.
* `+call` is used to pass a request in (“advance to target”).
* `+scry` exposes the read-only scry namespace of the vane.
* `+take` receives a response from another vane (“retreat along call stack”).

(Now the formerly-elided distinctions between `sign`, `gift`, `task`, and `note` start to matter.)

![](https://media.urbit.org/docs/arvo/cycle.png)

* A `note` is sent by a vane to the Arvo kernel's `+call` arm.
* Arvo dispatches a `task` to a vane's `+call` arm.
* The vane performs the work.
* If a result needs to be passed back, it is emitted as a `gift` along the `duct` back to Arvo's `+take` arm.
* Arvo dispatches a `sign` to the original caller's `+take` arm.

The actual mechanics of this are that the moves are placed into the appropriate `duct`, which is a `(list wire)`, simply an ordered collection of moves representing the causal history.

`/sys/arvo` tracks what little it knows about vanes at a few points, e.g.:

```hoon
::  van: vanes while we desire it (in larval stage)
van=(map term (trap vase))
::
++  grow
  |=  way=term
  ?+  way  way
    %a  %ames
    %b  %behn
    %c  %clay
    %d  %dill
    %e  %eyre
    %g  %gall
    %i  %iris
    %j  %jael
    %k  %khan
    %l  %lick
  ==
::
+$  vane  [=vase =worm]
```

* `vase` is of course a generic vase, but specifically it expects a noun with the correct `$type`.
* `worm` is the worm cache as discussed in [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04).

Arvo interacts with vanes in vase mode; for instance, a scry takes place via a call to the `+scry` arm via a `+slap` against the `%limb` named `%scry`: `(~(slap wa sac) rig [%limb %scry])`. As usual, working in vase mode permits dynamic updates to the source.

Vanes have as their subject:

* `/sys/hoon` for language definitions.
* `/sys/arvo` for message dispatch.
* `/sys/lull` for a shared interface definition.
* `/sys/zuse` for various stdlib utilities.

In particular, `/sys/lull` acts as a header so that vanes can “see” each other's interface.

### Updates <a href="#updates" id="updates"></a>

As with other parts of the system, vanes are rebuilt if the inner core on which they rely has been updated or if the vane itself has changed.

An update to a vane is triggered by `+mod:what:pith` in the `+le` event-loop engine. (Recall from `ca05` that `+what` is involved in a system upgrade.) While there are some unfamiliar types here, note particularly the `%=` centis clause building each vane.

```hoon
++  mod
  |=  [del=news all=?]
  ^+  ..pith
  =^  job=oped  fat.mod.sol  (~(adorn adapt fat.mod.sol) del all)
  =?  lul.mod.sol  ?=(^ lul.job)
    (smit:va "lull" pit /sys/lull/hoon u.lul.job)
  =?  zus.mod.sol  ?=(^ zus.job)
    (smit:va "zuse" lul.mod.sol /sys/zuse/hoon u.zus.job)
  %-  %+  need:wyrd   kel.ver.zen
      :~  lull/;;(@ud q:(slap lul.mod.sol limb/%lull))
          zuse/;;(@ud q:(slap zus.mod.sol limb/%zuse))
      ==
  %=    ..pith
      van.mod
    %+  roll  van.job
    |=  [[nam=term txt=cord] van=_van.mod.sol]
    ^+  van
    =/  nex  (create:va our zus.mod.sol nam /sys/vane/[nam]/hoon txt)
    =/  nav  (~(get by van) nam)
    =?  nex  ?=(^ nav)  (update:va vase.u.nav nex)
    (~(put by van) nam (settle:va nex))
  ==
```

The recompilation against `%zuse` takes place in `+adorn:adapt:part`. ([Arvo Pärt](https://www.youtube.com/watch?v=jNxbT0MESTY))

```hoon
::  kernel modules
::
::    %zuse is the subject of the vanes; force all if we have a new %zuse
::
=.  all  |(all ?=(^ zus))
=|  nav=(map term cord)
=?  nav  all
  %-  ~(gas by nav)
  %+  turn
    ~(tap by dir:(~(dip of fat) /sys/vane))
  |=([name=@ta _fat] [`@tas`name (sole (need fil))])
```

## Behn <a href="#behn" id="behn"></a>

Behn is a timer/wake-up call system. Since it's a simple vane, let's approach it obliquely, by looking at a generator that calls it.

* Open `/base/gen/timers/hoon` and examine the code.

```hoon
.^((list [date=@da =duct]) %bx (en-beam [our %$ [%da now]] /debug/timers))
```

* [“Behn Overview”](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/dill/behn/README.md)

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

The interface to Behn is defined in `/sys/lull`:

```hoon
::                                                      ::::
::::                    ++behn                            ::  (1b) timekeeping
  ::                                                    ::::
++  behn  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%doze p=(unit @da)]                            ::  next alarm
        [%wake error=(unit tang)]                       ::  wakeup or failed
        [%meta p=vase]
        [%heck syn=sign-arvo]                           ::  response to %huck
    ==
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  $>(%born vane-task)                             ::  new unix process
        [%rest p=@da]                                   ::  cancel alarm
        [%drip p=vase]                                  ::  give in next event
        [%huck syn=sign-arvo]                           ::  give back
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
        [%wait p=@da]                                   ::  set alarm
        [%wake ~]                                       ::  timer activate
    ==
  --  ::behn
```

(A `%vega` task informs the vane that the kernel has been upgraded.)

### Structure <a href="#structure" id="structure"></a>

```hoon
+$  behn-state
  $:  %2
      timers=(tree [key=@da val=(qeu duct)])
      unix-duct=duct
      next-wake=(unit @da)
      drips=drip-manager
  ==
```

* How does Behn think of timers?

`/sys/behn` presents three primary cores:

1. A type definition core.
2. A helper core.
3. The primary vane interface.

Behn only has two kinds of moves: `%wait` `task`s and `%wake` `gift`s. How are these processed and what do they result in? (See the `$timer-map` structure too.)

* What is a drip? How is it used?

> Say an app (the Target) is subscribed to updates from Clay (the Client). If Clay `%give`s updates to the app directly and the app crashes, this may cause Clay to crash as well. If instead Clay `%pass`es Behn a `%drip` `task` wrapping the update `gift`, Behn will set a timer for `now` that, when fired, will cause the update `gift` to be given. If it causes a crash then it will have been in response to the `%drip` move, thereby isolating Clay from the crash. Thus `%drip` acts as a sort of buffer against cascading sequences of crashes.

### The Nested Core Pattern <a href="#the-nested-core-pattern" id="the-nested-core-pattern"></a>

(\~rabsef-bicrym calls this the “`+abet` engine” and while it's not a popular term inside of the core development team, I like the pithiness of it. “Engine” as a term of art is frequently used in the kernel to refer to `+le` two-letter doors so this usage is not completely inconsistent, just more specialized in application.)

The basic concept of the nested core pattern is to have an outer core which builds a list of cards and state changes, then produces the queued changes all at once. (I always think about this as being one of those wind-up cars that you crank and then set down to whir away.)

Behn's nested core pattern is pretty simple: it has an alias to `this`, one `+emit` arm to prepend a move to a list of moves, and an `+abet` arm to yield the `[moves state]`. The `+per-event` core is used to script the neighboring `+scry` and `+call` arms for the vane without leaking state invariants. Behn's instantiation of the `+abet` pattern centralizes the helper outer core as a centralized state machine.

These are some common `+abet` pattern arms. These are not all unique, and many cores will omit all or most of these.

* `+abed`—initialize. Set up the state of the inner core.
* `+yoke`—initialize. Start from a particular value.
* `+abet`—finalize. Exit from an inner core to an outer core, taking changes. Commonly, take a modified valued and overwrite it into the final state with a `+put:by`.
* `+abut`—finalize. Alternative exit from `+abet` for a deletion.
* `+move`—send a move. Generalization for `+pass`/`+give`.
* `+pass`—request an action. Prepend a `%pass` move to the current list of moves.
* `+give`—return a result. Prepend a standard `%give` to the current list of moves.
* `+emit`—submit a card. Prepend a card to the current list of cards.
* `+emil`—submit cards. Prepend a list of cards to the current list of cards.

If some state needs to be maintained, this can be built in a door, but Behn's particular example is even more basic. In files with associated doors or with multiple nested core instances, it is common to prepend a two-letter identifier to disambiguate which outer core is being scripted at any given time, such as `+mo-abet` or `+ap-emil`.

* [The Engine Pattern](/hoon/engine-pattern)

### Vere I/O Driver: `vere/io/behn.c` <a href="#vere-io-driver-vereiobehnc" id="vere-io-driver-vereiobehnc"></a>

Arvo acquires its timer updates from Unix via `vere/io/behn.c`. This file presents its primary interface at `u3_behn_io_init()` to initialize a timer. This simply retrieves the current Unix time for a starting point and sets up the interface.

Each communicating vane is linked various I/O drivers in `vere/auto.c`. (These do not correspond one-to-one with vanes.) These are registered into `car_u`, a global `_u3_auto` used for I/O driver invocations and callbacks.

For Behn, the timer is set in `_behn_ef_doze()` using `uv_timer_start()`. The corresponding wakeup timer is emitted in `_behn_time_cb()` when the `libuv` main event loop handler.

Finally, we can examine how the injection comes back into Arvo in `_behn_time_cb()`. An `ovum` is produced by `u3_ovum_init()`, manually injected using `u3_auto_plan()`, and subscribed to with `u3_auto_peer()`. `/sys/behn` then processes this wakeup event as a `%wake` via its `+call` arm.

## Dill <a href="#dill" id="dill"></a>

Dill is Urbit's terminal driver.

```hoon
|pass [%d %text "hello world"]
```

Dill as a vane is mostly responsible for actually constructing terminal sessions and coordinating input and output. Thus most of the terminal stack actually lives in userspace (instrumented by Gall) rather than in `/sys/dill`.

What do we mean when we talk about a terminal? Originally, of course, computer were directly programmed by moving wires between vacuum tubes or chips; later, this evolved to the ability to read and output cards. Computer terminals with CRT-based character displays began to be used in the late 1950s and gradually became more common. In fact, the original plasma display screens were used with PLATO in the 1970s–1990s.

![](https://2.bp.blogspot.com/-T0k6GSlqlQE/UIYEXlThxMI/AAAAAAAAABs/Lwq27QO1TTQ/s1600/platoavatar.png)

When we refer to the terminal today, we typically mean a modern [terminal emulator](https://en.wikipedia.org/wiki/Terminal_emulator), which presents a terminal-like text user interface (TUI) for software to treat as if it were an actual character display. Terminal emulators need to track information like dimensions ($x$, $y$), content layout, active sessions or connexions, and cursor position. They provide affordances like a color space, escape codes, layout libraries, and scrollable sessions.

Dill is responsible for interfacing with keystrokes and with the terminal emulator session. Since Urbit can be run in a daemon mode, it's not necessary for Dill to actually have a terminal session for Urbit to run.

Due to terminal emulator limitations, Dill sessions are only properly supported today in the `%webterm` app.

* [“Developer Call: Urbit’s improved Terminal Stack”](https://www.youtube.com/watch?v=E-6E-l1SxFw)
* [“Dill Overview”](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/dill/dill/README.md)

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

The `/sys/lull` interface specification for Dill is more complicated than that of Behn. Unlike Behn, a number of supporting types are necessary to produce the basic pair of `gift`/`task` for Dill.

```hoon
::                                                      ::::
::::                    ++dill                            ::  (1d) console
  ::                                                    ::::
++  dill  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%blit p=(list blit)]                           ::  terminal output
        [%logo ~]                                       ::  logout
        [%meld ~]                                       ::  unify memory
        [%pack ~]                                       ::  compact memory
        [%trim p=@ud]                                   ::  trim kernel state
        [%logs =told]                                   ::  system output
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  [%boot lit=? p=*]                               ::  weird %dill boot
        [%crop p=@ud]                                   ::  trim kernel state
        [%flog p=flog]                                  ::  wrapped error
        [%heft ~]                                       ::  memory report
        $>(%init vane-task)                             ::  after gall ready
        [%logs p=(unit ~)]                              ::  watch system output
        [%meld ~]                                       ::  unify memory
        [%pack ~]                                       ::  compact memory
        [%seat =desk]                                   ::  install desk
        [%shot ses=@tas task=session-task]              ::  task for session
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
        [%verb ~]                                       ::  verbose mode
        [%knob tag=term level=?(%hush %soft %loud)]     ::  deprecated removeme
        session-task                                    ::  for default session
        told                                            ::  system output
    ==                                                  ::
  ::                                                    ::
  +$  session-task                                      ::  session request
    $%  [%belt p=belt]                                  ::  terminal input
        [%blew p=blew]                                  ::  terminal config
        [%flee ~]                                       ::  unwatch session
        [%hail ~]                                       ::  terminal refresh
        [%open p=dude:gall q=(list gill:gall)]          ::  setup session
        [%shut ~]                                       ::  close session
        [%view ~]                                       ::  watch session blits
    ==                                                  ::
  ::                                                    ::
  +$  told                                              ::  system output
    $%  [%crud p=@tas q=tang]                           ::  error
        [%talk p=(list tank)]                           ::  tanks (in order)
        [%text p=tape]                                  ::  tape
    ==                                                  ::
  ::
  ::::                                                  ::  (1d2)
    ::
  +$  blew  [p=@ud q=@ud]                               ::  columns rows
  +$  belt                                              ::  client input
    $?  bolt                                            ::  simple input
        [%mod mod=?(%ctl %met %hyp) key=bolt]           ::  w/ modifier
        [%txt p=(list @c)]                              ::  utf32 text
    ==                                                  ::
  +$  bolt                                              ::  simple input
    $@  @c                                              ::  simple keystroke
    $%  [%aro p=?(%d %l %r %u)]                         ::  arrow key
        [%bac ~]                                        ::  true backspace
        [%del ~]                                        ::  true delete
        [%hit x=@ud y=@ud]                              ::  mouse click
        [%ret ~]                                        ::  return
    ==                                                  ::
  +$  blit                                              ::  client output
    $%  [%bel ~]                                        ::  make a noise
        [%clr ~]                                        ::  clear the screen
        [%hop p=$@(@ud [x=@ud y=@ud])]                  ::  set cursor col/pos
        [%klr p=stub]                                   ::  put styled
        [%mor p=(list blit)]                            ::  multiple blits
        [%nel ~]                                        ::  newline
        [%put p=(list @c)]                              ::  put text at cursor
        [%sag p=path q=*]                               ::  save to jamfile
        [%sav p=path q=@]                               ::  save to file
        [%url p=@t]                                     ::  activate url
        [%wyp ~]                                        ::  wipe cursor line
    ==                                                  ::
  +$  dill-belt                                         ::  arvo input
    $%  belt                                            ::  client input
        [%cru p=@tas q=(list tank)]                     ::  errmsg (deprecated)
        [%hey ~]                                        ::  refresh
        [%rez p=@ud q=@ud]                              ::  resize, cols, rows
        [%yow p=gill:gall]                              ::  connect to app
    ==                                                  ::
  +$  dill-blit                                         ::  arvo output
    $%  blit                                            ::  client output
        [%qit ~]                                        ::  close console
    ==                                                  ::
  +$  flog                                              ::  sent to %dill
    $%  [%crop p=@ud]                                   ::  trim kernel state
        $>(%crud told)                                  ::
        [%heft ~]                                       ::
        [%meld ~]                                       ::  unify memory
        [%pack ~]                                       ::  compact memory
        $>(%text told)                                  ::
        [%verb ~]                                       ::  verbose mode
    ==                                                  ::
  ::                                                    ::
  +$  poke                                              ::  dill to userspace
    $:  ses=@tas                                        ::  target session
        dill-belt                                       ::  input
    ==                                                  ::
  --  ::dill
```

The main concepts to keep in mind:

* Dill receives `%belt` `task`s and sends `%blit` `gift`s.
* `%belt` `task`s result from keystrokes, terminal resizing,
* `%blit` `gift`s result from output events: putting a character, clearing the screen, placing the cursor.

### Structure <a href="#structure" id="structure"></a>

Dill's primary state is its `$axle` with a logging level:

```hoon
|%                                                      ::  console protocol
+$  axle                                                ::
  $:  %7                                                ::
      hey=(unit duct)                                   ::  default duct
      dug=(map @tas axon)                               ::  conversations
      eye=(jug @tas duct)                               ::  outside observers
      ear=(set duct)                                    ::  syslog listeners
      lit=?                                             ::  boot in lite mode
      egg=_|                                            ::  see +take, removeme
  ==                                                    ::
+$  axon                                                ::  dill session
  $:  ram=term                                          ::  console program
      tem=(unit (list dill-belt))                       ::  pending, reverse
      wid=_80                                           ::  terminal width
  ==                                                    ::
+$  log-level  ?(%hush %soft %loud)                     ::  none, line, full
--
```

As with `/sys/behn`, the primary cores of Dill include:

1. Type definitions (two cores).
2. Helper core (`+as` per-cause engine).
3. The primary vane interface.

A good way to familiarize yourself with Dill operations is to follow the full lifecycle of input and output in the next section.

Dill is the first vane in the boot sequence, and is used to boot Jael. (Compare `%aqua`, which does not need to start Dill and can initialize Jael directly.)

### Vere I/O Driver: `vere/io/term.c`, `ptty.c` <a href="#vere-io-driver-vereiotermc-pttyc" id="vere-io-driver-vereiotermc-pttyc"></a>

The main entrypoint for the terminal is `u3_term_io_init()`, which simply sets up the interface and callbacks.

As we noted before, each communicating vane is linked various I/O drivers in `vere/auto.c`. Here the global `car_u` has `term.c` connected for invocations and callbacks.

**`vere/vere.h`**:

```c
// u3_term_start_spinner(): prepare spinner state. RETAIN.
void u3_term_start_spinner(u3_noun say, c3_o del_o);

// u3_term_stop_spinner(): reset spinner state and restore input line.
void u3_term_stop_spinner(void);

// u3_term_get_blew(): return window size [columns rows].
u3_noun u3_term_get_blew(c3_l tid_l);

// u3_term_ef_winc(): window change.
void u3_term_ef_winc(void);

// u3_term_ef_ctlc(): send ^C.
void u3_term_ef_ctlc(void);

// u3_term_io_init(): initialize terminal I/O.
u3_auto* u3_term_io_init(u3_pier* pir_u);

// u3_term_io_hija(): hijack console for cooked print.
FILE* u3_term_io_hija(void);

// u3_term_it_log(): writes a log message
void u3_term_io_log(c3_c* line);

// u3_term_io_loja(): release console from cooked print.
void u3_term_io_loja(int x, FILE* f);

// u3_term_log_init(): initialize terminal for logging
void u3_term_log_init(void);

// u3_term_log_exit(): clean up terminal.
void u3_term_log_exit(void);

// u3_ptty_init(): initialize platform-specific tty.
u3_utty* u3_ptty_init(uv_loop_t* lup_u, const c3_c** err_c);
```

For instance, a keystroke is processed in the following way:

* `uv_read_start()` is the `libuv` event loop injector.
* `_term_read_cb()` is the character keystroke callback.
* `_term_suck()` processes input.
* `_term_io_suck_char()` decides if it's an `xterm` terminal emulator issue or something for Arvo to know about.
* `_term_io_spit()` inputs the buffer and belt.
* `_term_io_belt()` actually sends a value along the belt (as an `ovum`).

Now in Arvo, the keystroke is routed to Dill:

* `+call` takes the `task` from Arvo.
* `+call:as` receives the input and dispatches on `%belt`.
* `+send:as` sends the action to the proper session.
* `+deal:as` signals to pass the keystroke to Gall.
* `+pass:as` executes the pass to Gall.

Output can happen in three ways:

1. Some output traverses a path from (say) Gall outbound. These are conventionally known as `%slog`s.

   * Somehow a noun is marked for output using a `%slog` hint:
     * `~&` sigpam does this directly.
     * `~>` siggar can do this with a `%slog` hint and a priority value.
     * `+slog` wraps this as a function.

   Back in the runtime:

   * Once we have a `%slog` hint for the runtime, it can be emitted from the Nock processor via Nock Eleven. `noun/nock.c:_n_bint()` dispatches this via `SLOG` and thence `do_slog` in the bytecode processor.
   * `noun/trace.c:u3t_slog()` prints a value directly through the `u3C.slog_f` print handler, which is `_cw_serf_send_slog()`.
   * `vere/main.c::_cw_serf_send_slog()` sends the hint output to the serf.
   * `vere/main.c:_cw_serf_send()` is a plea handler to send pleas to the daemon.
   * `vere/newt.c:u3_newt_send()` sends a `+jam`med noun of the output (`u3s_jam_xeno()`) as a buffer to a stream. In this case, the value ends up at the `libuv` buffer using `uv_write()`.

   In this case, output results in the fact of the layout of the terminal handler rather than being explicitly known about by Dill. As a consequence, if you are building a TUI application and you don't want to have misaligned output, you need to build it directly using `%blit`s and suppress `%slog`s within your app. (See `tui-toys` for an example in `%snek`.)
2. Other output goes via Dill because the terminal vane explicitly needs to know about its position. This are `%blit`s.
   * A program (like `%snek`) can specify to manually output a `%blit` like `%klr` (styled text) or `%put` (plain text) at the cursor.
     * [`/lib/etui`](https://github.com/urbit/urbit/blob/wip/tui-toys/pkg/demo/lib/etui.hoon) offers some interesting demonstrations in this vein; see the `+zo` engine.
   * `vere/io/term.c:_term_io_kick()` applies effects sent to `term.c`, including blits. It was registerd as the effect handler when the I/O drivers were registered.
   * `vere/io/term.c:_term_ef_blit()` switches on the type of blit (notice it can also track a cursor position).
   * `vere/io/term.c:_term_it_show_tour()` emits UTF-32 to the cursor location.
   * Finally, `vere/io/term.c:_term_it_show_line()` prints at the actual cursor position.
3. The runtime terminal manager automatically handles aspects of layout such as maintaining the input line at the bottom of the screen. See `vere/io/term.c:_term_it_restore_line()` for details. (CSI = Control Sequence Introducer sequence) A different terminal handler (like `%webterm`) may handle these decisions differently.

Other output may follow yet a different path; for instance, `u3l_log()` directly prints using `vsnprintf()`to `stderr`. Some output goes via the `libuv` event loop, such as `u3_ptty_init()`..

* Examine `u3_term_log_init()` in `vere/io/term.c`.
* Examine `_term_it_show_line()` in the same file.

### Dojo, `%hood`, `%helm`, `%drum` <a href="#dojo-hood-helm-drum" id="dojo-hood-helm-drum"></a>

Dojo is Urbit's primary CLI interface, and while it is too complicated to delve deeply into here, the major parts to consider include:

1. Hoon parser. Real-time parsing of input, which evaluates Hoon code for syntactical correctness. (This is the reason that typing at the Dojo prompt is frequently slower than typing at other CLIs.)
2. Specialized syntax. Besides Hoon input, generators and pokes can be set up and invoked directly from the Dojo prompt. `%say` and `%ask` generators are head-tagged pairs with gates following that return `sole-result`s.
3. `+generator` prefixes cause Dojo to look in `/gen` for a particular generator file.
4. `|generator` is a Hood generator, on which more in a moment.
5. `+desk!generator` invokes a generator on a particular desk.
6. `:agent|generator` takes the output from a generator (at `/gen/agent/generator`) and feeds it as a noun to the agent's `+on-poke` arm.
7. `-thread` invokes a thread, with a similar desk-specific prefix as above.
8. Hood. Most of the interesting Urbit instrumentation is provided to Dojo by the Hood/Helm agent pipeline.
9. `%hood` is the overarching system app, to which Dojo redirects generator invocations prefixed with `|` bar such as `|pass` and `|install`. (Thus, `|pass` is in fact shorthand for `:hood|pass`.)
10. `%helm` provides the interface for kernel and system functionality, such as `|verb`, `|moon`, etc. Hood calls into Helm.
11. `%drum` manages the active CLI apps (`|dojo/link`, `Ctrl+X`).
12. `%kiln` instruments filesystem operations using Clay.

The overall call web of these is surprisingly tangled; as an example, let's trace `|dojo/link`, which tells Dojo to register a CLI interface to a Gall agent.

* `/gen/hood/link`
* `/app/hood`
* `/lib/drum`→`+poke` w/ `%drum-link`
* `/lib/drum`→`+poke-link`
* `/lib/drum`→`+se-link`
* `eel` (what is this?)

## Exercises <a href="#exercises" id="exercises"></a>

* Trace the entire lifecycle of `|pass [%d %text "Hello Mars!"]`. Include a function-by-function annotation and commentary.
* Write a basic vane, `/sys/vane/`. `~&` on receiving a `task`.


# 9. Vanes II: Ames

Core Academy lesson on Ames covering the encrypted peer-to-peer protocol, its packet structure, message flows, cryptographic operations, integration with Urbit ID, and the Fine remote scry protocol.

*This lesson covers Ames: Urbit's networking protocol. We'll see how it relates to Urbit ID and Fine, the remote scry protocol.*

Urbit knows about three kinds of networking: Ames and Fine over Ames and HTTP over Eyre. (That is, the network protocol over the implementing vane.) Ames is the name of both the network protocol and the implementing vane. Ames is particularly for ship-to-ship communication, while Fine acts as a dispatcher for efficient data requests (such as desk updates). In this lesson, we will focus on Ames first and then foray into Fine, since Fine is in more flux.

Ames is a good example of a vane that is conceptually straightforward but mechanically complicated. It handles networking, but has to track peer state, message flows, individual packets, network weather, etc. as gracefully as possible.

## Network Protocol: Ames <a href="#network-protocol-ames" id="network-protocol-ames"></a>

> Ames is an encrypted peer-to-peer network running as an overlay over UDP. Ames does not have separate addressing and identity layers (like IP and DNS). An Ames address is an identity, mapped to a phonemic string to create a memorable pseudonym, and bound to a public key for encrypted communication. (Whitepaper)

From a kernel perspective, the point of Ames is to extend `move` semantics across more than one Arvo instance. Local vanes (such as Gall) pass a `%plea` request to Ames, which sends the message to the peer Ames over the wire. That peer's Ames then dispatches the message to the destination vane on the peer. One advantage of Ames is that it wraps all of the peer negotiation and message delivery details such that the calling vane need not be aware of these. Among Ames’ guarantees:

1. Messages within a flow are processed in order.
2. Messages will be delivered only once to a destination vane. (“Ames can guarantee exactly-once delivery because both ships are transactional (so if they give an ack, we know for sure they have received it permanently and won't forget about it.” [\~wicdev-wisryt](https://groups.google.com/a/urbit.org/g/dev/c/y_gaSpn9mxM/m/zNy1l9ufAgAJ))

Networking in some ways is like a duct. It requires you to keep track of the forward and reverse causal history and content. However, there are two kinds of data transmissions: commands and facts.

> There is a categorical difference between a bus, which transports commands, and a network, which transports packets. You can drop a packet but not a command; a packet is a fact and a command is an order. Facts are inherently idempotent; learning fact *X* twice is the same as learning it once. You can drop a packet, because you can ignore a fact. Orders are inherently sequential; if you get two commands to do thing *X*, you do thing *X* twice. (Whitepaper)

The Ames vane is responsible for sending and receiving messages of arbitrary length. This means that it needs to know how to build and reconstruct component packets of a message, how to route, and how to encrypt and decrypt. Ames does handle some aspects of encryption and decryption but not all. For networking, Ames gets its public keys (and breach notifications) from Jael, which in turn gets them from an Azimuth userspace agent. The actual cryptographic operations may live in `/sys/zuse` but are applied by Ames as appropriate.

* [“Ames Overview”](/urbit-os/kernel/ames)
* [Curtis Yarvin `~sorreg-namtyv`, Philip Monk `~wicdev-wisryt`, Anton Dyudin, and Raymond Pasco, “Urbit: A Solid-State Interpreter” (“Whitepaper”)](http://media.urbit.org/whitepaper.pdf), sections 9–10
* [“Ames Security Audit and the Future of the Protocol”](https://urbit.org/blog/security-audit)

### Packet Protocol <a href="#packet-protocol" id="packet-protocol"></a>

> Ames receives packets as Arvo events and emits packets as Arvo effects. The runtime is responsible for transferring the bytes in an Ames packet across a physical network to another ship. (Ames Tutorial)

Ames packets have a 32-bit header followed by a variable-length body.

**Header**

| Bits  | Representative Value       | Meaning               |
| ----- | -------------------------- | --------------------- |
| 31–29 | `000`                      | reserved bits         |
| 28    | `1`                        | Ames or Fine?         |
| 27–25 | `000`                      | Ames protocol version |
| 24–23 | `11`                       | sender address size   |
| 22–21 | `11`                       | receiver address size |
| 20–1  | `1000.0001.1101.0010.1111` | checksum              |
| 0     | `1`                        | is this relayed?      |

The 2-bit address size refers to the address space rank (gathering galaxies and stars together as routers).

```hoon
++  ship-meta
  |=  =ship
  ^-  [size=@ =rank]
  =/  size=@  (met 3 ship)
  ?:  (lte size 2)  [2 %0b0]
  ?:  (lte size 4)  [4 %0b1]
  ?:  (lte size 8)  [8 %0b10]
  [16 %0b11]
```

A `relay` means that the packet is not at its destination here and should be passed forward. (This is handled by `+on-hear-forward` in `/sys/ames`.)

> If a relay responds to a scry request from its cache without asking the host, the relay should include an origin containing the last known IP and port of the host. … \[The] protocol should be resilient against the origin pointing at an unreachable IP and port.

**Body**

| Number of Bits | Representative Value                      | Meaning                                           |
| -------------- | ----------------------------------------- | ------------------------------------------------- |
| 4 bits         | `0000`                                    | sender life (mod 16)                              |
| 4 bits         | `0000`                                    | receiver life (mod 16)                            |
| variable       | `0110.0111.0110.1011`                     | sender address                                    |
| variable       | `1111.1100.0111.0011.0000.0101.0000.0000` | receiver address                                  |
| 48 bits        | —                                         | `origin` (if relayed)                             |
| 128 bits       | —                                         | `SIV` synthetic initialization vector for AES-256 |
| 16 bits        | —                                         | ciphertext size                                   |
| variable       | —                                         | ciphertext                                        |

Address size is determined by the header.

Here if the relay bit is set then 32 bits of the `origin` are the last known IPv4 address and 16 bits are the port.

> The ciphertext is formed by `+jam`ming a `$shut-packet` and then encrypting using [`+en:sivc:aes:crypto`](/hoon/cryptography#en).

The ciphertext results from `+jam`ming the message noun into an atom then breaking the result into 1 KB or smaller payloads. Packets are numbered so that they can be ordered upon receipt. These message fragments are then assembled into a single large atom and `+cue`d back into the noun.

Urbit messages result in raw nouns. Since Nock-derived languages are homoiconic, we could treat this noun as code directly, but instead we treat it as a cask (pair of mark and noun). We don't transmit vases over the network, but require the recipient to build the code locally.

> Ames messages are typed; the type itself is not sent, just a label (like a MIME type) that the recipient must map to a local source path. Validation failure causes a silent packet drop, because its normal cause is a recipient that has not yet received a new protocol update; we want the sender to back off. Ames also silently drops packets for encryption failure; error reports are just an attack channel.

#### UDP Packet Format

> There's a lot you can do with a stateful UDP server, especially one whose semantics are reasonably formal. (CGY)

At the host system level, the runtime communicates using the [User Datagram Protocol](https://en.wikipedia.org/wiki/User_Datagram_Protocol) (UDP) specification. UDP messages are “transaction oriented, and delivery and duplicate protection are not guaranteed.” (To compensate for this, Ames employs a unique system of acks and nacks, covered below.) Each UDP message has a brief header including destination, source, length, and checksum. It’s rather a “minimum viable” packet system.

> A UDP datagram consists of a datagram *header* followed by a *data* section (the payload data for the application). The UDP datagram header consists of 4 fields, each of which is 2 bytes (16 bits). UDP is faster but less reliable than TCP, another common transport protocol. In a TCP communication, the two computers begin by establishing a connection via an automated process called a ‘handshake.’ Only once this handshake has been completed will one computer actually transfer data packets to the other. ([Wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol))

Urbit compensates for this lower reliability by sending until receiving an appropriate ack or nack (negative acknowledgment) in reply.

> UDP is commonly used in time-sensitive communications where occasionally dropping packets is better than waiting. Voice and video traffic are sent using this protocol because they are both time-sensitive and designed to handle some level of loss. For example VOIP (voice over IP), which is used by many internet-based telephone services, operates over UDP. This is because a staticky phone conversation is preferable to one that is crystal clear but heavily delayed. ([Wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol))

* [RFC 768](https://tools.ietf.org/html/rfc768) (UDP specification)

### Acks and Nacks <a href="#acks-and-nacks" id="acks-and-nacks"></a>

```hoon
::  $ack: positive ack, nack packet, or nack trace
::
+$  ack
  $%  [%ok ~]
      [%nack ~]
      [%naxplanation =error]
  ==
```

If every message is a transaction (or event), then what is Ames acknowledging (ack) or negatively acknowledging (nack)? “A successful transaction has no result; a failed transaction is a negative ack and can contain an error dump.”

* An *ack* means that a piece of information has been received successfully.
* A *nack* means that a piece of information has been received but failed to process for some reason.

> Ames has an unusual system of acks and nacks (“negative acknowledgments”, but not like TCP’s packets of the same name; Ames nacks mean the packet was received but the message resulted in an error). In brief, each Ames packet of a message should get either an ack or a nack. In the current system, the nack may include an error message (e.g., an error code or a stack trace). ([\~wicdev-wisryt](https://groups.google.com/a/urbit.org/g/dev/c/y_gaSpn9mxM/m/njlRhYZHBwAJ))

Each Ames packet will merit either an ack or a nack, in other words. A nack may optionally include an error trace with it (`[tag=@tas =tang]`). Ames will adaptively continue to send messages until the appropriate acks or nacks have been received.

(A TCP nack means that the numbered packet was never received.)

Ames will send messages and acks until a replying ack is received. “Ames guarantees that a message will only be delivered once to the destination vane.” Thus nacks allow us to also guarantee notification that a request was completed or failed.

> 1. Always ack a dupe; never ack an ack. It's okay to ack a nack as long as you never nack a nack. ([Urbit Precepts B.1](https://urbit.org/blog/precepts))

If a remote ship sends a nack in response to a `%plea`, Ames waits until it receives a follow-up naxplanation and then delivers both to the local source vane. The flow blocks on needing to receive the naxplanation.

> When a new socket is opened, the client can resend (at-least-once delivery) or fail to resend (at-most-once). The programmer has to understand that the socket is not really a bus, and make sure the POST is actually an idempotent fact rather than an imperative command. (The idempotence problem is often punted to the human layer: “Please click only once to make your purchase.”) (Whitepaper)

Because Ames and Urbit assume several nines of uptime, sessions between ships are treated as persistent.

> The basic argument for including end-to-end acks (and by extension, nacks) is that they’re necessary for everything except those things which we don’t care whether the message was received at all. Thus, for Ames to give the guarantee that “if you give me a payload I will get it to the other side exactly once” isn’t useful in itself, because no application cares about that. They either (1) don’t care whether it gets there or (2) care whether the request itself was “completed”, in an application-defined sense. ([Phillip Monk, `~wicdev-wisryt`](https://groups.google.com/a/urbit.org/g/dev/c/y_gaSpn9mxM/m/njlRhYZHBwAJ))

Keep in mind Postel’s law, also known as the robustness principle: “Be conservative in what you send, and liberal in what you accept.”

* [Saltzer, Reed, and Clark, “End-to-End Arguments in System Design”](http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf) on nacks

### Cryptography <a href="#cryptography" id="cryptography"></a>

Almost every Ames packet is encrypted using [AES-256](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard). (The exception is comet self-attestation packets.)

Urbit's cryptographic suite for jets has been organized to present a uniform interface (currently in `urcrypt/`, migrating to its own repo). This eases the development of alternative runtimes since the C functions can be utilized as an FFI (foreign function interface) with uniform call signatures and behavior.

The details of the Azimuth PKI are discussed in `ca13`, *quod vide*.

* [“Ames: Cryptography”](/urbit-os/kernel/ames/cryptography)

### Routing <a href="#routing" id="routing"></a>

> The runtime tells Ames which physical address a packet came from, represented as an opaque atom. Ames can emit a packet effect to one of those opaque atoms or to the Urbit address of a galaxy (root node), which the runtime is responsible for translating to a physical address. (See `$lane`.)

```hoon
::  $address: opaque atomic transport address to or from unix
+$  address  @uxaddress
::  $lane: ship transport address; either opaque $address or galaxy
+$  lane  (each @pC address)
```

The `@uxaddress` value is an “opaque” address, in reality an IPv4 address for the runtime's use.

Since galaxy addresses are provided to the runtime on boot (from an RPC call to a roller or Ethereum node, see `vere/dawn.c:_dawn_eth_rpc()`), a route is always findable for any active point. Galaxy ports are hardcoded to be at 31337 or 13337 plus the galaxy numeric offset.

* What does `@pC` mean?

When we say that galaxies handle routing in Ames today (but stars will play a role later), this is the part of the system to which we refer.

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

The `/sys/lull` interface definition for Ames is quite long and should be reviewed in its entirety. A structural summary:

```hoon
++  ames  ^?
  |%
  ::  $task: job for ames
  +$  task
    $+  ames-task
    $%  [%hear =lane =blob]
        [%dear =ship =lane]
        [%heed =ship]
        [%jilt =ship]
        [%cork =ship]
        [%tame =ship]
        [%kroc bones=(list [ship bone])]
        $>(%plea vane-task)
        [%deep =deep]
    ::
        [%keen spar]
        [%yawn spar]
        [%wham spar]
    ::
        $>(%born vane-task)
        $>(%init vane-task)
        [%prod ships=(list ship)]
        [%sift ships=(list ship)]
        [%snub form=?(%allow %deny) ships=(list ship)]
        [%spew veb=(list verb)]
        [%cong msg=@ud mem=@ud]
        [%stir arg=@t]
        $>(%trim vane-task)
        $>(%vega vane-task)
    ==
  ::  $gift: effect from ames

  +$  gift
    $%  [%boon payload=*]
        [%clog =ship]
        [%done error=(unit error)]
        [%lost ~]
        [%send =lane =blob]
    ::
        [%tune spar roar=(unit roar)]
    ::
        [%turf turfs=(list turf)]
    ==
```

#### Tasks

```
::  $task: job for ames
::
::    Messaging Tasks
::
::    %hear: packet from unix
::    %dear: lane from unix
::    %heed: track peer's responsiveness; gives %clog if slow
::    %jilt: stop tracking peer's responsiveness
::    %cork: request to delete message flow
::    %tame: request to delete route for ship
::    %kroc: request to delete specific message flows, from their bones
::    %plea: request to send message
::    %deep: deferred calls to %ames, from itself
::
::    System and Lifecycle Tasks
::
::    %born: process restart notification
::    %init: vane boot
::    %prod: re-send a packet per flow, to all peers if .ships is ~
::    %sift: limit verbosity to .ships
::    %snub: set packet blocklist to .ships
::    %spew: set verbosity toggles
::    %cong: adjust congestion control parameters
::    %stir: recover from timer desync and assorted debug commands
::    %trim: release memory
::    %vega: kernel reload notification
```

Ames has a rather bohemian set of messaging names. Among others:

* `%hear` a packet
* `%heed` or `%jilt` a peer
* `%plea` to send a message (common from vanes)

Most other tasks are not used by userspace but by internal Ames state management. These are complemented by types like these:

* `$hoot` a request packet payload
* `$yowl` a serialized response packet payload

#### Notes

```
::    Messaging Gifts
::
::    %boon: response message from remote ship
::    %clog: notify vane that %boon's to peer are backing up locally
::    %done: notify vane that peer (n)acked our message
::    %lost: notify vane that we crashed on %boon
::    %send: packet to unix
::    %tune: peek result
::    %turf: domain report, relayed from jael
```

Every vane can receive a `%plea` note from Ames (except Behn, Dill, Iris, Khan, Lick). This is a redirection mechanism used to forward messages that a peer's vane passed to its own Ames en route to your peer's Ames and thence to your vane.

#### State

```
::  $ames-state: state for entire vane
+$  ames-state
  $+  ames-state
  $:  peers=(map ship ship-state)
      =unix=duct
      =life
      =rift
      crypto-core=acru:ames
      =bug
      snub=[form=?(%allow %deny) ships=(set ship)]
      cong=[msg=_5 mem=_100.000]
    ::
      $=  dead
      $:  flow=[%flow (unit dead-timer)]
          cork=[%cork (unit dead-timer)]
  ==  ==
```

* `$peers` are the state of connections to other ships, where `$ship-state` is either `%alien` or `%known`.
  * `%alien` means we have no PKI data and we must queue moves until we learn how contact that ship. The `$alien-agenda` stores messages, packets, and remote scry `keen`s.
  * `%known` means that we do have the peer state, on which more later.
* `$unix-duct` is a duct of moves to be sent to the host OS.
* `$life` is our own `life`, or how many times we rekeyed.
* `$crypto-core` is a handle to the cryptographic tools core.
* `$bug` describes the debug level (`|ames/verb`).
* `$snub` tracks a blocklist for incoming packets (`|ames/snub`).
* `$cong` tracks whether a flow should be considered clogged.
* `$dead` sets how long dead flows last and if they need to be restarted.

Ames maintains a duct (queue) of ordered messages. These are passed to and received from the runtime, and represent Arvo events. Each message is encrypted at the source and decrypted at the destination using a symmetric public-key system. A message may be a `%plea` (sent to another ship); in response, Ames can receive zero or more `%boon`s. The ack–nack system is explained above; note that nacks are in response to event crashes.

In `/sys/vane/ames`, there is a layer of versioning cruft to permit upgrades of the types (e.g. `$ames-state-5`).

#### Peer State

```hoon
::  $peer-state: state for a peer with known life and keys
::
::    route: transport-layer destination for packets to peer
::    qos: quality of service; connection status to peer
::    ossuary: bone<->duct mapper
::    snd: per-bone message pumps to send messages as fragments
::    rcv: per-bone message sinks to assemble messages from fragments
::    nax: unprocessed nacks (negative acknowledgments)
::         Each value is ~ when we've received the ack packet but not a
::         nack-trace, or an error when we've received a nack-trace but
::         not the ack packet.
::
::         When we hear a nack packet or an explanation, if there's no
::         entry in .nax, we make a new entry. Otherwise, if this new
::         information completes the packet+nack-trace, we remove the
::         entry and emit a nack to the local vane that asked us to send
::         the message.
::    heeds: listeners for %clog notifications
::    closing: bones closed on the sender side
::    corked:  bones closed on both sender and receiver
::
+$  peer-state
  $+  peer-state
  $:  $:  =symmetric-key
          =life
          =rift
          =public-key
          sponsor=ship
      ==
      route=(unit [direct=? =lane])
      =qos
      =ossuary
      snd=(map bone message-pump-state)
      rcv=(map bone message-sink-state)
      nax=(set [=bone =message-num])
      heeds=(set duct)
      closing=(set bone)
      corked=(set bone)
      keens=(map path keen-state)
  ==
```

### Structure <a href="#structure" id="structure"></a>

Ames’ formal interface is included more than once (like Arvo) as the “external vane interface” and the “adult ames”, for instance.

```hoon
++  call  :: handle request stack
++  take  :: handle response $sign
++  stay  :: extract state before reload
++  load  :: load in old state after reload
++  scry  :: dereference namespace
```

There is a collection of `ames-helper` cores as well to handle many specific cases for unpacking and routing messages. Ames uses a more sophisticated nested core pattern than Behn did. To that end, it presents five `+abet` cores:

* `ev` event handling core
* `mi` message receiver core
* `mu` message sender core
* `pe` per-peer processing core
* `pu` packet pump

There is a substantial amount of legacy Ames state upgrade debris in the file as well.

#### Scries

As typical, scries expose internal vane state. Ames has a richer inner life than some other vanes, so you can check on peer state and snubs and message flow details.

```hoon
.^((map ship ?(%alien %known)) %ax /=//=/peers)

.^(ship-state:ames %ax /=//=/peers/~zod)

!< message-pump-state:ames .^(vase %ax /=//=/snd-bones/~zod/0)
```

Most Ames scries aren't particularly useful to us directly unless we want to do direct network negotiation. Ames is used frequently by Gall but, from the agent's perspective, incidentally.

### Messages & Flows <a href="#messages-flows" id="messages-flows"></a>

We've looked at the packet protocol before; now let's look at Ames' mechanics of message management.

```hoon
+$  fragment       @uwfragment
+$  fragment-num   @udfragmentnum
+$  message-blob   @udmessageblob
+$  message-num    @udmessagenum
```

Messages are separated into 1 KB (or smaller) fragments and sequentially numbered.

* Examine `+split-message` to see how messages are broken up into pieces. (There's a neat optimization therein.)

Messages are of course sent and received in fragments. The messages from a `lane` accrue for a particular `bone` using `+mi`, the message receiver core (internal alias `sink`).

* Examine `+hear` and `+assemble-fragments`.
* Start the debug server (`|start %dbug`) and navigate to `/~debug`. Select `ames` to see message flows.

The message pump manages unsent messages, dispatching them to the packet pump when next in the queue.

> When we pop a message off .unsent-messages, we push as many fragments as we can into |packet-pump, which sends every packet it eats. Packets rejected by |packet-pump are placed in .unsent-fragments. When we hear a packet ack, we send it to |packet-pump to be removed from its queue of unacked packets. When we hear a message ack (positive or negative), we treat that as though all fragments have been acked.

There are a ton of other edge cases and consistency/sanity checks on messaging, one of the reasons that Ames is relatively complicated.

> At the end of a task, |message-pump sends a %halt task to |packet-pump, which can trigger a timer to be set or cleared based on congestion control calculations. When the timer fires, it will generally cause a packet to be re-sent. Message sequence numbers start at 1 so that the first message will be greater than .last-acked.message-sink-state on the receiver.

```hoon
+$  message-pump-state
  $+  message-pump-state
  $:  current=_`message-num`1
      next=_`message-num`1
      unsent-messages=(qeu message-blob)
      unsent-fragments=(list static-fragment)
      queued-message-acks=(map message-num ack)
      =packet-pump-state
  ==
::
+$  static-fragment
  $:  =message-num
      num-fragments=fragment-num
      =fragment-num
      =fragment
  ==
::
+$  partial-rcv-message
  $:  num-fragments=fragment-num
      num-received=fragment-num
      fragments=(map fragment-num fragment)
  ==
```

> A vane can pass Ames a `%heed` `task` to request Ames track a peer's responsiveness. If our `%boon`s to it start backing up locally, Ames will `give` a `%clog` back to the requesting vane containing the unresponsive peer's Urbit address.

```hoon
+$  qos
  $~  [%unborn *@da]
  [?(%live %dead %unborn) last-contact=@da]
```

To cork a flow (or “cork a bone”) closes the flow. A dangling bone refers to an incorrect bone (a message flow was closed on one side before all message fragments were received, for instance).

A message flow organizes a sequence of message fragments together. Within a flow, data order is guaranteed; however due to network traffic flows may arrive out of order.

```hoon
+$  bone           @udbone
::
+$  ossuary
  $:  =next=bone
      by-duct=(map duct bone)
      by-bone=(map bone duct)
  ==
```

A `$bone` is a duct handle, a way of identifying a particular message flow over the network.

Each `bone` increments by 4 since each flow includes a least-significant bit indicating if we send or receive pleas and a second-least-significant bit indicating if we are a diagnostic flow (naxplanation) or not.

```hoon
> *bone
0

> .^([snd=(set bone) rcv=(set bone)] %ax /=//=/bones/~nes)
[snd={0} rcv={}]
```

The `$ossuary` holds the bone↔duct bijection and the `next-bone` to map to a duct. (Thus the increment-by-four noted above.)

```hoon
::  $pump-metrics: congestion control state for a |packet-pump
::
::    This is an Ames adaptation of TCP's Reno congestion control
::    algorithm.  The information signals and their responses are
::    identical to those of the "NewReno" variant of Reno; the
::    implementation differs because Ames acknowledgments differ from
::    TCP's, because this code uses functional data structures, and
::    because TCP's sequence numbers reset when a peer becomes
::    unresponsive, whereas Ames sequence numbers only change when a
::    ship breaches.
::
::    A deviation from Reno is +fast-resend-after-ack, which re-sends
::    timed-out packets when a peer starts responding again after a
::    period of unresponsiveness.
::
::    If .skips reaches 3, we perform a fast retransmit and fast
::    recovery.  This corresponds to Reno's handling of "three duplicate
::    acks".
::
::    rto: retransmission timeout
::    rtt: roundtrip time estimate, low-passed using EWMA
::    rttvar: mean deviation of .rtt, also low-passed with EWMA
::    ssthresh: slow-start threshold
::    cwnd: congestion window; max unacked packets
::
+$  pump-metrics
  $:  rto=_~s1
      rtt=_~s1
      rttvar=_~s1
      ssthresh=_10.000
      cwnd=_1
      counter=@ud
  ==
```

### Vere I/O Driver: `vere/io/ames.c` <a href="#vere-io-driver-vereioamesc" id="vere-io-driver-vereioamesc"></a>

As elsewhere, the `libuv` event loop processor with callback functions responds to Ames-specific initiating events, in this case, the receipt of a UDP packet. The C side of Ames handles constructing and dispatching the UDP packets that underlie Ames communications, but perhaps surprisingly `ames.c` is actually less complicated and interesting than `ames.hoon`. (There's some serialization handling too.)

* `_ames_czar_cb()` for galaxy address resolution
* `_ames_send_cb()` for UDP transmission
* `_ames_recv_cb()` for UDP reception

## Network Protocol: Fine <a href="#network-protocol-fine" id="network-protocol-fine"></a>

A scry is a read-only request into the scry namespace. Historically, only local scries were supported, and these were instrumented synchronously using `.^` dotket. With the addition of remote scry, a new use case and use pattern emerged: asynchronous reads over the network.

> A ship that wants to read from a remote part of the namespace will have to pass a `%keen` task to its Ames, which then cooperates with Vere to produce the desired data. In some future event when the result is available, Ames gives it back as a `%tune` gift. From the requester's perspective, this is the entire default lifecycle of a remote scry request.

```hoon
::    Remote Scry Tasks
::
::    %keen: peek: [ship /vane/care/case/spur]
::    %yawn: cancel request from arvo
::    %wham: cancels all scry request from any vane
::
```

Fine maintains its own state, but other than having its own types its operation is not so different from Ames that we need to delve into it hear.

```hoon
+$  keen-state
  $+  keen-state
  $:  wan=((mop @ud want) lte)  ::  request packets, sent
      nex=(list want)           ::  request packets, unsent
      hav=(list have)           ::  response packets, backward
      num-fragments=@ud
      num-received=@ud
      next-wake=(unit @da)
      listeners=(set duct)
      metrics=pump-metrics
  ==
+$  want
  $:  fra=@ud
      =hoot
      packet-state
  ==
+$  have
  $:  fra=@ud
      meow
  ==
::
+$  meow  ::  response fragment
  $:  sig=@ux  ::  signature
      num=@ud  ::  number of fragments
      dat=@ux  ::  contents
  ==
::
+$  peep  ::  fragment request
  $:  =path
      num=@ud
  ==
::
+$  wail  ::  tagged request fragment
  $%  [%0 peep] :: unsigned
  ==
::
+$  roar  ::  response message
  (tale:pki:jael (pair path (unit (cask))))
::
+$  purr  ::  response packet payload
  $:  peep
      meow
  ==
```

(Having worked with remote scries some in userspace, I recommend tombstoning old endpoints when they are done being used.)

### Runtime Scry Dispatch <a href="#runtime-scry-dispatch" id="runtime-scry-dispatch"></a>

Remote scries are handled by the runtime rather than generating an Arvo event.

In `vere/io/ames.c`, a scry hashtable `sac_p` is created. `ames_hear()` decides whether to inject the packet into Arvo (Ames protocol) or handle in Vere (Fine protocol).

* `_fine_hear_request()` to receive a request
* `_fine_hear_response()` to receive the response
* `_fine_get_cache()`
* `_fine_put_cache()`

There are also provisions for Fine scry path length etc. therein.

* [“Guides: Remote Scry”](/build-on-urbit/userspace/remote-scry)
* [\~rovnys-ricfer, “Remote Scry Protocol Proposal”](https://gist.github.com/belisarius222/d9a9c164817d3e8bbda3c45f7d2000b9)


# 10. Vanes III: Eyre, Iris

Core Academy lesson on HTTP vanes covering the Eyre vane's HTTP server functionality, the Iris vane's HTTP client operations, web request/response patterns and authentication mechanisms.

*This lesson covers Arvo's HTTP server vane (Eyre) and its HTTP client vane, Iris. We discuss how Arvo interacts with clients over HTTP.*

## Servers <a href="#servers" id="servers"></a>

One of Urbit's primary use cases is to act as a “personal server”. To examine this statement, we need to consider what a server does. Etymologically, a server serves a service. Generally speaking, it is the locus of a computation and coordination process. A server program is a system daemon—and since Gall agents are essentially daemons in many respects, Urbit's execution model fulfills this niche nicely.

Some servers are physical or logical devices which talk to other devices as clients. Internet webpage and application servers typically follow this model. Other servers are software processes that run on the same hardware or local network as the client process, e.g. mail servers, print servers, and file servers.

![](https://storage.googleapis.com/media.urbit.org/core-academy/client-server.png)

The two major operational models for servers are the *request–response* model and the *publish–subscribe* (pub-sub) model. The request–response pattern corresponds to pokes and gifts in Arvo terms, while the pub-sub pattern is supplied by subscriptions and updates.

* A *client* originates and submits requests, and receives responses.
* A *server* accepts requests and replies with responses.

A client and a server need to agree on a communications protocol. There are many of these, but the basis for the World Wide Web is the HyperText Transfer Protocol (HTTP).

### Serving a Web Page <a href="#serving-a-web-page" id="serving-a-web-page"></a>

Two of the simplest actions one can take with a basic web server are to simply post a web page to any clients and to respond to interactions with that web page. Some interactions take place purely in the client session (form entry in the browser before submission), but then are propagated to the server.

#### Requests

HTTP requests are like Gall agent pokes: they are messages to trigger some action on the server. A method is specified (like `GET`, `PUT`, or `POST`) and the associated service-specific data follow.

It consists of a block of request headers, a block of general headers, and a block of representation headers. These may by followed by the body.

![](https://storage.googleapis.com/media.urbit.org/core-academy/http-request.png)

* `GET` means a read-only request for information (like an Urbit scry but without the bound namespace).
* `PUT` requests a state creation or update.
* `POST` asks for the server to process data. (Both `PUT` and `POST` are analogous to Urbit pokes.)

#### Responses

A [server response to a web page](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages) looks like this:

![](https://storage.googleapis.com/media.urbit.org/core-academy/http-response.png)

The [response code](https://en.wikipedia.org/wiki/HTTP_response_code) is normatively `200 OK` for a successful page access, but `404 Not Found` and other errors and special messages also occur frequently. (It would be very interesting if Urbit would implement `402 Payment requested`.)

The actual mechanics of communicating both kinds of communications are wrapped by `/lib/server` and Eyre. Generally speaking, an agent will receive HTTP requests in `+on-poke`, and commonly includes a `+handle-http` arm to deal with `inbound-request:eyre` values. `/lib/server` has request header parsers and response handlers which make it easy to respond appropriately (e.g. `(send:server ~ [%login-redirect './apps/my-agent'])`).

## Eyre <a href="#eyre" id="eyre"></a>

Eyre is an HTTP server, which receives HTTP messages from Unix and produces HTTP messages in reply. Your agent can register endpoints which a browser or other tool can interact with. Eyre can be instrumented to work with threads and generators.

HTTP requests include a method tag. While other methods exist, we are primarily interested in `POST`, `PUT`, and `GET` requests. Since we don't want to deal with client-side code yet, we're going to use `curl` to send requests here.

* `POST` is only used with Eyre to obtain a cookie.

```sh
$ curl -i localhost:8080/~/login -X POST -d "password=lidlut-tabwed-pillex-ridrup"
HTTP/1.1 204 ok
Date: Tue, 19 Jul 2022 16:28:05 GMT
Connection: keep-alive
Server: urbit/vere-1.9
set-cookie: urbauth-~nec=0v3.pis4a.sfdhv.f1p6i.lttba.gp93q; Path=/; Max-Age=604800
```

This cookie should be included in subsequent requests.

* `PUT` requests are used to send actions to Eyre: pokes, subscriptions, acks, unsubscribe requests, and channel deletions.
* `GET` requests are used to connect to a channel and receive any pending events. (Remember how Urbit prefers a dataflow computing model?)

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

```hoon
::                                                      ::::
::::                    ++eyre                            ::  (1e) http-server
  ::                                                    ::::
++  eyre  ^?
  |%
  +$  gift
    $%  $>(?(%boon %done) gift:ames)   ::  Ames responses
        [%set-config =http-config]   ::  configure external HTTP server
        [%sessions ses=(set @t)]     ::  valid auth cookies
        [%response =http-event:http]   ::  response to event from Earth
        [%bound accepted=? =binding]   ::  response to %connect or %serve
        [%grow =path]                ::  notification on cache entry change
    ==
  ::
  +$  task
    $~  [%vega ~]
    $%  $>(%init vane-task)        ::  initialize ourself with an identity
        $>(%born vane-task)        ::  new unix process
        $>(%plea vane-task)        ::  network request
        $>(%trim vane-task)        ::  trim state (memory pressure)
        $>(%vega vane-task)        ::  report upgrade
        ::
        [%live insecure=@ud secure=(unit @ud)]            ::  live HTTPS ports
        [%rule =http-rule]         ::  update HTTP configuration
        [%eauth-host host=(unit @t)]   ::  set base URL for eauth
        [%request secure=? =address =request:http]        ::  handle inbound
        [%request-local secure=? =address =request:http]  ::  handle backdoor
        [%cancel-request ~]        ::  cancel previous request
        [%connect =binding app=term]   ::  connects a binding to an app
        [%serve =binding =generator]   ::  connect binding to generator
        [%disconnect =binding]         ::  disconnect binding to generator
        ::
        [%code-changed ~]          ::  web login code changed
        [%approve-origin =origin]  ::  accept CORS requests from origin
        [%reject-origin =origin]   ::  reject CORS requests from origin
        [%spew veb=@]              ::  set verbosity
        [%set-response url=@t entry=(unit cache-entry)]   ::  cache mapping
    ==
--  ::eyre
```

Eyre is responsible for a few subsystems that facilitate userspace applications (unlike, say, Behn or Ames, most of what Eyre does is to support userspace).

* Authentication
* Channels
* Threads and generators
* HTTP request handling
* Scry interface

Several of these are handled by the `$action` dispatch system, invoked when a binding matches a known path.

> A `$binding` is a system unique mapping for a path to match. A `$binding` must be system unique because we don't want two handlers for a path; what happens if there are two different actions for `[~ /]`?

```hoon
::  +binding: A rule to match a path.
::
+$  binding
  $:  site=(unit @t)        ::  site: the site to match (~ for your.urbit.org)
      path=(list @t)        ::  path: matches this prefix path
  ==
::  +action: the action to take when a binding matches an incoming request
::
+$  action
  $%  [%gen =generator]     ::  dispatch to a generator
      [%app app=term]       ::  dispatch to an application
      [%authentication ~]   ::  internal authentication page
      [%eauth ~]            ::  cross-ship authentication handling
      [%logout ~]           ::  internal logout page
      [%channel ~]          ::  gall channel system
      [%scry ~]             ::  gall scry endpoint
      [%name ~]             ::  respond w/ @p requester is authenticated as
      [%host ~]             ::  respond w/ @p of the ship serving the response
      [%four-oh-four ~]     ::  respond with the default file not found page
  ==
```

`/sys/vane/eyre` is relatively more straightforward than (say) Ames. There is only one interface (search for `~% %http-server`).

#### Authentication

Client sessions typically require a login. (This is not true for materials served to the clearweb, e.g. via `%blog`.) A cookie is generated for each session in response to a login using `+code`.

```hoon
::  +authentication-state: state used in the login system
::
+$  authentication-state
  $:  sessions=(map @uv session)  ::  map of cookies to session information
      visitors=(map @uv visitor)  ::  visitors: in-progress incoming eauth flows
      visiting=(map ship logbook) ::  visiting: outgoing eauth state per ship
      endpoint=[user=(unit @t) auth=(unit @t) =time]  ::  endpoint: hardcoded 
                                                      ::  local eauth endpoint
                                                      ::  for %syn and %ack
  ==
```

(Visits are part of the EAuth system, *q.v.*)

Authentication is handled by `+authentication`.

* See `+authentication` in `/sys/vane/eyre`. Locate where the session cookie is created and logged.

Authentication is enforced by `+request-is-logged-in` and `+request-is-authenticated`.

* See `+request-is-logged-in` in `/sys/vane/eyre`.

#### Channels

> Channels are the main method where a webpage communicates with Gall apps. Subscriptions and pokes are issues with `PUT` requests on a path, while `GET` requests on that same path open a persistent `EventSource` channel. The `EventSource` API is a sequence number based API that browser provide which allow the server to push individual events to the browser over a connection held open. In case of reconnection, the browser will send a `'Last-Event-Id'` header to the server; the server then resends all events since then.

An [`EventSource`](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) interface is a way to track server-sent events for a client session. The JS on the browser/client-side receives `text/event-stream` formatted events. So a channel is a given connection to a browser including the `EventSource` connection.

```hoon
::  channel: connection to the browser
::
+$  channel
  $:  mode=?(%json %jam)
      =identity
      state=(each timer duct)
      next-id=@ud              ::  next-id: next sequence number to use
      last-ack=@da             ::  last-ack: time of last client ack
      events=(qeu [id=@ud request-id=@ud =channel-event])  ::  unacked events
      unacked=(map @ud @ud)    ::  unacked event counts by request-id
      subscriptions=(map @ud [ship=@p app=term =path duc=duct])  ::  gall subs
      heartbeat=(unit timer)   ::  sse heartbeat timer
  ==
::  +session: server side data about a session
::
+$  session
  $:  =identity                ::  authentication level & id of this session
      expiry-time=@da          ::  when this session expires
      channels=(set @t)        ::  channels opened by this session
  ==
::  channel-state: state used in the channel system
::
+$  channel-state
  $:  session=(map @t channel)   ::  mapping b/w an arbitrary key to a channel
      duct-to-key=(map duct @t)  ::  mapping from ducts to session key
  ==
::  channel-event: unacknowledged channel event, vaseless sign
::
+$  channel-event
  $%  $>(%poke-ack sign:agent:gall)
      $>(%watch-ack sign:agent:gall)
      $>(%kick sign:agent:gall)
      [%fact =desk =mark =noun]
  ==
```

Conventional channels communicate in JSON. Values passed into Urbit can be sent through a mark file to be transformed into a `%noun` or other type automatically. On the way out, a similar transformation can take the values back into MIME types.

* Trace `%poke` and `%poke-json` in `/sys/vane/eyre`.
* Examine [`/lib/schooner`](https://github.com/urbit/yard/blob/main/desk/lib/schooner.hoon) and the `/mar` files in [the `%yard` desk](https://github.com/urbit/yard). How does it handle JSON transformations? What about binary types like `audio/mpeg` (MP3)?

Noun channels make it possible for external applications to speak Urbit nouns. This means that you can communicate with an Urbit ship in a way other than using a JSON payload. The `content-type` is marked as `application/x-urb-jam`. Nouns are `+jam`med when sent into Eyre.

* Locate where `x-urb-jam` is processed in `/sys/vane/eyre` and in `/mar/noun`.

Thanks to the mark system and `+find-channel-mode`, it is straightforward on Urbit's side to implement noun channels. However, on the other side you need something that speaks nouns, such as [`noun.py`](https://github.com/urbit/tools/blob/master/pkg/pynoun/noun.py).

* [“Guide: Noun channels”](/urbit-os/kernel/eyre/noun-channels)
* [\~nordus- mocwyl, “bird-brained guide to noun channels”](https://bird.howm.art/guide/noun-channels)

#### HTTP request handling

All else set aside, the real purpose of Eyre is to act as the HTTP server for an Urbit ship. Eyre maintains a server configuration. There is an `$inbound-request` type to receive an HTTP request, but the main HTTP types are actually in another arm, `+http`.

A raw HTTP request handle happens like this:

* Eyre subscribes to an app at `/http-response/[eyre-id]`.
* Eyre pokes the app with `%handle-http-request` and the ID.
* The app produces `%fact`s of `?(%http-response-header %http-response-data %http-response-cancel)`.

```hoon
:: +http-config: full http-server configuration
::
+$  http-config
  $:  secure=(unit [key=wain cert=wain])  ::  PEM-encoded RSA private key and 
                                          ::  cert or cert chain
      proxy=_|                            ::  reverse TCP proxy HTTP(s)
      log=?                               ::  keep HTTP(s) access logs
      redirect=?               ::  send 301 redirects to upgrade HTTP to HTTPS
  ==
:: +http-rule: update configuration
::
+$  http-rule
  $%  [%cert cert=(unit [key=wain cert=wain])]  ::  set/clear cert and keypair
      [%turf action=?(%put %del) =turf]   ::  add/remove established dns binding
  ==
::  +address: client IP address
::
+$  address
  $%  [%ipv4 @if]
      [%ipv6 @is]
      ::  [%ames @p]
  ==
::  +inbound-request: +http-request and metadata
::
+$  inbound-request
  $:  authenticated=?          ::  has a valid session cookie
      secure=?                 ::  whether this request was encrypted (https)
      =address                 ::  the source address of this request
      =request:http            ::  the http-request itself
  ==
```

* Examine `+http`. Find the `request` and `response` handlers. In particular, see `$simple-payload`.
* Examine `/lib/server`, which contains wrapper arms for mere mortals.

In some ways, although this is the meat-and-potatoes of Eyre, it's all rather straightforward.

To actually get a value into userspace, Eyre sends the `response` to Gall:

* `+request-to-app` to dispatch an `%app` `$action` to Gall.
* `+deal-as` to `%pass` to Gall.

#### Threads and generators

We can treat a (local) ship as a “serverless” function call for a client.

A generator is a standalone computation based on arguments. Eyre supports generators explicitly:

```hoon
+$  generator
  $:  =desk           ::  desk on current ship that contains the generator
      path=(list @t)  ::  path on :desk to the generator's hoon file
      args=*          ::  arguments passed to the gate
  ==
```

and runs them in `+request`, branch `%gen`.

* How does the generator run? Note the `$roof` and the `+mock` call.

Having bound a generator `/gen/eyre-gen`

```hoon
|=  [[now=@da eny=@uvJ bec=beak] ~ ~]
|=  [authenticated=? =request:http]
^-  simple-payload:http
=/  msg=@t
  ?~  body.request
    (scot %da now)
  (cat 3 (cat 3 (scot %da now) 10) q.u.body.request)
=/  data=octs
  (as-octs:mimes:html msg)
=/  =response-header:http
  [200 ['Content-Type' 'text/plain']~]
[response-header `data]
```

to an endpoint `/mygen`,

```hoon
|pass [%e [%serve `/mygen %base /gen/eyre-gen/hoon ~]]
```

a client may initiate a generator call by posting a `PUT` request thus:

```sh
curl -i http://localhost:8080/mygen --data 'blah blah blah'
```

* Examine the [“Eyre Guide: Generators”](/urbit-os/kernel/eyre/guide#generators) example.

A thread is a transient standalone computation similar in some regards to a generator. Spider provides thread support using an Eyre binding.

```hoon
++  handle-http-request
  ~/  %handle-http-request
  |=  [eyre-id=@ta =inbound-request:eyre]
  ^-  (quip card _state)
  ?>  authenticated.inbound-request
  =/  url  (parse-request-line:server url.request.inbound-request)
  ?>  ?=([%spider @t @t @t @t ~] site.url)
  =*  desk         i.t.site.url
  =*  input-mark   i.t.t.site.url
  =*  thread       i.t.t.t.site.url
  =*  output-mark  i.t.t.t.t.site.url
  =/  =tid         (new-thread-id thread)
  =.  serving.state  (~(put by serving.state) tid [`eyre-id output-mark desk])
  =/  tube  (convert-tube %json input-mark desk bowl)
  ?>  ?=(^ body.request.inbound-request)
  =/  body=json  (need (de-json:html q.u.body.request.inbound-request))
  =/  input=vase  (slop !>(~) (tube !>(body)))
  =/  boc  bec
  =/  =start-args:spider  [~ `tid boc(q desk, r da+now.bowl) thread input]
  (handle-start-thread start-args)
```

* Examine the [“HTTP API”](/urbit-os/base/threads/http-api) example.

#### EAuth

Eyre's EAuth system “a mechanism by which HTTP clients may authenticate themselves as a specific urbit on HTTP endpoints served by any other urbit.” In other words, you can provide a comet-like client to an arbitrary client.

* How robust to collision is random EAuth assignment? ($\frac{1}{2^{128}-2^{64}} \approx 3^{-39}$, or one in 100 undecillion)
* If you are interested in investigating EAuth in detail, see `$visitor`, `$logbook`, `$eauth-plea`, `$eauth-boon` as well as the source description at [\~palfun-foslup, “mirage (eauth)”](https://gist.github.com/Fang-/41ed84b2a6dd96ca67c6a5dbda1fd35d) and the app [\~paldev, %chat-stream](https://pal.dev/lobby/).

### SSL <a href="#ssl" id="ssl"></a>

If you are working locally, you typically just have HTTP set up instead of secure HTTPS. SSL is a transport layer protocol formerly used for client–server encrypted channels, but now HTTPS actually uses TLS.

The `%acme` agent configures a certificate if you have a domain set up to use with Urbit.

* Read the `+install` arm in `/app/acme`.

#### Scry interface

Eyre exposes some information about bindings and connections, such as the sessions and cookies:

```hoon
.^(authentication-state:eyre %e /=authentication-state=)

.^((list [binding:eyre duct action:eyre]) %e /=bindings=)
```

### Vere I/O Driver: `vere/io/http.c` <a href="#vere-io-driver-vereiohttpc" id="vere-io-driver-vereiohttpc"></a>

The runtime counterpart to Eyre is `vere/io/http.c`, which is the HTTP server.

Vere's `http.c` uses `libh2o` as its HTTP server:

> H2O is a new generation HTTP server that provides quicker response to users with less CPU utilization when compared to older generation of web servers. Designed from ground-up, the server takes full advantage of HTTP/2 features including prioritized content serving and server push

Take especial note of the following functions:

* `u3_http_io_init()` to start the HTTP server manager.
* `_http_serv_listen_cb()`, the callback for receiving a value
* `_http_serv_accept()`
* `_http_seq_accept()` to process a new HTTP request
* `_http_hgen_send()` to send an HTTP response

## Iris <a href="#iris" id="iris"></a>

Iris is an HTTP client. It is not currently widely used since Urbit ships do not often serve as HTTP clients (rather as peers).

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

```hoon
::  %iris http-client interface
::
++  iris  ^?
  |%
  ::  +gift: effects the client can emit
  ::
  +$  gift
    $%  [%request id=@ud request=request:http]  ::  outbound http-request
        [%cancel-request id=@ud]                ::  tell earth to cancel a 
                                                ::  previous %request
        [%http-response =client-response]       ::  response to the caller
    ==
  ::
  +$  task
    $~  [%vega ~]
    $%  $>(%born vane-task)  ::  system started up; reset open connections
        $>(%trim vane-task)  ::  trim state (in response to memory pressure)
        $>(%vega vane-task)  ::  report upgrade
        [%request =request:http =outbound-config]  ::  fetches a remote resource
        [%cancel-request ~]  ::  cancels a previous fetch
        [%receive id=@ud =http-event:http]  ::  receives http data from outside
    ==
  ::  +client-response: one or more client responses given to the caller
  ::
  +$  client-response
    $%  $:  %progress        ::  periodic update along %fetch source duct
            =response-header:http     ::  full transaction header
            bytes-read=@ud            ::  bytes fetched so far
            expected-size=(unit @ud)  ::  size if response had content-length
            incremental=(unit octs)   ::  data received since last update
        ==
        [%finished =response-header:http full-file=(unit mime-data)] ::  final
                                                                     ::  response
        [%cancel ~]          ::  canceled by runtime system
    ==
  ::  mime-data: externally received but unvalidated mimed data
  ::
  +$  mime-data  [type=@t data=octs]
  ::  +outbound-config: configuration for outbound http requests
  ::
  +$  outbound-config
    $:  redirects=_5         ::  number of times to follow 300 before error
        retries=_3           ::  number of retries before failing
    ==
  --
```

### Structure <a href="#structure" id="structure"></a>

`/sys/vane/iris` is quite short and legible. Most of the vane is either tracking connection state as a client or sending updates for data transmission progress.

* Examine the [“Iris Guide: Example”](/urbit-os/kernel/iris/example) thread.
* Use this as a springboard for examining how the response header and body are constructed in Urbit.

Iris has relatively little information to expose at any given time, and has an extremely minimal scry interface:

* `%i %x %whey`, show memory usage.

### Vere I/O Driver: `vere/io/cttp.c` <a href="#vere-io-driver-vereiocttpc" id="vere-io-driver-vereiocttpc"></a>

The runtime counterpart to Iris is `vere/io/cttp.c`, which is the HTTP client.

* Read `u3_cttp_io_init()`, which initializes the client manager state.

Like Eyre, Vere's Iris uses [`libh2o`](https://powerdns.org/libh2o/) as its HTTP server/client library.

* Read `_cttp_creq_on_body()`, the callback upon receiving a response body
* `_cttp_creq_respond()`
* `_cttp_http_client_receive()`

## Exercise <a href="#exercise" id="exercise"></a>

* Produce an app which allows a clearweb login. This can be done using EAuth but it would be interesting to implement standard username/password login as well.


# 11. Vanes IV: Clay

Description of Clay, Arvo's revision-controlled filesystem vane. Also covers the build system, marks, scry cares, %kiln.

*This lesson covers Clay, Arvo's revision-controlled filesystem vane. We also look at the build system, marks, scry cares, software upgrades, and the `%kiln` agent.*

Clay is Urbit's version-controlled, referentially-transparent, globally-addressable filesystem. All data in Clay are typed and most are convertible to other types.

* *Version controlled*: Clay natively supports Git-like history, branching, merging, and checkpointing. A particular continuity (“desk”) is a series of numbered commits.
* *Referentially transparent*: for Clay, *referential transparency* means “a request must always yield the same result for all time.”
* *Globally addressable*: the standard resource identifier format includes the ship and time, which means that a fully-specified Clay path is similar to a [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier).
* *Persistent*: Clay inherits from Arvo that all events are persisted to disk.
* *Typed*: Clay attaches identification tags to any data and has ready to hand a set of conversion routines appropriate to the data type. These ID tags are called “marks,” and they act like MIME types.

To adequately address Clay we need to consider its quite varied functionality. To that end, we will not make an exposition for Clay per `/sys/lull` but instead break it up topically.

```hoon
::  clay (4c), revision control
::
::  The way to understand Clay is to take it section-by-section:
::
::  - Data structures.  You *must* start here; make sure you understand
::  the entire contents of +raft.
::
::  - Individual reads.  +aver is the entry point, follow it through
::  +read-at-tako to understand each kind of read.
::
::  - Subscriptions.  +wake is the center of this mechanism; nothing
::  else responds to subscriptions.  +wake has no arguments, which means
::  every subscription response happens when something in Clay's *state*
::  has changed.  No edge-triggered responses.
::
::  - Receiving foreign data.  For individual requests, this is
::  +take-foreign-answer.  For sync requests (%many, which is %sing %v
::  for a foreign desk), this is +foreign-update.
::
::  - Ford.  +ford builds hoon files and gives files their types.
::  Read +build-file for the first, and +read-file is the second.
::
::  - Writing to a desk.  Every write to a desk goes through +park, read
::  it thoroughly.
::
::  - Merges.  Control flow starts at +start-merge, then +merge, but
::  everything is scaffolding for +merge-by-germ, which is the ideal of
::  a merge function: it takes two commits and a merge strategy and
::  produces a new commit.
::
::  - Tombstoning.  This is in +tomb.
```

We will distribute the types as appropriate and organize these conceptually into our Core Academy approach:

1. File and desks
2. Subscriptions and desk distribution (including OTAs)
3. Marks, tubes, and `+ford`
4. Merges and desk writes
5. Move handler
6. The scry interface
7. Solid-state subscriptions

A warning: Clay is very old, and represents some of the darkest [jungle](https://www.youtube.com/watch?v=o1tj2zJ2Wvg) of Ye Olde Urbyt. The names and connexions are often obscure but we will bushwhack a trail through `/sys/vane/clay` together.

* [“Clay: Architecture”](/urbit-os/kernel/clay/architecture)

## Files and Desks <a href="#files-and-desks" id="files-and-desks"></a>

### Files <a href="#files" id="files"></a>

> What is a file system? It’s a way to pretend that a hard drive isn’t really a bunch of spinning magnetic platters that can store bits at certain locations, but rather a hierarchical system of folders-within-folders containing individual files that in turn consist of one or more strings of bytes. (Joel Spolsky, [“The Law of Leaky Abstractions”](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/))

What is a file? Per the context set by the quote above, it is a string of bytes at the level we wish to consider it. For all purposes in Urbit as a single-level store, we ignore file fragmentation and we try to ignore [endianness](https://en.wikipedia.org/wiki/Endianness) except in certain specific cases. The “hierarchical system of folders-within-folders” has little bearing on actual storage in Clay, as the identifying `path` is simply a tag. That tag is hashed and put into a lookup table, and when you request a resource Clay checks its `map` and produces the file as a noun for you.

Using a `%cx` scry, we can examine a particular resource on the `%base` desk as either a byte string or an ASCII text string:

```hoon
`@ux`.^(@ %cx /===/gen/cat/hoon)
.^(cord %cx /===/gen/cat/hoon)
```

A small fib in the above statement is that the path includes more than just an arbitrary “file path”. In particular, every file `path` really includes a `beak` at its head (as text). (There's some waffling here between values and `knot`s in various parts of the system, since a `path` is formally `(list knot)`.)

```hoon
+$  beak  [p=ship q=desk r=case]                      ::  path prefix
::
+$  beam  [[p=ship q=desk r=case] s=path]             ::  global name
::

+$  aeon  @ud                                         ::  version number

```

#### `beak` and `ship`

The top-level determiner for Clay is the `beak`: `(p=ship q=desk r=case)`. The ship is straightforward, merely the actual ship on which a resource resides. Clay is a globally-addressable filesystem so we can refer to a hypothetical resource on any ship. (This does not mean that the value actually exists, of course.)

```hoon
+$  ship  @p
```

#### `desk`

The second element of the `beak` is the `desk`. Clay organizes the world into `desk`s, which are also the most logical unit for app distribution. Essentially a desk is an organized collections of files with a common history.

```hoon
+$  desk  @tas
```

The structure of a desk matters for several parts of the Urbit system. Some parts are requirements: notably `/mar` which contains details on how to load the file resources as nouns, and its dependencies in `/sur` (structure files) and `/lib` (library files). By convention, the following are also present:

* `/app` (agent files, recognized by Gall)
* `/gen` (generators, recognized by Dojo)
* `/sys` (recognized by Arvo generally but only really active on `%base`)
* `/ted` (thread files, recognized by `%spider` and `%khan`)

(Keep in mind that inclusion of a file several times, like a mark file, need not lead to OS bloat since noun deduplication can store multiple references to a single identical resource.)

Some desks include `/tests` for unit tests. A few other agents like `%docs` use their own conventions as well (`/doc`).

#### `case`

```hoon
+$  case                                                                    
  $%  [%da p=@da]                                     ::  %da date
      [%tas p=@tas]                                   ::  %tas label
      [%ud p=@ud]                                     ::  %ud sequence
      [%uv p=@uv]                                     ::  %uv hash
  ==
```

Most commonly, the `case` is `now`, the timestamp which would refer to the file in its current state. Internally, Clay stores everything as a sequential value `aeon` in the `%ud` format and converts to equivalent `case`s when necessary (see e.g. `+aeon-to-tako`).

Any part of the `beak` can typically be replaced with `=` in a statement to get a default value. (Where in the parsers is this handled?)

```hoon
> /===
[~.~zod ~.base ~.~2022.7.17..23.50.01..3305 ~]

> /=landscape=
[~.~zod %landscape ~.~2022.7.17..23.50.05..9d0d ~]

> /~nec==
[~.~nec ~.base ~.~2022.7.17..23.50.13..dd5d ~]
```

### Desks and Commits <a href="#desks-and-commits" id="desks-and-commits"></a>

Why bother with the `case`? Enter the *commit*, which refers to a particular revision. A desk is a collection of commits as a particular continuity. (You can think of a desk as being like a Git branch.) Each commit, or `$yaki`, is a node in a historical state graph. It includes its parents and its namespace, as well links to any associated data to change. (While Clay can support file diffs, right now it does not have that feature turned on—it simply stores entire files; see `%info` below.)

```hoon
+$  yaki                                              ::  commit
  $:  p=(list tako)                                   ::  parents
      q=(map path lobe)                               ::  namespace
      r=tako                                          ::  self-reference
      t=@da                                           ::  date
  ==                                                  ::
::
+$  tako  @uvI                                        ::  yaki ref
+$  lobe  @uvI                                        ::  blob ref
```

In a while, we will look at how commits change desk state via merging, but we can punt on that question for now.

Knowing what commits are, we are finally prepared to examine desk state, the `$dome`:

```hoon
+$  dome
  $:  let=aeon                                        ::  top id
      hit=(map aeon tako)                             ::  versions by id
      lab=(map @tas aeon)                             ::  labels
      tom=(map tako norm)                             ::  tomb policies
      nor=norm                                        ::  default policy
      mim=(map path mime)                             ::  mime cache
      fod=flue                                        ::  ford cache
      wic=(map weft yoki)                             ::  commit-in-waiting
      liv=zest                                        ::  running agents
      ren=rein                                        ::  force agents on/off
  ==                                                  ::
```

* `let` is the version number at the latest point in the desk's history.
* `hit` is the collection of version numbers pointing to commit hashes to arrive at the current state.
* `lab` is the set of labels pointing to `aeon`s (for `case`).
* `tom` is the set of tombstone policies.
* `nor` is the default tombstone policy.
* `mim` is the MIME cache. Resources are converted to and from Unix frequently without changes so by caching we can sometimes short-circuit this.
* `fod` is `+ford`'s build cache.
* `wic` is the collection of commits pending to be applied (as in a system upgrade).
* `liv` is the list of agents currently active on this desk, derived from `desk.bill` and Gall.
* `ren` is the set of agents to force on or off.

### Notes & Gifts <a href="#notes-gifts" id="notes-gifts"></a>

There aren't many notes or gifts directly associated with file management since you often locally scry out individual files. (This is as opposed to desk management, which has ample moves associated therewith.)

```hoon
::                                                      ::::
::::                    ++clay                            ::  (1c) versioning
  ::                                                    ::::
++  clay  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%writ p=riot]                                  ::  response
        [%wris p=[%da p=@da] q=(set (pair care path))]  ::  many changes
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $%  [%warp wer=ship rif=riff]                       ::  internal file req
        [%werp who=ship wer=ship rif=riff-any]          ::  external file req
    ==                                                  ::
  --
```

Basically, a `%warp` request (to either a local or a foreign ship) may result in a `%writ` gift in response. (This does require setting permissions )

* [“`%warp` - Remote”](/urbit-os/kernel/clay/examples#foreign-ships)

Desk management has the following associated notes. (There are no gifts since per-desk information isn't exposed in the API this way.)

```hoon
::                                                      ::::
::::                    ++clay                            ::  (1c) versioning
  ::                                                    ::::
++  clay  ^?
  |%
  +$  task                                              ::  in request ->$
    $%  [%drop des=desk]                                ::  cancel pending merge
        [%info des=desk dit=nori]                       ::  internal edit
        [%into des=desk all=? fis=mode]                 ::  external edit
        $:  %merg                                       ::  merge desks
            des=desk                                    ::  target
            her=@p  dem=desk  cas=case                  ::  source
            how=germ                                    ::  method
        ==                                              ::
        $:  %fuse                                       ::  merge many
            des=desk                                    ::  target desk
            bas=beak                                    ::  base desk
            con=(list [beak germ])                      ::  merges
        ==                                              ::
        [%park des=desk yok=yoki ran=rang]              ::  synchronous commit
        [%pork ~]                                       ::  resume commit
        [%prep lat=(map lobe page)]                     ::  prime clay store
        [%rein des=desk ren=rein]                       ::  extra apps
        [%tire p=(unit ~)]                              ::  app state subscribe
        [%tomb =clue]                                   ::  tombstone specific
        [%zeal lit=(list [=desk =zest])]                ::  batch zest
        [%zest des=desk liv=zest]                       ::  live
    ==                                                  ::
  --
```

### Scries <a href="#scries" id="scries"></a>

The following scries specifically apply to files (single resources), commits, and desk-wide operations:

* [`%d` - List desks.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#d---list-desks)
* [`%p` - Return file permissions.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#p---file-permissions)
* [`%r` - Return file as vase.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#r---file-as-vase)
* [`%t` - List files.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#t---list-files)
* [`%u` - Check if resource exists.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#u---check-exists)
* [`%v` - Retrieve desk state.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#v---desk-state)
* [`%w` - Show revision number of a given `case`.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#w---revision-number)
* [`%x` - Read file.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#x---read-file)
* [`%y` - Read `arch` (file hash or directory contents).](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#y---read-arch)
* [`%z` - Show content hash.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#z---content-hash)

Most of the time you will use `%x` or `%y` from userspace, but in the kernel you may need more sophisticated information.

## Subscriptions and Desk Distribution <a href="#subscriptions-and-desk-distribution" id="subscriptions-and-desk-distribution"></a>

A desk can subscribe to another remote desk as its upstream, meaning that any changes on the remote are automatically propagated to subscribers. Apps are typically distributed this way (the alternative being an installation from source).

A subscription means that the upstream sponsor maintains a list of requested downstream sponsees in its `qyx.dojo` state for the appropriate desk. You can access this through a `%cx` `/cult` scry:

```hoon
.^((set [@p rave:clay]) %cx /=//=/cult/base)
```

The `%next` and `%mult` requests typically result from your own Gall agent wanting to know if the desk is updated (e.g. via a `|commit`). The `%sing` requests result from subscriptions and reads.

> `+wake` is the center of this mechanism; nothing else responds to subscriptions. `+wake` has no arguments, which means every subscription response happens when something in Clay's *state* has changed. No edge-triggered responses.

Desk distribution is largely the same for userspace app updates and system OTAs, except that userspace apps never require a system upgrade and do not have a separate `%kids` desk.

### OTAs <a href="#otas" id="otas"></a>

The most important desk subscription is your `%base` desk to your sponsor's `%kids` desk. This is how Urbit OS updates (over-the-air updates or OTAs) are propagated. The lifecycle of an OTA:

1. The sponsor syncs her `%base` desk to her `%kids` desk. (This takes place via a `%merg` task.) See [`MAINTAINERS.md`](https://github.com/urbit/urbit/blob/develop/MAINTAINERS.md) for one procedure.

* Why have a separate `%kids` desk?

2. The sponsor's `%kids` desk notifies all of its subscribers/sponsees (in `qyx.dojo`). (This notification comes over Ames to Clay.)
3. The sponsee receives the files via remote scry.
4. Once these have arrived, then the remote update is applied via a merge. (See `+apply-foreign-update`, in particular the definition of `nut` and `hut`.) See the discussion of merges below.
5. If a system update (to Hoon, Arvo, or the vanes) is involved, then handle the OTA as discussed in [*Arvo II: The Boot Sequence*](/build-on-urbit/core-academy/ca04). This may involve simply recompiling the vanes and migrating the state, or it may require updating everything all the way back to `/sys/hoon` including a worklist. (In this latter case, see `+sys-update`, `+park` (including the `kelvin` check), and how the `%pork` `%slip` is managed.)
6. This should unblock any desks that are blocked on the system kelvin version. See `+goad` and `+wick`.
7. If this ship has any sponsees, propagate the OTA to the `%kids` desk and thence forward to the daughter points.
8. If the state of the desk is requested, calculate and produce values like the `%cz` hash. (See `/gen/vats` and `/sur/hood`'s `+report-prep` and `+report-vat`, and `/sys/clay`'s `+content-hash`, for instance.)

## Merges and desk writes <a href="#merges-and-desk-writes" id="merges-and-desk-writes"></a>

A commit describes a set of changes to be made to a desk to result in a new desk state. Since desks are fundamentally collections of path-addressed resources, this is functionally similar to a regular Git-style version control system.

Thus we need to be able to produce commits (`$yaki`s) and apply them. This process is called a merge. A merge means that we have to decide how to reconcile two claims about reality into one. This results in several possible merge types in Clay. Most of the time you'll use `%init` implicitly (via `|new-desk`, for instance) or `%only-that` when you're trying to fix a desk mismatch.

These are ultimately concerned with reconciliation strategies involving commit types:

```hoon
+$  miso                                              ::  file delta
  $%  [%del ~]                                        ::  delete
      [%ins p=cage]                                   ::  insert
      [%dif p=cage]                                   ::  mutate from diff
      [%mut p=cage]                                   ::  mutate from raw
  ==                                                  ::
+$  soba  (list [p=path q=miso])                      ::  delta
::
+$  misu                                              ::  computed delta
  $%  [%del ~]                                        ::  delete
      [%ins p=cage]                                   ::  insert
      [%dif p=lobe q=cage]                            ::  mutate from diff
  ==                                                  ::
+$  suba  (list [p=path q=misu])                      ::  delta
::
+$  nori                                              ::  repository action
  $%  [%& p=soba]                                     ::  delta
      [%| p=@tas q=(unit aeon)]                       ::  label
  ==                                                  ::
+$  nuri                                              ::  repository action
  $%  [%& p=suba]                                     ::  delta
      [%| p=@tas]                                     ::  label
  ==                                                  ::
::
+$  mizu  [p=@u q=(map @ud tako) r=rang]              ::  new state
+$  moar  [p=@ud q=@ud]                               ::  normal change range
+$  moat  [from=case to=case =path]                   ::  change range
```

### Operations <a href="#operations" id="operations"></a>

#### Reads

`+aver` scaffolds read requests (`%sing`s); see also `+read-at-tako` (which is why we needed to see commit logic before we could really examine `+read-x`).

#### Changes

A change to a desk can originate from at least three sources:

1. Unix, via a mounted desk.
2. Userspace editing.
3. Update from a remote desk.

```hoon
::                                                      ::::
::::                    ++clay                            ::  (1c) versioning
  ::                                                    ::::
++  clay  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%mere p=(each (set path) (pair term tang))]    ::  merge result
        [%writ p=riot]                                  ::  response
        [%wris p=[%da p=@da] q=(set (pair care path))]  ::  many changes
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $%  [%drop des=desk]                                ::  cancel pending merge
        [%info des=desk dit=nori]                       ::  internal edit
        [%into des=desk all=? fis=mode]                 ::  external edit
        $:  %merg                                       ::  merge desks
            des=desk                                    ::  target
            her=@p  dem=desk  cas=case                  ::  source
            how=germ                                    ::  method
        ==                                              ::
        $:  %fuse                                       ::  merge many
            des=desk                                    ::  target desk
            bas=beak                                    ::  base desk
            con=(list [beak germ])                      ::  merges
        ==                                              ::
    ==                                                  ::
  --  ::clay
```

We consider these first as single-file updates (commits), then take a look at the merge process.

To modify a file, we must produce an `%info` write task. This requires a desk label and a `$nori` or repository action. In the case of writing a new file, the `$nori` looks like this:

```hoon
[%& ~[[/blade/runner/txt %ins %txt !>(~['Batty' 'Pris' 'Zhora' 'Leon'])]]]
```

passed into Clay like this:

```hoon
|pass [%c %info %base [%& ~[[/blade/runner/txt %ins %txt !>(~['Batty' 'Pris' 'Zhora' 'Leon'])]]]]
```

(A text file in Urbit is a `(list cord)` not a `cord`, per the `%txt` mark.)

![](https://media.urbit.org/docs/arvo/clay/ins.png)

* Trace how the `%info` task is dispatched into Clay: `+call`→`%info`→the worklist→`+abet`.

Merging desks is a more sophisticated operation, since it involves reconciling both current state and the parent commits. There are many ways to reconcile two versions of a desk:

```hoon
+$  germ                                              ::  merge style
  $?  %init                                           ::  new desk
      %fine                                           ::  fast forward
      %meet                                           ::  orthogonal files
      %mate                                           ::  orthogonal changes
      %meld                                           ::  force merge
      %only-this                                      ::  ours with parents
      %only-that                                      ::  hers with parents
      %take-this                                      ::  ours unless absent
      %take-that                                      ::  hers unless absent
      %meet-this                                      ::  ours if conflict
      %meet-that                                      ::  hers if conflict
  ==                                                  ::
```

> Control flow starts at `+start-merge`, then `+merge`, but everything is scaffolding for `+merge-by-germ`, which is the ideal of a merge function: it takes two commits and a merge strategy and produces a new commit.

The mechanics of the merge are handled by `+merge-by-germ`. For instance, an `%only-this` merge follows this rule:

> If this is an `%only-this` merge, we check to see if `ali`'s and `bob`'s commits are the same, in which case we're done. Otherwise, we create a new commit with `bob`'s data plus `ali` and `bob` as parents.

```hoon
++  merge-by-germ
  |=  [=ali=yaki bob-yaki=(unit yaki)]
  ^-  (unit merge-result)
  ?+    germ
      %only-this
    ?:  =(r.ali-yaki r.bob-yaki)
      ~
    :*  ~
        conflicts=~
        new=&+[[r.bob-yaki r.ali-yaki ~] (to-yuki q.bob-yaki)]
        lat=~
    ==
  --
```

* [“Merge Desks”](/urbit-os/kernel/clay/tasks#merge-desks)
* [“`%merg`”](/urbit-os/kernel/clay/examples#merg)

### Types <a href="#types" id="types"></a>

```hoon
+$  cone  (map [ship desk] dome)                      ::  domes
+$  crew  (set ship)                                  ::  permissions group
+$  dict  [src=path rul=real]                         ::  effective permission
+$  domo                                              ::  project state
  $:  let=@ud                                         ::  top id
      hit=(map @ud tako)                              ::  changes by id
      lab=(map @tas @ud)                              ::  labels
  ==                                                  ::
+$  germ                                              ::  merge style
  $?  %init                                           ::  new desk
      %fine                                           ::  fast forward
      %meet                                           ::  orthogonal files
      %mate                                           ::  orthogonal changes
      %meld                                           ::  force merge
      %only-this                                      ::  ours with parents
      %only-that                                      ::  hers with parents
      %take-this                                      ::  ours unless absent
      %take-that                                      ::  hers unless absent
      %meet-this                                      ::  ours if conflict
      %meet-that                                      ::  hers if conflict
  ==                                                  ::
+$  mode  (list [path (unit mime)])                   ::  external files
+$  mood  [=care =case =path]                         ::  request in desk
+$  mool  [=case paths=(set (pair care path))]        ::  requests in desk
+$  norm  (axal ?)                                    ::  tombstone policy
+$  open  $-(path vase)                               ::  get prelude
+$  page  ^page                                       ::  export for compat
+$  rang                                              ::  repository
  $+  rang
  $:  hut=(map tako yaki)                             ::  changes
      lat=(map lobe page)                             ::  data
  ==                                                  ::
+$  rant                                              ::  response to request
  $:  p=[p=care q=case r=desk]                        ::  clade release book
      q=path                                          ::  spur
      r=cage                                          ::  data
  ==                                                  ::
+$  rave                                              ::  general request
  $%  [%sing =mood]                                   ::  single request
      [%next =mood]                                   ::  await next version
      [%mult =mool]                                   ::  next version of any
      [%many track=? =moat]                           ::  track range
  ==                                                  ::
+$  real                                              ::  resolved permissions
  $:  mod=?(%black %white)                            ::
      who=(pair (set ship) (map @ta crew))            ::
  ==                                                  ::
+$  regs  (map path rule)                             ::  rules for paths
+$  rein  (map dude:gall ?)                           ::  extra apps
+$  riff  [p=desk q=(unit rave)]                      ::  request+desist
+$  riff-any                                          ::
  $%  [%1 =riff]                                      ::
  ==                                                  ::
+$  rite                                              ::  new permissions
  $%  [%r red=(unit rule)]                            ::  for read
      [%w wit=(unit rule)]                            ::  for write
      [%rw red=(unit rule) wit=(unit rule)]           ::  for read and write
  ==                                                  ::
+$  riot  (unit rant)                                 ::  response+complete
+$  rule  [mod=?(%black %white) who=(set whom)]       ::  node permission
+$  rump  [p=care q=case r=@tas s=path]               ::  relative path
+$  saba  [p=ship q=@tas r=moar s=dome]               ::  patch+merge
+$  toro  [p=@ta q=nori]                              ::  general change
++  unce                                              ::  change part
  |*  a=mold                                          ::
  $%  [%& p=@ud]                                      ::  skip[copy]
      [%| p=(list a) q=(list a)]                      ::  p -> q[chunk]
  ==                                                  ::
++  urge  |*(a=mold (list (unce a)))                  ::  list change
+$  waft                                              ::  kelvin range
  $^  [[%1 ~] p=(set weft)]                           ::
  weft                                                ::
+$  whom  (each ship @ta)                             ::  ship or named crew
+$  zest  $~(%dead ?(%dead %live %held))              ::  how live
::                                                    ::
```

## Building Code: `+ford` & Marks <a href="#building-code-ford-marks" id="building-code-ford-marks"></a>

Clay is responsible for assembling and building code. Building code differs from compiling code in that Clay's `+ford` arm must collect associated cores and code (referenced via `/` fas runes) and produce the appropriate Hoon source for `+ride` and friends to process into executable Nock. (The former standalone `%ford` vane was merged into `%clay` via [Ford Fusion](https://urbit.org/blog/ford-fusion) in 2020.)

Since Clay receives updates as source from remote desks, Clay is the *de facto* prime mover for internal state upgrades.

* `/sys/hoon` is stateless, so when it is updated it takes place first and just passes the worklist into the new world.
* `/sys/arvo` does maintain state, so the current state must be extracted and passed into the newly built program.
* `/sys/zuse` is stateless.
* Vanes are stateful and like Arvo may have a larval phase if necessary. (Notably Gall has one.)
* Userspace apps can then be updated by Gall using their `+on-save` and `+on-load` arms.

Ford produces several kinds of results, but these may be grouped into file-related types and mark-related types:

```hoon
+$  pour                                              ::  ford build w/content
  $%  [%file =path]
      [%nave =mark]
      [%dais =mark]
      [%cast =mars]
      [%tube =mars]
      ::  leafs
      ::
      [%vale =path =lobe]
      [%arch =path =(map path lobe)]
  ==
+$  soak                                              ::  ford result
  $%  [%cage =cage]
      [%vase =vase]
      [%arch dir=(map @ta vase)]
      [%dais =dais]
      [%tube =tube]
  ==
```

### File Builds <a href="#file-builds" id="file-builds"></a>

To see an example of how `+ford` works, trace the `%a` care:

* `+scry` → `%a`
* `+read-a`
* `+tako-ford` → `+tako-to-yaki`
* `+build-file` → `+build-dependency` (note the `+slap`) → `+read-file` → `+run-dependency` → `+parse-pile`
* Also see `+build-fit` for paths, noting how it handles `-`/`/`.

Files are built by `+ford` in vase mode. Arvo (Gall, etc.) can then drop them back into static mode once it has the core.

> Since building a file is a pure function, Clay memoizes the results of all builds, including builds of marks, mark conversions, and hoon source files. These memoization results are stored along with the desk and are used by later revisions of that desk.

Ford supplies several `/` fas “runes” to build code. (Formally these are not part of Hoon and are more aking to C's `#include` statements.) These are processed in `+parse-pile` → `+pile-rule`.

* `/?` faswut, pin kelvin version (currently ignored)
* `/-` import `/sur` files
* `/+` import `/lib` files
* `/=` arbitrary path to file
* `/~` arbitrary path to directory
* `/%` build and import mark core
* `/$` import mark conversion gate
* `/*` import file via specific mark

**Build cares, Part I**

* [`%a` - Build Hoon code.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#a---build-hoon)

### Marks <a href="#marks" id="marks"></a>

One of the roles of Clay in managing desks is to validate desk content. That is, does every resource in a desk have a definite way to convert to a noun representation (or to another representation, but at minimum to `%noun`)?

> If Clay has been asked to perform a commit, it needs to validate all the files in this desk and notify all subscribers to live queries of this desk's data. Gall, for example, maintains live queries on builds of its live agents. Validation uses the Ford build system.

A conventional DVCS filesystem like Git has special rules for handling text v. binary blob elements, Clay encourages the use of marks to identify filesystem data type and conversion routines. “It’s best defined as a symbolic mapping from a filesystem to a schema engine.” It's much like a [MIME type](https://en.wikipedia.org/wiki/Media_type), which specifies an intended data format (in a manner similar to a file extension).

> A mark is "like an executable MIME type." It's best defined as a symbolic mapping from a filesystem to a schema engine.

(You should get used to divorcing the conceptual relationship of data—what we could call it’s *form* in the Platonic sense or the *noun* in the Martian sense—from it’s *representation* or *instantiation*. For instance, one writes a JSON file a certain way in text, but when parsing it needs to think about it at a higher level of abstraction.)

A mark is a validated data structure, including rules for transformation between representations. In this regard, it is like a more rigorous file type. We frequently use marks in Gall agents to verify classes of operations (such as actions or updates) or to convert incoming data (such as via the JSON mark).

> Consider a file at `/web/foo/json`. In order to validate this file, Clay must load the mark definition core and use its validation routine to ensure the untyped value of `/web/foo/json` is in fact valid JSON. To obtain this core, Clay must build the file at `/mar/json/hoon` from source and then process the resulting raw mark core using some mild metaprogramming to get a standard interface core for dealing with marks, called a `$dais`, whose type is defined in Zuse. Since building a source file only makes sense if the file has been validated as a `%hoon` file, but mark definitions themselves must be built from source, there's a logical dependency cycle -- who validates the validators? To break this cycle, Clay hard-codes the validation of `%hoon` files. \[(See `+read-x` in `/sys/clay`.)] This allows mark definitions to be built from source, and in fact any file can depend on any other file of any mark as long as there are no cycles. As of Ford Fusion, Ford performs a cycle check to ensure acyclicity.

At a high level, files are validated using `+read-file`, which uses marks via `+validate-page`. So let's dive into marks.

The simplest way to use a mark is to simply supply Dojo with the names for source and target along with a value:

```hoon
&json &mime [/application/json (as-octs:mimes:html '"hey"')]
```

(In one sense, a mark is simply a label which nominally corresponds to a `/mar` file—but it is possible to have `cage`s that don't ever touch the filesystem.)

Marks expose several arms for converting between value representations:

* `+grab` cores convert *to* our `mark` *from* other `mark`s.
* `+grow` cores convert *from* our `mark` *to* another `mark`.
* `+grad` specify functions for revision control like creating diffs, patching files and so on. In our case, rather than writing all those functions, we've just delegated those tasks to the `%noun` `mark`.

To convert from mark `%alfa` to mark `%bravo`, Clay tries the following operations, in order:

* direct grow from `%alfa`
* direct grab from `%bravo`
* indirect jump from `%alfa` through `%charlie`
* indirect grab from `%bravo` through `%charlie`

You can see this logic instantiated in `+build-cast`.

* Construct a multi-step conversion between two marks that cannot grab/grow into each other (likely via `%noun`).

Note that marks don't have to perfectly round-trip: if you converted a `wain` to `json` back to `wain`, you won't necessarily have the same text.

* Examine the mark file `/mar/tape/hoon`.
* Examine the mark file `/mar/xml/hoon`.

As practically constructed, marks are typically either simple calls to outsource to other marks and `/sur` type validation, or they may involve JSON reparsing or construction. Only rarely do more complicated marks need to be built.

Marks can be built (using the right cares) to be either static or dynamic.

Static `mark` conversion gates only convert from one type directly to another. These have type `$-(from to)`.

```hoon
> =txt-to-mime .^($-(wain mime) %cf /===/txt/mime)

> (txt-to-mime ~['foo'])
[p=/text/plain q=[p=3 q=7.303.014]]
```

* See `+read-f` and `+build-nave`.

Static `mark` cores (`$nave`s) are more flexible than `%f` gates because they also supply the `+grad` arm to apply diffs.

```hoon
::  $nave: typed mark core
::
++  nave
  |$  [typ dif]
  $_
  ^?
  |%
  ++  diff  |~([old=typ new=typ] *dif)
  ++  form  *mark
  ++  join  |~([a=dif b=dif] *(unit (unit dif)))
  ++  mash
    |~  [a=[ship desk dif] b=[ship desk dif]]
    *(unit dif)
  ++  pact  |~([typ dif] *typ)
  ++  vale  |~(noun *typ)
  --
```

* See `+read-e` and `+build-tube`.

Dynamic `mark` conversion gates, or `$tube`s, process on `vase`s instead.

```hoon
::  $tube: mark conversion gate
::
+$  tube  $-(vase vase)
```

```hoon
> =txt-mime-tube .^(tube:clay %cc /===/txt/mime)

> !<  mime  (txt-mime-tube !>(~['foo']))
[p=/text/plain q=[p=3 q=7.303.014]]
```

* See `+read-c` and `+build-tube`.

Finally, dynamic `mark` cores (`$dais`) are the most powerful of all: they are doors operating in vase mode on files.

```hoon
::  $dais: processed mark core
::
+$  dais
  $_  ^|
  |_  sam=vase
  ++  diff  |~(new=_sam *vase)
  ++  form  *mark
  ++  join  |~([a=vase b=vase] *(unit (unit vase)))
  ++  mash
    |~  [a=[ship desk diff=vase] b=[ship desk diff=vase]]
    *(unit vase)
  ++  pact  |~(diff=vase sam)
  ++  vale  |~(noun sam)
  --
```

* See `+read-b` and `+build-dais`.
* [Curtis Yarvin \~sorreg-namtyv, “Towards a New Clay”](https://urbit.org/blog/toward-a-new-clay/) (very old content but interesting)
* [“Using Marks”](/urbit-os/kernel/clay/marks/using-marks)

**Build cares, Part II**

* [`%b` - Produce dynamic mark core.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#b---dyn-mark-core)
* [`%c` - Produce dynamic mark conversion gate.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#c---dyn-mark-convert)
* [`%e` - Produce static mark core.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#e---static-mark-core)
* [`%f` - Produce static mark conversion gate.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#f---stat-mark-convert)

## Clay as a Vane <a href="#clay-as-a-vane" id="clay-as-a-vane"></a>

Now we're ready to have a gander at the formal vane state.

```hoon
::  Formal vane state.
::
::  --  `rom` is our domestic state.
::  --  `hoy` is a collection of foreign ships where we know something about
::      their clay.
::  --  `ran` is the object store.
::  --  `mon` is a collection of mount points (mount point name to urbit
::      location).
::  --  `hez` is the unix duct that %ergo's should be sent to.
::  --  `cez` is a collection of named permission groups.
::  --  `pud` is an update that's waiting on a kernel upgrade
::
+$  raft                                                ::  filesystem
  $:  rom=room                                          ::  domestic
      hoy=(map ship rung)                               ::  foreign
      ran=rang                                          ::  hashes
      fad=flow                                          ::  ford cache
      mon=(map term beam)                               ::  mount points
      hez=(unit duct)                                   ::  sync duct
      cez=(map @ta crew)                                ::  permission groups
      tyr=(set duct)                                    ::  app subs
      tur=rock:tire                                     ::  last tire
      pud=(unit [=desk =yoki])                          ::  pending update
      sad=(map ship @da)                                ::  scry known broken
      bug=[veb=@ mas=@]                                 ::  verbosity
  ==                                                    ::
```

* `$room` is the domestic desk state.

```hoon
::
::  Domestic ship.
::
::  `hun` is the duct to dill, and `dos` is a collection of our desks.
::
+$  room                                                ::  fs per ship
          $:  hun=duct                                  ::  terminal duct
              dos=(map desk dojo)                       ::  native desk
          ==                                            ::
::
::  Domestic desk state.
::
::  Includes subscriber list, dome (desk content), possible commit state (for
::  local changes), possible merge state (for incoming merges), and permissions.
::
+$  dojo
  $:  qyx=cult                                          ::  subscribers
      dom=dome                                          ::  desk state
      per=regs                                          ::  read perms per path
      pew=regs                                          ::  write perms per path
      fiz=melt                                          ::  state for mega merges
  ==
```

### Move handler <a href="#move-handler" id="move-handler"></a>

There are several engine cores embedded in `/sys/clay`:

* `+de` desk engine to modify the desk (commits, merges, etc.) and metadata about the desk
* `+ze` utility engine to manipulate desk state itself
* `+lu` userspace agent management engine
* `+me` merge management core

The formal Arvo interface is located at `section 4cA, filesystem vane`. It is rather complex compared to the smaller vanes, and deserves a scan through the main arms.

The complete set of moves for Clay are:

```hoon
::                                                      ::::
::::                    ++clay                            ::  (1c) versioning
  ::                                                    ::::
++  clay  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%boon payload=*]                               ::  ames response
        [%croz rus=(map desk [r=regs w=regs])]          ::  rules for group
        [%cruz cez=(map @ta crew)]                      ::  permission groups
        [%dirk p=@tas]                                  ::  mark mount dirty
        [%ergo p=@tas q=mode]                           ::  version update
        [%hill p=(list @tas)]                           ::  mount points
        [%done error=(unit error:ames)]                 ::  ames message (n)ack
        [%mere p=(each (set path) (pair term tang))]    ::  merge result
        [%ogre p=@tas]                                  ::  delete mount point
        [%rule red=dict wit=dict]                       ::  node r+w permissions
        [%tire p=(each rock:tire wave:tire)]            ::  app state
        [%writ p=riot]                                  ::  response
        [%wris p=[%da p=@da] q=(set (pair care path))]  ::  many changes
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  [%boat ~]                                       ::  pier rebooted
        [%cred nom=@ta cew=crew]                        ::  set permission group
        [%crew ~]                                       ::  permission groups
        [%crow nom=@ta]                                 ::  group usage
        [%drop des=desk]                                ::  cancel pending merge
        [%info des=desk dit=nori]                       ::  internal edit
        $>(%init vane-task)                             ::  report install
        [%into des=desk all=? fis=mode]                 ::  external edit
        $:  %merg                                       ::  merge desks
            des=desk                                    ::  target
            her=@p  dem=desk  cas=case                  ::  source
            how=germ                                    ::  method
        ==                                              ::
        $:  %fuse                                       ::  merge many
            des=desk                                    ::  target desk
            bas=beak                                    ::  base desk
            con=(list [beak germ])                      ::  merges
        ==                                              ::
        [%mont pot=term bem=beam]                       ::  mount to unix
        [%dirk pot=term]                                ::  mark mount dirty
        [%ogre pot=$@(term beam)]                       ::  delete mount point
        [%park des=desk yok=yoki ran=rang]              ::  synchronous commit
        [%perm des=desk pax=path rit=rite]              ::  change permissions
        [%pork ~]                                       ::  resume commit
        [%prep lat=(map lobe page)]                     ::  prime clay store
        [%rein des=desk ren=rein]                       ::  extra apps
        [%stir arg=*]                                   ::  debug
        [%tire p=(unit ~)]                              ::  app state subscribe
        [%tomb =clue]                                   ::  tombstone specific
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
        [%warp wer=ship rif=riff]                       ::  internal file req
        [%werp who=ship wer=ship rif=riff-any]          ::  external file req
        [%wick ~]                                       ::  try upgrade
        [%zeal lit=(list [=desk =zest])]                ::  batch zest
        [%zest des=desk liv=zest]                       ::  live
        $>(%plea vane-task)                             ::  ames request
    ==                                                  ::
```

### The scry interface <a href="#the-scry-interface" id="the-scry-interface"></a>

#### Scries

Clay has more cares than any other vane because it needs to store and build Hoon code, as well as handle resource transformation using marks. We have already incidentally run into many of these scries, but for the sake of summary:

```hoon
+$  care                                              ::  clay submode
  $?  %a  %b  %c  %d  %e  %f                          ::
      %p  %q  %r  %s  %t  %u                          ::
      %v  %w  %x  %y  %z                              ::
  ==                                                  ::
```

**Resource cares**

* [`%d` - List desks.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#d---list-desks)
* [`%p` - Return file permissions.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#p---file-permissions)
* [`%r` - Return file as vase.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#r---file-as-vase)
* [`%t` - List files.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#t---list-files)
* [`%u` - Check if resource exists.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#u---check-exists)
* [`%v` - Retrieve desk state.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#v---desk-state)
* [`%w` - Show revision number of a given `case`.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#w---revision-number)
* [`%x` - Read file.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#x---read-file)
* [`%y` - Read `arch` (file hash or directory contents).](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#y---read-arch)
* [`%z` - Show content hash.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#z---content-hash)

**Build cares**

* [`%a` - Build Hoon code.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#a---build-hoon)
* [`%b` - Produce dynamic mark core.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#b---dyn-mark-core)
* [`%c` - Produce dynamic mark conversion gate.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#c---dyn-mark-convert)
* [`%e` - Produce static mark core.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#e---static-mark-core)
* [`%f` - Produce static mark conversion gate.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#f---stat-mark-convert)

**System care**

* [`%s` - Perform miscellaneous internal scries.](https://github.com/urbit/docs.urbit.org/blob/master/reference/arvo/clay/scry/README.md#s---misc-scries)

At this point, `%s` is the only new scry that we haven't looked at yet. (Clay unlike some other vanes certainly has a full complement of convenience scries.)

```hoon
> =/  =dome:clay  .^(dome:clay %cv %)
  =/  =tako:clay  (~(got by hit.dome) let.dome)
  .^(tako:clay %cs %/hash/(scot %uv tako))
0v16.er7uq.oke4u.cru7u.nglu9.q3su7.6ub1o.bh4qk.r5uav.ut12d.5rdl5
```

## `|mount` & `unix.c` <a href="#mount-unixc" id="mount-unixc"></a>

Urbit maintains its own single-level store including Clay via the runtime, but supports synchronizing Clay's vision with the underlying host OS. To mount a drive in this sense means to make a Unix-visible copy in the pier; the more recently timestamped of two files is considered the canonical instance.

```hoon
::                                                      ::::
::::                    ++clay                            ::  (1c) versioning
  ::                                                    ::::
++  clay  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%dirk p=@tas]                                  ::  mark mount dirty
        [%hill p=(list @tas)]                           ::  mount points
        [%ogre p=@tas]                                  ::  delete mount point
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $%  [%boat ~]                                       ::  pier rebooted
        [%mont pot=term bem=beam]                       ::  mount to unix
        [%dirk pot=term]                                ::  mark mount dirty
        [%ogre pot=$@(term beam)]                       ::  delete mount point
    ==                                                  ::
```

Mount point information is stored in the `$raft` at `mon=(map term beam)`. The actual procedure for mounting a drive is in `+mount`; note particularly the call out to `+ergo` which brokers file synchronization to Unix via the associated `hez` duct. The emission of an `%ergo` task to Unix is handled in `vere/io/unix.c`.

* Scan through `vere/io/unix.c`.
* How are on-Urbit edits be handled in the case of conflict with a base file? Explore this scenario.

## Solid-state subscriptions <a href="#solid-state-subscriptions" id="solid-state-subscriptions"></a>

Agents frequently need to synchronize all or some of their state via communication. To do this, they can either communicate their entire state when it changes, or they can send deltas indicating how to update the state to a particular point. (There could be checks on this like reporting a checksum or the hash.) Chat agents, for instance, send single messages rather than the total history of the chat channel to that point.

> The more efficient solution is … to only send out instructions on how to *update* the state, but then any subscribed Agent B has to manually interpret these, update its own state, and risk getting some detail wrong. Even if this is done correctly, reimplementing this common pattern in many agents is obviously both wasting wetware and cluttering codebases. \[Solid-state subscriptions are] how we … implement the second solution in kernelspace, reducing code overhead, network load and memory usage at the same time.

SSS will likely continue to evolve.

* [`/lib/sss`](https://github.com/urbit/urbit/blob/develop/pkg/base-dev/lib/sss.hoon)
* [`/app/simple`](https://github.com/wicrum-wicrun/sss/blob/master/urbit/app/simple.hoon)

## Permissions <a href="#permissions" id="permissions"></a>

Clay supports file permissions at the level of paths (and daughter paths). See `+perm` in `/sys/clay` for an example of setting permissions. This system does not appear to be used much at the current time. It does impact requests of remote file resources, which is not yet a common use pattern beyond simply publishing code.

```hoon
::                                                      ::::
::::                    ++clay                            ::  (1c) versioning
  ::                                                    ::::
++  clay  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%croz rus=(map desk [r=regs w=regs])]          ::  rules for group
        [%cruz cez=(map @ta crew)]                      ::  permission groups
        [%rule red=dict wit=dict]                       ::  node r+w permissions
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $%  [%cred nom=@ta cew=crew]                        ::  set permission group
        [%crew ~]                                       ::  permission groups
        [%crow nom=@ta]                                 ::  group usage
        [%perm des=desk pax=path rit=rite]              ::  change permissions
    ==                                                  ::
  --
::
+$  crew  (set ship)                                    ::  permissions group
+$  regs  (map path rule)                               ::  rules for paths
+$  rule  [mod=?(%black %white) who=(set whom)]         ::  node permission
+$  whom  (each ship @ta)                               ::  ship or named crew
+$  rite                                                ::  new permissions    
    $%  [%r red=(unit rule)]                            ::  for read     
        [%w wit=(unit rule)]                            ::  for write
        [%rw red=(unit rule) wit=(unit rule)]           ::  for read and write
    ==

```

* Trace how `|public` works.

Some related material for `%treaty` and the docket file system will be covered in `ca12`.

* [“Permissions”](/urbit-os/kernel/clay/tasks#permissions)

## Kiln <a href="#kiln" id="kiln"></a>

`%kiln` is the system affordance for interacting with Clay and Gall from userspace without composing direct tasks. It's a library inside of `%hood` and a set of associated generators.

A `%hood` generator (located in `/gen/hood`) that wants to interact with `%kiln` needs to send a poke indicating which predefined `%kiln` action should be taken, e.g.,

```hoon
;<  bind:m  ~  (poke-our:strandio %hood %kiln-mount !>([pax desk]))
(pure:m !>(~))
```

You can see the set of `%kiln` actions in the `+poke` arm of `/lib/kiln`. Most of these have an associated generator in `/gen/hood`.

* Trace the `%kiln` action for `|revive`.
* Trace the `%kiln` action for `|ota`.
* Trace the `%kiln` action for `|mount`.
* See [\~midden-fabler, `mount-all-desks.hoon`](https://gist.github.com/midden-fabler/5a6bd08d5e9b78113080b941b5d4e04b) for an example of using `%kiln` in another generator.

## Tombstoning <a href="#tombstoning" id="tombstoning"></a>

> Tombstoning is the deletion of data for old desk revisions. Clay has a single `%tomb` `task`, but its [`clue`](/urbit-os/kernel/clay/data-types#clue) has a number of different possible actions:

```hoon
+$  clue                                              ::  murder weapon
  $%  [%lobe =lobe]                                   ::  specific lobe
      [%all ~]                                        ::  all safe targets
      [%pick ~]                                       ::  collect garbage
      [%norm =ship =desk =norm]                       ::  set default norm
      [%worn =ship =desk =tako =norm]                 ::  set commit norm
      [%seek =ship =desk =cash]                       ::  fetch source blobs
  ==
::
+$  norm  (axal ?)
```

A tombstoned value can no longer be successfully returned from a scry. In this case, `[~ ~]` is a response meaning that you can never know the value.

The tombstone policy (`$norm`) affects a recursive directory structure.

* Does the cache retain a tombstoned value?
* [`%tomb` - Tombstoning](/urbit-os/kernel/clay/tasks#tomb---tombstoning)

## Story <a href="#story" id="story"></a>

Story is a set of generators to produce Clay commit messages. The actual messages are stored in a file in Clay, effectively using a Clay as a database. The generators are instrumented through `%hood`/`%helm` so they can pass notes to Arvo.

```hoon
> |new-desk %tale

> |mount %tale

> |cp /===/mar/story/hoon /=tale=/mar/story/hoon
+ /~zod/tale/2/mar/story/hoon

> |cp /===/sur/story/hoon /=tale=/sur/story/hoon
+ /~zod/tale/3/lib/story/hoon

> |cp /===/lib/story/hoon /=tale=/lib/story/hoon
+ /~zod/tale/4/lib/story/hoon

> |story-init, =desk %tale
+ /~zod/tale/5/story

> +story-read, =desk %tale

> |story-write 'Short message' 'Long descriptive message', =desk %tale
: /~zod/tale/6/story

> +story-read, =desk %tale
commit: 0vn.l7i50.emt3e.79vbv.tjuv6.ftaqk.pos61.iqa5q.j0jq4.7mn92.vjssn
Short message

Long descriptive message
```

Story is supported in `%base`, but you'll need to make the mark available on the target desk as done here.

## The Future of Clay <a href="#the-future-of-clay" id="the-future-of-clay"></a>

Clay does some things very well, but at the current scale of Urbit it hasn't really been stress-tested to its performance limits much. (There are some limits on the number of tokens that can be loaded from a single file, for instance.)

There are really two directions we can go with Clay: strip it back down towards source control and distribution, or scale it up into a full noun management system.

The first approach is rooted in an argument that Clay shouldn't do everything, but instead should push off aspects of file management and data storage to Gall instead. The details of this have not been laid out explicitly in any document I'm aware of, but it has been discuss in core architecture meetings.

The other possibility is that Gall and Clay merge into a hypothetical vane called Hume, which then manages agents and agent data in the same space as files and source.

## Exercise <a href="#exercise" id="exercise"></a>

* Produce a `%hood` generator that triggers `%kiln` to produce a file containing the line count of a supplied text file. This file should have the same name but a `.wc` suffix (which will require a mark). A `%txt` mark results in `(list cord)`.
* Walk through producing an OTA for a fake \~zod sponsor and a \~marzod sponsee. See [`MAINTAINERS.md`](https://github.com/urbit/urbit/blob/develop/MAINTAINERS.md) for details of that process (under “Release Next Release Candidate”); you will obviously need to change ship identities.

> There are always horrid exceptions, even in common use -- like extensionless Makefiles. The trivial solution is that if `%clay` finds a file mysterious, it won't track it.

* What happens to a `Makefile` today (that is, a file without a suffix)?


# 12. Vanes V: Gall and Userspace

Covers Gall, Arvo's userspace application framework vane, the agent type, running agents, and userspace software updates.

*This lesson covers Gall: the `$agent` type, running agents, and userspace software updates.*

## Gall <a href="#gall" id="gall"></a>

Currently, the end user zone of Urbit, denoted “userspace”, is supplied primarily by the `/sys/vane/gall` vane. Much as Arvo acts as a dispatcher and state manager for a functional OS in terms of transactions between vanes, Gall acts as a dispatcher and state manager for longrunning daemons. These daemons are referred to as “agents”, and comprise the main way that users actually use Urbit. (Other parts of userspace include generators and threads.)

Gall is responsible to manage userspace applications and their state, including subscribers. To fully understand agent userspace, we want to cover a few related topics:

1. Agents
2. `/sys/vane/gall`
3. Treaty/Docket publication
4. Agent wrappers
5. Historic (dynamic) Gall

We will cover threads in the lesson on Khan and Lick. We covered generators previously in the lesson on Dill and Dojo.

## Agents <a href="#agents" id="agents"></a>

> An agent is a piece of software that is primarily focused on maintaining and distributing a piece of state with a defined structure. It exposes an interface that lets programs read, subscribe to, and manipulate the state. Every event happens in an atomic transaction, so the state is never inconsistent. Since the state is permanent, when the agent is upgraded with a change to the structure of the state, the developer provides a migration function from the old state type to the new state type.

What is an agent in practice? In contemporary static Gall, an agent is a core that hews to the definition:

```hoon
::                                                      ::::
::::                    ++gall                            ::  (1g) extensions
  ::                                                    ::::
++  gall  ^?
  |%
  ::
  ::  +agent: app core
  ::
  ++  agent
    =<  form
    |%
    +$  step  (quip card form)
    +$  card  (wind note gift)
    +$  note
      $%  [%agent [=ship name=term] =task]
          [%arvo note-arvo]
          [%pyre =tang]
      ::
          [%grow =spur =page]
          [%tomb =case =spur]
          [%cull =case =spur]
      ==
    +$  task
      $%  [%watch =path]
          [%watch-as =mark =path]
          [%leave ~]
          [%poke =cage]
          [%poke-as =mark =cage]
      ==
    +$  gift
      $%  [%fact paths=(list path) =cage]
          [%kick paths=(list path) ship=(unit ship)]
          [%watch-ack p=(unit tang)]
          [%poke-ack p=(unit tang)]
      ==
    +$  sign
      $%  [%poke-ack p=(unit tang)]
          [%watch-ack p=(unit tang)]
          [%fact =cage]
          [%kick ~]
      ==
    ++  form
      $_  ^|
      |_  bowl
      ++  on-init
        *(quip card _^|(..on-init))
      ::
      ++  on-save
        *vase
      ::
      ++  on-load
        |~  old-state=vase
        *(quip card _^|(..on-init))
      ::
      ++  on-poke
        |~  [mark vase]
        *(quip card _^|(..on-init))
      ::
      ++  on-watch
        |~  path
        *(quip card _^|(..on-init))
      ::
      ++  on-leave
        |~  path
        *(quip card _^|(..on-init))
      ::
      ++  on-peek
        |~  path
        *(unit (unit cage))
      ::
      ++  on-agent
        |~  [wire sign]
        *(quip card _^|(..on-init))
      ::
      ++  on-arvo
        |~  [wire sign-arvo]
        *(quip card _^|(..on-init))
      ::
      ++  on-fail
        |~  [term tang]
        *(quip card _^|(..on-init))
      --
    --
  --  ::gall

```

A Gall agent must have ten arms. (There's a fascinating bit of self-reference in the state definition going on, and to be honest I'm a little surprised that it works, but it is an iron core.) The definitions here are for `|~` barsig arms in a `^|` ketbar core.

* [`|~` barsig](/hoon/rune/bar#barsig) yields an iron gate.
* [`^|` ketbar](/hoon/rune/ket#ketbar) yields an iron core from a gold core.

So we must at last really grapple with the [core variance model](/build-on-urbit/hoon-school/r-metals) in Urbit. This is often notorious to understand because we don't have great metaphors or analogues to type variance in real life. Right now, what we need to understand is that an iron/contravariant core is *opaque*: That is, we use this to define an interface in which the argument can be less specific than the interface and the result can be more specific. Contravariance is useful for flexibility in input values (`sample`s).

> An `%iron` core `i` has a write-only sample (payload head, `+6.i`) and an opaque context (payload tail, `+7.i`). A core `j` which nests within it must be a `%gold` or `%iron` core, such that `+6.i` nests within `+6.j`. Hence, **contravariant**.

The archetypal Gall agents in `/sys/lull` are composed using iron gates since they will be used as examples for building actual agent cores. Likewise, the `+rs` and sister gates in `/sys/hoon` are built using iron doors with specified rounding behavior so when you actually use the core (like `+add:rs`) the core you are using has been built as an example.

* How are the iron gate runes actually implemented in the Hoon type system? (See `+deem:nest:ut` and `+peel:ut`.)
* Try to implement an agent missing an arm, like `+on-fail`.
* Bonus question: what half-implemented rune produces an `%iron` core?

We construct an agent explicitly in an `/app` file by applying `%- agent:gall` to a correctly-shaped core.

* What does each arm produce?

```hoon
++  on-init   (quip card _agent)
++  on-save   (vase)
++  on-load   (quip card _agent)
++  on-poke   (quip card _agent)
++  on-watch  (quip card _agent)
++  on-leave  (quip card _agent)
++  on-peek   (unit (unit cage))
++  on-agent  (quip card _agent)
++  on-arvo   (quip card _agent)
++  on-fail   (quip card _agent)
```

Finally, we can take a gander at what that ubiquitous `+quip` is:

```hoon
++  quip
  |$  [item state]
  [(list item) state]
```

(It's just a wrapper for `(list item) state`.)

Basically, every arm must produces a list of effects and a state change, if any.

* What does each arm expect?

```hoon
++  on-init   ::  not a gate, only an arm
++  on-save   ::  not a gate, only an arm
++  on-load   |=  =vase
++  on-poke   |=  =cage
++  on-watch  |=  =path
++  on-leave  |=  =path
++  on-peek   |=  =path
++  on-agent  |=  [=wire =sign:agent:gall]
++  on-arvo   |=  [=wire =sign-arvo]
++  on-fail   |=  [=term =tang]
```

We'll need to differentiate the Gall `$sign`s and the Arvo `$sign`s in a moment.

When `+ford:clay` reads in a Gall agent file from `/app`, it automatically composes cores together using `=>` tisgar. (This leads to a slightly disconcerting situation in which the cores are simply present serially in a file.)

* Compare the definition of `$agent:shoe` with `$agent:gall`. How does this correctly extend the Gall agent definition for the type system?

## Vane <a href="#vane" id="vane"></a>

While Gall facilitates very complex userspace apps, the vane itself is rather modest, weighing in at less than half the size of Clay or Ames. Gall knows how to route events to the handler arms in a standard agent core, and it instruments upgrades and subscriptions.

However, we have to consider Gall at two levels: the vane level, which manages top-level state like the set of running agents and queued moves, and the agent level, which manages agents as doors.

Gall is a landlocked vane. It has no runtime counterpart.

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

```hoon
::                                                      ::::
::::                    ++gall                            ::  (1g) extensions
  ::                                                    ::::
++  gall  ^?
  |%
  +$  boar  (map [=wire =ship =term] nonce=@)           ::  and their nonces
  +$  dude  term                                        ::  server identity
  +$  gill  (pair ship term)                            ::  general contact
  +$  load  (list [=dude =beak =agent])                 ::  loadout
  +$  scar                                              ::  opaque duct
    $:  p=@ud                                           ::  bone sequence
        q=(map duct bone)                               ::  by duct
        r=(map bone duct)                               ::  by bone
    ==                                                  ::
  +$  suss  (trel dude @tas @da)                        ::  config report
  +$  well  (pair desk term)                            ::
  +$  deal
    $%  [%raw-poke =mark =noun]
        task:agent
    ==
  +$  unto
    $%  [%raw-fact =mark =noun]
        sign:agent
    ==
  --  ::gall
```

Most of the important types have been separated and are called out below.

(Variance again: [`^?` ketwut](/hoon/rune/ket#ketwut) is for a lead/bivariant core.)

### Vane State <a href="#vane-state" id="vane-state"></a>

```hoon
+$  state
  $:  system-duct=duct
      outstanding=(map [wire duct] (qeu remote-request))
      contacts=(set ship)
      yokes=(map term yoke)
      blocked=(map term (qeu blocked-move))
      =bug
  ==
```

* `system-duct` is the set of outbound moves to other vanes (like Ames for subscriptions) or remote agent contacts.
* `outstanding` is the outstanding request queue.
* `contacts` is the set of other ships with which we are in communication.
* `yokes` is the set of running agents.
* `blocked` is the set of moves to agents that haven't been started yet.
* `bug` is the debug print configuration.

### Vane Moves <a href="#vane-moves" id="vane-moves"></a>

```hoon
|%
+$  gift                                              ::  outgoing result
  $%  [%boon payload=*]                               ::  ames response
      [%done error=(unit error:ames)]                 ::  ames message (n)ack
      [%unto p=unto]                                  ::
  ==                                                  ::
+$  task                                              ::  incoming request
  $~  [%vega ~]                                       ::
  $%  [%deal p=sock q=term r=deal]                    ::  full transmission
      [%sear =ship]                                   ::  clear pending queues
      [%jolt =desk =dude]                             ::  (re)start agent
      [%idle =dude]                                   ::  suspend agent
      [%load =load]                                   ::  load agent
      [%nuke =dude]                                   ::  delete agent
      [%doff dude=(unit dude) ship=(unit ship)]       ::  kill subscriptions
      [%rake dude=(unit dude) all=?]                  ::  reclaim old subs
      $>(%init vane-task)                             ::  set owner
      $>(%trim vane-task)                             ::  trim state
      $>(%vega vane-task)                             ::  report upgrade
      $>(%plea vane-task)                             ::  network request
      [%spew veb=(list verb)]                         ::  set verbosity
      [%sift dudes=(list dude)]                       ::  per agent
  ==                                                  ::
--
```

### Agent State <a href="#agent-state" id="agent-state"></a>

```hoon
+$  bitt  (map duct (pair ship path))                 ::  incoming subs
+$  boat  (map [=wire =ship =term] [acked=? =path])   ::  outgoing subs
+$  bowl                                              ::  standard app state
  $:  $:  our=ship                                    ::  host
          src=ship                                    ::  guest
          dap=term                                    ::  agent
      ==                                              ::
      $:  wex=boat                                    ::  outgoing subs
          sup=bitt                                    ::  incoming subs
          $=  sky                                     ::  scry bindings
          %+  map  path                               ::
          ((mop @ud (pair @da (each page @uvI))) lte) ::
      ==                                              ::
      $:  act=@ud                                     ::  change number
          eny=@uvJ                                    ::  entropy
          now=@da                                     ::  current time
          byk=beak                                    ::  load source
  ==  ==                                              ::                     
```

Every agent needs two parts of its state: the `$bowl`, which is the information outside of the agent that Gall needs to communicate for the vane, and the

### Agent Moves <a href="#agent-moves" id="agent-moves"></a>

```hoon
|%
+$  card  (wind note gift)
+$  note
  $%  [%agent [=ship name=term] =task]
      [%arvo note-arvo]
      [%pyre =tang]
  ::
      [%grow =spur =page]
      [%tomb =case =spur]
      [%cull =case =spur]
  ==
+$  task
  $%  [%watch =path]
      [%watch-as =mark =path]
      [%leave ~]
      [%poke =cage]
      [%poke-as =mark =cage]
  ==
+$  gift
  $%  [%fact paths=(list path) =cage]
      [%kick paths=(list path) ship=(unit ship)]
      [%watch-ack p=(unit tang)]
      [%poke-ack p=(unit tang)]
  ==
+$  sign
  $%  [%poke-ack p=(unit tang)]
      [%watch-ack p=(unit tang)]
      [%fact =cage]
      [%kick ~]
  ==
--
```

A Gall `$card` differs from an Arvo card:

```hoon
::  Arvo
+$  card  (cask)                          ::  tagged, untyped event
++  cask  |$  [a]  (pair mark a)          ::  marked data builder
::
::  Gall
+$  card  (wind note gift)
++  wind
  |$  ::  a: forward
      ::  b: reverse
      ::
      [a b]
  $%  ::  %pass: advance
      ::  %slip: lateral
      ::  %give: retreat
      ::
      [%pass p=path q=a]
      [%slip p=a]
      [%give p=b]
  ==
+$  note
  $%  [%agent [=ship name=term] =task]
      [%arvo note-arvo]
      [%pyre =tang]
  ::
      [%grow =spur =page]
      [%tomb =case =spur]
      [%cull =case =spur]
  ==
+$  gift
  $%  [%fact paths=(list path) =cage]
      [%kick paths=(list path) ship=(unit ship)]
      [%watch-ack p=(unit tang)]
      [%poke-ack p=(unit tang)]
  ==
```

Gall does not permit a `%slip`, so a card is either:

* `[%pass path note]`
* `[%give gift]`

### Structure <a href="#structure" id="structure"></a>

The two main engine cores within `/sys/vane/gall` are:

* `+mo` Arvo move handler
* `+ap` agent-level core

The `+abet` pattern used in Gall prefixes each arm with the containing door abbreviation so you can remain more easily oriented within `/sys/vane/gall`.

#### `+mo` Arvo move handler

Many `+mo` calls resolve into `+ap` calls. It mainly sets things up around particular per-agent calls.

#### `+ap` agent-level core

To run an agent, we have to know the state of the agent, which includes its state and relevant bowl information:

```hoon
::  $yoke: agent runner state
::
+$  yoke
  $%  [%nuke sky=(map spur @ud)]
      $:  %live
          control-duct=duct
          run-nonce=@t
          sub-nonce=_1
          =stats
          =bitt
          =boat
          =boar
          code=*
          agent=(each agent vase)
          =beak
          marks=(map duct mark)
          sky=(map spur path-state)
          ken=(jug spar:ames wire)
  ==  ==
```

* `control-duct` is the duct of
* `run-nonce` is a unique nonce for each build.
* `sub-nonce` is global `%watch` nonce.
* `stats`
* `bitt` is the set of incoming subscriptions (for the `bowl`).
* `boat` is the set of outgoing subscriptions (for the `bowl`).
* `code` is the most recently loaded code as a noun.
* `agent` is the agent core, possibly as a vase.
* `beak` is the compilation source.
* `marks` is the map of mark conversion requests.
* `sky` is the map of scry bindings.
* `ken` is the map of sets of open `%keen` remote scry requests.

A typical call from `+mo` to `+ap` will be predicated on `+ap` setting up a Gall agent with its state and processing the incoming move through the appropriate arm.

For instance, this is the lifecycle of a scry call to Gall:

* A scry handler (`roof`) produces a call to Gall's `+scry` arm.
  * `+mo` as a door needs a duct and a set of moves.
    * `+mo-peek` sets up a call to `+ap-peek` along the given path with the appropriate care.
      * `+ap-abed` sets up the agent noun for evaluation.
        * `+ap-yoke` loads the actual agent state; an agent is a door with a state and bowl sample.
      * `+ap-peek` parses the scry path appropriately.
        * `+ap-mule-peek` evaluates the code using `[9 2 0 1]` and `+mock` (see `ca01` for a refresher).
          * `+ap-agent-core` sets up the agent core with its current bowl and state; this includes a `+on-peek` arm since we know the shape of the `$agent` core.
            * `+ap-construct-bowl` produces the agent-ready bowl from Gall-level information.

The lifecycle of a poke looks like this:

* A move is injected targeting Gall's `+call` arm as a `%deal` (indicating that the move goes to an agent).
  * `+call` dispatches to `+mo-handle-use` for an agent.
    * `+mo-handle-local` is for running local agents.
      * `+mo-apply` and `+mo-apply-sure` prepare to call `+ap`.
        * `+ap-abed` sets up the agent noun for evaluation.
          * `+ap-apply` dispatches several kinds of operations, including pokes.
            * `+ap-poke` queues a `%poke-ack` (since it's first among moves) and calls `+ap-ingest`.
              * `+ap-ingest` calls the agent arm.
                * `+ap-handle-result` and `+ap-handle-peers` take care of watches etc.
                  * `+ap-agent-core` sets up the agent core with its current bowl and state; this includes a `+on-poke` arm since we know the shape of the `$agent` core.
        * `+ap-abet` yields the list of cards to resolve back to `+mo`, but also the `$yoke`, which is the new agent state for Gall's `state`.
      * `+mo-abet` finalizes.

What about an agent modification like `|nuke`? Let's see the lifecycle of that call.

* `/gen/hood/nuke` → `%kiln-nuke`
  * `/lib/hood/kiln` → `+poke-nuke`
    * `[%pass /nuke %arvo %g [%nuke dude]]`
    * `/sys/vane/gall`:
      * `+call`
        * `+mo`
        * `+mo-nuke`
          * `+ap`
            * `+ap-abed`
            * `+ap-nuke` is where the real work is done. Review it.
              * `+ap-ingest`
            * `+ap-abet`
        * `+mo-abet`

Clay actually governs which agents can run on a given desk. How does `|install` instigate this?

* See `+goad` in `/sys/vane/clay`.

There is additionally some plumbing around Gall receiving responses in `+take` for the vane (`/sys`) versus for an agent (`/use`).

We'll need to differentiate the Gall `$sign`s and the Arvo `$sign`s in a moment.

### Scry interface <a href="#scry-interface" id="scry-interface"></a>

Gall brokers two kinds of scries: vane scries and agent scries.

> In order to hit the vane-level endpoints, the beginning of the the `spur` (e.g. the `path` after the `beak`) *must* be a `%$` empty element. For example:

```hoon
.^(desk %gd /=acme=/$)
.^((set [=dude:gall live=?]) %ge /=base=/$)
.^((list path) %gt /=acme=//foo)
```

* [`%d`: get desk of app](/urbit-os/kernel/gall/scry#d-get-desk-of-app)
* [`%e`: running apps](/urbit-os/kernel/gall/scry#e-running-apps)
* [`%f`: nonces of apps](/urbit-os/kernel/gall/scry#f-nonces-of-apps)
* [`%n`: get nonce of subscription](/urbit-os/kernel/gall/scry#n-get-nonce-of-subscription)
* [`%t`: remote scry subpaths](/urbit-os/kernel/gall/scry#t-remote-scry-subpaths)
* [`%u`: check if installed](/urbit-os/kernel/gall/scry#u-check-if-installed)
* [`%w`: latest revision of path](/urbit-os/kernel/gall/scry#w-latest-revision-of-path)
* [`%x`: remote scry file](/urbit-os/kernel/gall/scry#x-remote-scry-file)
* [`%z:` hash of value at path](/urbit-os/kernel/gall/scry#z-hash-of-value-at-path)

An agent scry has the form `/=agent=/path/to/scry` and may accept any care. `%x` cares must include a terminal mark in the path, however.

Gall also dispatches scries to agents' `+on-peek` arms. This takes place via `+mo-peek`→`+ap-peek`→`+on-peek`→`+ap-mule-peek`.

See `+scry` for details of both.

* [“Gall Scry Reference”](/urbit-os/kernel/gall/scry)

## Treaty and Docket <a href="#treaty-and-docket" id="treaty-and-docket"></a>

There are two ways to distribute nouns over Ames today:

1. Mark a desk `|public` and use Clay to directly synchronize desks.
2. Use the `/app/treaty` agent from `%landscape` to discover and install agents.

Landscape (formerly Grid) is formally a Tlon product. It primarily consists of two agents (and associated marks, libraries, etc.):

* `/app/treaty` handles publishing and advertising application desks.
* `/app/docket` handles retrieving, validating, and installing application desks.

Together they query a remote `%treaty` instance to `+install` a particular desk.

* Examine `$alliance` in `/sur/treaty`.
* Examine `+publish:so` and `+watch:tr` in `/app/treaty`.
* Examine `+install:ch` in `/app/docket`.

`/app/treaty` in particular has pretty tight construction and I commend its style to you.

* [“Software Distribution · Userspace”](/build-on-urbit/userspace/dist/software-distribution)

### Updates <a href="#updates" id="updates"></a>

> When Gall receives a newly rebuilt agent from Clay, it calls the gate produced by the `+on-load` arm of the new agent with the state extracted from the old agent. If there is a crash in any `+on-load` calls or in the handling of any effects they emit (which can include more agent activations), then the whole event crashes, canceling the commit. This effectively gives any agent the ability to abort a commit by crashing.

* Gall's `+call` arm receives a `%load` move with a noun of a core built by `+ford:clay`.
  * `+mo-core` is a handle to `+mo` because no `+abet` is needed.
    * `+mo-load` installs agents pretty mechanically, by simply `+skim`ming over the `%live` agents.
      * `+mo-receive-core` checks whether the agent is running.
      * If it is, then `+ap` is invoked to update the agent:
        * `+ap-abed`
        * `+ap-reinstall`
          * `+on-save:ap-agent-core`
          * `+ap-install` is the install wrapper.
            * `+ap-upgrade-state`
              * `+on-init:ap-agent-core`
              * `+on-load:ap-agent-core`
        * `+ap-abet`
        * `+mo-clear-queue` flushes the blocked tasks pending for a new agent.
      * If it isn't, then we have to create it in `+ap`:
        * `+ap-abed`
        * `+ap-upgrade-state`
          * `+on-init:ap-agent-core`
          * `+on-load:ap-agent-core`
      * `+mo-idle` puts the agent to sleep if it's in the kill list (to be retired due to `desk/bill`).
  * `+mo-abet` finalizes moves and state changes.

Note that this is independent of Treaty and Docket once the remote desk has been installed.

## Agent Wrappers as Core Modifiers <a href="#agent-wrappers-as-core-modifiers" id="agent-wrappers-as-core-modifiers"></a>

.An agent wrapper (like `dbug`) is a tool to wrap additional handlers around an agent core. These can wrap the internal Gall agent with new functionality by catching pokes and other standard moves, then re-dispatching to the normal arms if no special behavior is needed.

```hoon
> :my-agent +dbug [%state 'value']
```

* Examine `/lib/dbug`.
  * In particular, how does the `+on-poke` wrapper arm work?
  * How does the `+dbug` generator work?

> Using agent transformers to extend agents is a very nice conceptual pattern. But in practice, there are three pretty big problems with it:
>
> 1. You need to edit the agent code yourself.
> 2. Stateful transformers can break the agent.
> 3. The agent's world will also get transformed.

* [“Debugging Wrapper”](/build-on-urbit/userspace/examples/dbug)
* [\~wicrum-wicrun, “Gall Middleware (Assembly 2022”](https://www.quartus.co/blog/gall-middleware-wicrum-wicrun-assembly-2022)

## Dynamic Gall <a href="#dynamic-gall" id="dynamic-gall"></a>

An earlier incarnation of Gall, dynamic Gall, specified its arms in terms of the names of the move coming back or the mark of the poke coming in. (This is what I learned on, way back when.)

For instance, this agent was an earlier version of the egg timer app:

```hoon
::  |start %egg
::  :egg ~s5
|%
+$  effect   (pair bone syscall)
+$  syscall  [%wait path @da]
--
|_  [bowl:gall ~]
++  poke-noun
  |=  t=@dr
  ^+  [*(list effect) +>.$]
  :_  +>.$  :_  ~
  [ost %wait /egg-timer (add now t)]
++  wake
  |=  [=wire error=(unit tang)]
  ^+  [*(list effect) +>.$]
  ~&  "Timer went off!"
  [~ +>.$]
--
```

## Exercises <a href="#exercises" id="exercises"></a>

* Your assignment is to produce a minimalist Gall-like agent handler: a userspace framework for producing “toy” agent-like applications. Let's call them “scamps”.

  The scamp's state is defined in a `$state` block at the top of its file, e.g.:

  ```hoon
  +$  state
    $:  scores=(list @)
        hi-score=@
    ==
  ```

  Scamps do not support state upgrades, so no version tag is provided.

  A scamp requires the following arms for the developer:

  ```hoon
  |%
  ++  on-init
  ++  on-poke
  ++  on-peek
  --
  ```

  You should be able to poke and peek into a scamp. It has no subscription model.

  A scamp specification file is NOT implicitly chained with a running `=>` tisgal. Compose explicitly.

As a final aside, I believe that building an `%aqua`/`%pyro` testbed along the lines of Gall should also be feasible for you at this point.


# 13. Vanes VI: Khan, Lick

This lesson covers I/O with Khan, Arvo's thread runner vane, and Lick, Arvo's IPC vane.

*This lesson covers I/O with Khan, the thread-runner vane, and Lick, the IPC vane.*

Khan and Lick are both interprocess communications vanes with slightly different philosophies. We will also discuss `conn.c`. In brief:

1. `%khan` is a high-level thread interface, useful to both Arvo and external clients.
2. `%lick` is a low-level noun interface for domain sockets, treating Arvo as a server and earth software as a client.
3. `conn.c` provides full administrative control over Arvo and Vere.

## Khan & Threads <a href="#khan-threads" id="khan-threads"></a>

Khan allows threads to be triggered from outside of Urbit. To start off with, what exactly is a thread?

> A thread is a monadic function that takes arguments and produces a result. It may perform input and output while running, so it is not a pure function. A thread's strength is that it can easily perform complex IO operations. It uses what's often called the IO monad (plus the exception monad) to provide a natural framework for IO. A thread's weakness is that it's impermanent and may fail unexpectedly. In most of its intermediate states, it expects only a small number of events (usually one), so if it receives anything it didn't expect, it fails. When code is upgraded, it's impossible to upgrade a running thread, so it fails.

Threads can be run using Gall's `%spider` agent or Khan.

### `%spider`: Threads Before Khan <a href="#spider-threads-before-khan" id="spider-threads-before-khan"></a>

Arvo is an event handler for OS-level moves for vanes. Gall is an event handler for agent-level moves. Spider is an agent for transient thread-level operations.

#### Thread Definition

Ultimately, a `$thread` is a gate which accepts a `vase` and returns the `form` of a `strand` that produces a `vase`. In other words, the `$thread` doesn't (just) produce a result, it produces a strand that takes input and produces output from which a result can be extracted. This allows threads to chain friable computations together until a `%thread-done` is produced.

```hoon
+$  thread  $-(vase _*form:(strand ,vase))
```

* `+form` is the mold of the strand. It weaves together the notions of input and output thus:

```hoon
++  form  (strand-form-raw a)
++  strand-form-raw
  |*  a=mold
  $-(strand-input (strand-output-raw a))
+$  strand-input  [=bowl in=(unit input)]
++  strand-output-raw
  |*  a=mold
  $~  [~ %done *a]
  $:  cards=(list card)
      $=  next
      $%  [%wait ~]
          [%skip ~]
          [%cont self=(strand-form-raw a)]
          [%fail err=(pair term tang)]
          [%done value=a]
      ==
  ==
```

`+strand` is more complicated. It's an “asynchronous transaction mold”, which is basically a union of four different monads. It's a wet gate producing a core from a mold.

1. Reader for input.
2. Writer for cards.
3. Continuation for callbacks.
4. Exception.

This gate also produces a number of critical handlers, such as:

* `+form` is the main type of a strand computation.
* `+pure` is an identity computation, useful for binding.
* `+bind` is a combination of two computations.
* `+eval` maintains the monadic nature of the computation.

A simple thread (like `/ted/code`) is simply a wrapper for some check like a scry:

```hoon
/-  spider
/+  strandio
=,  strand=strand:spider
^-  thread:spider
|=  arg=vase
=/  m  (strand ,vase)
^-  form:m
;<  =bowl:spider  bind:m  get-bowl:strandio
;<  code=@p  bind:m  (scry:strandio @p /j/code/(scot %p our.bowl))
(pure:m !>(code))
```

* Read this thread with new eyes about the types involved.

[`;<` micgal](/hoon/rune/mic#micgal) serves to permit a sequence of computations in which each one depends on the output of the previous one.

```hoon
;<  mold  bind  expr1  expr2
```

which desugars to:

```hoon
%+  (bind mold)
  expr1
|=  mold
expr2
```

> `;<` can be used to glue a pipeline together to run an asynchronous function or event. This can be helpful when deferring parts of a computation based on external data.

The `main-loop` pattern provides a way of providing a list of functions to try a value against, and seems like an interesting way of handling an arbitrary number of `%facts`.

Threads can trigger daughter threads. `+handle-start-thread` does this by modifying Spider's thread `yarn`, but you need to see Spider now.

* [“Fetch JSON”](/urbit-os/base/threads/examples/get-json)
* [“Child Thread”](/urbit-os/base/threads/examples/child-thread)
* [“Main Loop”](/urbit-os/base/threads/examples/main-loop)
* [“Poke Agent”](/urbit-os/base/threads/examples/poke-agent)
* [“Scry”](/urbit-os/base/threads/examples/scry)
* [“Take Fact”](/urbit-os/base/threads/examples/take-fact)

#### `/app/spider`

`/app/spider` tracks threads at the highest level using a “spider core” and a state manager.

```hoon
+$  card         card:agent:gall
+$  thread       $-(vase shed:khan)
+$  tid          @tatid
+$  input        [=tid =cage]
+$  yarn         (list tid)
+$  thread-form  _*eval-form:eval:(strand ,vase)
+$  trying       ?(%build %none)
::
+$  state
  $:  starting=(map yarn [=trying =vase])
      running=(axal thread-form)
      tid=(map tid yarn)
      serving=(map tid [(unit @ta) =mark =desk])
      scrying=(jug tid [=wire =ship =path])
  ==
```

Fundamentally, each thread is an invocation of a list of thread IDs and their startup state, threads currently in progress, and some handlers for remote scries, etc.

* `starting` is the collection of threads pending successful execution.
* `running` contains currently-running threads identified by path.
* `tid` is a map for tracking child threads.
* `serving` has to do with the [HTTP API](/urbit-os/base/threads/http-api) for threads, allowing you to use the Urbit ship like a function-as-a-service server.
* `scrying` is a map of sets of remote scries.

Threads use a set of internal mark conventions (notably `%thread-done` and `%thread-fail`).

```hoon
++  strand-output-raw
  |*  a=mold
  $~  [~ %done *a]
  $:  cards=(list card)
      $=  next
      $%  [%wait ~]
          [%skip ~]
          [%cont self=(strand-form-raw a)]
          [%fail err=(pair term tang)]
          [%done value=a]
      ==
  ==
```

* `cards` is the set of cards to dispatch immediately.
* `%thread-wait` means to not move on but to stay awaiting a callback.
* `%thread-skip` is a drop because this should be handled elsewhere.
* `%thread-cont` means to continue the computation from a new callback.
* `%thread-fail` aborts a computation and doesn't send effects.
* `%thread-done` finishes a computation and sends effects.

The `%spider-helper` core has all the logic to handle HTTP, start and conclude threads, build code, handle input, etc. For instance:

```hoon
++  thread-done
  |=  [=yarn =vase silent=?]
  ^-  (quip card ^state)
  ::  %-  (slog leaf+"strand {<yarn>} finished" (sell vase) ~)
  =/  =tid  (yarn-to-tid yarn)
  =/  done-cards=(list card)
    :~  [%give %fact ~[/thread-result/[tid]] %thread-done vase]
        [%give %kick ~[/thread-result/[tid]] ~]
    ==
  =^  http-cards  state
    (thread-http-response tid vase)
  =^  scry-card  state  (cancel-scry tid silent)
  =^  cards      state  (thread-clean yarn)
  [:(weld done-cards cards http-cards scry-card) state]
```

It's not a proper `+abet` core.

Spider supports a few auxiliary scries to monitor thread state, such as the set of currently running threads:

```hoon
.^((list path) %gx /=spider=/tree/noun)
```

You can only subscribe to Spider for thread results.

* Look at `/lib/strand`. What surprises you?
* [“Spider API”](/urbit-os/base/threads/api)

### A New Interface <a href="#a-new-interface" id="a-new-interface"></a>

> Khan is the "control plane" and thread-runner vane. Its main purpose is to allow external applications to run [threads](/urbit-os/base/threads) via a Unix Socket and receive the result.

Khan was conceived as a way to control Urbit ships from the exterior using threads. The concept evolved a fair bit from proposal to implementation. In practice, Khan is essentially an interface wrapper for Spider-based threads, which produces a somewhat strange (but not unprecedented) situation in which a vane relies on a piece of userspace infrastructure to function correctly.

Khan can be internally invoked (using a `cage`) or externally invoked (using a `page`).

#### `/sys/lull` Definition

```hoon
::                                                      ::::
::::                    ++khan                            ::  (1i) threads
  ::                                                    ::::
++  khan  ^?
  |%
  +$  gift                                              ::  out result <-$
    $%  [%arow p=(avow cage)]                           ::  in-arvo result
        [%avow p=(avow page)]                           ::  external result
    ==                                                  ::
  +$  task                                              ::  in request ->$
    $~  [%vega ~]                                       ::
    $%  $>(%born vane-task)                             ::  new unix process
        [%done ~]                                       ::  socket closed
        [%fard p=(fyrd cage)]                           ::  in-arvo thread
        [%fyrd p=(fyrd cast)]                           ::  external thread
        [%lard =bear =shed]                             ::  inline thread
        $>(%trim vane-task)                             ::  trim state
        $>(%vega vane-task)                             ::  report upgrade
    ==                                                  ::
  ::                                                    ::
  ++  avow  |$  [a]  (each a goof)                      ::  $fyrd result
  +$  bear  $@(desk beak)                               ::  partial $beak
  +$  cast  (pair mark page)                            ::  output mark + input
  ++  fyrd  |$  [a]  [=bear name=term args=a]           ::  thread run request
  ::                                                    ::
  +$  shed  _*form:(strand:rand ,vase)                  ::  compute vase
  --  ::khan
```

* [“Khan”](/urbit-os/kernel/khan)

While `%khan` hasn't been thoroughly documented yet (we expect some minor API changes, such as the more recent addition of [inline thread invocation](https://github.com/urbit/urbit/pull/5981)), there are examples of its use in \~midsum-salrux's [Tendiebot price bot](https://github.com/midsum-salrux/tendiebot/blob/master/desk/tendiebot/app/tendiebot.hoon) and [Faux Urbit–Discord bridge](https://github.com/midsum-salrux/faux).

The basic conceit of Khan is that it instruments three ways to run a thread:

* `%fard` runs a thread from within Arvo directly.
* `%fyrd` runs a thread from outside Arvo (a connexion with the runtime).
* `%lard` runs an inline thread (rather than from `/ted`).

A `%fard` has the form:

```hoon
:*  %pass
    /path-name        ::  path
    %arvo  %k  %fard  ::  Arvo vane and %khan mode
    %namespace        ::  desk?
    %thread-name      ::  /ted/thread-name.hoon
    %noun             ::  mark (always %noun for now)
    !>  :*            ::  thread arguments:
      bowl            ::    bowl (entropy etc.)
      other-info      ::    other arguments for thread
    ==
==
```

A `%lard` has the form:

```hoon
=strandio -build-file %/lib/strandio/hoon
=sh |=  message=@t
    =/  m  (strand:rand ,vase)
    ;<  ~  bind:m  (poke:strandio [our %hood] %helm-hi !>('hi'))
    ;<  ~  bind:m  (poke:strandio [our %hood] %helm-hi !>(message))
    (pure:m !>('product'))
|pass [%k %lard %base (sh 'the message')]
```

Since `/sys/vane/khan` is a vane, you receive its gifts in `+on-arvo`.

* `[%arow p=(avow cage)]` is received in userspace. Note that it is a `cage`, or a pair of mark and vase.
* `[%avow p=(avow page)]` can only be received by an external process. It is a `page`, or a pair of mark and (unvased) data.

Compare Spider and Khan:

```hoon
:_  this
:~  [%pass /thread/[ta-now] %agent [our.bowl %spider] %watch /thread-result/[tid]]
    [%pass /thread/[ta-now] %agent [our.bowl %spider] %poke %spider-start !>([~ `tid byk.bowl %foo !>(~)])]
==
::
:_  this
:~  [%pass /thread[ta-now] %arvo %k %fard q.byk.bowl %foo %noun !>([bowl ~])]
==
```

As a vane, `/sys/vane/khan` is almost as simple as a vane can be: it simply `+call`s tasks and `+take`s gifts from Spider to dispatch back to its caller.

Khan currently supports no scries.

* [“Khan: API Reference”](/urbit-os/kernel/khan/tasks)
* [“Developer Call: The Future of `%khan`”](https://www.youtube.com/watch?v=cdSFvFNFqpI)

Speculatively, I believe that producing an improved CLI predicated on thread execution is feasible today on Urbit. Imagine a context which can dispatch moves either batched or singly, and queue return cards for processing.

In fact, although the vane evolved from its initial conception, Khan was originally proposed under the theory that pre-written threads would be the easiest way to bundle, distribute, and manage scripts for hosting and maintenance.

## Lick <a href="#lick" id="lick"></a>

Although also dealing with interprocess communication, Lick was designed for a very different scenario than Khan: to allow external processes, in particular hardware drivers, to intercommunicate with Urbit. (This breached the Earth/Mars divide.) Thus `/sys/vane/lick` focuses on instrumenting a low-level noun interfaces over domain sockets.

> Lick manages IPC ports, and the communication between Urbit applications and POSIX applications via these ports. Other vanes and applications ask Lick to open an IPC port, notify it when something is connected or disconnected, and transfer data between itself and the Unix application.

Lick works by opening a Unix socket for a particular process, which allows serialized IPC communications. These involve a jammed noun so the receiving process needs to know how to communicate in nouns.

> The IPC ports Lick creates are Unix domain sockets (`AF_UNIX` address family) of the `SOCK_STREAM` type.

The connexions are made via filepaths in `.urb/dev` of the pier.

The format is:

```
V.BBBB.JJJJ.JJJJ...
```

* `V` version
* `B` jam size in bytes (little endian)
* `J` jammed noun (little endian)

> The process on the host OS must therefore strip the first 5 bytes, [`+cue`](/hoon/stdlib/2p#cue) the jamfile, check the mark and (most likely) convert the noun into a native data structure.

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

```hoon
::                                              ::::
::::                    ++lick                    ::  (1j) IPC
  ::                                            ::::
++  lick  ^?
  |%
  +$  gift                                      ::  out result <-$
    $%  [%spin =name]                           ::  open an IPC port
        [%shut =name]                           ::  close an IPC port
        [%spit =name =mark =noun]               ::  spit a noun to the IPC port
        [%soak =name =mark =noun]               ::  soak a noun from the IPC port
    ==
  +$  task                                      ::  in request ->$
    $~  [%vega ~]                               ::
    $%  $>(%born vane-task)                     ::  new unix process
        $>(%trim vane-task)                     ::  trim state
        $>(%vega vane-task)                     ::  report upgrade
        [%spin =name]                           ::  open an IPC port
        [%shut =name]                           ::  close an IPC port
        [%spit =name =mark =noun]               ::  spit a noun to the IPC port
        [%soak =name =mark =noun]               ::  soak a noun from the IPC port
    ==
  ::
  +$  name  path
  --  ::lick
```

To evaluate what `/sys/vane/lick` is doing, we need to look at Unix's IPC model briefly. IPC (“interprocess communication“) describes any way that two processes in an operating system's shared context have to communicate with each other. Lick focuses on [Unix domain sockets](https://en.wikipedia.org/wiki/Unix_domain_socket), which are just [communication endpoints](https://man7.org/linux/man-pages/man7/unix.7.html). For instance, a valid use of `%lick` would use cards that look like this:

```hoon
++  init  [[%pass / %arvo %l %spin /control]~ this]
::
++  on-arvo
  |=  [=wire =sign-arvo]
  ?+  sign-arvo  (on-arvo:def wire sign-arvo)
      [%lick %soak *]
      ?+  mark.sign-arvo  [~ this]
      ::
        %connect
      ~&  >  "connect"
      :_  this  [%pass /spit %arvo %l %spit /control %init area.state]~
  ==  ==
::
++  send-state
  |=  =state
  ^-  card:agent:gall
  [%pass /spit %arvo %l %spit /control %state [slick:state face.state food.state live.state]]
```

* [\~mopfel-winrux, `%slick`](https://github.com/mopfel-winrux/slick)

The vane definition of `/sys/vane/lick` is even simpler than `/sys/vane/khan`: it has no `+abet` core and primarily communicates to the `unix-duct` in its state. The `owner` is a `duct` to handle the return `%soak`.

Lick takes several scries:

* [`%a` - Read ports](/urbit-os/kernel/lick/scry#a---read-ports)
* [`%d` - Port owner](/urbit-os/kernel/lick/scry#d---port-owner)
* [`%u` - Port existance](/urbit-os/kernel/lick/scry#u---port-existance)

Gall needs to wrap `%soak` and `%spit` to route properly. See e.g. `+ap-generic-take`. This lets multiple agents share sockets with the same name, and each agent can have its own folder.

### `vere/io/lick.c` <a href="#vereiolickc" id="vereiolickc"></a>

The hardware counterpart of `/sys/lick` is contained in `vere/io/lick.c` aside from its callback registration. As with other parts of the runtime event loop and callback system, the primary connexion is made using `libuv`, in this case via an instance of a [`uv_pipe_t`](https://docs.libuv.org/en/v1.x/pipe.html) descriptor.

* `_lick_ef_spit()`
  * `_lick_send_noun()`
  * `u3_newt_send()`
* `_lick_sock_cb()`, callback for connection from Earth.
* `_lick_moor_poke`, result of `%soak` from external process.

## `conn.c` <a href="#connc" id="connc"></a>

[`conn.c`](https://github.com/urbit/vere/blob/develop/pkg/vere/io/conn.c) is a driver in Vere. It is a part of the "King" (a.k.a. "Urth") process. It exposes a [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket) at `/path/to/pier/.urb/conn.sock` for sending/receiving data from external processes. The point of `conn.c` is to provide administrative control over Arvo and Vere: read ephemeral or persistent state, enqueue events, and send arbitrary commands (pack, meld, mass, \&c).

(`conn.c` is only loosely related to `/sys/khan`. Its main connection is special-casing some inputs for Khan.)

`conn.c` accepts a newt-encoded `+jam`med noun of the shape `[request-id command arguments]`, where:

* `request-id` is a client-supplied atomic identifier with type `@`. It exists entirely for the benefit of the client, allowing responses to be matched to requests. (The poor Earthling's wire.)
* `command` is one of:
  * `%peek`, namespace scry request into Arvo.
  * `%peel`, emulated namespace scry request into Vere.
  * `%ovum`, injection of a raw kernel move.
  * `%fyrd`, direct shortcut to Khan command.
  * `%urth`, subcommand to runtime to `%pack` or `%meld`.

See particularly:

* `_conn_moor_poke()` for the main message dispatcher.
* `_conn_peek_cb()` for the peek handler.
  * `_conn_send_noun()`
* `_conn_read_peel()` for the `%peel` handler.
* [“`conn.c` Usage Guide”](https://github.com/urbit/tools/wiki/conn.c-Usage-Guide)
* [Click](https://github.com/urbit/tools/tree/master/pkg/click)

## Exercises <a href="#exercises" id="exercises"></a>

* Run these valid commands on a fakeship from the outside (following examples in the `conn.c` usage guide).
  * Pack, meld, OTA, install, code, vats


# 14. Vanes VII: Jael, Azimuth

A look at Urbit's networking key infrastructure. Covers the Jael vane, the %eth-watcher agent, and Azimuth's Layer 2 rollup solution.

*This final lesson looks at Urbit's networking key infrastructure, including the Jael vane, `%eth-watcher` agent, and Azimuth's Layer 2 rollup.*

## Azimuth <a href="#azimuth" id="azimuth"></a>

Azimuth is the “Urbit ID” component of the system. It allows ships to securely communicate over Ames as peers, either using galaxy-based routing (today) or a direct route (if known from previous contact).

> Each ship in a two-way conversation computes the shared symmetric key for that conversation by computing the product of their own private key and the public key of the other party.

Azimuth is a [“public key infrastructure“](https://en.wikipedia.org/wiki/Public_key_infrastructure), meaning that binds public keys to entity identities, with each entity holding its own private key. For each point, Azimuth needs to know three things:

1. Identity (point)
2. Life (key revision number)
3. Rift (continuity number)

> These values are utilized by [Ames](/urbit-os/kernel/ames) and [Jael](/urbit-os/kernel/jael) to ensure that communication between ships is always done with the most recent set of networking keys, and that networking state is appropriately reset when a breach has occurred.

* [“Life and Rift”](/urbit-id/life-and-rift)

Today, Azimuth is instantiated as an NFT, technically an [ERC-721](https://ethereum.org/en/developers/docs/standards/tokens/erc-721/) token. The original Azimuth was “Layer 1” on Ethereum, meaning that the transactions and Azimuth state are directly verified by the Ethereum Virtual Machine. More recently, a “Layer 2” was added, which permits batched transactions very cheaply at the cost of eliding the EVM check (and external visibility).

Azimuth is strictly under the governance of the Galactic Senate and so the Ethereum state could be overridden. “Sovereign is he who decides the exception” (Carl Schmitt), and this provides an explicit safety valve against a chain state or contract exploit against Ethereum. (This would be controversial, I imagine, and it could break Azimuth-on-Ethereum, but there are circumstances under which it would make sense.)

> The Senate has no ability to touch this data directly. This is in direct contrast to all existing centralized services, where your account is always at risk of being taken away from you. What the Senate can change is the “business logic” that decides how you can interact with the data in Azimuth. These are mechanisms such as what powers various proxies have, how stars/planets are released over time, and how sponsorship works.

We will not discuss details of Layer 2 points other than to note that from Jael's perspective they behave the same as L1 points.

### Identity <a href="#identity" id="identity"></a>

You are doubtless familiar with the fivefold hierarchical nature of Urbit ID:

| **Type** | **Span**                                                              | **Number**       |
| -------- | --------------------------------------------------------------------- | ---------------- |
| Galaxy   | `0x0`–`0xff`                                                          | `256`            |
| Star     | `0x100`–`0xffff`                                                      | `65.280`         |
| Planet   | `0x1.0000`–`0xffff.ffff`                                              | `4.294.901.760`  |
| Moon     | `0x1.0000.0000`–`0xffff.ffff.ffff.ffff`                               | $2^{64}-2^{32}$  |
| Comet    | `0x1.0000.0000.0000.0000`–`0xffff.ffff.ffff.ffff.ffff.ffff.ffff.ffff` | $2^{128}-2^{64}$ |

(The counts are not generally even powers of two because of the carveout for the superior points.)

At various points in the past, these points had different names:

| **Type** | \*\*Feudal Type I \*\* (fort) | **Feudal Type II** (fort) | **Maritime Type** (ship) |
| -------- | ----------------------------- | ------------------------- | ------------------------ |
| Galaxy   | duke                          | czar                      | carrier                  |
| Star     | earl                          | king                      | cruiser                  |
| Planet   | lord                          | duke                      | destroyer                |
| Moon     | pawn                          | earl                      | yacht                    |
| Comet    | wolf                          | pawn                      | submarine                |

(\~zod was originally pope in the feudal terminology.)

Within Arvo, these are still baked into the core rank types, so you need occasionally to know the second set of feudal types:

```hoon
+$  rank  ?(%czar %king %duke %earl %pawn)        ::  ship width class
++  clan                                          ::  ship to rank
  |=  who=ship
  ^-  rank
  =/  wid  (met 3 who)
  ?:  (lte wid 1)   %czar
  ?:  =(2 wid)      %king
  ?:  (lte wid 4)   %duke
  ?:  (lte wid 8)   %earl
  ?>  (lte wid 16)  %pawn
```

```hoon
> (sein:title our now our)
~zod

> (clan:title our)
%czar
```

The `+title` core in `/sys/zuse` provides identity-related functions. Saliently:

* `+sein` for the actual sponsor of a particular point. (We distinguish because of emancipation.)
* `+clan` for the rank.
* `+name` for the responsible point (that point unless a moon).
* `+saxo` for the on-Azimuth sponsorship chain.

```hoon
> (saxo:title our now ~middev-middev)
[i=~middev-middev t=[i=~nodsel t=~[~sel]]]
```

#### Galaxies

Galaxies have a public key visible in Azimuth PKI. Their IP address is loaded on boot from a galaxy table supplied by Urbit (as organization). Then each ship can build the sponsorship chain it needs by construction.

#### Stars, Planets

Other points’ public keys in Azimuth PKI are obtained from Ethereum and stored in Jael, along with its sponsor, life, and rift.

#### Moons

Moons do not have an independent Azimuth presence, but rely on their parent ship's Jael to store their life and rift values.

#### Comets

Comets do not have a registration in Azimuth. They cannot cycle their networking keys, and they cannot breach, so their life and rift are always `0`.

* [“Comet self-attestation”](/urbit-os/kernel/ames/cryptography#comets)

### Ecliptic <a href="#ecliptic" id="ecliptic"></a>

There are two contract addresses for Azimuth:

* [`Azimuth.eth`](/urbit-id/azimuth-eth) is the contract data store containing the actual state of Azimuth on Ethereum.
* [`Ecliptic.eth`](/urbit-id/ecliptic) contains the operational logic (transfers, unlocking, etc.). It is separate from `Azimuth.eth` so that the contracts can be upgraded without losing chain state.

Ecliptic contracts are written in Solidity, the EVM contract programming language.

> * **Azimuth**: contains all on-chain state for azimuth. Most notably, ownership and public keys. Can't be modified directly, you must use the Ecliptic.
> * **Ecliptic**: is used as an interface for interacting with your points on-chain. Allows you to configure keys, transfer ownership, etc.
> * **Polls**: registers votes by the Galactic Senate on proposals. These can be either static documents or Ecliptic upgrades.
> * **Linear Star Release**: facilitates the release of blocks of stars to their owners over a period of time.
> * **Conditional Star Release**: facilitates the release of blocks of stars to their owners based on milestones.
> * **Claims**: allows point owners to make claims about (for example) their identity, and associate that with their point.
> * **Censures**: simple reputation management, allowing galaxies and stars to flag points for negative reputation.
> * **Delegated Sending**: enables network-effect like distributing of planets.
> * **Planet Sale**: gives an example of a way in which stars could sell planets on-chain.

* Demonstrate using the `READ` functions with [Etherscan](https://etherscan.io/address/ecliptic.eth#readContract).
* Read [`PlanetSale.sol`](https://github.com/urbit/azimuth/blob/master/contracts/PlanetSale.sol).

> There is no limit for galaxies. Instead, for most galaxies, all stars have already been spawned and placed into one of the lockup contracts: [Linear Star Release↗](https://etherscan.io/address/0x86cd9cd0992f04231751e3761de45cecea5d1801) and [Conditional Star Release↗](https://etherscan.io/address/0x8c241098c3d3498fe1261421633fd57986d74aea).\
> Beginning in 2019, stars may spawn at most 1024 planets. This limit doubles every subsequent year until the maximum is reached.

* [`Ecliptic.eth` contract reader](https://etherscan.io/address/0x86cd9cd0992f04231751e3761de45cecea5d1801#readContract)
* [“Azimuth's First Contract Upgrade”](https://urbit.org/blog/first-contract)
* [`urbit/azimuth`](https://github.com/urbit/azimuth)

### Bridge <a href="#bridge" id="bridge"></a>

Bridge is the primary interface to execute Ecliptic contracts.

* [“Using Bridge”](/user-manual/id/using-bridge)

### Urbit HD Wallet <a href="#urbit-hd-wallet" id="urbit-hd-wallet"></a>

A hierarchical deterministic (HD) wallet is a system of related key pairs with distinct delegated powers, à la [BIP-32](https://en.bitcoin.it/wiki/BIP_0032). A master key pair is used to deterministically generate subsidiary key pairs with specific capabilities on the chain.

As a point owner, you receive some means of claiming the point initially—a claim code or email or simply a transfer to an Ethereum address that you must accept. From that, you can either hold the point via the ownership address on Ethereum or via the master ticket (with seed `@q`).

(A `@q` is like a `@p` in that it is syllabic, but has no limitation on value or range unlike that implied by `@p`.)

The holder of the main ownership address and key pair can designate proxies who can only run certain Ecliptic transactions.

![](https://media.urbit.org/fora/proposals/UP-8.jpg)

The key pairs are:

* Ownership
* Transfer Proxy (set on a temporary basis to make transferring a point a two-step process less prone to error)
* Spawn Proxy (only galaxies and stars; can spawn new child identities)
* Management Proxy (planets, stars, and galaxies; can set life and rift, and manage sponsorship; receives boot keyfile)
* Voting Proxy (only galaxies)

The `/app/claz` command-line tool lets you perform specific sophisticated actions on L1 points from a fake ship. This provides an unmediated interface to Ecliptic.

* Read through `/app/claz`, `/ted/claz/prep-command`, and in particular `/lib/claz`.

Like `sudo`, this is riskier but direct.

* [“Urbit HD Wallet”](/urbit-id/hd-wallet)
* [“HD Wallet (Master Ticket)”](/user-manual/id/hd-wallet)
* [“HD Crypto Wallets: What Are They?” (Gemini)](https://www.gemini.com/cryptopedia/hd-crypto-wallets-hierachichal-deterministic)
* [“Operator's Manual: Proxies”](/user-manual/id/proxies)
* [“Advanced Azimuth Tools”](/urbit-id/advanced-azimuth-tools)

### Breaching/Factory Resets <a href="#breachingfactory-resets" id="breachingfactory-resets"></a>

What is a breach?

> Ships on the Ames network sometimes need to reset their continuity. A factory reset (hereafter just called a *reset*) is when an individual ship announces to the network: "I forgot who I am, let's start over from scratch." That is, it clears its own event log and sends an announcement to the network, asking all ships that have communicated with it to reset its networking information in their state. This makes it as though the ship was just started for the first time again, since everyone on the network has forgotten about it.

We call them “factory resets” for the public, but internally it's all `%breach`.

* See `/sys/vane/ames:++on-publ-breach`.

Since a breach happens to a ship, it cannot know about its own breach.

* [“Guide to Factory Resets”](/user-manual/id/guide-to-resets)

## Jael <a href="#jael" id="jael"></a>

Jael stores Azimuth information. `/sys/vane/jael` is intimately related to Azimuth. Azimuth represents what we may call the exoteric view of ownership, while Jael is the esoteric view. Jael’s primary role is ship networking management and cryptography, but it also supports promises.

`/sys/vane/jael` segregates state into two categories: absolute and relative. Absolute state refers to what is known about the Azimuth PKI, ship ownership, private keys, etc. Since not every ship is live on the network (such as a fakezod), there is also a notion of relative state, referring to what is known about the current ship only.

Jael's code has Azimuth hardcoded into it, but could be made to talk to a different agent pretty easily, as long as it communicates key change & sponsorship events in the same way.

### `/sys/lull` Definition <a href="#syslull-definition" id="syslull-definition"></a>

Jael defines a lot of different types to support what different parts of the system need to see. (For instance, `/app/eth-watcher` needs to communicate with `/sys/vane/jael`.)

#### Tasks and Gifts

```hoon
+$  gift                                            ::  out result <-$
  $%  [%done error=(unit error:ames)]               ::  ames message (n)ack
      [%boon payload=*]                             ::  ames response
      ::
      [%private-keys =life vein=(map life ring)]    ::  private keys
      [%public-keys =public-keys-result]            ::  ethereum changes
      ::
      [%turf turf=(list turf)]                      ::  domains
  ==                                                ::
+$  task                                            ::  in request ->$
  $~  [%vega ~]                                     ::
  $%  [%dawn dawn-event]                            ::  boot from keys
      [%fake =ship]                                 ::  fake boot
      ::
      [%listen whos=(set ship) =source]             ::  set ethereum source
      [%meet =ship =life =pass]                     ::  met after breach
      [%nuke whos=(set ship)]                       ::  cancel tracker from
      [%ruin ships=(set ship)]                      ::  pretend breach
      ::
      [%private-keys ~]                             ::  sub to privates
      [%public-keys ships=(set ship)]               ::  sub to publics
      [%rekey =life =ring]                          ::  update private keys
      [%resend ~]                                   ::  resend private key
      ::
      [%turf ~]                                     ::  view domains
      ::
      [%moon =ship =udiff:point]                    ::  register moon keys
      [%step ~]                                     ::  reset web login code
      ::
      $>(%trim vane-task)                           ::  trim state
      $>(%vega vane-task)                           ::  report upgrade
      $>(%plea vane-task)                           ::  ames request
  ==                                                ::
```

* `%private-keys` are used by Ames.
* `%public-keys` are used in a few places where messages need to be decrypted, e.g. Clay and `/app/ping`.
* `%turf` relates to domains for real ships.

#### Organization

`/sys/vane/jael` is not a particularly complicated vane.

`lex` is the durable state of Jael:

```hoon
=|  lex=state-2
+$  state-2
  $:  %2
      pki=state-pki-2                                   ::
      etn=state-eth-node                                ::  eth connection state
  ==                                                    ::
```

`$state-pki` is the known PKI state:

```hoon
+$  state-pki-2                                         ::  urbit metadata
  $:  $=  own                                           ::  vault (vein)
        $:  yen=(set duct)                              ::  trackers
            sig=(unit oath)                             ::  for a moon
            tuf=(list turf)                             ::  domains
            fak=_|                                      ::  fake keys
            lyf=life                                    ::  version
            step=@ud                                    ::  login code step
            jaw=(map life ring)                         ::  private keys
        ==                                              ::
      $=  zim                                           ::  public
        $:  yen=(jug duct ship)                         ::  trackers
            ney=(jug ship duct)                         ::  reverse trackers
            nel=(set duct)                              ::  trackers of all
            dns=dnses                                   ::  on-chain dns state
            pos=(map ship point)                        ::  on-chain ship state
        ==                                              ::
  == 
```

`$state-eth-node` is the state of a connexion to an Ethereum node.

```hoon
+$  source  (each ship term)
+$  source-id  @udsourceid
::
::  +state-eth-node: state of a connection to an ethereum node
::
+$  state-eth-node                                    ::  node config + meta
  $:  top-source-id=source-id
      sources=(map source-id source)
      sources-reverse=(map source source-id)
      default-source=source-id
      ship-sources=(map ship source-id)
      ship-sources-reverse=(jug source-id ship)
  ==                                                  ::
```

There are a number of ancillary cores that track the public-key/private-key state.

For the most part, you won't need to directly use Jael when working with Gall agents unless you are carrying out Azimuth operations.

There is a standard vane interface core which dispatches to an engine.

The main body of `/sys/vane/jael` has several engines:

* `+ez` Ethereum wallet algebra
* `+of` main engine for top-level semantics.
* `+su` subjective engine for all derived state, subscriptions, and actions.

The `+feel` core handles tracking and updating public keys.

* Trace an Azimuth update from `%fact` via `+new-event` to `+feel`'s operations.

Parenthetically, `/sys/vane/jael` uses a unique pattern for invoking `+abet`:

```hoon
%-  curd  =<  abet
(private-keys:~(feel su hen now pki etn) life.tac ring.tac)
```

#### Scries

Jael takes no cares, so all information is exposed via the path. Oddly, rather than a desk name, Jael expects an operation type. (This is a very old scry pattern and should be cleaned up.)

Reveal your web login code:

```hoon
.^(@p %j /=code=/(scot %p our))
```

Query who the ship's sponsor is:

```hoon
.^(@p %j /=sein=/~sampel-palnet)
```

Query the sponsorship chain:

```hoon
.^((list @p) %j /=saxo=/~sampel-palnet)
```

Get current state of subscriptions to public key updates; this won't have much to say if you are on a fakeship:

```hoon
.^([yen=(jug duct ship) ney=(jug ship duct) nel=(set duct)] %j /=subscriptions=/1)
```

* Examine the [`%step` exercise](/urbit-os/kernel/jael/examples#step) in the Jael docs.

### Azimuth Data Flow <a href="#azimuth-data-flow" id="azimuth-data-flow"></a>

Jael does not talk to Ethereum/Azimuth directly. It derives its knowledge of Azimuth as a PKI from some particular source. In the default case, this is `/app/eth-watcher` and friends.

> By default, Jael's primary source of information is the Gall agent `%azimuth`, which (using `%eth-watcher`) runs a thread that polls an Ethereum node for transactions in the Azimuth contract. It can also get updates from other ships on the network. (This is always the case for moons, where it subscribes to the moon's parent's Jael for updates.)

`/app/azimuth` is the agent responsible for subscribing to Azimuth transactions from `/app/eth-watcher`. (It unrolls L2 transactions as well.) It exposes this data via scries and subscription endpoints.

`/app/eth-watcher` is actually a generic Ethereum state listener. It works for Azimuth because what Jael needs is conveyed via `/app/azimuth`.

A few relevant types, eliding the point algebra:

```hoon
::                                                      ::::
::::                    ++jael                          ::  (1h) security
  ::                                                    ::::
++  jael  ^?
  |%
  +$  public-keys-result
    $%  [%full points=(map ship point)]
        [%diff who=ship =diff:point]
        [%breach who=ship]
    ==
  ::
  ++  block
    =<  block
    |%
    +$  hash    @uxblockhash
    +$  number  @udblocknumber
    +$  id      [=hash =number]
    +$  block   [=id =parent=hash]
    --
  ::
  ++  point
    =<  point
    |%
    +$  point
      $:  =rift
          =life
          keys=(map life [crypto-suite=@ud =pass])
          sponsor=(unit @p)
      ==
    --
  --  ::  jael
```

* [“Azimuth Data Flow: `%eth-watcher`”](/urbit-id/flow#eth-watcher)
* [“Azimuth Data Flow: `naive.hoon`](/urbit-id/flow#naive)

#### `/sys/lull` Shared Types

```hoon
::                                                      ::::
::::                    ++ethereum-types                  ::  eth surs for jael
  ::                                                    ::::
++  ethereum-types
  |%
  ++  address  @ux                              ::  ethereum address, 20 bytes.
  +$  event-id  [block=@ud log=@ud]             ::  event location
  ++  events  (set event-id)
  --
::                                                      ::::
::::                    ++azimuth-types                   ::  az surs for jael
  ::                                                    ::::
++  azimuth-types
  =,  ethereum-types
  |%
  ++  point
    $:  ::  ownership
        $=  own
        $:  owner=address
            management-proxy=address
            voting-proxy=address
            transfer-proxy=address
        ==
      ::
        ::  networking
        $=  net
        %-  unit
        $:  =life
            =pass
            continuity-number=@ud
            sponsor=[has=? who=@p]
            escape=(unit @p)
        ==
      ::
        ::  spawning
        $=  kid
        %-  unit
        $:  spawn-proxy=address
            spawned=(set @p)
        ==
    ==
  ::
  +$  dnses  [pri=@t sec=@t ter=@t]
  ::
  ++  diff-azimuth
    $%  [%point who=@p dif=diff-point]
        [%dns dnses]
    ==
  ::
  ++  diff-point
    $%  [%full new=point]                           ::
        [%owner new=address]                        ::  OwnerChanged
        [%activated who=@p]                         ::  Activated
        [%spawned who=@p]                           ::  Spawned
        [%keys =life =pass]                         ::  ChangedKeys
        [%continuity new=@ud]                       ::  BrokeContinuity
        [%sponsor new=[has=? who=@p]]               ::  EscapeAcc/LostSpons
        [%escape new=(unit @p)]                     ::  EscapeReq/Can
        [%management-proxy new=address]             ::  ChangedManagementPro
        [%voting-proxy new=address]                 ::  ChangedVotingProxy
        [%spawn-proxy new=address]                  ::  ChangedSpawnProxy
        [%transfer-proxy new=address]               ::  ChangedTransferProxy
    ==
  --
  ::                                                    ::
  ::::                  ++pki:jael                      ::  (1h2) certificates
    ::                                                  ::::
  ++  pki  ^?
    |%
    +$  hand  @uvH                                      ::  128-bit hash
    +$  mind  [who=ship lyf=life]                       ::  key identifier
    +$  name  (pair @ta @t)                             ::  ascii / unicode
    +$  oath  @                                         ::  signature
    ++  tale                                            ::  urbit-signed *
      |$  [typ]                                         ::  payload mold
      $:  dat=typ                                       ::  data
          syg=(map ship (pair life oath))               ::  signatures
      ==                                                ::
    --  ::  pki
```

* What is a change on Azimuth? (See `+diff-point`.)

### Bootstrapping <a href="#bootstrapping" id="bootstrapping"></a>

Jael rises early in the booting of a new ship, since we bake identity into Arvo very soon. This results from `%dawn` task passed with a `$dawn-event`:

```hoon
+$  dawn-event
  $:  =seed
      spon=(list [=ship point:azimuth-types])
      czar=(map ship [=rift =life =pass])
      turf=(list turf)
      bloq=@ud
      node=(unit purl:eyre)
  ==
```

Jael is a landlocked vane, but does receive a `%dawn` event constructed by `vere/dawn.c:u3_dawn_vent()`.

* Review that function and `_dawn_post_json()` for the actual information retrieval.

## Example <a href="#example" id="example"></a>

* Boot a comet using a star as its sponsor if you have a running star. (Maybe try with the next option.)
* Boot a fake comet. (See `vere/main.c` and override.)
* Build a fake sponsorship chain. Can you change the sponsorship chain to another point (emancipate) on a fakenet? Why or why not?


# Runtime

Documentation for Urbit's runtime and Nock interpreter (Vere), including guides for U3, jetting, C programming in Urbit, and cryptography implementation.

These guides cover Urbit's runtime and Nock interpreter, which is called Vere.

* [U3 Overview](/build-on-urbit/runtime/u3) - An overview of the noun-wrangling part of the runtime, U3.
* [Conn.c Guide](/build-on-urbit/runtime/conn) - Using `conn.c` to interact with a running ship from the outside.
* [How to Write a Jet](/build-on-urbit/runtime/jetting) - A jetting guide by for new Urbit developers.
* [C3: C in Urbit](/build-on-urbit/runtime/c) - Under u3 is the simple c3 layer, which is just how we write C in Urbit.
* [U3: Land of Nouns](/build-on-urbit/runtime/nouns) - The division between c3 and u3 is that you could theoretically imagine using c3 as just a generic C environment. Anything to do with nouns is in u3.
* [U3: API Overview](/build-on-urbit/runtime/api) - A walkthrough of each of the u3 modules.
* [Cryptography](/build-on-urbit/runtime/cryptography) - References on the cryptography libraries utilized by jets.

## Additional Resources <a href="#additional-resources" id="additional-resources"></a>

* [User Reference](/user-manual/running/vere) - Reference for the utilities and options the runtime takes from the terminal.
* [The Vere Repo](https://github.com/urbit/vere) - Github repository for the Urbit runtime.


# U3

Introduction to u3, the Nock runtime system. C library for making and storing nouns. Solid-state interpreter.

The Urbit interpreter is built on a Nock runtime system written in C, `u3`. This section is a relatively complete description.

You should keep reading if (a) you're planning to work on the Urbit interpreter; (b) you're a language implementation geek; or (c) you don't really understand anything until you've seen the actual structs.

## u3: Noun processing in C <a href="#u3-noun-processing-in-c" id="u3-noun-processing-in-c"></a>

`u3` is the C library that makes Urbit work. If it wasn't called `u3`, it might be called `libnoun` - it's a library for making and storing nouns.

What's a noun? A noun is either a cell or an atom. A cell is an ordered pair of any two nouns. An atom is an unsigned integer of any size.

To the C programmer, this is not a terribly complicated data structure, so why do you need a library for it?

One: nouns have a well-defined computation kernel, Nock, whose spec fits on a page and gzips to 340 bytes. But the only arithmetic operation in Nock is increment. So it's nontrivial to compute both efficiently and correctly.

Two: `u3` is designed to be a "solid-state interpreter," ie, a single-level store which is transparently snapshotted. This implies a specialized memory-management model, etc, etc.

(Does `u3` depend on the higher levels of Urbit, Arvo and Hoon? Yes and no. `u3` expects you to load something shaped like an Arvo kernel, and use it as an event-processing function. But you don't need to use this feature if you don't want, and your kernel doesn't have to be Arvo proper - just Arvo-compatible. Think of `u3` as the BIOS and Arvo as the boot kernel. And there are no dependencies at all between Hoon the language and `u3`.)


# Conn.c Guide

Guide for using conn.c driver. Unix domain socket for external ship interaction. Khan vane, urbit eval utility.

## Background <a href="#background" id="background"></a>

Prior to 2022, the only way to interact with a running ship from Earth was via HTTP requests sent to the `%lens` agent. In addition to `%lens`, there was a Python script helper named `herb` which would automatically format HTTP requests for `%lens` based on user inputs. However, there were several pain points for using `%lens` and `herb`. `%lens`, `herb`, and the difficulties of using them are documented more fully [here](https://github.com/urbit/urbit/issues/6418).

Unrelated to the above, inconveniences around writing boilerplate code to interact with the `%spider` agent was hampering adoption of threads outside of the Arvo kernel.

## Present <a href="#present" id="present"></a>

Starting in 2022, tools for solving the above issues began to appear (though work on them began in 2021). In order, they are:

* `conn.c`
* The Khan vane
* The `urbit eval` utility
* `-eval` and `-khan-eval`
* The click thin client

Together, these tools are the building blocks for performing any action on a running ship (poke, scry, or command) from Earth, and receiving back programmatically usable output.

### `conn.c` <a href="#connc" id="connc"></a>

[`conn.c`](https://github.com/urbit/vere/blob/develop/pkg/vere/io/conn.c) is a driver in Vere. It is a part of the "King" (a.k.a. "Urth") process. It exposes a [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket) at `/path/to/pier/.urb/conn.sock` for sending/receiving data from external processes.

Input to `conn.c` must be a newt-encoded jammed noun that fits mold `[request-id command arguments]`, where:

* `request-id` is a client-supplied atomic identifier with type `@`. It exists entirely for the benefit of the client, allowing responses to be matched to requests.
* `command` is one of:
  * `%peek`
  * `%peel`
  * `%ovum`
  * `%fyrd`
  * `%urth`

These commands cover all possible cases for the following 2x2 matrix:

|          | poke             | scry    |
| -------- | ---------------- | ------- |
| **vere** | `%urth`          | `%peel` |
| **arvo** | `%ovum`, `%fyrd` | `%peek` |

For a valid command, the output from `conn.c` is a newt-encoded jammed noun with type `[request-id output]`, where:

* `request-id` matches the input `request-id`
* `output` depends on the `command`

For an invalid command, the output from `conn.c` is a newt-encoded jammed noun with type `[0 %bail error-code error-string]`. However, `conn.c` makes no guarantees that:

* It will be able to sufficiently recover from the error to guarantee this output
* It will produce a meaningful error code and message

#### `%ovum`

The argument to an `%ovum` command is a raw kernel move which is injected directly into the Arvo event loop. This is a very powerful - and potentially dangerous - tool. For example, if a ship somehow got into a state where Clay was no longer working properly (meaning new files could not be compiled to fix the state of the kernel), the source code for a new, working Clay could be directly injected into the ship using an `%ovum`.

The output of an `%ovum` command is:

* `[%news %done]` if the move completed successfully
* `[%news %drop]` if the move was dropped
* `[%bail goof]` if an error occurred

#### `%fyrd`

`%fyrd` is a direct shortcut to the Khan vane. The Khan vane coordinates and manages threads, and is described in further detail below. The arguments to a `%fyrd` command are (in order):

1. The name of the desk in which the thread lives (e.g. `%base`) or `beak` for the thread (e.g. `[%zod %base %10]`)
2. The name of the thread (e.g. `%hi`)
3. Mark to which the output should be cast (e.g. `%tape`)
4. Mark for how to interpret the input argument to thread (e.g. `%ship`)
5. Input argument to thread (e.g. `~zod`)

The output of a `%fyrd` command is `[%avow (each page goof)]`, the value of `each` depending on whether the thread succeeded or not.

#### `%urth`

The argument to the `%urth` command is a subcommand for the action to perform. Currently, the only valid commands are `%pack` and `%meld`.

`%urth` will return `%&` if given a valid command as input, otherwise it will return `[0 %bail 0xfffffff9 %urth-bad]`. No other output is emitted.

#### `%peek`

The `%peek` command is used to perform a namespace read request (a.k.a. scry) using Arvo's external peek interface ([arm +22 in arvo.hoon](https://github.com/urbit/urbit/blob/develop/pkg/arvo/sys/arvo.hoon#L1774)). The argument to `%peek` is the `nom` input to `+peek` in `arvo.hoon` (`lyc` is auto-filled as `[~ ~]`, i.e. "request from self"). That is to say that the argument to `%peek` must have type:

```
$+  each  path
$%  [%once vis=view syd=desk tyl=spur]
    [%beam vis=view bem=beam]
==
```

Practically speaking, this means that the input will look like one of these three examples:

```
[%& p=path]
[%| p=[%once vis=view syd=desk tyl=spur]]
[%| p=[%beam vis=view bem=beam]]
```

Where:

* `path` is a `[view beam]`, with the `view` passed in as a `coin`
* `view` is the vane code for the scry, as well as an optional care, possibly appended to the vane (e.g. `%j`, `%gx`, etc.)
* `beam` is a `[beak spur]`
* `desk` is used to auto-generate a `beak`: `[our desk now]`
* `spur` is the scry endpoint for the agent or vane

The output of a `%peek` command is `[%peek (unit (unit scry-output))]`, where `~` means that the scry endpoint is invalid, and `[~ ~]` means that the scry resolved to nothing.

See [here](/build-on-urbit/app-school/10-scry) for more information on scrying.

#### `%peel`

`%peel` attempts to emulate a scry-like namespace, like the one used by Arvo and accessed by `%peek`. The argument to `%peel` should be a path. Valid paths result in a non-null `unit` containing the result of the scry. Invalid paths result in null (i.e. `~`). The valid paths and the data they return are:

```
/help   (unit (list path))  Supported %peel paths
/live   (unit ~)            Pier health check; succeeds if pier is running
/khan   (unit ~)            Khan health check; succeeds if Khan vane is running
/info   (unit mass)         Pier info as a mass
/v      (unit @t)           Returns version of the Vere binary as a cord
/who    (unit @)            Returns the Azimuth identity of the ship as an atom
```

Note that the pier info above is returned as a `mass` report, i.e. type `(pair cord (each * (list mass)))`. This is not the same as the `|mass` memory report. `/mass` is meant to be a valid `%peel` path which returns the `|mass` memory report, but it is currently unimplemented.

### Khan <a href="#khan" id="khan"></a>

The Khan vane is a command / response interface for running threads. Khan was introduced to make running threads a kernel-level feature, as simple as poking an agent or setting a timer. Threads allow users to run arbitrarily complex code on their ships in the same way that bash allows them to do so on Linux.

Khan's API exposes three thread requests:

* `%fard`: Kernel thread requests
* `%fyrd`: External thread requests
* `%lard`: "Inline" thread requests

"Kernel" above doesn't mean that this interface is hidden or protected from userspace agents; thread requests by userspace agents should almost certainly use `%fard`. It just means that `%fard` thread requests are expected to originate from within the kernel or a userspace agent. Specifically:

* `%fard` commands take the thread input argument as a `cage`
* The data in the `vase` of the `cage` is a `unit` (as expected by `%spider`)
* The output is also a `cage` (see below for more information)

`%fyrd` thread requests, on the other hand, perform some extra services that are useful when running threads from the dojo or via `conn.c`. Specifically:

* `%fyrd` commands take the thread input as a raw `noun`
* Khan performs mark conversion on both the input and output for `%fyrd` requests
* Khan automatically lifts the converted input into a `unit`

"Inline" threads are a particularly specialized Khan thread request where the thread has already been compiled and is passed as a part of the input.

Khan requests expect the following input:

* `%fard`: `p=[=bear name=term args=cage]`
* `%fyrd`: `p=[=bear name=term args=(pair mark page)]`
* `%lard`: `[=bear =shed]`

Where:

* `bear` is a `desk` or a `beak`; if `bear` is a `desk`, then the it will be converted to a `beak` using `our` and `now` as default values
* `shed` is a pre-computed chain of strands that produce a `vase` (the canonical thread)

All three produce the same output if an error occured while running the thread: `[vow %| goof]`, where `vow` is `%arow` for `%fard` and `%lard`, and `%avow` for `%fyrd`.

If the thread succeeded, `%fard` and `%lard` produce `[%arow %& %noun vase]`. `%fyrd` produces `[%avow %& mark noun]`, where `mark` is the output mark and `noun` is the output as a raw noun after mark conversion.

See [here](/urbit-os/base/threads/basics/fundamentals) for more information about threads.

### `urbit eval` <a href="#urbit-eval" id="urbit-eval"></a>

`eval` is a utility command in the Urbit binary. Originally, it was introduced to evaluate snippets of Hoon code using the binary to emulate Arvo from the associated ivory pill. This allowed it to run any Hoon code fragments that used kernel and STL functions (e.g. anything in `hoon.hoon`, `arvo.hoon`, `lull.hoon`, and `zuse.hoon`). Notably, this did not (and does not) evaluate any Hoon fragments that require pier state (e.g. scries, `our`, `now`, etc.).

Example:

```
$ echo '(add 2 2)' | ./urbit eval
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
eval (run):
4
```

The result (i.e. `4`) is printed to `stdout`. If the command had failed to compile, the stack trace would have been printed to `stdout` instead. All other messages are printed to `stderr`.

#### Options

`eval` was extended with several options that make it useful for processing Hoon nouns as input to or output from `conn.c`:

* `-j`, `--jam`: output result as a jammed noun
* `-c`, `--cue`: read input as a jammed noun
* `-n`, `--newt`: write output / read input as a newt-encoded jammed noun, when paired with `-j` or `-c` respectively
* `-k`: treat the input as the jammed noun input of a `%fyrd` request to `conn.c`; if the result is a `goof`, pretty-print it to `stderr` instead of returning it

### `-eval` and `-khan-eval` <a href="#eval-and--khan-eval" id="eval-and--khan-eval"></a>

Two threads that evaluate arbitrary Hoon were added to the suite of threads included with Arvo: [`ted/eval.hoon`](https://github.com/urbit/urbit/blob/develop/pkg/arvo/ted/eval.hoon) and [`ted/khan-eval.hoon`](https://github.com/urbit/urbit/blob/develop/pkg/arvo/ted/khan-eval.hoon).

Both threads take the same input: Hoon code as a `cord` and an optional `(list path)`. The optional `(list path)` is a list of Clay file dependencies which need to be included for the Hoon to be evaluated (i.e. if the Hoon code includes libraries or types defined outside of the kernel). Each `path` can be a `beam` (i.e. `[beak spur]`) or just a `spur`, in which case the default `beak` (i.e. `[our %base now]`) will be prepended.

`ted/eval.hoon` expects the input to be a Hoon expression. It's very similar to `urbit eval`, except that it has access to ship state: `now`, `our`, vane & agent state, etc.

`ted/khan-eval.hoon` expects the input to be a thread. It attempts to compile the thread using the dependencies (if any) and then sends it to Khan as a `%lard` thread request.

Both threads return regular thread output, i.e. a `vase`.

Examples:

* `-eval '(add 2 2)'`
* `-eval '(my-add 2 2)' [/lib/my-add/hoon ~]`
  * Where `my-add` is defined in `lib/my-add.hoon` in `%base`
* `-eval '(my-add 2 2)' [/(scot %p our)/my-desk/(scot %da now)/lib/my-add/hoon ~]`
  * Where `my-add` is defined in `lib/my-add.hoon` in `%my-desk`
* `-khan-eval '=/ m (strand ,vase) ;< ~ bind:m (poke [~zod %hood] %helm-hi !>(\'\')) (pure:m !>(\'success\'))'`

### click <a href="#click" id="click"></a>

[click](https://github.com/urbit/tools/tree/master/pkg/click) is a `bash` thin client which auto-formats `-eval` and `-khan-eval` thread calls via `%fyrd` requests to `conn.c` and coordinates chaining together the appropriate commands to execute those requests on a running ship.

Using click, a call like:

```
echo $'[0 %fyrd %base %khan-eval %noun %ted-eval \'=/  m  (strand ,vase)  ;<  ~  bind:m  (poke [~zod %hood] %helm-hi !>(\\\'\\\'))  (pure:m !>(\\\'success\\\'))\']' |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

instead looks like:

```
/path/to/click -k /path/to/zod $'=/  m  (strand ,vase)  ;<  ~  bind:m  (poke [~zod %hood] %helm-hi !>(\\\'\\\'))  (pure:m !>(\\\'success\\\'))'
```

or even more conveniently:

```
/path/to/click -k -i threads/poke.hoon /path/to/zod
```

```
Usage:
    click [options] <path-to-pier> <hoon> [<dependencies> ...]
    click [options] -i <path-to-file> <path-to-pier> [<dependencies> ...]
    click [-o|-p] -e -i <path-to-file> <path-to-pier>

    Thin client for interacting with running Urbit ship via conn.c

    options:
        -e                  Execute jammed Hoon
        -h                  Show usage info
        -i <path-to-file>   Read input from file
        -j                  Jam only
        -k                  Execute command using "khan-eval" thread
        -o <path-to-file>   Output to file
        -p                  Filter failure stack traces from result and pretty-print them to stderr
        -x                  Jam to hex
```

## Using these tools <a href="#using-these-tools" id="using-these-tools"></a>

Below are examples of how to execute common commands on a running ship from Earth.

### `|mass` <a href="#mass" id="mass"></a>

Blocked by issues; not currently doable in a way that returns the results as data.

### `|pack` <a href="#pack" id="pack"></a>

```
echo "[0 %urth %pack]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
echo "[0 %ovum %d /test %pack ~]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  ~  bind:m  (flog [%pack ~])  (pure:m !>(\\\'success\\\'))'
```

### `|meld` <a href="#meld" id="meld"></a>

```
echo "[0 %urth %meld]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
echo "[0 %ovum %d /test %meld ~]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  ~  bind:m  (flog [%meld ~])  (pure:m !>(\\\'success\\\'))'
```

### `|ota` <a href="#ota" id="ota"></a>

#### `|ota ~bus`

```
echo "[0 %ovum [%g /test [%deal [~zod ~zod] %hood %raw-poke %kiln-install %base ~bus %kids]]]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  ~  bind:m  (poke [our %hood] %kiln-install !>([%base ~bus %kids]))  (pure:m !>(\\\'success\\\'))'
```

#### `|ota %disable`

```
echo "[0 %ovum [%g /test [%deal [~zod ~zod] %hood %raw-poke %kiln-install %base ~zod %base]]]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  ~  bind:m  (poke [our %hood] %kiln-install !>([%base our %base]))  (pure:m !>(\\\'success\\\'))'
```

#### `|ota ~bus %desk`

```
echo "[0 %ovum [%g /test [%deal [~zod ~zod] %hood %raw-poke %kiln-install %base ~zod %desk]]]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  ~  bind:m  (poke [our %hood] %kiln-install !>([%base ~bus %desk]))  (pure:m !>(\\\'success\\\'))'
```

### `|install` <a href="#install" id="install"></a>

#### `|install ~sampel-palnet %desk`

```
echo "[0 %ovum [%g /test [%deal [~zod ~zod] %hood %raw-poke %kiln-install %desk ~sampel-palnet %desk]]]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  ~  bind:m  (poke [our %hood] %kiln-install !>([%desk ~sampel-palnet %desk]))  (pure:m !>(\\\'success\\\'))'
```

#### `|install ~sampel-palnet %desk, =local %my-desk`

```
echo "[0 %ovum [%g /test [%deal [~zod ~zod] %hood %raw-poke %kiln-install %my-desk ~sampel-palnet %desk]]]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  ~  bind:m  (poke [our %hood] %kiln-install !>([%my-desk ~sampel-palnet %desk]))  (pure:m !>(\\\'success\\\'))'
```

### `+code` <a href="#code" id="code"></a>

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  code=@p  bind:m  (scry @p /j/code/(scot %p our))  (pure:m !>((crip (slag 1 (scow %p code)))))'
```

### `+vats` <a href="#vats" id="vats"></a>

#### `+vats %base %kids`

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  now=@da  bind:m  get-time  (pure:m !>((crip ~(ram re [%rose [~ ~ ~] (report-vats our now [%base %kids ~] %$ |)]))))' \
'/sur/hood/hoon'
```

#### `+vats, =filt %exists`

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  now=@da  bind:m  get-time  (pure:m !>((crip ~(ram re [%rose [~ ~ ~] (report-vats our now ~ %exists |)]))))' \
'/sur/hood/hoon'
```

#### `+vats %base, =verb &, =filt %running`

```
/path/to/click -kp /path/to/pier/zod \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  now=@da  bind:m  get-time  (pure:m !>((crip ~(ram re [%rose [~ ~ ~] (report-vats our now [%base ~] %exists &)]))))' \
'/sur/hood/hoon'
```

### Additional Notes <a href="#additional-notes" id="additional-notes"></a>

#### Alternative click calls

Any example above that uses click has two additional options that have been omitted for brevity, since the actual code for the call would be identical in each example:

1. Custom `-thread` in `%desk`:

```
echo "[0 %fyrd %desk %thread %noun %noun ~]" |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -cn
```

2. Pass inline thread to click from file:

```
/path/to/bin/click -k -i path/to/thread.hoon /path/to/pier/zod
```

#### Undocked ships

click assumes that the ship at the given pier is docked (i.e. that `/path/to/pier/.run` exists). If for whatever reason the running ship is undocked, it's still possible to work around this assumption using the click-format helper script. For example, the call for `+vats` becomes:

```
/path/to/click-format -k \
$'=/  m  (strand ,vase)  ;<  our=@p  bind:m  get-our  ;<  now=@da  bind:m  get-time  (pure:m !>((crip ~(ram re [%rose [~ ~ ~] (report-vats our now)]))))' \
'/sur/hood/hoon' |
/path/to/urbit eval -jn |
nc -U -W 1 /path/to/pier/zod/.urb/conn.sock |
/path/to/urbit eval -ckn
```

## Issues and Future Work <a href="#issues-and-future-work" id="issues-and-future-work"></a>

Currently, there are a number of minor issues and one major issue impacting interactions between Earth and Mars.

The minor issues are:

* `conn.c`'s simulated namespace for `%peel`
  * Unprincipled namespace simulation for no reason other than consistency with Arvo scry
* `/mass` path for `conn.c` `%peel` not implemented
* No `mass` mark in Arvo, so attempting to scry for `|mass` with `%peek` crashes the ship

The major issue is the lack of "thick" clients which are able to consume the newt-encoded jammed nouns emitted by `conn.c` as input. Though not officially codified yet, it makes sense for newt-encoded jammed nouns to be the [narrow waist](https://www.oilshell.org/blog/2022/02/diagrams.html) of Urbit, and [recent design decisions appear to be heading in this direction](https://github.com/urbit/urbit/pull/6396). Unfortunately, the narrow waist of `bash` is text, and it's not always easy or useful to convert nouns to text (particularly stack traces).

There exist already two external noun libraries, in [Rust](https://github.com/urbit/noun) and Haskell (link to Haskell lib coming soon). Adding more, while not trivial, is not difficult. The proliferation of noun representation libraries in other languages would open many doors with regards to the support, hosting, and application opportunities available (the ever-fabled "Quake over Urbit").


# How to Write a Jet

Complete tutorial on writing jets in Urbit. C implementations of Hoon gates for performance. Developer environment setup.

## Writing Jets

Many operations are inefficient when implemented in Nock, and it is efficacious to treat Nock as a standard of behavior rather than the implementation. This tutorial aims to teach you how to read existing jet code; produce a jet matching a Hoon gate with a single argument; and produce a more complex jet involving multiple values and floating-point arithmetic. It will then discuss jetting more generically.

Since jetting code requires modifying the binary runtime, we will work some in Hoon but much more in C. While you can build whatever you like as experimental or personal work, if you intend to submit your jetted code back to the main Urbit developer community then you should coordinate with the Urbit Foundation.

**Additional Resources**

* [\~timluc-miptev, “Jets in the Urbit Runtime”](https://github.com/timlucmiptev/docs-runtime/blob/master/jets1_current.md) (recommended to start here first)
* [“`u3`: Land of Nouns”](/build-on-urbit/runtime/nouns) (recommended as supplement to this document)
* [“API overview by prefix”](/build-on-urbit/runtime/api) (recommended as supplement after this document)

### Developer Environment <a href="#developer-environment" id="developer-environment"></a>

#### Basic Setup (*Mise en place*) <a href="#basic-setup-mise-en-place" id="basic-setup-mise-en-place"></a>

All of Urbit's source code is available in the main Github repo. We will presumptively work in a folder called `~/jetting` which contains a copy of the full Urbit repo. Create a new branch within the repo named `example-jet`.

```sh
$ cd
$ mkdir jetting
$ cd jetting
$ git clone https://github.com/urbit/vere.git
```

The Urbit runtime build stack is based on [Bazel](https://bazel.build/). This suffices unless you intend to include some other third-party library, which must be [linked statically](https://en.wikipedia.org/wiki/Static_library) due to how the Urbit binary is distributed. There is also a bias towards software implementations of processes which hew to a specified reference implementation, such as [SoftFloat](http://www.jhauser.us/arithmetic/SoftFloat.html) rather than hardware floating-point for [IEEE 754](https://ieeexplore.ieee.org/document/30711) floating-point mathematics.

Since jet development requires booting ships many times as one iterates, a pill can make the Urbit-side development process much faster, and is actually required for kernel jets.

Test your build process to produce a local executable binary of Vere:

```sh
$ cd ~/jetting/vere
$ bazel build :urbit
```

This invokes Nix to build the Urbit binary. Take note of where that binary is located (typically in `/tmp` on your main file system) and create a new fakezod using a downloaded pill. (You should check the current binary version and use the appropriate pill instead of `v1.9`.)

```sh
$ cd ~/jetting
$ wget https://bootstrap.urbit.org/urbit-v1.9.pill
$ <Nix build path>/bin/urbit -B urbit-v1.9.pill -F zod
```

> As you work through this guide, version numbers will likely be older than the contemporary release version due to the pace of release. We will update this guide if a version breaks the instructions.

We will primarily work in the development ship (a fakeship or moon) on the files just mentioned, and in the `pkg/urbit` directory of the main Urbit repository, so we need a development process that allows us to quickly access each of these, move them into the appropriate location, and build necessary components. The basic development cycle will look like this:

1. Compose correct Hoon code.
2. Hint the Hoon code.
3. Register the jets in the Vere C code.
4. Compose the jets.
5. Compile and troubleshoot.
6. Repeat as necessary.

* [“Contributor Guidelines”](https://github.com/urbit/vere/blob/master/CONTRIBUTING.md)

#### Conveniences <a href="#conveniences" id="conveniences"></a>

You should consider using a terminal utility like `tmux` or `screen` which allows you to work in several locations on your file system simultaneously: one for file system operations (copying files in and out of the `home` directory), one for running the development ship, and one for editing the files, or an IDE or text editor if preferred.

![](https://raw.githubusercontent.com/sigilante/pixiesticks/master/layout.png)

Inside of your development ship, sync `%clay` to Unix,

```hoon
> |mount %
```

Then copy the entire `%base` desk out so that you can work with it and copy it back in as necessary.

```sh
$ cp -r zod/base .
```

In addition, making a backup copy of a fakeship will make it *much* faster to reset if memory gets corrupted. (This is regrettably common when developing jets.)

```sh
$ cp -r zod zod-backup
```

To reset, simply delete the `.urb/` directory and replace it:

```sh
$ rm -rf zod/.urb
$ cp -r zod-backup/.urb zod
```

### Jet Walkthrough: `+add` <a href="#jet-walkthrough-add" id="jet-walkthrough-add"></a>

Given a Hoon gate, how can a developer produce a matching C jet? Let us illustrate the process using a simple `|%` core. We assume the reader has achieved facility with both Hoon code and C code. This tutorial aims to communicate the practical process of producing a jet, and many [`u3` noun concepts](/build-on-urbit/runtime/nouns) are only briefly discussed or alluded to.

To this end, we begin by examining the Hoon `+add` gate, which accepts two values in its sample.

The Hoon code for `+add` decrements one of these values and adds one to the other for each decrement until zero is reached. This is because all atoms in Hoon are unsigned integers and Nock has no simple addition operation. The source code for `+add` is located in `hoon.hoon`:

```hoon
|%
+|  %math
++  add
  ~/  %add
  ::  unsigned addition
  ::
  ::  a: augend
  ::  b: addend
  |=  [a=@ b=@]
  ::  sum
  ^-  @
  ?:  =(0 a)  b
  $(a (dec a), b +(b))
```

or in a more compact form (omitting the parent core and chapter label)

```hoon
++  add
  ~/  %add
  |=  [a=@ b=@]  ^-  @
  ?:  =(0 a)  b
  $(a (dec a), b +(b))
```

The jet hint `%add` allows Hoon to hint to the runtime that a jet *may* exist. By convention, the jet hint name matches the gate label. Jets must be registered elsewhere in the runtime source code for the Vere binary to know where to connect the hint; we elide that discussion until we take a look at jet implementation below. We will expand on the jet registration runes [`~/` sigfas](/hoon/rune/sig#sigfas) and [`~%` sigcen](/hoon/rune/sig#sigcen) later.

The following C code implements `+add` as a significantly faster operation including handling of >31-bit atoms. It may be found in `urbit/pkg/noun/jets/a/add.c`:

```c
u3_noun
u3qa_add(u3_atom a,
         u3_atom b)
{
  if ( _(u3a_is_cat(a)) && _(u3a_is_cat(b)) ) {
    c3_w c = a + b;

    return u3i_words(1, &c);
  }
  else if ( 0 == a ) {
    return u3k(b);
  }
  else {
    mpz_t a_mp, b_mp;

    u3r_mp(a_mp, a);
    u3r_mp(b_mp, b);

    mpz_add(a_mp, a_mp, b_mp);
    mpz_clear(b_mp);

    return u3i_mp(a_mp);
  }
}
u3_noun
u3wa_add(u3_noun cor)
{
  u3_noun a, b;

  if ( (c3n == u3r_mean(cor, u3x_sam_2, &a, u3x_sam_3, &b, 0)) ||
       (c3n == u3ud(a)) ||
       (c3n == u3ud(b) && a != 0) )
  {
    return u3m_bail(c3__exit);
  } else {
    return u3qa_add(a, b);
  }
}
```

The main entry point for a call into the function is `u3wa_add`. `u3w` functions are translator functions which accept the entire sample as a `u3_noun` (or Nock noun). `u3q` functions take custom combinations of nouns and atoms and generally correspond to unpacked samples.

`u3wa_add` defines two nouns `a` and `b` which will hold the unpacked arguments from the sample. The sample elements are copied out by reference into `a` from sample address 2 (`u3x_sam_2`) and into `b` from sample address 3 (`u3x_sam_3`). A couple of consistency checks are made; if these fail, `u3m_bail` yields a runtime error. Else `u3qa_add` is invoked on the C-style arguments.

`u3qa_add` has the task of adding two Urbit atoms. There is a catch, however! An atom may be a *direct* atom (meaning the value as an unsigned integer fits into 31 bits) or an *indirect* atom (anything higher than that). Direct atoms, called `cat`s, are indicated by the first bit being zero.

```
0ZZZ.ZZZZ.ZZZZ.ZZZZ.ZZZZ.ZZZZ.ZZZZ.ZZZZ
```

Any atom value which may be represented as $2^{31}-1 = 2.147.483.647$ or less is a direct atom. The `Z` bits simply contain the value.

```hoon
> `@ub`2.147.483.647
0b111.1111.1111.1111.1111.1111.1111.1111
> `@ux`2.147.483.647
0x7fff.ffff
```

However, any atom with a value *greater* than this (including many cords, floating-point values, etc.) is an indirect atom (or `dog`) marked with a prefixed bit of one.

```
11YX.XXXX.XXXX.XXXX.XXXX.XXXX.XXXX.XXXX
```

where bit 31 indicates indirectness, bit 30 is always set, and bit 29 (`Y`) indicates if the value is an atom or a cell. An indirect atom contains a pointer into the loom from bits 0–28 (bits `X`).

What does this mean for `u3qa_add`? It means that if the atoms are both direct atoms (`cat`s), the addition is straightforward and simply carried out in C. When converted back into an atom, a helper function `u3i_words` deals with the possibility of overflow and the concomitant transformation to a `dog`.

```c
c3_w c = a + b;               # c3_w is a 32-bit C word.

return u3i_words(1, &c);
```

There's a second trivial case to handle one of the values being zero. (It is unclear to the author of this tutorial why both cases as-zero are not being handled; the speed change may be too trivial to matter.)

Finally, the general case of adding the values at two loom addresses is dealt with. This requires general pointer-based arithmetic with GMP multi-precision integer operations.

```c
mpz_t a_mp, b_mp;             # mpz_t is a GMP multi-precision integer type

u3r_mp(a_mp, a);              # read the atoms out of the loom into the MP type
u3r_mp(b_mp, b);

mpz_add(a_mp, a_mp, b_mp);    # carry out MP-correct addition
mpz_clear(b_mp);              # clear the now-unnecessary `b` value from memory

return u3i_mp(a_mp);          # write the value back into the loom and return it
```

The procedure to solve the problem in the C jet does not need to follow the same algorithm as the Hoon code. (In fact, it is preferred to use native C implementations where possible to avoid memory leaks in the `u3` noun system.)

In general, jet code feels a bit heavy and formal. Jet code may call other jet code, however, so much as with Hoon layers of complexity can be appropriately encapsulated. Once you are used to the conventions of the u3 library, you will be in a good position to produce working and secure jet code.

### Jet Composition: Integer `+factorial` <a href="#jet-composition-integer-factorial" id="jet-composition-integer-factorial"></a>

Similar to how we encountered recursion way back in [Hoon School](/build-on-urbit/hoon-school/f-cores) to talk about gate mechanics, let us implement a C jet of the `+factorial` example code. We will call this library `trig` in a gesture to some subsequent functions you should implement as an exercise. Create a file `lib/trig.hoon` with the following contents:

**`/lib/trig.hoon`**

```hoon
~%  %trig  ..part  ~
|%
:: Factorial, $x!$
::
++  factorial
  ~/  %factorial
  |=  x=@ud  ^-  @ud
  =/  t=@ud  1
  |-  ^-  @rs
  ?:  =(x 0)  t
  ?:  =(x 1)  t
  $(x (sub x 1), t (mul t x))
--
```

We will create a generator `gen/trig.hoon` which will help us quickly check the library's behavior.

**`/gen/trig.hoon`**

```hoon
/+  *trig
!:
:-  %say
|=  [[* eny=@uv *] [x=@rs n=@rs ~] ~]
::
~&  (factorial n)
~&  (absolute x)
~&  (exp x)
~&  (pow-n x n)
[%verb ~]
```

We will further define a few unit tests as checks on arm behavior in `tests/lib/trig.hoon`:

**`/tests/lib/trig.hoon`**

```hoon
/+  *test, *trig
::
::::
  ::
|%
++  test-factorial  ^-  tang
  ;:  weld
    %+  expect-eq
      !>  1
      !>  (factorial 0)
    %+  expect-eq
      !>  1
      !>  (factorial 1)
    %+  expect-eq
      !>  120
      !>  (factorial 5)
    %+  expect-eq
      !>  720
      !>  (factorial 6)
  ==
--
```

(Here we are eliding a key point about contemporary Urbit development: `/lib` code is considered userspace and thus ineligible for jet inclusion in the runtime. This is a matter of development policy rather than technical capability. We will zoom out to consider how to modify kernel code later.)

Save the foregoing library code in `base/lib` and the generator code in `base/gen`; also, don't forget the unit tests! Whenever you work in your preferred editor, you should work on the `base` copies, then move them back into the fakezod and synchronize before execution.

```sh
$ cp -r base zod
```

```sh
> |commit %base

> -test %/tests/lib/trig ~
built   /tests/lib/trig/hoon
OK      /lib/trig/test-factorial
```

#### Jet construction <a href="#jet-construction" id="jet-construction"></a>

Now that you have a developer cycle in place, let's examine what's necessary to produce a jet. A jet is a C function which replicates the behavior of a Hoon (Nock) gate. Jets have to be able to manipulate Urbit quantities within the binary, which requires both the proper affordances within the Hoon code (the interpreter hints) and support for manipulating Urbit nouns (atoms and cells) within C.

Make a development branch for the jet changes first:

```sh
$ cd ~/jetting/vere
$ git branch example-jet
$ git checkout example-jet
```

Jet hints must provide a trail of symbols for the interpreter to know how to match the Hoon arms to the corresponding C code. Think of these as breadcrumbs. Here we have a two-deep scenario. Specifically, we mark the outermost arm with `~%` and an explicit reference to the Arvo core (the parent of `part`). We mark the inner arms with `~/` because their parent symbol can be determined from the context. The `@tas` token will tell the runtime (Vere) which C code matches the arm. All symbols in the nesting hierarchy must be included.

```hoon
~%  %trig  ..part  ~
|%
++  factorial
  ~/  %factorial
  |=  x=@ud  ^-  @ud
  ...
--
```

We also need to add appropriate handles for the C code. This consists of several steps:

1. Register the jet symbols and function names in `tree.c`.
2. Declare function prototypes in headers `q.h` and `w.h`.
3. Produce functions for compilation and linking in the `pkg/noun/jets/e` directory.

The first two steps are fairly mechanical and straightforward.

**Register the jet symbols and function names.** A jet registration may be carried out at in point in `tree.c`. The registration consists of marking the core in the Hoon source and including the name in the C source.

```c
/* Jet registration of ++factorial arm under trig */
static u3j_harm _140_hex__trig_factorial_a[] = {{".2", u3we_trig_factorial, c3y}, {}};
/* Associated hash */
static c3_c* _140_hex__trig_factorial_ha[] = {
  "903dbafb8e59427eced0b35379ad617c2eb6083a235075e9cdd9dd80e732efa4",
  0
};

static u3j_core _140_hex__trig_d[] =
  { { "factorial", 7, _140_hex__trig_factorial_a, 0, _140_hex__trig_factorial_ha },
  {}
  };
static c3_c* _140_hex__trig_ha[] = {
  "0bac9c3c43634bb86f6721bbcc444f69c83395f204ff69d3175f3821b1f679ba",
  0
};

/* Core registration by token for trig */
static u3j_core _140_hex_d[] =
{ /* ... pre-existing jet registrations ... */
  { "trig",   31, 0, _140_hex__trig_d, _140_hex__trig_ha  },
  {}
};
```

The numeric component of the title, `140`, indicates the Hoon Kelvin version. Library jets of this nature are registered as `hex` jets, meaning they live within the Arvo core. Other, more inner layers of `%zuse` and `%lull` utilize `pen` and other three-letter jet tokens. (These are loosely mnemonic from Greek antecedents.) The core is conventionally included here, then either a `d` suffix for the function association or a `ha` suffix for a jet hash. (Jet hashes are a way of “signing” code. They are not as of this writing actively used by the binary runtimes.) Arms are marked with `_a` and child cores with `_d`. The structs used are defined in `jets.h`.

The particular flavor of C mandated by the Vere kernel is quite lapidary, particularly when shorthand functions (such as `u3z`) are employed. In this code, we see the following `u3` elements:

1. `c3_c`, the platform C 8-bit `char` type
2. `c3y`, loobean true, `%.y` (similarly `c3n`, loobean false, `%.n`)
3. `u3j_core`, C representation of Hoon/Nock cores
4. `u3j_harm`, an actual C jet ("Hoon arm")

The numbers `7` and `31` refer to relative core addresses. In most cases—unless you're building a particularly complicated jet or modifying `%zuse` or `%lull`—you can follow the pattern laid out here. `".2"` is a label for the axis in the core `[battery sample]`, so just the battery. The text labels for the `|%` core and the arm are included at their appropriate points. Finally, the jet function entry point `u3we_trig_factorial` is registered.

For more information on `u3`, please check out the `u3` summary below or the official documentation at [`“u3`: Land of Nouns”](/build-on-urbit/runtime/nouns).

**Declare function prototypes in headers.**

A `u3w` function is always the entry point for a jet. Every `u3w` function accepts a `u3noun` (a Hoon/Nock noun), validates it, and invokes the `u3q` function that implements the actual logic. The `u3q` function needs to accept the same number of atoms as the defining arm (since these same values will be extricated by the `u3w` function and passed to it).

In this case, we have cited `u3we_trig_factorial` in `tree.c` and now must declare both it and `u3qe_trig_factorial`:

In `w.h`:

```c
u3_noun u3we_trig_factorial(u3_noun);
```

In `q.h`:

```c
u3_noun u3qe_trig_factorial(u3_atom);
```

**Produce functions for compilation and linking.**

Given these function prototype declarations, all that remains is the actual definition of the function. Both functions will live in their own file; we find it the best convention to associate all arms of a core in a single file. In this case, create a file `pkg/noun/jets/e/trig.c` and define all of your `trig` jets therein. (Here we show `+factorial` only.)

As with `+add`, we have to worry about direct and indirect atoms when carrying out arithmetic operations, prompting the use of GMP `mpz` operations.

```c
/* jets/e/trig.c
**
*/
#include "all.h"
#include <stdio.h>      // helpful for debugging, removable after development

/* factorial of @ud integer
*/
  u3_noun
  u3qe_trig_factorial(u3_atom a)  /* @ud */
  {
    fprintf(stderr, "u3qe_trig_factorial\n\r");  // DELETE THIS LINE LATER
    if (( 0 == a ) || ( 1 == a )) {
      return 1;
    }
    else if ( _(u3a_is_cat(a))) {
      c3_d c = ((c3_d) a) * ((c3_d) (a-1));

      return u3i_chubs(1, &c);
    }
    else {
      mpz_t a_mp, b_mp;

      u3r_mp(a_mp, a);
      mpz_sub(b_mp, a_mp, 1);
      u3_atom b = u3qe_trigrs_factorial(u3i_mp(b_mp));
      u3r_mp(b_mp, b);

      mpz_mul(a_mp, a_mp, b_mp);
      mpz_clear(b_mp);

      return u3i_mp(a_mp);
    }
  }

  u3_noun
  u3we_trig_factorial(u3_noun cor)
  {
    fprintf(stderr, "u3we_trig_factorial\n\r");  // DELETE THIS LINE LATER
    u3_noun a;

    if ( c3n == u3r_mean(cor, u3x_sam, &a, 0) ||
         c3n == u3ud(a) )
    {
      return u3m_bail(c3__exit);
    }
    else {
      return u3qe_trig_factorial(a);
    }
  }
```

This code merits ample discussion. Without focusing on the particular types used, read through the logic and look for the skeleton of a standard simple factorial algorithm.

`u3r` operations are used to extract Urbit-compatible types as C values.

`u3i` operations wrap C values back into Urbit-compatible types.

### `u3` Overview <a href="#u3-overview" id="u3-overview"></a>

Before proceeding to compose a more complicated floating-point jet, we should step back and examine the zoo of `u3` functions that jets use to formally structure atom access and manipulation.

**`u3` Functions**

`u3` defines a number of functions for extracting data from Urbit types into C types for ready manipulation, then wrapping those same values back up for Urbit to handle. These fall into several categories:

| Prefix | Mnemonic                                    | Source File   | Example of Function |
| ------ | ------------------------------------------- | ------------- | ------------------- |
| `u3a_` | Allocation                                  | `allocate.c`  | `u3a_malloc`        |
| `u3e_` | Event (persistence)                         | `events.c`    | `u3e_foul`          |
| `u3h_` | Hash table                                  | `hashtable.c` | `u3h_put`           |
| `u3i_` | Imprisonment (noun construction)            | `imprison.c`  |                     |
| `u3j_` | Jet control                                 | `jets.c`      | `u3j_boot`          |
| `u3k_` | Jets (transfer semantics, C arguments)      | `[a-g]/*.c`   |                     |
| `u3l_` | Logging                                     | `log.c`       | `u3l_log`           |
| `u3m_` | System management                           | `manage.c`    | `u3m_bail`          |
| `u3n_` | Nock computation                            | `nock.c`      | `u3nc`              |
| `u3q_` | Jets (retain semantics, C arguments)        | `[a-g]/*.c`   |                     |
| `u3r_` | Retrieval; returns on error                 | `retrieve.c`  | `u3r_word`          |
| `u3t_` | Profiling and tracing                       | `trace.c`     | `u3t`               |
| `u3v_` | Arvo operations                             | `vortex.c`    | `u3v_reclaim`       |
| `u3w_` | Jets (retain semantics, Nock core argument) | `[a-g]/*.c`   |                     |
| `u3x_` | Retrieval; crashes on error                 | `xtract.c`    | `u3x_cell`          |
| `u3z_` | Memoize                                     | `zave.c`      | `u3z_uniq`          |

**`u3` Nouns**

The `u3` system allows you to extract Urbit nouns as atoms or cells. Atoms may come in one of two forms: either they fit in 31 bits or less of a 32-bit unsigned integer, or they require more space. In the former case, you will use the singular functions such as `u3r_word` and `u3a_word` to extract and store information. If the atom is larger than this, however, you need to treat it a bit more like a C array, using the plural functions `u3r_words` and `u3a_words`. (For native sizes larger than 32 bits, such as double-precision floating-point numbers, replace `word` with `chub` in these.) Confusing a 31-bit-or-less integer with a 32+-bit integer means confusing a value with a pointer! Bad things will happen!

An audit of the jet source code shows that the most commonly used `u3` functions include:

1. `u3a_free` frees memory allocated on the loom (Vere memory model).
2. `u3a_malloc` allocates memory on the loom (Vere memory model).\
   (Never use regular C `malloc` in `u3`.)
3. `u3i_bytes` writes an array of bytes into an atom.
4. `u3i_chub` is the ≥32-bit equivalent of `u3i_word`.
5. `u3i_chubs` is the ≥32-bit equivalent of `u3i_words`.
6. `u3i_word` writes a single 31-bit or smaller atom.
7. `u3i_words` writes an array of 31-bit or smaller atoms.
8. `u3m_bail` produces an error and crashes the process.
9. `u3m_p` prints a message and a `u3` noun.
10. `u3r_at` retrieves data values stored at locations in the sample.
11. `u3r_byte` retrieves a byte from within an atom.
12. `u3r_bytes` retrieves multiple bytes from within an atom.
13. `u3r_cell` produces a cell `[a b]`.
14. `u3r_chub` is the >32-bit equivalent of `u3r_word`.
15. `u3r_chubs` is the >32-bit equivalent of `u3r_words`.
16. `u3r_mean` deconstructs a noun by axis address.
17. `u3r_met` reports the total size of an atom.
18. `u3r_trel` factors a noun into a three-element cell `[a b c]`.
19. `u3r_word` retrieves a value from an atom as a C `uint32_t`.
20. `u3r_words` is the multi-element (array) retriever like `u3r_word`.

**`u3` Samples**

Defining jets which have a different sample size requires querying the correct nodes of the sample as binary tree:

```
1.  1 argument → `u3x_sam`
2.  2 arguments → `u3x_sam_2`, `u3x_sam_3`
3.  3 arguments → `u3x_sam_2`, `u3x_sam_6`, `u3x_sam_7`
4.  4 arguments → `u3x_sam_2`, `u3x_sam_6`, `u3x_sam_14`, `u3x_sam_15`
5.  5 arguments → `u3x_sam_2`, `u3x_sam_6`, `u3x_sam_14`, `u3x_sam_30`, `u3x_sam_31`
6.  6 arguments → `u3x_sam_2`, `u3x_sam_6`, `u3x_sam_14`, `u3x_sam_30`, `u3x_sam_62`, `u3x_sam_63`
```

A more complex argument structure requires grabbing other entries; e.g.,

```hoon
|=  [u=@lms [ia=@ud ib=@ud] [ja=@ud jb=@ud]]
```

requires

```c
u3x_sam_2, u3x_sam_12, u3x_sam_13, u3x_sam_14, u3x_sam_15
```

**Exercise: Review Jet Code**

* We commend to the reader the exercise of selecting particular Hoon-language library functions provided with the system, such as [`+cut`](https://github.com/urbit/urbit/blob/ceed4b78d068d7cb70350b3cd04e7525df1c7e2d/pkg/arvo/sys/hoon.hoon#L854), locating the corresponding jet code in:

  * [`tree.c`](https://github.com/urbit/urbit/blob/cd400dfa69059e211dc88f4ce5d53479b9da7542/pkg/urbit/jets/tree.c#L1575)
  * [`w.h`](https://github.com/urbit/urbit/blob/cd400dfa69059e211dc88f4ce5d53479b9da7542/pkg/urbit/include/jets/w.h#L53)
  * [`q.h`](https://github.com/urbit/urbit/blob/cd400dfa69059e211dc88f4ce5d53479b9da7542/pkg/urbit/include/jets/q.h#L51)
  * [`cut.c`](https://github.com/urbit/urbit/blob/cd400dfa69059e211dc88f4ce5d53479b9da7542/pkg/urbit/jets/c/cut.c)

  and learning in detail how particular operations are realized in `u3` C. Note in particular that jets do not need to follow the same solution algorithm and logic as the Hoon code; they merely need to reliably produce the same result.

### Jet Composition: Floating-Point `+factorial` <a href="#jet-composition-floating-point-factorial" id="jet-composition-floating-point-factorial"></a>

Let us examine jet composition using a more complicated floating-point operation. The Urbit runtime uses [SoftFloat](http://www.jhauser.us/arithmetic/SoftFloat-3/doc/SoftFloat.html) to provide a reference software implementation of floating-point mathematics. This is slower than hardware FP but more portable.

This library `lib/trig-rs.hoon` provides a few transcendental functions useful in many mathematical calculations. The `~%` "sigcen" rune registers the jets (with explicit arguments, necessary at the highest level of inclusion). The `~/` "sigfas" rune indicates which arms will be jetted.

**`/lib/trig-rs.hoon`**

```hoon
::  Transcendental functions library, compatible with @rs
::
=/  tau  .6.28318530717
=/  pi   .3.14159265358
=/  e    .2.718281828
=/  rtol  .1e-5
~%  %trig  ..part  ~
|%
:: Factorial, $x!$
::
++  factorial
  ~/  %factorial
  |=  x=@rs  ^-  @rs
  =/  t=@rs  .1
  |-  ^-  @rs
  ?:  =(x .0)  t
  ?:  =(x .1)  t
  $(x (sub:rs x .1), t (mul:rs t x))
:: Absolute value, $|x|$
::
++  absolute
  |=  x=@rs  ^-  @rs
  ?:  (gth:rs x .0)
    x
  (sub:rs .0 x)
:: Exponential function, $\exp(x)$
::
++  exp
  ~/  %exp
  |=  x=@rs  ^-  @rs
  =/  rtol  .1e-5
  =/  p   .1
  =/  po  .-1
  =/  i   .1
  |-  ^-  @rs
  ?:  (lth:rs (absolute (sub:rs po p)) rtol)
    p
  $(i (add:rs i .1), p (add:rs p (div:rs (pow-n x i) (factorial i))), po p)
:: Integer power, $x^n$
::
++  pow-n
  ~/  %pow-n
  |=  [x=@rs n=@rs]  ^-  @rs
  ?:  =(n .0)  .1
  =/  p  x
  |-  ^-  @rs
  ?:  (lth:rs n .2)
    p
  ::~&  [n p]
  $(n (sub:rs n .1), p (mul:rs p x))
--
```

We will create a generator which will pull the arms and slam each gate such that we can assess the library's behavior. Later on we will create unit tests to validate the behavior of both the unjetted and jetted code.

**`/gen/trig-rs.hoon`**

```hoon
/+  *trig-rs
!:
:-  %say
|=  [[* eny=@uv *] [x=@rs n=@rs ~] ~]
::
~&  (factorial n)
~&  (absolute x)
~&  (exp x)
~&  (pow-n x n)
[%verb ~]
```

We will further define a few unit tests as checks on arm behavior:

**`/tests/lib/trig-rs.hoon`**

```hoon
/+  *test, *trig-rs
::
::::
  ::
|%
++  test-factorial  ^-  tang
  ;:  weld
    %+  expect-eq
      !>  .1
      !>  (factorial .0)
    %+  expect-eq
      !>  .1
      !>  (factorial .1)
    %+  expect-eq
      !>  .120
      !>  (factorial .5)
    %+  expect-eq
      !>  .720
      !>  (factorial .6)
  ==
--
```

#### Jet Composition <a href="#jet-composition" id="jet-composition"></a>

As before, the jet hints must provide a breadcrumb trail of symbols for the interpreter to know how to match the Hoon arms to the corresponding C code.

```hoon
~%  %trig-rs  ..part  ~
|%
++  factorial
  ~/  %factorial
  |=  x=@rs  ^-  @rs
  ...
++  exp
  ~/  %exp
  |=  x=@rs  ^-  @rs
  ...
++  pow-n
  ~/  %pow-n
  |=  [x=@rs n=@rs]  ^-  @rs
  ...
--
```

1. Register the jet symbols and function names in `tree.c`.
2. Declare function prototypes in headers `q.h` and `w.h`.
3. Produce functions for compilation and linking in the`pkg/noun/jets/e` directory.

**Register the jet symbols and function names.**

A jet registration may be carried out at any point in `tree.c`. The registration consists of marking the core

In **`pkg/noun/jets/tree.c`**:

```c
/* Jet registration of ++factorial arm under trig-rs */
static u3j_harm _140_hex__trigrs_factorial_a[] = {{".2", u3we_trigrs_factorial, c3y}, {}};
/* Associated hash */
static c3_c* _140_hex__trigrs_factorial_ha[] = {
  "903dbafb8e59427eced0b35379ad617c2eb6083a235075e9cdd9dd80e732efa4",
  0
};

static u3j_core _140_hex__trigrs_d[] =
  { { "factorial", 7, _140_hex__trigrs_factorial_a, 0, _140_hex__trigrs_factorial_ha },
  {}
  };
static c3_c* _140_hex__trigrs_ha[] = {
  "0bac9c3c43634bb86f6721bbcc444f69c83395f204ff69d3175f3821b1f679ba",
  0
};

/* Core registration by token for trigrs */
static u3j_core _140_hex_d[] =
{ /* ... pre-existing jet registrations ... */
  { "trig-rs",   31, 0, _140_hex__trigrs_d, _140_hex__trigrs_ha  },
  {}
};
```

**Declare function prototypes in headers.**

We must declare `u3we_trigrs_factorial` and `u3qe_trigrs_factorial`:

In `w.h`:

```c
u3_noun u3we_trigrs_factorial(u3_noun);
```

In `q.h`:

```c
u3_noun u3qe_trigrs_factorial(u3_atom);
```

**Produce functions for compilation and linking.**

Given these function prototype declarations, all that remains is the actual definition of the function. Both functions will live in their own file; we find it the best convention to associate all arms of a core in a single file. In this case, create a file `pkg/noun/jets/e/trig-rs.c` and define all of your `trig-rs` jets therein. (Here we show `+factorial` only.)

**`pkg/noun/jets/e/trig-rs.c`**

```c
/* jets/e/trig-rs.c
**
*/
#include "all.h"
#include <softfloat.h>  // necessary for working with software-defined floats
#include <stdio.h>      // helpful for debugging, removable after development
#include <math.h>       // provides library fabs() and ceil()

  union sing {
    float32_t s;    //struct containing v, uint_32
    c3_w c;         //uint_32
    float b;        //float_32, compiler-native, useful for debugging printfs
  };

/* ancillary functions
*/
  bool isclose(float a,
               float b)
  {
    float atol = 1e-6;
    return ((float)fabs(a - b) <= atol);
  }

/* factorial of @rs single-precision floating-point value
*/
  u3_noun
  u3qe_trigrs_factorial(u3_atom u)  /* @rs */
  {
    fprintf(stderr, "u3qe_trigrs_factorial\n\r");  // DELETE THIS LINE LATER
    union sing a, b, c, e;
    u3_atom bb;
    a.c = u3r_word(0, u);  // extricate value from atom as 32-bit word

    if (ceil(a.b) != a.b) {
      // raise an error if the float has a nonzero fractional part
      return u3m_bail(c3__exit);
    }

    if (isclose(a.b, 0.0)) {
      a.b = (float)1.0;
      return u3i_words(1, &a.c);
    }
    else if (isclose(a.b, 1.0)) {
      a.b = (float)1.0;
      return u3i_words(1, &a.c);
    }
    else {
      // naive recursive algorithm
      b.b = a.b - 1.0;
      bb = u3i_words(1, &b.c);
      c.c = u3r_word(0, u3qe_trig_factorial(bb));
      e.s = f32_mul(a.s, c.s);
      u3m_p("result", u3i_words(1, &e.c));  // DELETE THIS LINE LATER
      return u3i_words(1, &e.c);
    }
  }

  u3_noun
  u3we_trigrs_factorial(u3_noun cor)
  {
    fprintf(stderr, "u3we_trigrs_factorial\n\r");  // DELETE THIS LINE LATER
    u3_noun a;

    if ( c3n == u3r_mean(cor, u3x_sam, &a, 0) ||
         c3n == u3ud(a) )
    {
      return u3m_bail(c3__exit);
    }
    else {
      return u3qe_trigrs_factorial(a);
    }
  }
```

This code deviates from the integer implementation in two ways: because all `@rs` atoms are guaranteed to be 32-bits, we can assume that `c3_w` can always contain them; and we are using software-defined floating-point operations with SoftFloat.

We have made use of `u3r_word` to convert a 32-bit (really, 31-bit or smaller) Hoon atom (`@ud`) into a C `uint32_t` or `c3_w`. This unsigned integer may be interpreted as a floating-point value (similar to a cast to `@rs`) by the expedient of a C `union`, which allows multiple interpretations of the same bit pattern of data; in this case, as an unsigned integer, as a SoftFloat `struct`, and as a C single-precision `float`.

`f32_mul` and its sisters (`f32_add`, `f64_mul`, `f128_div`, etc.) are floating-point operations defined in software ([Berkeley SoftFloat](http://www.jhauser.us/arithmetic/SoftFloat.html)). These are not as efficient as native hardware operations would be, but allow Urbit to guarantee cross-platform compatibility of operations and not rely on hardware-specific implementations. Currently all Urbit floating-point operations involving `@r` values use SoftFloat.

#### Compiling and Using the Jet <a href="#compiling-and-using-the-jet" id="compiling-and-using-the-jet"></a>

With this one jet for `+factorial` in place, compile the jet and take note of where Nix produces the binary.

```sh
$ make
```

Copy the affected files back into the ship's pier:

```sh
$ cp base/lib/trig-rs.hoon zod/base/lib
$ cp base/gen/trig-rs.hoon zod/base/gen
```

Restart your fakezod using the new Urbit binary and synchronize these to the `%home` desk:

```hoon
> |commit %base
```

If all has gone well to this point, you are prepared to test the jet using the `%say` generator from earlier:

```hoon
> +trig 5
120
```

Among the other output values, you should observe any `stderr` messages emitted by the jet functions each time they are called.

**`pkg/noun/jets/e/trig.c`**

```c
/* integer power of @rs single-precision floating-point value
*/
  u3_noun
  u3qe_trigrs_pow_n(u3_atom x,  /* @rs */
                  u3_atom n)  /* @rs */
  {
    fprintf(stderr, "u3qe_trig_pow_n\n\r");
    union sing x_, n_, f_;
    x_.c = u3r_word(0, x);  // extricate value from atom as 32-bit word
    n_.c = u3r_word(0, n);

    f_.b = (float)pow(x_, n_);

    return u3i_words(1, &f_.c);
  }

  u3_noun
  u3w_trigrs_pow_n(u3_noun cor)
  {
    fprintf(stderr, "u3w_trig_pow_n\n\r");
    u3_noun a, b;

    if ( c3n == u3r_mean(cor, u3x_sam_2, &a,
                              u3x_sam_3, &b, 0) ||
         c3n == u3ud(a) || c3n == u3ud(b) )
    {
      return u3m_bail(c3__exit);
    }
    else {
      return u3q_trigrs_pow_n(a, b);
    }
  }
```

The type union `sing` remains necessary to easily convert the floating-point result back into an unsigned integer atom.

**Exercise: Implement the Other Jets**

* We leave the implementation of the other jets to the reader as an exercise. (Please do not skip this: the exercise will both solidify your understanding and raise new important situational questions.)

  Again, the C jet code need not follow the same logic as the Hoon source code; in this case, we simply use the built-in `math.h` `pow` function. (We could—arguably should—have used SoftFloat-native implementations, but that is more involved than this tutorial intends.)

### Jetting the Kernel <a href="#jetting-the-kernel" id="jetting-the-kernel"></a>

Hoon jets are compiled into the Vere binary for distribution with the Urbit runtime. Per current development policy, this is the only way to actually share jets with other developers.

Jets are registered with the runtime so that Vere knows to check whether a particular jet exists when it encounters a marked Hoon arm.

* [`~/` sigfas](/hoon/rune/sig#sigfas) registers a jet simply (using defaults).
* [`~%` sigcen](/hoon/rune/sig#sigcen) registers a jet with all arguments specified.

Typically we use `~/` sigfas to register jets within a core under the umbrella of a `~%` sigcen registration. For instance, `+add` is registered under the Kelvin tag of `hoon.hoon`:

```hoon
~%  %k.140  ~  ~                                        ::
|%
++  hoon-version  +
--  =>
~%  %one  +  ~
|%
++  add
  ~/  %add
  |=  [a=@ b=@]
  ^-  @
  ?:  =(0 a)  b
  $(a (dec a), b +(b))
```

As a generic example, let us consider three nested arms within cores. We intend to jet only `+ccc`, but we need to give Vere a way of tracking the jet registration for all containing cores.

```hoon
  ++  aaa
    ~%  %aaa  ..is  ~
    ...
    ++  bbb
      ~/  %bbb
      ++  ccc
        ~/  %ccc
        |=  dat=@
        ^-  pont
        =+  x=(end 3 w a)
        =+  y=:(add (pow x 3) (mul a x) b)
        =+  s=(rsh 3 32 dat)
        :-  x
        ?:  =(0x2 s)  y
        ?:  =(0x3 s)  y
        ~|  [`@ux`s `@ux`dat]
        !!
```

We hint `+ccc` with `%ccc` and add a trail of hints up the enclosing tree of arms. `~/` sigfas takes only the `term` symbol used to label the hint because it knows the context, but `~%` sigcen needs two more fields: the parent jet and some core registration information (which is often `~` null). We here use the parent of `..is`, a system-supplied jet, as the parent jet. Since `+is` is an arm of the Arvo core, `..is` is a reference to the entire Arvo core. The whole Arvo core is hinted with the jet label `%hex`, which is used as the parent for all the top-level jet hints in %zuse.

When hinting your own code, make sure to hint each nesting arm. Skipping any nesting core will result in the jet code not being run.

You do not need to provide C implementations for everything you hint. In the above, we hint `%aaa`, `%bbb`, and `%ccc`—even if our intent is only to jet `+ccc`.

#### Editing the C Source Code <a href="#editing-the-c-source-code" id="editing-the-c-source-code"></a>

Having hinted our Hoon, we now need to write the matching C code. If we don't, there isn't a problem—hinting code merely tells the interpreter to look for a jet, but if a jet is not found, the Hoon still runs just fine.

This whole process recapitulates what you've done above, but in a generic way.

There are two distinct tasks to be done C-side:

1. Write the jet.
2. Register the jet.

For each jet you will write one `u3we()` function and one `u3qe()` function.

**Edit the C Source Code to Add Registration**

* Edit the header file `include/jets/w.h` to have a declaration for each of your `u3we()` functions. Every `u3we()` function looks the same, e.g.

  ```c
  u3_noun u3we_xxx(u3_noun);
  ```
* Edit the header file `~/jetting/urbit/pkg/urbit/include/jets/q.h` to have a declaration for your `u3qe()` function. `u3qe()` functions can differ from each other, taking distinct numbers of `u3_nouns` and/or `u3_atoms`, e.g.

  ```c
  u3_noun u3qe_yyy(u3_atom, u3_atom);
  u3_noun u3qe_zzz(u3_noun, u3_noun, u3_atom, u3_atom);
  ```
* Create a new `.c` file to hold your jets; both the `u3we_()` and `u3qe_()` functions go in the same file, for instance `~/jetting/urbit/pkg/noun/jets/e/secp.c`. The new file should include at least the following three things:
  * `#include "all.h"`
  * the new `u3we()` function
  * the new `u3qe()` function
* Edit `~/jetting/urbit/pkg/urbitjets/tree.c` to register the jet.

In the Hoon code we hinted some leaf node functions (`%ccc` for `+ccc` in our example) and then hinted each parent node up to the root `%aaa`/`+aaa`). We need to replicate this structure in C. Here's example C code to jet our above example Hoon:

```c
      // 1: register a C func u3we_ccc()
      static u3j_harm _143_hex_hobo_reco_d[] =
        {
        {".2", u3we_ccc, c3y},
        {}
        };

      // 2: that implements a jet for Hoon arm 'ccc'
      static u3j_core _143_hex_hobo_bbb_d[] =
        {
        { "ccc",            _143_hex_hobo_ccc_d },
        {}
        };

      // 3: ... that is inside a Hoon arm 'bbb'
      static u3j_core _143_hex_hobo_hobo_d[] =
        {
        { "bbb", 0,   _143_hex_hobo_bbb_d },
        {}
        };

      // 4: ... that is inside a Hoon arm 'aaa'
      static u3j_core _143_hex_d[] =
      { { "down", 0, _143_hex_down_d },
        { "lore", _143_hex_lore_a },
        { "loss", _143_hex_loss_a },
        { "lune", _143_hex_lune_a },
        { "coed", 0, _143_hex_coed_d },
        { "aes", 0, _143_hex_aes_d },
        { "hmac", 0, _143_hex_hmac_d },
        { "aaa", 0, _143_hex_hobo_d },
        {}
      };
```

There are 4 steps here. Let's look at each in turn.

1. Section 1 names the C function that we want to invoke: `u3we_ccc()`. The precise manner in which it does this is by putting entries in an array of `u3j_harm`s. The first one specifies the jet; the second one is empty and serves as a termination to the array, similar to how a C string is null terminated with a zero. The jet registration supplies two fields `{".2", u3we_secp}`, but this does not initialize all of the fields of `u3j_harm`. Other fields can be specified.

   The first field, with value ".2" in this example, is "arm 2". `".2"` labels the axis of the arm in the core. With a `%fast` hint (`~/` sigfas ), we're hinting a gate, so the relevant arm formula is always just the entire battery at `+2`.

   The second field, with value `u3we_ccc` in this example, is a function pointer (to the C implementation of the jet).

   The third field (absent here) is a flag to turn on verification of C jet vs Hoon at run time. It can take value `c3n` (which means verify at run time) or `c3y` (which means don't verify). If not present, it is set to don't verify.

   There are additional flags; see \~/tlon/urbit/include/noun/jets.h
2. Section 2 associated the previous jet registration with the name `"ccc"`. This must be the same symbol used in the Hoon hint. We again have a “null terminated” (metaphorically) list, ending with `{}`.

   Section 3 references structure built in step 2 and slots it under `bbb` (again, note that this is exactly the same symbol used in the hinting in Hoon).

   The line in section 2

   ```c
         { "ccc",            _143_hex_hobo_ccc_a },
   ```

   looks very similar to the line in section 3

   ```c
       { "bbb", 0,   _143_hex_hobo_bbb_d },
   ```

   But note that the line in section 2 fill in the first 2 fields in the struct, and the line in section 3 fills in the first three fields. Section 2 is registering an array of `u3j_harm`, i.e. is registering an actual C jet.
3. Section 3 specifies `0` for the array of `u3j_harm` and is instead specifying an array of `u3j_core`, i.e. it is registering nesting of another core which is not a leaf node.
4. Section 4 is much like section 3, but it's the root of this particular tree. Section 4 is also an example of how a given node in the jet registration tree may have multiple children.

You should be able to register jets whether your nesting is 2 layers deep, 3 (like this example), or more. You should also be able to register multiple jets at the same nesting level (e.g. a function `u3we_ddd()` which is a sibling of `u3we_ccc()` inside data structure `_143_hex_hobo_reco_d[]` ).

**Edit the C Source Code to Add the `u3we_()` Function**

There are two C functions per jet, because separation of concerns is a good thing.

The first C function—named `u3we_xxx()`—unpacks arguments from the Hoon code and gets them ready.

The second C function -- named `u3qe_xxx()`—takes those arguments and actually performs the operations that parallel the Hoon code being jetted.

Let's write the `u3we_xxx()` function first. This function accepts one argument, of type `u3_noun`. This is the same type as a Hoon noun (`*`). This one argument is the payload. The payload is a tree, obviously.

The payload consists of (on the right branch) the context (you'd think of “global variables and available methods”, if analogies to other programming languages were allowed!) and on the left branch the sample (the arguments to this particular function call).

Your `u3we_xxx()` function does one thing: unpacks the sample from `cor`, sanity checks them, and passes them to the `u3qe_xxx()` function.

To unpack the sample, we use the function `u3r_mean()` to do this, thusly:

```c
      u3_noun arg_a, arg_b, arg_c  ... ;

      u3r_mean(cor,
               axis_a, & arg_a,
               axis_b, & arg_b,
               axis_c, & arg_c
         ...
         0)
```

If we want to to assign the data located at axis 3 of cor to `arg_a`, we'd set `axis_a = 3`.

`u3r_mean()` takes [varargs](https://en.wikipedia.org/wiki/Variadic_function), so we can pass in as many axis/return-argument pairs as we wish, terminated with a `0`. You saw above how to pull the sample arguments out of the right-descending trees (because a linked list is a degenerate case of a tree).

If the Hoon that you're jetting looks like this

```hoon
++  make-k
~/  %make-k
=,  mimes:html
|=  [aaa=@ bbb=@ ccc=@]
```

In the C code you'd fetch them out of the payload with

```c
u3r_mean(cor,
         u3x_sam_2, & arg_aaa,
         u3x_sam_5, & arg_bbb,
         u3x_sam_6, & arg_ccc
         ...
         0)
```

If you're confident, go ahead and write code. If you want to inspect your arguments to see what's going on, you can pretty print the sample.

You could in theory inspect/pretty-print the noun by calling

```c
u3m_p("description", cor);  :: DO NOT DO THIS !!!
```

… but you don't want to do this, because, recall, `cor` contains the *entire* context.

Do instead, perhaps,

```c
c3_o ret;
u3_noun sample;

ret = u3r_mean(sample, u3x_sam_1,  &sample, 0);
fprintf(stderr, "ret = %i\n\r", ret); // we want ret = 0 = yes

u3m_p("sample", sample);      // pretty print the entire sample
```

After our C function pulls out the arguments it needs to typecheck them.

If `arg_a` is supposed to be a atom, trust but verify:

```c
u3ud(arg_a);  // checks for atomicity; alias for u3a_is_atom()
```

If it's supposed to be a cell:

```c
u3du(arg_a);  // checks for cell-ness
```

There are other tests you might need to use

```c
u3a_is_cat()  // check whether the noun is a direct atom (31 bits or less)
u3a_is_dog()  // check whether the noun is an indirect noun (32+ bits)

u3a_is_pug()  // check whether noun is indirect atom
u3a_is_pom()  // check whether noun is indirect cell
```

All of these tests return Hoon loobeans (`yes` 0/`no` 1 vs. `TRUE`/`FALSE`), so check return values vs `c3n` / `c3y`. If any of these `u3_mean()`, `u3ud()` etc return `u3n` you have an error and should return

```c
return u3m_bail(c3__exit);
```

Otherwise, pass the arguments into your inner jet function and return the results of that.

**Edit the C Source Code to Add the `u3qe_()` Function**

**Unpacking Nouns**

The `u3qe_xxx()` function is the real jet—the C code that replaces the Hoon.

First, you may need to massage your inputs a bit to get them into types that you can use.

You have received a bunch of `u3_nouns` or `u3_atoms`, but you presumably want to do things in a native C/non-Hoon manner: computing with raw integers, etc.

A `u3_noun` will want to be further disassembled into atoms.

A `u3_atom` represents a simple number, but the implementation may or may not be simple. If the value held in the atom is 31 bits or less, it's stored directly in the atom. If the value is 32 bits the atom holds a pointer into the loom where the actual value is stored. ( see [Nouns](/build-on-urbit/runtime/nouns) )

You don't want to get bogged down in the details of this—you just want to get data out of your atoms.

If you know that the data fits in 32 bits or less, you can use

```c
u3r_word(c3_w    a_w, u3_atom b);
```

If it is longer than 32 bits, use

```c
u3r_words(c3_w    a_w, c3_w    b_w, c3_w*   c_w, u3_atom d);
```

or

```c
  u3r_bytes(c3_w    a_w, c3_w    b_w, c3_y*   c_y, u3_atom d)
```

If you need to get the size, use

```c
u3r_met(3, a);
```

Cells have their own set of characteristic functions for accessing interior nouns: `u3r_cell`, `u3r_trel`, `u3r_qual`, `u3h`, `u3t`, and the like.

The actual meat of the function is up to you. What is the function supposed to do for Hoon?

**Packing Nouns**

Now we move on to return semantics.

First, you can transfer raw values into nouns using

```c
u3_noun u3i_words(c3_w a_w, const c3_w* b_w)
```

and you can build cells out of nouns using

```c
u3nc();  // pair
u3nt();  // triple
u3nq();  // quad
```

There are two facets here:

1. **Data format.** If the Hoon is expected to return a single atom (e.g. if the Hoon looks like this:)

   ```hoon
   ++  make-k
     ~/  %make-k
     |=  [has=@uvI prv=@]     ::  <---- input parguments
     ^-  @                    ::  <---- return value is a single value of type '@' (atom)
     ...
   ```

   then your C code—at least when you're stubbing it out—can do something like

   ```c
   return(123);
   ```

   Or, if you want to create an atom more formally, you can build it like this

   ```c
   // this variable is on the stack and will disappear
   unsigned char nonce32[32];

   // this allocates an indirect (> 31 bits) atom in the loom,
   // does appropriate reference count, and returns the 32 bit handle to the atom
   u3_noun nonce = u3i_words(8, (const c3_w*) nonce32);

   // this returns the 32 bit handle to the atom
   return(nonce);
   ```

   If, on the other hand, your Hoon looks like

   ```hoon
   ++  ecdsa-raw-sign
     ~/  %ecdsa-raw-sign
     |=  [has=@uvI prv=@]     ::  <---- input parguments
     ^-  [v=@ r=@ s=@]        ::  <---- return value is a cell
     ...
   ```

   ending your C code with

   ```c
   return(123);
   ```

   is wrong and will result in a runtime error because you are returning a single atom, instead of a tuple containing three atoms.

   Instead do one of these:

   ```c
   return(u3nc(a, b));        // for two atoms
   return(u3nt(a, b, c));     // for three atoms
   return(u3nq(a, b, c, d));  // for four atoms
   ```

   If you need to return a longer tuple, you can compose your own. Look at the definitions of these three functions and you will see that they are just recursive calls to the cell constructor `u3i_cell()` e.g.

   ```c
   u3i_cell(a, u3i_cell(b, u3i_cell(c, d));
   ```

   This implies that, to create a list instead of a cell, you will need to append `u3_nul` to the appropriately-sized tuple constructor:

   ```c
   return(u3nt(a, b, u3_nul));    // for two atoms as a list
   ```
2. **Memory allocation.** Understanding the memory model, allocation, freeing, and ownership ('transfer' vs 'retain' semantics) is important. More information is available in [the “Nouns” docs](/build-on-urbit/runtime/nouns).

### Pills <a href="#pills" id="pills"></a>

A *pill* is a Nock “binary blob”, really a parsed Hoon abstract syntax tree. Pills are used to bypass the bootstrapping procedure for a new ship, and are particularly helpful when jetting code in `hoon.hoon`, `%zuse`, `%lull`, or the main Arvo vanes.

An Urbit ship has to boot into the Arvo kernel—a Nock core with a particular interface. While it would be possible to make some ad-hoc procedure to initialize Arvo, it would be a drastic layering violation and couple Urbit to all sorts of internal implementation details of Arvo and Hoon. In contrast, a pill is basically a serialized set of declarative steps to initialize Arvo.

You don't strictly need to use pills in producing jets in `/lib`, but it can speed up your development cycle significantly. However, you *must* use pills when working on the core kernel (`hoon.hoon`, `zuse.hoon`, `arvo.hoon`).

#### Producing a Pill <a href="#producing-a-pill" id="producing-a-pill"></a>

Having edited the C code, you now need to compile it to build a new runtime executable.

```sh
$ cd ~/jetting/vere
$ bazel build :urbit
```

You need to compile this in C and in Hoon, however. When the Urbit executable runs, the first thing it does is load the complete Arvo operating system. That step is much faster if it can load a jammed pill, where all of the Hoon has already been parsed from text file into Hoon abstract syntax tree, and then compiled from the Hoon into the Nock equivalent.

Critically, this means that if you edit `hoon.hoon`, `zuse.hoon`, `arvo.hoon`, `lull.hoon`, `/sys/vane/ames.hoon`, etc., and then restart the executable, **you are not running your new code**.

The only way to run the new code is to follow the following process:

1. Start up a new fakeship (typically `~zod`) which knows where your edited Arvo files are (although it will not execute them, as discussed above).
2. From the Dojo command line, load the Hoon files and compile them into a `pill` file:

   ```hoon
   > .pill +pill/solid
   %solid-start
   %solid-loaded
   %solid-parsed
   %solid-compiled
   %solid-arvo
   [%solid-kernel 0x6aa7.627e]
   %arvo-assembly
   [%solid-veer p=%$ q=/zuse]
   [%tang /~zod/home/~2018.7.25..20.47.51..0027/sys/zuse ~mondyr-rovmes]
   ```

   If this is successful, then you are ready to move forwards. Otherwise, correct the syntax errors and iterate.
3. Exit the ship with `Ctrl+D` or `|exit`.
4. Save the pill file.

   ```sh
   $ cd ~/jetting
   $ cp zod/.urb/put/.pill  ./mypill.pill
   ```

**Run the Compiled C/Compiled Hoon Pill**

5. Prepare a new fakezod (you can't use a backup fakezod here because the point is to boot from scratch again):

   ```sh
   $ cd ~/jetting
   $ rm -rf zod
   $ /path/to/new/urbit -F zod -B ~/tlon/mypill.pill`
   ```

If booting takes more than about 90 seconds, you may have created a ‘poison pill’, which hangs things. Try booting without the `-B` flag, and/or reverting your Hoon changes, generating a new pill based on that, and launching `urbit` with the known-clean pill. If these steps and boot in <90 seconds, but a boot with a pill created from your own Hoon does not, you have a Hoon bug of some sort.

Hoon bugs that disable booting can be as simple as the wrong number of spaces. Many, but not all of them, will result in compile errors during the `.pill +pill/solid` step. If your booting takes >90 seconds, abort it, and debug at your Hoon code.

* Inside the Dojo, `|commit` the changed `hoon.hoon` or other system file. It should automatically recompile if correct.

You now have created a galaxy fakezod, on its own detached network, running your own strange variant of the OS.

6. Run and test your jetting code, e.g. `(ccc:bbb:aaa 1 2 3)`.

(As an aside, should you see “biblical” names like `noah`, this means that you are using a feature of the kernel in a core before it is available. You'll need to move things to a later point in the file or change your code if that happens.)

### Testing Jets <a href="#testing-jets" id="testing-jets"></a>

All nontrivial code should be thoroughly tested to ensure software quality. To rigorously verify the jet's behavior and performance, we will combine live testing in a single Urbit session, comparative behavior between a reference Urbit binary and our modified binary, and unit testing.

1. Live spot checks rely on you modifying the generator `trig-rs.hoon` and observing whether the jet works as expected.

   When producing a library, one may use the `-build-file` thread to build and load a library core through a face. Two fakezods can be operated side-by-side in order to verify consistency between the Hoon and C code.

   ```hoon
   > =trig-rs -build-file %/lib/trig-rs/hoon

   > (exp:trig-rs .5)
   ```
2. Comparison to the reference Urbit binary can be done with a second development ship and the same Hoon library and generator.
3. Unit tests rely on using the `-test` thread as covered in [Hoon School](/build-on-urbit/hoon-school/i-testing) and the [testing guide](/build-on-urbit/userspace/unit-tests).

   ```hoon
   > -test %/tests/lib/trig-rs ~
   ```
4. One of the arguments to the C function registration forces comparison of the results of the Hoon/Nock code and the C jet.

   It can take value `c3n` (which means to verify the jet's behavior at run time) or `c3y` (which means to not verify). If not present, it will not verify.

   Why is `c3y` ("yes") used to turn OFF verification? Because the flag is actually asking, “Is this jet already known to be correct?”
5. There are integration tests available for the Urbit repository; you should investigate the now-current standard of practice for implementing and including these with your jetted code submission.

### Et Cetera <a href="#et-cetera" id="et-cetera"></a>

We omit from the current discussion a few salient points:

1. Reference counting with transfer and retain semantics. (For everything the new developer does outside of real kernel shovel work, one will use transfer semantics.) These are discussed in [the “Noun” docs](/build-on-urbit/runtime/nouns).
2. The structure of memory: the loom, with outer and inner roads. This is discussed in [the “Noun” docs](/build-on-urbit/runtime/nouns).
3. Many details of C-side atom declaration and manipulation from the `u3` library. These are discussed in [the API docs](/build-on-urbit/runtime/api).
4. `fprintf`-based output should be done using `fprintf()` to `stderr`. Use both and to achieve line feed (move cursor down one line) and carriage return (move it to the left). You can also use `u3l_log` which does not require `\r`, but should not be used in cases where the IO drivers have not yet been initialized or can no longer be relied upon, e.g. crashing or shutdown.
5. A jet can be partial: it can solve certain cases efficiently but leave others to the Hoon implementation. A `u3w_*` jet interface function takes the entire core as one noun argument and returns a `u3_weak` result. If the return value is `u3_none` (distinct from `u3_nul`, `~` null), the core is evaluated; otherwise the resulting noun is produced in place of the nock.


# API Overview by Prefix

API overview by prefix for u3 system modules. Function reference for noun manipulation, jets, allocation, Nock computation.

Let's run through the `u3` modules one by one. All public functions are commented, but the comments may be cryptic.

## u3m: main control <a href="#u3m-main-control" id="u3m-main-control"></a>

To start `u3`, run

```c
/* u3m_boot(): start the u3 system.
*/
  void
  u3m_boot(c3_o nuu_o, c3_o bug_o, c3_c* dir_c);
```

`nuu_o` is `c3y` (yes, `0`) if you're creating a new pier, `c3n` (no, `1`) if you're loading an existing one. `bug_o` is `c3y` if you want to test the garbage-collector, `c3n` otherwise. `dir_c` is the directory for the pier files.

`u3m_boot()` expects an `urbit.pill` file to load the kernel from. It will try first `$dir/.urb.urbit.pill`, then `U3_LIB`.

Any significant computation with nouns, certainly anything Turing complete, should be run (a) virtualized and (b) in an inner road. These are slightly different things, but at the highest level we bundle them together for your convenience, in `u3m_soft()`:

```c
/* u3m_soft(): system soft wrapper.  unifies unix and nock errors.
**
**  Produces [%$ result] or [%error (list tank)].
*/
  u3_noun
  u3m_soft(c3_w sec_w, u3_funk fun_f, u3_noun arg);
```

`sec_w` is the number of seconds to time out the computation. `fun_f` is a C function accepting `arg`.

The result of `u3m_soft()` is a cell whose head is an atom. If the head is `%$` - ie, `0` - the tail is the result of `fun_f(arg)`. Otherwise, the head is a `term` (an atom which is an LSB first string), and the tail is a `(list tank)` (a list of `tank` printables - see `+tank` in `hoon.hoon`). Error terms should be the same as the exception terms above.

If you're confident that your computation won't fail, you can use `u3m_soft_sure()`, `u3m_soft_slam()`, or `u3m_soft_nock()` for C functions, Hoon function calls, and Nock invocations. Caution - this returns just the result, and asserts globally.

All the `u3m_soft` functions above work **only on the surface**. Within the surface, virtualize with `u3m_soft_run()`. Note that this takes a `fly` (a namespace gate), thus activating the `11` super-operator in the nock virtualizer, `+mock`. When actually using the `fly`, call `u3m_soft_esc()`. Don't do either unless you know what you're doing!

For descending into a subroad **without** Nock virtualization, use `u3m_hate()` and `u3m_love` respectively. Hating enters a subroad; loving leaves it, copying out a product noun.

Other miscellaneous tools in `u3m`: `u3m_file()` loads a Unix file as a Nock atom; `u3m_water()` measures the boundaries of the loom in current use (ie, watermarks); and a variety of prettyprinting routines, none perfect, are available, mainly for debugging printfs: `u3m_pretty()`, `u3m_p()`, `u3m_tape()` and `u3m_wall()`.

It's sometimes nice to run a mark-and-sweep garbage collector, `u3m_grab()`, which collects the world from a list of roots, and asserts if it finds any leaks or incorrect refcounts. This tool is for debugging and long-term maintenance only; refcounts should never err.

## u3j: jets <a href="#u3j-jets" id="u3j-jets"></a>

The jet system, `u3j`, is what makes `u3` and `nock` in any sense a useful computing environment. Except perhaps `u3a` (there is really no such thing as a trivial allocator, though `u3a` is dumber than most) - `u3j` is the most interesting code in `u3`.

Let's consider the minor miracle of driver-to-battery binding which lets `u3j` work - and decrement not be `O(n)` - without violating the precisely defined semantics of pure Nock, **ever**.

It's easy to assume that jets represent an architectural coupling between Hoon language semantics and Nock interpreter internals. Indeed such a coupling would be wholly wrongtious and un-Urbit. But the jet system is not Hoon-specific. It is specific to nock runtime systems that use a design pattern we call a `core`.

### u3j: core structure <a href="#u3j-core-structure" id="u3j-core-structure"></a>

A core is no more than a cell `[code data]`, in which a `code` is either a Nock formula or a cell of `code`s, and `data` is anything. In a proper core, the subject each formula expects is the core itself.

Except for the arbitrary decision to make a core `[code data]`, (or as we sometimes say, `[battery payload]`), instead of `[data code]`, any high-level language transforming itself to Nock would use this design.

So jets are in fact fully general. Broadly speaking, the jet system works by matching a C **driver** to a battery. When the battery is invoked with Nock operator `9`, it must be found in associative memory and linked to its driver. Then we link the formula axis of the operation (`a` in `[9 a b]`) to a specific function in the driver.

To validate this jet binding, we need to know two things. One, we need to know the C function actually is a perfect semantic match for the Nock formula. This can be developed with driver test flags, which work, and locked down with a secure formula hash in the driver, which we haven't bothered with just yet. (You could also try to develop a formal method for verifying that C functions and Nock formulas are equivalent, but this is a research problem for the future.)

Two, we need to validate that the payload is appropriate for the battery. We should note that jets are a Nock feature and have no reference to Hoon. A driver which relies on the Hoon type system to only pair it with valid payloads is a broken driver, and breaks the Nock compliance of the system as a whole. So don't.

Now, a casual observer might look at `[battery payload]` and expect the simplest case of it to be `[formula subject]`. That is: to execute a simple core whose battery is a single formula, we compute

```
nock(+.a -.a)
```

Then, naturally, when we go from Hoon or a high-level language containing functions down to Nock, `[function arguments]` turns into `[formula subject]`. This seems like an obvious design, and we mention it only because it is **completely wrong**.

Rather, to execute a one-armed core like the above, we run

```
nock(a -.a)
```

and the normal structure of a `gate`, which is simply Urbitese for "function," is:

```
[formula [sample context]]
```

where `sample` is Urbitese for "arguments" - and `context`, any Lisper will at once recognize, is Urbitese for "environment."

To `slam` or call the gate, we simply replace the default sample with the caller's data, then nock the formula on the entire gate.

What's in the context? Unlike in most dynamic languages, it is not some secret system-level bag of tricks. Almost always it is another core. This onion continues until at the bottom, there is an atomic constant, conventionally is the kernel version number.

Thus a (highly desirable) `static` core is one of the form

```
[battery constant]
[battery static-core]
```

ie, a solid stack of nested libraries without any dynamic data. The typical gate will thus be, for example,

```
[formula [sample [battery battery battery constant]]]
```

but we would be most foolish to restrict the jet mechanism to cores of this particular structure. We cannot constrain a payload to be `[sample static-core]`, or even `[sample core]`. Any such constraint would not be rich enough to handle Hoon, let alone other languages.

### u3j: jet state <a href="#u3j-jet-state" id="u3j-jet-state"></a>

There are two fundamental rules of computer science: (1) every system is best understood through its state; (2) less state is better than more state. Sadly, a pier has three different jet state systems: `cold`, `warm` and `hot`. It needs all of them.

Hot state is associated with this particular Unix process. The persistent pier is portable not just between process and process, but machine and machine or OS and OS. The set of jets loaded into a pier may itself change (in theory, though not in the present implementation) during the lifetime of the process. Hot state is a pure C data structure.

Cold state is associated with the logical execution history of the pier. It consists entirely of nouns and ignores restarts.

Warm state contains all dependencies between cold and hot state. It consists of C structures allocated on the loom.

Warm state is purely a function of cold and hot states, and we can wipe and regenerate it at any time. On any restart where the hot state might have changed, we clear the warm state with `u3j_ream()`.

There is only one hot state, the global jet dashboard `u3j_Dash` or `u3D` for short. In the present implementation, u3D is a static structure not modified at runtime, except for numbering itself on process initialization. This structure - which embeds function pointers to all the jets - is defined in `j/tree.c`. The data structures:

```c
    /* u3j_harm: driver arm.
    */
      typedef struct _u3j_harm {
        c3_c*               fcs_c;            //  `.axe` or name
        u3_noun           (*fun_f)(u3_noun);  //  compute or 0 / semitransfer
        c3_o                ice;              //  perfect (don't test)
        c3_o                tot;              //  total (never punts)
        c3_o                liv;              //  live (enabled)
      } u3j_harm;

    /* u3j_core: C core driver.
    */
      typedef struct _u3j_core {
        c3_c*             cos_c;              //  control string
        struct _u3j_harm* arm_u;              //  blank-terminated static list
        struct _u3j_core* dev_u;              //  blank-terminated static list
        struct _u3j_core* par_u;              //  dynamic parent pointer
        c3_l              jax_l;              //  dynamic jet index
      } u3j_core;

    /* u3e_dash, u3_Dash, u3D: jet dashboard singleton
    */
      typedef struct _u3e_dash {
        u3j_core* dev_u;                      //  null-terminated static list
        c3_l      len_l;                      //  ray_u filled length
        c3_l      all_l;                      //  ray_u allocated length
        u3j_core* ray_u;                      //  dynamic driver array
      } u3j_dash;
```

Warm and cold state is **per road**. In other words, as we nest roads, we also nest jet state. The jet state in the road is:

```c
      struct {                                //  jet dashboard
        u3p(u3h_root) har_p;                  //  warm state
        u3_noun       das;                    //  cold state
      } jed;
```

In case you understand Hoon, `das` (cold state) is a `+dash`, and `har_p` (warm state) is a map from battery to `+calx`:

```hoon
    ++  bane  ,@tas                                 ::  battery name
    ++  bash  ,@uvH                                 ::  label hash
    ++  bosh  ,@uvH                                 ::  local battery hash
    ++  batt  ,*                                    ::  battery
    ++  calf                                        ::
      $:  jax=,@ud                                  ::  hot core index
          hap=(map ,@ud ,@ud)                       ::  axis/hot arm index
          lab=path                                  ::  label as path
          jit=*                                     ::  arbitrary data
      ==                                            ::
    ++  calx  (trel calf (pair bash cope) club)     ::  cached by battery
    ++  clog  (pair cope (map batt club))           ::  identity record
    ++  club  (pair corp (map term nock))           ::  battery pattern
    ++  cope  (trel bane axis (each bash noun))     ::  core pattern
    ++  core  ,*                                    ::  core
    ++  corp  (each core batt)                      ::  parent or static
    ++  dash  (map bash clog)                       ::  jet system
```

The driver index `jax` in a `+calx` is an index into `ray_u` in the dashboard - ie, a pointer into hot state. This is why the warm state has to be reset when we reload the pier in a new process.

Why is jet state nested? Nock of course is a functional system, so as we compute we don't explicitly create state. Jet state is an exception to this principle (which works only because it can't be semantically detected from Nock/Hoon) - but it can't violate the fundamental rules of the allocation system.

For instance, when we're on an inner road, we can't allocate on an outer road, or point from an outer road to an inner. So if we learn something - like a mapping from battery to jet - in the inner road, we have to keep it in the inner road.

Mitigating this problem, when we leave an inner road (with `u3m_love()`), we call `u3j_reap()` to promote jet information in the dying road. Reaping promotes anything we've learned about any battery that either (a) already existed in the outer road, or (b) is being saved to the outer road.

### u3j: jet binding <a href="#u3j-jet-binding" id="u3j-jet-binding"></a>

Jet binding starts with a `%fast` hint. (In Hoon, this is produced by the runes `~%`, for the general case, or `~/` for simple functions.) To bind a jet, execute a formula of the form:

```
[10 [%fast clue-formula] core-formula]
```

`core-formula` assembles the core to be jet-propelled. `clue-formula` produces the hint information, or `+clue` above, which we want to annotate it with.

A clue is a triple of name, parent, and hooks:

```hoon
++  clue  (trel chum nock (list (pair term nock)))
```

The name, or `+chum`, has a bunch of historical structure which we don't need (cleaning these things up is tricky), but just gets flattened into a term.

The parent axis is a nock formula, but always reduces to a simple axis, which is the address of this core's **parent**. Consider again an ordinary gate

```
[formula [sample context]]
```

Typically the `context` is itself a library core, which itself has a jet binding. If so, the parent axis of this gate is `7`.

If the parent is already bound - and the parent **must** be already bound, in this road or a road containing it - we can hook this core bottom-up into a tree hierarchy. Normally the child core is produced by an arm of the parent core, so this is not a problem - we wouldn't have the child if we hadn't already made the parent.

The clue also contains a list of **hooks**, named nock formulas on the core. Usually these are arms, but they need not be. The point is that we often want to call a core from C, in a situation where we have no type or other source information. A common case of this is a complex system in which we're mixing functions which are jet-propelled with functions that aren't.

In any case, all the information in the `%fast` hint goes to `u3j_mine()`, which registers the battery in cold state (`das` in `jed` in `u3R`), then warm state (`har_p` in `jed`).

It's essential to understand that the `%fast` hint has to be, well, fast - because we apply it whenever we build a core. For instance, if the core is a Hoon gate - a function - we will call `u3j_mine` every time the function is called.

### u3j: the cold jet dashboard <a href="#u3j-the-cold-jet-dashboard" id="u3j-the-cold-jet-dashboard"></a>

For even more fun, the jet tree is not actually a tree of batteries. It's a tree of battery **labels**, where a label is an \[axis term] path from the root of the tree. (At the root, if the core pattern is always followed properly, is a core whose payload is an atomic constant, conventionally the Hoon version.)

Under each of these labels, it's normal to have an arbitrary number of different Nock batteries (not just multiple copies of the same noun, a situation we **do** strive to avoid). For instance, one might be compiled with debugging hints, one not.

We might even have changed the semantics of the battery without changing the label - so long as those semantics don't invalidate any attached driver.

For instance, it's normal to have two equivalent Nock batteries at the same time in one pier: one battery compiled with debugging hints, one not.

Rather, the jet tree is a semantic hierarchy. The root of the hierarchy is a constant, by convention the Hoon kernel version because any normal jet-propelled core has, at the bottom of its onion of libraries, the standard kernel. Thus if the core is

```
[foo-battery [bar-battery [moo-battery 164]]]
```

we can reverse the nesting to construct a hierarchical core path. The static core

```
164/moo/bar/foo
```

extends the static core `164/moo/bar` by wrapping the `foo` battery (ie, in Hoon, `|%`) around it. With the core above, you can compute `foo` stuff, `bar` stuff, and `moo` stuff. Rocket science, not.

Not all cores are static, of course - they may contain live data, like the sample in a gate (ie, argument to a function). Once again, it's important to remember that we track jet bindings not by the core, which may not be static, but by the battery, which is always static.

(And if you're wondering how we can use a deep noun like a Nock formula or battery as a key in a key-value table, remember `mug_w`, the lazily computed short hash, in all boxed nouns.)

In any case, `das`, the dashboard, is a map from `bash` to jet location record (`+clog`). A `clog` in turn contains two kinds of information: the `+cope`, or per-location noun; and a map of batteries to a per-battery `+club`.

The `cope` is a triple of `+bane` (battery name, right now just a `term`); `+axis`, the axis, within **this** core, of the parent; and `(each bash noun)`, which is either `[0 bash]` if the parent is another core, or `[1 noun]`, for the constant noun (like `164`) if there is no parent core.

A `bash` is just the noun hash (`+sham`) of a `cope`, which uniquely expresses the battery's hierarchical location without depending on the actual formulas.

The `club` contains a `+corp`, which we use to actually validate the core. Obviously jet execution has to be perfectly compatible with Nock. We search on the battery, but getting the battery right is not enough - a typical battery is dependent on its context. For example, your jet-propelled library function is very likely to call `+dec` or other advanced kernel technology. If you've replaced the kernel in your context with something else, we need to detect this and not run the jet.

There are two cases for a jet-propelled core - either the entire core is a static constant, or it isn't. Hence the definition of `corp`:

```hoon
++  corp  (each core batt)                ::  parent or static
```

Ie, a `corp` is `[0 core]` or `[1 batt]`. If it's static - meaning that the jet only works with one specific core, ie, the parent axis of each location in the hierarchy is `3` - we can validate with a single comparison. Otherwise, we have to recurse upward by checking the parent.

Note that there is at present no way to force a jet to depend on static **data**.

### u3j: the warm jet dashboard <a href="#u3j-the-warm-jet-dashboard" id="u3j-the-warm-jet-dashboard"></a>

We don't use the cold state to match jets as we call them. We use the cold state to register jets as we find them, and also to rebuild the warm state after the hot state is reset.

What we actually use at runtime is the warm state, `jed->har_p`, which is a `u3h` (built-in hashtable), allocated on the loom, from battery to `+calx`.

A `calx` is a triple of a `+calf`, a `[bash cope]` cell, and a `club`. The latter two are all straight from cold state.

The `calf` contains warm data dependent on hot state. It's a quadruple: of `jax`, the hot driver index (in `ray_u` in `u3j_dash`); `hap`, a table from arm axis (ie, the axis of each formula within the battery) to driver arm index (into `arm_u` in `u3j_core`); `lab`, the complete label path; and `jit`, any other dynamic data that may speed up execution.

We construct `hap`, when we create the calx, by iterating through the arms registered in the `u3j_core`. Note the way a `u3j_harm` declares itself, with the string `fcs_c` which can contain either an axis or a name. Most jetted cores are of course gates, which have one formula at one axis within the core: `fcs_c` is `".3"`.

But we do often have fast cores with more complex arm structure, and it would be sad to have to manage their axes by hand. To use an `fcs_c` with a named arm, it's sufficient to make sure the name is bound to a formula `[0 axis]` in the hook table.

`jit`, as its name suggests, is a stub where any sort of optimization data computed on battery registration might go. To use it, fill in the `_cj_jit()` function.

### u3j: the hot dashboard <a href="#u3j-the-hot-dashboard" id="u3j-the-hot-dashboard"></a>

Now it should be easy to see how we actually invoke jets. Every time we run a nock `9` instruction (pretty often, obviously), we have a core and an axis. We pass these to `u3j_kick()`, which will try to execute them.

Because nouns with a reference count of 1 are precious, `u3j_kick()` has a tricky reference control definition. It reserves the right to return `u3_none` in the case where there is no driver, or the driver does not apply for this case; in this case, it retains argument `cor`. If it succeeds, though, it transfers `cor`.

`u3j_kick()` searches for the battery (always the head of the core, of course) in the hot dashboard. If the battery is registered, it searches for the axis in `hap` in the `calx`. If it exists, the core matches a driver and the driver jets this arm. If not, we return `u3_none`.

Otherwise, we call `fun_f` in our `u3j_harm`. This obeys the same protocol as `u3j_kick()`; it can refuse to function by returning `u3_none`, or consume the noun.

Besides the actual function pointer `fun_f`, we have some flags in the `u3j_harm` which tell us how to call the arm function.

If `ice` is yes (`&`, `0`), the jet is known to be perfect and we can just trust the product of `fun_f`. Otherwise, we need to run **both** the Nock arm and `fun_f`, and compare their results.

(Note that while executing the C side of this test, we have to set `ice` to yes; on the Nock side, we have to set `liv` to no. Otherwise, many non-exponential functions become exponential. When auto-testing jets in this way, the principle is that the test is on the outermost layer of recursion.)

(Note also that anyone who multi-threads this execution environment has a slight locking problem with these flags if arm testing is multi-threaded.)

If `tot` is yes, (`&`, `0`), the arm function is **total** and has to return properly (though it can still return **u3\_none**). Otherwise, it is **partial** and can `u3_cm_bail()` out with c3\_\_punt. This feature has a cost: the jet runs in a subroad.

Finally, if `liv` is no (`|`, 1), the jet is off and doesn't run.

It should be easy to see how the tree of cores gets declared - precisely, in `j/dash.c`. We declare the hierarchy as a tree of `u3j_core` structures, each of which comes with a static list of arms `arm_u` and sub-cores `dev_u`.

In `u3j_boot()`, we traverse the hierarchy, fill in parent pointers `par_u`, and enumerate all `u3j_core` structures into a single flat array `u3j_dash.ray_u`. Our hot state then appears ready for action.

### u3j: jet functions <a href="#u3j-jet-functions" id="u3j-jet-functions"></a>

At present, all drivers are compiled statically into `u3`. This is not a long-term permanent solution or anything. However, it will always be the case with a certain amount of core functionality.

For instance, there are some jet functions that we need to call as part of loading the Arvo kernel - like `+cue` to unpack a noun from an atom. And obviously it makes sense, when jets are significant enough to compile into `u3`, to export their symbols in headers and the linker.

There are three interface prefixes for standard jet functions: `u3k`, `u3q`, and `u3w`. All jets have `u3w` interfaces; most have `u3q`; some have `u3k`. Of course the actual logic is shared.

`u3w` interfaces use the same protocol as `fun_f` above: the caller passes the entire core, which is retained if the function returns `u3_none`, transferred otherwise. Why? Again, use counts of 1 are special and precious for performance hackers.

`u3q` interfaces break the core into C arguments, **retain** noun arguments, and **transfer** noun returns. `u3k` interfaces are the same, except with more use of `u3_none` and other simple C variations on the Hoon original, but **transfer** both arguments and returns. Generally, `u3k` are most convenient for new code.

Following `u3k/q/w` is `[a-f]`, corresponding to the 6 logical tiers of the kernel, or `g` for user-level jets. Another letter is added for functions within subcores. The filename, under `j/`, follows the tier and the function name.

For instance, `+add` is `u3wa_add(cor)`, `u3qa_add(a, b)`, or `u3ka_add(a, b)`, in `j/a/add.c`. `+get` in `+by` is `u3wdb_get(cor)`, `u3kdb_get(a, b)`, etc, in `j/d/by_get.c`.

For historical reasons, all internal jet code in `j/[a-f]` **retains** noun arguments, and **transfers** noun results. Please do not do this in new `g` jets! The new standard protocol is to transfer both arguments and results.

## u3a: allocation functions <a href="#u3a-allocation-functions" id="u3a-allocation-functions"></a>

`u3a` allocates on the current road (u3R). Its internal structures are uninteresting and typical of a naive allocator.

The two most-used `u3a` functions are `u3a_gain()` to add a reference count, and `u3a_lose()` to release one (and free the noun, if the use count is zero). For convenience, `u3a_gain()` returns its argument. The pair are generally abbreviated with the macros `u3k()` and `u3z()` respectively.

Normally we create nouns through `u3i` functions, and don't call the `u3a` allocators directly. But if you do:

One, there are **two** sets of allocators: the word-aligned allocators and the fully-aligned (ie, malloc compatible) allocators. For instance, on a typical OS X setup, malloc produces 16-byte aligned results - needed for some SSE instructions.

These allocators are **not compatible**. For 32-bit alignment as used in nouns, call

```c
    /* u3a_walloc(): allocate storage measured in words.
    */
      void*
      u3a_walloc(c3_w len_w);

    /* u3a_wfree(): free storage.
    */
      void
      u3a_wfree(void* lag_v);

    /* u3a_wealloc(): word realloc.
    */
      void*
      u3a_wealloc(void* lag_v, c3_w len_w);
```

For full alignment, call:

```c
    /* u3a_malloc(): aligned storage measured in bytes.
    */
      void*
      u3a_malloc(size_t len_i);

    /* u3a_realloc(): aligned realloc in bytes.
    */
      void*
      u3a_realloc(void* lag_v, size_t len_i);

    /* u3a_realloc2(): gmp-shaped realloc.
    */
      void*
      u3a_realloc2(void* lag_v, size_t old_i, size_t new_i);

    /* u3a_free(): free for aligned malloc.
    */
      void
      u3a_free(void* tox_v);

    /* u3a_free2(): gmp-shaped free.
    */
      void
      u3a_free2(void* tox_v, size_t siz_i);
```

There are also a set of special-purpose allocators for building atoms. When building atoms, please remember that it's incorrect to have a high 0 word - the word length in the atom structure must be strictly correct.

Of course, we don't always know how large our atom will be. Therefore, the standard way of building large atoms is to allocate a block of raw space with `u3a_slab()`, then chop off the end with `u3a_malt()` (which does the measuring itself) or `u3a_mint()` in case you've measured it yourself.

Once again, **do not call `malloc()`** (or C++ `new`) within any code that may be run within a jet. This will cause rare sporadic corruption when we interrupt execution within a `malloc()`. We'd just override the symbol, but `libuv` uses `malloc()` across threads within its own synchronization primitives - for this to work with `u3a_malloc()`, we'd have to introduce our own locks on the surface-level road (which might be a viable solution).

## u3n: nock execution <a href="#u3n-nock-execution" id="u3n-nock-execution"></a>

The `u3n` routines execute Nock itself. On the inside, they have a surprising resemblance to the spec proper (the only interesting detail is how we handle tail-call elimination) and are, as one would expect, quite slow. (There is no such thing as a fast tree interpreter.)

There is only one Nock, but there are lots of ways to call it. (Remember that all `u3n` functions **transfer** C arguments and returns.)

The simplest interpreter, `u3n_nock_on(u3_noun bus, u3_noun fol)` invokes Nock on `bus` (the subject) and `fol` (the formula). (Why is it`[subject formula]`, not `[formula subject]`? The same reason `0` is true and `1` is false.)

A close relative is `u3n_slam_on(u3_noun gat, u3_noun sam)`, which slams a **gate** (`gat`) on a sample (`sam`). (In a normal programming language which didn't talk funny and was retarded, `u3n_slam_on()` would call a function on an argument.) We could write it most simply as:

```c
    u3_noun
    u3n_slam_on(u3_noun gat, u3_noun sam)
    {
      u3_noun pro = u3n_nock_on
                      (u3nc(u3k(u3h(gat)),
                            u3nc(sam, u3k(u3t(u3t(gat))))),
                       u3k(u3h(gat)));
      u3z(gat);
      return pro;
    }
```

Simpler is `u3n_kick_on(u3_noun gat)`, which slams a gate (or, more generally, a **trap** - because sample structure is not even needed here) without changing its sample:

```c
    u3_noun
    u3n_kick_on(u3_noun gat, u3_noun sam)
    {
      return u3n_nock_on(gat, u3k(u3h(gat)));
    }
```

The `_on` functions in `u3n` are all defined as pure Nock. But actually, even though we say we don't extend Nock, we do. But we don't. But we do.

Note that `u3` has a well-developed error handling system - `u3m_bail()` to throw an exception, `u3m_soft_*` to catch one. But Nock has no exception model at all. That's okay - all it means if that if an `_on` function bails, the exception is an exception in the caller.

However, `u3`'s exception handling happens to match a convenient virtual super-Nock in `hoon.hoon`, the infamous `+mock`. Of course, Nock is slow, and `mock` is Nock in Nock, so it is (logically) super-slow. Then again, so is decrement.

With the power of `u3`, we nest arbitrary layers of `mock` without any particular performance cost. Moreover, we simply treat Nock proper as a special case of `mock`. (More precisely, the internal VM loop is `+mink` and the error compiler is `+mook`. But we call the whole sandbox system `mock`.)

The nice thing about `mock` functions is that (by executing within `u3m_soft_run()`, which as you may recall uses a nested road) they provide both exceptions and the namespace operator - `.^` in Hoon, which becomes operator `11` in `mock`.

`11` requires a namespace function, or `fly`, which produces a `+unit` - `~` (`0`) for no binding, or `[0 value]`. The sample to a `fly` is a `+path`, just a list of text `span`.

`mock` functions produce a `+toon`. Fully elaborated:

```hoon
    ++  noun  ,*                                      ::  any noun
    ++  path  (list ,@ta)                             ::  namespace path
    ++  span  ,@ta                                    ::  text-atom (ASCII)
    ++  toon  $%  [%0 p=noun]                         ::  success
                  [%1 p=(list path)]                  ::  blocking paths
                  [%2 p=(list tank)]                  ::  stack trace
              ==                                      ::
    ++  tank                                          ::  printable
              $%  [%leaf p=tape]                      ::  flat text
                  $:  %palm                           ::  backstep list
                      p=[p=tape q=tape r=tape s=tape] ::  mid cap open close
                      q=(list tank)                   ::  contents
                  ==                                  ::
                  $:  %rose                           ::  straight list
                      p=[p=tape q=tape r=tape]        ::  mid open close
                      q=(list tank)                   ::  contents
                  ==                                  ::
              ==
```

(Note that `tank` is overdesigned and due for replacement.)

What does a `toon` mean? Either your computation succeeded (`[0 noun]`, or could not finish because it blocked on one or more global paths (`[1 (list path)]`), or it exited with a stack trace (`[2 (list tank)]`).

Note that of all the `u3` exceptions, only `%exit` is produced deterministically by the Nock definition. Therefore, only `%exit` produces a `2` result. Any other argument to `u3m_bail()` will unwind the virtualization stack all the way to the top - or to be more exact, to `u3m_soft_top()`.

In any case, the simplest `mock` functions are `u3n_nock_un()` and `u3n_slam_un()`. These provide exception control without any namespace change, as you can see by the code:

```c
    /* u3n_nock_un(): produce .*(bus fol), as ++toon.
    */
    u3_noun
    u3n_nock_un(u3_noun bus, u3_noun fol)
    {
      u3_noun fly = u3nt(u3nt(11, 0, 6), 0, 0);  //  |=(a=* .^(a))

      return u3n_nock_in(fly, bus, fol);
    }

    /* u3n_slam_un(): produce (gat sam), as ++toon.
    */
    u3_noun
    u3n_slam_un(u3_noun gat, u3_noun sam)
    {
      u3_noun fly = u3nt(u3nt(11, 0, 6), 0, 0);  //  |=(a=* .^(a))

      return u3n_slam_in(fly, gat, sam);
    }
```

The `fly` is added as the first argument to `u3n_nock_in()` and `u3n_slam_in()`. Of course, logically, `fly` executes in the caller's exception layer. (Maintaining this illusion is slightly nontrivial.) Finally, `u3n_nock_an()` is a sandbox with a null namespace.

## u3e: persistence <a href="#u3e-persistence" id="u3e-persistence"></a>

The only `u3e` function you should need to call is `u3e_save()`, which saves the loom. As it can be restored on any platform, please make sure you don't have any state in the loom that is bound to your process or architecture - except for exceptions like the warm jet state, which is actively purged on reboot.

## u3r: reading nouns (weak) <a href="#u3r-reading-nouns-weak" id="u3r-reading-nouns-weak"></a>

As befits accessors they don't make anything, `u3r` noun reading functions always retain their arguments and their returns. They never bail; rather, when they don't work, they return a `u3_weak` result.

Most of these functions are straightforward and do only what their comments say. A few are interesting enough to discuss.

`u3r_at()` is the familiar tree fragment function, `/` from the Nock spec. For taking complex nouns apart, `u3r_mean()` is a relatively funky way of deconstructing nouns with a varargs list of `axis`, `u3_noun *`. For cells, triples, etc, decompose with `u3r_cell()`, `u3r_trel()`, etc. For the tagged equivalents, use `u3r_pq()` and friends.

`u3r_sing(u3_noun a, u3_noun b)` (true if `a` and `b` are a **single** noun) are interesting because it uses mugs to help it out. Clearly, different nouns may have the same mug, but the same nouns cannot have a different mug. It's important to understand the performance characteristics of `u3r_sing()`: the worst possible case is a comparison of duplicate nouns, which have the same value but were created separately. In this case, the tree is traversed

`u3r_sung()` is a deeply funky and frightening version of `u3r_sing()` that unifies pointers to the duplicate nouns it finds, freeing the second copy. Obviously, do not use `u3r_sung()` when you have live, but not reference counted, noun references from C - if they match a noun with a refcount of 1 that gets freed, bad things happen.

It's important to remember that `u3r_mug()`, which produces a 31-bit, nonzero insecure hash, uses the `mug_w` slot in any boxed noun as a lazy cache. There are a number of variants of `u3r_mug()` that can get you out of building unneeded nouns.

## u3x: reading nouns (bail) <a href="#u3x-reading-nouns-bail" id="u3x-reading-nouns-bail"></a>

`u3x` functions are like `u3r` functions, but instead of returning `u3_none` when (for instance) we try to take the head of an atom, they bail with `%exit`. In other words, they do what the same operation would do in Nock.

## u3h: hash tables. <a href="#u3h-hash-tables" id="u3h-hash-tables"></a>

We can of course use the Hoon `map` structure as an associative array. This is a balanced treap and reasonably fast. However, it's considerably inferior to a custom structure like an HAMT (hash array-mapped trie). We use `u3_post` to allocate HAMT structures on the loom.

(Our HAMT implements the classic Bagwell algorithm which depends on the `gcc` standard directive `__builtin_popcount()`. On a CPU which doesn't support popcount or an equivalent instruction, some other design would probably be preferable.)

There's no particular rocket science in the API. `u3h_new()` creates a hashtable; `u3h_free()` destroys one; `u3h_put()` inserts, `u3h_get()` retrieves. You can transform values in a hashtable with `u3h_walk()`.

The only funky function is `u3h_gut()`, which unifies keys with `u3r_sung()`. As with all cases of `u3r_sung()`, this must be used with extreme caution.

## u3z: memoization <a href="#u3z-memoization" id="u3z-memoization"></a>

Connected to the `~+` rune in Hoon, via the Nock `%memo` hint, the memoization facility is a general-purpose cache.

(It's also used for partial memoization - a feature that'll probably be removed, in which conservative worklist algorithms (which would otherwise be exponential) memoize everything in the subject **except** the worklist. This is used heavily in the Hoon compiler jets (j/f/\*.c). Unfortunately, it's probably not possible to make this work perfectly in that it can't be abused to violate Nock, so we'll probably remove it at a later date, instead making `+ut` keep its own monadic cache.)

Each `u3z` function comes with a `c3_m` mote which disambiguates the function mapping key to value. For Nock itself, use 0. For extra speed, small tuples are split out in C; thus, find with

```c
    u3_weak u3z_find(c3_m, u3_noun);
    u3_weak u3z_find_2(c3_m, u3_noun, u3_noun);
    u3_weak u3z_find_3(c3_m, u3_noun, u3_noun, u3_noun);
    u3_weak u3z_find_4(c3_m, u3_noun, u3_noun, u3_noun, u3_noun);
```

and save with

```c
    u3_noun u3z_save(c3_m, u3_noun, u3_noun);
    u3_noun u3z_save_2(c3_m, u3_noun, u3_noun, u3_noun);
    u3_noun u3z_save_3(c3_m, u3_noun, u3_noun, u3_noun, u3_noun);
    u3_noun u3z_save_4(c3_m, u3_noun, u3_noun, u3_noun, u3_noun, u3_noun);
```

where the value is the last argument. To eliminate duplicate nouns, there is also

```c
    u3_noun
    u3z_uniq(u3_noun);
```

`u3z` functions retain keys and transfer values.

The `u3z` cache, built on `u3h` hashes, is part of the current road, and goes away when it goes away. (In future, we may wish to promote keys/values which outlive the road, as we do with jet state.) There is no cache reclamation at present, so be careful.

## u3t: tracing and profiling. <a href="#u3t-tracing-and-profiling" id="u3t-tracing-and-profiling"></a>

TBD.

## u3v: the Arvo kernel <a href="#u3v-the-arvo-kernel" id="u3v-the-arvo-kernel"></a>

An Arvo kernel - or at least, a core that compiles with the Arvo interface - is part of the global `u3` state. What is an Arvo core? Slightly pseudocoded:

```hoon
    ++  arvo
      |%
      ++  come  |/  {yen/@ ova/(list ovum) nyf/pone}  ::  11
                ^-  {(list ovum) _+>}
                !!
      ++  keep  |/  {now/@da hap/path}                ::  4
                ^-  (unit ,@da)
                !!
      ++  load  |/  {yen/@ ova/(list ovum) nyf/pane}  ::  86
                ^-  {(list ovum) _+>}
                !!
      ++  peek  |/  {now/@da path}                    ::  87
                ^-  (unit)
                !!
      ++  poke  |/  {now/@da ovo/ovum}                ::  42
                ^-  {(list ovum) _+>}
                !!
      ++  wish  |/  txt/@ta                           ::  20
                ^-  *
                !!
      --
    ++  card  {p/@tas q/*}                           ::  typeless card
    ++  ovum  {p/wire q/card}                        ::  Arvo event
    ++  wire  path                                    ::  event cause
```

This is the Arvo ABI in a very real sense. Arvo is a core with these six arms. To use these arms, we hardcode the axis of the formula (`11`, `4`, `86`, etc) into the C code that calls Arvo, because otherwise we'd need type metadata - which we can get, by calling Arvo.

It's important to understand the Arvo event/action structure, or `+ovum`. An `ovum` is a `card`, which is any `[term noun]` cell, and a `+wire`, a `path` which indicates the location of the event. At the Unix level, the `wire` corresponds to a system module or context. For input events, this is the module that caused the event; for output actions, it's the module that performs the action.

`+poke` sends Arvo an event `ovum`, producing a cell of action ova and a new Arvo core.

`+peek` dereferences the Arvo namespace. It takes a date and a key, and produces `~` (`0`) or `[~ value]`.

`+keep` asks Arvo the next time it wants to be woken up, for the given `wire`. (This input will probably be eliminated in favor of a single global timer.)

`+wish` compiles a string of Hoon source. While just a convenience, it's a very convenient convenience.

`+come` and `+load` are used by Arvo to reset itself (more precisely, to shift the Arvo state from an old kernel to a new one); there is no need to call them from C.

Now that we understand the Arvo kernel interface, let's look at the `u3v` API. As usual, all the functions in `u3v` are commented, but unfortunately it's hard to describe this API as clean at present. The problem is that `u3v` remains design coupled to the old `vere` event handling code written for `u2`. But let's describe the functions you should be calling, assuming you're not writing the next event system. There are only two.

`u3v_wish(str_c)` wraps the `+wish` functionality in a cache (which is read-only unless you're on the surface road).

`u3v_do()` uses `wish` to provide a convenient interface for calling Hoon kernel functions by name. Even more conveniently, we tend to call `u3v_do()` with these convenient aliases:

```c
    #define  u3do(txt_c, arg)         u3v_do(txt_c, arg)
    #define  u3dc(txt_c, a, b)        u3v_do(txt_c, u3nc(a, b))
    #define  u3dt(txt_c, a, b, c)     u3v_do(txt_c, u3nt(a, b, c))
    #define  u3dq(txt_c, a, b, c, d)  u3v_do(txt_c, u3nt(a, b, c, d))
```

## Code Mnemonics <a href="#code-mnemonics" id="code-mnemonics"></a>

| Shorthand | Expansion              | Meaning              |
| --------- | ---------------------- | -------------------- |
| `u3A`     | `(&(u3v_Home->arv_u))` | Arvo                 |
| `u3C`     | `u3o_Config`           | command line options |
| `u3D`     | `u3j_Dash`             | jet dashboard        |
| `u3H`     | `u3v_Home`             | home road            |
| `u3P`     | `u3e_Pool`             | snapshotting system  |
| `u3R`     | `u3a_Road`             | current road         |
| `u3T`     | `u3t_Trace`            | tracing profiler     |


# C in Urbit

C conventions and style used in Urbit. Integer types, variable naming, loobeans. The c3 layer under u3.

Under `u3` is the simple `c3` layer, which is just how we write C in Urbit.

When writing C in u3, please of course follow the conventions of the code around you as regards indentation, etc. It's especially important that every function have a header comment, even if it says nothing interesting.

But some of our idiosyncrasies go beyond convention. Yes, we've done awful things to C. Here's what we did and why we did.

## c3: integer types <a href="#c3-integer-types" id="c3-integer-types"></a>

First, it's generally acknowledged that underspecified integer types are C's worst disaster. C99 fixed this, but the `stdint` types are wordy and annoying. We've replaced them with:

```c
    /* Good integers.
    */
      typedef uint64_t c3_d;  // double-word
      typedef int64_t c3_ds;  // signed double-word
      typedef uint32_t c3_w;  // word
      typedef int32_t c3_ws;  // signed word
      typedef uint16_t c3_s;  // short
      typedef int16_t c3_ss;  // signed short
      typedef uint8_t c3_y;   // byte
      typedef int8_t c3_ys;   // signed byte
      typedef uint8_t c3_b;   // bit

      typedef uint8_t c3_t;   // boolean
      typedef uint8_t c3_o;   // loobean
      typedef uint8_t c3_g;   // 5-bit atom for a 32-bit log.
      typedef uint32_t c3_l;  // little; 31-bit unsigned integer
      typedef uint32_t c3_m;  // mote; also c3_l; LSB first a-z 4-char string.

    /* Bad integers.
    */
      typedef char      c3_c; // does not match int8_t or uint8_t
      typedef int       c3_i; // int - really bad
      typedef uintptr_t c3_p; // pointer-length uint - really really bad
      typedef intptr_t c3_ps; // pointer-length int - really really bad
```

Some of these need explanation. A loobean is a Nock boolean - Nock, for mysterious reasons, uses 0 as true (always say "yes") and 1 as false (always say "no").

Nock and/or Hoon cannot tell the difference between a short atom and a long one, but at the `u3` level every atom under `2^31` is direct. The `c3_l` type is useful to annotate this. A `c3_m` is a **mote** - a string of up to 4 characters in a `c3_l`, least significant byte first. A `c3_g` should be a 5-bit atom. Of course, C cannot enforce these constraints, only document them.

Use the "bad" - ie, poorly specified - integer types only when interfacing with external code that expects them.

An enormous number of motes are defined in `i/c/motes.h`. There is no reason to delete motes that aren't being used, or even to modularize the definitions. Keep them alphabetical, though.

## c3: variables and variable naming <a href="#c3-variables-and-variable-naming" id="c3-variables-and-variable-naming"></a>

The C3 style uses Hoon style TLV variable names, with a quasi Hungarian syntax. This is weird, but works really well, as long as what you're doing isn't hideously complicated. (Then it works badly, but we shouldn't need anything hideous in u3.)

A TLV variable name is a random pronounceable three-letter string, sometimes with some vague relationship to its meaning, but usually not. Usually CVC (consonant-vowel-consonant) is a good choice.

You should use TLVs much the way math people use Greek letters. The same concept should in general get the same name across different contexts. When you're working in a given area, you'll tend to remember the binding from TLV to concept by sheer power of associative memory. When you come back to it, it's not that hard to relearn. And of course, when in doubt, comment it.

Variables take pseudo-Hungarian suffixes, matching in general the suffix of the integer type:

```c
c3_w wor_w;     //  32-bit word
```

Unlike in standard Hungarian, there is no change for pointer variables. C structure variables take a `_u` suffix.

## c3: loobeans <a href="#c3-loobeans" id="c3-loobeans"></a>

The code (from `defs.h`) tells the story:

```c
    #     define c3y      0
    #     define c3n      1

    #     define _(x)        (c3y == (x))
    #     define __(x)       ((x) ? c3y : c3n)
    #     define c3a(x, y)   __(_(x) && _(y))
    #     define c3o(x, y)   __(_(x) || _(y))
```

In short, use `_()` to turn a loobean into a boolean, `__` to go the other way. Use `!` as usual, `c3y` for yes and `c3n` for no, `c3a` for and and `c3o` for or.


# Cryptography

Cryptographic primitives used by Arvo. Ed25519 signatures, AES-SIV encryption. All jetted for performance.

All cryptographic primitives utilized by Arvo are [jetted](/nock/jetting). This is done for performance-related reasons in other parts of the system, but for cryptography this is also extremely important because it allows us to utilize standard reference implementations for the primitives written in C.

All jets related to encryption may be found in `pkg/urbit/jets/e/`.

In this section we review what specific implementations are utilized. At the moment, only libraries directly related to Ames are documented here, though we note that there are jets for other cryptographic functions such as the [SHA Hash Family](/hoon/stdlib/3d) as well.

## Ed25519 <a href="#ed" id="ed"></a>

Urbit implements [Ed25519](http://ed25519.cr.yp.to/) based on the SUPERCOP "ref10" implementation. Additionally there is key exchanging and scalar addition included to further aid building a PKI using Ed25519. All code is licensed under the permissive zlib license.

All code is pure ANSI C without any dependencies, except for the random seed generation which uses standard OS cryptography APIs (CryptGenRandom on Windows, `/dev/urandom` on nix).

## AES-SIV <a href="#aes" id="aes"></a>

The library we utilize for AES-SIV is an [RFC5297](https://tools.ietf.org/html/rfc5297)-compliant C implementation of AES-SIV written by Daniel Franke on behalf of [Akamai Technologies](https://www.akamai.com). It is published under the [Apache License (v2.0)](https://www.apache.org/licenses/LICENSE-2.0). It uses OpenSSL for the underlying [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) and [CMAC](https://en.wikipedia.org/wiki/One-key_MAC) implementations and follows a similar interface style.

While the jets are found in `pkg/urbit/jets/e`, the statically-linked package is found at `pkg/urcrypt/`.


# Land of Nouns

Comprehensive guide to u3 noun processing in C. Memory model, reference counting, road system architecture.

The division between `c3` and `u3` is that you could theoretically imagine using `c3` as just a generic C environment. Anything to do with nouns is in `u3`.

## u3: a map of the system <a href="#u3-a-map-of-the-system" id="u3-a-map-of-the-system"></a>

These are the symbols you'll need to know about to program in `u3`. All files listed below are found in the [`pkg/noun`](https://github.com/urbit/vere/tree/develop/pkg/noun) directory. Symbols follow this pattern:

```
prefix    purpose                      .h            .c
-------------------------------------------------------------------
u3a_      allocation                   allocate.h    allocate.c
u3e_      persistence                  events.h      events.c
u3h_      hashtables                   hashtable.h   hashtable.c
u3i_      noun construction            imprison.h    imprison.c
u3j_      jet control                  jets.h        jets.c
u3l_      logging                      log.h         log.c
u3m_      system management            manage.h      manage.c
u3n_      nock computation             nock.h        nock.c
u3o_      command-line options         options.h     options.c
u3r_      noun access (error returns)  retrieve.h    retrieve.c
u3s_      noun serialization           serial.h      serial.c
u3t_      profiling                    trace.h       trace.c
u3u_      urth (memory management)     urth.h        urth.c
u3v_      arvo                         vortex.h      vortex.c
u3x_      noun access (error crashes)  xtract.h      xtract.c
u3z_      memoization                  zave.h        zave.c
u3k[a-g]  jets (transfer, C args)      jets/k.h      jets/[a-g]/*.c
u3q[a-g]  jets (retain, C args)        jets/q.h      jets/[a-g]/*.c
u3w[a-g]  jets (retain, nock core)     jets/w.h      jets/[a-g]/*.c
```

Additionally, various noun type definition are found in `pkg/noun/types.h`.

## u3: noun internals <a href="#u3-noun-internals" id="u3-noun-internals"></a>

A noun is a `u3_noun` - currently defined as a 32-bit `c3_w`. (This is zero-indexed so bit `31` is the high bit.)

If your `u3_noun` is less than `(1 << 31)`, it's a direct atom. Every unsigned integer between `0` and `0x7fffffff` inclusive is its own noun.

If bit `31` is set in a `u3_noun` and bit `30` is `1` the noun is an indirect cell. If bit `31` is set and bit `30` is `0` the noun is an indirect atom. Bits `29` through `0` are a word pointer into the loom - see below. The structures are:

```c
typedef struct {
  c3_w mug_w;
  c3_w len_w;
  c3_w buf_w[0];    //  actually [len_w]
} u3a_atom;

typedef struct {
  c3_w    mug_w;
  u3_noun hed;
  u3_noun tel;
} u3a_cell;
```

The only thing that should be mysterious here is `mug_w`, which is a 31-bit lazily computed nonzero short hash ([Murmur3](https://github.com/PeterScott/murmur3)). If `mug_w` is 0, the hash is not yet computed. We also hijack this field for various hacks, such as saving the new address of a noun when copying over.

Also, the value `0xffffffff` is `u3_none`, which is never a valid noun. Use the type `u3_weak` to express that a noun variable may be `u3_none`.

## u3: reference counts <a href="#u3-reference-counts" id="u3-reference-counts"></a>

The only really essential thing you need to know about `u3` is how to handle reference counts. Everything else, you can skip and just get to work.

u3 deals with reference-counted, immutable, acyclic nouns. Unfortunately, we are not Apple and can't build reference counting into your C compiler, so you need to count by hand.

Every allocated noun (or any allocation object, because our allocator is general-purpose) contains a counter which counts the number of references to it - typically variables with type `u3_noun`. When this counter goes to 0, the noun is freed.

To tell `u3` that you've added a reference to a noun, call the function `u3a_gain()` or its shorthand `u3k()`. (For your convenience, this function returns its argument.) To tell `u3` that you've destroyed a reference, call `u3a_lose()` or `u3z()`.

(If you screw up by decrementing the counter too much, `u3` will dump core in horrible ways. If you screw up by incrementing it too much, `u3` will leak memory. To check for memory leaks, set the `bug_o` flag in `u3e_boot()` - eg, run `vere` with `-g`. Memory leaks are difficult to debug - the best way to handle leaks is just to revert to a version that didn't have them, and look over your code again.)

(You can gain or lose a direct atom. It does nothing.)

## u3: reference protocols <a href="#u3-reference-protocols" id="u3-reference-protocols"></a>

**THIS IS THE MOST CRITICAL SECTION IN THE `u3` DOCUMENTATION.**

The key question when calling a C function in a refcounted world is what the function will do to the noun refcounts - and, if the function returns a noun, what it does to the return.

There are two semantic patterns, `transfer` and `retain`. In `transfer` semantics, the caller "gives" a use count to the callee, which "gives back" any return. For instance, if I have

```c
    {
      u3_noun foo = u3i_string("foobar");
      u3_noun bar;

      bar = u3f_futz(foo);
      [...]
      u3z(bar);
    }
```

Suppose `u3f_futz()` has `transfer` semantics. At `[...]`, my code holds one reference to `bar` and zero references to `foo` - which has been freed, unless it's part of `bar`. My code now owns `bar` and gets to work with it until it's done, at which point a `u3z()` is required.

On the other hand, if `u3f_futz()` has `retain` semantics, we need to write

```c
    {
      u3_noun foo = u3i_string("foobar");
      u3_noun bar;

      bar = u3f_futz(foo);
      [...]
      u3z(foo);
    }
```

because calling `u3f_futz()` does not release our ownership of `foo`, which we have to free ourselves.

But if we free `bar`, we are making a great mistake, because our reference to it is not in any way registered in the memory manager (which cannot track references in C variables, of course). It is normal and healthy to have these uncounted C references, but they must be treated with care.

The bottom line is that it's essential for the caller to know the refcount semantics of any function which takes or returns a noun. (In some unusual circumstances, different arguments or returns in one function may be handled differently.)

Broadly speaking, as a design question, retain semantics are more appropriate for functions which inspect or query nouns. For instance, `u3h()` (which takes the head of a noun) retains, so that we can traverse a noun tree without constantly incrementing and decrementing.

Transfer semantics are more appropriate for functions which make nouns, which is obviously what most functions do.

In general, though, in most places it's not worth thinking about what your function does. There is a convention for it, which depends on where it is, not what it does. Follow the convention.

## u3: reference conventions <a href="#u3-reference-conventions" id="u3-reference-conventions"></a>

The `u3` convention is that, unless otherwise specified, **all functions have transfer semantics** - with the exception of the prefixes: `u3r`, `u3x`, `u3z`, `u3q` and `u3w`. Also, within jet directories `a` through `f` (but not `g`), internal functions retain (for historical reasons).

If functions outside this set have retain semantics, they need to be commented, both in the `.h` and `.c` file, with `RETAIN` in all caps. Yes, it's this important.

## u3: system architecture <a href="#u3-system-architecture" id="u3-system-architecture"></a>

If you just want to tinker with some existing code, it might be enough to understand the above. If not, it's probably worth taking the time to look at `u3` as a whole.

`u3` is designed to work as a persistent event processor. Logically, it computes a function of the form

```
    f(event, old state) -> (actions, new state)
```

Obviously almost any computing model - including, but not limited to, Urbit - can be defined in this form. To create the illusion of a computer that never loses state and never fails, we:

* log every event externally before it goes into u3
* keep a single reference to a permanent state noun.
* can abort any event without damaging the permanent state.
* snapshot the permanent state periodically, and/or prune logs.

## u3: the road model <a href="#u3-the-road-model" id="u3-the-road-model"></a>

`u3` uses a memory design which I'm sure someone has invented somewhere before, because it's not very clever, but I've never seen it anywhere in particular.

Every allocation starts with a solid block of memory, which `u3` calls the `loom`. How do we allocate on the loom? You're probably familiar with the Unix heap-stack design, in which the stack grows downward and the heap (malloc arena) grows upward:

```
    0           brk                                          ffff
    |   heap     |                                    stack    |
    |------------#################################+++++++++++++|
    |                                             |            |
    0                                             sp         ffff
```

A road is a normal heap-stack system, except that the heap and stack can point in **either direction**. Therefore, inside a road, we can nest another road in the **opposite direction**.

When the opposite road completes, its heap is left on top of the opposite heap's stack. It's no more than the normal behavior of a stack machine for all subcomputations to push their results on the stack.

The performance tradeoff of "leaping" - reversing directions in the road - is that if the outer computation wants to preserve the results of the inner one, not just use them for temporary purposes, it has to **copy them**.

This is a trivial cost in some cases, a prohibitive cost in others. The upside, of course, is that all garbage accrued in the inner computation is discarded at zero cost.

The goal of the road system is the ability to **layer** memory models. If you are allocating on a road, you have no idea how deep within a nested road system you are - in other words, you have no idea exactly how durable your result may be. But free space is never fragmented within a road.

Roads do not reduce the generality or performance of a memory system, since even the most complex GC system can be nested within a road at no particular loss of performance - a road is just a block of memory.

Each road (`u3a_road` to be exact) uses four pointers: `rut` is the bottom of the arena, `hat` the top of the arena, `mat` the bottom of the stack, `cap` the top of the stack. (Bear in mind that the road "stack" is not actually used as the C function-call stack, though it probably should be.)

A "north" road has the stack high and the heap low:

```
    0           rut   hat                                    ffff
    |            |     |                                       |
    |~~~~~~~~~~~~-------##########################+++++++$~~~~~|
    |                                             |      |     |
    0                                            cap    mat  ffff
```

A "south" road is the other way around:

```
    0           mat   cap                                    ffff
    |            |     |                                       |
    |~~~~~~~~~~~~$++++++##########################--------~~~~~|
    |                                             |      |     |
    0                                            hat    rut  ffff
```

Legend: `-` is durable storage (heap); `+` is temporary storage (stack); `~` is deep storage (immutable); `$` is the allocation frame `#` is free memory.

Pointer restrictions: pointers stored in `+` can point anywhere. Pointers in `-` can only point to `-` or `~`; pointers in `~` only point to `~`.

To "leap" is to create a new inner road in the `###` free space. but in the reverse direction, so that when the inner road "falls" (terminates), its durable storage is left on the temporary storage of the outer road.

`u3` keeps a global variable, `u3_Road` or its alias `u3R`, which points to the current road. (If we ever run threads in inner roads - see below - this will become a thread-local variable.) Relative to `u3R`, `+` memory is called `junior` memory; `-` memory is `normal` memory; `~` is `senior` memory.

## u3: explaining the road model <a href="#u3-explaining-the-road-model" id="u3-explaining-the-road-model"></a>

But... why?

We're now ready to understand why the road system works so logically with the event and persistence model.

The key is that **we don't update refcounts in senior memory.** A pointer from an inner road to an outer road is not counted. Also, the outmost, or `surface` road, is the only part of the image that gets checkpointed.

So the surface road contains the entire durable state of `u3`. When we process an event, or perform any kind of complicated or interesting calculation, **we process it in an inner road**. If its results are saved, they need to be copied.

Since processing in an inner road does not touch surface memory, (a) we can leave the surface road in a read-only state and not mark its pages dirty; (b) we can abort an inner calculation without screwing up the surface; and (c) because inner results are copied onto the surface, the surface doesn't get fragmented.

All of (a), (b) and (c) are needed for checkpointing to be easy. It might be tractable otherwise, but easy is even better.

Moreover, while the surface is most definitely single-threaded, we could easily run multiple threads in multiple inner roads (as long as the threads don't have pointers into each others' memory, which they obviously shouldn't).

Moreover, in future, we'll experiment more with adding road control hints to the programmer's toolbox. Reference counting is expensive. We hypothesize that in many - if not most - cases, the programmer can identify procedural structures whose garbage should be discarded in one step by copying the results. Then, within the procedure, we can switch the allocator into `sand` mode, and stop tracking references at all.

## u3: rules for C programming <a href="#u3-rules-for-c-programming" id="u3-rules-for-c-programming"></a>

There are two levels at which we program in C: (1) above the interpreter; (2) within the interpreter or jets. These have separate rules which need to be respected.

## u3: rules above the interpreter <a href="#u3-rules-above-the-interpreter" id="u3-rules-above-the-interpreter"></a>

In its relations with Unix, Urbit follows a strict rule of "call me, I won't call you." We do of course call Unix system calls, but only for the purpose of actually computing.

Above Urbit, you are in a normal C/Unix programming environment and can call anything in or out of Urbit. Note that when using `u3`, you're always on the surface road, which is not thread-safe by default. Generally speaking, `u3` is designed to support event-oriented, single-threaded programming.

If you need threads which create nouns, you could use `u3m_hate()` and `u3m_love()` to run these threads in subroads. You'd need to make the global road pointer, `u3R`, a thread-local variable instead. This seems perfectly practical, but we haven't done it because we haven't needed to.

## u3: rules within the interpreter <a href="#u3-rules-within-the-interpreter" id="u3-rules-within-the-interpreter"></a>

Within the interpreter, your code can run either in the surface road or in a deep road. You can test this by testing

```c
    (&u3H->rod_u == u3R)
```

ie: does the pier's home road equal the current road pointer? Normally in this context you assume you're obeying the rules of running on an inner road, ie, "deep memory." Remember, however, that the interpreter **can** run on surface memory - but anything you can do deep, you can do on the surface. The converse is by no means the case.

In deep memory, think of yourself as if in a signal handler. Your execution context is extremely fragile and may be terminated without warning or cleanup at any time (for instance, by `Ctrl+C`).

For instance, you can't call `malloc` (or C++ `new`) in your C code, because you don't have the right to modify data structures at the global level, and will leave them in an inconsistent state if your inner road gets terminated. (Instead, use our drop-in replacements, `u3a_malloc()`, `u3a_free()`, `u3a_realloc()`.)

A good example is the different meaning of `c3_assert()` inside and outside the interpreter. At either layer, you can use regular assert(), which will just kill your process. On the surface, `c3_assert()` will just... kill your process.

In deep execution, `c3_assert()` will issue an exception that queues an error event, complete with trace stack, on the Arvo event queue. Let's see how this happens.

## u3: exceptions <a href="#u3-exceptions" id="u3-exceptions"></a>

You produce an exception with

```c
    /* u3m_bail(): bail out.  Does not return.
    **
    **  Bail motes:
    **
    **    %exit               ::  semantic failure
    **    %evil               ::  bad crypto
    **    %intr               ::  interrupt
    **    %fail               ::  execution failure
    **    %foul               ::  assert failure
    **    %need               ::  network block
    **    %meme               ::  out of memory
    **    %time               ::  timed out
    **    %oops               ::  assertion failure
    */
      c3_i
      u3m_bail(c3_m how_m);
```

Broadly speaking, there are two classes of exception: internal and external. An external exception begins in a Unix signal handler. An internal exception begins with a call to longjmp() on the main thread.

There are also two kinds of exception: mild and severe. An external exception is always severe. An internal exception is normally mild, but some (like `c3__oops`, generated by `c3_assert()`) are severe.

Either way, exceptions come with a stack trace. The `u3` nock interpreter is instrumented to retain stack trace hints and produce them as a printable `(list tank)`.

Mild exceptions are caught by the first virtualization layer and returned to the caller, following the behavior of the Nock virtualizer `+mock` (in `hoon.hoon`)

Severe exceptions, or mild exceptions at the surface, terminate the entire execution stack at any depth and send the cumulative trace back to the `u3` caller.

For instance, `vere` uses this trace to construct a `%crud` event, which conveys our trace back toward the Arvo context where it crashed. This lets any UI component anywhere, even on a remote node, render the stacktrace as a consequence of the user's action - even if its its direct cause was (for instance) a Unix SIGINT or SIGALRM.

## u3: C structures on the loom <a href="#u3-c-structures-on-the-loom" id="u3-c-structures-on-the-loom"></a>

Normally, all data on the loom is nouns. Sometimes we break this rule just a little, though - eg, in the `u3h` hashtables.

To point to non-noun C structs on the loom, we use a `u3_post`, which is just a loom word offset. A macro lets us declare this as if it was a pointer:

```c
    typedef c3_w       u3_post;
    #define u3p(type)  u3_post
```

Some may regard this as clever, others as pointless. Anyway, use `u3to()` and `u3of()` to convert to and from pointers.

When using C structs on the loom - generally a bad idea - make sure anything which could be on the surface road is structurally portable, eg, won't change size when the pointer size changes. (Note also: we consider little-endian, rightly or wrongly, to have won the endian wars.)


# Tools

Developer documentation for additional tools, external libraries, and third-party applications for Urbit development, including JavaScript libraries and documentation tools.

This section of the docs is for developer documentation of additonal, external and/or third-party apps, libraries and tools.

* [Useful Links](/build-on-urbit/tools/links) - Links to external app and library repos, documentation, and other tools.
* [JS Libraries](/build-on-urbit/tools/js-libs) - Javascript libraries for interacting with an Urbit ship.
* [Docs App](/build-on-urbit/tools/docs-app) - Reference documentation for distributing app documentation for the docs app.


# Useful Links

Curated collection of external Urbit development resources including app repositories, Hoon libraries, JavaScript packages, documentation sites, and development tools.

A collection of links to external tools, libraries, app repos, and documentation.

## App Repos <a href="#app-repos" id="app-repos"></a>

[Autoprop](https://github.com/urbit/urbit/tree/develop/pkg/autoprop) - automatically build pills.

[Citadel](https://github.com/ryjm/citadel) - desk management on Urbit.

[Deskjam](https://github.com/tinnus-napbus/deskjam) - import and export desks.

[Docs](https://github.com/tinnus-napbus/docs-app) - include documentation with your app.

[Emissary](https://github.com/sigilante/emissary) - allow a star to designate a planet as its representative.

[Herd](https://github.com/midsum-salrux/herd) - desk dependency management.

[Paldev Suite](https://github.com/Fang-/suite) - apps and libraries by \~paldev including Pals and Rumors.

## External Docs <a href="#external-docs" id="external-docs"></a>

[GroundSeg Docs](https://manual.groundseg.app/) - documentation for GroundSeg by Native Planet.

## Hoon Libraries <a href="#hoon-libraries" id="hoon-libraries"></a>

[John](https://github.com/litlep-nibbyt/john) - automatic JSON reparsing.

[/lib/string](https://github.com/sigilante/string) - a string library for mortals.

[Manx Utils](https://github.com/tinnus-napbus/manx-utils) - tools for working with XML structures.

[Mast](https://github.com/R-JG/mast) - a library for building fully dynamic Sail front-ends.

[Paldev Suite](https://github.com/Fang-/suite) - apps and libraries by \~paldev including Rudder & Gossip.

[Regex](https://github.com/lynko/re.hoon) - an implementation of POSIX extended regular expressions.

[Saloon](https://github.com/sigilante/saloon) - transcendental and algebraic functions in Hoon (WIP).

[Yard](https://github.com/urbit/yard) - helpful utilities for app development.

## JS Libraries <a href="#js-libraries" id="js-libraries"></a>

[create-landscape-app](https://github.com/urbit/create-landscape-app) - get started building an Urbit app.

[HTTP API](https://github.com/urbit/js-http-api) - connect to an Urbit ship via a JS application.

[nockjs](https://github.com/urbit/nockjs) - A JS implementation of Nock, plus some useful functions.

[sigil-js](https://github.com/urbit/sigil-js) - A library for generating sigil profile pictures from `@p`s.

[urbit-ob](https://github.com/urbit/urbit-ob) - Utilities for Hoon-style atom printing and conversion.

## Other Languages <a href="#other-languages" id="other-languages"></a>

[go-urbit](https://github.com/stephenlacy/go-urbit/) - Golang Urbit interface over Ames.

[pynoun](https://github.com/urbit/tools) - jam and cue nouns into Python data types.

## Other Tools <a href="#other-tools" id="other-tools"></a>

[Bridge](https://github.com/urbit/bridge) - an application for interacting with Azimuth.

[Click](https://github.com/urbit/tools) - bash thin client that runs threads.

[GroundSeg](https://github.com/Native-Planet/GroundSeg) - a tool for managing Urbit instances.


# JS Libraries

JavaScript libraries for Urbit development including @urbit/http-api for communicating with a Gall agent backend from a React web frontend.

This section contains documentation of JS libraries for interacting with an Urbit ship.

* [HTTP API Guide](/build-on-urbit/tools/js-libs/http-api-guide) - A detailed guide to using the `@urbit/http-api` library for talking to an Urbit ship from a JS front-end.


# HTTP API

Guide to the @urbit/http-api JavaScript library for building web frontends that communicate with Urbit ships, covering authentication, pokes, subscriptions, scries, threads, and practical examples.

Urbit's Eyre vane is an HTTP server which our web frontends can talk to. In this guide, we'll create a simple Urbit app and use the [`@urbit/http-api`](https://github.com/urbit/js-http-api) JavaScript module to interact with it from a web app.

## Background <a href="#background" id="background"></a>

Eyre's API is a fairly thin overlay on some of Arvo's internal systems, so there's some basic things to understand.

### Clay <a href="#clay" id="clay"></a>

Clay is the filesystem vane. It's typed, and it's revision-controlled in a similar way to git. Clay contains a number of desks, which are a bit like git repositories. Each app on your ship's home screen corresponds to a desk in Clay. That desk contains the source code and resources for that app.

#### Marks

Most of Clay's workings aren't relevant to frontend development, but there's one important concept to understand: marks. Clay is a typed filesystem, and marks are the filetypes. There's a mark for `.hoon` files, a mark for `.txt` files, and so on. The mark specifies the datatype for those files, and it also specifies conversion methods between different types. Marks aren't just used for files saved in Clay, but also for data that goes to and from the web through Eyre.

When you send a [poke](#pokes) or run a [thread](#threads) through Eyre's HTTP API, Clay will look at the mark specified (for example `ui-action`, `contact-action-1`, etc.) and use the corresponding mark file in the relevant desk to convert the given JSON to that type, before passing it to the target [agent](#gall-agents) or thread. The same conversion will happen in reverse for responses.

Note that Eyre makes a best effort attempt to convert data to and from JSON. If the marks in question do not contain appropriate JSON conversion functions, it will fail. Not all [scry endpoints](#scry-endpoints), [subscription paths](#subscriptions), and pokes are intended to be used from a frontend, so not all of them use marks which can convert to and from JSON. (The `noun` mark for example). The majority of things you'll want to interact with through Eyre will work with JSON.

### Gall agents <a href="#gall-agents" id="gall-agents"></a>

An *agent* is a userspace application managed by the Gall vane. A desk may contain multiple agents that do different things. The [Tlon Messenger](https://github.com/tloncorp/tlon-apps) app, for example, has the `%contacts`, `%profile`, and `%lanyard` agents in its desk, among others. Agents are the main thing you'll interact with through Eyre. They have a simple interface with three main parts:

| Interface     | Description                                                                                                                                    |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Pokes         | One-off message to an agent. Pokes often represent actions, commands, requests, etc.                                                           |
| Subscriptions | An agent may have a number of different paths to which you may subscribe.                                                                      |
| Scries        | Scries are one-time, read-only requests for data. Like subscriptions, they are organized by path. Scry requests will be fulfilled immediately. |

#### Pokes <a href="#pokes" id="pokes"></a>

Pokes are single, standalone messages to agents. Pokes are how you send data and requests to agents. Agents will send back either a positive acknowledgement (ack) or a negative acknowledgement (nack). The agent can't send actual data back in the acks. If they have any response to give back, it will be sent out to subscribers on a subscription path instead.

The pokes an agent accepts will be defined in the `+on-poke` section of its source code in the desk's `/app` directory, or maybe in its type definition file in the `/sur` directory.

`@urbit/http-api` includes a `poke()` function which allows you to perform pokes through Eyre, and is [detailed below](#poke).

#### Subscriptions <a href="#subscriptions" id="subscriptions"></a>

Agents define subscription paths which you can subscribe to through Eyre. A path might be simple and fixed like `/foo/bar`, or it might have dynamic elements where you can specify a date, a user, a key in a key-value store, etc. Each agent will define its subscription paths in the `+on-watch` section of its source code.

You can subscribe by sending a request to the agent with the desired path specifed. The agent will apply some logic (such as checking permissions) to the request and then ack or nack it. If acked, you'll be subscribed. You might receive an initial payload of data defined in `+on-watch`. Then you'll begin receiving any updates the agent sends out on that path in future. What you'll receive on a given path depends entirely on the agent.

Agents can kick subscribers, and you can unsubscribe at any time.

`@urbit/http-api` includes a `subscribe()` function which allows you to subscribe and unsubscribe to paths through Eyre, and is [detailed below](#subscribe-and-unsubscribe).

#### Scry Endpoints <a href="#scry-endpoints" id="scry-endpoints"></a>

Pokes and subscriptions can modify the state of the agent. A third kind of interaction called a *scry* does not. It simply retrieves data from the agent without any side-effects. Agents can define *scry endpoints* which, like subscriptions, are paths. A scry to one of these endpoints will retrieve some data as determined by the agent. Like subscription paths, scry paths can be simple like `/foo/bar` or contain dynamic elements. Unlike subscriptions, a scry is a one-off request and the data will come back immediately.

Scry endpoints are defined in the `+on-peek` section of an agent's source code. Scry endpoints will be written with a leading letter like `/x/foo/bar`. That letter is a `$care` which tells Gall what kind of request this is. All scries through Eyre have a care of `/x`, so that letter needn't be specified.

`@urbit/http-api` includes a `scry()` function which allows you to perform scries through Eyre, and is [detailed below](#scry).

### Threads <a href="#threads" id="threads"></a>

A thread is a monadic function in Arvo that takes arguments and produces a result. Threads are conceptually similar to Javascript promises: they can perform one or more asynchronous I/O operations, which can be chained together, and will notify the agent that started them whether they succeedeed or failed. Threads are often used to handle complex I/O operations for agents. Threads live in the `/ted` directory of a desk.

`@urbit/http-api` includes a `thread()` function which allows you to run threads through Eyre, and is [detailed below](#thread).

## HTTP API basics <a href="#http-api-basics" id="http-api-basics"></a>

Now that we've covered the backend concepts, let's see how `@urbit/http-api` communicates with the server.

### The `Urbit()` object <a href="#the-urbit-object" id="the-urbit-object"></a>

All functionality is contained within the `Urbit()` object. There are two ways to instantiate it, depending on whether your web app is served directly from the ship or whether it's served externally. The reason for the difference is that you require a session cookie to talk to the ship.

If your app is served from the ship, the user will already be logged in and they'll have a session cookie that `Urbit()` will use automatically.

If your app isn't served from the ship, you'll need to authenticate with the user's ship, which is [detailed separately below](#authenticate).

In the case of a frontend served from the ship, the `Urbit()` class contains a constructor which takes 1-3 arguments:

| Argument | Type     | Description                                                                                                                                                                                                                                                                                                                            | Example                                          |
| -------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| `url`    | `string` | The host of the ship. This string is mandatory, but is typically left empty as requests will still work if they're root-relative paths.                                                                                                                                                                                                | `"example.com"`, `"http://localhost:8080"`, `""` |
| `code`   | `string` | (Optional.) The web login code of the ship. Not needed if your frontend is served from the ship. In practice this should never be set by the frontend (if you need the user to log into their ship, use EAuth), but if you had to you'd want to import this from a secure environment variable to avoid putting it in the source code. | `""`, `"lidlut-tabwed-pillex-ridrup"`            |
| `desk`   | `string` | (Optional.) The desk on which you want to run threads. This is only used if you want to run threads from the frontend, rather than run them from the agent.                                                                                                                                                                            | `"landscape"`, `""`                              |

To create an `Urbit()` instance, you can simply do:

```javascript
const api = new Urbit("");
```

If you want to specify a desk, you can do:

```javascript
const api = new Urbit("", "", "landscape");
```

### `/session.js` <a href="#sessionjs" id="sessionjs"></a>

Most functions of `Urbit()` need to know the ship's Urbit ID or they will fail. This is given explicitly with the external authentication method [detailed below](#authenticate), but that's unnecessary when using the `Urbit()` object in a web app served directly from the ship, because the ship serves a JS library at `/session.js` that contains the following:

```javascript
window.ship = "zod";
```

`"zod"` will be replaced with the actual name of the ship in question. You can import this file like so:

```html
<script src="/session.js"></script>
```

Then you need to set the `ship` field in the `Urbit()` object. You would typically do it immediately after instantiating it:

```javascript
const api = new Urbit("");
api.ship = window.ship;
```

### Channels <a href="#channels" id="channels"></a>

With the exception of scries and threads, all communication with Eyre happens through its channel system.

When it's constructed, the `Urbit()` object will generate a random channel ID like `1646295453-e1bdfd`, and use a path of `/~/channel/1646295453-e1bdfd` to talk to Eyre. Pokes and subscription requests will be sent to that channel. Responses and subscription updates will be sent out to the frontend on that channel too.

Eyre sends out updates and responses on an SSE (Server Sent Event) stream for that channel. The `Urbit()` object handles this internally with an `eventSource` object, so you won't deal with it directly. Eyre requires all events it sends out be acknowledged by the client, and will eventually close the channel if enough unacknowledged events accumulate. The `Urbit()` object handles event acknowledgement automatically.

Eyre automatically creates a channel when a poke or subscription request is first sent to `/~/channel/[unknown-channel-id]`. If your web app is served outside a ship, you could use the `authenticate()` function [described below](#authenticate) which will automatically send a poke and open the new channel. If your web app is served directly from the ship and you use the `Urbit()` object, it won't open the channel right away. Instead, the channel will be opened whenever you first send a poke or subscription request.

### Connection state <a href="#connection-state" id="connection-state"></a>

The `Urbit()` object includes three optional callback functions that fire when the SSE connection state changes:

| Callback    | Description                                                                                            |
| ----------- | ------------------------------------------------------------------------------------------------------ |
| `onOpen()`  | Called when an SSE channel connection is successfully established.                                     |
| `onRetry()` | Called when a reconnection attempt is made due to an interruption, e.g. if there are network problems. |
| `onError()` | Called when there is an unrecoverable error, e.g. after enough reconnection attemps have failed.       |

As mentioned in the previous section, typically a channel will be opened and an SSE connection established after you first poke the ship or make a subscription request. If successful, whatever function you provided to `onOpen()` will be called. If at some point the connection is interrupted, a reconnection attempt will be made three times:

1. Instantly.
2. 750ms after the first.
3. 3000ms after the second.

Each attempt will call the function you provided to `onRetry()`, if any. If all three reconnection attempts failed, or if a fatal error occurred, the function you provided `onError()` will be called with an `Error` object containing an error message as its argument.

How you use these, if at all, is up to you. If you want to try reconnecting when `onError()` fires, note that Eyre will delete a channel if it's had no messages from the client in the last 12 hours. The timeout is reset whenever it receives a message, including the acks that are automatically sent by the `Urbit()` object in response to subscription updates.

If you don't want to account for the possibility of the channel having been deleted, you can just call the [`reset()`](#reset) function before you try reconnecting and consequently open a brand new channel.

## Tutorial setup <a href="#tutorial-setup" id="tutorial-setup"></a>

Start a fake \~zod and we'll add a Gall agent to its `%base` desk. This agent will store a trivial key-value database. It will provide endpoints for pokes, scries, and subscriptions. We'll define its poke and update types in a `/sur` file, and create marks that convert those types to and from JSON. We'll use `@urbit/http-api` to interact with the agent.

### Types <a href="#types" id="types"></a>

In the `/sur` folder of the `%base` desk, create a file `/api-demo.hoon` and define the following types:

* `$api-action`: User actions sent from the frontend to the Gall agent. We just want to put new k-v pairs into the state, and delete them by their keys.
* `$api-update`: Updates sent from the Gall agent to the frontend. Updates tagged with `%store` will contain the entire updated k-v store. Updates tagged with `%key-value` will contain one key and a unit of a value.

{% code title="/sur/api-demo.hoon" lineNumbers="true" %}

```hoon
|%
+$  api-action
  $%  [%put key=@tas val=@t]
      [%del key=@tas]
  ==
+$  api-update
  $%  [%store store=(map @tas @t)]
      [%key-value key=@tas val=(unit @t)]
  ==
--
```

{% endcode %}

### Marks <a href="#marks" id="marks"></a>

In the `/mar` folder of the `%base` desk, create two new files `/api-action.hoon` and `/api-update.hoon`. Both of these marks will contain functions for converting their respective types in `/sur` to and from JSON. Don't worry if you don't understand every line of this.

{% code title="/mar/api-action.hoon" lineNumbers="true" %}

```hoon
/-  *api-demo
|_  act=api-action
++  grab
  |%
  ++  noun  api-action
  ++  json
    |=  jon=^json
    %-  api-action
    =,  format
    %.  jon
    %-  of:dejs
    :~  :-  %put
        %-  ot:dejs
        :~  [%key (se:dejs %tas)]
            [%val so:dejs]
        ==
        :-  %del
        %-  ot:dejs
        :~  [%key (se:dejs %tas)]
        ==
    ==
  --
++  grow
  |%
  ++  noun  act
  ++  json
    ^-  ^json
    =,  format
    ?-  -.act
        %put
      %-  frond:enjs
      :-  'put'
      %-  pairs:enjs
      :~  ['key' [%s key.act]]
          ['val' [%s val.act]]
      ==
    ::
        %del
      %-  frond:enjs
      :-  'del'
      %-  frond:enjs
      :-  'key'
      [%s key.act]
    ==
  --
++  grad  %noun
--
```

{% endcode %}

{% code title="/mar/api-update.hoon" lineNumbers="true" %}

```hoon
/-  *api-demo
|_  upd=api-update
++  grab
  |%
  ++  noun  api-update
  --
++  grow
  |%
  ++  noun  upd
  ++  json
    =,  format
    ^-  ^json
    ?-  -.upd
        %store
      %-  frond:enjs
      :-  'store'
      %-  pairs:enjs
      %+  turn
        ~(tap by store.upd)
      |=  [k=@tas v=@t]
      [(@t k) [%s v]]
    ::
        %key-value
      %-  frond:enjs
      :-  'key-value'
      %-  pairs:enjs
      :~  ['key' [%s (@t key.upd)]]
          ['val' ?~(val.upd [%s ''] [%s u.val.upd])]
      ==
    ==
  --
++  grad  %noun
--
```

{% endcode %}

### Agent <a href="#agent" id="agent"></a>

In the `%base` desk's `/app` directory, create a new file called `/api-demo.hoon` and paste in the code below.

{% code title="/app/api-demo.hoon" lineNumbers="true" %}

```hoon
/-  *api-demo
/+  default-agent, dbug
::
|%
+$  card  card:agent:gall
+$  versioned-state
  $%  state-0
  ==
+$  state-0
  $:  %0
      store=(map @tas @t)
  ==
--
::
%-  agent:dbug
=|  state-0
=*  state  -
::
^-  agent:gall
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
::
++  on-init
  ^-  (quip card _this)
  :-  ~
  %=  this
    state  [%0 ~]
  ==
::
++  on-save  !>(state)
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
      %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?+  mark
    (on-poke:def mark vase)
  ::
      %api-action
    =/  act  !<(api-action vase)
    ?-  -.act
        %put
      ~&  >  "api-demo: putting [{<key.act>} {<val.act>}]"
      =/  new-store
        (~(put by store) key.act val.act)
      :_  %=  this
            store  new-store
          ==
      :~  :*  %give
              %fact
              [/updates]~
              %api-update
              !>  ^-  api-update
              [%store new-store]
          ==
          :*  %give
              %fact
              [(welp /updates [key.act]~)]~
              %api-update
              !>  ^-  api-update
              [%key-value key.act (some val.act)]
          ==
      ==
    ::
        %del
      ~&  >  "api-demo: deleting {<key.act>}"
      :_  %=  this
            store  (~(del by store) key.act)
          ==
      :~  :*  %give
              %fact
              [(welp /updates [key.act]~)]~
              %api-update
              !>  ^-  api-update
              [%key-value key.act ~]
          ==
      ==
    ==
  ==
::
++  on-watch
  |=  =(pole knot)
  ^-  (quip card _this)
  ?+  pole
    (on-watch:def pole)
  ::
      [%updates ~]
    ~&  >  "api-demo: subscribed to /updates"
    :_  this
    :~  :*  %give
            %fact
            ~
            %api-update
            !>  ^-  api-update
            [%store store]
        ==
    ==
  ::
      [%updates key=@tas ~]
    ~&  >  "api-demo: subscribed to {<`path`(welp /updates [key.pole]~)>}"
    :_  this
    :~  :*  %give
            %fact
            ~
            %api-update
            !>  ^-  api-update
            [%key-value key.pole (~(get by store) key.pole)]
        ==
    ==
  ==
::
++  on-peek
  |=  =(pole knot)
  ^-  (unit (unit cage))
  ~&  >  "api-demo: scry on {<`path`pole>}"
  ?+  pole
    (on-peek:def pole)
  ::
      [%x %store ~]
    %-  some
    %-  some
    :-  %api-update
    !>  ^-  api-update
    [%store store]
  ::
      [%x %store key=@tas ~]
    %-  some
    %-  some
    :-  %api-update
    !>  ^-  api-update
    [%key-value key.pole (~(get by store) key.pole)]
  ==
::
++  on-leave
  |=  =(pole knot)
  ~&  >  "api-demo: unsubscribed from {<`path`pole>}"
  `this
::
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
--
```

{% endcode %}

### Committing the code <a href="#committing-the-code" id="committing-the-code"></a>

Finally, run `|commit %base` in the ship's dojo to commit your changes to the desk, then run `|start %api-demo` to initialise the agent.

## Using the HTTP API <a href="#using-the-http-api" id="using-the-http-api"></a>

Create and serve the HTML examples below from a local URL to talk to your fake \~zod via the HTTP API.

An Urbit ship will deny CORS requests from external URLs by default. **In order to run the examples below, you'll need to serve them from a URL (for example, with Python's http.server module) and approve that URL in the ship's dojo.** If serving the example page from `http://localhost:8000`, you'll need to run:

```
|eyre/cors/approve 'http://localhost:8000'
```

### Importing the HTTP API <a href="#importing-the-http-api" id="importing-the-http-api"></a>

The `http-api` module is available in npm as `@urbit/http-api`, and can be installed with:

```
npm i @urbit/http-api
```

Once installed, you can import it into your app with:

```javascript
import Urbit from '@urbit/http-api';
```

Note that the examples in this guide are simple HTML documents with vanilla Javascript in `<script>` tags, so they use [unpkg.com](https://unpkg.com) to import `@urbit/http-api`. This is not typical, and is just done here for purposes of simplicity.

### Authenticate <a href="#authenticate" id="authenticate"></a>

**If your frontend is served directly from the Urbit ship, this can be skipped.**

If your web app is served externally to the ship, you must authenticate and obtain a session cookie before commencing communications with the ship.

The `Urbit()` object includes an `authenticate()` function which does the following:

1. Login to the user's ship with their `code` and obtain a session cookie.
2. Generate a random channel ID for the connection.
3. Poke the user's ship and print "opening airlock" in the dojo to initialize the channel.

The `authenticate()` function takes four arguments in an object: `ship`, `url`, `code` and `verbose`:

| Argument  | Type      | Description                                                                                        | Example                                      |
| --------- | --------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `ship`    | `string`  | (Optional.) The ship ID (`@p`) without the leading `~`.                                            | `"sampel-palnet"` or `"zod"`                 |
| `url`     | `string`  | The base URL for the ship.                                                                         | `"http://localhost:8080"` or `"example.com"` |
| `code`    | `string`  | (Optional.) The user's web login code.                                                             | `"lidlut-tabwed-pillex-ridrup"`              |
| `verbose` | `boolean` | (Optional.) Whether to log details to the console. This field is optional and defaults to `false`. | `true`                                       |

This function returns a promise that if successful, produces an `Urbit()` object which can then be used for communications with the ship.

#### `authenticate()` example <a href="#authenticate-example" id="authenticate-example"></a>

{% code title="auth-test.html" lineNumbers="true" %}

```html
<html>
  <head>
    <script src="https://unpkg.com/@urbit/http-api"></script>
  </head>
  <body>
    <button id="start" type="button" onClick="connect()" >Connect</button>
  </body>
  <script>
    async function connect() {
      window.api = await UrbitHttpApi.Urbit.authenticate({
          ship: "zod",
          url: "http://localhost:8080",
          code: "lidlut-tabwed-pillex-ridrup",
          verbose: true
      });
      document.body.innerHTML = "Connected!";
    };
  </script>
</html>
```

{% endcode %}

### Poke <a href="#poke" id="poke"></a>

For poking a ship, `Urbit()` includes a `poke()` function. The `poke()` function takes six arguments in a object:

| Argument    | Type        | Description                                                                                                                                | Example                               |
| ----------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------- |
| `app`       | `string`    | The Gall agent to poke.                                                                                                                    | `"api-demo"`                          |
| `mark`      | `string`    | The mark of the data to poke the agent with.                                                                                               | `"api-action"`                        |
| `json`      | any JSON    | The data to poke the agent with.                                                                                                           | `{ put: { key: "foo", val: "bar" } }` |
| `ship`      | `string`    | (Optional.) The Urbit ID (`@p`) of the ship without the `~`. This may be ommitted if it's already been set for the whole `Urbit()` object. | `"zod"`                               |
| `onSuccess` | A function. | (Optional.) This is called if the poke succeeded (the ship ack'd the poke).                                                                | `someFunction()`                      |
| `onError`   | A function. | (Optional.) This is called if the poke failed (the ship nack'd the poke).                                                                  | `anotherFunction()`                   |

#### `poke()` example <a href="#poke-example" id="poke-example"></a>

{% code title="poke-test.html" lineNumbers="true" %}

```html
<html>
  <head>
    <script src="https://unpkg.com/@urbit/http-api"></script>
  </head>
  <body>
    <h2>HTTP API - Pokes</h2>
    <br>
    <div>
      <input id="put-key" type="text" placeholder="Key" />
      <input id="put-value" type="text" placeholder="Value" />
      <button id="put-button" type="button" onClick="putByKey()">Store Value</button>
    </div>
    <br>
    <div>
      <input id="del-key" type="text" placeholder="Key to delete" />
      <button id="del-button" type="button" onClick="delByKey()">Delete Value</button>
    </div>
    <br>
    <p id="status"></p>
  </body>
  <script>
    const api = new UrbitHttpApi.Urbit("");
    api.ship = "zod";
    api.url = "http://localhost:8080";
    api.code = "lidlut-tabwed-pillex-ridrup";

    function putByKey() {
      const key = document.getElementById("put-key").value;
      const value = document.getElementById("put-value").value;
      
      if (!key) {
        document.getElementById("status").innerHTML = "Error: Key is required";
        return;
      }

      if (!value) {
        document.getElementById("status").innerHTML = "Error: Value is required for deletion";
        return;
      }

      try {
        console.log(key)
        console.log(value)

        api.poke({
          app: "api-demo",
          mark: "api-action",
          json: { put: { key: key, val: value } },
          onSuccess: pokeSuccess,
          onError: pokeError,
        });
      } catch (err) {
        document.getElementById("status").innerHTML = "Poke error: " + err.message;
      }
    }

    function delByKey() {
      const key = document.getElementById("del-key").value;

      if (!key) {
        document.getElementById("status").innerHTML = "Error: Key is required for deletion";
        return;
      }

      try {
        api.poke({
          app: "api-demo",
          mark: "api-action",
          json: { del: { key: key } },
          onSuccess: successDelete,
          onError: pokeError,
        });
      } catch (err) {
        document.getElementById("status").innerHTML = "Poke error: " + err.message;
      }
    }

    function pokeSuccess() {
      document.getElementById("put-key").value = "";
      document.getElementById("put-value").value = "";
      document.getElementById("del-key").value = "";
      document.getElementById("status").innerHTML = "Value stored successfully!";
    }

    function successDelete() {
      document.getElementById("put-key").value = "";
      document.getElementById("put-value").value = "";
      document.getElementById("del-key").value = "";
      document.getElementById("status").innerHTML = "Value deleted successfully!";
    }

    function pokeError() {
      document.getElementById("status").innerHTML = "Poke failed, see dojo";
    }
  </script>
</html>
```

{% endcode %}

### Scry <a href="#scry" id="scry"></a>

To scry agents on the ship, `Urbit()` includes a `scry()` function. The `scry()` function takes two arguments in a object:

| Argument | Type     | Description                        | Example      |
| -------- | -------- | ---------------------------------- | ------------ |
| `app`    | `string` | The agent to scry.                 | `"api-demo"` |
| `path`   | `string` | The path to scry, sans the `care`. | `"/store"`   |

The `scry()` function returns a promise that, if successful, contains the requested data as JSON. If the scry failed, for example due to a non-existent scry endpoint, connection problem, or mark conversion failure, the promise will fail.

#### `scry()` example <a href="#scry-example" id="scry-example"></a>

{% code title="scry-test.html" lineNumbers="true" %}

```html
<html>
  <head>
    <script src="https://unpkg.com/@urbit/http-api"></script>
  </head>
  <body>
    <h1>HTTP API - Scry Endpoints</h1>
    <div>
      <h2>Fetch entire store</h2>
      <div>
        <button
          id="store-scry-btn"
          type="button"
          onClick="scryStore()"
        >
          Scry /store
        </button>
      </div>
      <br />
      <div id="store-status">No data requested yet</div>
      <pre id="store-data">Results will appear here</pre>
    </div>
    <br />
    <div>
      <h2>Fetch a specific key</h2>
      <input
        id="key-input"
        type="text"
        placeholder="Enter key to fetch"
      />
      <div>
      <br />
        <button
          id="key-scry-btn"
          type="button"
          onClick="scryKey()"
        >
          Scry key
        </button>
      </div>
      <br />
      <div id="key-status">No data requested yet</div>
      <pre id="key-data">Results will appear here</pre>
    </div>
  </body>
  <script>
    const api = new UrbitHttpApi.Urbit("");
    api.ship = "zod";
    api.url = "http://localhost:8080";
    api.code = "lidlut-tabwed-pillex-ridrup";

    // Scry the entire store
    async function scryStore() {
      const statusEl = document.getElementById("store-status");
      const dataEl = document.getElementById("store-data");
      
      statusEl.innerText = "Fetching data...";
      
      try {
        const result = await api.scry({
          app: "api-demo",
          path: "/store"
        });
        
        statusEl.innerText = "Data fetched successfully";
        dataEl.innerText = JSON.stringify(result, null, 2);
      } catch (err) {
        statusEl.innerText = "Error fetching data";
        dataEl.innerText = "Error: " + (err.message || "Unknown error");
      }
    }

    // Scry a specific key
    async function scryKey() {
      const key = document.getElementById("key-input").value;
      const statusEl = document.getElementById("key-status");
      const dataEl = document.getElementById("key-data");
      
      if (!key) {
        statusEl.innerText = "Error: Please enter a key";
        return;
      }
      
      statusEl.innerText = "Fetching data...";
      
      try {
        const result = await api.scry({
          app: "api-demo",
          path: `/store/${key}`
        });
        
        statusEl.innerText = "Data fetched successfully";
        if (result === null) {
          dataEl.innerText = "Key not found";
        } else {
          dataEl.innerText = JSON.stringify(result, null, 2);
        }
      } catch (err) {
        statusEl.innerText = "Error fetching data";
        dataEl.innerText = "Error: " + (err.message || "Unknown error");
      }
    }
  </script>
</html>
```

{% endcode %}

### Subscribe and unsubscribe <a href="#subscribe-and-unsubscribe" id="subscribe-and-unsubscribe"></a>

For subscribing to a particular path in an agent, `Urbit()` includes a `subscribe` function. The `subscribe` function takes six arguments in a object:

| Argument | Type        | Description                                                                                                                                  | Example                |
| -------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `app`    | `string`    | The Gall agent to which you'll subscribe.                                                                                                    | `"api-demo"`           |
| `path`   | `string`    | The subscription path.                                                                                                                       | `"/updates"`           |
| `ship`   | `string`    | (Optional.) The Urbit ID (`@p`) of the ship without the `~`. This may be ommitted if it's already been set for the whole `Urbit()` object.   | `"zod"`                |
| `err`    | A function. | (Optional.) This is called if the subscription request fails.                                                                                | `someFunction()`       |
| `event`  | A function. | (Optional.) This is the function to handle each update you receive for this subscription. The function's argument is the update's JSON data. | `anotherFunction()`    |
| `quit`   | A function. | (Optional.) This is called if you are kicked from the subscription.                                                                          | `yetAnotherFunction()` |

The `subscribe` function returns a subscription ID, which is just a number. This ID can be used to unsubscribe down the line.

If the subscription request is successful, you'll continue to receive updates until you either unsubscribe or are kicked by the agent. You may subscribe to multiple different agents and subscription paths by calling the `subscribe` function for each one.

If you wish to unsubscribe from a particular subscription, `Urbit()` includes an `unsubscribe` function. This function just takes a single argument: the subscription ID number of an existing subscription. Once unsubscribed, you'll stop receiving updates for the specified subscription.

#### `subscribe()` example <a href="#subscribe-example" id="subscribe-example"></a>

{% code title="subscribe-test.html" lineNumbers="true" %}

```html
<html>
  <head>
    <script src="https://unpkg.com/@urbit/http-api"></script>
  </head>
  <body>
    <h1>HTTP API - Subscriptions</h1>
    <div>
      <h2>Subscribe to the /updates wire</h2>
      <div>
        <button
          id="store-sub-btn"
          type="button"
          onClick="toggleStoreSubscription()"
        >
          Subscribe to /updates
        </button>
      </div>
      <br />
      <div id="store-status">Not subscribed</div>
      <pre id="store-data">No data yet</pre>
    </div>
    <br />
    <div>
      <h2>Subscribe to a wire for a specific key</h2>
      <input
        id="key-input"
        type="text"
        placeholder="Enter key to subscribe to"
      />
      <div>
      <br />
        <button
          id="key-sub-btn"
          type="button"
          onClick="toggleKeySubscription()"
        >
          Subscribe to key
        </button>
      </div>
      <br />
      <div id="key-status">Not subscribed</div>
      <pre id="key-data">No data yet</pre>
    </div>
  </body>
  <script>
    const api = new UrbitHttpApi.Urbit("");
    api.ship = "zod";
    api.url = "http://localhost:8080";
    api.code = "lidlut-tabwed-pillex-ridrup";

    let storeSubId = null;
    let keySubId = null;
    let keyName = null;

    // Subscribe to all store updates
    function toggleStoreSubscription() {
      const statusEl = document.getElementById("store-status");
      const btnEl = document.getElementById("store-sub-btn");
      const dataEl = document.getElementById("store-data");

      if (storeSubId === null) {
        // Subscribe
        statusEl.innerText = "Subscribing...";

        storeSubId = api.subscribe({
          app: "api-demo",
          path: "/updates",
          err: () => {
            storeSubId = null;
            statusEl.innerText = "Subscription failed!";
            btnEl.innerText = "Subscribe to /updates";
          },
          event: (update) => {
            statusEl.innerText = "Receiving updates";
            dataEl.innerText = JSON.stringify(update, null, 2);
          },
          quit: () => {
            storeSubId = null;
            statusEl.innerText = "Kicked from subscription";
            btnEl.innerText = "Subscribe to /updates";
            dataEl.innerText = "No data - subscription ended";
          },
        });

        btnEl.innerText = "Unsubscribe from /updates";
        statusEl.innerText = "Subscribed, awaiting events...";
      } else {
        // Unsubscribe
        api.unsubscribe(storeSubId);
        storeSubId = null;
        btnEl.innerText = "Subscribe to /updates";
        statusEl.innerText = "Not subscribed";
        dataEl.innerText = "No data - unsubscribed";
      }
    }

    // Subscribe to a specific key
    function toggleKeySubscription() {
      const key = document.getElementById("key-input").value;
      const statusEl = document.getElementById("key-status");
      const btnEl = document.getElementById("key-sub-btn");
      const dataEl = document.getElementById("key-data");

      if (!key) {
        statusEl.innerText = "Error: Please enter a key";
        return;
      }

      if (keySubId === null) {
        // Subscribe
        statusEl.innerText = "Subscribing...";

        keySubId = api.subscribe({
          app: "api-demo",
          path: `/updates/${key}`,
          err: () => {
            keySubId = null;
            keyName = null;
            statusEl.innerText = "Subscription failed!";
            btnEl.innerText = "Subscribe to key";
          },
          event: (update) => {
            statusEl.innerText = "Receiving updates";
            dataEl.innerText = JSON.stringify(update, null, 2);
          },
          quit: () => {
            keySubId = null;
            keyName = null;
            statusEl.innerText = "Kicked from subscription";
            btnEl.innerText = "Subscribe to key";
            dataEl.innerText = "No data";
          },
        });

        keyName = key;
        btnEl.innerText = `Unsubscribe from ${key}`;
        statusEl.innerText = "Subscribed, awaiting events...";
      } else {
        // Unsubscribe
        api.unsubscribe(keySubId);
        keySubId = null;
        keyName = null;
        btnEl.innerText = "Subscribe to key";
        statusEl.innerText = "Unsubscribed";
        dataEl.innerText = "No data";
      }
    }
  </script>
</html>
```

{% endcode %}

### Subscribe once <a href="#subscribe-once" id="subscribe-once"></a>

The `subscribeOnce()` function is a variation on the ordinary [`subscribe`](#subscribe-and-unsubscribe) function. Rather than keeping the subscription going and receiving an arbitrary number of updates, instead it waits to receive a single update and then closes the subscription. This is useful if, for example, you send a poke and just want a response to that one poke.

The `subscribeOnce()` function also takes an optional `timeout` argument, which specifies the number of milliseconds to wait for an update before closing the subscription. If omitted, `subscribeOnce()` will wait indefinitely.

`subscribeOnce()` takes three arguments (these can't be in an object like most other `Urbit()` functions):

| Argument  | Type     | Description                                                                                                                          | Example      |
| --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------ |
| `app`     | `string` | The Gall agent to which you'll subscribe.                                                                                            | `"api-demo"` |
| `path`    | `string` | The subscription path.                                                                                                               | `"/updates"` |
| `timeout` | `number` | (Optional.) The number of milliseconds to wait for an update before closing the subscription. If omitted, it will wait indefinitely. | `5000`       |

`subscribeOnce()` returns a Promise. If successful, the Promise produces the JSON data of the update it received. If it failed due to either timing out or getting kicked from the subscription, it will return an error message of either `"timeout"` or `"quit"`.

#### `subscribeOnce()` example <a href="#subscribe-once-example" id="subscribe-once-example"></a>

{% code title="auth-test.html" lineNumbers="true" %}

```html
<html>
  <head>
    <script src="https://unpkg.com/@urbit/http-api"></script>
  </head>
  <body>
    <h1>HTTP API - Subscribe Once</h1>
    <div>
      <h2>Subscribe once to the /updates wire</h2>
      <div>
        <button
          id="store-sub-btn"
          type="button"
          onClick="subscribeOnceToStore()"
        >
          Subscribe to /updates and unsubscribe
        </button>
      </div>
      <br />
      <div id="store-status">No data requested yet</div>
      <pre id="store-data">Results will appear here</pre>
    </div>
  </body>
  <script>
    const api = new UrbitHttpApi.Urbit("");
    api.ship = "zod";
    api.url = "http://localhost:8080";
    api.code = "lidlut-tabwed-pillex-ridrup";

    // Subscribe once to all store updates
    async function subscribeOnceToStore() {
      const statusEl = document.getElementById("store-status");
      const dataEl = document.getElementById("store-data");

      statusEl.innerText = "Subscribing once...";
      dataEl.innerText = "Waiting for one update...";

      try {
        const result = await api.subscribeOnce("api-demo", "/updates", 5000);
        statusEl.innerText = "Received one update and closed subscription";
        dataEl.innerText = JSON.stringify(result, null, 2);
      } catch (err) {
        if (err === "timeout") {
          statusEl.innerText = "Subscription timed out";
          dataEl.innerText = "No update received within the timeout period";
        } else if (err === "quit") {
          statusEl.innerText = "Kicked from subscription";
          dataEl.innerText = "The agent kicked us from the subscription";
        } else {
          statusEl.innerText = "Subscription error";
          dataEl.innerText = "Error: " + (err.message || "Unknown error");
        }
      }
    }
  </script>
</html>
```

{% endcode %}

### Run a thread <a href="#thread" id="thread"></a>

To run a thread, `Urbit()` includes a `thread` function. The `thread` function takes five arguments in an object:

| Argument     | Type     | Description                                                                                                              | Example  |
| ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------ | -------- |
| `inputMark`  | `string` | The mark to convert your JSON data to before giving it to the thread as its argument.                                    | `"ship"` |
| `outputMark` | `string` | The result of the thread should be converted to this mark before being converted to JSON and returned to you.            | `"tang"` |
| `threadName` | `string` | The name of the thread to run.                                                                                           | `"hi"`   |
| `body`       | any JSON | The data to give to the thread as its argument.                                                                          | `"~bud"` |
| `desk`       | `string` | (Optional.) The desk in which the thread resides. This may be ommitted if previously set for the whole `Urbit()` object. | `"base"` |

The `thread` function will produce a promise that, if successful, contains the JSON result of the thread. If the thread failed, a connection error occurred, or mark conversion failed, the promise will fail.

#### `thread()` example <a href="#thread-example" id="thread-example"></a>

{% code title="thread-test.html" lineNumbers="true" %}

```html
<html>
  <head>
    <script src="https://unpkg.com/@urbit/http-api"></script>
  </head>
  <body>
    <h1>HTTP API - Threads</h1>
    <div>
      <h2>Run the "hi" thread in %base</h2>
      <div>
        <input id="name-input" type="text" placeholder="~bud" />
        <br />
        <br />
        <button id="run-thread-btn" type="button" onClick="runHiThread()">
          Say hi
        </button>
      </div>
      <br />
      <div id="thread-status">No thread run yet</div>
      <pre id="thread-result">Results will appear here</pre>
    </div>
  </body>
  <script>
    const api = new UrbitHttpApi.Urbit("");
    api.ship = "zod";
    api.desk = "base";
    api.url = "http://localhost:8080";
    api.code = "lidlut-tabwed-pillex-ridrup";

    async function runHiThread() {
      const name = document.getElementById("name-input").value;
      const statusEl = document.getElementById("thread-status");
      const resultEl = document.getElementById("thread-result");

      if (!name) {
        statusEl.innerText = "Input cannot be empty";
        return;
      }

      statusEl.innerText = "Running thread...";
      resultEl.innerText = "Waiting for thread to complete...";

      try {
        const result = await api.thread({
          inputMark: "ship",
          outputMark: "tang",
          threadName: "hi",
          body: name,
        });

        if (result) {
          statusEl.innerText = "Thread completed!";
          resultEl.innerText = "Check the recipient's dojo";
        }
      } catch (err) {
        statusEl.innerText = "Thread error";
        resultEl.innerText = "Error: " + (err.message || "Unknown error");
      }
    }
  </script>
</html>
```

{% endcode %}

### Delete a channel <a href="#delete-a-channel" id="delete-a-channel"></a>

Rather than just closing individual subscriptions, the entire channel can be closed with the `delete()` function in `Urbit()`. When a channel is closed, all subscriptions are cancelled and all pending updates are discarded. The function takes no arguments, and can be called like `api.delete()`.

### Reset <a href="#reset" id="reset"></a>

An existing `Urbit()` object can be reset with its `reset()` function. This function takes no arguments, and can be called like `api.reset()`. When a channel is reset, all subscriptions are cancelled and all pending updates are discarded. Additionally, all outstanding outbound pokes to the agent will be discarded, and a fresh channel ID will be generated.

## Further reading <a href="#further-reading" id="further-reading"></a>

* [`@urbit/http-api` on Github](https://github.com/urbit/js-http-api) - The source code for the JS HTTP API package.
* [Eyre External API Reference](/urbit-os/kernel/eyre/external-api-ref) - Lower-level documentation of Eyre's external API.
* [Eyre Guide](/urbit-os/kernel/eyre/guide) - Lower-level examples of using Eyre's external API with `curl`.


# Docs App

Documentation application for distributing docs with Urbit desks.

The `%docs` app allows you to include documentation with the desks you distribute, making them available for easy browsing by users and developers.

The `%docs` app will automatically detect and publish any docs included with any installed desks. As long as they're in the correct place and of a filetype with appropriate `$mark` conversion methods, they'll be picked up.

A `doc.toc` file may be included in the root of the desk, specifying the files to be included, their `$mark`s, and their titles. See the [Index File](/build-on-urbit/tools/docs-app/index-file) section for details.

The files will be in a `/doc` directory in the root of the desk. They may be of any filetype, as long as it has conversion methods to the `%docu` mark used by the `%docs` app. The `%docs` app includes parsers and conversion methods for the following marks by default:

* `%txt` - Ordinary `.txt` text files.
* `%udon` - A markdown-like format that supports embedded hoon.
* `%gmi` - An ultra-minimalist markup format called "gemtext".
* `%html` - An ordinary `.html` file.

For more details of these file formats, as well as details of the `%docu` mark and other format requirements of the `%docs` app, see the [File Format](/build-on-urbit/tools/docs-app/file-format) section.

Apart from the `$mark` and location requirements described above, there are no particular restrictions on how you organise your docs, or what docs you include. There are, however, some general recommendations detailed in the [Suggested Structure](/build-on-urbit/tools/docs-app/structure) section.

## Dev desk <a href="#dev-desk" id="dev-desk"></a>

To include docs in your own desk, there are a few files you'll need (mark files, etc). These are included in the `/docs-dev` directory in the [github repo](https://github.com/tinnus-napbus/docs-app). These files are all symlinked, so you'll need to clone the whole repo, then copy them across with something like:

```
cp -rL git/docs-app/docs-dev/* /path/to/your/development/desk
```


# File Format

File format specifications for the %docs app including %docu mark requirements, supported formats (udon, txt, html, gmi), syntax rules, and conversion guidelines for documentation publishing.

## `%docu` mark <a href="#docu-mark" id="docu-mark"></a>

{% hint style="info" %}
**Note**: this section is mostly useful if you're writing a mark conversion method. For marks that are already supported and you can use directly, see further down this page.
{% endhint %}

The `%docs` app supports any mark, as long as it has a conversion method to its `%docu` mark. The `%docu` mark is not expected to be used directly to write documentation, its purpose is to be a mark conversion target.

The `%docu` mark expects a `$manx`.

A `$manx` is how an XML node structure is represented in hoon. See [Section 5e](/hoon/stdlib/5e#manx) of the standard library reference for details. A `$manx` is what `+de-xml:html` and `+en-xml:html` decode/encode raw XML strings from/to.

The `%docu` mark will technically accept any `$manx`, but the `%docs` agent itself makes some changes and imposes some additional rules:

* The root element must be a `<div>`.
* `<h1>`, `<h2>`, and `<h3>` elements that are direct children of the root `<div>` will be used to make the table of contents. Other header levels will not be included in the table of contents, but they can still be used. `<h1>`, through `<h3>` can also be used at deeper levels, but they also won't be included in the table of contents.
* Only these tags are allowed: `<a>`, `<address>`, `<b>`, `<br>`, `<blockquote>`, `<code>`, `<del>`, `<div>`, `<em>`, `<h1>`, `<h2>`, `<h3>`, `<h4>`, `<h5>`, `<h6>`, `<hr>`, `<i>`, `<img>`, `<ins>`, `<li>`, `<ol>`, `<p>`, `<pre>`, `<q>`, `<small>`, `<span>`, `<strong>`, `<sub>`, `<sup>`, `<time>`, `<ul>`, `<var>`.
* Inside the `<h1>`, `<h2>`, and `<h3>` headers that are direct children of the root `<div>` (and will therefore be used in the table of contents), only a subset of the tags listed above are allowed: `<b>`, `<code>`, `<del>`, `<em>`, `<i>`, `<ins>`, `<q>`, `<small>`, `<span>`, `<strong>`, `<sub>`, `<sup>`, `<time>`, `<var>`.
* All attributes will be stripped from all elements (you can still include them but they'll be removed), with the following exceptions:
  * The `src` and `alt` attributes in an `<img>` tag.
  * The `href` attribute in an `<a>` tag.
  * A `class` attribute in a `<pre>` tag beginning with `language-` (e.g. `class="language-hoon"`). This is not currently used for anything but will be used for syntax highlighting in the future.

{% hint style="info" %}
**Note**: table elements are not currently supported but will likely be added in a future release.
{% endhint %}

***

## Included marks <a href="#included-marks" id="included-marks"></a>

The following marks are supported by the %docs app and you can use them to write docs right away.

### `%udon` <a href="#udon" id="udon"></a>

Udon is a markdown-like language native to hoon, with a parser built into the hoon compiler. Here is its syntax in brief:

* The first line of the `.udon` document **must** be a single rune: `;>`. This tells the compiler to interpret everything following as udon.
* **Paragraphs**: Content on a single line will be made into a paragraph. Paragraphs may be hard-wrapped, so consecutive lines of text will become a single paragraph. The paragraph will be ended by an empty line or other block element.
* **Headers**: lines beginning with 1-6 `#`s followed by a single space and then some content (e.g. `## foo`) will be made into headers. The number of `#`s dictates the header level.
* **Italics**: content wrapped in single `_`s (e.g. `_foo_`) will be made italic.
* **Bold**: content wrapped in single `*`s (e.g. `*foo*`) will be made bold.
* **Unordered lists**: lines beginning with `-` followed by a space will be made into items in a list. List lines can be hard-wrapped, with two spaces beginning each subsequent line to be included in the list. Lists can be nested by indenting the `-`s a further two spaces for each level of nesting.
* **Ordered lists**: lines beginning with `+` followed by a space will be made into ordered lists, and numbered in the order they appear. These have the same wrapping and nesting logic as unordered lists.
* **Links**: this is standard markdown syntax: square bracks containing the display content and then parentheses containing the URL, e.g. `[foo](http://example.com)`. The URL may also be a relative link or an anchor link.
* **Images**: this is also standard markdown; a link with an exclamation mark at the beginning, e.g. `![foo](http://example.com/image.png)`. The square brackets contain the alt-text and the the parentheses contain the image URL.
* **Inline code**: text wrapped in single backticks will be rendered verbatim in a monospace font.
* **Fenced codeblocks**: Triple-backticks on their own line begin and end a codeblock. All lines in between will be rendered verbatim in a monospace font. Note that udon does not support a language specification after the opening backticks like markdown does.
* **Horizontal rules**: Three or more hyphens (`---`) will create a horizontal rule.
* **Block quotes**: a line beginning with `>` creates a block quote. This may be hard-wrapped, as long as the next line is indented two spaces. Block quotes may contain anything, including other blockquotes.
* **Line breaks**: A line ending in a single backslash will have a line break inserted at the end, so it will not flow together with the subsequent line as is usually the case.
* **Escape characters**: You may prefix Udon syntax with a backslash to have it treated as the literal text.
* **Hoon constants**: Udon will automatically render any values with atom aura syntax as inline code. It'll also render arms like `+foo:bar`, `$baz`, and `+*foo:bar:baz`, as inline code.
* **Sail**: this is hoon's native XML syntax. Udon will parse it, execute it, and include the `$manx`es produced in the resulting document. This means you can embed arbitrary hoon in the document. There is little formal sail documentation, but you can refer to the [`;` (mic) rune reference](/hoon/rune/mic) for most of its runes and some rudimentary examples.

{% hint style="info" %}
**Note**: Udon is quite strict on its syntax, and may fail to parse if it's incorrect.
{% endhint %}

### `%txt` <a href="#txt" id="txt"></a>

The `%docs` app supports plain `.txt` files. The file will be rendered as a preformatted codeblock with wrapping.

### `%html` <a href="#html" id="html"></a>

Ordinary HTML files may be used, but note the tag and structural restrictions described in the `%docu` mark description above.

### `%gmi` <a href="#gmi" id="gmi"></a>

Gemtext is an ultra-minimal markup format developed for the [Gemini project](https://en.wikipedia.org/wiki/Gemini_\(protocol\)), an internet protocol for servinglight-weight hypertext, inspired by Gopher. Its file extension is `.gmi`.

Gemtext interprets things on a line-by-line basis, and does not support different types on a single line. Every line is a separate element, with the exception of fenced codeblocks which may span multiple lines. In brief, here is the syntax:

* **Paragraphs**: Plain text on a single line constitutes a paragraph. Note hard-wrapping is not supported.
* **Links**: lines beginning with `=>` followed by a space create a link. After the space, the target URL is given. After the URL, there may optionally be a space and then some display text for the link. If no displace text is given, the URL itself will be displayed.
* **Codeblocks**: triple-backticks at the beginning of a line begin and end a codeblock. All text in between will be rendered verbatim in a monospace font. The opening backticks may optionally be followed by some text, which will be used as the language tag.
* **Headings**: 1-3 `#`s followed by text create a heading. The number of `#`s determine the heading level.
* **Lists**: lines beginning with `*` followed by a space and then text will create a list item.
* **Quotes**: lines beginning with `>` followed by a space creates a blockquote.


# Index File

Configuration guide for doc.toc index files in the %docs app, covering structure definition, hierarchical organization, file/directory specification, and automatic table of contents generation.

Each desk with docs may include a `doc.toc` file in its root. A `doc.toc` file looks something like this:

```
/dev                  Developer
  /hark-store         Hark-store
    /overview/udon    Overview
    /types/udon       Data Types
    /pokes/udon       Pokes
    /paths/udon       Subscription Paths
    /scry/udon        Scry Endpoints
  /settings-store     Settings-store
    /overview/udon    Overview
    /types/udon       Data Types
    /pokes/udon       Pokes
    /paths/udon       Subscription Paths
    /scry/udon        Scry Endpoints
```

Each line contains a pair of path and title with at least one space in between. A path with a single element (such as `/dev` and `/hark-store`) denotes a directory. A path with two elements (such as `/overview/udon`) denotes a file, where the first element is the name and the second is the mark.

Any directory structure and depth is allowed. The hierarchy is determined by indentation. Each double-space indent denotes a nesting level. Two spaces is one level, four spaces is two levels, etc.

The root is the `/doc` directory of your desk, it should not be explicitly listed.

The title specified will be displayed at the top of the document and in the table of contents for the desk.

If no `doc.toc` file is included, but a desk does have other files under `/doc`, the Docs app will infer a table of contents from the directory structure, and *all* files will be included. Files are ordered alphabetically, except that 'overview' always come first. For titles, hyphens in the filename are replaced with spaces and the whole is converted to Title Case.


# Suggested Structure

Best practices for organizing documentation in the %docs app, covering user documentation, developer API references, changelog structure, and data type documentation patterns.

Here we'll discuss recommendations and best practices for your docs. The %docs app doesn't impose any restrictions on the structure of your docs, so you're free to do things differently if you feel it's appropriate.

## User docs <a href="#user-docs" id="user-docs"></a>

For the most part, user documentation depends on the nature of your app, so it's up to you how you document it. There are, however, a couple of things that are advisable to include:

### General info <a href="#general-info" id="general-info"></a>

You should include some of the following information (as appropriate) in either a distinct informational document or at the top of the first document:

* **Publisher/developer/contributors**: Your `@p`, the `@p` you distribute the app from, people who have helped build the app, etc.
* **License**: MIT, GPL, etc.
* **Source**: Github link (or similar) for the repo containing the source code.
* **Issues**: Where to file issues (typically also the Github repo).
* **Group**: If you have a group associated with your app (for support, discussion, or what have you), you can include the `~host/group-name`.
* Any other metadata, links, info, etc, you feel are appropriate to include.

### Changelog <a href="#changelog" id="changelog"></a>

Another useful document is a changelog. For each version you release, you can add a new section at the top with the version number, release date, and a brief summary of the changes.

***

## Developer docs <a href="#developer-docs" id="developer-docs"></a>

Like user docs, you're free to document things as you see fit, but it's helpful to include particular things with a standard format. These are documented below.

{% hint style="info" %}
**Note**: If your desk has a dev version, it's advisable to link to it somewhere sensible in the developer documentation.
{% endhint %}

### API Reference <a href="#api-reference" id="api-reference"></a>

If there's any chance other developers will want to interface with agents on your desk, you should include reference information about the scry endpoints, pokes, subscription paths, and (if appropriate) data types of each agent.

These things can either be on a single page, or each have their own separate page, depending on their complexity.

#### Scry endpoints

Scry endpoints may either be organised by their paths or by short summaries of each endpoint's purpose. Scry paths should be in the format `/[care]/rest/of/path`, like `/x/foo/bar`. Any variable parts of the scry path should have square brackets like `/x/foo/[ship]`. The data type of the response should be specified, and it's helpful to include an example of a scry and the pretty-printed response in a codeblock.

#### Pokes

Any poke actions an agent will accept should be documented. It's helpful to include an example poke. You may also wish to provide a JSON example if people can interact with your agent through Eyre's channel system or similar.

If your agent's pokes are organised around something like an "action" tagged union structure, the poke reference might work better as a kind of data type reference for that action structure rather than specifically pokes.

#### Subscriptions

Subscription paths should be documented in a similar manner to the scry reference. If the updates your agent can send out to subscribers are organised around something like an "update" tagged union structure, the subscription reference might work better as just a brief path reference, and documentation of the updates can be done separately.

#### Data types

The inclusion of a data types section depends on the complexity of an agent's data types. If such a reference is appropriate, it's best to structure it with a separate section for each type. The type definition should be included in a codeblock, as well as a brief description of its purpose.


# Userspace

Guides for building and distributing userspace applications on Urbit, including examples, exercises, and tutorials for app development.

Gall is one of the ten vanes (kernel modules) of Arvo, Urbit's operating system. Gall's purpose is to manage userspace applications called *agents*. **Agents** are the main kind of userspace application on Urbit. They have a persistent state and API that handles events and produces effects.Gall agents can variously be treated as databases with developer-defined logic, services, daemons, or a kind of state machine.

One or more Gall agents can be put together in a "desk" in Clay (the filesystem vane) and, along with a front-end, can be published and distributed as a cohesive app for users to install.

This section of the docs contains guides related to writing apps and [distributing them](/build-on-urbit/userspace/dist/software-distribution). It also contains a number of different [examples and exercises](/build-on-urbit/userspace/examples) you can work through to learn how to build different kinds of apps.

Additionally, the [courses](/build-on-urbit/contents) section of the docs contain two in-depth app-building tutorials, which are the recommended place to start learning Urbit app development:

* [App School](/build-on-urbit/app-school) - Learn the basics of app development.
* [App School II](/build-on-urbit/app-school-full-stack) - Learn to build and publish a full app, back-end and front-end.


# Command-Line App Tutorial

Complete tutorial for building CLI applications using /lib/shoe.hoon. Command parsing, session management, the %shoe example app.

## Introduction <a href="#introduction" id="introduction"></a>

In this walkthrough we will go in-depth on how to build command line interface (CLI) applications in Urbit using `/lib/shoe.hoon` in the `%base` desk.

There are three CLI apps that currently ship with Urbit: `%dojo`, `%chat-cli`, and `%shoe`. You should be familiar with the former two, the latter is an example app that shows off how the Shoe library works that we will be looking at closely. These are all Gall apps, and their source can be found in the `/app` folder of your `%base` desk.

In [the Shoe library](#the-shoe-library) we take a closer look at the Shoe library and its cores and how they are utilized in CLI apps. Then in [the sole library](#the-sole-library) we look at what Shoe effects ultimately break down into. Finally in [`%shoe` app walkthrough](#shoe-app-walkthrough) we explore the functionality of the `%shoe` agent and then go through the code line-by-line.

This tutorial can be considered to be an application equivalent of the [Hoon school lesson](/build-on-urbit/hoon-school/p-stdlib-io#ask-generators) on `%sole` and `%ask` generators, which only covers the bare minimum necessary to write generators that take user input.

## The Shoe library <a href="#the-shoe-library" id="the-shoe-library"></a>

Here we describe how sessions are identified, the specialized `$card:shoe`s that Gall agents with the Shoe library are able to utilize, and the different cores of `/lib/shoe.hoon` and their purpose.

### Session identifiers <a href="#session-identifiers" id="session-identifiers"></a>

An app using the Shoe library will automatically track sessions by their `$sole-id`. A `$sole-id` is made up of a "host" ship name and some identifier string (`[who=@p ses=@ta]`). Generally, however, the `$sole-id` as a whole should be treated as an opaque identifier, generated by the connecting client. An app using the Shoe library may be connected to by a local or remote ship in order to send commands, and each of these connections is assigned a unique `$sole-id` that identifies the ship and which session on that ship if there are multiple.

### `%shoe` `$card`s <a href="#shoe-cards" id="shoe-cards"></a>

Gall agents with the Shoe library are able to utilize `$card:shoe`s. These additions to Gall's standard set of `$card:agent:gall`s have the following shape:

{% code title="/lib/shoe.hoon" %}

```hoon
[%shoe sole-ids=(list sole-id) effect=shoe-effect]
```

{% endcode %}

The Shoe card's `sole-ids` is the list of session ids that the following `$shoe-effect` is emitted to. An empty `sole-ids` list sends the effect to all connected sessions. `$shoe-effect`s, for now, are always of the shape `[%sole effect=sole-effect]`, where `$sole-effect`s are basic console events such as displaying text, changing the prompt, beeping, etc. These are described in the section on the [sole library](#the-sole-library).

For example, a `$card:shoe` that causes all connected sessions to beep would be `[%shoe ~ %sole %bel ~]`.

### Shoe core <a href="#shoe-core" id="shoe-core"></a>

An iron (contravariant) door that defines an interface for Gall agents utilizing the Shoe library. Use this core whenever you want to receive input from the user and run a command. The input will get put through the parser (`+command-parser`) and results in a noun of `$command-type` that the underlying application specifies, which Shoe then feeds back into the underlying app as an `+on-command` callback.

In addition to the ten arms that all Gall core apps possess, `+shoe` defines and expects a few more, tailored to common CLI logic. Thus you will need to wrap the `+shoe:shoe` core using the `+agent:shoe` function to obtain a standard 10-arm Gall agent core. See the [Shoe example app walkthrough](#shoe-example-app-walkthrough) for how to do this.

The additional arms are described below. The Hoon code shows their expected type signature. As we'll see [later](#shoe-app-walkthrough), the `$command-type` can differ per application. Note also that most of these take a session identifier as an argument. This lets applications provide different users (at potentially different "places" within the application) with different affordances.

#### `+command-parser`

{% code title="/lib/shoe.hoon" %}

```hoon
  ++  command-parser
    |~  =sole-id
    |~(nail *(like [? command-type]))
```

{% endcode %}

Input parser for a specific command-line session. Will be run on whatever the user tries to input into the command prompt, and won't let them type anything that doesn't parse. If the head of the result is true, instantly run the command. If it's false, require the user to press return.

#### `+tab-list`

{% code title="/lib/shoe.hoon" %}

```hoon
  ++  tab-list
    |~  =sole-id
    *(list (option:auto tank))
```

{% endcode %}

Autocomplete options for the command-line session (to match `+command-parser`).

#### `+on-command`

{% code title="/lib/shoe.hoon" %}

```hoon
  ++  on-command
    |~  [=sole-id command=command-type]
    *(quip card _this)
```

{% endcode %}

Called when a valid command is run.

#### `+can-connect`

{% code title="/lib/shoe.hoon" %}

```hoon
  ++  can-connect
    |~  =sole-id
    *?
```

{% endcode %}

Called to determine whether a session may be opened or connected to. For example, you may only want the local ship to be able to connect.

#### `+on-connect`

{% code title="/lib/shoe.hoon" %}

```hoon
  ++  on-connect
    |~  =sole-id
    *(quip card _^|(..on-init))
```

{% endcode %}

Called when a session is opened or connected to.

#### `+on-disconnect`

{% code title="/lib/shoe.hoon" %}

```hoon
  ++  on-disconnect
    |~  =sole-id
    *(quip card _^|(..on-init))
```

{% endcode %}

Called when a previously made session gets disconnected from.

### `+default` core <a href="#default-core" id="default-core"></a>

This core contains the bare minimum implementation of the additional Shoe arms beyond the 10 standard Gall app ams. It is used analogously to how the `+default` agent core is used for regular Gall apps.

### `+agent` core <a href="#agent-core" id="agent-core"></a>

This is a function for wrapping a Shoe core, which has too many arms to be a valid Gall agent core. This turns it into a standard Gall agent core by integrating the additional arms into the standard ones.

## The Sole library <a href="#the-sole-library" id="the-sole-library"></a>

Shoe apps may create specialized `$card:shoe`s of the `[%shoe (list sole-id) shoe-effect]` shape, where `$shoe-effect` currently just wrap `$sole-effect`s, i.e. instructions for displaying text and producing other effects in the console.

The list of possible `$sole-effect`s can be found in `/sur/sole.hoon`. A few commonly used ones are as follows.

* `[%txt tape]`: display a line of text.
* `[%bel ~]`: emit a beep.
* `[%pro sole-prompt]`: set the prompt.
* `[%mor (list sole-effect)]`: emit multiple effects.

For example, a `$sole-effect` that displays `This is some text.` and beeps would be structured as

```hoon
[%mor [%txt "This is some text."] [%bel ~] ~]
```

## `%shoe` app walkthrough <a href="#shoe-app-walkthrough" id="shoe-app-walkthrough"></a>

Here we explore the capabilities of the `%base` desk's example Shoe agent `/app/shoe.hoon`, and then go through the code, explaining what each line does.

### Playing with `%shoe` <a href="#playing-with-shoe" id="playing-with-shoe"></a>

First let's test the functionality of the `%shoe` agent so we know what we're getting into.

Start two fake ships, one named \~zod. The other may have any name, let's go with \~nus. Fake ships run locally are able to see each other, and our intention is to connect their `%shoe` apps.

On each fake ship start `%shoe` by entering `|start %shoe` into their Dojos. Now that the agents are running, you can run `|dojo/link %shoe` to connect to the CLI interface. This will change the prompt to `~zod:shoe>` and `~nus:shoe>`. Type `demo` and watch the following appear:

{% code title="Dojo" %}

```
~zod ran the command
~zod:shoe>
```

{% endcode %}

`~zod ran the command` should be displayed in bold green text, signifying that the command originated locally.

Now we will connect the sessions. Switch \~zod back to dojo by pressing `Ctrl+X` and enter `|dojo/link ~nus %shoe`. If this succeeds you will see the following.

{% code title="Dojo" %}

```
>=
; ~nus is your neighbor
[linked to [p=~nus q=%shoe]]
```

{% endcode %}

Now \~zod will have two `%shoe` sessions running - one local one on \~zod and one remote one on \~nus, which you can access by pressing `Ctrl+X` until you see `~nus:shoe>` from \~zod's console. On the other hand, you should not see `~zod:shoe>` on \~nus's side, since you have not connected \~nus to \~zod's `%shoe` agent. When you enter `demo` from `~nus:shoe>` on \~zod's console you will again see `~zod ran the command`, but this time it should be in the ordinary font used by the console, signifying that the command is originating from a remote session. Contrast this with entering `demo` from `~nus:shoe>` in \~nus's console, which will display `~nus ran the command` in bold green text.

Now try to link to \~zod's `%shoe` session from \~nus by switching to the dojo on \~nus and entering `|dojo/link ~zod %shoe`. You should see

{% code title="Dojo" %}

```
>=
[unlinked from [p=~zod q=%shoe]]
```

{% endcode %}

And if you press `Ctrl+X` you will not get a `~zod:shoe>` prompt. This is because the example app is set up to always allow \~zod to connect (as well as subject moons if the ship happens to be a planet) but not \~nus, so this message means that \~nus failed to connect to \~zod's `%shoe` session.

### `%shoe`'s code <a href="#shoes-code" id="shoes-code"></a>

{% code title="/app/shoe.hoon" %}

```hoon
::  shoe: example usage of /lib/shoe
::
::    the app supports one command: "demo".
::    running this command renders some text on all sole clients.
::
/+  shoe, verb, dbug, default-agent
```

{% endcode %}

`/+` is the Ford rune which imports libraries from the local desk's `/lib` directory into the subject.

* `shoe`: the Shoe library.
* `verb`: a library used to print what a Gall agent is doing.
* `dbug`: a library of debugging tools.
* `default-agent`: a Gall agent core with minimal implementations of required Gall arms.

{% code title="/app/shoe.hoon" %}

```hoon
|%
+$  state-0  [%0 ~]
+$  command  ~
::
+$  card  card:shoe
--
```

{% endcode %}

The types used by the app.

`$state-0` stores the state of the app, which is `~` (null) as there is no state to keep track of. It is good practice to include a version number anyways, in case the app is made stateful at a later time.

`$command` is typically a set of tagged union types that represent the possible commands that can be entered by the user. Since this app only supports one command, it is unnecessary for it to have any associated data, thus the command is represented by `~`.

In a non-trivial context, a `$command` is commonly given by a `$page` (`(pair term noun)`), where the `$term` is the identifier for the type of command and the `$noun` is any type or list of types that contain the data needed to execute the command. See `/app/chat-cli.hoon` for examples of commands, such as `[%say letter:store]` and `[%delete path]`. This is not required though, and you could use something like `[chat-room=@t =action]`.

`$card:shoe` is either an ordinary `$card:agent:gall` or a Shoe card, which takes the shape `[%shoe sole-ids=(list sole-id) effect=shoe-effect]`. A Shoe card is sent to all sessions listed in its `sole-ids`, making them run its `$sole-effect` (e.g. printing some text). Here we can reference `$card:shoe` because of `/+ shoe` at the beginning of the agent file.

{% code title="/app/shoe.hoon" %}

```hoon
=|  state-0
=*  state  -
::
```

{% endcode %}

The above adds the bunt (default value) of `$state-0` to the head of the subject, then give it the macro `state`. The `-` here is a lark expression referring to the head of the subject. This allows us to use `state` to refer to the state elsewhere in the code no matter what version we're using, while also getting direct access to the contents of `.state` (if it had any).

{% code title="/app/shoe.hoon" %}

```hoon
%+  verb  |
%-  agent:dbug
^-  agent:gall
%-  (agent:shoe command)
^-  (shoe:shoe command)
```

{% endcode %}

The `^-` type casts here are just reminders of what is being produced. So let's focus on what the `%` runes are doing, from bottom to top. We call the `(agent:shoe command)` function on what follows (in this case, the rest of the app), producing a standard Gall agent core. Then we call wrap the Gall agent core with `+agent:dbug`, endowing it with additional arms useful for debugging, and then wrap again with `+verb`.

{% code title="/app/shoe.hoon" %}

```hoon
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %|) bowl)
    des   ~(. (default:shoe this command) bowl)
```

{% endcode %}

This is boilerplate Gall agent core code. We set `.this` to be a macro for the subject, which is the Gall agent core itself. We set `def` and `des` to be macros for initialized `+default-agent` and `+default:shoe` doors respectively.

Next we implement all of the arms required for a Shoe agent. Starting with the standard Gall arms:

{% code title="/app/shoe.hoon" %}

```hoon
++  on-init   on-init:def
++  on-save   !>(state)
++  on-load
  |=  old=vase
  ^-  (quip card _this)
  [~ this]
::
++  on-poke   on-poke:def
++  on-watch  on-watch:def
++  on-leave  on-leave:def
++  on-peek   on-peek:def
++  on-agent  on-agent:def
++  on-arvo   on-arvo:def
++  on-fail   on-fail:def
```

{% endcode %}

These are minimalist Gall app arm implementations using the default behavior found in `def`.

Here begins the implementation of the additional arms required by the `(shoe:shoe command)` function.

{% code title="/app/shoe.hoon" %}

```hoon
++  command-parser
  |=  =sole-id
  ^+  |~(nail *(like [? command]))
  (cold [& ~] (jest 'demo'))
```

{% endcode %}

`+command-parser` is of central importance - it is what is used to parse user input and transform it into `$command`s for the app to execute. Writing a proper command parser requires understanding of the Hoon parsing functions found in the standard library. How to do so may be found in the [parsing tutorial](/hoon/parsing). For now, it is sufficient to know that this arm matches the text "demo" and produces a `[? command]`-shaped noun in response. Note how the `&` signifies that the command will be run as soon as it has been entered, without waiting for the user to press return.

{% code title="/app/shoe.hoon" %}

```hoon
++  tab-list
  |=  =sole-id
  ^-  (list [@t tank])
  :~  ['demo' leaf+"run example command"]
  ==
```

{% endcode %}

`+tab-list` is pretty much plug-and-play. For each command you want to be tab completed, add an entry to the list begun by `:~` of the form `[%command leaf+"description"]`. Now whenever the user types a partial command and presses tab, the console will display the list of commmands that match the partial command as well as the descriptions given here.

Thus here we have that starting to type `demo` and pressing tab will result in the following output in the console...

{% code title="Dojo" %}

```
demo  run example command
~zod:shoe> demo
```

{% endcode %}

...with the remainder of `demo` now added to the input line.

Next we have `+on-command`, which is called whenever `+command-parser` recognizes that `demo` has been entered by a user.

{% code title="/app/shoe.hoon" %}

```hoon
++  on-command
  |=  [=sole-id =command]
  ^-  (quip card _this)
```

{% endcode %}

This is a gate that takes in the `$sole-id` corresponding to the session and the `command` noun parsed by `+command-parser` and returns a list of `$card:shoe`s and `_this`, which is our shoe agent core including its state.

{% code title="/app/shoe.hoon" %}

```hoon
  =-  [[%shoe ~ %sole -]~ this]
```

{% endcode %}

This creates a cell of a `$card:shoe` that triggers a `$sole-effect` given by the head of the subject `-`, then the Gall agent core `.this` - the return result of this gate. The use of the `=-` rune means that what follows this expression is actually run first, which puts the desired `$sole-effect` into the head of the subject.

{% code title="/app/shoe.hoon" %}

```hoon
  =/  =tape  "{(scow %p src.bowl)} ran the command"
```

{% endcode %}

We define the `$tape` that we want to be printed.

{% code title="/app/shoe.hoon" %}

```hoon
  ?.  =(src our):bowl
    [%txt tape]
  [%klr [[`%br ~ `%g] [(crip tape)]~]~]
```

{% endcode %}

We cannot just produce the `$tape` we want printed, - it needs to fit the `$sole-effect` type. This tells us that if the origin of the command is not our ship to just print it normally with the `%txt` `$sole-effect`. Otherwise we use `%klr`, which prints it stylistically (here it makes the text green and bold).

The following allows either \~zod, or the host ship and its moons, to connect to this app's command line interface using `|dojo/link`.

{% code title="/app/shoe.hoon" %}

```hoon
++  can-connect
  |=  =sole-id
  ^-  ?
  ?|  =(~zod src.bowl)
      (team:title [our src]:bowl)
  ==
```

{% endcode %}

We use the minimal implementations for the final two Shoe arms, since we don't want to do anything special when users connect or disconnect.

{% code title="/app/shoe.hoon" %}

```hoon
++  on-connect      on-connect:des
++  on-disconnect   on-disconnect:des
--
```

{% endcode %}

This concludes our review of the code of the `%shoe` app.

To continue learning how to build your own CLI app, we recommend checking out [`%chat-cli`](https://github.com/urbit/urbit/blob/759c28f772aeeb7581c4804b2e58159d7d22ff86/pkg/landscape/app/chat-cli.hoon).


# Remote Scry

Remote scrying between ships since kernel v413. Publishing with %grow, encryption options, access control, %keen/%chum/%tune lifecycle.

To scry is to perform a *read* from Urbit's referentially transparent namespace. In other words, it's a function from a `$path` to a `$noun` (although in some cases, the resulting type may be more constrained). Previously we only supported scrying within the same ship, but from Kernel version `[%zuse 413]`, it is possible to scry from *other* ships.

## Lifecycle of a scry <a href="#lifecycle-of-a-scry" id="lifecycle-of-a-scry"></a>

When you think of scry, you probably think of `.^` [dotket](/hoon/rune/dot#dotket). However, since networking is asynchronous, this is not a suitable interface for remote scry. Instead, a ship that wants to read from a remote part of the namespace will have to (directly or indirectly) ask Ames to perform the scry, which then cooperates with Vere to produce the desired data. In some future event when the result is available, Ames gives it back as a `%tune` gift. From the requester's perspective, this is the entire default lifecycle of a remote scry request.

Of course, you need to know how Ame's `%chum` and `%tune` look, as well as Gall's `%keen` note, to be able to use them. There are also a few exceptions to this default lifecycle. We'll go through all of this in a moment, but first, let's look at what kind of data is possible to scry.

## Publishing <a href="#publishing" id="publishing"></a>

At the moment, there are two vanes that can handle remote scry requests: Clay and Gall. Clay uses it to distribute source code in a more efficient manner than is possible with conventional Ames, but conceptually it only extends its [local scries](/urbit-os/kernel/clay/scry) over the network, with the notable difference that you can't scry at the *current* time, since the requester doesn't know when the request reaches the publisher. Additionally, the paths are modified so that the vane and care are specified separately, like so: `/c/x/1/base/sys/hoon/hoon`.

Gall is more interesting. First, let's clear up a possible misunderstanding that could easily come up: remote scry does *not* involve calling an agent's `+on-peek` arm. `+on-peek` scries always happen at the current time, and since the requester can't know at which time the publisher handles the request, these aren't possible to reliably serve.

Instead, agents *ask* Gall to `%grow` nouns to paths in the namespace on their behalf, and Gall stores the data in *its* state (not in the agent's state). Gall will take care of incrementing version numbers, so that the same path never maps to different nouns. The agent can also ask Gall to delete data, either at a specific version number, or everything up to and including a version number.

{% hint style="info" %}
Note: we'll only discuss the basic case of unencrypted and two-party encrypted scries here. Gall also supports multi-party encrypted scries with access control, which we'll look at in the next section.
{% endhint %}

`$note:agent:gall` includes the following cases:

{% code title="/sys/lull.hoon" %}

```hoon
+$  note
  $%  ::  ...
      [%grow =spur =page]  ::  publish
      [%tomb =case =spur]  ::  delete one
      [%cull =case =spur]  ::  delete up to
      ::  ...
  ==
```

{% endcode %}

Here's an example sequence of cards that use these:

```hoon
[%pass /call/back/path %grow /foo atom+'lorem']  ::  /foo version 0
[%pass /call/back/path %grow /foo atom+'ipsum']  ::  /foo version 1
[%pass /call/back/path %grow /foo atom+'dolor']  ::  /foo version 2
[%pass /call/back/path %grow /foo atom+'sit']    ::  /foo version 3

[%pass /call/back/path %tomb ud+3 /foo]          ::  delete /foo version 3
[%pass /call/back/path %cull ud+1 /foo]          ::  delete /foo 0 through 1

[%pass /call/back/path %grow /foo atom+'amet']   ::  /foo version 4
[%pass /call/back/path %grow /foo/bar atom+123]  ::  /foo/bar version 0
```

After this sequence of cards we would have the following mappings (assuming the agent that emits them is named `%test`):

```
/g/x/2/test//foo     -> [%atom 'dolor']
/g/x/4/test//foo     -> [%atom 'amet']
/g/x/0/test//foo/bar -> [%atom 123]
```

Let's pick apart the first one of these paths.

```hoon
/g     ::  g for Gall
/x     ::  a care of %x generally means "normal read"
/2     ::  version number
/test  ::  the agent that published the data
/      ::  ???
/foo   ::  the path that the data is published on
```

What's that lone `/` before the path? It signifies that this data is published by *Gall* itself, instead of the `+on-peek` arm in the `%test` agent. As part of the remote scry release, we have *reserved* part of the scry namespace for Gall, effectively *preventing* any agents from directly publishing at those paths. Though as we've seen, they can do it indirectly, by asking Gall to do it for them using `%grow`.

As long as the extra `/` is included, Gall will serve scries with care `%x` at both specific revision numbers and at arbitrary times. If the extra `/` is not included, the scry has to happen at the current time, since we don't cache old results of calling `+on-peek`.

### Additional Gall cares <a href="#additional-gall-cares" id="additional-gall-cares"></a>

Apart from supporting reads using the `%x` care, Gall now also supports three new cares:

* `%t` lists all subpaths that are bound under a path (only supported at the current time, i.e. not remotely!).
* `%w` gives the latest revision number for a path (only supported at the current time, i.e. not remotely!).
* `%z` gives the hash identifier of the value bound at the path (supported at any time and at specific revisions, but not remotely).

All of these require the extra `/` to be present in the path, just as with `%x`.

## Encryption <a href="#encryption" id="encryption"></a>

As well as ordinary unencrypted scries, Ames also supports two-party and multi-party encrypted scries. Two-party encryption doesn't require any additional steps on the publisher's side, but multi-party encryption does:

1. A *security context* must be created.
2. You must implement an access-control scry handler for that security context in the `+on-peek` arm.
3. Data must be published to that security context.

A security context is called a `$coop`, which is just a path of your choosing, like `/foo/bar/baz`.

`$note:agent:gall` includes the following two `$note`s for managing security contexts and publishing data to them:

```hoon
$%  ...
    [%tend =coop =path =page]
    [%germ =coop]
    ...
==
```

#### `%germ`

{% code title="/sys/lull.hoon" %}

```hoon
[%germ =coop]
```

{% endcode %}

The `%germ` note creates the *security context* specified in the `$coop`. It's just a path of your choice, like `/foo/bar/baz`. Once created, you can publish data to it with a [`%tend`](#tend) note.

Example:

```hoon
[%pass /call/back/path %germ /foo/bar/baz]
```

#### `%tend`

{% code title="/sys/lull.hoon" %}

```hoon
[%tend =coop =path =page]
```

{% endcode %}

The `%tend` note publishes the given `$page` to the given path in the given `$coop` security context. This is the same as a `%grow` note, just with the addition of the security context. The only difference is that access is limited to those allowed in the `$coop`.

### Access control <a href="#access-control" id="access-control"></a>

For each security context created with the `%tend` task described above, the `+on-peek` arm of the agent should provide a scry handler for it, to decide whether a ship is allowed to access the resource or not. The scry path looks like:

```hoon
/c/your/security/context/~sampel-palnet
```

It has a `%c` `$care`, the security context (in this case `/your/security/context`), and then the ship in question (\~sampel-palnet). It must return a `?` boolean in a `%noun` mark which is true if the ship is allowed to access that security context, and false if not. How you determine whether a ship is allowed is up to you. Here's a trivial example:

```hoon
+  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ?.  ?=([%c %your %security %context @ ~] path)
    ~
  =/  =ship  (slav %p i.t.t.t.t.path)
  ?:  =(~dinleb-rambep ship)  :: your whitelist logic here
    ``[%noun !>(%.y)]
  ``[%noun !>(%.n)]
```

Note this is unnecessary for unencrypted and two-party encrypted remote scries, only for files you publish in a security context with the [`%tend`](#tend) note.

## Scrying <a href="#scrying" id="scrying"></a>

Now we've looked at the publisher side, let's look at actually performing remote scries.

It's not currently possible to perform a remote scry on yourself, unless your ship has an Ames route to itself which is an edge-case. If you need to perform a dotket scry on a remote scry path, you can do so with this syntax.

{% code title="Dojo" %}

```
.^(* %gx /=/test/1//1/foo/bar)
```

{% endcode %}

{% hint style="info" %}
Notice the `//` empty path element differentiating an agent scry from a Gall vane scry.

Additionally, notice the `1` after the `//`. This is a path format version number introduced in `[%zuse 411]` to facilitate easier path format changes in the future. *All remote scries to Gall agents must include the path format version number.* Scries to places other than Gall agents are unaffected.
{% endhint %}

There is one `$note:agent:gall` for performing unencrypted and multi-party encrypted remote scries, one Ames task for performing two-party encrypted remote scries, and two Ames tasks for cancelling pending remote scries. We'll look at each of these.

### Tasks and Notes <a href="#tasks-and-notes" id="tasks-and-notes"></a>

#### `%keen`

{% code title="/sys/lull.hoon" %}

```hoon
[%keen secret=? spar:ames]
```

{% endcode %}

The `%keen` note performs either an unencrypted scry or a multi-party encrypted scry.

{% hint style="info" %}
Note that this is a `$note:agent:gall`, and is not to be confused with the Ames task of the same name. Under the hood, Gall will still use the `%keen` Ames task, but this way you don't have to deal with encryption keys. You shouldn't use the Ames task directly.
{% endhint %}

The `secret` boolean specifies whether it should be a multi-party encrypted scry or an ordinary unencrypted scry. The `$spar` is a `(pair ship path)`.

For an unencrypted remote scry to read (`%x` care) the `/sys/hoon/hoon` file from the `%base` desk at revision `4` in Clay (`%c`) on the \~sampel ship, it would look like:

```hoon
[%pass /your/wire %keen %.n ~sampel /c/x/4/base/sys/hoon/hoon]
```

For an unencrypted scry to the `%example` agent in Gall (`%g`) of the \~sampel ship at `/foo` path, revision `4`, it would look like:

```hoon
[%pass /your/wire %keen %.n ~sampel /g/x/4/example//1/foo]
```

For a multi-party encrypted scry to the `%example` agent in Gall (`%g`) of the \~sampel ship at the `/foo` path, revision `4` in the `/my/context` security context, it would look like:

```hoon
[%pass /your/wire %keen %.y ~sampel /g/x/4/example//1/my/context/foo]
```

Notice the `/my/context` security context and `/foo` path are combined into a single continuous path.

You will receive a [`%tune`](#tune) gift from Ames with the response once completed.

#### `%chum`

{% code title="/sys/lull.hoon" %}

```hoon
[%chum spar]
```

{% endcode %}

The Ames `%chum` task performs a two-party encrypted remote scry. It behaves exactly the same as an unencrypted remote scry except that it's encrypted. You don't need a security context for this kind of remote scry & an unencrypted `%keen` can be swapped out for this without the publisher having to change any of their app logic. For details of the `$spar` format, see the [`%keen` note entry above](#keen).

Example:

```hoon
[%pass /your/wire %arvo %a %chum ~sampel /g/x/4/example//1/foo]
```

You will receive a [`%tune`](#tune) gift from Ames with the response once completed.

#### `%yawn`

{% code title="/sys/lull.hoon" %}

```hoon
[%yawn spar]
```

{% endcode %}

A `%yawn` Ames task tells Ames that *we're* no longer interest in a response from a pending request to the given `$spar`. Ames uses the `$duct` to determine which requests to cancel, which means the `$wire` must be the same as the original `%chum` task or `%keen` note.

Example:

```hoon
[%pass /call/back/path %arvo %a %yawn ~sampel /g/x/4/test//foo]
```

You will receive a [`%tune`](#tune) gift from Ames with a null `$roar` for any pending requests.

#### `%wham`

{% code title="/sys/lull.hoon" %}

```hoon
[%wham spar]
```

{% endcode %}

A `%wham` task to Ames tells Ames to cancel all pending requests to the given `$spar`, regardless of where it came from on our ship. This will cancel pending requests from other agents or vanes too, so be careful.

Example:

```hoon
[%pass /call/back/path %arvo %a %wham ~sampel /g/x/4/test//foo]
```

Everything on the ship with pending requests to the given `$spar` will receive a [`%tune`](#tune) gift from Ames with a null `$roar`.

### Gifts <a href="#gifts" id="gifts"></a>

There is only one kind of response you can receive from Ames for any kind of remote scry: a [`%tune`](#tune) gift.

#### `%tune`

In response to any kind of remote scry, Ames returns a `%tune` gift, which looks like:

{% code title="/sys/lull.hoon" %}

```hoon
[%tune spar roar=(unit roar)]
```

{% endcode %}

The `$spar` is the `[ship path]` the request was made to, and the `$roar` is the response. The outer `$unit` of `$roar` will be `~` if Ames doesn't have a response, but may have one in the future. Otherwise, it will contain a signature and the data. The data in the [`$roar`](/urbit-os/kernel/ames/data-types#roar) may be `~`, meaning that there is no value at this path and will never be one.

You'll receive a `%tune` whether it failed or succeeded on the target ship, as well as if the request was cancelled locally.

## `-peek` <a href="#peek" id="peek"></a>

In addition to the above interface offered to agents, there is also a thread `/ted/peek.hoon` in the `%base` desk. You can run it from the Dojo to read the `%noun` mark's source code out of \~zod's `%kids` desk.

{% code title="Dojo" %}

```
-peek [[~zod /c/x/1/kids/mar/noun/hoon] ~]
```

{% endcode %}

That trailing `~` specifies the task to use, which depends on the security status: either `[%chum ~]` for a two-party encrypted remote scry, `[%shut idx=@ key=@]` for a multi-party encrypted remote scry, or `~` to do a regular `%keen` to a public path.

If there were a `%test` agent on \~sampel with path `/foo/bar` that you wanted to scry with this thread, you'd do so like this:

{% code title="Dojo" %}

```
-peek [[~sampel /g/x/1/test//1/foo/bar] ~]
```

{% endcode %}

## Additional reading <a href="#additional-reading" id="additional-reading"></a>

* [Gall scry reference](/urbit-os/kernel/gall/scry): Reference documentation of Gall's vane-level and agent-level scry interface.
* [Ames API reference](/urbit-os/kernel/ames/tasks): Reference documentation of tasks that can be passed to Ames, including those for remote scries.


# Unit Tests

Unit testing framework using -test thread and /lib/test.hoon. Write test arms with +test-\* pattern that return $tang.

## Structure <a href="#structure" id="structure"></a>

The `%base` desk includes a `-test` thread which can run unit tests you've written. A test is a Hoon file which produces a core. The `-test` thread will look for any arms in the core whose name begin with `test-`, e.g:

```hoon
|%
++  test-foo
  ::  ...
++  test-bar
  ::  ...
++  test-foo-bar
  ::  ...
--
```

Any arms that don't begin with `test-` will be ignored. Each `+test-*` arm must produce a `$tang` (a `(list tank)`). If the `$tang` is empty (`~`), it indicates success. If the `$tang` is non-empty, it indicates failure, and the contents of the `$tang` is the error message.

To make test-writing easier, the `%base` desk includes the `/lib/test.hoon` library which you can import into your test file. The library contains the following functions, which all produce `$tang`s:

* `+expect-eq` - test whether an expression produces the expected value. This function takes `[expected=vase actual=vase]`, comparing `.expected` to `.actual`.
* `+expect` - test whether an expression produces `%.y`. This function takes a `$vase` containing the result to check.
* `+expect-fail` - tests whether the given `$trap` crashes, failing if it succeeds.
* `+expect-success` - the converse of `+expect-fail`: tests whether the given `$trap` succeeds, failing if it crashes.
* `+expect-fail-message` - takes `[msg=@t a=(trap)]`. Like `+expect-fail`, but also requires that `.msg` appear in the resulting error message, so you can assert on *why* something failed rather than just that it did.
* `+run-chain` - takes an `$a-test-chain` and runs a sequence of tests, stopping at the first failure. Note that arms in the chain should not begin with `test-`, so that `-test %/... ~` does not also run them individually.
* `+category` - this is a utility that prepends an error message to a failed test (non-null `$tang`), passing through an empty `$tang` (successful test) unchanged.

The most commonly used function is `+expect-eq`, which is used like:

```hoon
++  test-foo
  %+  expect-eq
  !>  'the result I expect'
  !>  (function-i-want-to-test 'some argument')
```

Of course, you'll want to test something else you've written rather than just expressions in the test file itself. To do that, you'd just import the file with `/=` or a similar Ford rune, and then call its functions in the test arms. You're free to do any compositions, import types, etc, as long as the file ultimately produces a `$core` with `+test-*` arms.

## Running <a href="#running" id="running"></a>

The `-test` thread takes a `(list path)` in the Dojo, where each path is a path to a test file. The path *must* include the full path prefix (`/[ship]/[desk]/[case]`). The path *may* omit the mark, since a `.hoon` file is assumed. The path *may* include the name of a test arm after the filename. In that case, only the specified test arm will be run.

The conventional location for tests is a `/tests` directory in the root of a desk.

The output of the `-test` thread will note which arms were tested and whether they succeeded. It will also include:

* The number of micro-seconds it took to execute each test arm.
* A `?` specifying whether all tests succeeded.
* A message confirming the file was built successfully.

Here's an example of running the tests for `/lib/naive.hoon`:

{% code title="Dojo" %}

```
> -test %/tests/lib/naive ~
built   /tests/lib/naive/hoon
>   test-zod-spawn-to-zero: took 81359µs
OK      /lib/naive/test-zod-spawn-to-zero
>   test-zod-spawn-proxy: took 128125µs
OK      /lib/naive/test-zod-spawn-proxy
.............................
....truncated for brevity....
.............................
>   test-approval-for-all: took 647403µs
OK      /lib/naive/test-approval-for-all
>   test-address-padding: took 75104µs
OK      /lib/naive/test-address-padding
ok=%.y
```

{% endcode %}

Here's an example of running just a single test for `/lib/naive.hoon`, the `+test-deposit` arm:

{% code title="Dojo" %}

```
> -test %/tests/lib/naive/test-deposit ~
built   /tests/lib/naive/hoon
>   test-deposit: took ms/45.542
OK      /lib/naive/test-deposit
ok=%.y
```

{% endcode %}


# Software Distribution

Urbit supports peer-to-peer distribution and installation of applications. A user can click on a link to an app published on another ship to install.

Urbit supports peer-to-peer distribution and installation of applications. A user can click on a link to an app published on another ship to install that app. Landscape lets users manage their installed apps and launch their interfaces in new tabs.

This document describes the architecture of Urbit's app distribution system. For a walkthrough of creating and distributing an app, see the [`Guide`](/build-on-urbit/userspace/dist/software-distribution) document.

## Architecture <a href="#architecture" id="architecture"></a>

The unit of software distribution is the desk. A desk is a lot like a git branch, but full of typed files, and designed to work with the Arvo kernel. In addition to files full of source code, a desk specifies the version(s) of the kernel that it's compatible with, and it includes a manifest describing which of its Gall agents should be run by default.

Every desk is self-contained: the result of validating and building its files is a pure function of those files and the standard libraries provided by the running kernel. A desk on one ship will build the same as on any other ship.

For the moment, every live desk must support the running kernel version. Kernel backwards compatibility is an intended feature in the future, but for now, app developers will need to publish an app update supporting new kernels each time they're released.

Each desk defines its own filetypes (called `$mark`s), in its `/mar` folder. There are no shared system marks that all userspace code knows, nor common libraries in `/lib` or `/sur` — each desk is completely self-contained.

It's common for a apps to interact with apps on other desks, and therefore need marks and libraries by the other desk's developer. Since direct cross-desk dependencies are not currently supported, the typical approach is for app developers to publish a "dev" folder on their git repo with the files need for developers interfacing with their app.

Most desks will want to include the `base-dev` folder from the [urbit/urbit](https://github.com/urbit/urbit) repo so they can easily interact with system apps in the `%base` desk.

Landscape apps (those with frontends launched from a tile in Landscape), should also include `desk-dev` from the [Landscape repo](https://github.com/tloncorp/landscape). This folder includes the `%docket-0` mark, which the app needs in order to include a `/desk/docket-0` file.

The `%docket` agent in Landscape reads the `/desk/docket-0` file to display an app tile on the home screen and hook up other front-end functionality, such as downloading the app's client bundle ([glob](/build-on-urbit/userspace/dist/glob)). Docket also manages app installs, serves the home screen, downloads client bundles, and communicates with Kiln to configure the apps on your system.

### Anatomy of a Desk <a href="#anatomy-of-a-desk" id="anatomy-of-a-desk"></a>

Desks contain helper files in `/lib` and `/sur`, generators in `/gen`, marks in `/mar`, threads in `/ted`, tests in `/tests`, and Gall agents in `/app`. In addition, desks also contain these files:

```
/sys/kelvin     ::  Kernel kelvin, e.g. [%zuse 408] (mandatory)
/desk/bill      ::  list of agents to start on install (optional, read by Kiln) 
/desk/docket-0  ::  app metadata (optional, read by Docket) 
/desk/ship      ::  ship of original desk publisher (optional, read by Docket) 
```

Only the `%base` desk contains a `/sys` directory with the standard library and vanes. All other desks simply specify the kernel version(s) they're compatible with in the `/sys/kelvin` file.

### Updates <a href="#updates" id="updates"></a>

Kiln listens to the publisher for any new commits to the desk. If an update is compatible with the current kernel, it will be installed immediately. Any running Gall agents on the desk will be upgraded and if there's a new front-end glob version, it will be fetched. If the update is only compatible with a future kelvin, it will be queued until a compatible kernel upgrade is applied.

A kernel update with a new kelvin version will not be applied until all running apps have compatible updates queued for installation. Users have the option to force the kernel update, which will suspend any incompatible apps. Once they receive a compatible update, they'll automatically be upgraded and started back up. It's therefore important for app publishers to push updates ahead of any new kernel versions. The Urbit Foundation will announce new kernel updates in advance, so app developers have time to update their apps. Often the only necessary change is signalling compatibility with the new version in `sys.kelvin`, but sometimes further changes are necessary. Breaking changes will be included in the release notes of release candidates on the [urbit/urbit](https://github.com/tloncorp/landscape) Github repo.

### Managing Apps and Desks in Kiln <a href="#managing-apps-and-desks-in-kiln" id="managing-apps-and-desks-in-kiln"></a>

The default agents to run are listed in the `/desk/bill` file of the desk. These can be forced off, or additional agents forced on, with commands to Clay.

For details of the generators for managing desks and agents, see the [`Dojo Tools`](/user-manual/os/dojo-tools) document.

### Landscape apps <a href="#landscape-apps" id="landscape-apps"></a>

It's possible to create and distribute desks without a front-end, but most developers will want a web UI for their app. There are a couple of options for this:

* Have a Gall agent handle directly HTTP requests through, doing server-side page rendering and/or serving front-end files out of Clay. You can refer to the [Sail guide](/hoon/sail) for more information about generating HTML in Hoon.
* Have the Gall agents perform back-end functions only, and create a separate client bundle called a [`glob`](/build-on-urbit/userspace/dist/glob), which contains the front-end files like HTML, CSS, JS, images, and so forth.

When an app is started, `%docket` will read the `desk.docket-0` file in the desk and, if it specifies a glob, fetch and serve it. For more details of the docket file, see the [Docket File](/build-on-urbit/userspace/dist/docket) document.

### Globs <a href="#globs" id="globs"></a>

If the glob is to be served over Ames, the Docket globulator at the `/docket/upload` URL will let you upload the front-end files. It'll automatically update the `desk.docket-0` file with the name and hash so people who install the desk know where to get it.

If the glob is to be served over HTTP, the [`-make-glob`](/build-on-urbit/userspace/dist/glob#make-glob) thread can be used to assemble it from the individual front-end files and output the resulting `.glob` file to the host filesystem. You can then upload the glob to an s3 bucket or wherever else you'd like to serve it from. You can then update the `desk.docket-0` file with the URL.

Note that serving a glob over Ames might increase the install time for your app, since Ames is currently pretty slow compared to HTTP — but being able to serve a glob from your ship allows you to serve your whole app, both server-side and client-side, without setting up a CDN or any other external web tooling. Your ship can do it all on its own.

For further details of globs, see the [Glob](/build-on-urbit/userspace/dist/glob) document.

## Sections <a href="#sections" id="sections"></a>

* [Guide](/build-on-urbit/userspace/dist/software-distribution) - A detailed guide to software distribution.
* [Glob](/build-on-urbit/userspace/dist/glob) - Documentation of `glob`s (client bundles).
* [Docket Files](/build-on-urbit/userspace/dist/docket) - Documentation of docket files.


# Software Distribution Guide

Complete tutorial for creating and publishing Urbit apps. Walk through Hello World example from desk creation to remote installation via Treaty.

In this document we'll walk through an example of creating and publishing a desk that others can install. We'll create a simple "Hello World!" front-end with a "Hello" tile to launch it. For simplicity, the desk won't include an actual Gall agent, but we'll note everything necessary if there were one.

## Install binary <a href="#install-binary" id="install-binary"></a>

*Note: You can skip this step if you already have the latest binary installed.*

Follow the [CLI install instructions](/get-on-urbit#get-the-urbit-runtime) appropriate for your OS to download the `urbit` binary.

## Spin up fake ship <a href="#spin-up-fake-ship" id="spin-up-fake-ship"></a>

*Note: You can skip this step if you already have a fake \~zod.*

With the `urbit` binary installed, run the following in the terminal to spin up a fake \~zod:

{% code title="Unix" %}

```sh
./urbit -F zod
```

{% endcode %}

It will take a couple of minutes to spin up, but it should eventually take you to a Dojo prompt like:

{% code title="Dojo" %}

```
~zod:dojo>
```

{% endcode %}

## Create desk <a href="#create-desk" id="create-desk"></a>

In the Dojo of your fake \~zod, run the following to create a new desk called `%hello`:

{% code title="Dojo" %}

```
|new-desk %hello
```

{% endcode %}

If you now run `+tree` on the desk, you'll see it now exists and has a handful of files in it:

{% code title="Dojo" %}

```
> +tree /=hello=
/mar/hoon/hoon
/mar/kelvin/hoon
/mar/noun/hoon
/mar/txt/hoon
/sys/kelvin
```

{% endcode %}

## Copy in extra files <a href="#copy-in-extra-files" id="copy-in-extra-files"></a>

Each desk must be self-contained in terms of its mark files and libraries. There's a couple of extra mark files and such that we'll need to add, so run the following commands in the Dojo:

{% code title="Dojo" %}

```
|cp %/mar/mime/hoon /=hello=/mar/mime/hoon
|cp %/mar/txt-diff/hoon /=hello=/mar/txt-diff/hoon
|cp %/mar/ship/hoon /=hello=/mar/ship/hoon
|cp %/mar/bill/hoon /=hello=/mar/bill/hoon
|cp /=landscape=/mar/docket-0/hoon /=hello=/mar/docket-0/hoon
|cp /=landscape=/sur/docket/hoon /=hello=/sur/docket/hoon
|cp /=landscape=/lib/docket/hoon /=hello=/lib/docket/hoon
```

{% endcode %}

## Mount the desk <a href="#mount-the-desk" id="mount-the-desk"></a>

Now we have all the basic files we need, we can mount the desk to the host. Run the following in the Dojo:

{% code title="Dojo" %}

```
|mount %hello
```

{% endcode %}

Now, in an ordinary terminal (not the Dojo), navigate to the mounted desk:

{% code title="Unix" %}

```sh
cd zod/hello
```

{% endcode %}

## Config files <a href="#config-files" id="config-files"></a>

### `sys.kelvin` <a href="#syskelvin" id="syskelvin"></a>

Our desk must include a `sys.kelvin` file which specifies the kernel version it's compatible with. The `|new-desk` generator automatically added that, so we can leave it as-is.

### `desk.ship` <a href="#deskship" id="deskship"></a>

We can optionally add a `desk.ship` file to specify the original publisher of this desk. We're using a fake \~zod so let's just add \~zod as the publisher:

{% code title="Unix" %}

```sh
echo "~zod" > desk.ship
```

{% endcode %}

### `desk.bill` <a href="#deskbill" id="deskbill"></a>

If we had Gall agents in this desk which should be automatically started when the desk is installed, we'd add them to a list in the `desk.bill` file. It would look something like this:

{% code title="desk.bill" %}

```hoon
:~  %some-agent
    %another-agent
==
```

{% endcode %}

In this example we're not adding any agents, so we'll simply omit the `desk.bill` file.

### `desk.docket-0` <a href="#deskdocket-0" id="deskdocket-0"></a>

The final file we need is `desk.docket-0`. This one's more complicated, so we'll open it in our preferred text editor:

{% code title="Unix" %}

```sh
nano desk.docket-0
```

{% endcode %}

In the text editor, we'll add the following:

{% code title="desk.docket-0" %}

```hoon

:~  title+'Hello'
    info+'A simple hello world app.'
    color+0x81.88c9
    image+'https://media.urbit.org/guides/additional/dist/wut.svg'
    base+'hello'
    glob-ames+[~zod 0v0]
    version+[0 0 1]
    website+'https://urbit.org'
    license+'MIT'
==
```

{% endcode %}

You can refer to the [Docket File](/build-on-urbit/userspace/dist/docket) documentation for more details of what is required. In brief, the `desk.docket-0` file contains a Hoon list of [clauses](/build-on-urbit/userspace/dist/docket) which configure the appearance of the app tile, the source of the [glob](/build-on-urbit/userspace/dist/glob), and some other metadata.

We've given the app a [`%title`](/build-on-urbit/userspace/dist/docket#title) of "Hello", which will be displayed on the app tile and will be the name of the app when others browse to install it. We've given the app tile a [`%color`](/build-on-urbit/userspace/dist/docket#color) of `#8188C9`, and also specified the URL of an [`%image`](/build-on-urbit/userspace/dist/docket#image) to display on the tile.

The [`%base`](/build-on-urbit/userspace/dist/docket#base) clause specifies the base URL path for the app. We've specified `'hello'` so it'll be `http://localhost:8080/apps/hello/...` in the browser. For the [glob](/build-on-urbit/userspace/dist/glob), we've used a clause of [`%glob-ames`](/build-on-urbit/userspace/dist/docket#glob-ames), which means the glob will be served from a ship over Ames, as opposed to being served over HTTP with a [`%glob-http`](/build-on-urbit/userspace/dist/docket#glob-http) clause or having an Eyre binding with a [`%site`](/build-on-urbit/userspace/dist/docket#site) clause. You can refer to the [glob](/build-on-urbit/userspace/dist/glob) documentation for more details of the glob options. In our case we've specified `[~zod 0v0]`. Since \~zod is the fakeship we'll install it on, the `%docket` agent will await a separate upload of the glob, so we can just specify `0v0` here as it'll get overwritten later.

The [`%version`](/build-on-urbit/userspace/dist/docket#version) clause specifies the version as a triple of major version, minor version and patch version. The rest is just some additional informative metadata which will be displayed in *App Info*.

So let's save that to the `desk.docket-0` file and have a look at our desk:

{% code title="Unix" %}

```sh
ls
desk.docket-0  desk.ship  lib  mar  sur  sys.kelvin
```

{% endcode %}

## Commit <a href="#commit" id="commit"></a>

Now we've added out configuration files, we can commit them to the desk. Back in the fake \~zod's Dojo, run the following:

{% code title="Dojo" %}

```
|commit %hello
```

{% endcode %}

You should see something like this as the result:

{% code title="Dojo" %}

```
>=
+ /~zod/hello/9/desk/ship
+ /~zod/hello/9/desk/docket-0
```

{% endcode %}

## Install <a href="#install" id="install"></a>

The next step is to install the desk as an app. Run the following in the Dojo:

{% code title="Dojo" %}

```
|install our %hello
```

{% endcode %}

You should see a message like:

{% code title="Dojo" %}

```
docket: awaiting manual glob for %hello desk
```

{% endcode %}

That's because our `desk.docket-0` file includes a [`%glob-ames`](/build-on-urbit/userspace/dist/docket#glob-ames) clause which specifies our ship as the source, so it's waiting for us to upload the glob of front-end files.

## Create files for glob <a href="#create-files-for-glob" id="create-files-for-glob"></a>

We'll now create the files for the glob. We'll use a very simple static HTML page that just displayes "Hello World!" and an image. Typically we'd have a more complex JS web app that talked to apps on our ship through Eyre's channel system, but for the sake of simplicity we'll forgo that. Let's hop back in the Unix terminal and run a few commands:

{% code title="Unix" %}

```sh
cd ~
mkdir hello-glob
cd hello-glob
mkdir img
wget -P img https://media.urbit.org/guides/additional/dist/pot.svg
```

{% endcode %}

We should now have the following file structure in our home directory:

{% code title="Unix" %}

```
hello-glob
└── img
    └── pot.svg
```

{% endcode %}

We've grabbed an image to use in our "Hello world!" page. The next thing we need to add is an `index.html` file in the root of the folder. The `index.html` file is mandatory; it's what will be loaded when the app's tile is clicked. Let's open our preferred editor and create it:

{% code title="Unix" %}

```sh
nano index.html
```

{% endcode %}

In the editor, paste in the following HTML and save it:

{% code title="index.html" %}

```html
<!DOCTYPE html>
<html>
  <head>
    <style>
      div {
        text-align: center;
      }
    </style>
  </head>
  <title>Hello World</title>
  <body>
    <div>
      <h1>Hello World!</h1>
      <img src="img/pot.svg" alt="pot" width="219" height="196" />
    </div>
  </body>
</html>
```

{% endcode %}

Our `/hello-glob` folder should now look like this:

{% code title="Unix" %}

```
hello-glob
├── img
│   └── pot.svg
└── index.html
```

{% endcode %}

## Login to Landscape <a href="#login-to-landscape" id="login-to-landscape"></a>

Open a web browser and navigate to `localhost:8080`, or just `localhost` if port `8080` doesn't work. It should take you to the fake \~zod's login screen. Login with the fake \~zod's default code of `lidlut-tabwed-pillex-ridrup`.

Once you login, you'll notice you have the Hello app's tile, but it still says "installing" because we haven't uploaded the glob yet:

![](https://media.urbit.org/guides/additional/dist/2-installing.png)

## Upload to glob <a href="#upload-to-glob" id="upload-to-glob"></a>

We can now create a glob from the `/hello-glob` directory we previously created. To do so, navigate to `http://localhost:8080/docket/upload` in the browser. This will bring up the `%docket` app's [Globulator](/build-on-urbit/userspace/dist/glob#globulator) tool:

![](https://media.urbit.org/guides/additional/dist/2-globulator.png)

Simply select "hello" from the drop-down, click "Choose file" and select the `/hello-glob` folder in the the file browser, then click the "glob!" button.

Now if we return to our ship's homescreen, we should see the tile looks as we specified in the docket file:

![](https://media.urbit.org/guides/additional/dist/2-installed-tile.png)

And if we click on the tile, it'll load the `index.html` in our glob:

![](https://media.urbit.org/guides/additional/dist/2-hello-world.png)

Our app is working!

## Publish <a href="#publish" id="publish"></a>

The final step is publishing our desk with the `%treaty` agent so others can install it. To do this, there's a simple command in the dojo:

{% code title="Dojo" %}

```
> :treaty|publish %hello
>=
```

{% endcode %}

Note: For desks without a docket file (and therefore without a tile and glob), `%treaty` can't be used. Instead you can make the desk public with `|public %desk-name`, which will enable other ships to install it with the Dojo command `|install ~sampel %desk-name`.

## Remote install <a href="#remote-install" id="remote-install"></a>

Let's spin up another fake ship in the terminal so we can try install it:

{% code title="Unix" %}

```sh
cd ~
./urbit -F bus
```

{% endcode %}

In the browser, navigate to `localhost:8081` (or `localhost:8080` if that doesn't work) and login with \~bus's code `riddec-bicrym-ridlev-pocsef`. Next, type "\~zod" in the search bar and click on the matching result. It should pop up a list of \~zod's published apps, which in this case is our "Hello" app:

![](https://media.urbit.org/guides/additional/dist/2-zod-apps.png)

When we click on the app, it'll show some of the information from the clauses in the docket file:

![](https://media.urbit.org/guides/additional/dist/2-hello-info.png)

Click "Get App" and it'll ask as if we want to install it:

![](https://media.urbit.org/guides/additional/dist/2-install-hello.png)

Finally, click "Get 'Hello'" and it'll be installed as a tile on \~bus which can then be opened:

![](https://media.urbit.org/guides/additional/dist/2-hello-installed-bus.png)


# Docket File

Docket file configuration for Landscape apps. Defines tile appearance, glob source, version, and metadata using clauses like %title, %color.

The docket file sets various options for desks with a tile and (usually) a browser-based front-end of some kind. Mainly it configures the appearance of an app's tile, the source of its [glob](/build-on-urbit/userspace/dist/glob), and some additional metadata.

The docket file is read by the `%docket` agent when a desk is `|install`ed. The `%docket` agent will fetch the glob if applicable and create the tile as specified on the homescreen. If the desk is published with `:treaty|publish`, the information specified in the docket file will also be displayed for others who are browsing apps to install on your ship.

The docket file is *optional* in the general case. If it is omitted, however, the app cannot have a tile in Landscape, nor can it be published with the `%treaty` agent, so others will not be able to browse for it in Landscape.

The docket file must be named `desk.docket-0`. The `%docket` mark is versioned to facilitate changes down the line, so the `-0` suffix may be incremented in the future.

The file must contain a Hoon list with a series of clauses. The clauses are defined in `/sur/docket.hoon` as:

```hoon
+$  clause
  $%  [%title title=@t]
      [%info info=@t]
      [%color color=@ux]
      [%glob-http url=cord hash=@uvH]
      [%glob-ames =ship hash=@uvH]
      [%image =url]
      [%site =path]
      [%base base=term]
      [%version =version]
      [%website website=url]
      [%license license=cord]
  ==
```

The `%image` clause is optional. It is mandatory to have exactly one of either `%site`, `%glob-http` or `%glob-ames`. All other clauses are mandatory.

Here's what a typical docket file might look like:

```hoon
:~
  title+'Foo'
  info+'An app that does a thing.'
  color+0xf9.8e40
  glob-ames+[~zod 0v0]
  image+'https://example.com/tile.svg'
  base+'foo'
  version+[0 0 1]
  license+'MIT'
  website+'https://example.com'
==
```

Details of each clause and their purpose are described below.

***

## `%title` <a href="#title" id="title"></a>

*required*

The `%title` field specifies the name of the app. The title will be the name shown on the app's tile, as well as the name of the app when others search for it.

#### Type

```hoon
[%title title=@t]
```

#### Example

```hoon
title+'Bitcoin'
```

***

## `%info` <a href="#info" id="info"></a>

*required*

The `%info` field is a brief summary of what the app does. It will be shown as the subtitle in *App Info*.

#### Type

```hoon
[%info info=@t]
```

#### Example

```hoon
info+'A Bitcoin Wallet that lets you send and receive Bitcoin directly to and from other Urbit users'
```

***

## `%color` <a href="#color" id="color"></a>

*required*

The `%color` field specifies the color of the app tile as a `@ux`-formatted hex value.

#### Type

```hoon
[%color color=@ux]
```

#### Example

```hoon
color+0xf9.8e40
```

***

## `%glob-http` <a href="#glob-http" id="glob-http"></a>

*exactly one of either this,* [*glob-ames*](#glob-ames) *or* [*site*](#site) *is required*

The `%glob-http` field specifies the URL and hash of the app's [glob](/build-on-urbit/userspace/dist/glob) if it is distributed via HTTP.

#### Type

```hoon
[%glob-http url=cord hash=@uvH]
```

#### Example

```hoon
glob-http+['https://example.com/glob-0v1.s0me.h4sh.glob' 0v1.s0me.h4sh]
```

***

## `%glob-ames` <a href="#glob-ames" id="glob-ames"></a>

*exactly one of either this,* [*glob-http*](#glob-http) *or* [*site*](#site) *is required*

The `%glob-ames` field specifies the ship and hash of the app's [glob](/build-on-urbit/userspace/dist/glob) if it is distributed from a ship over Ames. If the glob will be distributed from our ship, the hash can initially be `0v0` as it will be overwritten with the hash produced by the [Globulator](/build-on-urbit/userspace/dist/glob#globulator).

#### Type

```hoon
[%glob-ames =ship hash=@uvH]
```

#### Example

```hoon
glob-ames+[~zod 0v0]
```

***

## `%site` <a href="#site" id="site"></a>

*exactly one of either this,* [*glob-ames*](#glob-ames) *or* [*glob-http*](#glob-http) *is required*

It's possible for an app to handle HTTP requests from the client directly rather than with a separate [glob](/build-on-urbit/userspace/dist/glob). In that case, the `%site` field specifies the `$path` of the Eyre endpoint the app will bind. If `%site` is used, clicking the app's tile will simply open a new tab with a GET request to the specified Eyre endpoint.

For more information on direct HTTP handling with a Gall agent or generator, see the [Eyre Internal API Reference](/urbit-os/kernel/eyre/tasks) documentation.

#### Type

```hoon
[%site =path]
```

#### Example

```hoon
site+/foo/bar
```

***

## `%image` <a href="#image" id="image"></a>

*optional*

The `%image` field specifies the URL of an image to be displayed on the app's tile. This field is optional and may be omitted entirely.

The given image will be displayed on top of the [color](#color)ed tile. The app [title](#title) (and hamburger menu upon hover) will be displayed on top of the given image, in small rounded boxes with the same background color as the main tile. The given image will be displayed at 100% of the width of the tile. The image's corners will be hidden by the rounded corners of the tile, so the image itself needn't have rounded corners. The tile is a perfect square, so if the image should occupy the whole tile, it should also be a perfect square. If the image should be a smaller icon in the center of the tile (like the bitcoin tile), it should just have a square of transparent negative space around it.

It may be tempting to set the image URL as a root-relative path like `/apps/myapp/img/tile.svg` and bundle it in the glob. While this would work locally, it means the image would fail to load for those browsing apps to install. Therefore, the image should be hosted somewhere globally available.

#### Type

```hoon
[%image =url]
```

The `url` type is a simple `$cord`:

```hoon
+$  url  cord
```

#### Example

```hoon
image+'http://example.com/icon.svg'
```

***

## `%base` <a href="#base" id="base"></a>

*required*

The `%base` field specifies the base of the URL path of the glob resources. In the browser, the path will begin with `/apps`, then the specified base, then the rest of the path to the particular glob resource like `http://localhost:8080/apps/my-base/index.html`. Note the `$path`s of the glob contents themselves should not include this base element.

#### Type

```hoon
[%base base=term]
```

#### Example

```hoon
base+'bitcoin'
```

***

## `%version` <a href="#version" id="version"></a>

*required*

The `%version` field specifies the current version of the app. It's a triple of three `@ud` numbers representing the major version, minor version and patch version. In the client, `[1 2 3]` will be rendered as `1.2.3`. You would typically increase the appropriate number each time you published a change to the app.

#### Type

```hoon
[%version =version]
```

The `version` type is just a triple of three numbers:

```hoon
+$  version
  [major=@ud minor=@ud patch=@ud]
```

#### Example

```hoon
version+[0 0 1]
```

***

## `%website` <a href="#website" id="website"></a>

*required*

The `%website` field is for a link to a relevant website. This might be a link to the app's github repo, company website, or whatever is appropriate. This field will be displayed when people are browsing apps to install.

#### Type

```hoon
[%website website=url]
```

The `url` type is a simple `$cord`:

```hoon
+$  url  cord
```

#### Example

```hoon
website+'https://example.com'
```

***

## `%license` <a href="#license" id="license"></a>

*required*

The `%license` field specifies the license for the app in question. It would typically be a short name like `MIT`, `GPLv2`, or what have you. The field just takes a `$cord` so any license can be specified.

#### Type

```hoon
[%license license=cord]
```

#### Example

```hoon
license+'MIT'
```


# Glob

Globs contain client-side resources (HTML, JS, CSS) for Landscape apps. Distribution over Ames or HTTP, Globulator tool, make-glob thread.

A `glob` contains the client bundle—client-side resources like HTML, JS, and CSS files—for a Landscape app distributed in a desk. Globs are managed separately from other files in desks because they often contain large files that frequently change, and would therefore bloat a ship's state if they were subject to Clay's revision control mechanisms.

The hash and source of an app's glob is defined in a desk's [docket file](/build-on-urbit/userspace/dist/docket). The `%docket` agent reads the docket file, obtains the glob from the specified source, and makes its contents available to the browser client. On a desk publisher's ship, if the glob is to be distributed over Ames, the glob is also made available to desk subscribers.

## The `glob` type <a href="#the-glob-type" id="the-glob-type"></a>

The `%docket`agent defines the type of a `glob` as:

```hoon
+$  glob  (map path mime)
```

Given the following file heirarchy:

```
foo
├── css
│   └── style.css
├── img
│   ├── favicon.png
│   ├── foo.svg
│   └── bar.svg
├── index.html
└── js
    └── baz.js
```

...its `$glob` form would look like:

```hoon
{ [p=/img/foo/svg q=[p=/image/svg+xml q=[p=0 q=0]]]
  [p=/css/style/css q=[p=/text/css q=[p=0 q=0]]]
  [p=/img/favicon/png q=[p=/image/png q=[p=0 q=0]]]
  [p=/js/baz/js q=[p=/application/javascript q=[p=0 q=0]]]
  [p=/img/bar/svg q=[p=/image/svg+xml q=[p=0 q=0]]]
  [p=/index/html q=[p=/text/html q=[p=0 q=0]]]
}
```

Note: The mime byte-length and data are 0 in this example because it was made with empty dummy files.

A glob may contain any number of files and folders in any kind of heirarchy. The one important thing is that an `index.html` file is present in its root. The `index.html` file is automatically served when the app is opened in the browser and will fail if it is missing.

In addition to the `$glob` type, a glob can also be output to Unix with a `.glob` file extension for distribution over HTTP. This file simply contains a [`+jam`](/hoon/stdlib/2p#jam)med `$glob` structure.

## Docket file clause <a href="#docket-file-clause" id="docket-file-clause"></a>

The `desk.docket-0` file must include exactly one of the following clauses:

#### `site+/some/path`

If an app binds an Eyre endpoint and handles HTTP directly, for example with a [`%connect` task:eyre](/urbit-os/kernel/eyre/tasks#connect), the `%site` clause is used, specifying the Eyre binding. In this case a glob is omitted entirely.

#### `glob-ames+[~zod 0vs0me.h4sh]`

If the glob is to be distributed over Ames, the `%glob-ames` clause is used, with a cell of the `$ship` which has the glob and the `@uv` hash of the glob. If it's our ship, the hash can just be `0v0` and the glob can instead be created with the [Globulator](#globulator).

#### `glob-http+['https://example.com/some.glob' 0vs0me.h4sh]`

If the glob is to be distributed over HTTP, for example from an s3 instance, the `%glob-http` clause is used. It takes a cell of a `$cord` with the URL serving the glob and the `@uv` hash of the glob.

## Making a glob <a href="#making-a-glob" id="making-a-glob"></a>

There are a couple of different methods depending on whether the glob will be distributed over HTTP or Ames.

### Globulator <a href="#globulator" id="globulator"></a>

For globs distributed over Ames from our ship, the client bundle can be uploaded directly with `%docket`'s Globulator tool, which is available in the browser at `http[s]://[host]/docket/upload`. It looks like this:

![](https://media.urbit.org/docs/userspace/dist/globulator.png)

Simply select the target desk, select the folder to be globulated, and hit `glob!`.

Note the target desk must have been `|install`ed before uploading its glob. When installed, `%docket` will print `docket: awaiting manual glob for %desk-name desk` in the terminal and wait for the upload. The hash in the `%ames-glob` clause of the docket file will be overwritten by the hash of the new glob. As a result, there's no need to specify the actual glob hash in `desk.docket` - you can just use any `@uv` like `0v0`. Once uploaded, the desk can then be published with `:treaty|publish %desk-name` and the glob will become available for download by subscribers.

### `-make-glob` <a href="#make-glob" id="make-glob"></a>

There's a different process for globs to be distributed over HTTP from a webserver rather than over Ames from a ship. For this purpose, the `%landscape` desk includes a `%make-glob` thread. The thread takes a folder in a desk and produces a glob of the files it contains, which it then saves to Unix in a [`+jam`](/hoon/stdlib/2p#jam)file with a `.glob` extension.

To begin, you'll need to spin up a ship (typically a fake ship) and `|mount` a desk for which to add the files. In order for Clay to add the files, the desk must contain mark files in its `/mar` directory for all file extensions your folder contains. The `%landscape` desk is a good bet because it includes mark files for `.js`, `.html`, `.png`, `.svg`, `.woff2` and a couple of others. If there's no desk with a mark for a particular file type you want included in your glob, you may need to add a new mark file. A very rudimentary mark file like the `png.hoon` mark will suffice.

With the desk mounted, add the folder to be globbed to the root of the desk in Unix. It's imporant it's in the root because the `%make-glob` thread will only strip the first level of the folder heirarchy. Additionally ensure that all file names in the folder you're globbing are lowercase, otherwise this next step will not work correctly.

Next, `|commit` the files to the desk, then run `-landscape!make-glob %the-desk /folder-name`, where `%the-desk` is the desk containing the folder to be globbed and `/folder-name` is its name.

On Unix, if you look in `/path/to/pier/.urb/put`, you'll now see a file which looks like:

```
glob-0v1.7vpqa.r8pn5.6t0s1.rhc7r.5e9vo.glob
```

This file can be uploaded to your webserver and the `desk.docket-0` file of the desk you're publishing can be updated with:

```hoon
glob-http+['https://s3.example.com/glob-0v1.7vpqa.r8pn5.6t0s1.rhc7r.5e9vo.glob' 0v1.7vpqa.r8pn5.6t0s1.rhc7r.5e9vo]
```


# Examples

Examples and exercises for building Urbit userspace apps including CLI tools, websites, and games.

This section contains examples of building simple apps and additional exercises to help hone your skills.

* [Building a CLI App](/build-on-urbit/userspace/examples/rpn) - Building a simple calculator you can use from the command line.
* [Debugging Wrapper](/build-on-urbit/userspace/examples/dbug) - How to use the `/lib/dbug.hoon` wrapper library to debug your Gall agents.
* [Host a Website](/build-on-urbit/userspace/examples/feature) - How to build a simple app that serves a website,
* [Serving a JS Game](/build-on-urbit/userspace/examples/flap) - How to connect an off-the-shelf JS game to an Urbit back-end.
* [Ship Monitoring](/build-on-urbit/userspace/examples/ahoy) - Examining and modifying an app that monitors ship activity.
* [Styled Text](/build-on-urbit/userspace/examples/track7) - How to produce styled text in an agent and output it to the terminal.


# Building a CLI App

Tutorial on building a reverse Polish notation calculator CLI app using Hoon and the shoe library.

We will utilize the basic calculator app logic from the [parsing guide](/hoon/parsing#recursive-parsers) to produce a linked calculator agent `%rpn` supporting the following operators by the appropriate parsers:

* numbers (as `@rs` without `.` dot prefix) (`royl-rs:so`)
* `+` lus, addition (`lus`)
* `-` hep, subtraction (`hep`)
* `*` tar, multiplication (`tar`)
* `/` fas, division (`fas`)
* `.` dot, display top of stack (`dot`)

We will leave all regular Gall arms as their defaults, but of course poking, subscribing, and peeking should be supported in a full application.

## Agent Logic <a href="#agent-logic" id="agent-logic"></a>

**`/sur/rpn.hoon`**

We just need to define the expected operators that will show up in the stack. These are `@t` text constants.

```hoon
|%
+$  op  $?  [%op %add]
            [%op %sub]
            [%op %mul]
            [%op %div]
            [%op %sho]
        ==
+$  num  @rs
+$  command  ?(@rs op)
--
```

(`$command` doesn't really feel like the right name here, but we're pattern-matching with the demo `/app/shoe.hoon`.)

**`/lib/rpn.hoon`**

These are the parsing rules that the CLI agent will use. We could include these directly in the agent file but we'll post them to a library file.

```hoon
|%
++  num     royl-rs:so
++  op-add  (cook |=(p=@ ?:(=('+' p) op+%add ~)) lus)
++  op-sub  (cook |=(p=@ ?:(=('-' p) op+%sub ~)) hep)
++  op-mul  (cook |=(p=@ ?:(=('*' p) op+%mul ~)) tar)
++  op-div  (cook |=(p=@ ?:(=('/' p) op+%div ~)) fas)
++  op-sho  (cook |=(p=@ ?:(=('.' p) op+%sho ~)) dot)
++  ops     ;~(pose op-add op-sub op-mul op-div op-sho)
--
```

**`/app/rpn.hoon`**

```hoon
++  state-0
  $:  %0
      stack=(list ?(@rs op:rpn))
  ==
```

**`+command-parser`**

We want this arm to wait until `RETURN` is pressed so we `+stag` the value with `|` `FALSE`/`%.n`.

```hoon
++  command-parser
  |=  =sole-id:shoe
  ^+  |~(nail *(like [? command:rpn]))
  %+  stag  |
  (cook command:rpn ;~(pose num:rpnlib ops:rpnlib))
```

**`+on-command`**

This arm pushes values onto the stack, displays the stack, then checks to parse for the result of an operation.

```hoon
++  on-command
  |=  [=sole-id:shoe =command:rpn]
  ^-  (quip card _this)
  =/  old-stack  (weld stack ~[command])
  =/  new-stack  (process:rpnlib old-stack)
  :_  this(stack new-stack)
  :~  [%shoe ~ sole+klr+~[(crip "{<old-stack>} →")]]
      [%shoe ~ sole+klr+~[[[`%br ~ `%g] (crip "{<new-stack>}") ~]]]
  ==
```

For this we add a helper arm to `/lib/rpn.hoon` which takes each entry, makes sure it is a `@rs` atom, and carries out the operation. (This could probably be made more efficient.)

**`/lib/rpn.hoon`**

```hoon
/-  rpn
:: * * *
++  process
  |=  stack=(list command:rpn)
  ^-  (list command:rpn)
  ~|  "Failure processing operation on stack {<stack>}"
  ?~  stack  !!
  ?-    `command:rpn`(snag 0 (flop stack))
      [%op %add]
    =/  augend        ;;(@rs `command:rpn`(snag 1 (flop stack)))
    =/  addend        ;;(@rs `command:rpn`(snag 2 (flop stack)))
    (flop (weld ~[(add:rs augend addend)] (slag 3 (flop stack))))
    ::
      [%op %sub]
    =/  minuend       ;;(@rs `command:rpn`(snag 1 (flop stack)))
    =/  subtrahend    ;;(@rs `command:rpn`(snag 2 (flop stack)))
    (flop (weld ~[(sub:rs minuend subtrahend)] (slag 3 (flop stack))))
    ::
      [%op %mul]
    =/  multiplicand  ;;(@rs `command:rpn`(snag 1 (flop stack)))
    =/  multiplier    ;;(@rs `command:rpn`(snag 2 (flop stack)))
    (flop (weld ~[(mul:rs multiplicand multiplier)] (slag 3 (flop stack))))
    ::
      [%op %div]
    =/  numerator     ;;(@rs `command:rpn`(snag 1 (flop stack)))
    =/  denominator   ;;(@rs `command:rpn`(snag 2 (flop stack)))
    (flop (weld ~[(div:rs numerator denominator)] (slag 3 (flop stack))))
    ::
      [%op %sho]
    ~&  >  "{<(snag 1 (flop stack))>}"
    (flop (slag 1 (flop stack)))
    ::
      @rs
    stack
  ==
```

### Linking <a href="#linking" id="linking"></a>

After a `%sole` agent has been `|install`ed, it should be registered for Dojo to cycle input to it using `|dojo/link`.

```hoon
|dojo/link %rpn
```

Now `Ctrl+X` allows you to switch to that app and evaluate expressions using it.

```hoon
gall: booted %rpn
> 50
~ →
~[.50]

> 25
~[.50] →
~[.50 .25]

> -
~[.50 .25] →
~[.-25]

> 5
~[.-25] →
~[.-25 .5]

> /
~[.-25 .5] →
~[.-0.19999999]

> 5
~[.-0.19999999] →
~[.-0.19999999 .5]

> *
~[.-0.19999999 .5] →
~[.-0.99999994]

> 1
~[.-0.99999994] →
~[.-0.99999994 .1]

> /
~[.-0.99999994 .1] →
~[.-1]
```

## Exercises <a href="#exercises" id="exercises"></a>

* Extend the calculator app to support modulus as `%` cen.
* Extend the calculator app so it instead operates on `@rd` values. Either use `+cook` to automatically convert the input values from a `1.23`-style input to the `.~1.23` `@rd` style or build a different input parser from the entries in `+royl:so`.
* Extend the calculator app so that it can support named variables (using `@tas`) with `=` tis. What new data structure do you need? For convenience, expose the result of the last operation as `ans` (a feature of TI graphing calculators and MATLAB, among other programs).
* The calculator app stack isn't really a proper CS stack with push and pop operations. Refactor it to use such a type.


# Debugging Wrapper

Guide to using the dbug library to debug Gall agents and inspect their state.

The `/lib/dbug.hoon` agent wrapper adds support to view the state of a Gall agent. It is applied to an existing Gall agent as a single drop-in line, `%- agent:dbug`.

Before we look at the code, let's consider the functionality it exposes. By supplying `%- agent:dbug`, an associated `+dbug` generator can be invoked against the agent state.

For instance, using the `%azimuth` agent, we can expose the current state of the agent:

```hoon
> :azimuth +dbug
[ %7
  url=~.
  net=%default
  refresh=~m5
  whos={}
  nas=[%0 points={} operators={} dns=<||>]
  own={}
  spo={}
  logs=~
  sap=[%0 id=[hash=0x0 number=0] nas=[%0 points={} operators={} dns=<||>] owners={} sponsors={}]
]

> :azimuth +dbug %bowl
>   [ [our=~zod src=~zod dap=%azimuth]
  [wex={} sup={}]
  act=3
    eny
  0v1rn.n49dr.2u8t5.h7be5.6dcq7.9hon5.6m3pr.3hcb8.u7tmv.qddpq.kent7.1ftc7.9tao6.hfsht.4i0c3.ak3t7.t8d8j.nn4eb.b7eh3.4d5pr.t8ftg
  now=~2023.2.3..20.03.23..f60e
  byk=[p=~zod q=%base r=[%da p=~2023.1.26..02.41.25..926a]]
]

> :azimuth +dbug [%incoming ~]
>   no matching subscriptions

> :azimuth +dbug [%state '(lent whos)']
>   0
```

There are four actions exposed by the wrapper via the `+dbug` generator:

1. `:app +dbug` exposes the entire state, just dumping the current agent state.
2. `:app +dbug %bowl` shows the agent's `$bowl`. The Gall `$bowl` consists of:

   ```hoon
   +$  bowl              ::  standard app state
     $:  $:  our=ship    ::  host
             src=ship    ::  guest
             dap=term    ::  agent
         ==              ::
         $:  wex=boat    ::  outgoing subs
             sup=bitt    ::  incoming subs
         ==              ::
         $:  act=@ud     ::  change number
             eny=@uvJ    ::  entropy
             now=@da     ::  current time
             byk=beak    ::  load source
     ==  ==              ::
   ```
3. `:app +dbug [%state 'hoon']` exposes data in the state, including evaluated Hoon like `(lent values)`.
4. `:app +dbug [?(%incoming outgoing) specifics]` reveals details about the subscriptions.

## The Code <a href="#the-code" id="the-code"></a>

<details>

<summary>/gen/dbug.hoon</summary>

```hoon
/+  *dbug
:-  %say
|=  $:  ::  environment
        *
        ::  inline arguments
        args=?(~ [what=?(%bowl %state) ~] [=poke ~])
        ::  named arguments
        depth=@ud
    ==
:-  %dbug
=;  =poke
  ?:  =(0 depth)  poke
  [%skip depth poke]
?-  args
  ~          [%state '']
  [@ ~]      ?-(what.args %bowl [%bowl ~], %state [%state ''])
  [[@ *] ~]  poke.args
==
```

</details>

<details>

<summary>/lib/dbug.hoon</summary>

```hoon
::  dbug: agent wrapper for generic debugging tools
::
::    usage: %-(agent:dbug your-agent)
::
|%
+$  poke
  $%  [%skip depth=@ud =poke]
      [%bowl ~]
      [%state grab=cord]
      [%incoming =about]
      [%outgoing =about]
  ==
::
+$  about
  $@  ~
  $%  [%ship =ship]
      [%path =path]
      [%wire =wire]
      [%term =term]
  ==
::
++  agent
  |=  =agent:gall
  ^-  agent:gall
  !.
  |_  =bowl:gall
  +*  this  .
      ag    ~(. agent bowl)
  ::
  ++  on-poke
    |=  [=mark =vase]
    ^-  (quip card:agent:gall agent:gall)
    ?.  ?=(%dbug mark)
      =^  cards  agent  (on-poke:ag mark vase)
      [cards this]
    =/  dbug
      !<(poke vase)
    =?  dbug  ?=([%skip %0 *] dbug)  poke.dbug
    ?:  ?=(%skip -.dbug)
      =^  cards  agent
        %+  on-poke:ag  %dbug
        !>(`poke`dbug(depth (dec depth.dbug)))
      [cards this]
    =;  =tang
      ((%*(. slog pri 1) tang) [~ this])
    ?-  -.dbug
      %bowl   [(sell !>(bowl))]~
    ::
        %state
      =?  grab.dbug  =('' grab.dbug)  '-'
      =;  product=^vase
        [(sell product)]~
      =/  state=^vase
        ::  if the underlying app has implemented a /dbug/state scry endpoint,
        ::  use that vase in place of +on-save's.
        ::
        =/  result=(each ^vase tang)
          (mule |.(q:(need (need (on-peek:ag /x/dbug/state)))))
        ?:(?=(%& -.result) p.result on-save:ag)
      %+  slap
        (slop state !>([bowl=bowl ..zuse]))
      (ream grab.dbug)
    ::
        %incoming
      =;  =tang
        ?^  tang  tang
        [%leaf "no matching subscriptions"]~
      %+  murn
        %+  sort  ~(tap by sup.bowl)
        |=  [[* a=[=ship =path]] [* b=[=ship =path]]]
        (aor [path ship]:a [path ship]:b)
      |=  [=duct [=ship =path]]
      ^-  (unit tank)
      =;  relevant=?
        ?.  relevant  ~
        `>[path=path from=ship duct=duct]<
      ?:  ?=(~ about.dbug)  &
      ?-  -.about.dbug
        %ship  =(ship ship.about.dbug)
        %path  ?=(^ (find path.about.dbug path))
        %wire  %+  lien  duct
               |=(=wire ?=(^ (find wire.about.dbug wire)))
        %term  !!
      ==
    ::
        %outgoing
      =;  =tang
        ?^  tang  tang
        [%leaf "no matching subscriptions"]~
      %+  murn
        %+  sort  ~(tap by wex.bowl)
        |=  [[[a=wire *] *] [[b=wire *] *]]
        (aor a b)
      |=  [[=wire =ship =term] [acked=? =path]]
      ^-  (unit tank)
      =;  relevant=?
        ?.  relevant  ~
        `>[wire=wire agnt=[ship term] path=path ackd=acked]<
      ?:  ?=(~ about.dbug)  &
      ?-  -.about.dbug
        %ship  =(ship ship.about.dbug)
        %path  ?=(^ (find path.about.dbug path))
        %wire  ?=(^ (find wire.about.dbug wire))
        %term  =(term term.about.dbug)
      ==
    ==
  ::
  ++  on-peek
    |=  =path
    ^-  (unit (unit cage))
    ?.  ?=([@ %dbug *] path)
      (on-peek:ag path)
    ?+  path  [~ ~]
      [%u %dbug ~]                 ``noun+!>(&)
      [%x %dbug %state ~]          ``noun+!>(on-save:ag)
      [%x %dbug %subscriptions ~]  ``noun+!>([wex sup]:bowl)
    ==
  ::
  ++  on-init
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  on-init:ag
    [cards this]
  ::
  ++  on-save   on-save:ag
  ::
  ++  on-load
    |=  old-state=vase
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  (on-load:ag old-state)
    [cards this]
  ::
  ++  on-watch
    |=  =path
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  (on-watch:ag path)
    [cards this]
  ::
  ++  on-leave
    |=  =path
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  (on-leave:ag path)
    [cards this]
  ::
  ++  on-agent
    |=  [=wire =sign:agent:gall]
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  (on-agent:ag wire sign)
    [cards this]
  ::
  ++  on-arvo
    |=  [=wire =sign-arvo]
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  (on-arvo:ag wire sign-arvo)
    [cards this]
  ::
  ++  on-fail
    |=  [=term =tang]
    ^-  (quip card:agent:gall agent:gall)
    =^  cards  agent  (on-fail:ag term tang)
    [cards this]
  --
--
```

</details>

As we examine this code, there are two particularly interesting aspects:

1. How `/lib/dbug.hoon` modifies an agent's arms by adding functionality over the top of them.
2. How `/gen/dbug.hoon` utilizes the modified arms with an elegant and simple invocation.

There is also extensive use of `$tank`/`$tang` formatted error messaging.

## How the library works <a href="#how-the-library-works" id="how-the-library-works"></a>

By applying this door builder using `%-` censig, the `+on-poke` and `+on-peek` arms can be modified. (In fact, all of the arms can be modified but most of the arms are pass-throughs to the modified agent.)

#### `+on-poke`

<details>

<summary>++on-poke</summary>

```hoon
++  on-poke
    |=  [=mark =vase]
    ^-  (quip card:agent:gall agent:gall)
    ?.  ?=(%dbug mark)
      =^  cards  agent  (on-poke:ag mark vase)
      [cards this]
    =/  dbug
      !<(poke vase)
    =?  dbug  ?=([%skip %0 *] dbug)  poke.dbug
    ?:  ?=(%skip -.dbug)
      =^  cards  agent
        %+  on-poke:ag  %dbug
        !>(`poke`dbug(depth (dec depth.dbug)))
      [cards this]
    =;  =tang
      ((%*(. slog pri 1) tang) [~ this])
    ?-  -.dbug
      %bowl   [(sell !>(bowl))]~
    ::
        %state
      =?  grab.dbug  =('' grab.dbug)  '-'
      =;  product=^vase
        [(sell product)]~
      =/  state=^vase
        ::  if the underlying app has implemented a /dbug/state scry endpoint,
        ::  use that vase in place of +on-save's.
        ::
        =/  result=(each ^vase tang)
          (mule |.(q:(need (need (on-peek:ag /x/dbug/state)))))
        ?:(?=(%& -.result) p.result on-save:ag)
      %+  slap
        (slop state !>([bowl=bowl ..zuse]))
      (ream grab.dbug)
    ::
        %incoming
      =;  =tang
        ?^  tang  tang
        [%leaf "no matching subscriptions"]~
      %+  murn
        %+  sort  ~(tap by sup.bowl)
        |=  [[* a=[=ship =path]] [* b=[=ship =path]]]
        (aor [path ship]:a [path ship]:b)
      |=  [=duct [=ship =path]]
      ^-  (unit tank)
      =;  relevant=?
        ?.  relevant  ~
        `>[path=path from=ship duct=duct]<
      ?:  ?=(~ about.dbug)  &
      ?-  -.about.dbug
        %ship  =(ship ship.about.dbug)
        %path  ?=(^ (find path.about.dbug path))
        %wire  %+  lien  duct
               |=(=wire ?=(^ (find wire.about.dbug wire)))
        %term  !!
      ==
    ::
        %outgoing
      =;  =tang
        ?^  tang  tang
        [%leaf "no matching subscriptions"]~
      %+  murn
        %+  sort  ~(tap by wex.bowl)
        |=  [[[a=wire *] *] [[b=wire *] *]]
        (aor a b)
      |=  [[=wire =ship =term] [acked=? =path]]
      ^-  (unit tank)
      =;  relevant=?
        ?.  relevant  ~
        `>[wire=wire agnt=[ship term] path=path ackd=acked]<
      ?:  ?=(~ about.dbug)  &
      ?-  -.about.dbug
        %ship  =(ship ship.about.dbug)
        %path  ?=(^ (find path.about.dbug path))
        %wire  ?=(^ (find wire.about.dbug wire))
        %term  =(term term.about.dbug)
      ==
    ==
```

</details>

The `+on-poke` arm has several branches added to it after a check to see whether it is being used through the `+dbug` generator. If it isn't (as determined by the associated `$mark`), then the poke is passed through to the base agent.

```hoon
?.  ?=(%dbug mark)
  =^  cards  agent  (on-poke:ag mark vase)
  [cards this]
```

The following `?-` wuthep handles the input arguments: `%state` is the most interesting code in this library. The code first checks whether the base agent has a `/dbug/state` peek endpoint already (in which case it passes it through), otherwise it evaluates the requested Hoon expression against the agent's state (obtained via `+on-save:ag`).

```hoon
  %state
=?  grab.dbug  =('' grab.dbug)  '-'
=;  product=^vase
  [(sell product)]~
=/  state=^vase
  ::  if the underlying app has implemented a /dbug/state scry endpoint,
  ::  use that vase in place of +on-save's.
  ::
  =/  result=(each ^vase tang)
    (mule |.(q:(need (need (on-peek:ag /x/dbug/state)))))
  ?:(?=(%& -.result) p.result on-save:ag)
%+  slap
  (slop state !>([bowl=bowl ..zuse]))
(ream grab.dbug)
```

This branch includes the use of a rare [`=?` tiswut](/hoon/rune/tis#tiswut) conditional leg change and the reversed `=/` tisfas, [`=;` tismic](/hoon/rune/tis#tismic). There is also some direct compilation of `$cord`s taking place:

* [`+sell`](/hoon/stdlib/5c#sell) is a `$vase` pretty-printer.
* [`+slop`](/hoon/stdlib/5c#slop) conses two `$vase`s together as a cell.
* [`+slap`](/hoon/stdlib/5c#slap) compiles a Hoon expression and produces a `$vase` of the result.
* [`+ream`](/hoon/stdlib/5d#ream) parses a `$cord` to a Hoon expression.

#### `+on-peek`

<details>

<summary>++on-peek</summary>

```hoon
  ++  on-peek
    |=  =path
    ^-  (unit (unit cage))
    ?.  ?=([@ %dbug *] path)
      (on-peek:ag path)
    ?+  path  [~ ~]
      [%u %dbug ~]                 ``noun+!>(&)
      [%x %dbug %state ~]          ``noun+!>(on-save:ag)
      [%x %dbug %subscriptions ~]  ``noun+!>([wex sup]:bowl)
    ==
```

</details>

The `+on-peek` arm adds several peek endpoints which expose the state (via `+on-save:ag`) and the subscriptions.

```hoon
> .^(noun %gx /(scot %p our)/azimuth/(scot %da now)/dbug/subscriptions/noun)
[0 0]
```

## How the generator works <a href="#how-the-generator-works" id="how-the-generator-works"></a>

The generator explicitly injects the `%dbug` mark in its return `+cask` (`[mark noun]`). This is a valid if uncommon operation, and it works here because the mark is never used as a transforming gate but only as a marker to see whether the arms need to pass through the values. The no-argument input is routed through the `%state` with an empty `$cord`.

```hoon
:-  %dbug
?-  args
  ~          [%state '']
  [@ ~]      ?-(what.args %bowl [%bowl ~], %state [%state ''])
  [[@ *] ~]  poke.args
==
```

Library authors should consider augmenting developer capabilities by exposing appropriate functionality using a wrapper agent similar to `/lib/dbug`.


# Host a Website

Tutorial on building a simple website hosting app using the Schooner library to serve HTML pages from Urbit.

[`%feature`](https://github.com/hanfel-dovned/Feature) by \~hanfel-dovned hosts a simple HTML page from an Urbit ship at an associated URL. This tutorial examines how it uses the middleware [`%schooner`](https://github.com/dalten-collective/schooner/) library by Quartus to return a web page when contacted by a web browser. You will learn how a basic site hosting app can handle HTTP requests and render a page using an `%html` mark.

`%feature` presents a web page from `/app/feature-ui` at `/apps/feature/feature-ui`. These paths are both configurable by the developer.

## `/sur` Structure Files <a href="#sur-structure-files" id="sur-structure-files"></a>

Our primary event in this case is simply an `%action` to create a page.

**`/sur/feature.hoon`**:

```hoon
|%
+$  action
  $%  [%new-page html=@t]
  ==
--
```

No special mark files are necessary for `%feature` other than `%html`.

## `/app` Agent Files <a href="#app-agent-files" id="app-agent-files"></a>

The agent only maintains a state containing the page contents as a `cord.`

The system only handles pokes: there are no subscriptions or Arvo calls except for the Eyre binding.

<details>

<summary>/app/feature.hoon</summary>

```hoon
/-  feature
/+  dbug, default-agent, server, schooner
/*  feature-ui  %html  /app/feature-ui/html
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  [%0 page=@t]
+$  card  card:agent:gall
--
%-  agent:dbug
^-  agent:gall
=|  state-0
=*  state  -
|_  =bowl:gall
+*  this  .
    def  ~(. (default-agent this %.n) bowl)
++  on-init
  ^-  (quip card _this)
  :_  this(page 'Hello World')
  :~
    :*  %pass  /eyre/connect  %arvo  %e
        %connect  `/apps/feature  %feature
    ==
  ==
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %0  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  |^
  ?+    mark  (on-poke:def mark vase)
      %handle-http-request
    ?>  =(src.bowl our.bowl)
    =^  cards  state
      (handle-http !<([@ta =inbound-request:eyre] vase))
    [cards this]
  ==
  ++  handle-http
    |=  [eyre-id=@ta =inbound-request:eyre]
    ^-  (quip card _state)
    =/  ,request-line:server
      (parse-request-line:server url.request.inbound-request)
    =+  send=(cury response:schooner eyre-id)
    ::
    ?+    method.request.inbound-request  
      [(send [405 ~ [%stock ~]]) state]
      ::
        %'POST'
      ?.  authenticated.inbound-request
        :_  state
        %-  send
        [302 ~ [%login-redirect './apps/feature']]
      ?~  body.request.inbound-request
        [(send [405 ~ [%stock ~]]) state]
      =/  json  (de:json:html q.u.body.request.inbound-request)
      =/  action  (dejs-action +.json)
      (handle-action action) 
      :: 
        %'GET'
      ?+    site  
          :_  state 
          (send [404 ~ [%plain "404 - Not Found"]])
        ::
          [%apps %feature %public ~]
        :_  state
        %-  send
        :+  200  ~  
        :-  %html  page
        ::
          [%apps %feature ~]
        ?.  authenticated.inbound-request
          :_  state
          %-  send
          [302 ~ [%login-redirect './apps/feature']]
        :_  state
        %-  send
        :+  200  ~  
        :-  %html  feature-ui
      == 
    ==
  ::
  ++  dejs-action
    =,  dejs:format
    |=  jon=json
    ^-  action:feature
    %.  jon
    %-  of
    :~  new-page+so
    ==
  ::
  ++  handle-action
    |=  =action:feature
    ^-  (quip card _state)
    ?-    -.action
        %new-page
      ?>  =(src.bowl our.bowl)
      `state(page html:action)
    ==
  --
++  on-peek  on-peek:def
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?+    path  (on-watch:def path)
      [%http-response *]
    `this
  ==
::
++  on-leave  on-leave:def
++  on-agent  on-agent:def
++  on-arvo  on-arvo:def
++  on-fail  on-fail:def
--
```

</details>

### Pokes <a href="#pokes" id="pokes"></a>

`+on-poke` only responds to `%handle-http-request`, which is dealt with in a `|^` barket core.

The most interesting part of the whole app is the `+handle-http` arm:

<details>

<summary>+handle-http</summary>

```hoon
++  handle-http
    |=  [eyre-id=@ta =inbound-request:eyre]
    ^-  (quip card _state)
    =/  ,request-line:server
      (parse-request-line:server url.request.inbound-request)
    =+  send=(cury response:schooner eyre-id)
    ::
    ?+    method.request.inbound-request  
      [(send [405 ~ [%stock ~]]) state]
      ::
        %'POST'
      ?.  authenticated.inbound-request
        :_  state
        %-  send
        [302 ~ [%login-redirect './apps/feature']]
      ?~  body.request.inbound-request
        [(send [405 ~ [%stock ~]]) state]
      =/  json  (de:json:html q.u.body.request.inbound-request)
      =/  action  (dejs-action +.json)
      (handle-action action) 
      :: 
        %'GET'
      ?+    site  
          :_  state 
          (send [404 ~ [%plain "404 - Not Found"]])
        ::
          [%apps %feature %public ~]
        :_  state
        %-  send
        :+  200  ~  
        :-  %html  page
        ::
          [%apps %feature ~]
        ?.  authenticated.inbound-request
          :_  state
          %-  send
          [302 ~ [%login-redirect './apps/feature']]
        :_  state
        %-  send
        :+  200  ~  
        :-  %html  feature-ui
        ::
        ::    [%apps %feature %state ~]
        ::  :_  state
        ::  %-  send
        ::  :+  200  ~ 
        ::  [%json (enjs-state +.state)]
      == 
    ==
```

</details>

This arm uses the server and schooner libraries to produce a response of a server state and associated data. HTTP requests to `/apps/feature` are checked for login authentication, while `/apps/feature/public` are not.

### `POST` <a href="#post" id="post"></a>

In response to a `POST` request, the default page in the state can be changed. This is the only state change supported by the agent.

### `GET` <a href="#get" id="get"></a>

A `GET` request defaults to a `404` error.

* `/apps/feature/public` returns `200` success and the default page in the state.
* `/apps/feature` returns `200` success and the target page, statically compiled on agent build.

### `/lib/schooner` <a href="#libschooner" id="libschooner"></a>

The [Schooner library](https://github.com/dalten-collective/schooner/) simplifies raw HTTP handling for Gall agents, in particular for MIME returns.


# Serving a JS Game

Tutorial on connecting a JavaScript Flappy Bird game to an Urbit backend with leaderboard and file serving.

In this tutorial, we will take an off-the-shelf JavaScript game which runs in the browser and connect it to an Urbit back-end. This page assumes that you have completed some version of [Hoon School](/build-on-urbit/hoon-school) and [App School](/build-on-urbit/app-school). Our goal is to show you one way of directly serving client code from an Urbit ship as server.

*Flappy Bird* is an "insanely irritating, difficult and frustrating game which combines a super-steep difficulty curve with bad, boring graphics and jerky movement" ([Huffington Post](https://web.archive.org/web/20140205084251/http://www.huffingtonpost.com/2014/02/03/flappy-bird-tips_n_4717406.html)). We are going to implement `%flap`, a *Flappy Bird* leaderboard using \~paldev’s `%pals` peer tracking agent. The approach given in this tutorial will apply to any game which is primarily run in the browser and has some persistent state to retain across sessions or communicate between players at discrete intervals. Direct player-v.-player games will require other techniques to implement.

Our objective is to illustrate a minimum viable set of changes necessary to implement the task. We should the following components when complete:

1. A front end. We will start with the JS browser app and make some adjustments so it can communicate with Urbit and display shared results.
2. A data model. Structure and mark files will allow the components of the system to seamlessly communicate.
3. A back end. Urbit will serve as the database for storing and propagating scores. Urbit will also serve the front end.
4. A communications protocol. The leaderboard will need to know who to watch and track as peers. We will utilize \~paldev's `%pals` contact list.

We will conceive of this app's communications structure as consisting of a *vertical* component (which is the communication between the client in the browser and the Urbit ship as database) and a *horizontal* component (which is the communication between Urbit peers). Vertical communication will take place using JSON via the `%flap-action` mark, while horizontal communication will take place using the `%flap-update` mark. Apps can achieve good data modularity using this separation.

![](https://media.urbit.org/developers/vert-horz.svg)

## Desk Setup <a href="#desk-setup" id="desk-setup"></a>

As with all Urbit development, you should set up a development ship. In this case, it will be more convenient to have this ship be on the live network: a comet or a moon. Below, we refer to this as `comet`.

On that ship, `|install ~paldev %pals`. Optionally, download \~paldev's [Suite repo](https://github.com/Fang-/suite) in case you need to refer to `%pals`-related code.

Download the [Flappy Bird repo](https://github.com/CodeExplainedRepo/Original-Flappy-bird-JavaScript/). Although we will follow this clone, other versions and other games follow a similar structure and the below approach is highly transferrable.

At this point, you should have a directory structure containing the following:

```
├── Original-Flappy-bird-JavaScript/
├── suite/
└── comet/
```

We now need to create a new clean desk in the development ship. In the Dojo:

```hoon
|mount %base
|mount %landscape
|merge %flap our %base
|mount %flap
```

Back on Earth:

```sh
rm -rf comet/flap/*
echo "~[%flap]" > comet/flap/desk.bill
echo "[%zuse 408]" > comet/flap/sys.kelvin
```

At this point, we need to take stock of what kind of file marks and libraries we need to make available: `kelvin`, `docket-0`, and so forth. While there are marks for `js` and `png`, there is no `wav` so we'll handle that directly.

```sh
# Copy necessary %base files to %flap
cp -r comet/base/lib comet/flap
cp -r comet/base/sur comet/flap
cp -r comet/base/mar comet/flap

# Copy necessary %landscape files to %flap
yes | cp -r comet/landscape/lib comet/flap
yes | cp -r comet/landscape/sur comet/flap
yes | cp -r comet/landscape/mar comet/flap

# Create empty directories for planned files
mkdir -p comet/flap/app/flap
mkdir -p comet/flap/mar/flap

# Include the Schooner library by Quartus/Dalten for HTTP handling.
#git clone https://github.com/dalten-collective/schooner.git
# We actually should instead pull a slightly modified version of `schooner.hoon` pending some upstream changes
git clone https://github.com/hoon-school/schooner.git
cp schooner/lib/schooner.hoon comet/flap/lib
```

Make an appropriate docket file `desk.docket-0`, e.g.:

```hoon
:~
  title+'Flappy Bird'
  info+'An insanely irritating, difficult and frustrating game which combines a super-steep difficulty curve with bad, boring graphics and jerky movement.'
  color+0xea.c124
  version+[0 0 1]
  website+'https://urbit.org'
  license+''
  base+'flap'
  site+/apps/flap
==
```

At this point, your overall directory structure (not showing most of the files) should look like this:

```
├── Original-Flappy-bird-JavaScript/
├── suite/
└── comet/
    ├── base/
    ├── flap/
    │   ├── app/
    │   │   └── flap/
    │   ├── mar/
    │   │   └── flap/
    │   ├── sur/
    │   ├── desk.bill
    │   ├── desk.docket-0
    │   └── sys.kelvin
    └── landscape/
```

`|commit %flap` to include all of these files.

## Front End <a href="#front-end" id="front-end"></a>

If you open `index.html` in `Original-Flappy-bird-JavaScript/` in a web browser, the game should work interactively. Only mouse clicks are recorded as events. The only termination condition is death.

The original authors of this clone, CodeExplained, provide [a walkthrough video](https://youtu.be/0ArCFchlTq4) which explains how the front-end code works in detail. For our purposes now, you don't need to know much JavaScript—just enough to be able to interpret and modify some simple statements and functions.

Since the game is served directly from the `index.html` file, we can simply copy that file into Urbit and have things work. Other files also have appropriate marks except for the sound files which are `wav` files. Since `wav` files are a standard MIME type for web browsers to handle, we don't need to do very much to represent them correctly: by copying `/mar/png.hoon` to `/mar/wav.hoon` and modifying it to present an `/audio/wav` MIME type, you can correctly include

```hoon
|_  dat=@
++  grow
  |%
  ++  mime  [/audio/wav (as-octs:mimes:html dat)]
  --
++  grab
  |%
  ++  mime  |=([p=mite q=octs] q.q)
  ++  noun  @
  --
++  grad  %mime
--
```

With `/mar/wav.hoon` present, you should be able to directly copy in the game content to the Urbit ship:

```sh
cp -r Original-Flappy-bird-JavaScript/* comet/flap/app/flap
```

and `|commit %flap`.

The `index.html` file will still work if you open it in the browser directly, but it doesn't have any connection to Urbit yet. Clay doesn't know where to build everything and hook it up, so at a minimum we have to load and display the front-end using `/app/flap.hoon`.

## Data Model <a href="#data-model" id="data-model"></a>

Different parts of the system need to converge on their shared vision of the world. Thus, `/sur` and `/mar`. We aren't interested in calculating the gameplay mechanics, only in the scores. So we expect to be able to track our state including:

* our current score (last game) (`score`)
* our all-time high score (`hiscore`)
* the all-time high score of our `%pals` (`scores`)

We don't need to actively track friends *except* that they will have entries in `scores`, even if zero.

**`/sur/flap.hoon`**:

The basic structure file defines friendship, which it will derive from `%pals`, and scores. Scores are simple, so they're just a matter of a single `@ud` number.

We `%gain` a `score` at the end of each game by an `%action`, and track our own `hiscore`. We `%lord` a high score over others (or they over us) by sending and receiving `%update`s. (So `%action`s are vertical between client and server, while `%update`s are horizontal between servers.)

```hoon
|%
+$  fren    @p
+$  score   @ud
+$  scores  (map fren score)
::
+$  action
  $%  [%gain =score]
  ==
::
+$  update
  $%  [%lord =score =fren]
  ==
--
```

**`/mar/flap/action.hoon`**:

Actions are sent from the client to the Urbit ship. An incoming JSON will be of the form

```json
{
  "gain": {
    "score": 15
  }
}
```

Given an action to `%gain` a score as a JSON, we process it in the mark and yield it as a `%flap-action`.

```hoon
/-  flap
|_  =action:flap
++  grab
  |%
  ++  noun  action:flap
  ++  json
    =,  dejs:format
    |=  jon=json
    ^-  action
    %.  jon
    %-  of
    :~  [%gain (ot ~[score+ni])]
    ==
  --
++  grow
  |%
  ++  noun  action
  --
++  grad  %noun
--
```

**`/mar/flap/update.hoon`**:

Updates are sent between Urbit peers.

```hoon
/-  flap
|_  =update:flap
++  grab
  |%
  ++  noun  update:flap
  --
++  grow
  |%
  ++  noun  update
  --
++  grad  %noun
--
```

## Back End <a href="#back-end" id="back-end"></a>

The main app implements the logic for exposing and tracking data.

<details>

<summary>/app/flap.hoon (version 1)</summary>

```hoon
  ::  flap.hoon
::::  Maintains leaderboard for Flappy Bird on Mars.
::
/-  *flap
/+  default-agent               :: agent arm defaults
/+  dbug                        :: debug wrapper for agent
/+  schooner                    :: HTTP request handling
/+  server                      :: HTTP request processing
/+  verb                        :: support verbose output for agent
/*  flapui      %html  /app/flap/index/html
/*  flapjs      %js    /app/flap/game/js
/*  flapsprite  %png   /app/flap/img/sprite/png
/*  flapaudios  %wav   /app/flap/audio/sfx-point/wav
/*  flapaudiof  %wav   /app/flap/audio/sfx-flap/wav
/*  flapaudioh  %wav   /app/flap/audio/sfx-hit/wav
/*  flapaudiow  %wav   /app/flap/audio/sfx-swooshing/wav
/*  flapaudiod  %wav   /app/flap/audio/sfx-die/wav
|%
+$  versioned-state
  $%  state-zero
  ==
+$  state-zero  $:
      %zero
      =score
      hiscore=score
    ==
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-zero
=*  state  -
%+  verb  |
^-  agent:gall
|_  bol=bowl:gall
+*  this     .
    default  ~(. (default-agent this %.n) bol)
::
++  on-init
  ^-  (quip card _this)
  ~&  >  "%flap initialized successfully."
  :_  this
  :~  [%pass /eyre %arvo %e %connect [~ /apps/flap] %flap]
  ==
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %zero  `this(state old)
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?>  =(our src):bol
  |^
  ?+    mark  (on-poke:default mark vase)
    ::
      %flap-action
    =/  axn  !<(action vase)
    ?>  ?=(%gain -.axn)
    ?.  (gth score.axn hiscore)
      `this(score score.axn)
    `this(score score.axn, hiscore score.axn)
    ::
      %handle-http-request
    (handle-http !<([@ta =inbound-request:eyre] vase))
  ==
  ::
  ++  handle-http
    |=  [eyre-id=@ta =inbound-request:eyre]
    ^-  (quip card _this)
    =/  ,request-line:server
      (parse-request-line:server url.request.inbound-request)
    =+  send=(cury response:schooner eyre-id)
    ?.  authenticated.inbound-request
      :_  this
      %-  send
      [302 ~ [%login-redirect './apps/flap']]
    ::
    ?+    method.request.inbound-request
      [(send [405 ~ [%stock ~]]) this]
      ::
        %'POST'
      ?~  body.request.inbound-request
        [(send [405 ~ [%stock ~]]) this]
      =/  json  (de:json:html q.u.body.request.inbound-request)
      =/  axn  `action`(dejs-action +.json)
      (on-poke %flap-action !>(axn))
      ::
        %'GET'
      ?+  site  :_  this
                %-  send
                :+  404
                  ~
                [%plain "404 - Not Found"]
          [%apps %flap ~]
        :_  this
        %-  send
        :+  200
          ~
        [%html flapui]
        ::
          [%apps %flap %whoami ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (scow %p our.bol)]
        ::
          [%apps %flap %score ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (scow %ud score)]
        ::
          [%apps %flap %hiscore ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (scow %ud hiscore)]
        ::
          [%apps %flap %game %js ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (trip flapjs)]
        ::
          [%apps %flap %img %sprite %png ~]
        :_  this
        %-  send
        :+  200
          ~
        [%image-png flapsprite]
        ::
          [%apps %flap %audio %sfx-point %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudios]
        ::
          [%apps %flap %audio %sfx-flap %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudiof]
        ::
          [%apps %flap %audio %sfx-hit %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudioh]
        ::
          [%apps %flap %audio %sfx-swooshing %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudiow]
        ::
          [%apps %flap %audio %sfx-die %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudiod]
      ==
    ==
  ++  dejs-action
    =,  dejs:format
    |=  jon=json
    ^-  action
    %.  jon
    %-  of
    :~  [%gain (ot ~[score+ni])]
    ==
  --
::
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?+    path  (on-watch:default path)
      [%http-response *]
    ?:  =(our src):bol
      `this
    (on-watch:default path)
  ==
::
++  on-leave  on-leave:default
::
++  on-peek  on-peek:default
::
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:default wire sign)
      [%flap ~]
    ?+    -.sign  (on-agent:default wire sign)
      ::
        %fact
      ?+    p.cage.sign  (on-agent:default wire sign)
          %flap-update
        =/  upd  !<(update q.cage.sign)
        ?-    -.upd
            %lord
          !!
        ==
      ==
      ::
        %kick
      :_  this
      :~  [%pass /flap %agent [src.bol %flap] %watch /updates/out]
      ==
      ::
        %watch-ack
      ?~  p.sign
        ((slog '%flap: Subscribe succeeded!' ~) `this)
      ((slog '%flap: Subscribe failed!' ~) `this)
    ==
  ==
::
++  on-arvo
|=  [=wire =sign-arvo]
  ^-  (quip card _this)
  ?.  ?=([%eyre %bound *] sign-arvo)
    (on-arvo:default [wire sign-arvo])
  ?:  accepted.sign-arvo
    %-  (slog leaf+"/apps/flap bound successfully!" ~)
    `this
  %-  (slog leaf+"Binding /apps/flap failed!" ~)
  `this
::
++  on-fail   on-fail:default
--
```

</details>

Then `|install our %flap` to install the app.

Now when we navigate to `localhost:8080/apps/flap`, what do we see? The game canvas is merely an empty box. What can we do to fix this?

### Serving Correctly <a href="#serving-correctly" id="serving-correctly"></a>

If we investigate the Developer Tools console in our browser, we see messages to the effect that resources are unable to be located. Resource paths (for `js`, `png`, and `wav` files) tell the browser from whence the resources will come when they are loaded. We have two options here as well: hot-link the resource from its GitHub or other source or serve the resource from Urbit.

If we hot-link the resources, the corresponding lines will look like this:

```js
const sprite = new Image();
sprite.src = "https://raw.githubusercontent.com/CodeExplainedRepo/Original-Flappy-bird-JavaScript/master/img/sprite.png";
```

This is easiest if less elegant than serving the files from Urbit.

If we want to serve our files from Urbit, we need to build the files and serve them to particular endpoints so that they are visible to the browser. This means importing and serving these to make these available. It also means renaming the files so that they are compatible with `@ta`-style path entries.

```sh
mv comet/flap/app/flap/audio/sfx_point.wav comet/flap/app/flap/audio/sfx-point.wav
mv comet/flap/app/flap/audio/sfx_flap.wav comet/flap/app/flap/audio/sfx-flap.wav
mv comet/flap/app/flap/audio/sfx_hit.wav comet/flap/app/flap/audio/sfx-hit.wav
mv comet/flap/app/flap/audio/sfx_swooshing.wav comet/flap/app/flap/audio/sfx-swooshing.wav
mv comet/flap/app/flap/audio/sfx_die.wav comet/flap/app/flap/audio/sfx-die.wav
```

The import lines at the top of `/app/flap.hoon` build each file according to its mark:

```hoon
/*  flapjs  %js    /app/flap/game/js
/*  flapsprite  %png   /app/flap/img/sprite/png
/*  flapaudios  %wav   /app/flap/audio/sfx-point/wav
/*  flapaudiof  %wav   /app/flap/audio/sfx-flap/wav
/*  flapaudioh  %wav   /app/flap/audio/sfx-hit/wav
/*  flapaudiow  %wav   /app/flap/audio/sfx-swooshing/wav
/*  flapaudiod  %wav   /app/flap/audio/sfx-die/wav
```

Later in `/app/flap.hoon` we serve the files at particular endpoints:

<details>

<summary>file serving code</summary>

```hoon
  [%apps %flap %game %js ~]
:_  this
%-  send
:+  200
  ~
[%plain (trip flapjs)]
::
  [%apps %flap %img %sprite %png ~]
:_  this
%-  send
:+  200
  ~
[%image-png flapsprite]
::
  [%apps %flap %audio %sfx-point %wav ~]
:_  this
%-  send
:+  200
  ~
[%audio-wav flapaudios]
::
  [%apps %flap %audio %sfx-flap %wav ~]
:_  this
%-  send
:+  200
  ~
[%audio-wav flapaudiof]
::
  [%apps %flap %audio %sfx-hit %wav ~]
:_  this
%-  send
:+  200
  ~
[%audio-wav flapaudioh]
::
  [%apps %flap %audio %sfx-swooshing %wav ~]
:_  this
%-  send
:+  200
  ~
[%audio-wav flapaudiow]
::
  [%apps %flap %audio %sfx-die %wav ~]
:_  this
%-  send
:+  200
  ~
[%audio-wav flapaudiod]
```

</details>

In `index.html`:

```html
<script src="flap/game/js"></script>
```

In `game.js`, all we need to change are the source paths:

```js
// LOAD SPRITE IMAGE
const sprite = new Image();
sprite.src = "flap/img/sprite/png";

// LOAD SOUNDS
const SCORE_S = new Audio();
SCORE_S.src = "flap/audio/sfx-point/wav";

const FLAP = new Audio();
FLAP.src = "flap/audio/sfx-flap/wav";

const HIT = new Audio();
HIT.src = "flap/audio/sfx-hit/wav";

const SWOOSHING = new Audio();
SWOOSHING.src = "flap/audio/sfx-swooshing/wav";

const DIE = new Audio();
DIE.src = "flap/audio/sfx-die/wav";
```

Flappy Bird in HTML+JS needs Urbit affordances so it knows how to talk to the app backend. Changes like the above will be common in adapting non-Hoon-compliant code due to assumptions about file location and naming conventions, but most of the time your process of adaptation will look much like the above.

You can check out the API endpoints listed in the code above: `/apps/flap/whoami`, `/apps/flap/img/sprite/png` and so forth, to see what they serve to the browser as.

This version of the app should run in your browser correctly once you `|commit %flap`. Urbit knows where all resources are and how to serve them. However, while the vertical communications work (between client and server), the horizontal communications are still missing (between Urbit peers).

> ### Cross-Origin Resource Sharing
>
> CORS is a security policy that allows you to load resources dynamically with prior approval from the browser and server. Sometimes when you are serving various game configurations during development, you may arrive a situation in which things aren't loading correctly to due to the CORS policy. (When things are entirely served from your Urbit ship then this shouldn't be an issue.) If you encounter this problem on your way, however, you can [set up CORS origins](/urbit-os/kernel/eyre/guide#managing-cors-origins) for Eyre by telling your Urbit ship to allow `localhost` files to be served on the appropriate port.
>
> ```hoon
> |eyre/cors/approve 'http://localhost:8080'
> ```

## Communications Protocol <a href="#communications-protocol" id="communications-protocol"></a>

### Changes to Back End <a href="#changes-to-back-end" id="changes-to-back-end"></a>

#### Urbit as Database

Our app's state should retain a list of *all* high scores it receives. To that end, we will transition from a single high-score state to using a `(map fren score)`.

```hoon
/-  *flap, pals
::  * * *
::
|%
+$  versioned-state
  $%  state-zero
      state-one
  ==
+$  state-one  $:
      %one
      =score
      =scores
    ==
+$  state-zero  $:
      %zero
      =score
      hiscore=score
    ==
+$  card  card:agent:gall
--
```

We will include an upgrade path for the old agent:

```hoon
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %one   `this(state old)
    %zero  `this(scores (~(put by scores) our.bol hiscore.old))
  ==
```

These values will be changed when peer state changes are received using `%pals`.

#### Adding Friends

`%pals` is a very simple contact manager which recognizes outgoing requests, incoming requests, and mutually recognized peers. Let's integrate knowledge of other friends (and thus the ability to maintain a leaderboard of our friends). We will base our leaderboard on outgoing requests for simplicity.

You should copy `%pals` support files over to our working desk.

```sh
# Include the %pals tooling by ~paldev.
git clone https://github.com/Fang-/suite.git
cp suite/lib/pals.hoon comet/flap/lib
cp suite/sur/pals.hoon comet/flap/sur
cp -r suite/mar/pals comet/flap/mar
```

When dealing with `%pals`, we need to maintain some list of our friends and their current known high scores (which means modifying the app's state) and dealing with sending and receiving notifications. That is reflected in adding `scores` to the new state in the state definition in `/app/flap.hoon`.

Subscriptions will all take place over the `/flap` path. In the `+on-poke` arm, we need to issue notices along that arm with the `%flap-update` mark.

```hoon
  %flap-action
=/  axn  !<(action vase)
?>  ?=(-.axn %gain)
?.  (gth score.axn hiscore)
  `this(score score.axn)
:_  this(score score.axn, hiscore score.axn)
:~  [%give %fact ~[/flap] %flap-update !>(`update`lord+[score=score.axn fren=our.bol])]
==
```

We also need to process incoming `%fact`s with a `%flap-update` cage in `+on-agent`:

```hoon
  %flap-update
=/  upd  !<(update q.cage.sign)
?>  ?=(-.upd %lord)
?:  (gth (~(got by scores) fren.upd) score.upd)
  `this
~&  >  "%flap:  new high score {<score.upd>} from {<fren.upd>}"
`this(scores (~(put by scores) fren.upd score.upd))
```

Changes to `%pals` come in along the path `/newpals`, so we need to watch for incoming values there. `%pals` organizes friends to `%meet` for the first time or `%part` when they separate. Thus we add or remove entries from our `scores.state`.

```hoon
  [%newpals ~]
?+    -.sign  `this
    %fact
  ?+    p.cage.sign  `this
      %pals-effect
    =/  fx  !<(effect:pals q.cage.sign)
    ?+    -.fx  (on-agent:default wire sign)
        %meet
      :_  this(scores (~(put by scores) +.fx 0))
      :~  [%pass /flap %agent [+.fx %flap] %watch /flap]
      ==
        %part
      :_  this(scores (~(del by scores) +.fx))
      :~  [%pass /flap %agent [+.fx %flap] %leave ~]
      ==
    ==
  ==
    %kick
  :_  this
  :~  [%pass /flap %agent [src.bol dap.bol] %watch /flap]
  ==
==
```

With all of the above, you should have a working `%flappy` instance at `http://localhost:8080/apps/flappy`. Use `:flappy +dbug` to check that the score is being communicated back.

<details>

<summary>/app/flap.hoon (version 2)</summary>

```hoon
  ::  flap.hoon
::::  Maintains leaderboard for Flappy Bird on Mars.
::
/-  *flap, pals
/+  default-agent               :: agent arm defaults
/+  dbug                        :: debug wrapper for agent
/+  schooner                    :: HTTP request handling
/+  server                      :: HTTP request processing
/+  verb                        :: support verbose output for agent
/*  flapui      %html  /app/flap/index/html
/*  flapjs      %js    /app/flap/game/js
/*  flapsprite  %png   /app/flap/img/sprite/png
/*  flapaudios  %wav   /app/flap/audio/sfx-point/wav
/*  flapaudiof  %wav   /app/flap/audio/sfx-flap/wav
/*  flapaudioh  %wav   /app/flap/audio/sfx-hit/wav
/*  flapaudiow  %wav   /app/flap/audio/sfx-swooshing/wav
/*  flapaudiod  %wav   /app/flap/audio/sfx-die/wav
|%
+$  versioned-state
  $%  state-zero
      state-one
  ==
+$  state-one  $:
      %one
      =score
      =scores
      hiscore=score
    ==
+$  state-zero  $:
      %zero
      =score
      hiscore=score
    ==
+$  card  card:agent:gall
--
%-  agent:dbug
=|  state-one
=*  state  -
%+  verb  |
^-  agent:gall
|_  bol=bowl:gall
+*  this     .
    default  ~(. (default-agent this %.n) bol)
::
++  on-init
  ^-  (quip card _this)
  ~&  >  "%flap initialized successfully."
  :_  this
  :~  [%pass /eyre %arvo %e %connect [~ /apps/flap] %flap]
  ==
::
++  on-save
  ^-  vase
  !>(state)
::
++  on-load
  |=  old-state=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state old-state)
  ?-  -.old
    %one   `this(state old)
    %zero  `this(scores (~(put by scores) our.bol hiscore.old))
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?>  =(our src):bol
  |^
  ?+    mark  (on-poke:default mark vase)
    ::
      %flap-action
    =/  axn  !<(action vase)
    ?>  ?=(-.axn %gain)
    ?.  (gth score.axn (~(gut by scores) our.bol 0))
      `this(score score.axn)
    :_  this(score score.axn, scores (~(put by scores) our.bol score.axn))
    :~  [%give %fact ~[/flap] %flap-update !>(`update`lord+[score=score.axn fren=our.bol])]
    ==
    ::
      %handle-http-request
    (handle-http !<([@ta =inbound-request:eyre] vase))
  ==
  ::
  ++  handle-http
    |=  [eyre-id=@ta =inbound-request:eyre]
    ^-  (quip card _this)
    =/  ,request-line:server
      (parse-request-line:server url.request.inbound-request)
    =+  send=(cury response:schooner eyre-id)
    ?.  authenticated.inbound-request
      :_  this
      %-  send
      [302 ~ [%login-redirect './apps/flap']]
    ::
    ?+    method.request.inbound-request
      [(send [405 ~ [%stock ~]]) this]
      ::
        %'POST'
      ?~  body.request.inbound-request
        [(send [405 ~ [%stock ~]]) this]
      =/  json  (de:json:html q.u.body.request.inbound-request)
      =/  axn  `action`(dejs-action +.json)
      (on-poke %flap-action !>(axn))
      ::
        %'GET'
      ?+  site  :_  this
                %-  send
                :+  404
                  ~
                [%plain "404 - Not Found"]
          [%apps %flap ~]
        :_  this
        %-  send
        :+  200
          ~
        [%html flapui]
        ::
          [%apps %flap %whoami ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (scow %p our.bol)]
        ::
          [%apps %flap %score ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (scow %ud score)]
        ::
          [%apps %flap %hiscore ~]
        :_  this
        %-  send
        :+  200
          ~
        [%plain (scow %ud (~(gut by scores) our.bol 0))]
        ::
          [%apps %flap %game %js ~]
        :_  this
        %-  send
        :+  200
          ~
        [%application-javascript (trip flapjs)]
        ::
          [%apps %flap %img %sprite %png ~]
        :_  this
        %-  send
        :+  200
          ~
        [%image-png flapsprite]
        ::
          [%apps %flap %audio %sfx-point %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudios]
        ::
          [%apps %flap %audio %sfx-flap %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudiof]
        ::
          [%apps %flap %audio %sfx-hit %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudioh]
        ::
          [%apps %flap %audio %sfx-swooshing %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudiow]
        ::
          [%apps %flap %audio %sfx-die %wav ~]
        :_  this
        %-  send
        :+  200
          ~
        [%audio-wav flapaudiod]
        ::
          [%apps %flap %frens ~]
        :_  this
        %-  send
        :+  200
          ~
        [%json (enjs-scores scores)]
      ==
    ==
  ++  dejs-action
    =,  dejs:format
    |=  jon=json
    ^-  action
    %.  jon
    %-  of
    :~  [%gain (ot ~[score+ni])]
    ==
  ++  enjs-scores
    =,  enjs:format
    |=  =^scores
    ^-  json
    :-  %a
    :*
    %+  turn  ~(tap by scores)
    |=  point=[@p @ud]
    %-  pairs
    :~  ['fren' s+(scot %p -.point)]
        ['score' (numb +.point)]
    ==  ==
  --
::
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?+    path  (on-watch:default path)
      [%http-response *]
    ?:  =(our src):bol
      `this
    (on-watch:default path)
  ==
::
++  on-leave  on-leave:default
::
++  on-peek  on-peek:default
::
++  on-agent
  |=  [=wire =sign:agent:gall]
  ^-  (quip card _this)
  ?+    wire  (on-agent:default wire sign)
      [%flap ~]
    ?+    -.sign  (on-agent:default wire sign)
      ::
        %fact
      ?+    p.cage.sign  (on-agent:default wire sign)
          %flap-update
        =/  upd  !<(update q.cage.sign)
        ?>  ?=(-.upd %lord)
        =/  hiscore  (~(gut by `(map @p @ud)`scores) fren.upd 0)
        ?:  (gth hiscore score.upd)
          `this
        ~&  >  "%flap:  new high score {<score.upd>} from {<fren.upd>}"
        `this(scores (~(put by scores) fren.upd score.upd))
      ==
      ::
        %kick
      :_  this
      :~  [%pass /flap %agent [src.bol %flap] %watch /updates/out]
      ==
      ::
        %watch-ack
      ?~  p.sign
        ((slog '%flap: Subscribe succeeded!' ~) `this)
      ((slog '%flap: Subscribe failed!' ~) `this)
    ==
    ::
      [%newpals ~]
    ?+    -.sign  `this
        %fact
      ?+    p.cage.sign  `this
          %pals-effect
        =/  fx  !<(effect:pals q.cage.sign)
        ?+    -.fx  (on-agent:default wire sign)
            %meet
          :_  this(scores (~(put by scores) +.fx 0))
          :~  [%pass /flap %agent [+.fx %flap] %watch /flap]
          ==
            %part
          :_  this(scores (~(del by scores) +.fx))
          :~  [%pass /flap %agent [+.fx %flap] %leave ~]
          ==
        ==
      ==
    ==
      %kick
    :_  this
    :~  [%pass /flap %agent [src.bol dap.bol] %watch /flap]
    ==
  ==
::
++  on-arvo
|=  [=wire =sign-arvo]
  ^-  (quip card _this)
  ?.  ?=([%eyre %bound *] sign-arvo)
    (on-arvo:default [wire sign-arvo])
  ?:  accepted.sign-arvo
    %-  (slog leaf+"/apps/flap bound successfully!" ~)
    `this
  %-  (slog leaf+"Binding /apps/flap failed!" ~)
  `this
::
++  on-fail   on-fail:default
--
```

</details>

### Changes to Front End <a href="#changes-to-front-end" id="changes-to-front-end"></a>

Now that the a leaderboard is supported, we need a way to display it alongside the browser game. To wit, we will add some `async` functions to retrieve key bits of information from the ship and display it in a table.

```js
// URBIT STATE
async function getmyship() {
        const response = await fetch('/apps/flap/whoami');
        return response.text();
    }
var myshipname = await getmyship();
document.getElementById("ship").innerHTML = myshipname;

async function gethiscore() {
        const response = await fetch('/apps/flap/hiscore');
        return response.text();
    }
var myhiscore = await gethiscore();

async function getscore() {
        const response = await fetch('/apps/flap/score');
        return response.text();
    }
var myscore = await getscore();

document.getElementById("hiscore").innerHTML = myhiscore;
document.getElementById("score").innerHTML = myscore;
```

along with similar modest changes in the final file (included below in its entirety). E.g. on death we modify some value we placed in tags in `index.html`:

```js
if(state.current == state.game){
    state.current = state.over;
    sendscore(score);
    document.getElementById("hiscore").innerHTML = myhiscore;
    document.getElementById("score").innerHTML = myscore;
    drawtable(myfrens);
}
```

We'll also include a way to post data back as a `flap-action`.

```js
//  Send score to Gall agent
function sendscore(score) {
    fetch('/apps/flap', {
        method: 'POST',
        body: JSON.stringify({'gain': {'score': score.value}})
    })
}
```

If you examine `+on-poke` in `/app/flap.hoon`, you will see that HTTP `POST` requests are accepted, examined, and if valid passed back through as a self-poke with appropriate mark.

```hoon
  %'POST'
?~  body.request.inbound-request
  [(send [405 ~ [%stock ~]]) this]
=/  json  (de:json:html q.u.body.request.inbound-request)
=/  axn  `action`(dejs-action +.json)
(on-poke %flap-action !>(axn))
```

<details>

<summary>/app/flap/index.html (version 2)</summary>

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Original Flappy Bird -JavaScript</title>
    <link href="https://fonts.googleapis.com/css?family=Teko:700" rel="stylesheet">
    <style>        
        canvas{
            border: 1px solid #000;
            display: block;
            margin: 0 auto;
        }
    </style>
</head>
<body>
    <canvas id="bird" width="320" height="480"></canvas>

    <p>
    <span id="ship"></span>:  <span id="score"></span>/<span id="hiscore"></span>
    </p>
    
    <p>
    <div id="frens"></div>
    </p>
        
    <script src="flap/game/js" type="module"></script>
</body>
</html>
```

</details>

<details>

<summary>/app/flap/game.js (version 2)</summary>

```javascript
// URBIT STATE
async function getmyship() {
    const response = await fetch('/apps/flap/whoami');
    return response.text();
}
var myshipname = await getmyship();
document.getElementById("ship").innerHTML = myshipname;

async function gethiscore() {
    const response = await fetch('/apps/flap/hiscore');
    return response.text();
}
var myhiscore = await gethiscore();

document.getElementById("hiscore").innerHTML = myhiscore;

async function getscore() {
    const response = await fetch('/apps/flap/score');
    return response.text();
}
var myscore = await getscore();
document.getElementById("score").innerHTML = myscore;

//  Send score to Gall agent
function sendscore(score) {
fetch('/apps/flap', {
    method: 'POST',
    body: JSON.stringify({'gain': {'score': score.value}})
})
}

//  Draw table of frens
async function getfrens() {
    const response = await fetch('/apps/flap/frens');
    return response.text();
}
function drawtable(myfrens) {
console.log(myfrens);
var frens = JSON.parse(myfrens);

var table = document.createElement("table");
var titleRow = table.insertRow();
var frenCell = titleRow.insertCell();
frenCell.innerHTML = "Ship";
var scoreCell = titleRow.insertCell();
scoreCell.innerHTML = "Score";

for (let key in frens) {
    var row = table.insertRow();
    var cell = row.insertCell();
    cell.classList += "ship";
    cell.innerHTML = frens[key]['fren'];
    cell = row.insertCell();
    cell.innerHTML = frens[key]['score'];
}

// Clear the old table
const list = document.getElementById("frens");
while (list.hasChildNodes()) {
    list.removeChild(list.firstChild);
}
// Add the new one
document.getElementById("frens").appendChild(table);
}
var myfrens = await getfrens();
console.log(myfrens);
drawtable(myfrens);

// SELECT CVS
const cvs = document.getElementById("bird");
const ctx = cvs.getContext("2d");

// GAME VARS AND CONSTS
let frames = 0;
const DEGREE = Math.PI/180;

// LOAD SPRITE IMAGE
const sprite = new Image();
sprite.src = "flap/img/sprite/png";

// LOAD SOUNDS
const SCORE_S = new Audio();
SCORE_S.src = "flap/audio/sfx-point/wav";

const FLAP = new Audio();
FLAP.src = "flap/audio/sfx-flap/wav";

const HIT = new Audio();
HIT.src = "flap/audio/sfx-hit/wav";

const SWOOSHING = new Audio();
SWOOSHING.src = "flap/audio/sfx-swooshing/wav";

const DIE = new Audio();
DIE.src = "flap/audio/sfx-die/wav";

// GAME STATE
const state = {
    current : 0,
    getReady : 0,
    game : 1,
    over : 2
}

// START BUTTON COORD
const startBtn = {
    x : 120,
    y : 263,
    w : 83,
    h : 29
}

// CONTROL THE GAME
cvs.addEventListener("click", function(evt){
    switch(state.current){
        case state.getReady:
            state.current = state.game;
            SWOOSHING.play();
            break;
        case state.game:
            if(bird.y - bird.radius <= 0) return;
            bird.flap();
            FLAP.play();
            break;
        case state.over:
            let rect = cvs.getBoundingClientRect();
            let clickX = evt.clientX - rect.left;
            let clickY = evt.clientY - rect.top;
            
            // CHECK IF WE CLICK ON THE START BUTTON
            if(clickX >= startBtn.x && clickX <= startBtn.x + startBtn.w && clickY >= startBtn.y && clickY <= startBtn.y + startBtn.h){
                pipes.reset();
                bird.speedReset();
                score.reset();
                state.current = state.getReady;
            }
            break;
    }
});


// BACKGROUND
const bg = {
    sX : 0,
    sY : 0,
    w : 275,
    h : 226,
    x : 0,
    y : cvs.height - 226,
    
    draw : function(){
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);
        
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x + this.w, this.y, this.w, this.h);
    }
    
}

// FOREGROUND
const fg = {
    sX: 276,
    sY: 0,
    w: 224,
    h: 112,
    x: 0,
    y: cvs.height - 112,
    
    dx : 2,
    
    draw : function(){
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);
        
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x + this.w, this.y, this.w, this.h);
    },
    
    update: function(){
        if(state.current == state.game){
            this.x = (this.x - this.dx)%(this.w/2);
        }
    }
}

// BIRD
const bird = {
    animation : [
        {sX: 276, sY : 112},
        {sX: 276, sY : 139},
        {sX: 276, sY : 164},
        {sX: 276, sY : 139}
    ],
    x : 50,
    y : 150,
    w : 34,
    h : 26,
    
    radius : 12,
    
    frame : 0,
    
    gravity : 0.25,
    jump : 4.6,
    speed : 0,
    rotation : 0,
    
    draw : function(){
        let bird = this.animation[this.frame];
        
        ctx.save();
        ctx.translate(this.x, this.y);
        ctx.rotate(this.rotation);
        ctx.drawImage(sprite, bird.sX, bird.sY, this.w, this.h,- this.w/2, - this.h/2, this.w, this.h);
        
        ctx.restore();
    },
    
    flap : function(){
        this.speed = - this.jump;
    },
    
    update: function(){
        // IF THE GAME STATE IS GET READY STATE, THE BIRD MUST FLAP SLOWLY
        this.period = state.current == state.getReady ? 10 : 5;
        // WE INCREMENT THE FRAME BY 1, EACH PERIOD
        this.frame += frames%this.period == 0 ? 1 : 0;
        // FRAME GOES FROM 0 To 4, THEN AGAIN TO 0
        this.frame = this.frame%this.animation.length;
        
        if(state.current == state.getReady){
            this.y = 150; // RESET POSITION OF THE BIRD AFTER GAME OVER
            this.rotation = 0 * DEGREE;
        }else{
            this.speed += this.gravity;
            this.y += this.speed;
            
            if(this.y + this.h/2 >= cvs.height - fg.h){
                this.y = cvs.height - fg.h - this.h/2;
                if(state.current == state.game){
                    state.current = state.over;
                    sendscore(score);
                    document.getElementById("hiscore").innerHTML = myhiscore;
                    document.getElementById("score").innerHTML = myscore;
                    drawtable(myfrens);
                }
            }
            
            // IF THE SPEED IS GREATER THAN THE JUMP MEANS THE BIRD IS FALLING DOWN
            if(this.speed >= this.jump){
                this.rotation = 90 * DEGREE;
                this.frame = 1;
            }else{
                this.rotation = -25 * DEGREE;
            }
        }
        
    },
    speedReset : function(){
        this.speed = 0;
    }
}

// GET READY MESSAGE
const getReady = {
    sX : 0,
    sY : 228,
    w : 173,
    h : 152,
    x : cvs.width/2 - 173/2,
    y : 80,
    
    draw: function(){
        if(state.current == state.getReady){
            ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);
        }
    }
    
}

// GAME OVER MESSAGE
const gameOver = {
    sX : 175,
    sY : 228,
    w : 225,
    h : 202,
    x : cvs.width/2 - 225/2,
    y : 90,
    
    draw: function(){
        if(state.current == state.over){
            ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);   
        }
    }
    
}

// PIPES
const pipes = {
    position : [],
    
    top : {
        sX : 553,
        sY : 0
    },
    bottom:{
        sX : 502,
        sY : 0
    },
    
    w : 53,
    h : 400,
    gap : 85,
    maxYPos : -150,
    dx : 2,
    
    draw : function(){
        for(let i  = 0; i < this.position.length; i++){
            let p = this.position[i];
            
            let topYPos = p.y;
            let bottomYPos = p.y + this.h + this.gap;
            
            // top pipe
            ctx.drawImage(sprite, this.top.sX, this.top.sY, this.w, this.h, p.x, topYPos, this.w, this.h);  
            
            // bottom pipe
            ctx.drawImage(sprite, this.bottom.sX, this.bottom.sY, this.w, this.h, p.x, bottomYPos, this.w, this.h);  
        }
    },
    
    update: function(){
        if(state.current !== state.game) return;
        
        if(frames%100 == 0){
            this.position.push({
                x : cvs.width,
                y : this.maxYPos * ( Math.random() + 1)
            });
        }
        for(let i = 0; i < this.position.length; i++){
            let p = this.position[i];
            
            let bottomPipeYPos = p.y + this.h + this.gap;
            
            // COLLISION DETECTION
            // TOP PIPE
            if(bird.x + bird.radius > p.x && bird.x - bird.radius < p.x + this.w && bird.y + bird.radius > p.y && bird.y - bird.radius < p.y + this.h){
                state.current = state.over;
                sendscore(score);
                document.getElementById("hiscore").innerHTML = myhiscore;
                document.getElementById("score").innerHTML = myscore;
                drawtable(myfrens);
                HIT.play();
            }
            // BOTTOM PIPE
            if(bird.x + bird.radius > p.x && bird.x - bird.radius < p.x + this.w && bird.y + bird.radius > bottomPipeYPos && bird.y - bird.radius < bottomPipeYPos + this.h){
                state.current = state.over;
                sendscore(score);
                document.getElementById("hiscore").innerHTML = myhiscore;
                document.getElementById("score").innerHTML = myscore;
                drawtable(myfrens);
                HIT.play();
            }
            
            // MOVE THE PIPES TO THE LEFT
            p.x -= this.dx;
            
            // if the pipes go beyond canvas, we delete them from the array
            if(p.x + this.w <= 0){
                this.position.shift();
                score.value += 1;
                SCORE_S.play();
                score.best = Math.max(score.value, score.best);
                localStorage.setItem("best", score.best);
            }
        }
    },
    
    reset : function(){
        this.position = [];
    }
    
}

// SCORE
const score= {
    best : parseInt(localStorage.getItem("best")) || (myhiscore ? myhiscore : 0),
    value : 0,
    
    draw : function(){
        ctx.fillStyle = "#FFF";
        ctx.strokeStyle = "#000";
        
        if(state.current == state.game){
            ctx.lineWidth = 2;
            ctx.font = "35px Teko";
            ctx.fillText(this.value, cvs.width/2, 50);
            ctx.strokeText(this.value, cvs.width/2, 50);
            
        }else if(state.current == state.over){
            // SCORE VALUE
            ctx.font = "25px Teko";
            ctx.fillText(this.value, 225, 186);
            ctx.strokeText(this.value, 225, 186);
            // BEST SCORE
            ctx.fillText(this.best, 225, 228);
            ctx.strokeText(this.best, 225, 228);
        }
    },
    
    reset : function(){
        this.value = 0;
    }
}

// DRAW
function draw(){
    ctx.fillStyle = "#70c5ce";
    ctx.fillRect(0, 0, cvs.width, cvs.height);
    
    bg.draw();
    pipes.draw();
    fg.draw();
    bird.draw();
    getReady.draw();
    gameOver.draw();
    score.draw();
}

// UPDATE
function update(){
    bird.update();
    fg.update();
    pipes.update();
}

// LOOP
function loop(){
    update();
    draw();
    frames++;
    
    requestAnimationFrame(loop);
}
loop();
```

</details>

At this point, if we refresh the page we will see our `%pals` data visible in the table. You can set up multiple development comets with the desk and test it out.

```hoon
|install our %flap
:treaty|publish %flap
```

### Beautification <a href="#beautification" id="beautification"></a>

There's a final set of changes we can make to the styling which makes this a much prettier app while still keeping these simple:

<details>

<summary>/app/flap/index.html (version 3)</summary>

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Original Flappy Bird -JavaScript</title>
    <link rel="preconnect" href="https://fonts.googleapis.com"> 
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> 
    <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600&family=Teko:wght@700&display=swap" rel="stylesheet">
    <style>
        body, html {
            font-family: "Inter", sans-serif;
            height: 100%;
            width: 100%;
            margin: 0;
        }
        body {
            display: flex;
            flex-direction: column;
            align-items: center;
            justify-content: center;
        }
        body > div {
            max-width: 320px;
            width: 100%;
        }
        table {
            width: 100%;
            text-align: center;
        }
        tr:first-of-type {
            font-weight: 700;
            color: #918C84;
        }
        canvas {
            border: 1px solid #000;
            display: block;
            margin: 0 auto;
        }
        #ship, #score, #hiscore {
            display: block;
            color: #000;
            font-weight: 400;
        }
        #bird {
            image-rendering: pixelated;
        }
        #ship, .ship {
            font-family: monospace;
        }
        #our {
            display: flex;
        }
        #our > p {
            margin: 1rem;
            font-weight: 700;
            color: #918C84;
            text-align: center;
        }
    </style>
</head>
<body>
    <div>
<canvas id="bird" width="320" height="480"></canvas>

<div id="our">
<p>Ship
    <span id="ship"></span>
</p>
<p>
    Last score
    <span id="score"></span>
</p>
<p>High score <span id="hiscore"></span></p>
</div>

<p style="font-weight: 700; text-align: center; margin-top: 2rem;">Leaderboard</p>
<p id="board">
<div id="frens"></div>
</p>
</div>

<script src="flap/game/js" type="module"></script>
</body>
</html>
```

</details>

<details>

<summary>/app/flap/game.js (version 3)</summary>

```hoon
// URBIT STATE
async function getmyship() {
    const response = await fetch('/apps/flap/whoami');
    return response.text();
}
var myshipname = await getmyship();
document.getElementById("ship").innerHTML = myshipname;

async function gethiscore() {
    const response = await fetch('/apps/flap/hiscore');
    return response.text();
}
var myhiscore = await gethiscore();

async function updateHiScore() {
    let newHiScore = await gethiscore();
    myhiscore = newHiScore;
}
document.getElementById("hiscore").innerHTML = myhiscore;

async function getscore() {
    const response = await fetch('/apps/flap/score');
    return response.text();
}
var myscore = await getscore();
document.getElementById("score").innerHTML = myscore;

//  Send score to Gall agent
function sendscore(score) {
  fetch('/apps/flap', {
    method: 'POST',
    body: JSON.stringify({'gain': {'score': score.value}})
  })
}

//  Draw table of frens
async function getfrens() {
        const response = await fetch('/apps/flap/frens');
        return response.text();
    }
function drawtable(myfrens) {
    console.log(myfrens);
    var frens = JSON.parse(myfrens);

    var table = document.createElement("table");
    var titleRow = table.insertRow();
    var frenCell = titleRow.insertCell();
    frenCell.innerHTML = "Ship";
    var scoreCell = titleRow.insertCell();
    scoreCell.innerHTML = "Score";

    for (let key in frens) {
        var row = table.insertRow();
        var cell = row.insertCell();
        cell.classList += "ship";
        cell.innerHTML = frens[key]['fren'];
        cell = row.insertCell();
        cell.innerHTML = frens[key]['score'];
    }

    // Clear the old table
    const list = document.getElementById("frens");
    while (list.hasChildNodes()) {
        list.removeChild(list.firstChild);
    }
    // Add the new one
    document.getElementById("frens").appendChild(table);
}
var myfrens = await getfrens();
console.log(myfrens);
drawtable(myfrens);

// SELECT CVS
const cvs = document.getElementById("bird");
const ctx = cvs.getContext("2d");

// GAME VARS AND CONSTS
let frames = 0;
const DEGREE = Math.PI/180;

// LOAD SPRITE IMAGE
const sprite = new Image();
sprite.src = "flap/img/sprite/png";

// LOAD SOUNDS
const SCORE_S = new Audio();
SCORE_S.src = "flap/audio/sfx-point/wav";

const FLAP = new Audio();
FLAP.src = "flap/audio/sfx-flap/wav";

const HIT = new Audio();
HIT.src = "flap/audio/sfx-hit/wav";

const SWOOSHING = new Audio();
SWOOSHING.src = "flap/audio/sfx-swooshing/wav";

const DIE = new Audio();
DIE.src = "flap/audio/sfx-die/wav";

// GAME STATE
const state = {
    current : 0,
    getReady : 0,
    game : 1,
    over : 2
}

// START BUTTON COORD
const startBtn = {
    x : 120,
    y : 263,
    w : 83,
    h : 29
}

// CONTROL THE GAME
cvs.addEventListener("click", function(evt){
    switch(state.current){
        case state.getReady:
            state.current = state.game;
            SWOOSHING.play();
            break;
        case state.game:
            if(bird.y - bird.radius <= 0) return;
            bird.flap();
            FLAP.play();
            break;
        case state.over:
            let rect = cvs.getBoundingClientRect();
            let clickX = evt.clientX - rect.left;
            let clickY = evt.clientY - rect.top;
            
            // CHECK IF WE CLICK ON THE START BUTTON
            if(clickX >= startBtn.x && clickX <= startBtn.x + startBtn.w && clickY >= startBtn.y && clickY <= startBtn.y + startBtn.h){
                pipes.reset();
                bird.speedReset();
                score.reset();
                state.current = state.getReady;
            }
            break;
    }
});


// BACKGROUND
const bg = {
    sX : 0,
    sY : 0,
    w : 275,
    h : 226,
    x : 0,
    y : cvs.height - 226,
    
    draw : function(){
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);
        
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x + this.w, this.y, this.w, this.h);
    }
    
}

// FOREGROUND
const fg = {
    sX: 276,
    sY: 0,
    w: 224,
    h: 112,
    x: 0,
    y: cvs.height - 112,
    
    dx : 2,
    
    draw : function(){
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);
        
        ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x + this.w, this.y, this.w, this.h);
    },
    
    update: function(){
        if(state.current == state.game){
            this.x = (this.x - this.dx)%(this.w/2);
        }
    }
}

// BIRD
const bird = {
    animation : [
        {sX: 276, sY : 112},
        {sX: 276, sY : 139},
        {sX: 276, sY : 164},
        {sX: 276, sY : 139}
    ],
    x : 50,
    y : 150,
    w : 34,
    h : 26,
    
    radius : 12,
    
    frame : 0,
    
    gravity : 0.25,
    jump : 4.6,
    speed : 0,
    rotation : 0,
    
    draw : function(){
        let bird = this.animation[this.frame];
        
        ctx.save();
        ctx.translate(this.x, this.y);
        ctx.rotate(this.rotation);
        ctx.drawImage(sprite, bird.sX, bird.sY, this.w, this.h,- this.w/2, - this.h/2, this.w, this.h);
        
        ctx.restore();
    },
    
    flap : function(){
        this.speed = - this.jump;
    },
    
    update: function(){
        // IF THE GAME STATE IS GET READY STATE, THE BIRD MUST FLAP SLOWLY
        this.period = state.current == state.getReady ? 10 : 5;
        // WE INCREMENT THE FRAME BY 1, EACH PERIOD
        this.frame += frames%this.period == 0 ? 1 : 0;
        // FRAME GOES FROM 0 To 4, THEN AGAIN TO 0
        this.frame = this.frame%this.animation.length;
        
        if(state.current == state.getReady){
            this.y = 150; // RESET POSITION OF THE BIRD AFTER GAME OVER
            this.rotation = 0 * DEGREE;
        }else{
            this.speed += this.gravity;
            this.y += this.speed;
            
            if(this.y + this.h/2 >= cvs.height - fg.h){
                this.y = cvs.height - fg.h - this.h/2;
                if(state.current == state.game){
                    state.current = state.over;
                    sendscore(score);
                    document.getElementById("hiscore").innerHTML = myhiscore;
                    document.getElementById("score").innerHTML = myscore;
                    drawtable(myfrens);
                }
            }
            
            // IF THE SPEED IS GREATER THAN THE JUMP MEANS THE BIRD IS FALLING DOWN
            if(this.speed >= this.jump){
                this.rotation = 90 * DEGREE;
                this.frame = 1;
            }else{
                this.rotation = -25 * DEGREE;
            }
        }
        
    },
    speedReset : function(){
        this.speed = 0;
    }
}

// GET READY MESSAGE
const getReady = {
    sX : 0,
    sY : 228,
    w : 173,
    h : 152,
    x : cvs.width/2 - 173/2,
    y : 80,
    
    draw: function(){
        if(state.current == state.getReady){
            ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);
        }
    }
    
}

// GAME OVER MESSAGE
const gameOver = {
    sX : 175,
    sY : 228,
    w : 225,
    h : 202,
    x : cvs.width/2 - 225/2,
    y : 90,
    
    draw: function(){
        if(state.current == state.over){
            ctx.drawImage(sprite, this.sX, this.sY, this.w, this.h, this.x, this.y, this.w, this.h);   
        }
    }
    
}

// PIPES
const pipes = {
    position : [],
    
    top : {
        sX : 553,
        sY : 0
    },
    bottom:{
        sX : 502,
        sY : 0
    },
    
    w : 53,
    h : 400,
    gap : 85,
    maxYPos : -150,
    dx : 2,
    
    draw : function(){
        for(let i  = 0; i < this.position.length; i++){
            let p = this.position[i];
            
            let topYPos = p.y;
            let bottomYPos = p.y + this.h + this.gap;
            
            // top pipe
            ctx.drawImage(sprite, this.top.sX, this.top.sY, this.w, this.h, p.x, topYPos, this.w, this.h);  
            
            // bottom pipe
            ctx.drawImage(sprite, this.bottom.sX, this.bottom.sY, this.w, this.h, p.x, bottomYPos, this.w, this.h);  
        }
    },
    
    update: function(){
        if(state.current !== state.game) return;
        
        if(frames%100 == 0){
            this.position.push({
                x : cvs.width,
                y : this.maxYPos * ( Math.random() + 1)
            });
        }
        for(let i = 0; i < this.position.length; i++){
            let p = this.position[i];
            
            let bottomPipeYPos = p.y + this.h + this.gap;
            
            // COLLISION DETECTION
            // TOP PIPE
            if(bird.x + bird.radius > p.x && bird.x - bird.radius < p.x + this.w && bird.y + bird.radius > p.y && bird.y - bird.radius < p.y + this.h){
                state.current = state.over;
                sendscore(score);
                document.getElementById("hiscore").innerHTML = myhiscore;
                document.getElementById("score").innerHTML = myscore;
                drawtable(myfrens);
                HIT.play();
            }
            // BOTTOM PIPE
            if(bird.x + bird.radius > p.x && bird.x - bird.radius < p.x + this.w && bird.y + bird.radius > bottomPipeYPos && bird.y - bird.radius < bottomPipeYPos + this.h){
                state.current = state.over;
                sendscore(score);
                document.getElementById("hiscore").innerHTML = myhiscore;
                document.getElementById("score").innerHTML = myscore;
                drawtable(myfrens);
                HIT.play();
            }
            
            // MOVE THE PIPES TO THE LEFT
            p.x -= this.dx;
            
            // if the pipes go beyond canvas, we delete them from the array
            if(p.x + this.w <= 0){
                this.position.shift();
                score.value += 1;
                SCORE_S.play();
                score.best = Math.max(score.value, score.best);
                localStorage.setItem("best", score.best);
            }
        }
    },
    
    reset : function(){
        updateHiScore();
        this.position = [];
    }
    
}

// SCORE
const score= {
    best : parseInt(localStorage.getItem("best")) || (myhiscore ? myhiscore : 0),
    value : 0,
    
    draw : function(){
        ctx.fillStyle = "#FFF";
        ctx.strokeStyle = "#000";
        
        if(state.current == state.game){
            ctx.lineWidth = 2;
            ctx.font = "35px Teko";
            ctx.fillText(this.value, cvs.width/2, 50);
            ctx.strokeText(this.value, cvs.width/2, 50);
            
        }else if(state.current == state.over){
            // SCORE VALUE
            ctx.font = "25px Teko";
            ctx.fillText(this.value, 225, 186);
            ctx.strokeText(this.value, 225, 186);
            // BEST SCORE
            ctx.fillText(this.best, 225, 228);
            ctx.strokeText(this.best, 225, 228);
        }
    },
    
    reset : function(){
        this.value = 0;
    }
}

// DRAW
function draw(){
    ctx.fillStyle = "#70c5ce";
    ctx.fillRect(0, 0, cvs.width, cvs.height);
    
    bg.draw();
    pipes.draw();
    fg.draw();
    bird.draw();
    getReady.draw();
    gameOver.draw();
    score.draw();
}

// UPDATE
function update(){
    bird.update();
    fg.update();
    pipes.update();
}

// LOOP
function loop(){
    update();
    draw();
    frames++;
    
    requestAnimationFrame(loop);
}
loop();
```

</details>

## What's Next? <a href="#whats-next" id="whats-next"></a>

After completing this tutorial, you should think about how to apply what you've seen to other applications. Some things to think about:

* What do other games need?
* What internal state do other games maintain?
* What makes sense to share via Urbit peers?
* How can we serve other components directly from the ship?


# Ship Monitoring

Tutorial on building a ship monitoring app with Sail frontend, CLI generators, and Behn timers for notifications.

> **Not to be confused with the kernel's `%ahoy`.** This page is about a third-party desk. Since February 2026 there is also an `%ahoy` subsystem in the kernel, for migrating peers to Directed Messaging, reached through the [`|ahoy/prob`](/user-manual/os/dojo-tools#ahoyprob) and [`|ahoy/comb`](/user-manual/os/dojo-tools#ahoycomb) generators. The two are unrelated.

The `%ahoy` desk by \~midden-fabler provides a number of agents to automatically monitor ship activity such as breaching and network uptime. This tutorial examines the `%ahoy` agent specifically with some slight simplifications to demonstrate how an Urbit-native app can be constructed. You will see how to render a front-end using Sail, employ the `+abet` nested core design pattern, construct CLI generators, and set wakeup timers using Behn.

`%ahoy` presents a web UI at `/ahoy` rendered using [Sail](/hoon/sail) and \~paldev's Rudder library alongside command-line generators to add, delete, and modify ship watches. Notifications are sent using `%hark-store` if a ship hasn't been contacted after a specified amount of time.

```hoon
:ahoy|add-watch ~sampel ~d1
:ahoy|del-watch ~sampel
:ahoy|set-update-interval ~m30
```

## `/sur` Structure Files <a href="#sur-structure-files" id="sur-structure-files"></a>

As with other agents, we think about our data structures and actions before we dive into the agent code. The structure file here defines the state for the agent, `records`, which is a collection of ships to watch and the update interval for sending notifications.

```hoon
+$  records
  $:  watchlist=(map ship @dr)
      update-interval=@dr
  ==
```

Three commands are supported: to add a ship to the watchlist at a given watch interval, to delete the ship, or to change the check interval. (Modifying a ship is the same as adding it.)

```hoon
+$  command
  $%  [%add-watch =ship t=@dr]
      [%del-watch =ship]
      [%set-update-interval t=@dr]
  ==
```

**`/sur/ahoy.hoon`**:

```hoon
|%
+$  records
  $:  watchlist=(map ship @dr)
      update-interval=@dr
  ==
+$  command
  $%  [%add-watch =ship t=@dr]
      [%del-watch =ship]
      [%set-update-interval t=@dr]
  ==
--
```

No special mark files are necessary for `%ahoy`.

## `/app` Agent Files <a href="#app-agent-files" id="app-agent-files"></a>

The agent itself is simple: it maintains `records` as state and processes pokes from generators or the front-end and gifts from `%behn` in particular.

In addition, `%ahoy` sends notifications using `%hark-store`, the notification process integrated with Landscape and Grid.

### Pokes <a href="#pokes" id="pokes"></a>

At the macro level, `+on-poke` recognizes three poke cages:

1. `%noun` for pinging a ship.
2. `%ahoy-command` for commands per `/sur/ahoy.hoon`.
3. `handle-http-request` for displaying the webpage.

Most of the poke work takes place through `%ahoy-command`, which checks on the ship state per Ames’ scheme of `%alien` and `%known` ships, then maintains the agent state by its watchlist.

```hoon
  %ahoy-command
=+  !<(cmd=command vase)
?-    -.cmd
    %add-watch
  =/  ss=(unit ship-state:ames)
    (~(ship-state ahoy bowl) ship.cmd)
  ?~  ss
    ~&  >>  [%ahoy '%alien ship not added']
    [~ this]
  :-  [(send-plea:hc ship.cmd)]~
  this(watchlist (~(put by watchlist) ship.cmd t.cmd))
::
    %del-watch
  `this(watchlist (~(del by watchlist) ship.cmd))
::
    %set-update-interval  
  `this(update-interval t.cmd)
==
```

HTTP requests are processed into a form useful to `rudder`, a front-end rendering library for native Hoon webpages. `rudder` facilitates a Sail-based webpage being exposed through three arms:

1. `+argue` responds to `POST` requests.
2. `+final` is called after `POST` requests.
3. `+build` responds to `GET` requests, most commonly just yielding the webpage.

A number of other facilities in `rudder` are employed here as well:

* `+order:rudder` is a type for handling inbound requests from Eyre.
* `+steer:rudder` is the helper constructor for producing pages.
* `+point:rudder` is a routing arm.
* `+fours:rudder` is a 404 error handler.
* `+brief:rudder` is a type union, `?(~ @t)`.

```hoon
  %handle-http-request
=;  out=(quip card _+.state)
  [-.out this(+.state +.out)]
%.  [bowl !<(order:rudder vase) +.state]
%-  (steer:rudder _+.state command)
:^    pages
    (point:rudder /[dap.bowl] & ~(key by pages))
  (fours:rudder +.state)
|=  cmd=command
^-  $@  brief:rudder
    [brief:rudder (list card) _+.state]
=^  cards  this
  (on-poke %ahoy-command !>(cmd))
['Processed succesfully.' cards +.state]
```

### Gifts <a href="#gifts" id="gifts"></a>

The agent expects to receive a `%wake` gift periodically from Behn on the wire `%update-interval`. It handles this by means of an arm in the agent's helper core, `+on-update-interval`.

```hoon
  [%update-interval ~]
=^  cards  state
  on-update-interval:hc
[cards this]
```

This helper core arm notably employs the `+abet` nested core pattern for handling cards. The `+abet` nested core is a design pattern rather than a specific core. It is designed to accumulate cards, often using `+emit` and `+emil`, then send them all at once.

The `+abet` pattern itself is rather simple to construct. It enables other arms to construct a list of cards rather than having to produce complex `=^`-style constructions. This instance of the nested core pattern consists of three arms (omitting an `+abed` arm):

* `+emit` is used to submit a card to a collection of cards in the helper core.
* `+emil` is similar but accepts a list of cards.
* `+abet` issues the list of cards back along with the state to be updated. (Note that the core must be scoped such that the Gall agent's state is visible.)

Other arms (such as `+set-timer`) then simply construct cards which are inserted into the `+abet` core's list.

<details>

<summary>Helper Core code</summary>

```hoon
=|  cards=(list card)
|_  =bowl:gall
++  this  .
++  abet
  ^-  (quip card _state)
  [(flop cards) state]
::
++  emit
  |=  car=card
  this(cards [car cards])
::
++  emil
  |=  rac=(list card)
  |-  ^+  this
  ?~  rac
    this
  =.  cards  [i.rac cards]
  $(rac t.rac)
::
++  on-update-interval
  ^-  (quip card _state)
  ::  reset timer
  =.  this  (emit (set-timer update-interval))
  ::  send pleas
  =.  this
    %-  emil
    %+  turn  ~(tap in ~(key by watchlist))
    |=  [who=ship]
    (send-plea who)
  ::  send notifications
  =.  this
    %-  emil
    %-  zing
    %+  turn  ~(tap in down-status)
    |=  [who=ship]
    (send-notification who)
  abet
::
++  set-timer
  |=  t=@dr
  ^-  card
  =/  when=@da  (add now.bowl t)
  [%pass /update-interval %arvo %b %wait when]
::
++  send-plea
  |=  [who=ship]
  ^-  card
  [%pass /ahoy/(scot %p who) %arvo %a %plea who %evil-vane / ~]
::
++  down-status
  ^-  (set ship)
  %-  silt
  %+  murn  ~(tap in ~(key by watchlist))
  |=  [who=ship]
  =/  when=(unit @dr)  (~(last-contact ahoy bowl) who)
  ?~  when  ~
  ?.  (gte u.when (~(got by watchlist) who))
    ~
  `who
::
++  send-notification
  |=  [who=ship]
  ^-  (list card)
  ?.  .^(? %gu /(scot %p our.bowl)/hark-store/(scot %da now.bowl))  ~
  =/  when=@dr  (need (~(last-contact ahoy bowl) who))
  =/  title=(list content:hark)
    =-  [ship+who - ~]
    text+(crip " has not been contacted in {<when>}")
  =/  =bin:hark     [/[dap.bowl] q.byk.bowl /(scot %p who)]
  =/  =action:hark  [%add-note bin title ~ now.bowl / /[dap.bowl]]
  =/  =cage         [%hark-action !>(action)]
  [%pass /hark %agent [our.bowl %hark-store] %poke cage]~
--
```

</details>

For `%ahoy`, the main arm we need to examine is `+on-update-interval`. This arm resets the timer, sends checks to all of the ships, and then sends notifications to `%hark-store` for anything unresponsive.

```hoon
++  on-update-interval
  ^-  (quip card _state)
  ::  reset timer
  =.  this  (emit (set-timer update-interval))
  ::  send pleas
  =.  this
    %-  emil
    %+  turn  ~(tap in ~(key by watchlist))
    |=  [who=ship]
    (send-plea who)
  ::  send notifications
  =.  this
    %-  emil
    %-  zing
    %+  turn  ~(tap in down-status)
    |=  [who=ship]
    (send-notification who)
  abet
```

The `+send-plea` status check is interesting: it checks whether Ames is responsive on a particular ship without doing anything to the remote ship except eliciting an error. (`|hi` or similar would unnecessarily spam the recipient's Dojo.)

```hoon
++  send-plea
  |=  [who=ship]
  ^-  card
  [%pass /ahoy/(scot %p who) %arvo %a %plea who %evil-vane / ~]
```

`%hark-store` is the standard cross-agent notification store provided by Grid and recognized by Landscape. The notification message requires a little bit of explicit construction as `$action` but can be treated as boilerplate code aside from the text.

```hoon
++  send-notification
  |=  [who=ship]
  ^-  (list card)
  ?.  .^(? %gu /(scot %p our.bowl)/hark-store/(scot %da now.bowl))  ~
  =/  when=@dr  (need (~(last-contact ahoy bowl) who))
  =/  title=(list content:hark)
    =-  [ship+who - ~]
    text+(crip " has not been contacted in {<when>}")
  =/  =bin:hark     [/[dap.bowl] q.byk.bowl /(scot %p who)]
  =/  =action:hark  [%add-note bin title ~ now.bowl / /[dap.bowl]]
  =/  =cage         [%hark-action !>(action)]
  [%pass /hark %agent [our.bowl %hark-store] %poke cage]~
```

<details>

<summary>/app/ahoy.hoon</summary>

```hoon
::  ahoy: ship monitoring
::
::    get notified if last-contact with a ship
::    exceeds a specified amount of time
::
::  usage:
::    :ahoy|add-watch ~sampel ~d1
::    :ahoy|del-watch ~sampel
::    :ahoy|set-update-interval ~m30
::
::  scrys:
::    .^((map @p @dr) %gx /=ahoy=/watchlist/noun)
::    .^((set ship) %gx /=ahoy=/watchlist/ships/noun)
::    .^(@dr %gx /=ahoy=/update-interval/noun)
::
/-  *ahoy, hark=hark-store
/+  default-agent, 
    agentio, 
    rudder,
    dbug,
    ahoy 
/~  pages  (page:rudder records command)  /app/ahoy/webui
::
=>  |%
    +$  card  card:agent:gall
    +$  versioned-state
      $%  state-0
      ==
    +$  state-0  [%0 records]
    --
::
=|  state-0
=*  state  -
%-  agent:dbug
^-  agent:gall
=<
|_  =bowl:gall
+*  this  .
    def   ~(. (default-agent this %.n) bowl)
    hc    ~(. +> bowl)
    io    ~(. agentio bowl)
    pass  pass:io
::
++  on-init
  ^-  (quip card _this)
  =/  interval=@dr  ~m5
  =+  sponsor=(sein:title [our now our]:bowl)
  :_  this(update-interval interval)
  :~  (~(connect pass /eyre/connect) [~ /[dap.bowl]] dap.bowl)
      (poke-self:pass %ahoy-command !>([%add-watch sponsor ~d1]))
      (set-timer interval)
  ==
::
++  on-save  !>(state)
++  on-load
  |=  ole=vase
  ^-  (quip card _this)
  =/  old  !<(versioned-state ole)
  ?-  -.old
    %0  [~ this(state old)]
  ==
::
++  on-poke
  |=  [=mark =vase]
  ^-  (quip card _this)
  ?>  =(our src):bowl
  ?+    mark  (on-poke:def mark vase)
      %noun
    =+  !<(who=ship vase)
    :_  this
    [(send-plea:hc who)]~
  ::
      %ahoy-command
    =+  !<(cmd=command vase)
    ?-    -.cmd
        %add-watch
      =/  ss=(unit ship-state:ames)
        (~(ship-state ahoy bowl) ship.cmd)
      ?~  ss
        ~&  >>  [%ahoy '%alien ship not added']
        [~ this]
      :-  [(send-plea:hc ship.cmd)]~
      this(watchlist (~(put by watchlist) ship.cmd t.cmd))
    ::
        %del-watch
      `this(watchlist (~(del by watchlist) ship.cmd))
    ::
        %set-update-interval  
      `this(update-interval t.cmd)
    ==
  ::
      %handle-http-request
    =;  out=(quip card _+.state)
      [-.out this(+.state +.out)]
    %.  [bowl !<(order:rudder vase) +.state]
    %-  (steer:rudder _+.state command)
    :^    pages
        (point:rudder /[dap.bowl] & ~(key by pages))
      (fours:rudder +.state)
    |=  cmd=command
    ^-  $@  brief:rudder
        [brief:rudder (list card) _+.state]
    =^  cards  this
      (on-poke %ahoy-command !>(cmd))
    ['Processed succesfully.' cards +.state]
  ==
::
++  on-watch
  |=  =path
  ^-  (quip card _this)
  ?+    path  (on-watch:def path)
      [%http-response *]
    ?>  =(our src):bowl
    [~ this]
  ==
::
++  on-arvo
  |=  [=wire =sign-arvo]
  ^-  (quip card _this)
  ?+    wire  (on-arvo:def wire sign-arvo)
      [%ahoy @ ~]  [~ this]
  ::
      [%update-interval ~]
    =^  cards  state
      on-update-interval:hc
    [cards this]
  ::
      [%eyre %connect ~]
    ?+  sign-arvo  (on-arvo:def wire sign-arvo)
        [%eyre %bound *]
      ~?  !accepted.sign-arvo
        [dap.bowl 'eyre bind rejected!' binding.sign-arvo]
      [~ this]
    ==
  ==
::
++  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ?>  =(our src):bowl
  ?+  path  (on-peek:def path)
    [%x %watchlist ~]         ``noun+!>(watchlist)
    [%x %watchlist %ships ~]  ``noun+!>(~(key by watchlist))
    [%x %update-interval ~]   ``noun+!>(update-interval)
  ==
::
++  on-leave  on-leave:def
++  on-agent  on-agent:def
++  on-fail   on-fail:def
--
::
=|  cards=(list card)
|_  =bowl:gall
++  this  .
++  abet
  ^-  (quip card _state)
  [(flop cards) state]
::
++  emit
  |=  car=card
  this(cards [car cards])
::
++  emil
  |=  rac=(list card)
  |-  ^+  this
  ?~  rac
    this
  =.  cards  [i.rac cards]
  $(rac t.rac)
::
++  on-update-interval
  ^-  (quip card _state)
  ::  reset timer
  =.  this  (emit (set-timer update-interval))
  ::  send pleas
  =.  this
    %-  emil
    %+  turn  ~(tap in ~(key by watchlist))
    |=  [who=ship]
    (send-plea who)
  ::  send notifications
  =.  this
    %-  emil
    %-  zing
    %+  turn  ~(tap in down-status)
    |=  [who=ship]
    (send-notification who)
  abet
::
++  set-timer
  |=  t=@dr
  ^-  card
  =/  when=@da  (add now.bowl t)
  [%pass /update-interval %arvo %b %wait when]
::
++  send-plea
  |=  [who=ship]
  ^-  card
  [%pass /ahoy/(scot %p who) %arvo %a %plea who %evil-vane / ~]
::
++  down-status
  ^-  (set ship)
  %-  silt
  %+  murn  ~(tap in ~(key by watchlist))
  |=  [who=ship]
  =/  when=(unit @dr)  (~(last-contact ahoy bowl) who)
  ?~  when  ~
  ?.  (gte u.when (~(got by watchlist) who))
    ~
  `who
::
++  send-notification
  |=  [who=ship]
  ^-  (list card)
  ?.  .^(? %gu /(scot %p our.bowl)/hark-store/(scot %da now.bowl))  ~
  =/  when=@dr  (need (~(last-contact ahoy bowl) who))
  =/  title=(list content:hark)
    =-  [ship+who - ~]
    text+(crip " has not been contacted in {<when>}")
  =/  =bin:hark     [/[dap.bowl] q.byk.bowl /(scot %p who)]
  =/  =action:hark  [%add-note bin title ~ now.bowl / /[dap.bowl]]
  =/  =cage         [%hark-action !>(action)]
  [%pass /hark %agent [our.bowl %hark-store] %poke cage]~
--
```

</details>

The `/lib/ahoy.hoon` library file provides helper logic for determining ship status. In particular, scries are simplified. For instance, `(~(last-contact ahoy bowl) ship)` can be used instead of the scry below.

<details>

<summary>/lib/ahoy.hoon</summary>

```hoon
|_  =bowl:gall
++  ship-state
  |=  [who=ship]
  ^-  (unit ship-state:ames)
  ?.  (~(has in peers) who)
    ~
  `.^(ship-state:ames %ax /(scot %p our.bowl)//(scot %da now.bowl)/peers/(scot %p who))
::
++  peers
  ^-  (set ship)
  =/  mips
    .^((map ship ?(%alien %known)) %ax /(scot %p our.bowl)//(scot %da now.bowl)/peers)
  ~(key by mips)
::
++  last-contact
  |=  [who=ship]
  ^-  (unit @dr)
  =/  ss=(unit ship-state:ames)  (ship-state who)
  ?~  ss  ~
  ?.  ?=([%known *] u.ss)
    ~
  =/  last-contact=@da  last-contact.qos.u.ss
  =/  when=@dr  (sub now.bowl last-contact)
  `when
--
```

</details>

<details>

<summary>/app/ahoy/webui/index.hoon</summary>

```hoon
/-  *ahoy, contact=contact-store
/+  ahoy, rudder, ahoy-style, sigil-svg=sigil
::
^-  (page:rudder records command)
|_  [=bowl:gall =order:rudder records]
++  argue
  |=  [headers=header-list:http body=(unit octs)]
  ^-  $@(brief:rudder command)
  =/  args=(map @t @t)
    ?~(body ~ (frisk:rudder q.u.body))
  ?~  what=(~(get by args) 'what')  ~
  ?+    u.what  ~
      %add-watch
    ?~  who=(slaw %p (~(gut by args) 'who' ''))     ~
    ?~  when=(slaw %dr (~(gut by args) 'when' ''))  ~
    [%add-watch u.who u.when]
  ::
      %del-watch
    ?~  who=(slaw %p (~(gut by args) 'who' ''))  ~
    [%del-watch u.who]
  ==
::
++  final  (alert:rudder (cat 3 '/' dap.bowl) build)
++  build
  |=  $:  arg=(list [k=@t v=@t])
          msg=(unit [o=? =@t])
      ==
  ^-  reply:rudder
  |^  [%page page]
  ++  page
    ^-  manx
    ;html
      ;head
        ;title:"%ahoy"
        ;meta(charset "utf-8");
        ;meta(name "viewport", content "width=device-width, initial-scale=1");
        ;style:"{(trip style:ahoy-style)}"
      ==
      ;body
        ;a/"/ahoy"
          ;h2:"%ahoy"
        ==

        ;h4:"ship monitoring (tutorial)"

        get notified if last-contact with a ship
        exceeds a specified amount of time
        
        ;+  ?~  msg  ;p:""
            ?:  o.u.msg
              ;p.green:"{(trip t.u.msg)}"
            ;p.red:"{(trip t.u.msg)}"
        ;table#ahoy
          ;form(method "post")
            ::  table header
            ;tr(style "font-weight: bold")
              ;td(align "center"):"~"
              ;td(align "center"):"@p"
              ;td(align "center"):"notify after @dr"
              ;td(align "center"):"last-contact `@dr"
            ==
            ::  first row for adding new ships
            ;tr
              ;td
                ;button(type "submit", name "what", value "add-watch"):"+"
              ==
              ;td
                ;input(type "text", name "who", placeholder "~sampel");
              ==
              ;td
                ;input(type "text", name "when", placeholder "~d1.h12.m30");
              ==
              ;td(align "center"):"~"
            ==  ::  first row
          ==    ::  form
          ;*  work
        ==
      ==  ::  body
    ==    ::  html
  ++  work
    ^-  (list manx)
    %+  turn  ~(tap by watchlist)
    |=  [=ship t=@dr]
    ;tr
      ;td
        ::  %del-watch
        ;form(method "post")
          ;button(type "submit", name "what", value "del-watch"):"-"
          ;input(type "hidden", name "who", value "{(scow %p ship)}");
        ==
        ::  ship
        ;td
          ;+  (sigil ship)
          ; {(scow %p ship)}
        ==
        ::  when to notify
        ;form(method "post")
          ;td
            ;input(type "hidden", name "what", value "add-watch");
            ;input(type "hidden", name "who", value "{(scow %p ship)}");
            ;input(type "text", name "when", value "{(scow %dr t)}");
          ==
        ==
        ::  last-contact
        ;td(align "right")
          ; {<(~(last-contact ahoy bowl) ship)>}
        ==
      ==
    ==
  ::
  ++  contacts  ~+
    =/  base=path
      /(scot %p our.bowl)/contact-store/(scot %da now.bowl)
    ?.  .^(? %gu base)  *rolodex:contact
    .^(rolodex:contact %gx (weld base /all/noun))
  ::
  ++  sigil
    |=  =ship
    ^-  manx
    =/  bg=@ux
      ?~(p=(~(get by contacts) ship) 0xff.ffff color.u.p)
    =/  fg=tape
      =+  avg=(div (roll (rip 3 bg) add) 3)
      ?:((gth avg 0xc1) "black" "white")
    =/  bg=tape
      ((x-co:co 6) bg)
    ;div.sigil(style "background-color: #{bg}; width: 20px; height: 20px;")
      ;img@"/ahoy/sigil.svg?p={(scow %p ship)}&fg={fg}&bg=%23{bg}&icon&size=20";
    ==
  --  ::  |^
--    ::  |_
```

</details>

The CSS styling is included via a library core:

<details>

<summary>/lib/ahoy/style.hoon</summary>

```hoon
|%
++  style
  '''

  * { margin: 0.2em; padding: 0.2em; font-family: monospace; }

  body {
    background-color: black; 
    color: white; 
  }

  h2 { color: red; }

  p { max-width: 50em; }

  form { margin: 0; padding: 0; }

  .red { font-weight: bold; color: #dd2222; }
  .green { font-weight: bold; color: #229922; }

  a {
    display: inline-block;
    color: inherit;
    padding: 0;
    margin-top: 0;
  }

  table#ahoy tr td:nth-child(2) {
    padding: 0 0.5em;
  }

  .label {
    display: inline-block;
    background-color: #ccc;
    border-radius: 3px;
    margin-right: 0.5em;
    padding: 0.1em;
  }
  .label input[type="text"] {
    max-width: 100px;
  }
  .label span {
    margin: 0 0 0 0.2em;
  }

  button {
    padding: 0.2em 0.5em;
  }

  .sigil {
    display: inline-block;
    vertical-align: middle;
    margin: 0 0.5em 0 0;
    padding: 0.2em;
    border-radius: 0.2em;
  }

  .sigil * {
    margin: 0;
    padding: 0;
  }
  '''
--
```

</details>

### Rendering Sigils <a href="#rendering-sigils" id="rendering-sigils"></a>

[Sigils](https://urbit.org/blog/creating-sigils) are unique visual representations of `@p` ship identifiers. Many Urbit apps use sigils in small or large sizes as ship icons.

A sigil library is provided with \~paldev's Suite tools. We do not include the contents of `/lib/sigil.hoon` or `/lib/sigil/symbols.hoon` here due to their length.

* [`/lib/sigil.hoon`](https://github.com/Fang-/suite/blob/master/lib/sigil.hoon)
* [`/lib/sigil/symbols.hoon`](https://github.com/Fang-/suite/blob/master/lib/sigil/symbols.hoon)

The sigils are rendered in `/app/ahoy/webui/index.hoon`.

## `/gen` Generator Files <a href="#gen-generator-files" id="gen-generator-files"></a>

Some agents (notably `%helm`, a Dojo tool) are instrumented to work directly with generators at the command line. The `%ahoy` agent demonstrates this with several generator files such as `/gen/add-watch.hoon`, used thus:

```hoon
:ahoy|add-watch ~sampel-palnet ~h2
```

**`/gen/ahoy/add-watch.hoon`**

```hoon
  ::  :ahoy|add-watch ~sampel ~d1
  ::
  :-  %say
  |=  $:  ^
          [who=ship t=@dr ~]
          ~
      ==
  [%ahoy-command [%add-watch who t]]
```

As you can see here, an `%ahoy-command` is generated which is then passed to the `%ahoy` agent as a poke using Dojo's `|` logic. (A generator called with Dojo's `+` logic would be located in `/gen`, whereas `|` tells Dojo to look inside the agent's folder, much like a `/mar` mark file.)

Such agent-specific generator files can be much cleaner than manual poke logic:

```hoon
:ahoy|add-watch ~sampel-palnet ~h2
```

is the equivalent of

```hoon
:ahoy &ahoy-command [%add-watch ~zod ~h2]
```

### Exercise: Compose a Generator <a href="#exercise-compose-a-generator" id="exercise-compose-a-generator"></a>

* Without consulting the `%ahoy` source code, compose a generator `/gen/ahoy/del-watch.hoon` which removes a ship from the watchlist.


# Styled Text

Tutorial on producing styled text with styx strings and outputting them from Urbit agents to the terminal.

In this tutorial, we examine how to produce `$styx` styled text strings and output them to the terminal from an agent.

## `%shoe` CLI Session Manager <a href="#shoe-cli-session-manager" id="shoe-cli-session-manager"></a>

`%shoe` is responsible to manage attached agent sessions. It adds a few arms to the standard Gall agent, namely:

* `+command-parser` is the input parser, similar to the work we were carrying out just above. This parses every input and only permits valid keystrokes (think of Dojo real-time parsing).
* `+tab-list` provides autocompletion options. We can ignore for now.
* `+on-command` is called whenever a valid command is run. This produces the actual effects.
* `+can-connect` supports `|dojo/link` connexion to the app.
* `+on-connect` provides particular session support when a user connects. We can ignore for now.
* `+on-disconnect` provides particular session support when a user disconnects. We can ignore for now.

To get started with text parsers and CLI agents, we need to focus on `+command-parser` and `+on-command`. But first, the agent's structure and state:

The agent will adopt a two-stage process, wherein a value is put on the stack then the stack is checked for any valid operations.

### `+command-parser` <a href="#command-parser" id="command-parser"></a>

The input parser can simply accept whole words or single inputs, or parse complex expressions (as Dojo does with Hoon).

This results in a noun of `$command-type` based on the specific application. The example `/app/shoe.hoon` agent defines:

```hoon
+$  command
  $?  %demo
      %row
      %table
  ==
```

and later uses this as:

```hoon
  ++  command-parser                                                                            
    |=  =sole-id:shoe
    ^+  |~(nail *(like [? command]))
    %+  stag  &
    (perk %demo %row %table ~)
```

where the unfamiliar parser components are:

* `+stag` adds a label, here `&` pam `TRUE`/`%.y` to indicate that the command should be run immediately when it matches. (We won't want this below so we will `+stag` a `|` `FALSE`/`%.n`.)
* `+perk` parses a fork in the type.

### `+on-command` <a href="#on-command" id="on-command"></a>

This arm accepts a session ID and a command resulting from `+command-parser`. It produces a regular `(quip card _this)` so you can modify agent state and produce effects here.

## `%sole` Effects <a href="#sole-effects" id="sole-effects"></a>

`%sole` is responsible for producing effects. If you want to yield effects to the command line from your CLI agent (which you often do), this is a great place to work.

`%sole-effect`s are head-tagged by time and produce a variety of terminal effects, from text to bells, colors, and other screen effects.

## `$styx` Styled Text String <a href="#styx-styled-text-string" id="styx-styled-text-string"></a>

A `+klr` effect uses a `$styx`, or styled text string. The relevant data structures are in `/sys/lull.hoon`:

```hoon
+$  deco  ?(~ %bl %br %un)                              ::  text decoration
+$  stye  (pair (set deco) (pair tint tint))            ::  decos/bg/fg
+$  styl  %+  pair  (unit deco)                         ::  cascading style
          (pair (unit tint) (unit tint))                ::
+$  styx  (list $@(@t (pair styl styx)))                ::  styled text       
+$  tint  $@  ?(%r %g %b %c %m %y %k %w %~)             ::  text color
          [r=@uxD g=@uxD b=@uxD]                        ::  24bit true color
```

* `$deco` is a text decoration, here `%bl` blinking, `%br` bright (bold), and `%un` underlined.
* `$tint` is a color, either explicitly the terminal red/green/blue/cyan etc. or a 24-bit true color value.
* `$stye` composes these into a style which will be applied to a string.
* `$styl` similarly composes styles together.
* `$styx` pairs styles with cords.

This means that composing styled text correctly can require explicitly nesting statements in rather a complicated way.

For instance, to produce a bold string with hex color `#123456`, we could produce the `sole-effect`:

```hoon
^-  sole-effect:sole
:-  %klr
^-  styx
~[[[`%br ~ `[r=0x12 g=0x34 b=0x56]] 'Hello Mars!' ~]]
```

* [\~ropdeb-sormyr, "Styled output - requirements and progress" \~2016.8.2 Urbit fora post](https://github.com/urbit/fora-posts/blob/0238536650dfc284f14295d350f9acada0341480/archive/posts/~2016.8.2..21.19.29..2ab8~.md)

## Agent Logic <a href="#agent-logic" id="agent-logic"></a>

Here is an agent that will accept a single character and produce a line with varying random colors of that character.

**`/app/track7.hoon`**

```hoon
/+  default-agent, dbug, shoe, sole
|%
+$  versioned-state
  $%  state-0
  ==
+$  state-0  %0
+$  card  card:agent:shoe
+$  command  @t
--
%-  agent:dbug
=|  state-0
=*  state  -
^-  agent:gall
%-  (agent:shoe command)
^-  (shoe:shoe command)
|_  =bowl:gall
+*  this     .
    default  ~(. (default-agent this %|) bowl)
    leather  ~(. (default:shoe this command) bowl)
++  on-init   on-init:default
++  on-save   !>(state)
++  on-load
  |=  old=vase
  ^-  (quip card _this)
  `this(state !<(state-0 old))
++  on-poke   on-poke:default
++  on-peek   on-peek:default
++  on-arvo   on-arvo:default
++  on-watch  on-watch:default
++  on-leave  on-leave:default
++  on-agent  on-agent:default
++  on-fail   on-fail:default
++  command-parser
  |=  =sole-id:shoe
  ^+  |~(nail *(like [? command]))
  (stag & (boss 256 (more gon qit)))
++  on-command
  |=  [=sole-id:shoe =command]
  ^-  (quip card _this)
  :_  this
  ^-  (list card)
  :~  :+  %shoe  ~
  ^-  shoe-effect:shoe
  :-  %sole
  ^-  sole-effect:sole  :-  %klr
  ^-  styx
  =/  idx  0
  =|  fx=styx
  =/  rng  ~(. og eny:bowl)
  |-
  ?:  =(80 idx)  fx
  =^  huer  rng  (rads:rng 256)
  =^  hueg  rng  (rads:rng 256)
  =^  hueb  rng  (rads:rng 256)
  %=  $
    idx  +(idx)
    fx   `styx`(weld fx `styx`~[[[`%br ~ `[r=`@ux`huer g=`@ux`hueg b=`@ux`hueb]] command ~]])
  ==  ==
++  can-connect
  |=  =sole-id:shoe
  ^-  ?
  ?|  =(~zod src.bowl)
      (team:title [our src]:bowl)
  ==
++  on-connect     on-connect:leather
++  on-disconnect  on-disconnect:leather
++  tab-list       tab-list:leather
--
```


# What is Urbit ID?

Overview of Urbit ID, a decentralized identity standard built on Ethereum. Covers network topology, smart contracts, and Layer 2 rollup functionality.

Urbit ID is Urbit's identity standard. It's decentralized, secure, and human-meaningful. It's basis is [a public-key infrastructure (PKI) implemented with NFTs on the Ethereum blockchain](https://urbit.org/blog/pki-maze).

You need an Urbit ID to get on the Urbit network. Your Urbit ID allows your Urbit OS server to cryptographically sign every message it sends over the network: everything you see on Urbit really comes from who it says it does.

### Network topology <a href="#network-topology" id="network-topology"></a>

Like any peer-to-peer network, Urbit needs topographic affordances for bootstrapping peer discovery. Some notions of governance, ownership, and reputation are nice-to-haves. Urbit solves these issues by dividing the address space into five "ranks":

* **Galaxies:** These 2^8 (256) nodes make up Urbit's Galactic Senate, which ratifies governance proposals and votes on smart contract upgrades. The Senate also elects the Board of Directors of the Urbit Foundation. They spawn and sponsor stars.
* **Stars:** These 2^16 (65,536) nodes distribute software updates to their sponsees. Eventually they may be used for peer discovery. They spawn and sponsor planets.
* **Planets:** There are 2^32 (\~4 billion) nodes are for day-to-day individual usage. They spawn and sponsor moons.
* **Moons:** Every planet can spawn \~4 billion moons, which are ideal for identifying machines tied to their owner's planet. Today, these are mostly used by app developers to host software distribution nodes on cloud servers.
* **Comets:** Urbit ID also has room for 2^64 (\~18 quintillion) free, anonymous, disposable identities. They aren't sponsored and don't interact with the blockchain at all.

Technically, "sponsorship" just means that the sponsor is responsible for distributing software updates to the sponsee. Whether this sponsorship chain means that one node endorses another as trustworthy is totally implicit.

### Smart contracts <a href="#smart-contracts" id="smart-contracts"></a>

Urbit ID ownership is stored on a smart contract at `azimuth.eth`, and that store is goverend by a contract at `ecliptic.eth`. Other contracts provide supplementary functionality.

You can read these contracts on Etherscan:

* [Azimuth.eth](https://etherscan.io/address/azimuth.eth) `0x223c067f8cf28ae173ee5cafea60ca44c335fecb`: Contains all on-chain state for Urbit ID. Most notably, ownership and public keys. Can't be modified directly, you must use Ecliptic.
* [Ecliptic.eth](https://etherscan.io/address/ecliptic.eth) `0x33EeCbf908478C10614626A9D304bfe18B78DD73`: An interface for interacting with Azimuth. Allows you to configure keys, transfer ownership, etc.
* [Polls](https://etherscan.io/address/0x7fecab617c868bb5996d99d95200d2fa708218e4) `0x7fecab617c868bb5996d99d95200d2fa708218e4`: Registers votes on governance proposals by Urbit's Galactic Senate. These can be either static documents or Ecliptic upgrades.
* [Linear Star Release](https://etherscan.io/address/0x86cd9cd0992f04231751e3761de45cecea5d1801) `0x86cd9cd0992f04231751e3761de45cecea5d1801`: Facilitates the release of blocks of stars to their owners over a period of time.
* [Conditional Star Release](https://etherscan.io/address/0x8c241098c3d3498fe1261421633fd57986d74aea) `0x8c241098c3d3498fe1261421633fd57986d74aea`: Facilitates the release of blocks of stars to their owners based on milestones.
* [Claims](https://etherscan.io/address/0xe7e7f69b34d7d9bd8d61fb22c33b22708947971a) `0xe7e7f69b34d7d9bd8d61fb22c33b22708947971a`: Allows point owners to make claims about (for example) their identity, and associate that with their point.
* [Censures](https://etherscan.io/address/0x325f68d32bdee6ed86e7235ff2480e2a433d6189) `0x325f68d32bdee6ed86e7235ff2480e2a433d6189`: Simple reputation management, allowing galaxies and stars to flag points for negative reputation.
* [Delegated Sending](https://etherscan.io/address/0xf6b461fe1ad4bd2ce25b23fe0aff2ac19b3dfa76) `0xf6b461fe1ad4bd2ce25b23fe0aff2ac19b3dfa76`: Enables network-effect like distributing of planets.

### General Azimuth Resources <a href="#general-azimuth-resources" id="general-azimuth-resources"></a>

These documents pertain to L1 and other general aspects of Azimuth. For L2 docs, [see below](#naive-rollups).

* [HD Wallet](/urbit-id/hd-wallet) - Azimuth has its own optional hierarchical deterministic wallet system, often referred to as a "master ticket".
* [Data Flow](/urbit-id/flow) - Diagrams and explanations of how data flows between Bridge and the various components inside Urbit involved with Azimuth and L2.
* [Azimuth.eth](/urbit-id/azimuth-eth) - A description of the azimuth.eth smart contract, which is the data store for Azimuth.
* [Ecliptic.eth](/urbit-id/ecliptic) - A description of the ecliptic.eth smart contract, which is the business logic for azimuth.eth. This includes an overview of all function calls available.
* [Advanced Azimuth Tools](/urbit-id/advanced-azimuth-tools) - Expert-level tooling for generating, signing, and sending layer 1 Azimuth transactions from within Urbit itself.
* [Life and Rift](/urbit-id/life-and-rift) - An explanation of how Azimuth indexes networking keys revisions and breaches to keep track of the most recent set of networking keys necessary to communicate with a ship.

### Naive rollups <a href="#naive-rollups" id="naive-rollups"></a>

In 2021, Tlon introduced a new component to Azimuth called **naive rollups**, and often referred to as Urbit ID's "Layer 2" or "L2". It was intended to reduce gas costs for working with Urbit ID and the friction associated with using cryptocurrency in general. This system allows batches of transactions with `azimuth.eth` to be submitted together as a single transaction, using an Urbit node known as a "roller". The PKI state transitions resulting from these transactions are computed locally by your urbit rather than by the [Ethereum Virtual Machine](https://ethereum.org/en/developers/docs/evm/).

Due to the extremely low cost, Tlon offers their own roller that is free for ordinary public use. This enables new users to get started with a permanent Urbit ID without any prior knowledge of Ethereum, cryptocurrency, or blockchains. Anybody can run a roller, and you can even use your own ship as a roller to submit single transactions.

#### Layer 2 resources <a href="#layer-2-resources" id="layer-2-resources"></a>

* [Layer 2 Overview](/urbit-id/l2) - An overview of how naive rollups work.
* [Custom Roller Tutorial](/urbit-id/roller-tutorial) - A guide to running your own L2 roller locally.
* [Actions Reference](/urbit-id/l2/l2-actions) - Details of the L2 API's possible actions.
* [Transaction Format Reference](/urbit-id/l2/bytestring) - Details of the bytestring format for L2 transactions and batches.
* [Roller](/urbit-id/l2/roller) - Overview of the L2 roller system.


# Azimuth Data Flow

Technical overview of Azimuth's data flow, including Bridge, Gall agents, the Layer 2 naive rollup solution, and how components in Urbit OS communicate in Urbit's PKI system on Ethereum.

This document summarizes the various components involved with Azimuth and how they communicate with each other. This also constitutes an explanation for how Urbit implements the data flow of naive rollups.

## Bridge <a href="#bridge" id="bridge"></a>

The primary way in which users interact with Azimuth is via [Bridge](https://bridge.urbit.org). Bridge is responsible for collecting transactions from users, signing them, and forwarding them to a roller via an HTTP API.

## Azimuth <a href="#azimuth" id="azimuth"></a>

Azimuth was originally defined as a set of smart contracts on Ethereum that defines the [state](/urbit-id/azimuth-eth) and [business logic](/urbit-id/ecliptic) of the PKI for layer 1. With the introduction of naive rollups, this has also come to include the set of components used for dealing with the PKI within Urbit, as now the complete PKI state is stored offchain (though this state is derived entirely from on-chain data). The following sections outline what each component is responsible for and how it communicates with the others.

The Gall agents involved with Azimuth are summarized as follows:

* [`%azimuth`](#azimuth) - obtains and holds PKI state.
* [`%azimuth-rpc`](#azimuth-rpc) - JSON RPC-API for `%azimuth`.
* [`%eth-watcher`](#eth-watcher) - Ethereum event log collector.
* [`%roller`](#roller) - submits batches of L2 transactions to Ethereum.
* [`%roller-rpc`](#roller-rpc) - JSON RPC-API for `%roller`.

The transaction processing library is [`/lib/naive.hoon`](#naive).

## Gall agents <a href="#gall-agents" id="gall-agents"></a>

### `%azimuth` <a href="#azimuth" id="azimuth"></a>

`%azimuth`, located at `/app/azimuth.hoon`, is a Gall agent and thread handler responsible for finding Azimuth transactions gathered by `%eth-watcher`, keeping track of the PKI state, and exposing that data via scries.

The following diagram illustrates `%azimuth`'s and `%eth-watcher`'s role in the system.

![](https://media.urbit.org/docs/layer2/l2-azimuth-azimuth.png)

The state held by `%azimuth` is the following.

```hoon
++  app-state
  $:  %3
      url=@ta
      whos=(set ship)
      nas=^state:naive
      own=owners
      logs=(list =event-log:rpc:ethereum)
  ==
```

`whos` is the set of ships currently known by Azimuth. `nas` is the PKI state, as defined in [`naive.hoon`](#naive). `own` is a `+jug` of Ethereum addresses and the set of ships owned by that address. `logs` is a list of all Azimuth-related Ethereum event logs known by the ship.

Scries can be inferred from the `+on-peek` arm:

```hoon
++  on-peek
  |=  =path
  ^-  (unit (unit cage))
  ?+  path  (on-peek:def path)
      [%x %logs ~]  ``noun+!>(logs.state)
      [%x %nas ~]   ``noun+!>(nas.state)
      [%x %dns ~]   ``noun+!>(dns.nas.state)
      [%x %own ~]   ``noun+!>(own.state)
  ==
```

### `%azimuth-rpc` <a href="#azimuth-rpc" id="azimuth-rpc"></a>

`%azimuth-rpc`, located at `app/azimuth-rpc.hoon`, is a JSON RPC-API for getting `point` and `dns` data from the Azimuth PKI state kept by `%azimuth`.

### `%eth-watcher` <a href="#eth-watcher" id="eth-watcher"></a>

`%eth-watcher`, located at `/app/eth-watcher.hoon`, is responsible for listening to an Ethereum node and collecting event logs from it. It is general-purpose and not particular to Azimuth. It sends collected transactions to `+on-agent` in `%azimuth`, which then obtains the resulting PKI state transitions by passing them to [`naive.hoon`](#naive).

![](https://media.urbit.org/docs/layer2/roller-agents.png)

### `%roller` <a href="#roller" id="roller"></a>

`%roller`, stored at `/app/roller.hoon`, is a Gall agent responsible for collecting and submitting batches of layer 2 transactions to the Ethereum blockchain. Among other things, it keeps track of a list of pending transactions to be sent, transactions it has sent that are awaiting confirmation, history of transactions sent organized by Ethereum address, and when the next batch of transactions will be sent. See also [Rollers](/urbit-id/l2/roller) for more information on the roller.

The following diagram illustrates how the roller interacts with Bridge and Ethereum at a high level.

![](https://media.urbit.org/docs/layer2/l2-high.png)

The relationship between the roller and other agents is outlined in the following diagram.

![](https://media.urbit.org/docs/layer2/roller-agents.png)

`%roller` has a number of scries available, intended primarily to display data to the end user in Bridge. They can be inferred from the `+on-peek` arm:

```hoon
  ++  on-peek
    |=  =path
    ^-  (unit (unit cage))
    |^
    ?+  path  ~
      [%x %pending ~]       ``noun+!>(pending)
      [%x %pending @ ~]     (pending-by i.t.t.path)
      [%x %tx @ %status ~]  (status i.t.t.path)
      [%x %history @ ~]     (history i.t.t.path)
      [%x %nonce @ @ ~]     (nonce i.t.t.path i.t.t.t.path)
      [%x %spawned @ ~]     (spawned i.t.t.path)
      [%x %next-batch ~]    ``atom+!>(next-batch)
      [%x %point @ ~]       (point i.t.t.path)
      [%x %points @ ~]      (points i.t.t.path)
      [%x %config ~]        config
      [%x %chain-id ~]      ``atom+!>(chain-id)
    ==
```

This app is not responsible for communicating with Bridge via HTTP. Instead, that is handled by `%roller-rpc`. The scries are also communicated to Bridge via `%roller-rpc`.

### `%roller-rpc` <a href="#roller-rpc" id="roller-rpc"></a>

`%roller-rpc`, stored at `/app/roller-rpc.hoon`, is a very simple Gall app responsible for receiving HTTP RPC-API calls, typically sent from other Urbit ID users via Bridge. It then translates these API calls from JSON to a format understood by `%roller` and forwards them to `%roller`. This app does not keep any state - its only purpose is to act as an intermediary between Bridge and `%roller`. See [here](/urbit-id/l2/layer2-api) for more information on the JSON RPC-API.

## `naive.hoon` <a href="#naive" id="naive"></a>

`/lib/naive.hoon` consists of a gate whose sample is a `verifier`, `chain-id=@ud`, `state`, and `input`, which outputs a cell of `[effects state]`. This is the transition function which updates the state of the PKI stored in `%azimuth` which handles state transitions caused by both layer 1 and layer 2 transactions. A high-level overview of how `naive.hoon` functions can be found [here](#naive).

A `verifier` is a gate whose sample is of the form `[dat=octs v=@ r=@ s=@]` and which returns `(unit address)`:

```hoon
+$  verifier  $-([dat=octs v=@ r=@ s=@] (unit address))
```

The `verifier` in use by `naive.hoon` runs the keccak hash function on `dat` to verify that `dat` is data signed by the ECDSA signature given by the `[v r s]` tuple, according to the format for signed transactions outlined in the [bytestring format](/urbit-id/l2/bytestring) documentation.

`chain-id` is the ID used by the Ethereum blockchain, which is `1337`. See [bytestring format](/urbit-id/l2/bytestring) for more information. This is used so that e.g. transactions on a test network cannot be replayed on the mainnet.

`state` is the current state of the PKI. This is structured similarly to the state held in [Azimuth.eth](/urbit-id/azimuth-eth), but will differ in general since `state` takes into account layer 2 transactions as well. See the [Layer 2 Overview](/urbit-id/l2) for more on how PKI state is handled.

```hoon
+$  state
  $:  =points
      =operators
      dns=(list @t)
  ==
+$  points     (tree [ship point])
++  point
  $:  ::  domain
      ::
      =dominion
      ::
      ::  ownership
      ::
      $=  own
      $:  owner=[=address =nonce]
          spawn-proxy=[=address =nonce]
          management-proxy=[=address =nonce]
          voting-proxy=[=address =nonce]
          transfer-proxy=[=address =nonce]
      ==
      ::
      ::  networking
      ::
      $=  net
      $:  rift=@ud
          =keys
          sponsor=[has=? who=@p]
          escape=(unit @p)
      ==
  ==
+$  dominion  ?(%l1 %l2 %spawn)
+$  operators  (jug address address)
```

`points` should be self-explanatory if you are already familiar with the structure of [Azimuth.eth](/urbit-id/azimuth-eth). The only new addition is `dominion`, whose value says whether a ship is on layer 1, layer 2, or layer 1 with a layer 2 spawn proxy. See [Layer 2 actions](/urbit-id/l2/l2-actions) for an overview of how `dominion` determines the PKI actions available to a ship.

`operators` already existed on layer 1 and are defined as a part of the [ERC-721 standard](https://eips.ethereum.org/EIPS/eip-721).

`dns` is a list of DNS entries by which galaxy IP addresses may be looked up. At present, this is always `~['urbit.org' 'urbit.org' 'urbit.org']`.


# Life and Rift

Explanation of life and rift numbers in Azimuth, how they track networking key revisions and breach counts for secure peer-to-peer networking on Urbit.

Associated to every Azimuth point are two non-negative integers known as *life* and *rift*. This numbering system partition messages according to the quantity of networking key changes and quantity of [breaches](/user-manual/id/guide-to-resets), respectively. This is explained in more detail below. These values are utilized by [Ames](/urbit-os/kernel/ames) and [Jael](/urbit-os/kernel/jael) to ensure that communication between ships is always done with the most recent set of networking keys, and that networking state is appropriately reset when a breach has occurred.

Every ship begins with a `$life` and `$rift` of 0. For galaxies, stars, and planets, these values are stored in the [Azimuth PKI](/urbit-id/azimuth-eth#points) as `keyRevisionNumber` and `continuityNumber`, while for moons, these values are stored by their parent. Comets cannot change their networking keys, nor can they breach, and so their `$life` and `$rift` are always 0.

You can check your current `$life` and `$rift` number by running the `+keys our` generator in dojo. You can inspect another ship's `$life` and `$rift` by running `+keys ~sampel-palnet`.

## Life <a href="#life" id="life"></a>

A ship's `$life`, or *key revision number*, is a count of the number of times which a ship's networking keys have been altered. The initial value of each key is always zero, and the initial `$life` is always 0.

Thus, setting the keys of a ship to a nonzero value for the first time will increment the `$life` from 0 to 1. Rotating to a new set of keys will then increment the `$life` to 2. Setting the keys back to zero would increment the `$life` once more, to 3.

## Rift <a href="#rift" id="rift"></a>

A ship's `$rift`, or *continuity number*, is a count of the number of times that a ship has breached, also known as broken continuity.

In other words, a ship's `$rift` will remain at 0 until the first time it is breached.

Network-wide breaches do not affect the `$rift` of any ship. They are only used to count the number of personal breaches.

## Edge cases <a href="#edge-cases" id="edge-cases"></a>

Configuring the keys to the same value they already were (i.e. a no-op) is possible, but has no effect on the `$life`. Thus `$life` is actually a measure of networking key *revisions*, and not the number of times they've been set.

Thus under ordinary circumstances, a breach will increment both `$life` and `$rift` since a breach typically also involves changing the networking keys. However, it is possible to breach without changing the networking keys. If the new keys specified as part of a breach are the same as the old, `$rift` will increment while `$life` will not.


# Urbit HD Wallet

Guide to Urbit's Hierarchical Deterministic wallet system. Master tickets, proxy keys, wallet generation, and security best practices for Urbit ID ownership.

Owners of Urbit identities need safeguards that allow for the use of Urbit without jeopardizing cryptographic ownership of their assets. Toward this end, we created the **Urbit Hierarchical Deterministic (HD) Wallet** for the storage of identities. The Urbit HD Wallet is not one key-pair, but a system of related key-pairs that each have distinct powers, from setting networking keys for communicating in the Arvo network to transferring ownership of identities.

The Urbit HD Wallet's derivation paths have a hierarchical structure, so that keys with different powers can be physically separated. A "master ticket" can [re-derive the entire wallet](#hd-wallet-generation) in case of loss. The encryption and authentication keys that identities ships use to sign messages within the network are also derived from the wallet.

Another HD wallet option you may wish to utilize to store your Urbit are hardware wallets such as Trezor or Ledger. We compare this method to the Urbit HD wallet [below](#hardware-hd-wallet).

Urbit HD wallets are composed of the following items, which are each assigned to their own individual Ethereum key-pairs.

## Master Ticket <a href="#master-ticket" id="master-ticket"></a>

Think of your master ticket like a very high-value password. The master ticket is the secret code from which all of your other keys are derived. Technically, your master ticket is a cryptographic seed. You should never share it with anyone, and store it very securely. This ticket can derive all of your other keys: your ownership key and all of the related proxies.

## Ownership Address <a href="#ownership-address" id="ownership-address"></a>

An ownership address has all rights over the assets deeded to it. These rights are on-chain actions described and implemented in Ecliptic, Azimuth's suite of governing smart-contracts.

## Proxies <a href="#proxies" id="proxies"></a>

Each permanent Urbit ID can designate one or more [proxies](/user-manual/id/proxies), which are Ethereum addresses capable of a limited subset of Urbit ID transactions, such as spawning planets or rotating keys. The HD wallet automatically generates additional addresses utilized as proxies according to what is appropriate for your Urbit ID.

## HD wallet generation <a href="#hd-wallet-generation" id="hd-wallet-generation"></a>

Your Urbit HD wallet is generated from a `@q` seed called `T`, which looks something like `~sampel-ticket-bucbel-sipnem`. This is the string known as your "Master Ticket" that you input into Bridge to sign in. This is put through a series of algorithms that ultimately generate your keys and the Ethereum addresses at which they are stored.

![](https://media.urbit.org/fora/proposals/UP-8.jpg)

First, your `@q` is converted into a numeric value `E` as an intermediary step by adding [salt](https://en.wikipedia.org/wiki/Salt_\(cryptography\)). Then by adding additional salts, `E` is converted into a set of BIP39 seed phrases - these are 24 word mnemonic sequences used to generate Ethereum wallets. You end up with one seed phrase for each proxy associated with your ship, and these seed phrases are then used to generate Ethereum wallets.

One of the wallets will store your Azimuth point, an [ERC-721](#erc-721) token, which will be known as your ownership address. Bridge then automatically uses your ownership address to assign the other proxies to the other wallets generated.

## ERC-721 <a href="#erc-721" id="erc-721"></a>

Most Ethereum tokens use the ERC-20 standard for smart contracts. Urbit identities are, however, essentially different from most Ethereum tokens, due to identities not being fungible. Since any two stars will handle social-networking realities in a different way, they will carry a different reputation.

The ERC-721 standard, having been made specifically to provide a smart-contract interface for non-fungible assets, serves our needs well. This is the standard that we use for deeding Urbit identities.

Identities, and all of their blockchain operations, are governed by Ecliptic. Ecliptic is an Ethereum smart-contract that governs identity state and the ownership, spawn, management, and voting rights affiliated with your identities.

For the technical implementation details, take a look at Azimuth's [Github repository](https://github.com/urbit/azimuth).

## Hardware HD wallet <a href="#hardware-hd-wallet" id="hardware-hd-wallet"></a>

You may also store your Urbit using a hardware HD wallet such as a Trezor or Ledger. This option is used instead of a master ticket - it is a totally separate process from that outlined above. How hardware wallets work are outside of the scope of this document, but here we explain how you may utilize your hardware wallet in a similar fashion to the Urbit HD wallet.

A hardware HD wallet is generated from a seed phrase that should be thought of as analogous to the `@q` `T` used to generate the Urbit HD wallet in the above diagram. This seed phrase is a human readable sequence of words that may be used to generate private keys to a near-limitless number of wallets created by adding salt. Some hardware wallets also allow an additional passphrase to go into the generation of these wallets. The seed phrase and passphrase should be considered equal in value to the master ticket `T`- never share these with anyone!

In order to use your hardware wallet as your Urbit wallet, you first need to generate a set of addresses on which you would like to store your Azimuth point and proxies. Consult your wallet vendor on how to do this. Then login to Bridge using your current method (likely a master ticket) and transfer ownership to the wallet address you generated on your hardware wallet for ownership.

Next, login to Bridge using your hardware wallet. While Bridge supports Trezor and Ledger natively, this may require using Metamask as an intermediary anyways, depending on which firmware you are running. Then [accept the transfer](/user-manual/id/using-bridge#accept-your-transfer). Your Azimuth point is now stored on your hardware wallet.

To finish the process, use Bridge to set your management, voting, and spawn proxies to the other addresses you generated as applicable. You are now using your hardware HD wallet in an analogous fashion to the Urbit HD wallet.

## Heightened security practices <a href="#heightened-security-practices" id="heightened-security-practices"></a>

A common security pattern is to have "hot" and "cold" wallets. For higher value points such as stars and galaxies, you may consider having your ownership address live on a "cold" wallet that never touches an internet-connected computer, and the various proxies on a "hot" wallet that is permitted to connect to internet-connected devices. This could be accomplished with multiple hardware wallets, a combination of paper and hardware wallet, a hardware cold wallet and Metamask hot wallet, etc. See the [User Manual](/user-manual/id/hd-wallet) for concrete suggestions on security practices.

One tool useful for this setup is `claz`, located at `app/claz.hoon`. `claz` is used for making and signing Bridge transactions from an offline computer. A guide to using `claz` is available [here](/urbit-id/advanced-azimuth-tools).


# Advanced Azimuth Tools

Advanced techniques for generating, signing, and sending Azimuth transactions from within Urbit using the %claz and %eth-sender agents for Layer 1 operations.

This document outlines some advanced techniques for generating, signing, and sending Azimuth-related transactions from within Urbit itself. It's strongly discouraged to do this unless you're very confident as Azimuth transactions are permanent and will cost ETH to complete. However, until you upload a signed transaction to the blockchain no transaction will take place so it's safe to explore these tools. It will also be very helpful to have at least a rudimentary understanding of Hoon in order to read and modify parts of the code.

These tools are currently only available for layer 1 transactions. Tooling for layer 2 transactions is forthcoming.

First you will want to [boot a fakezod](/build-on-urbit/environment#creating-a-fake-ship). You can do this with your live running ship but at this time this will need you to modify the code so it's best to do this on a ship you can make mistakes on.

Be sure to have mounted your base desk so you can easily edit files on it. The first edit that will probably need to be made is in [`app/claz.hoon`](https://github.com/urbit/urbit/blob/85435e9a81e105809d5d381b5d34fae1d4daa3b8/pkg/arvo/app/claz.hoon#L14). The listed url will need to be changed to a valid Ethereum endpoint. `infura.io` provides an Ethereum API that you can use.

The second change that may potentially be needed is to modify the [gas limit](https://github.com/urbit/urbit/blob/85435e9a81e105809d5d381b5d34fae1d4daa3b8/pkg/arvo/app/claz.hoon#L179). This number here is the maximum number of gas units that will be used. In future this will be configurable but for now you may need to change this manually. For a guide to what to set it to you can use the [constants in bridge](https://github.com/urbit/bridge/blob/29f4a14869489481a950a1af53f583d751897444/src/lib/constants.js#L23) as a guide based on what action you are going to be performing. The maximum cost of your transaction will be the maximum gas units multiplied by the gwei you're going to set later. You'll need at least this much in your account.

Once you have made these changes be sure to run `|commit %base` to get them into your ship.

Next run `|start %claz` after which you'll be able to generate transactions.

```
:claz [%generate %/spawn/eth-txs %mainnet author %single %spawn ~ship to]
```

Here we have an example of generating a transaction to spawn a ship. Something to note is that for this to be successful you'll need to actually have the right to spawn the ship in question. Let's break down the command.

`%generate` is how all commands to `claz` are going to start.

`%/spawn/eth-txs` is simply the path for the output file

`%mainnet` specifies which network this is going to sent out on. The full set of options is `%mainnet`, `%goerli`, `%fakenet` and `[%other id=@]` (`+network` in `/sur/claz.hoon`), where `[%other id]` lets you give a chain ID directly. Note that Goerli has since been deprecated by the Ethereum Foundation, so `[%other id]` is the practical choice for a testnet today.

`author` here is an Ethereum address that is issuing the transaction. It will need to be written in `@ux` notation, e.g. `0x3b17.d097.d9dd.711e.4ef8.517a.bbf1.8b2b.a643.81fe`.

`%single` specifies what kind of batch that will be created. If you are going to do multiple transactions you will need to either set this to `%more` followed by a *list* of batches, or generate the next transaction after having submitted the previous one to the blockchain.

`[%spawn ~ship to]` is the transaction we want to issue. The arguments are will vary based on the transaction. `~ship` will be the ship to be spawned. `+to` is the Ethereum address to spawn the ship to.

You will want to read [`/sur/claz.hoon`](https://github.com/urbit/urbit/blob/85435e9a81e105809d5d381b5d34fae1d4daa3b8/pkg/arvo/sur/claz.hoon). This will contain more details about the various kinds of transactions you can perform particularly `+call`

After running this command you should find a csv file at the output path you specified. You can examine this file to learn how they are constructed. You are able to modify the nonces, gas limit and price without changing the functionality of the transaction but it's not important for this guide.

This file can now be signed and uploaded. At this point, you can transfer this file to an airgapped machine which has your keys, if you want. If you choose to do that you will need to download the `urbit` binary as well as the most recent pill file or a copy of the fakezod pier you have already booted.

For this guide, we'll continue on the same ship.

Run `|start %eth-sender` and then you can use it to sign the transaction file.

```
:eth-sender [%sign %/spawn-signed/txt %/spawn/eth-txs %/mykey/txt ~[40 80]]
```

Let's break this down again.

`%sign` is the action we are going to perform here. `eth-sender` can also be used to submit batches of transactions to the network in various ways, but details on that are outside of the scope of this guide.

`%/spawn-signed/txt` is the path to the output file.

`%/spawn/eth-txs` is the path to the transaction file we generated earlier. It will need to be accessible so don't forget to `|commit %base` if you moved it to a different pier.

`%/mykey/txt` is the path to your private key file. It should contain your private key beginning with 0x and followed by a hex string with no dots, and where capital letters are fine. This file is most likely the reason you will want to use an airgapped machine so that your private key never touches a networked computer. Opsec is left to the reader.

`~[40 80]` is a list of gas prices, in gwei; a signed transaction file will be created for each one listed. Here we just specify 40 and 80 gwei. You may want to adjust this list based on current Ethereum network traffic and your time preference for the transactions.

Now you should have one file in your pier for each gwei costs.

To actually transmit the transaction, paste the contents of one of these files into <https://etherscan.io/pushTx> or whatever other transmission system you would like to use. **This is the action that will actually cost you ETH and perform the transaction on the network. Only do this step if you want to actually perform the action.**




---

[Next Page](/llms-full.txt/1)