9. Vanes II: Ames

Urbit knows about three kinds of networking: Ames and Fine over Ames and HTTP over Eyre. (That is, the network protocol over the implementing vane.) Ames is the name of both the network protocol and the implementing vane. Ames is particularly for ship-to-ship communication, while Fine acts as a dispatcher for efficient data requests (such as desk updates). In this lesson, we will focus on Ames first and then foray into Fine, since Fine is in more flux.

Ames is a good example of a vane that is conceptually straightforward but mechanically complicated. It handles networking, but has to track peer state, message flows, individual packets, network weather, etc. as gracefully as possible.

Network Protocol: Ames

Ames is an encrypted peer-to-peer network running as an overlay over UDP. Ames does not have separate addressing and identity layers (like IP and DNS). An Ames address is an identity, mapped to a phonemic string to create a memorable pseudonym, and bound to a public key for encrypted communication. (Whitepaper)

From a kernel perspective, the point of Ames is to extend move semantics across more than one Arvo instance. Local vanes (such as Gall) pass a %plea request to Ames, which sends the message to the peer Ames over the wire. That peer's Ames then dispatches the message to the destination vane on the peer. One advantage of Ames is that it wraps all of the peer negotiation and message delivery details such that the calling vane need not be aware of these. Among Ames’ guarantees:

1. Messages within a flow are processed in order.
2. Messages will be delivered only once to a destination vane. (“Ames can guarantee exactly-once delivery because both ships are transactional (so if they give an ack, we know for sure they have received it permanently and won't forget about it.” ~wicdev-wisryt↗)

Networking in some ways is like a duct. It requires you to keep track of the forward and reverse causal history and content. However, there are two kinds of data transmissions: commands and facts.

There is a categorical difference between a bus, which transports commands, and a network, which transports packets. You can drop a packet but not a command; a packet is a fact and a command is an order. Facts are inherently idempotent; learning fact X twice is the same as learning it once. You can drop a packet, because you can ignore a fact. Orders are inherently sequential; if you get two commands to do thing X, you do thing X twice. (Whitepaper)

The Ames vane is responsible for sending and receiving messages of arbitrary length. This means that it needs to know how to build and reconstruct component packets of a message, how to route, and how to encrypt and decrypt. Ames does handle some aspects of encryption and decryption but not all. For networking, Ames gets its public keys (and breach notifications) from Jael, which in turn gets them from an Azimuth userspace agent. The actual cryptographic operations may live in /sys/zuse but are applied by Ames as appropriate.

• “Ames Overview”
• Curtis Yarvin ~sorreg-namtyv, Philip Monk ~wicdev-wisryt, Anton Dyudin, and Raymond Pasco, “Urbit: A Solid-State Interpreter” (“Whitepaper”)↗, sections 9–10
• “Ames Security Audit and the Future of the Protocol”↗

Packet Protocol

Ames receives packets as Arvo events and emits packets as Arvo effects. The runtime is responsible for transferring the bytes in an Ames packet across a physical network to another ship. (Ames Tutorial)

Ames packets have a 32-bit header followed by a variable-length body.

Header

The 2-bit address size refers to the address space rank (gathering galaxies and stars together as routers).

++  ship-meta
  |=  =ship
  ^-  [size=@ =rank]
  =/  size=@  (met 3 ship)
  ?:  (lte size 2)  [2 %0b0]
  ?:  (lte size 4)  [4 %0b1]
  ?:  (lte size 8)  [8 %0b10]
  [16 %0b11]

A relay means that the packet is not at its destination here and should be passed forward. (This is handled by ++on-hear-forward in /sys/ames.)

If a relay responds to a scry request from its cache without asking the host, the relay should include an origin containing the last known IP and port of the host. … [The] protocol should be resilient against the origin pointing at an unreachable IP and port.

Body

Address size is determined by the header.

Here if the relay bit is set then 32 bits of the origin are the last known IPv4 address and 16 bits are the port.

The ciphertext is formed by +jamming a $shut-packet and then encrypting using +en:sivc:aes:crypto.

The ciphertext results from ++jamming the message noun into an atom then breaking the result into 1 KB or smaller payloads. Packets are numbered so that they can be ordered upon receipt. These message fragments are then assembled into a single large atom and ++cued back into the noun.

Urbit messages result in raw nouns. Since Nock-derived languages are homoiconic, we could treat this noun as code directly, but instead we treat it as a cask (pair of mark and noun). We don't transmit vases over the network, but require the recipient to build the code locally.

Ames messages are typed; the type itself is not sent, just a label (like a MIME type) that the recipient must map to a local source path. Validation failure causes a silent packet drop, because its normal cause is a recipient that has not yet received a new protocol update; we want the sender to back off. Ames also silently drops packets for encryption failure; error reports are just an attack channel.

UDP Packet Format

There's a lot you can do with a stateful UDP server, especially one whose semantics are reasonably formal. (CGY)

At the host system level, the runtime communicates using the User Datagram Protocol↗ (UDP) specification. UDP messages are “transaction oriented, and delivery and duplicate protection are not guaranteed.” (To compensate for this, Ames employs a unique system of acks and nacks, covered below.) Each UDP message has a brief header including destination, source, length, and checksum. It’s rather a “minimum viable” packet system.

A UDP datagram consists of a datagram header followed by a data section (the payload data for the application). The UDP datagram header consists of 4 fields, each of which is 2 bytes (16 bits). UDP is faster but less reliable than TCP, another common transport protocol. In a TCP communication, the two computers begin by establishing a connection via an automated process called a ‘handshake.’ Only once this handshake has been completed will one computer actually transfer data packets to the other. (Wikipedia↗)

Urbit compensates for this lower reliability by sending until receiving an appropriate ack or nack (negative acknowledgment) in reply.

UDP is commonly used in time-sensitive communications where occasionally dropping packets is better than waiting. Voice and video traffic are sent using this protocol because they are both time-sensitive and designed to handle some level of loss. For example VOIP (voice over IP), which is used by many internet-based telephone services, operates over UDP. This is because a staticky phone conversation is preferable to one that is crystal clear but heavily delayed. (Wikipedia↗)

• RFC 768↗ (UDP specification)

Acks and Nacks

::  $ack: positive ack, nack packet, or nack trace
::
+$  ack
  $%  [%ok ~]
      [%nack ~]
      [%naxplanation =error]
  ==

If every message is a transaction (or event), then what is Ames acknowledging (ack) or negatively acknowledging (nack)? “A successful transaction has no result; a failed transaction is a negative ack and can contain an error dump.”

• An ack means that a piece of information has been received successfully.
• A nack means that a piece of information has been received but failed to process for some reason.

Ames has an unusual system of acks and nacks (“negative acknowledgments”, but not like TCP’s packets of the same name; Ames nacks mean the packet was received but the message resulted in an error). In brief, each Ames packet of a message should get either an ack or a nack. In the current system, the nack may include an error message (e.g., an error code or a stack trace). (~wicdev-wisryt↗)

Each Ames packet will merit either an ack or a nack, in other words. A nack may optionally include an error trace with it ([tag=@tas =tang]). Ames will adaptively continue to send messages until the appropriate acks or nacks have been received.

(A TCP nack means that the numbered packet was never received.)

Ames will send messages and acks until a replying ack is received. “Ames guarantees that a message will only be delivered once to the destination vane.” Thus nacks allow us to also guarantee notification that a request was completed or failed.

1. Always ack a dupe; never ack an ack. It's okay to ack a nack as long as you never nack a nack. (Urbit Precepts B.1↗)

If a remote ship sends a nack in response to a %plea, Ames waits until it receives a follow-up naxplanation and then delivers both to the local source vane. The flow blocks on needing to receive the naxplanation.

When a new socket is opened, the client can resend (at-least-once delivery) or fail to resend (at-most-once). The programmer has to understand that the socket is not really a bus, and make sure the POST is actually an idempotent fact rather than an imperative command. (The idempotence problem is often punted to the human layer: “Please click only once to make your purchase.”) (Whitepaper)

Because Ames and Urbit assume several nines of uptime, sessions between ships are treated as persistent.

The basic argument for including end-to-end acks (and by extension, nacks) is that they’re necessary for everything except those things which we don’t care whether the message was received at all. Thus, for Ames to give the guarantee that “if you give me a payload I will get it to the other side exactly once” isn’t useful in itself, because no application cares about that. They either (1) don’t care whether it gets there or (2) care whether the request itself was “completed”, in an application-defined sense. (Phillip Monk, ~wicdev-wisryt↗)

Keep in mind Postel’s law, also known as the robustness principle: “Be conservative in what you send, and liberal in what you accept.”

• Saltzer, Reed, and Clark, “End-to-End Arguments in System Design”↗ on nacks

Cryptography

Almost every Ames packet is encrypted using AES-256↗. (The exception is comet self-attestation packets.)

Urbit's cryptographic suite for jets has been organized to present a uniform interface (currently in urcrypt/, migrating to its own repo). This eases the development of alternative runtimes since the C functions can be utilized as an FFI (foreign function interface) with uniform call signatures and behavior.

The details of the Azimuth PKI are discussed in ca13, quod vide.

• “Ames: Cryptography”

Routing

The runtime tells Ames which physical address a packet came from, represented as an opaque atom. Ames can emit a packet effect to one of those opaque atoms or to the Urbit address of a galaxy (root node), which the runtime is responsible for translating to a physical address. (See +$lane.)

::  $address: opaque atomic transport address to or from unix
+$  address  @uxaddress
::  $lane: ship transport address; either opaque $address or galaxy
+$  lane  (each @pC address)

The @uxaddress value is an “opaque” address, in reality an IPv4 address for the runtime's use.

Since galaxy addresses are provided to the runtime on boot (from an RPC call to a roller or Ethereum node, see vere/dawn.c:_dawn_eth_rpc()), a route is always findable for any active point. Galaxy ports are hardcoded to be at 31337 or 13337 plus the galaxy numeric offset.

• What does @pC mean?

When we say that galaxies handle routing in Ames today (but stars will play a role later), this is the part of the system to which we refer.

/sys/lull Definition

The /sys/lull interface definition for Ames is quite long and should be reviewed in its entirety. A structural summary:

++  ames  ^?
  |%
  ::  $task: job for ames
  +$  task
    $+  ames-task
    $%  [%hear =lane =blob]
        [%dear =ship =lane]
        [%heed =ship]
        [%jilt =ship]
        [%cork =ship]
        [%tame =ship]
        [%kroc bones=(list [ship bone])]
        $>(%plea vane-task)
        [%deep =deep]
    ::
        [%keen spar]
        [%yawn spar]
        [%wham spar]
    ::
        $>(%born vane-task)
        $>(%init vane-task)
        [%prod ships=(list ship)]
        [%sift ships=(list ship)]
        [%snub form=?(%allow %deny) ships=(list ship)]
        [%spew veb=(list verb)]
        [%cong msg=@ud mem=@ud]
        [%stir arg=@t]
        $>(%trim vane-task)
        $>(%vega vane-task)
    ==
  ::  $gift: effect from ames

  +$  gift
    $%  [%boon payload=*]
        [%clog =ship]
        [%done error=(unit error)]
        [%lost ~]
        [%send =lane =blob]
    ::
        [%tune spar roar=(unit roar)]
    ::
        [%turf turfs=(list turf)]
    ==

Tasks

::  $task: job for ames
::
::    Messaging Tasks
::
::    %hear: packet from unix
::    %dear: lane from unix
::    %heed: track peer's responsiveness; gives %clog if slow
::    %jilt: stop tracking peer's responsiveness
::    %cork: request to delete message flow
::    %tame: request to delete route for ship
::    %kroc: request to delete specific message flows, from their bones
::    %plea: request to send message
::    %deep: deferred calls to %ames, from itself
::
::    System and Lifecycle Tasks
::
::    %born: process restart notification
::    %init: vane boot
::    %prod: re-send a packet per flow, to all peers if .ships is ~
::    %sift: limit verbosity to .ships
::    %snub: set packet blocklist to .ships
::    %spew: set verbosity toggles
::    %cong: adjust congestion control parameters
::    %stir: recover from timer desync and assorted debug commands
::    %trim: release memory
::    %vega: kernel reload notification

Ames has a rather bohemian set of messaging names. Among others:

• %hear a packet
• %heed or %jilt a peer
• %plea to send a message (common from vanes)

Most other tasks are not used by userspace but by internal Ames state management. These are complemented by types like these:

• +$hoot a request packet payload
• +$yowl a serialized response packet payload

Notes

::    Messaging Gifts
::
::    %boon: response message from remote ship
::    %clog: notify vane that %boon's to peer are backing up locally
::    %done: notify vane that peer (n)acked our message
::    %lost: notify vane that we crashed on %boon
::    %send: packet to unix
::    %tune: peek result
::    %turf: domain report, relayed from jael

Every vane can receive a %plea note from Ames (except Behn, Dill, Iris, Khan, Lick). This is a redirection mechanism used to forward messages that a peer's vane passed to its own Ames en route to your peer's Ames and thence to your vane.

State

::  $ames-state: state for entire vane
+$  ames-state
  $+  ames-state
  $:  peers=(map ship ship-state)
      =unix=duct
      =life
      =rift
      crypto-core=acru:ames
      =bug
      snub=[form=?(%allow %deny) ships=(set ship)]
      cong=[msg=_5 mem=_100.000]
    ::
      $=  dead
      $:  flow=[%flow (unit dead-timer)]
          cork=[%cork (unit dead-timer)]
  ==  ==

• +$peers are the state of connections to other ships, where +$ship-state is either %alien or %known.
  • %alien means we have no PKI data and we must queue moves until we learn how contact that ship. The +$alien-agenda stores messages, packets, and remote scry keens.
  • %known means that we do have the peer state, on which more later.
• +$unix-duct is a duct of moves to be sent to the host OS.
• +$life is our own life, or how many times we rekeyed.
• +$crypto-core is a handle to the cryptographic tools core.
• +$bug describes the debug level (|ames-verb).
• +$snub tracks a blocklist for incoming packets (|ames-snub).
• +$cong tracks whether a flow should be considered clogged.
• +$dead sets how long dead flows last and if they need to be restarted.

Ames maintains a duct (queue) of ordered messages. These are passed to and received from the runtime, and represent Arvo events. Each message is encrypted at the source and decrypted at the destination using a symmetric public-key system. A message may be a %plea (sent to another ship); in response, Ames can receive zero or more %boons. The ack–nack system is explained above; note that nacks are in response to event crashes.

In /sys/vane/ames, there is a layer of versioning cruft to permit upgrades of the types (e.g. +$ames-state-5).

Peer State

::  $peer-state: state for a peer with known life and keys
::
::    route: transport-layer destination for packets to peer
::    qos: quality of service; connection status to peer
::    ossuary: bone<->duct mapper
::    snd: per-bone message pumps to send messages as fragments
::    rcv: per-bone message sinks to assemble messages from fragments
::    nax: unprocessed nacks (negative acknowledgments)
::         Each value is ~ when we've received the ack packet but not a
::         nack-trace, or an error when we've received a nack-trace but
::         not the ack packet.
::
::         When we hear a nack packet or an explanation, if there's no
::         entry in .nax, we make a new entry. Otherwise, if this new
::         information completes the packet+nack-trace, we remove the
::         entry and emit a nack to the local vane that asked us to send
::         the message.
::    heeds: listeners for %clog notifications
::    closing: bones closed on the sender side
::    corked:  bones closed on both sender and receiver
::
+$  peer-state
  $+  peer-state
  $:  $:  =symmetric-key
          =life
          =rift
          =public-key
          sponsor=ship
      ==
      route=(unit [direct=? =lane])
      =qos
      =ossuary
      snd=(map bone message-pump-state)
      rcv=(map bone message-sink-state)
      nax=(set [=bone =message-num])
      heeds=(set duct)
      closing=(set bone)
      corked=(set bone)
      keens=(map path keen-state)
  ==

Structure

Ames’ formal interface is included more than once (like Arvo) as the “external vane interface” and the “adult ames”, for instance.

++  call  :: handle request stack
++  take  :: handle response $sign
++  stay  :: extract state before reload
++  load  :: load in old state after reload
++  scry  :: dereference namespace

There is a collection of ames-helper cores as well to handle many specific cases for unpacking and routing messages. Ames uses a more sophisticated nested core pattern than Behn did. To that end, it presents five ++abet cores:

• ev event handling core
• mi message receiver core
• mu message sender core
• pe per-peer processing core
• pu packet pump

There is a substantial amount of legacy Ames state upgrade debris in the file as well.

Scries

As typical, scries expose internal vane state. Ames has a richer inner life than some other vanes, so you can check on peer state and snubs and message flow details.

.^((map ship ?(%alien %known)) %ax /=//=/peers)

.^(ship-state:ames %ax /=//=/peers/~zod)

!< message-pump-state:ames .^(vase %ax /=//=/snd-bones/~zod/0)

Most Ames scries aren't particularly useful to us directly unless we want to do direct network negotiation. Ames is used frequently by Gall but, from the agent's perspective, incidentally.

Messages & Flows

We've looked at the packet protocol before; now let's look at Ames' mechanics of message management.

+$  fragment       @uwfragment
+$  fragment-num   @udfragmentnum
+$  message-blob   @udmessageblob
+$  message-num    @udmessagenum

Messages are separated into 1 KB (or smaller) fragments and sequentially numbered.

• Examine ++split-message to see how messages are broken up into pieces. (There's a neat optimization therein.)

Messages are of course sent and received in fragments. The messages from a lane accrue for a particular bone using ++mi, the message receiver core (internal alias sink).

• Examine ++hear and ++assemble-fragments.
• Start the debug server (|start %dbug) and navigate to /~debug. Select ames to see message flows.

The message pump manages unsent messages, dispatching them to the packet pump when next in the queue.

When we pop a message off .unsent-messages, we push as many fragments as we can into |packet-pump, which sends every packet it eats. Packets rejected by |packet-pump are placed in .unsent-fragments. When we hear a packet ack, we send it to |packet-pump to be removed from its queue of unacked packets. When we hear a message ack (positive or negative), we treat that as though all fragments have been acked.

There are a ton of other edge cases and consistency/sanity checks on messaging, one of the reasons that Ames is relatively complicated.

At the end of a task, |message-pump sends a %halt task to |packet-pump, which can trigger a timer to be set or cleared based on congestion control calculations. When the timer fires, it will generally cause a packet to be re-sent. Message sequence numbers start at 1 so that the first message will be greater than .last-acked.message-sink-state on the receiver.

+$  message-pump-state
  $+  message-pump-state
  $:  current=_`message-num`1
      next=_`message-num`1
      unsent-messages=(qeu message-blob)
      unsent-fragments=(list static-fragment)
      queued-message-acks=(map message-num ack)
      =packet-pump-state
  ==
::
+$  static-fragment
  $:  =message-num
      num-fragments=fragment-num
      =fragment-num
      =fragment
  ==
::
+$  partial-rcv-message
  $:  num-fragments=fragment-num
      num-received=fragment-num
      fragments=(map fragment-num fragment)
  ==

A vane can pass Ames a %heed task to request Ames track a peer's responsiveness. If our %boons to it start backing up locally, Ames will give a %clog back to the requesting vane containing the unresponsive peer's Urbit address.

+$  qos
  $~  [%unborn *@da]
  [?(%live %dead %unborn) last-contact=@da]

To cork a flow (or “cork a bone”) closes the flow. A dangling bone refers to an incorrect bone (a message flow was closed on one side before all message fragments were received, for instance).

A message flow organizes a sequence of message fragments together. Within a flow, data order is guaranteed; however due to network traffic flows may arrive out of order.

+$  bone           @udbone
::
+$  ossuary
  $:  =next=bone
      by-duct=(map duct bone)
      by-bone=(map bone duct)
  ==

A +$bone is a duct handle, a way of identifying a particular message flow over the network.

Each bone increments by 4 since each flow includes a least-significant bit indicating if we send or receive pleas and a second-least-significant bit indicating if we are a diagnostic flow (naxplanation) or not.

> *bone
0

> .^([snd=(set bone) rcv=(set bone)] %ax /=//=/bones/~nes)
[snd={0} rcv={}]

The +$ossuary holds the bone↔duct bijection and the next-bone to map to a duct. (Thus the increment-by-four noted above.)

::  $pump-metrics: congestion control state for a |packet-pump
::
::    This is an Ames adaptation of TCP's Reno congestion control
::    algorithm.  The information signals and their responses are
::    identical to those of the "NewReno" variant of Reno; the
::    implementation differs because Ames acknowledgments differ from
::    TCP's, because this code uses functional data structures, and
::    because TCP's sequence numbers reset when a peer becomes
::    unresponsive, whereas Ames sequence numbers only change when a
::    ship breaches.
::
::    A deviation from Reno is +fast-resend-after-ack, which re-sends
::    timed-out packets when a peer starts responding again after a
::    period of unresponsiveness.
::
::    If .skips reaches 3, we perform a fast retransmit and fast
::    recovery.  This corresponds to Reno's handling of "three duplicate
::    acks".
::
::    rto: retransmission timeout
::    rtt: roundtrip time estimate, low-passed using EWMA
::    rttvar: mean deviation of .rtt, also low-passed with EWMA
::    ssthresh: slow-start threshold
::    cwnd: congestion window; max unacked packets
::
+$  pump-metrics
  $:  rto=_~s1
      rtt=_~s1
      rttvar=_~s1
      ssthresh=_10.000
      cwnd=_1
      counter=@ud
  ==

Vere I/O Driver: vere/io/ames.c

As elsewhere, the libuv event loop processor with callback functions responds to Ames-specific initiating events, in this case, the receipt of a UDP packet. The C side of Ames handles constructing and dispatching the UDP packets that underlie Ames communications, but perhaps surprisingly ames.c is actually less complicated and interesting than ames.hoon. (There's some serialization handling too.)

• _ames_czar_cb() for galaxy address resolution
• _ames_send_cb() for UDP transmission
• _ames_recv_cb() for UDP reception

Network Protocol: Fine

A scry is a read-only request into the scry namespace. Historically, only local scries were supported, and these were instrumented synchronously using .^ dotket. With the addition of remote scry, a new use case and use pattern emerged: asynchronous reads over the network.

A ship that wants to read from a remote part of the namespace will have to pass a %keen task to its Ames, which then cooperates with Vere to produce the desired data. In some future event when the result is available, Ames gives it back as a %tune gift. From the requester's perspective, this is the entire default lifecycle of a remote scry request.

::    Remote Scry Tasks
::
::    %keen: peek: [ship /vane/care/case/spur]
::    %yawn: cancel request from arvo
::    %wham: cancels all scry request from any vane
::

Fine maintains its own state, but other than having its own types its operation is not so different from Ames that we need to delve into it hear.

+$  keen-state
  $+  keen-state
  $:  wan=((mop @ud want) lte)  ::  request packets, sent
      nex=(list want)           ::  request packets, unsent
      hav=(list have)           ::  response packets, backward
      num-fragments=@ud
      num-received=@ud
      next-wake=(unit @da)
      listeners=(set duct)
      metrics=pump-metrics
  ==
+$  want
  $:  fra=@ud
      =hoot
      packet-state
  ==
+$  have
  $:  fra=@ud
      meow
  ==
::
+$  meow  ::  response fragment
  $:  sig=@ux  ::  signature
      num=@ud  ::  number of fragments
      dat=@ux  ::  contents
  ==
::
+$  peep  ::  fragment request
  $:  =path
      num=@ud
  ==
::
+$  wail  ::  tagged request fragment
  $%  [%0 peep] :: unsigned
  ==
::
+$  roar  ::  response message
  (tale:pki:jael (pair path (unit (cask))))
::
+$  purr  ::  response packet payload
  $:  peep
      meow
  ==

(Having worked with remote scries some in userspace, I recommend tombstoning old endpoints when they are done being used.)

Runtime Scry Dispatch

Remote scries are handled by the runtime rather than generating an Arvo event.

In vere/io/ames.c, a scry hashtable sac_p is created. ames_hear() decides whether to inject the packet into Arvo (Ames protocol) or handle in Vere (Fine protocol).

• _fine_hear_request() to receive a request
• _fine_hear_response() to receive the response
• _fine_get_cache()
• _fine_put_cache()

There are also provisions for Fine scry path length etc. therein.

• “Guides: Remote Scry”
• ~rovnys-ricfer, “Remote Scry Protocol Proposal”↗