10. Vanes III: Eyre, Iris

This lesson covers Arvo's HTTP server vane (Eyre) and its HTTP client vane, Iris. We discuss how Arvo interacts with clients over HTTP.

Servers

One of Urbit's primary use cases is to act as a “personal server”. To examine this statement, we need to consider what a server does. Etymologically, a server serves a service. Generally speaking, it is the locus of a computation and coordination process. A server program is a system daemon—and since Gall agents are essentially daemons in many respects, Urbit's execution model fulfills this niche nicely.

Some servers are physical or logical devices which talk to other devices as clients. Internet webpage and application servers typically follow this model. Other servers are software processes that run on the same hardware or local network as the client process, e.g. mail servers, print servers, and file servers.

The two major operational models for servers are the request–response model and the publish–subscribe (pub-sub) model. The request–response pattern corresponds to pokes and gifts in Arvo terms, while the pub-sub pattern is supplied by subscriptions and updates.

• A client originates and submits requests, and receives responses.

• A server accepts requests and replies with responses.

A client and a server need to agree on a communications protocol. There are many of these, but the basis for the World Wide Web is the HyperText Transfer Protocol (HTTP).

Serving a Web Page

Two of the simplest actions one can take with a basic web server are to simply post a web page to any clients and to respond to interactions with that web page. Some interactions take place purely in the client session (form entry in the browser before submission), but then are propagated to the server.

Requests

HTTP requests are like Gall agent pokes: they are messages to trigger some action on the server. A method is specified (like GET, PUT, or POST) and the associated service-specific data follow.

It consists of a block of request headers, a block of general headers, and a block of representation headers. These may by followed by the body.

• GET means a read-only request for information (like an Urbit scry but without the bound namespace).

• PUT requests a state creation or update.

• POST asks for the server to process data. (Both PUT and POST are analogous to Urbit pokes.)

Responses

A server response to a web page looks like this:

The response code is normatively 200 OK for a successful page access, but 404 Not Found and other errors and special messages also occur frequently. (It would be very interesting if Urbit would implement 402 Payment requested.)

The actual mechanics of communicating both kinds of communications are wrapped by /lib/server and Eyre. Generally speaking, an agent will receive HTTP requests in ++on-poke, and commonly includes a ++handle-http arm to deal with inbound-request:eyre values. /lib/server has request header parsers and response handlers which make it easy to respond appropriately (e.g. (send:server ~ [%login-redirect './apps/my-agent'])).

Eyre

Eyre is an HTTP server, which receives HTTP messages from Unix and produces HTTP messages in reply. Your agent can register endpoints which a browser or other tool can interact with. Eyre can be instrumented to work with threads and generators.

HTTP requests include a method tag. While other methods exist, we are primarily interested in POST, PUT, and GET requests. Since we don't want to deal with client-side code yet, we're going to use curl to send requests here.

• POST is only used with Eyre to obtain a cookie.

$ curl -i localhost:8080/~/login -X POST -d "password=lidlut-tabwed-pillex-ridrup"
HTTP/1.1 204 ok
Date: Tue, 19 Jul 2022 16:28:05 GMT
Connection: keep-alive
Server: urbit/vere-1.9
set-cookie: urbauth-~nec=0v3.pis4a.sfdhv.f1p6i.lttba.gp93q; Path=/; Max-Age=604800

This cookie should be included in subsequent requests.

• PUT requests are used to send actions to Eyre: pokes, subscriptions, acks, unsubscribe requests, and channel deletions.

• GET requests are used to connect to a channel and receive any pending events. (Remember how Urbit prefers a dataflow computing model?)

/sys/lull Definition

::                                                      ::::
::::                    ++eyre                            ::  (1e) http-server
  ::                                                    ::::
++  eyre  ^?
  |%
  +$  gift
    $%  $>(?(%boon %done) gift:ames)   ::  Ames responses
        [%set-config =http-config]   ::  configure external HTTP server
        [%sessions ses=(set @t)]     ::  valid auth cookies
        [%response =http-event:http]   ::  response to event from Earth
        [%bound accepted=? =binding]   ::  response to %connect or %serve
        [%grow =path]                ::  notification on cache entry change
    ==
  ::
  +$  task
    $~  [%vega ~]
    $%  $>(%init vane-task)        ::  initialize ourself with an identity
        $>(%born vane-task)        ::  new unix process
        $>(%plea vane-task)        ::  network request
        $>(%trim vane-task)        ::  trim state (memory pressure)
        $>(%vega vane-task)        ::  report upgrade
        ::
        [%live insecure=@ud secure=(unit @ud)]            ::  live HTTPS ports
        [%rule =http-rule]         ::  update HTTP configuration
        [%eauth-host host=(unit @t)]   ::  set base URL for eauth
        [%request secure=? =address =request:http]        ::  handle inbound
        [%request-local secure=? =address =request:http]  ::  handle backdoor
        [%cancel-request ~]        ::  cancel previous request
        [%connect =binding app=term]   ::  connects a binding to an app
        [%serve =binding =generator]   ::  connect binding to generator
        [%disconnect =binding]         ::  disconnect binding to generator
        ::
        [%code-changed ~]          ::  web login code changed
        [%approve-origin =origin]  ::  accept CORS requests from origin
        [%reject-origin =origin]   ::  reject CORS requests from origin
        [%spew veb=@]              ::  set verbosity
        [%set-response url=@t entry=(unit cache-entry)]   ::  cache mapping
    ==
--  ::eyre

Eyre is responsible for a few subsystems that facilitate userspace applications (unlike, say, Behn or Ames, most of what Eyre does is to support userspace).

• Authentication

• Channels

• Threads and generators

• HTTP request handling

• Scry interface

Several of these are handled by the +$action dispatch system, invoked when a binding matches a known path.

A +$binding is a system unique mapping for a path to match. A +$binding must be system unique because we don't want two handlers for a path; what happens if there are two different actions for [~ /]?

::  +binding: A rule to match a path.
::
+$  binding
  $:  site=(unit @t)        ::  site: the site to match (~ for your.urbit.org)
      path=(list @t)        ::  path: matches this prefix path
  ==
::  +action: the action to take when a binding matches an incoming request
::
+$  action
  $%  [%gen =generator]     ::  dispatch to a generator
      [%app app=term]       ::  dispatch to an application
      [%authentication ~]   ::  internal authentication page
      [%eauth ~]            ::  cross-ship authentication handling
      [%logout ~]           ::  internal logout page
      [%channel ~]          ::  gall channel system
      [%scry ~]             ::  gall scry endpoint
      [%name ~]             ::  respond w/ @p requester is authenticated as
      [%host ~]             ::  respond w/ @p of the ship serving the response
      [%four-oh-four ~]     ::  respond with the default file not found page
  ==

/sys/vane/eyre is relatively more straightforward than (say) Ames. There is only one interface (search for ~% %http-server).

Authentication

Client sessions typically require a login. (This is not true for materials served to the clearweb, e.g. via %blog.) A cookie is generated for each session in response to a login using +code.

::  +authentication-state: state used in the login system
::
+$  authentication-state
  $:  sessions=(map @uv session)  ::  map of cookies to session information
      visitors=(map @uv visitor)  ::  visitors: in-progress incoming eauth flows
      visiting=(map ship logbook) ::  visiting: outgoing eauth state per ship
      endpoint=[user=(unit @t) auth=(unit @t) =time]  ::  endpoint: hardcoded 
                                                      ::  local eauth endpoint
                                                      ::  for %syn and %ack
  ==

(Visits are part of the EAuth system, q.v.)

Authentication is handled by ++authentication.

• See ++authentication in /sys/vane/eyre. Locate where the session cookie is created and logged.

Authentication is enforced by ++request-is-logged-in and ++request-is-authenticated.

• See ++request-is-logged-in in /sys/vane/eyre.

Channels

Channels are the main method where a webpage communicates with Gall apps. Subscriptions and pokes are issues with PUT requests on a path, while GET requests on that same path open a persistent EventSource channel. The EventSource API is a sequence number based API that browser provide which allow the server to push individual events to the browser over a connection held open. In case of reconnection, the browser will send a 'Last-Event-Id' header to the server; the server then resends all events since then.

An EventSource interface is a way to track server-sent events for a client session. The JS on the browser/client-side receives text/event-stream formatted events. So a channel is a given connection to a browser including the EventSource connection.

::  channel: connection to the browser
::
+$  channel
  $:  mode=?(%json %jam)
      =identity
      state=(each timer duct)
      next-id=@ud              ::  next-id: next sequence number to use
      last-ack=@da             ::  last-ack: time of last client ack
      events=(qeu [id=@ud request-id=@ud =channel-event])  ::  unacked events
      unacked=(map @ud @ud)    ::  unacked event counts by request-id
      subscriptions=(map @ud [ship=@p app=term =path duc=duct])  ::  gall subs
      heartbeat=(unit timer)   ::  sse heartbeat timer
  ==
::  +session: server side data about a session
::
+$  session
  $:  =identity                ::  authentication level & id of this session
      expiry-time=@da          ::  when this session expires
      channels=(set @t)        ::  channels opened by this session
  ==
::  channel-state: state used in the channel system
::
+$  channel-state
  $:  session=(map @t channel)   ::  mapping b/w an arbitrary key to a channel
      duct-to-key=(map duct @t)  ::  mapping from ducts to session key
  ==
::  channel-event: unacknowledged channel event, vaseless sign
::
+$  channel-event
  $%  $>(%poke-ack sign:agent:gall)
      $>(%watch-ack sign:agent:gall)
      $>(%kick sign:agent:gall)
      [%fact =desk =mark =noun]
  ==

Conventional channels communicate in JSON. Values passed into Urbit can be sent through a mark file to be transformed into a %noun or other type automatically. On the way out, a similar transformation can take the values back into MIME types.

• Trace %poke and %poke-json in /sys/vane/eyre.

• Examine /lib/schooner and the /mar files in the %yard desk. How does it handle JSON transformations? What about binary types like audio/mpeg (MP3)?

Noun channels make it possible for external applications to speak Urbit nouns. This means that you can communicate with an Urbit ship in a way other than using a JSON payload. The content-type is marked as application/x-urb-jam. Nouns are ++jammed when sent into Eyre.

• Locate where x-urb-jam is processed in /sys/vane/eyre and in /mar/noun.

Thanks to the mark system and ++find-channel-mode, it is straightforward on Urbit's side to implement noun channels. However, on the other side you need something that speaks nouns, such as noun.py.

• “Guide: Noun channels”

• ~nordus- mocwyl, “bird-brained guide to noun channels”

HTTP request handling

All else set aside, the real purpose of Eyre is to act as the HTTP server for an Urbit ship. Eyre maintains a server configuration. There is an +$inbound-request type to receive an HTTP request, but the main HTTP types are actually in another arm, ++http.

A raw HTTP request handle happens like this:

• Eyre subscribes to an app at /http-response/[eyre-id].

• Eyre pokes the app with %handle-http-request and the ID.

• The app produces %facts of ?(%http-response-header %http-response-data %http-response-cancel).

:: +http-config: full http-server configuration
::
+$  http-config
  $:  secure=(unit [key=wain cert=wain])  ::  PEM-encoded RSA private key and 
                                          ::  cert or cert chain
      proxy=_|                            ::  reverse TCP proxy HTTP(s)
      log=?                               ::  keep HTTP(s) access logs
      redirect=?               ::  send 301 redirects to upgrade HTTP to HTTPS
  ==
:: +http-rule: update configuration
::
+$  http-rule
  $%  [%cert cert=(unit [key=wain cert=wain])]  ::  set/clear cert and keypair
      [%turf action=?(%put %del) =turf]   ::  add/remove established dns binding
  ==
::  +address: client IP address
::
+$  address
  $%  [%ipv4 @if]
      [%ipv6 @is]
      ::  [%ames @p]
  ==
::  +inbound-request: +http-request and metadata
::
+$  inbound-request
  $:  authenticated=?          ::  has a valid session cookie
      secure=?                 ::  whether this request was encrypted (https)
      =address                 ::  the source address of this request
      =request:http            ::  the http-request itself
  ==

• Examine ++http. Find the request and response handlers. In particular, see +$simple-payload.

• Examine /lib/server, which contains wrapper arms for mere mortals.

In some ways, although this is the meat-and-potatoes of Eyre, it's all rather straightforward.

To actually get a value into userspace, Eyre sends the response to Gall:

• ++request-to-app to dispatch an %app +$action to Gall.

• ++deal-as to %pass to Gall.

Threads and generators

We can treat a (local) ship as a “serverless” function call for a client.

A generator is a standalone computation based on arguments. Eyre supports generators explicitly:

+$  generator
  $:  =desk           ::  desk on current ship that contains the generator
      path=(list @t)  ::  path on :desk to the generator's hoon file
      args=*          ::  arguments passed to the gate
  ==

and runs them in ++request, branch %gen.

• How does the generator run? Note the +$roof and the ++mock call.

Having bound a generator /gen/eyre-gen

|=  [[now=@da eny=@uvJ bec=beak] ~ ~]
|=  [authenticated=? =request:http]
^-  simple-payload:http
=/  msg=@t
  ?~  body.request
    (scot %da now)
  (cat 3 (cat 3 (scot %da now) 10) q.u.body.request)
=/  data=octs
  (as-octs:mimes:html msg)
=/  =response-header:http
  [200 ['Content-Type' 'text/plain']~]
[response-header `data]

to an endpoint /mygen,

|pass [%e [%serve `/mygen %base /gen/eyre-gen/hoon ~]]

a client may initiate a generator call by posting a PUT request thus:

curl -i http://localhost:8080/mygen --data 'blah blah blah'

• Examine the “Eyre Guide: Generators” example.

A thread is a transient standalone computation similar in some regards to a generator. Spider provides thread support using an Eyre binding.

++  handle-http-request
  ~/  %handle-http-request
  |=  [eyre-id=@ta =inbound-request:eyre]
  ^-  (quip card _state)
  ?>  authenticated.inbound-request
  =/  url  (parse-request-line:server url.request.inbound-request)
  ?>  ?=([%spider @t @t @t @t ~] site.url)
  =*  desk         i.t.site.url
  =*  input-mark   i.t.t.site.url
  =*  thread       i.t.t.t.site.url
  =*  output-mark  i.t.t.t.t.site.url
  =/  =tid         (new-thread-id thread)
  =.  serving.state  (~(put by serving.state) tid [`eyre-id output-mark desk])
  =/  tube  (convert-tube %json input-mark desk bowl)
  ?>  ?=(^ body.request.inbound-request)
  =/  body=json  (need (de-json:html q.u.body.request.inbound-request))
  =/  input=vase  (slop !>(~) (tube !>(body)))
  =/  boc  bec
  =/  =start-args:spider  [~ `tid boc(q desk, r da+now.bowl) thread input]
  (handle-start-thread start-args)

• Examine the “HTTP API” example.

EAuth

Eyre's EAuth system “a mechanism by which HTTP clients may authenticate themselves as a specific urbit on HTTP endpoints served by any other urbit.” In other words, you can provide a comet-like client to an arbitrary client.

• How robust to collision is random EAuth assignment? ($\frac{1}{2^{128}-2^{64}} \approx 3^{-39}$, or one in 100 undecillion)

• If you are interested in investigating EAuth in detail, see +$visitor, +$logbook, +$eauth-plea, +$eauth-boon as well as the source description at ~palfun-foslup, “mirage (eauth)” and the app ~paldev, %chat-stream.

SSL

If you are working locally, you typically just have HTTP set up instead of secure HTTPS. SSL is a transport layer protocol formerly used for client–server encrypted channels, but now HTTPS actually uses TLS.

The %acme agent configures a certificate if you have a domain set up to use with Urbit.

• Read the ++install arm in /app/acme.

Scry interface

Eyre exposes some information about bindings and connections, such as the sessions and cookies:

.^(authentication-state:eyre %e /=authentication-state=)

.^((list [binding:eyre duct action:eyre]) %e /=bindings=)

Vere I/O Driver: vere/io/http.c

The runtime counterpart to Eyre is vere/io/http.c, which is the HTTP server.

Vere's http.c uses libh2o as its HTTP server:

H2O is a new generation HTTP server that provides quicker response to users with less CPU utilization when compared to older generation of web servers. Designed from ground-up, the server takes full advantage of HTTP/2 features including prioritized content serving and server push

Take especial note of the following functions:

• u3_http_io_init() to start the HTTP server manager.

• _http_serv_listen_cb(), the callback for receiving a value

• _http_serv_accept()

• _http_seq_accept() to process a new HTTP request

• _http_hgen_send() to send an HTTP response

Iris

Iris is an HTTP client. It is not currently widely used since Urbit ships do not often serve as HTTP clients (rather as peers).

/sys/lull Definition

::  %iris http-client interface
::
++  iris  ^?
  |%
  ::  +gift: effects the client can emit
  ::
  +$  gift
    $%  [%request id=@ud request=request:http]  ::  outbound http-request
        [%cancel-request id=@ud]                ::  tell earth to cancel a 
                                                ::  previous %request
        [%http-response =client-response]       ::  response to the caller
    ==
  ::
  +$  task
    $~  [%vega ~]
    $%  $>(%born vane-task)  ::  system started up; reset open connections
        $>(%trim vane-task)  ::  trim state (in response to memory pressure)
        $>(%vega vane-task)  ::  report upgrade
        [%request =request:http =outbound-config]  ::  fetches a remote resource
        [%cancel-request ~]  ::  cancels a previous fetch
        [%receive id=@ud =http-event:http]  ::  receives http data from outside
    ==
  ::  +client-response: one or more client responses given to the caller
  ::
  +$  client-response
    $%  $:  %progress        ::  periodic update along %fetch source duct
            =response-header:http     ::  full transaction header
            bytes-read=@ud            ::  bytes fetched so far
            expected-size=(unit @ud)  ::  size if response had content-length
            incremental=(unit octs)   ::  data received since last update
        ==
        [%finished =response-header:http full-file=(unit mime-data)] ::  final
                                                                     ::  response
        [%cancel ~]          ::  canceled by runtime system
    ==
  ::  mime-data: externally received but unvalidated mimed data
  ::
  +$  mime-data  [type=@t data=octs]
  ::  +outbound-config: configuration for outbound http requests
  ::
  +$  outbound-config
    $:  redirects=_5         ::  number of times to follow 300 before error
        retries=_3           ::  number of retries before failing
    ==
  --

Structure

/sys/vane/iris is quite short and legible. Most of the vane is either tracking connection state as a client or sending updates for data transmission progress.

• Examine the “Iris Guide: Example” thread.

• Use this as a springboard for examining how the response header and body are constructed in Urbit.

Iris has relatively little information to expose at any given time, and has an extremely minimal scry interface:

• %i %x %whey, show memory usage.

Vere I/O Driver: vere/io/cttp.c

The runtime counterpart to Iris is vere/io/cttp.c, which is the HTTP client.

• Read u3_cttp_io_init(), which initializes the client manager state.

Like Eyre, Vere's Iris uses libh2o as its HTTP server/client library.

• Read _cttp_creq_on_body(), the callback upon receiving a response body

• _cttp_creq_respond()

• _cttp_http_client_receive()

Exercise

• Produce an app which allows a clearweb login. This can be done using EAuth but it would be interesting to implement standard username/password login as well.