Servers
One of Urbit's primary use cases is to act as a “personal server”. To examine this statement, we need to consider what a server does. Etymologically, a server serves a service. Generally speaking, it is the locus of a computation and coordination process. A server program is a system daemon—and since Gall agents are essentially daemons in many respects, Urbit's execution model fulfills this niche nicely.
Some servers are physical or logical devices which talk to other devices as clients. Internet webpage and application servers typically follow this model. Other servers are software processes that run on the same hardware or local network as the client process, e.g. mail servers, print servers, and file servers.
The two major operational models for servers are the request–response model and the publish–subscribe (pub-sub) model. The request–response pattern corresponds to pokes and gifts in Arvo terms, while the pub-sub pattern is supplied by subscriptions and updates.
- A client originates and submits requests, and receives responses.
- A server accepts requests and replies with responses.
A client and a server need to agree on a communications protocol. There are many of these, but the basis for the World Wide Web is the HyperText Transfer Protocol (HTTP).
Serving a Web Page
Two of the simplest actions one can take with a basic web server are to simply post a web page to any clients and to respond to interactions with that web page. Some interactions take place purely in the client session (form entry in the browser before submission), but then are propagated to the server.
Requests
HTTP requests are like Gall agent pokes: they are messages to trigger some action on the server. A method is specified (like GET, PUT, or POST) and the associated service-specific data follow.
It consists of a block of request headers, a block of general headers, and a block of representation headers. These may by followed by the body.
GETmeans a read-only request for information (like an Urbit scry but without the bound namespace).PUTrequests a state creation or update.POSTasks for the server to process data. (BothPUTandPOSTare analogous to Urbit pokes.)
Responses
A server response to a web page↗ looks like this:
The response code↗ is normatively 200 OK for a successful page access, but 404 Not Found and other errors and special messages also occur frequently. (It would be very interesting if Urbit would implement 402 Payment requested.)
The actual mechanics of communicating both kinds of communications are wrapped by /lib/server and Eyre. Generally speaking, an agent will receive HTTP requests in ++on-poke, and commonly includes a ++handle-http arm to deal with inbound-request:eyre values. /lib/server has request header parsers and response handlers which make it easy to respond appropriately (e.g. (send:server ~ [%login-redirect './apps/my-agent'])).
Eyre
Eyre is an HTTP server, which receives HTTP messages from Unix and produces HTTP messages in reply. Your agent can register endpoints which a browser or other tool can interact with. Eyre can be instrumented to work with threads and generators.
HTTP requests include a method tag. While other methods exist, we are primarily interested in POST, PUT, and GET requests. Since we don't want to deal with client-side code yet, we're going to use curl to send requests here.
POSTis only used with Eyre to obtain a cookie.
$ curl -i localhost:8080/~/login -X POST -d "password=lidlut-tabwed-pillex-ridrup"HTTP/1.1 204 okDate: Tue, 19 Jul 2022 16:28:05 GMTConnection: keep-aliveServer: urbit/vere-1.9set-cookie: urbauth-~nec=0v3.pis4a.sfdhv.f1p6i.lttba.gp93q; Path=/; Max-Age=604800
This cookie should be included in subsequent requests.
PUTrequests are used to send actions to Eyre: pokes, subscriptions, acks, unsubscribe requests, and channel deletions.GETrequests are used to connect to a channel and receive any pending events. (Remember how Urbit prefers a dataflow computing model?)
/sys/lull Definition
:: :::::::: ++eyre :: (1e) http-server:: ::::++ eyre ^?|%+$ gift$% $>(?(%boon %done) gift:ames) :: Ames responses[%set-config =http-config] :: configure external HTTP server[%sessions ses=(set @t)] :: valid auth cookies[%response =http-event:http] :: response to event from Earth[%bound accepted=? =binding] :: response to %connect or %serve[%grow =path] :: notification on cache entry change==::+$ task$~ [%vega ~]$% $>(%init vane-task) :: initialize ourself with an identity$>(%born vane-task) :: new unix process$>(%plea vane-task) :: network request$>(%trim vane-task) :: trim state (memory pressure)$>(%vega vane-task) :: report upgrade::[%live insecure=@ud secure=(unit @ud)] :: live HTTPS ports[%rule =http-rule] :: update HTTP configuration[%eauth-host host=(unit @t)] :: set base URL for eauth[%request secure=? =address =request:http] :: handle inbound[%request-local secure=? =address =request:http] :: handle backdoor[%cancel-request ~] :: cancel previous request[%connect =binding app=term] :: connects a binding to an app[%serve =binding =generator] :: connect binding to generator[%disconnect =binding] :: disconnect binding to generator::[%code-changed ~] :: web login code changed[%approve-origin =origin] :: accept CORS requests from origin[%reject-origin =origin] :: reject CORS requests from origin[%spew veb=@] :: set verbosity[%set-response url=@t entry=(unit cache-entry)] :: cache mapping==-- ::eyre
Eyre is responsible for a few subsystems that facilitate userspace applications (unlike, say, Behn or Ames, most of what Eyre does is to support userspace).
- Authentication
- Channels
- Threads and generators
- HTTP request handling
- Scry interface
Several of these are handled by the +$action dispatch system, invoked when a binding matches a known path.
A
+$bindingis a system unique mapping for a path to match. A+$bindingmust be system unique because we don't want two handlers for a path; what happens if there are two different actions for[~ /]?
:: +binding: A rule to match a path.::+$ binding$: site=(unit @t) :: site: the site to match (~ for your.urbit.org)path=(list @t) :: path: matches this prefix path==:: +action: the action to take when a binding matches an incoming request::+$ action$% [%gen =generator] :: dispatch to a generator[%app app=term] :: dispatch to an application[%authentication ~] :: internal authentication page[%eauth ~] :: cross-ship authentication handling[%logout ~] :: internal logout page[%channel ~] :: gall channel system[%scry ~] :: gall scry endpoint[%name ~] :: respond w/ @p requester is authenticated as[%host ~] :: respond w/ @p of the ship serving the response[%four-oh-four ~] :: respond with the default file not found page==
/sys/vane/eyre is relatively more straightforward than (say) Ames. There is only one interface (search for ~% %http-server).
Authentication
Client sessions typically require a login. (This is not true for materials served to the clearweb, e.g. via %blog.) A cookie is generated for each session in response to a login using +code.
:: +authentication-state: state used in the login system::+$ authentication-state$: sessions=(map @uv session) :: map of cookies to session informationvisitors=(map @uv visitor) :: visitors: in-progress incoming eauth flowsvisiting=(map ship logbook) :: visiting: outgoing eauth state per shipendpoint=[user=(unit @t) auth=(unit @t) =time] :: endpoint: hardcoded:: local eauth endpoint:: for %syn and %ack==
(Visits are part of the EAuth system, q.v.)
Authentication is handled by ++authentication.
- See
++authenticationin/sys/vane/eyre. Locate where the session cookie is created and logged.
Authentication is enforced by ++request-is-logged-in and ++request-is-authenticated.
- See
++request-is-logged-inin/sys/vane/eyre.
Channels
Channels are the main method where a webpage communicates with Gall apps. Subscriptions and pokes are issues with
PUTrequests on a path, whileGETrequests on that same path open a persistentEventSourcechannel. TheEventSourceAPI is a sequence number based API that browser provide which allow the server to push individual events to the browser over a connection held open. In case of reconnection, the browser will send a'Last-Event-Id'header to the server; the server then resends all events since then.
An EventSource interface is a way to track server-sent events for a client session. The JS on the browser/client-side receives text/event-stream formatted events. So a channel is a given connection to a browser including the EventSource connection.
:: channel: connection to the browser::+$ channel$: mode=?(%json %jam)=identitystate=(each timer duct)next-id=@ud :: next-id: next sequence number to uselast-ack=@da :: last-ack: time of last client ackevents=(qeu [id=@ud request-id=@ud =channel-event]) :: unacked eventsunacked=(map @ud @ud) :: unacked event counts by request-idsubscriptions=(map @ud [ship=@p app=term =path duc=duct]) :: gall subsheartbeat=(unit timer) :: sse heartbeat timer==:: +session: server side data about a session::+$ session$: =identity :: authentication level & id of this sessionexpiry-time=@da :: when this session expireschannels=(set @t) :: channels opened by this session==:: channel-state: state used in the channel system::+$ channel-state$: session=(map @t channel) :: mapping b/w an arbitrary key to a channelduct-to-key=(map duct @t) :: mapping from ducts to session key==:: channel-event: unacknowledged channel event, vaseless sign::+$ channel-event$% $>(%poke-ack sign:agent:gall)$>(%watch-ack sign:agent:gall)$>(%kick sign:agent:gall)[%fact =desk =mark =noun]==
Conventional channels communicate in JSON. Values passed into Urbit can be sent through a mark file to be transformed into a %noun or other type automatically. On the way out, a similar transformation can take the values back into MIME types.
- Trace
%pokeand%poke-jsonin/sys/vane/eyre. - Examine
/lib/schoonerand the/marfiles in the%yarddesk↗. How does it handle JSON transformations? What about binary types likeaudio/mpeg(MP3)?
Noun channels make it possible for external applications to speak Urbit nouns. This means that you can communicate with an Urbit ship in a way other than using a JSON payload. The content-type is marked as application/x-urb-jam. Nouns are ++jammed when sent into Eyre.
- Locate where
x-urb-jamis processed in/sys/vane/eyreand in/mar/noun.
Thanks to the mark system and ++find-channel-mode, it is straightforward on Urbit's side to implement noun channels. However, on the other side you need something that speaks nouns, such as noun.py.
HTTP request handling
All else set aside, the real purpose of Eyre is to act as the HTTP server for an Urbit ship. Eyre maintains a server configuration. There is an +$inbound-request type to receive an HTTP request, but the main HTTP types are actually in another arm, ++http.
A raw HTTP request handle happens like this:
- Eyre subscribes to an app at
/http-response/[eyre-id]. - Eyre pokes the app with
%handle-http-requestand the ID. - The app produces
%facts of?(%http-response-header %http-response-data %http-response-cancel).
:: +http-config: full http-server configuration::+$ http-config$: secure=(unit [key=wain cert=wain]) :: PEM-encoded RSA private key and:: cert or cert chainproxy=_| :: reverse TCP proxy HTTP(s)log=? :: keep HTTP(s) access logsredirect=? :: send 301 redirects to upgrade HTTP to HTTPS==:: +http-rule: update configuration::+$ http-rule$% [%cert cert=(unit [key=wain cert=wain])] :: set/clear cert and keypair[%turf action=?(%put %del) =turf] :: add/remove established dns binding==:: +address: client IP address::+$ address$% [%ipv4 @if][%ipv6 @is]:: [%ames @p]==:: +inbound-request: +http-request and metadata::+$ inbound-request$: authenticated=? :: has a valid session cookiesecure=? :: whether this request was encrypted (https)=address :: the source address of this request=request:http :: the http-request itself==
- Examine
++http. Find therequestandresponsehandlers. In particular, see+$simple-payload. - Examine
/lib/server, which contains wrapper arms for mere mortals.
In some ways, although this is the meat-and-potatoes of Eyre, it's all rather straightforward.
To actually get a value into userspace, Eyre sends the response to Gall:
++request-to-appto dispatch an%app+$actionto Gall.++deal-asto%passto Gall.
Threads and generators
We can treat a (local) ship as a “serverless” function call for a client.
A generator is a standalone computation based on arguments. Eyre supports generators explicitly:
+$ generator$: =desk :: desk on current ship that contains the generatorpath=(list @t) :: path on :desk to the generator's hoon fileargs=* :: arguments passed to the gate==
and runs them in ++request, branch %gen.
- How does the generator run? Note the
+$roofand the++mockcall.
Having bound a generator /gen/eyre-gen
|= [[now=@da eny=@uvJ bec=beak] ~ ~]|= [authenticated=? =request:http]^- simple-payload:http=/ msg=@t?~ body.request(scot %da now)(cat 3 (cat 3 (scot %da now) 10) q.u.body.request)=/ data=octs(as-octs:mimes:html msg)=/ =response-header:http[200 ['Content-Type' 'text/plain']~][response-header `data]
to an endpoint /mygen,
|pass [%e [%serve `/mygen %base /gen/eyre-gen/hoon ~]]
a client may initiate a generator call by posting a PUT request thus:
curl -i http://localhost:8080/mygen --data 'blah blah blah'
- Examine the “Eyre Guide: Generators” example.
A thread is a transient standalone computation similar in some regards to a generator. Spider provides thread support using an Eyre binding.
++ handle-http-request~/ %handle-http-request|= [eyre-id=@ta =inbound-request:eyre]^- (quip card _state)?> authenticated.inbound-request=/ url (parse-request-line:server url.request.inbound-request)?> ?=([%spider @t @t @t @t ~] site.url)=* desk i.t.site.url=* input-mark i.t.t.site.url=* thread i.t.t.t.site.url=* output-mark i.t.t.t.t.site.url=/ =tid (new-thread-id thread)=. serving.state (~(put by serving.state) tid [`eyre-id output-mark desk])=/ tube (convert-tube %json input-mark desk bowl)?> ?=(^ body.request.inbound-request)=/ body=json (need (de-json:html q.u.body.request.inbound-request))=/ input=vase (slop !>(~) (tube !>(body)))=/ boc bec=/ =start-args:spider [~ `tid boc(q desk, r da+now.bowl) thread input](handle-start-thread start-args)
- Examine the “HTTP API” example.
EAuth
Eyre's EAuth system “a mechanism by which HTTP clients may authenticate themselves as a specific urbit on HTTP endpoints served by any other urbit.” In other words, you can provide a comet-like client to an arbitrary client.
- How robust to collision is random EAuth assignment? ($\frac{1}{2{128}-2{64}} \approx 3^{-39}$, or one in 100 undecillion)
- If you are interested in investigating EAuth in detail, see
+$visitor,+$logbook,+$eauth-plea,+$eauth-boonas well as the source description at ~palfun-foslup, “mirage (eauth)”↗ and the app ~paldev, %chat-stream↗.
SSL
If you are working locally, you typically just have HTTP set up instead of secure HTTPS. SSL is a transport layer protocol formerly used for client–server encrypted channels, but now HTTPS actually uses TLS.
The %acme agent configures a certificate if you have a domain set up to use with Urbit.
- Read the
++installarm in/app/acme.
Scry interface
Eyre exposes some information about bindings and connections, such as the sessions and cookies:
.^(authentication-state:eyre %e /=authentication-state=).^((list [binding:eyre duct action:eyre]) %e /=bindings=)
Vere I/O Driver: vere/io/http.c
The runtime counterpart to Eyre is vere/io/http.c, which is the HTTP server.
Vere's http.c uses libh2o as its HTTP server:
H2O is a new generation HTTP server that provides quicker response to users with less CPU utilization when compared to older generation of web servers. Designed from ground-up, the server takes full advantage of HTTP/2 features including prioritized content serving and server push
Take especial note of the following functions:
u3_http_io_init()to start the HTTP server manager._http_serv_listen_cb(), the callback for receiving a value_http_serv_accept()_http_seq_accept()to process a new HTTP request_http_hgen_send()to send an HTTP response
Iris
Iris is an HTTP client. It is not currently widely used since Urbit ships do not often serve as HTTP clients (rather as peers).
/sys/lull Definition
:: %iris http-client interface::++ iris ^?|%:: +gift: effects the client can emit::+$ gift$% [%request id=@ud request=request:http] :: outbound http-request[%cancel-request id=@ud] :: tell earth to cancel a:: previous %request[%http-response =client-response] :: response to the caller==::+$ task$~ [%vega ~]$% $>(%born vane-task) :: system started up; reset open connections$>(%trim vane-task) :: trim state (in response to memory pressure)$>(%vega vane-task) :: report upgrade[%request =request:http =outbound-config] :: fetches a remote resource[%cancel-request ~] :: cancels a previous fetch[%receive id=@ud =http-event:http] :: receives http data from outside==:: +client-response: one or more client responses given to the caller::+$ client-response$% $: %progress :: periodic update along %fetch source duct=response-header:http :: full transaction headerbytes-read=@ud :: bytes fetched so farexpected-size=(unit @ud) :: size if response had content-lengthincremental=(unit octs) :: data received since last update==[%finished =response-header:http full-file=(unit mime-data)] :: final:: response[%cancel ~] :: canceled by runtime system==:: mime-data: externally received but unvalidated mimed data::+$ mime-data [type=@t data=octs]:: +outbound-config: configuration for outbound http requests::+$ outbound-config$: redirects=_5 :: number of times to follow 300 before errorretries=_3 :: number of retries before failing==--
Structure
/sys/vane/iris is quite short and legible. Most of the vane is either tracking connection state as a client or sending updates for data transmission progress.
- Examine the “Iris Guide: Example” thread.
- Use this as a springboard for examining how the response header and body are constructed in Urbit.
Iris has relatively little information to expose at any given time, and has an extremely minimal scry interface:
%i %x %whey, show memory usage.
Vere I/O Driver: vere/io/cttp.c
The runtime counterpart to Iris is vere/io/cttp.c, which is the HTTP client.
- Read
u3_cttp_io_init(), which initializes the client manager state.
Like Eyre, Vere's Iris uses libh2o as its HTTP server/client library.
- Read
_cttp_creq_on_body(), the callback upon receiving a response body _cttp_creq_respond()_cttp_http_client_receive()
Exercise
- Produce an app which allows a clearweb login. This can be done using EAuth but it would be interesting to implement standard username/password login as well.