3. JSON
Data sent between our agent and our front-end will all be encoded as JSON. In this section, we'll briefly look at how JSON works in Urbit, and write a library to convert our agent's structures to and from JSON for our front-end.
JSON data comes into Eyre as a string, and Eyre parses it with the +de:json:html function in zuse.hoon. The hoon type it's parsed to is $json, which is defined as:
+$ json :: normal json value
$@ ~ :: null
$% [%a p=(list json)] :: array
[%b p=?] :: boolean
[%o p=(map @t json)] :: object
[%n p=@ta] :: number
[%s p=@t] :: string
== ::Once Eyre has converted the raw JSON string to a $json structure, it will be converted to the mark the web client specified and then delivered to the target agent (unless the mark specified is already %json, in which case it will be delivered directly). Outbound facts will go through the same process in reverse - converted from the agent's native mark to $json, then encoded in a string by Eyre using +en:json:html and delivered to the web client. The basic flow for both inbound messages (pokes) and outbound messages (facts and scry results) looks like this:
The mark conversion will be done by the corresponding mark file in /mar on the agent's desk. In our case it would be /mar/journal/action.hoon and /mar/journal/update.hoon in the %journal desk for our %journal-action and %journal-update marks, which are for the $action and $update structures we defined previously.
Mark conversion functions can be included directly in the mark file, or they can be written in a separate library, then imported and called by the mark file. We will do the latter in this case, so before we create the mark files themselves, we'll write a library called /lib/journal.hoon with the conversion functions.
$json utilities
$json utilitieszuse.hoon contains three main cores for converting to and from $json:
+enjs:format- Functions to help encode data structures as$json.+dejs:format- Functions to decode$jsonto other data structures.+dejs-soft:format- Mostly the same as+dejs:formatexcept the functions produce units which are null if decoding fails, rather than just crashing.
+enjs:format
+enjs:formatThis contains ten functions for encoding $json. Most of them are for specific hoon data types, such as +tape:enjs:format, +ship:enjs:format, +path:enjs:format, etc. We'll just have a look at the two most general and useful ones: +frond:enjs:format and +pairs:enjs:format.
+frond
+frondThis function is for forming a JSON object from a single key-value pair. For example:
> (frond:enjs:format 'foo' s+'bar')
[%o p={[p='foo' q=[%s p='bar']]}]When stringified by Eyre, this will look like:
{ "foo": "bar" }+pairs
+pairsThis is similar to +frond and also forms a JSON object, but it takes multiple key-value pairs rather than just one:
> (pairs:enjs:format ~[['foo' n+~.123] ['bar' s+'abc'] ['baz' b+&]])
[%o p={[p='bar' q=[%s p='abc']] [p='baz' q=[%b p=%.y]] [p='foo' q=[%n p=~.123]]}]When stringified by Eyre, this will look like:
{
"foo": 123,
"baz": true,
"bar": "abc"
}Notice that we used a knot for the value of foo (n+~.123). Numbers in JSON can be signed or unsigned and integers or floating point values. The $json structure uses a knot so that you can decide whether a particular number should be treated as @ud, @sd, @rs, etc.
+dejs:format
+dejs:formatThis core contains many functions for decoding $json. We'll touch on some useful families of +dejs functions in brief, but because there's so many, in practice you'll need to look through the +dejs reference to find the correct functions for your use case.
Number functions
+ne- decode a number to a@rd.+ni- decode a number to a@ud.+no- decode a number to a@ta.+nu- decode a hexadecimal string to a@ux.
For example:
> (ni:dejs:format n+'123')
123String functions
+sa- decode a string to a$tape.+sd- decode a string containing a@daaura date value to a@da.+se- decode a string containing the specified aura to that aura.+so- decode a string to a@t.+su- decode a string by parsing it with the given parsing rule.
Array functions
+ar, +as, and +at decode a $json array to a +list, +set, and n-tuple respectively. These gates take other +dejs functions as an argument, producing a new gate that will then take the $json array. For example:
> ((ar so):dejs:format a+[s+'foo' s+'bar' s+'baz' ~])
<|foo bar baz|>Notice that +so is given as the argument to +ar. +so is a +dejs function that decodes a $json string to a $cord. The gate resulting from (ar so) is then called with a $json array as its argument, and its product is a (list @t) of the elements of the array.
Many +dejs functions take other +dejs functions as their arguments. A complex nested $json decoding function can be built up in this manner.
Object functions
+of- decode an object containing a single key-value pair to a head-tagged cell.+ot- decode an object to a n-tuple.+ou- decode an object to an n-tuple, replacing optional missing values with a given value.+oj- decode an object of arrays to a+jug.+om- decode an object to a+map.+op- decode an object to a+map, and also parse the object keys with a parsing rule.
For example:
> =js %- need %- de:json:html
'''
{
"foo": "hello",
"baz": true,
"bar": 123
}
'''
> %- (ot ~[foo+so bar+ni]):dejs:format js
['hello' 123]Our types as JSON
We need to decide how our $action and $update types will be represented as JSON in order to write our conversion functions. There are many ways to do this, but in this case we'll do it as follows:
Actions
{"add":{"id":1648366311070,"txt":"some text"}}
[%add id=1.648.366.034.844 txt='some text']
{"edit":{"id":1648366311070,"txt":"some text"}}
[%edit id=1.648.366.034.844 txt='some text']
{"del":{"id":1648366311070}}
[%del id=1.648.366.034.844]
Updates
[1.648.366.492.459 %add id=1.648.366.034.844 txt='some text']
{time:1648366481425,"add":{"id":1648366311070,"txt":"some text"}}
[1.648.366.492.459 %edit id=1.648.366.034.844 txt='some text']
{time:1648366481425,"edit":{"id":1648366311070,"txt":"some text"}}
[1.648.366.492.459 %del id=1.648.366.034.844]
{time:1648366481425,"del":{"id":1648366311070}}
[1.648.366.492.459 %jrnl ~[[id=1.648.366.034.844 txt='some text'] ...]
{time:1648366481425,"entries":[{"id":1648366311070,"txt":"some text"},...]}
[1.648.366.492.459 %logs ~[[1.648.366.492.459 %add id=1.648.366.034.844 txt='some text'] ...]
{time:1648366481425,"logs":[{time:1648366481425,"add":{id":1648366311070,"txt":"some text"}},...]}
Now let's write our library of encoding/decoding functions.
/lib/journal.hoon
/lib/journal.hoon/- *journal
|%First, we'll import the /sur/journal.hoon structures we previously created. Next, we'll create two arms in our core, +dejs-action and +enjs-update, to handle incoming poke $actions and outgoing facts or scry result $updates.
$json to $action
$json to $action++ dejs-action
=, dejs:format
|= jon=json
^- action
%. jon
%- of
:~ [%add (ot ~[id+ni txt+so])]
[%edit (ot ~[id+ni txt+so])]
[%del (ot ~[id+ni])]
==The first thing we do is use the =, rune to expose the +dejs:format namespace. This allows us to reference "ot", "ni", etc. rather than having to write "ot:dejs:format" every time. Note that you should be careful using =, generally as the exposed wings can shadow previous wings if they have the same name.
We then create a gate that takes $json and returns a $action structure. Since we'll only take one action at a time, we can use the +of function, which takes a single key-value pair. +of takes a list of all possible $json objects it will receive, tagged by key.
For each key, we specify a function to handle its value. Ours will be objects, so we use +ot and specify the pairs of the key and +dejs function to decode it. We then cast the output to our $action structure.
You'll notice the nesting of these +dejs functions approximately reflects the nested structure of the $json it's decoding.
$update to $json
$update to $json++ enjs-update
=, enjs:format
|= upd=update
^- json
|^
?+ -.q.upd (logged upd)
%jrnl
%- pairs
:~ ['time' (numb p.upd)]
['entries' a+(turn list.q.upd entry)]
==
::
%logs
%- pairs
:~ ['time' (numb p.upd)]
['logs' a+(turn list.q.upd logged)]
==
==
++ entry
|= ent=^entry
^- json
%- pairs
:~ ['id' (numb id.ent)]
['txt' s+txt.ent]
==
++ logged
|= lgd=^logged
^- json
?- -.q.lgd
%add
%- pairs
:~ ['time' (numb p.lgd)]
:- 'add'
%- pairs
:~ ['id' (numb id.q.lgd)]
['txt' s+txt.q.lgd]
== ==
%edit
%- pairs
:~ ['time' (numb p.lgd)]
:- 'edit'
%- pairs
:~ ['id' (numb id.q.lgd)]
['txt' s+txt.q.lgd]
== ==
%del
%- pairs
:~ ['time' (numb p.lgd)]
:- 'del'
(frond 'id' (numb id.q.lgd))
==
==
--
--Our $update encoding function's a little more complex than our $action decoding function, since our $update structure is more complex.
Like the previous one, we use =, to expose the namespace of +enjs:format.
Our gate takes an $update and returns a $json structure. We use |^ so we can separate out the encoding functions for individual entries (+entry) and individual logged actions (+logged).
We first test the head of the $update, and if it's %jrnl (a list of entries), we +turn over the entries and call +entry to encode each one. If it's %logs, we do the same, but call +logged for each item in the list. Otherwise, if it's just a single update, we encode it with +logged.
We primarily use +pairs to form the object, though sometimes +frond if it only contains a single key-value pair. We also use +numb to encode numerical values.
You'll notice more of our encoding function is done manually than our previous decoding function. For example, we form arrays by tagging an ordinary +list with %a, and strings by tagging an ordinary $cord with %s. This is typical when you write $json encoding functions, and is the reason there are far fewer +enjs functions than +dejs functions.
Resources
The JSON Guide - The stand-alone JSON guide covers JSON encoding/decoding in great detail.
The Zuse reference - The
/sys/zuse.hoonreference documents all JSON-related functions in detail.+enjs:formatreference - This section of thezuse.hoondocumentation covers all JSON encoding functions.+dejs:formatreference - This section of thezuse.hoondocumentation covers all JSON decoding functions.Eyre overview - This section of the Eyre vane documentation goes over the basic features of the Eyre vane.
Last updated