<article>
<h2 id="overview">Overview</h2>
<p>In the <a
href="https://hackerpublicradio.org/eps/hpr4104/index.html">last
episode</a> we looked at how JSON data is structured and saw how
<code>jq</code> could be used to format and print this type of data.</p>
<p>In this episode we'll visit a few of the options to the
<code>jq</code> command and then start on the filters written in the
<code>jq</code> language.</p>
<h2 id="options-used-by-jq">Options used by <code>jq</code></h2>
<p>In general the <code>jq</code> command is invoked thus:</p>
<pre><code>jq [options...] filter [files...]</code></pre>
<p>It can be given data in files or sent to it via the STDIN (standard
in) channel. We saw data being sent this way in the last episode, having
been downloaded by <code>curl</code>.</p>
<p>There are many options to the command, and these are listed in the
manual page and in the <a
href="https://jqlang.github.io/jq/manual/">online manual</a>. We will
look at a few of them here:</p>
<dl>
<dt><code>--help</code> <em>or</em> <code>-h</code></dt>
<dd>
<p>Output the <code>jq</code> help and exit with zero.</p>
</dd>
<dt><code>-f filename</code> <em>or</em>
<code>--from-file filename</code></dt>
<dd>
<p>Read filter from the file rather than from a command line, like awk´s
-f option. You can also use ´#´ to make comments in the file.</p>
</dd>
<dt><code>--compact-output</code> <em>or</em> <code>-c</code></dt>
<dd>
<p>By default, <code>jq</code> pretty-prints JSON output. Using this
option will result in more compact output by instead putting each JSON
object on a single line.</p>
</dd>
<dt><code>--color-output</code> <em>or</em> <code>-C</code> <em>and</em>
<code>--monochrome-output</code> <em>or</em> <code>-M</code></dt>
<dd>
<p>By default, <code>jq</code> outputs colored JSON if writing to a
terminal. You can force it to produce color even if writing to a pipe or
a file using <code>-C</code>, and disable color with
<code>-M</code>.</p>
</dd>
<dt><code>--tab</code></dt>
<dd>
<p>Use a tab for each indentation level instead of two spaces.</p>
</dd>
<dt><code>--indent n</code></dt>
<dd>
<p>Use the given number of spaces (no more than 7) for indentation.</p>
</dd>
</dl>
<h3 id="notes">Notes</h3>
<ul>
<li><p>The <code>-C</code> option is useful when printing output to the
<code>less</code> command with the colours that <code>jq</code> normally
generates. Use this:</p>
<pre><code>jq -C &#39;.&#39; file.json | less -R</code></pre>
<p>The <code>-R</code> option to less allows colour escape sequences to
pass through.</p></li>
<li><p>Do not do what I did recently. Accidentally leaving the
<code>-C</code> option on the command caused <code>formatted.json</code>
to contain all the escape codes used to colour the output:</p>
<pre><code>$ jq -C &#39;.&#39; file.json &gt; formatted.json</code></pre>
<p>This is why <code>jq</code> normally only generates coloured output
when writing to the terminal.</p></li>
</ul>
<h2 id="filters-in-jq">Filters in <code>jq</code></h2>
<p>As we saw in the last episode JSON can contain arrays and objects.
Arrays are enclosed in square brackets and their elements can be any of
the data types we saw last time. So, arrays of arrays, arrays of
objects, and arrays of both of these are all possible.</p>
<p>Objects contain collections of keyed items where the keys are strings
of various types and the values they are associated with can be any of
the data types.</p>
<h3 id="json-examples">JSON Examples</h3>
<p>Simple arrays:</p>
<pre><code>[1,2,3]
[1,2,3,[4,5,6]]
[&quot;Hacker&quot;,&quot;Public&quot;,&quot;Radio&quot;]
[&quot;Sunday&quot;,&quot;Monday&quot;,&quot;Tuesday&quot;,&quot;Wednesday&quot;,&quot;Thursday&quot;,&quot;Friday&quot;,&quot;Saturday&quot;]</code></pre>
<p>Simple object:</p>
<pre><code>{ &quot;name&quot;: &quot;Hacker Public Radio&quot;, &quot;type&quot;: &quot;podcast&quot;}</code></pre>
<p>This more complex object was generated by the <a
href="https://randomuser.me/documentation">Random User Generator
API</a>. It is a subset of what can be obtained from this site.</p>
<pre><code>{
  &quot;gender&quot;: &quot;female&quot;,
  &quot;name&quot;: {
    &quot;title&quot;: &quot;Mrs&quot;,
    &quot;first&quot;: &quot;Jenny&quot;,
    &quot;last&quot;: &quot;Silva&quot;
  },
  &quot;dob&quot;: {
    &quot;date&quot;: &quot;1950-01-03T21:38:19.583Z&quot;,
    &quot;age&quot;: 74
  },
  &quot;nat&quot;: &quot;GB&quot;
}</code></pre>
<p>This one comes from the file <code>countries.json</code> from the <a
href="https://github.com/mledoze/countries">Github project
<code>mledoze/countries</code></a>. It is a subset of the entry for
<em>Mexico</em>.</p>
<pre><code>{
  &quot;name&quot;: {
    &quot;common&quot;: &quot;Mexico&quot;,
    &quot;official&quot;: &quot;United Mexican States&quot;,
    &quot;native&quot;: {
      &quot;spa&quot;: {
        &quot;official&quot;: &quot;Estados Unidos Mexicanos&quot;,
        &quot;common&quot;: &quot;México&quot;
      }
    }
  },
  &quot;capital&quot;: [
    &quot;Mexico City&quot;
  ],
  &quot;borders&quot;: [
    &quot;BLZ&quot;,
    &quot;GTM&quot;,
    &quot;USA&quot;
  ]
}</code></pre>
<h3 id="identity-filter">Identity filter</h3>
<p>This is the simplest filter which we already encountered in episode
1: <code>'.'</code>. It takes its input and produces the same value as
output. Since the default action is to pretty-print the output it
formats the data:</p>
<pre><code>$ echo &#39;[&quot;Hacker&quot;,&quot;Public&quot;,&quot;Radio&quot;]&#39; | jq .
[
  &quot;Hacker&quot;,
  &quot;Public&quot;,
  &quot;Radio&quot;
]</code></pre>
<p>Notice that the filter is not enclosed in quotes in this example.
This is usually fine for the simplest filters which don't contain any
characters which are of significance to the shell. It's probably a good
idea to always use (single) quotes however.</p>
<p>There may be considerations regarding how <code>jq</code> handles
numbers. Consult the <a
href="https://jqlang.github.io/jq/manual/"><code>jq</code>
documentation</a> for details.</p>
<h3 id="object-identifier-index-filter">Object Identifier-Index
filter</h3>
<p>This form of filter refers to object keys. A particular key is
usually referenced with a full-stop followed by the name of the key.</p>
<p>In the HPR statistics data there is a top-level key "hosts" which
refers to the number of currently registered hosts. This can be obtained
thus (assuming the JSON is in the file <code>stats.json</code>):</p>
<pre><code>$ jq &#39;.hosts&#39; stats.json
357</code></pre>
<p>The statistics file contains a key <code>'stats_generated'</code>
which marks a Unix time value (seconds since the Unix Epoch 1970-01-01).
This can be decoded on the command line like this:</p>
<pre><code>$ date -d &quot;@$(jq &#39;.stats_generated&#39; stats.json)&quot; +&#39;%F %T&#39;
2024-04-18 15:30:07</code></pre>
<p>Here the <code>'-d'</code> option to <code>date</code> provides the
date to print, and if it begins with a <code>'@'</code> character it's
interpreted as seconds since the Epoch. Note that the result is in my
local time zone which is currently UTC + 0100 (aka BST).</p>
<p>Using object keys in this way only works if the keys contain only
ASCII characters and underscores and don't start with a digit. To use
other characters it's necessary to enclose the key in double quotes or
square brackets and double quotes. So, assuming the key we used earlier
had been altered to <code>'stats-generated'</code> we could use either
of these expressions:</p>
<pre><code>.&quot;stats-generated&quot;
.[&quot;stats-generated&quot;]</code></pre>
<p>Of course, the <code>.[&lt;string&gt;]</code> form is valid in all
contexts. Here <code>&lt;string&gt;</code> represents a JSON string in
double quotes. The <code>jq</code> documentation refers to this as an
<em>Object Index</em>.</p>
<p>What if you want the <code>next_free</code> value discussed in the
last episode (number of shows until the next free slot)? Just typing the
following will not work:</p>
<pre><code>$ jq &#39;.next_free&#39; stats.json
null</code></pre>
<p>This is showing that there is no key <code>next_free</code> at the
top level of the object, the key we want is in the object with the key
<code>slot</code>.</p>
<p>If you request the <code>slot</code> key this will happen:</p>
<pre><code>$ jq &#39;.slot&#39; stats.json
{
  &quot;next_free&quot;: 8,
  &quot;no_media&quot;: 0
}</code></pre>
<p>Here an object has been returned, but we actually want the value
within it, as we know.</p>
<p>This is where we can <em>chain</em> filters like this:</p>
<pre><code>$ jq &#39;.slot | .next_free&#39; stats.json
8</code></pre>
<p>The <em>pipe</em> symbol causes the result of the first filter to be
passed to the second filter. Note that the pipe here is not the same as
the Unix pipe, although it looks the same</p>
<p>There is a shorthand way of doing this "chaining":</p>
<pre><code>$ jq &#39;.slot.next_free&#39; stats.json
8</code></pre>
<p>This is a bit like a file system path, and makes the extraction of
desired data easier to visualise and therefore quite straightforward, I
think.</p>
<h3 id="array-index-filter">Array index filter</h3>
<p>We have seen the object index filter <code>.[&lt;string&gt;]</code>
where <code>&lt;string&gt;</code> represents a key in the object we are
working with.</p>
<p>It makes sense for array indexing to be
<code>.[&lt;number&gt;]</code> where <code>&lt;number&gt;</code>
represents an integer starting at zero, or a negative integer. The
meaning of the negative number is to count backwards from the last
element of the array (which is <code>-1</code>).</p>
<p>So, some examples might be:</p>
<pre><code>$ echo &#39;[&quot;Sunday&quot;,&quot;Monday&quot;,&quot;Tuesday&quot;,&quot;Wednesday&quot;,&quot;Thursday&quot;,&quot;Friday&quot;,&quot;Saturday&quot;]&#39; | jq &#39;.[1]&#39;
&quot;Monday&quot;

$ echo &#39;[&quot;Sun&quot;,&quot;Mon&quot;,&quot;Tue&quot;,&quot;Wed&quot;,&quot;Thu&quot;,&quot;Fri&quot;,&quot;Sat&quot;]&#39; | jq &#39;.[-1]&#39;
&quot;Sat&quot;

$ echo &#39;[1, 2, 3, [4, 5, 6]]&#39; | jq &#39;.[-1]&#39;
[
  4,
  5,
  6
]</code></pre>
<p>We will look at more of the basic filters in the next episode.</p>
<h2 id="links">Links</h2>
<ul>
<li><code>jq</code>:
<ul>
<li><a href="https://jqlang.github.io/jq/">GitHub page</a></li>
<li><a href="https://jqlang.github.io/jq/download/">Downloading
<code>jq</code></a></li>
<li><a href="https://jqlang.github.io/jq/manual/">The <code>jq</code>
manual</a></li>
<li><a
href="https://en.wikipedia.org/wiki/Jq_(programming_language)">Wikipedia
page about the jq programming language</a></li>
</ul></li>
</ul>
<ul>
<li>Test data sources:
<ul>
<li><a href="https://randomuser.me/documentation">Random User Generator
API</a></li>
<li><a href="https://github.com/mledoze/countries">Github project
<code>mledoze/countries</code></a></li>
</ul></li>
</ul>
</article>
