cat9

Cat9
Login

Cat9

What is it?

Cat9 is a user shell script for LASH - a command-line shell that discriminates against terminal emulators, written in Lua. You probably have not heard of LASH before. If you really must know, check the Backstory section below.

LASH just provides some basic shared infrastructure and a recovery shell. It then runs a user provided script that actually provides most of the rules for how the command line is supposed to look and behave.

That brings us back to Cat9, which is my such script. You can use it as is or remix it into something different that fits you - see HACKING.md for more tips.

What can it do?

One of the bigger conveniences, on top of being quite snappy, is to run and cleanly separate multiple concurrent jobs asynchronously, with the results from 'out' and 'err' being kept until you decide to reuse or forget it. At the same time, traditionally noisy tasks like changing directories are blocked from polluting your view with irrelevant information.

https://user-images.githubusercontent.com/5888792/161494772-2abccac4-bb92-4a12-9a69-987e66201719.mp4

This allows for neat visuals like changing layouts, reordering presentation, folding and unfolding. It also allows for reusing results of a previous job without thinking much about it - caching is the default and you don't need to redirect results to files just for reuse or re-execute pipelines when all you wanted was different processing of its old outputs.

It is also designed with the intention of being able to frontend- legacy cli tools with little effort - the set of builtins that the shell provides can be contextually swapped for something specific to working with some domain or tool specific context. In this way, famously unfriendly tools can be worked around and integrated into your workflow as seemless as possible.

It cooperates with your desktop window manager (should you have one), letting it take care of splitting out things into new windows or even embedding media or graphical application output into clipped regions of its own window.

https://user-images.githubusercontent.com/5888792/161494402-9e5636e3-dd78-4fcf-bff0-fe5c3dd0a369.mp4

Installation

Building and setting this up is currently not for the faint of heart. It might look simple at first glance, but going against the grain of decades of accumulated legacy comes with some friction.

For starters you want an Arcan build that comes straight from the source. The things needed here are new, unlikely covered by any release, and is actively worked on. Arcan is unpleasant to build from source and, if you want it to replace your entire display server, also a pain to setup. Twice the fun.

For our ends here, it works just fine as a window that looks strangely much like a terminal emulator would look, but its innards are entirely different.

If you managed to build Arcan to your liking, you then need to start Arcan with a suitable window manager.

There are several to chose from, notable ones being:

Then there is the much more humble 'console' that mimics the BSD/Linux console with just fullscreen and invisible tabs. Since it comes included with Arcan itself, we will go for that one. The way to actually start lash from within these vary, for console it is easy:

arcan console lash

This should convince Arcan to setup a simple fullscreen graphical shell that then runs the textual command-line shell in lash. Alas the shell will be kindof useless. This is where Cat9 comes in.

Underneath the surface it actually runs:

ARCAN_ARG=cli=lua afsrv_terminal

copy or link cat9.lua to $HOME/.arcan/lash/default.lua or cat9.lua (make the directory should it not exist) as well as the cat9 subdirectory so that there is a $HOME/.arcan/lash/cat9.

Similarly, in Durden it would be global/open/terminal=cli=lua and for safespaces, tack on cli=lua to the terminal spawn line, e.g. layers/current/terminal=cli=lua. In recent Durden versions this has a shortcut as global/open/lash, and is also bound to m1+m2+enter.

Next time you start the arcan console like above, if you picked the default.lua route it will start immediately - otherwise you have to manually tell lash to run the cat9 rulset with the shell command:

shell cat9

This scans LASH_BASE, HOME/.arcan/lash and XDG_CONFIG_HOME/arcan/lash for a matching cat9.lua and switches over to that. It is also possible to set LASH_SHELL=cat9 and cat9.lua will be tried immediately instead of default.lua

This extra set of steps is to allow multiple shells to coexist, so that there is a premade path for other rulesets to join the scene with less of a disadvantage.

Use

By default, commands will get tracked as a 'job'. These get numeric identifiers and are referenced by a pound sign:

find /tmp
repeat #0 flush

These pounds can also use relative addresses, #-2 would point to the second latest job to be created. There are also special jobs, like #csel pointing to the currently cursor-selected job, and #last pointing to the latest created job.

Most builtin commands use job references in one way or another. The context of these jobs, e.g. environment variables and path is tracked separately. By starting a command with a job reference, the current context is temporarily set to that of a previous job.

cd /usr/share
find . -> becomes job #0
cd /tmp
#0 pwd -> /usr/share

Most strings entered will be executed as non-tty jobs. To run something specifically as a tty (ncurses or other 'terminal-tui' like application), mark it with a ! to spawn a new window bound to a terminal emulator.

!vim

Window creation and process environment can be controlled (e.g. vertical split (v) or swallow (s)):

v!vim

To forego any parsing or internal pipelineing and run the entire line verbatim (forwarded to sh -c) use !!:

!!find /usr |grep share

This is also useful when you need to do shell expansions:

!!rm -rf *

Certain clients really want a pseudoterminal or they refuse to do anything, for those clients, start with p! like so:

p!vim

These will default switch to 'view wrap vt100' that has a rudimentary terminal emulator state machine that need some more work (see base/vt100*).

Data can be sliced out of a job into the current command-line with ctrl+space:

rm #0(1,3)

and pressing ctrl+space would copy lines 1 and 3 and expand them into the current command line. This argument format is also supported by some builtins, typing:

copy #0(1-5)

and pressing enter would copy the lines 1 to 5 into a new job.

Customisation

The config/config.lua file can be edited to change presentation, layout and similar options, including the formats for prompts and titlebars. Most of these options can also be reached at runtime via the 'config' builtin, further below.

Keybindings and mouse button presses act just like lines typed on the prompt, but are defined in the config/bindings.lua file. Keybindings are activated on ctrl+[key]. The following:

bnd[tui.keys.A] = "forget #-1"

would create a binding that whenever ctrl-A is pressed, the latest job created would be removed.

Mouse buttons are similar, but the 'key' is slightly more complex as they are split based on where you are clicking (titlebar, data body, ...). The currently most complex binding looks something like this:

bnd.m2_data_col_click = "view #csel select $=crow"

Would apply the builtin view command on the mouse-selected job at the current row offset if the second mouse button was clicked on the first column (line-number).

It is also possible to work with 'chorded' input bindings where one binding sets a 'consume on use' input, e.g. Ctrl-w to Ctrl-a. See bindings.lua for examples on how that is configured.

Builtins

There are a number of 'builtin' commands. This is traditionally commands implemented by the shell itself rather than outsourcing it to some external program. Most shells has this as a global namespace, but in cat9 it is hierarchical. There is a small default set, and then a number of interchangeable ones.

These are defined by a basename, that is reflected in how files are organised in the project:

cat9/default.lua
cat9/default/open.lua
	cat9/config/default.lua
...

The reason for this structure is to allow lash to be reused for building CLI frontends to other tools, maintain a unified look and feel and swap between them quickly at will.

You can also modify/extend this to mimic the behaviour of other common shells.

These also include a set of views. A view is a script that defines how to present the data within a job, and controls things like colour and formatting, optional elements like line numbers as well as wrapping behaviour.

These have a separate config store that follows the pattern:

cat9/config/(basedir).lua

Two sets of builtins are of additional importance, 'default' that contains cat9 related commands and 'system' which contains many of the normal commands one would expect from a traditional shell such as 'cd'.

Many of these builtins have a toggleable help descriptor for their commands and subcommands. This can be toggled by pressing F1 at any time during the readline prompt.

Builtins:Default

The commands included in the default set are: builtin, input, view, open, config, forget, repeat, trigger

Builtin

The command for switching between sets of builtins is also a builtin, albeit a special one that is always present. It takes the arguments:

builtin [setname=system] [nodef]

This will cause a hot reload of the set of builtins, including the default unless nodef is provided as the second argument.

Input

The input command is for controlling how parts of the UI responds to mouse or keyboard inputs. By default the keyboard is grabbed by the command-line. This grab can be toggled with CTRL+ESCAPE.

input #jobid

This will only trigger for jobs that have a working input sink, and retain current focus if it does not.

Config

config key value

The config options changes the runtime shell behavior configuration. It is populated by the keys and values in config/config.lua.

Certain targets also allow properties to set, e.g. persistence or an alias:

config #id alias myname
config #myname persist auto

It can be used to hot-reload settings and the code for default builtins:

config =reload

It can be used to define aliases:

config myalias "my longer command"

When ctrl+space is used with readline, the last word will be swapped for the alias. This will not have a commit action, so the expansion is visible to avoid unpleasant surprises.

Open

open file or #job [hex] [new | vnew | tab | swallow]

This tries to open the contents of [file] through a designated handler. For the job mode specifically, it either switches the window to a text or hex buffer. It is also possible to pop it out as a new window or tab.

Each

each (sequential OR merge) #job !! cmd $arg $dir

This will slice data from #job and for each line in the output, parse the string after !! substituting arg for the line and and dir for the directory of #job, or if the line contains a path (/to/somewhere) the /to part. The later form is useful when combined with stash. By default each runs all commands in parallel.

Contain

contain new
contain #job add #job1 #job2 ..
contain #job show n1 n2 ..
             show append n1 n2 ..
contain #job release n1 n2 ..
contain #job forget n1 n2 ..
contain #job capture [on | off | toggle]

This creates a container job which absorbs other visible ones into itself. It is useful both for reducing visible clutter and for treating a collection of data sources as a single discrete one.

You spawn a new container job via 'contain new' and either manually add the jobs that should be treated this way into itself through add #id #job, or by enabling job capture via 'contain #id capture on'.

It is not possible to nest containers, and some builtins that spawn jobs will refuse to be captured in this way. Jobs that are bound to a different window will also refuse capture.

Forget

forget #job1 #job2
forget #job1  .. #job3
forget all-bad
forget all-passive
forget all-hard

This will remove the contents and tracking of previous jobs, either by specifying one or several distinct jobs, or a range. If any marked job is still active and tied to a running process, that process will be killed. There are also special presets: all-bad, all-hard and all-passive; all-bad forgets the completed jobs with !0 return status code; all-hard removes all jobs, passive and running; all-passive removes all jobs that have completed running.

Repeat

repeat #job [flush | edit]

This will re-execute a previously job that has completed. If the flush argument is specified, any collected data from its previous run will be discarded. If the edit argument is specified, the command-line which spawned the job will be copied into the command line - and if executed, will append or flush into the existing job identifier.

View

view #job stream [opt1] [opt2] .. [optn]

Changes job presentation based on the provided set of options. The possible options are:

There are also a number of dynamic views that can apply higher level transforms on the contents to change how it presents. One such view is 'wrap':

view #job wrap [vt100] [max-col]

This implements word wrap, optionally filtered through a terminal state machine (vt100) and with a custom column cap.

Copy

copy src [opts] [dst]

The copy command is used to copy data in and out of Cat9 itself, such as taking the output of a previous job and storing it in a file, into another running job or into a new job (if dst is not provided).

By default, src and dst are treated as just names with normal semantics for absolute or relative paths. Depending on a prefix used, the role can change:

Using # will treat it as a job reference.

Using pick: will treat it as a request to an outer graphical shell (i.e. your wm) to provide a file, optionally with an extension hint:

copy pick:iso test.iso

The optional source arguments can be used to slice out subranges, e.g.

copy #0 (1-10,20) dst

Which would copy lines 1 to 10 and line 20 of the current view buffer into the destination.

Copy destinations do not have to be files, they can also be other interactive jobs, or special ones like clipboard: that would forward to the outer WM clipboard.

Trigger

trigger #job condition [delay n] action

It is possible to attach several event triggers to a job that has an external action attached. The command for that is 'trigger'. The condition can be either 'ok' or 'fail', with an optional delay in seconds. The action is any regular command-line string (remember to encapsulate with "").

To remove previously set triggers, use 'flush'.

A common case for trigger is to repeat a job that finished:

trigger #0 ok delay 10 run "repeat #0"

Would keep the job #0 relaunching 10 seconds after completing until removed:

trigger #0 ok flush

Builtins:System

The commands included in the system set are:

Signal

signal [signal: kill, hup, user1, user2, stop, quit, continue] #jobid or pid

The signal commands send a signal to an active running job or a process based on a process identifier (number).

Cd

cd directory
cd #job

Change the current directory to the specified one, with regular absolute or relative syntax. It is also possible to cd back into the directory that was current when a specific job was created.

Cd also tracks which directories commands are being run from and adds them to a history. This can be accessed via the special:

cd f ...

Where the f will be omitted and the set of suggested completions will come from the list of favourites. This can be manually altered, using f- to remove a path and f+ . or f+ /some/path to add it to the favorites.

Env

env [#job] key value

This is used to change the environment for new jobs. It can also be used to update the cached environment for an existing job. This environment will be applied if the job is repeated, or if a new job is derived from the context of one:

env #0 LS_COLOR yes
#0 ls /tmp

List

list [path]

List is similar to that of traditional 'ls', but adds mouse navigation, re-use of the same job window to step in / out of directories and automatic refresh on changes to the directory (if inotifytools are present).

Term (prefix + !)

This exposes a number of ways to spawn a legacy terminal device and emulator, with a prefix which hints at how the command line is translated and how the window management should work.

the !! form disables parsing, expansion and so on and forwards the line verbatim through /bin.sh (config:sh_runner).

The different prefixes are:

(a and l are reserved prefixes that will be used for arcan clients specifically)

Stash

The stash command lets you queue filenames and then applying group actions on them all at once. It also monitors the queue of drag-and-drop items.

This is useful for hand-picking sets of files.

stash add [file] [file2] ...

This adds the specified files to an existing stash (or create a new if there isn't one) that is visible as a job.

It is also possible to detect changes to the stash.

stash verify

This runs a checksum tool (e.g. sha256sum) for each item. Repeating the verification would mark (-) those that have changes since last.

It is also used to build a virtual filesystem:

stash map /some/stash/entry new/path/name

Then build an archive:

stash archive tar myfile.tar

Omit the destination file to get the contents into a job.

Builtins:Spreadsheet

The spreadsheet is a neat way of organising and processing data, and some other commands will create one automatically as needed. It has a basic programming language with the same syntax as the regular Cat9 CLI.

New

To create a new spreadsheet you run:

new

Insert / Replace

The generic format is:

insert location item1 [...]

Where location can be a single row reference, or a cell reference (e.g. C4). The item1 are subject to the same argument parsing and job slicing rules as everything else, but can also be populated by an external command that will run through /bin/sh:

insert 4 [separate "ptn"="%s+"] [split "ptn"="\n"] !shell-command

Where (ptn) is a valid Lua pattern. 'Split' will be used to split into new rows and separate to divide up into columns.

Set

To populate a single cell, you use the 'set' command:

set location value

Where (value) can be a string, number or expression. Expressions are distinguished by the '=' prefix:

set C1 =max(A1:A5)

Would populate C1 with the maximum value found in the range of cells from A1,A2..A5.

Remove

The remove command is used to remove a full row or column.

remove [noshift] row or column [count=1]

If the shift option is specified, everything below the row or to the right of the column will be shifted up/left (count) number of times.

Plot

The plot command combines embedded media job creation like 'open' would, with a copy operation, forwarding the sliced data to an external renderer like 'gnuplot' or 'dot' to generate a chart or graph and finally afsrv_decode to render it back into the job.

Dev

The dev builtin set is experimental and ongoing. It aims to consolidate developer tooling such as build systems, debuggers, revision control and so on.

Debug

This implements debugger protocols such as DAP. It is a fairly complex one as it is a complete debugger frontend with the many views and controls that entails.

It starts simple enough:

debug launch /path/to/program arg1 ..
debug attach pid

This will spawn a number of views (or a warning in the case of attach where permission restrictions may prevent you from attaching to a process) and the command-line will default to forward commands to the debugger process itself.

Debug:thread

Most debug commands requires a thread and a frame reference.

Debug:files

Debug:maps

Backstory

Arcan is what some would call an ambitious project that takes more than a few minutes to wrap your head around. Among its many subprojects are SHMIF and TUI. SHMIF is an IPC system -- initially to compartment and sandbox media parsing that quickly evolved to encompass all inter-process communication needed for something on the scale of a full desktop.

TUI is an API layered on top of SHMIF client side, along with a text packing format (TPACK). It was first used to write a terminal emulator that came bundled with Arcan, and then evolved towards replacing all uses of ECMA-48 and related escape codes, as well as kernel-tty and userspace layers. The end goal being completely replacing all traces of ncurses, readline, in-band signalling and so on -- to get much needed improved CLIs and TUIs that cooperate with an outer graphical desktop shell rather than obliviously combat it.

This journey has been covered in several articles, the most important ones being 'the dawn of a new command line interface' and the more recent 'the day of a new command line interface: shell'.

The later steps then has been a migration toggle in the previous arcan terminal emulator that allows a switch over to scripts with embedded Lua based bindings to the TUI API and its widgets. This later mode and support scripts is what we refer to as Lash. Lash in turn is too barebones to be useful, and a set of user scripts are plugged in, Cat9 is one such set.