Help

Building Benben (v0.7.1 and Prior)

These instructions will explain how to build Benben from source. Benben is written in Common Lisp, and building it is very different from building a C, Crystal, or Rust program, so please read these instructions carefully. Remember that AppImages are available as an alternative to building it yourself.

Note: these instructions are only for v0.7.1 and earlier. If you are building the trunk code or a later version, please consult Building Benben instead.

Dependencies

There are four conceptual sets of dependencies:

  1. A Common Lisp implementation.
  2. C/C++ libraries.
  3. Common Lisp libraries that you can get via Quicklisp
  4. Common Lisp libraries you have to get manually. These are mostly the ones that Remilia (the author of Benben) wrote.

You'll need SBCL, preferably a recent version. SBCL 2.3.11 or later is required, and SBCL 2.5.x or later is recommended. Older SBCL versions may work, but are not officially supported. Other Lisp implementations will NOT work, are NOT supported, and will NOT be supported in the future.

Your copy of SBCL will need these packages included with it, otherwise Benben will not build (they are usually included by default):

  • sb-posix
  • sb-thread

These are likely good to have as well:

  • sb-unicode
  • sb-simd (on x86-64 platforms)

You can test for these by starting SBCL and typing (require 'sb-posix), replacing the sb-posix part as-needed.

C/C++ Dependencies

Required C/C++ libraries:

  • libzstd
  • S-Lang (2.3.2 or higher)
  • libmpg123 (1.29 or higher)
  • libxmp (4.6.0 or higher recommended)
  • libsidplayfp (2.1.0 or higher recommended)
  • libwavpack (5.0 or higher recommended)
  • libopus (1.3.0 or higher recommended)
  • libao (1.2.0 or higher recommended)
  • libremicsid

Optional C libraries:

  • libasound (only when the ALSA driver is enabled)
  • libout123 (only when the out123 driver is enabled)

Some of the Quicklisp-installable Common Lisp dependencies may have additional C/C++ library requirements and are not listed here.

Important: static libraries are not supported.

Dependencies You Can (Probably) Get via Quicklisp or Similar

These dependencies should be available via Quicklisp, or if they aren't they should be easy to find. You can check out the file dev-scripts/build-benben-lisp-x86-64-appimage.rb for recommended repository URLs and tags/commits/checkouts.

If you don't want to use QuickLisp (Remilia doesn't), then the Ruby script that's used to build the official AppImages may be useful, as it has an updated list of URLs for these dependencies.

A large chunk of these dependencies are for the FXML or Dexador libraries, or are subdependencies.

  • uiop (part of ASDF)
  • alexandria
  • babel
  • cffi
  • closer-mop
  • cl-ppcre
  • cl-cpus
  • trivial-indent
  • computable-reals
  • st-json
  • lparallel
  • atomics
  • documentation-utils
  • trivial-garbage
  • zstd (cl-zstd)
  • quri
  • dexador
  • fxml
  • chipz
  • trivial-features
  • trivial-gray-streams
  • parse-number
  • bordeaux-threads
  • global-vars
  • cl-octet-streams
  • split-sequence
  • cl-utilities
  • idna
  • flexi-streams
  • named-readtables (git commit 6eea56674442b884a4fee6ede4c8aad63541aa5b or later)
  • mgl-pax-bootstrap (part of mgl-pax)
  • cl-who
  • serapeum
  • trivia
  • type-i
  • iterate
  • introspect-environment
  • lisp-namespace
  • trivial-cltl2
  • string-case
  • parse-declarations
  • trivial-file-size
  • trivial-macroexpand-all
  • mmap
  • pathname-utils
  • fast-http
  • proc-parse
  • xsubseq
  • smart-buffer
  • fast-io
  • static-vectors
  • chunga
  • cl-cookie
  • local-time
  • trivial-mimes
  • cl-base64
  • usocket
  • cl+ssl (cl-plus-ssl)

Dependencies You Cannot Yet Get via Quicklisp

Follow these links, then clone the repositories somewhere that ASDF can find them. See the section of the ASDF manual on where ASDF looks for dependencies for more information. Tarballs for these are also available on their respective release pages.

If you do clone them with Fossil, then after they are cloned (and before you build Benben), for each one, run fossil checkout <version>, replacing <version> with the required version listed below. For example, Benben expects CL-RemiMarshal v0.4.3, so run fossil checkout v0.4.3 where you cloned CL-RemiMarshal. If you just download the tarballs, then you can skip this step, as they are already at the correct version.

These dependencies are all maintained by Remilia, so you can contact her if you have any issues with them. You can find her contact details on the Benben repository homepage.

Optional Dependencies Needed for Building

These programs are only needed if you want to build the manual.

  • rst2man
  • gzip
  • texi2any

A Special Note About the C/C++ Dependencies

SBCL binaries behave differently than, say, C or Crystal binaries when it comes to shared libraries.

For the C/C++ dependencies, it is highly recommended that they be installed where the dlopen() function can find them. This function is used by the Lisp system to load shared libraries both at build time, and at run time. See man 3 dlopen for information on where dlopen() searches for shared library files. Usually this means one of the directories listed in the $LD_LIBRARY_PATH environment variable.

If libraries are not installed where dlopen() can automatically find them, then you must specify the path with the $LD_LIBRARY_PATH environment variable both at build time, and every time you run Benben. This information is not stored in the binary. For example:

# When building Benben
LD_LIBRARY_PATH="/path/to/so/directory:$LD_LIBRARY_PATH" ./build.sh <build options...>

# And then when running it
LD_LIBRARY_PATH="/path/to/so/directory:$LD_LIBRARY_PATH" benben <music files>

So if (as an example) you install libremicsid into /home/user/libs, then the $LD_LIBRARY_PATH environment must include /home/user/libs, and this must be set both when you build Benben, and every time you run it. You may want to set the environment variable in your shell's startup script (e.g. ~/.bashrc or similar) to make it easy.

tl;dr: make sure the C/C++ libraries are in one of the directories listed in your $LD_LIBRARY_PATH environment variable. When in doubt, the safest bet for most Linux distributions is to install them into /usr/lib.

A Note for Package Maintainers

It's highly recommended you clear out your fasl cache prior to building Benben. This will ensure Benben gets built with the correct compiler policies applied. A common way to do this is to run rm -rf ~/.cache/common-lisp/ prior to running the build script.

Getting Benben's Sources

$ fossil clone https://chiselapp.com/user/MistressRemilia/repository/benben/
$ cd benben

Benben is, itself, a Common Lisp system (actually three of them), so be sure you clone the repository where ASDF can locate it. When in doubt, put it in the same location as you put the other Common Lisp dependencies. Again, see the section of the ASDF manual on where ASDF looks for Common Lisp systems. If ASDF can't find the Benben source code, then you will get an error that it can't find benben-lib.

Building With the Build Script

Using the provided build script, build.sh, is the expected way to build Benben. For a default build, you can just run ./build.sh from the root repository directory. The binary will be in the bin/ directory.

Important: Do NOT strip the resulting binary. It will not function if you do. Have I mentioned Common Lisp binaries behave differently?

You can run ./build.sh --help to see a list of available options. In many cases you won't need to use any of them, though you may want to do ./build.sh --manual to build Benben's man page, Info page, and manual files.

If your copy of SBCL does not include the SB-SIMD package, you're on a non-x86-64 platform (e.g. ARM AArch64), or you're on a CPU older than Intel's Broadwell microarchitecture, then you will have to pass the --no-simd option to the build script. This will result in a slightly slower binary.

Building Without the Build Script

The first step is to set up some environment variables:

  • BENBEN_OPT_MODE: This controls the build type. It can be one of: normal, unsafe, or debug.
  • BENBEN_NO_SIMD: If this is set to 1, then various blocks of SIMD code are disabled.
  • BENBEN_NO_OUT123: If this is set to 1, then the out123 driver is disabled.
  • BENBEN_NO_ALSA: If this is set to 1, then the ALSA driver is disabled. Note that the ALSA driver is always disabled on non-Linux systems anyway.
  • BENBEN_COMPRESS: If this is set to 1, then a compressed binary is built. Setting this to 0 produces an uncompressed binary.
  • BENBEN_DONT_RELOAD_LIBRARIES: Normally the build script will rebuild a large chunk of Common Lisp libraries to ensure that they are built with the same compiler policies as Benben. Setting this to 1 will tell the build script to NOT rebuild these libraries. This saves a large chunk of time, especially if you're rebuilding Benben quite a bit and you know you haven't changed things in Slime or elsewhere. It's recommended you leave this unset or set to 0, however.

Note: this environment variable can also be used with the Rakefile.

Next, you'll want to determine the maximum heap size you want. The Rakefile defaults to 8192 (that is, 8 gigabytes), and is the recommended size. Benben won't use this much, but it basically puts a cap on how much the Lisp runtime is allowed to allocate. Do not set this too low or Benben won't even start.

Finally, start SBCL like this from the root of the repository so that it loads the build file:

$ sbcl --dynamic-space-size <MAX HEAP SIZE HERE> --disable-ldb --merge-core-pages --disable-debugger --load build-scripts/build-benben.lisp

The finished binary will be in the bin/ directory. Important: Do NOT strip the resulting binary. It will not function if you do.

Additional Debug Info

Benben contains various internal debugging instruments that may be useful if you're hacking on the code. Here are some tips for debugging:

  • Benben's Minimal UI is great for debugging most of the code. All messages get logged to your REPL's output. In fact, that's why and how the Minimal UI started life.
  • To debug Benben's Original UI code, you'll want to start your Lisp process with the start-with-swank.sh file. This will load Benben, then start up a new Swank listener that you can connect to via Slime or similar. The script takes a few options as well, so check ./start-with-swank.sh --help.
  • Benben (and SatouSynth) redefines CL-SDM's DEFINE-TYPED-FN function and MUFFLING macro to better aid in debugging. Look at src/lib/redefines.lisp to see how they differ.
  • If you want to always invoke the Lisp debugger every time a player has an error, push :BENBEN-DEBUG-PLAYER-ERRORS onto *FEATURES* and recompile Benben. This will occur within a HANDLER-BIND that exists within the main PLAYER-MANAGER's SUBSYSTEM-START method.
  • Unless you're debugging the Original UI, you'll almost certainly want to load the startup-scripts/benben-debug.lisp into your REPL. This sets up most of the debugging options by pushing these symbols onto *FEATURES*: :REMIAUDIO-EXTRA-CHECKS, :NEVER-APPLY-BLOCK-COMPILATION, :SATOU-NEVER-INLINE, :SATOU-DEBUG, :SATOU-VERBOSE-DEBUG, :BENBEN-NEVER-INLINE, :BENBEN-DEBUG, :BENBEN-VERBOSE-DEBUG.

Building the AppImage

Building an AppImage of Benben is, in general, very easy, though it does assume that you are on an x86-64 CPU that supports the FMA3 SIMD instructions. This means a CPU that's at least as new as Intel's Broadwell architecture, released in 2014. It also assumes that your copy of SBCL includes the SB-SIMD package.

If you want to build the AppImage from scratch, then you should use the build-benben-lisp-x86-64-appimage.rb script from the dev-scripts/ directory. As indicated by the name of the script, it only builds an x86-64 binary. Using this script won't require you to download any of the Common Lisp dependencies (Quicklisp or otherwise) listed above since this will automatically be handled by the script. You will still need to install the C/C++ dependencies yourself, however.

This script is specifically built to be run on a Slackware Linux 14.2 system. This was chosen for two reasons:

  1. Remilia is extremely familiar with Slackware, having used it almost exclusively since 2002.
  2. It uses an old glibc version, which is important for producing AppImages that run on a wide variety of systems.

Newer versions of Slackware are known to work as well. Other Linux distributions may work are but are not supported by this script. So you may need to modify it to suit your needs.

The script clones repos and builds things in the system's /tmp directory. You'll need a couple hundred megabytes of free space for this.

To use the script, simply run it:

$ ./build-benben-lisp-x86-64-appimage.rb

The resulting AppImage will be at /tmp/benben-<version>-x86-64.Appimage.