\documentclass[11pt]{article}
\begin{document}
\title{How to build CSL and REDUCE}
\author{Arthur Norman, Codemist Ltd}
\maketitle
\section{Very Brief Instructions}
Start with a ``sh''-compatible shell. On Windows start in a
cygwin shell. You will need a reasonably recent version of
automake as well as a C compiler etc installed.
\begin{verbatim}
# Configure, build and install support packages
cd r38/lisp/csl/support-packages
chmod +x build*
./build-all.sh
cd ..
# Ensure some files are executable when they need to be.
# This is in case the file-transfer scheme used has not done
# so already for you.
pushd cslbase; chmod +x stamp auto configure; ./stamp; popd
pushd ../util; . ./setexec.sh; popd
# Create and select directory for building REDUCE
mkdir somedir
cd somedir
#
# Configure it, using the support packages built above
../cslbase/configure --with-fox=../fox
# Compile REDUCE and build its image file.
make r38
make r38.img
# Run all tests scripts and compare with reference output
make testall
make checkall
# Admire the speed of your computer
more log/times.log
\end{verbatim}
\section{Introduction}
CSL can be built on a range of platforms. This document is to assist those
who have access to the sources to recompile it for themselves. The platforms
that are mainly used for testing are
\begin{enumerate}
\item Linux: the exact distribution used may change over time! When this
document was initially drafted it was RedHat 9.0. Shortly after that it
moved to Fedora Core, and then to SuSE 9.0 in its 64-bit edition. Some attempt
is made to test across a range of systems, but usually these will concentrate
on those that are most up to date, so support for old versions of libraries
and so on may cause trouble at times. Note also that increased security
techniques that Fedora and perhaps some other versions of Linux seem to
be introducing make much mode more delicate.
\item Windows XP: this uses the cygwin/mingw package, ie a version of gcc and
its support tools tuned so that Microsoft's libraries are used for everything.
I take care not to link in the main cygwin DLL which comes under the GPL.
\item Macintosh Darwin: used with X windows and hence as a very distinctly
Unix-like environment. Note that earlier Macintosh systems are quite different
and are not so easily supported. Testing was started on Darwin 10.1 but
is now on 10.2.
\item Linux-64: an emerging platform that ensures that the code is 64-bit
clean.
\end{enumerate}
Support of these and other platforms is achieved through use of the GNU
development tools autoheader, automake and autoconf. These provide a scheme
where build options can be set up fairly easily for any system that supports
a version of the Bourne Shell (ie \verb+/bin/sh+). Two master files
\verb+configure.ac+ and \verb+Makefile.am+ contain all information
about what to do. These get processed
by the GNU tools to create a script called \verb+configure+. This script only
needs adjustment when \verb+configure.ac+ or \verb+Makefile.am+ change.
It can then be run on any system and creates two files: \verb+Makefile+ and
\verb+config.h+. These
encapsulate system-specific information about the target. When a new
target machine is to be supported the hope is that in very many cases the
tests done in the configure script will adapt everything for it totally
automatically. In some cases a new target will reveal the need for extra
testing to be established by \verb+configure.ac+.
When a system has been configured and built, a file \verb+config.status+ will
exist and it lists the configuration options that were used in the build.
Extract the one key line from that and keep it in acse you need to
reproduce a situation. A file \verb+config.h+ shows the various aspects of
your system that may have been considered.
\section{Preconditions}
The body of this code would like to have a number of support packages
available to it. These are:
\begin{enumerate}
\item The FOX GUI toolkit. It is fairly unlikely that this is
already installed for you, so you are probably going to have to build
it for yourself. Notes on how to do that are given later on here;
\item Threads. Either windows native threads or Posix Threads;
\item tercap, terminfo, curses, ncurses or pdcurses for screen control
in command-line mode on Unix-like systems.
\end{enumerate}
If and or all of these are not found then a system can still be built --
it will just not have as many facilities and may feel clumsy in use.
\section{The Ideal Case}
In an idea case you can first identify the directory
\ldots\verb+/lisp/csl/cslbase+,
which I will now refer to as \verb+$CSL+. Create a totally fresh new directory
parallel to \verb.$CSL., and select it as current. In particular do not
have \verb+$CSL+ as your current directory. Eg select \verb.$CSL. and go
\begin{verbatim}
mkdir ../local-build
cd ../local-build
\end{verbatim}
Run the command
``\verb+$CSL/configure+'', which should generate several screens-full
of trace output and end up having created ``\verb+Makefile+'' and
``\verb+config.h+'' in your current directory.
If you already have a working \verb+r38.img+ file you can now just go
``\verb+make r38+'' (or on Windows ``\verb+make r38.exe+''). If you need
to bootstrap everything from scratch the sequence is
\begin{description}
\item{\verb+make slowr38.img+:} Builds a bootstrap version of Reduce which
will be significantly slower than the final one;
\item{\verb+make profile+:} This step runs for a significant time, and collects
measurements from all the Reduce tests. The output is used to tune the
final version. It is only necessary to re-do this analysis when rather
substantial changes to Reduce have been made;
\item{\verb+make prof-inst+:} A step to install the statistics gathered above.
This is done as a separate step to allow one to check that the previous
job ran, and make a manual sanity check of what it did;
\item{\verb+make c-code+:} Compile selected parts of Reduce from Lisp into C.
The generated C is placed in the directory ``\verb+../csl-c+'';
\item{\verb+make r38.img+:} The final version of Reduce can be built.
\item{\verb+make install+:} The status of this target is uncertain at present,
but in due course it ought to work!
\end{description}
\section{Installing or building the support packages}
The short version of this section is ``See the script
\verb+build-all.sh+'' in the support-packages directory. That is
a simple script that does not require administrator priviledge and it
builds FOX, installing it in \verb+../fox+.
Note that I include a short patch script in the build script I have
for FOX 1.4 or 1.6. This makes a few patches to FOX that help me build under
cygwin, support systems as old as Windows 95 and cope with some Macintosh
oddities that otherwise pained me. The patch utility used should be
reasonably resilient and modest changes to FOX releases should not confuse
it, but if FOX changes in substantial ways the adjustments I propose may
become irrelevant or just plain different. The version current at the
date of the REDUCE release is FOX 1.4.27 and the patches certainly apply
there and to 1.6.0-rc-2.
I have collected original versions of the various tools and
sub-packages that I use, and put them in a support-packages
directory. These tools come under a variety of licenses: some are
fully public domain, some under BSD license, some under LGPL or
a modification. FOX is LGPL with an adjustment that permits distribution
of statically linked binaries. Where I have GPL code it is not used
in any release package but just in the process of building things -
for instance the autoconf tools (and note that they have a special
exception to the GPL for some of their components that may have to
be re-distributed). Please respect the licenses. You should probably
search for and visit the web sites of the various too providers: if you
fetch and install any version of their code different from the one that
I use then it is your responsibility not mine to sort out any bugs or
incompatibilities that arise. That said, my intention is that my code
should work with a reasonable degree of portability across systems and
versions.
\subsection{Cygwin and MinGW}
To install \verb+cygwin+ visit \verb+http://www.cygwin.com+, fetch the
\verb+setup+ program and run it. You get a chance to select which packages
you want installed, and you must select all the development tools, such
as \verb+automake+, \verb+gcc+ and \verb+make+, as well as all the
parts of \verb+mingw+ shown there. Once \verb+cygwin+ is installed you should
launch a \verb+bash+ shell, and go
\begin{verbatim}
mkdir /mingw
cd /mingw
tar xvfz <path-to-files>/mingw...tar.gz
tar xvfz <path-to-files>/w32api...tar.gz
\end{verbatim}
\noindent where the two files to be unpacked are in my \verb+support-files+
directory under \verb+mingw+.
To get a working setup on a clean machine I used the expanded view of all the
packages that the cygwin installer could support, and selected (at least)
\begin{verbatim}
autoconf, automake 1.9, binutils, bison, gcc, gcc-mingw,
glib, glib-devel, glib2-devel, libncurses-devel,
libtool1.5, libXft, make, mingw-zlib, patchutils,
xorg-x11-devel, zip
\end{verbatim}
\noindent Of these, the X11 tools are only useful for building a
private cygwin-only version. I also need some editor (eg \verb.emacs.
or \verb.vi.) and find \verb.gdb. and \verb.openssh. useful tools
to have around.
\subsection{FOX}
Even with the patch that I apply my code will not run on
a freshly-installed copy of Windows 95 --- you need (I believe) to have
installed a suitable version of Internet Explorer to get some system
libraries updated. I found that I had to find copies of \verb+ws2_32.dll+
and \verb+wshelp.dll+ at least. I expect that few people will now be
concerned about Windows 95 and those who are will already have installed
enough other packages that they will have all the libraries that they need!
In most respects the use of FOX that I make is rather simple and should
not stress things. I can build using FOX 1.4.x and the ``1.6'' branch.
In the past I had versions using FOX 1.1 and 1.2.
The flag \verb+--with-fox=<dir>+
for \verb+configure+ can be used to indicate the location
of the FOX libraries.
It seems at present that FOX must be configured and built in its main
directory. If you use my ``build-all.sh'' or ``build-fox-1.4.sh'' scripts
you do not have to worry. Otherwise first ensure that no config.cache files
are left over possibly storing stale information. Then run ``configure''.
You can give an option --using-opengl=no to disable support for 3D graphics,
and on some platforms you need to do that if you are to have a hope of
success. You might need to specify the C and C++ compilers you use, or some
flags that they need. You can select a debug or release build. The
configure script tries to make good guesses and selections for you.
If you do not have system rights on your computer you will need to
select a directory to install the version of FOX that you build, and
specify it using \verb+--prefix=$HOME/support+ (or some such). On some
machines going \verb+--disable-dynamic+ to get a build that does not even try
to create dynamically linked libraries may help. On Darwin there is a patch
that may or may not already have been applied to the libtool that you have:
scan the FOX web-site to find comments about it or hope that by the time
you read this my comments are outdated and irrelevant.
After configuring just go ``\verb+make+'' and go and brew some coffee.
As well as making the FOX library this build a collection of tests and
demonstrations (in the test directory). Check some of them out and if
they all fail you have some more work to do! If the tests appear to behave
well you can go ``\verb+make install+'', which unless you have used
\verb+--prefix+ will need root access and may put files
into \verb+/usr/local+. If it does then you should review
\verb+/etc/ld.so.conf+ and run ldconfig since otherwise applications may
not find your library. On some systems you may need to arrange that
\verb+LD_LIBRARY_PATH+ gets set to include the directories where FOX has
been placed, or add \verb+/usr/local/lib+ to the list of placed in
\verb+/etc/ld.so.conf+. At present my recommendation is that you use
\verb+--prefix+ so that FOX is installed in a private location, and that you
use just static libraries so all the complication about run-time access to
them is avoided. Indeed I suggest that unless you have real need to do
something else that you just use my scripts to build things.
I note that I expect that in many cases it will prove least confusing to build
FOX to be installed in a private location such as \verb+../fox+, and to
make only a statically-linkable version. That is what the scripts that I
provide do for you.
\subsection{pthreads etc.}
My own code uses native Windows threads on Windows, and Posix Threads
(pthreads) on all other platforms. In fact the use of threads is about
as primitive and simple as it could be, so alternative threading options
might be manageable on awkward platforms. I need the ability to fork
a new thread, join, and to implement a mutex. For inter-thread
communication I create a pipe and write bytes from one thread to
the other. I do not make use of counting semaphores or condition variables.
Threads are only used at all when running with FOX in windowed mode, so if
they are not available on some system a simple terminal-mode system can
still be built.
\subsection{gcc and other C compilers}
The C and C++ compiler used is expected to be (say) less than a decade
old. In C++ mode it must support C++ namespaces (at present FOX declares
a namespace but then sets things up so that it does not separate things off
very much. Future releases of FOX are liable to be more aggressive in this
respect). In some cases the C compiler may need extra command-line flags
to tell it to generate thread-safe code. Debugging options can make a
noticable difference to the speed of your code, but a vast different to the
size of compiled binaries. In many cases I expect the \verb+configure+ script
to sort out everything you need.
I link the FOX code in statically (and maybe in a way that leads
to somewhat bloated executable binaries) since I do not expect other
people to have those libraries installed in a way that allows arbitrary
applications to link to them. The libraries for curses or ncurses, X11,
network access and general C language support are left with their default
dynamic linking (unless \verb+--enable-static+ is passed to \verb+configure+).
That means that those libraries must be present on the machine than things
eventually run on. New or different systems may end up needing different
libraries linked in, but across a reasonable range of systems the
\verb+configure+ scripts should get things right.
\section{64-bit Windows}
I can build on the Windows-XP 64-bit edition, but the code most be considered
experimental at present. The build process is a little strange:
\begin{enumerate}
\item Fetch and install the Windows SDK from Microsoft. Ensure that you
install components that build for the x64 platform. At the time of writing the
SDK is a free download that includes command-line versions of Microsoft's
optimising compilers.
\item Fetch and install the cygwin environment. My experience so far has been
that this runs well on Windows XP-64, but running as 32-bit code.
\item Set up your path and environment so that you can run the Microsoft
compiler but are running under the \verb.bash. shell from cygwin, with access
to \verb.make., \verb.patch. and all the other GNU tools.
\item Run the \verb+build-all-win64.sh+ script in the support packages
directory to build FOX in a \verb.fox64. directory. Check that the FOX
test programs run properly.
\item Configure CSL/Reduce using the \verb.--with-xp64. option that sets up
ad hoc treatment to support using the Microsoft C compiler. Continue
from there as normal.
\end{enumerate}
\section{Tested platforms}
Here are the the systems I have tested on reasonably recently, with a few
comments.
\begin{enumerate}
\item RedHat Linux 9.0 (32-bit) with the particular
collection of installed packages that I happen to have collected. This was
once one of my main development machines and so things were well-tested on it.
However it will also have all the libraries etc that I need installed,
and will not cause me any compatibility woes with itself! The issue
of dynamic libraries and other people's versions of even RedHat Linux 9.0
remain to be addressed!
\item Fedora Core (32-bit) releases 1-5 have been tested, but generally only
the most recent will be used more than a little.
\item OpenSuSE 10.0 64-bit. This has taken over as the main development
platform.
\item Windows XP using Cygwin/MinGW32 running on an AMD Athlon. This is my
second main development machine. There may again be issues of dynamic
libraries and MinGW32 specials, but basically the compiled code should rely
on just the standard Microsoft base.
\item A Macintosh with Darwin (panther) and using the X windows system that is
native to the platform. Note that this needs the X server activated.
\item Debian 3.0r2, Suse 9.0 (32-bit), RedHat 9.0 and Fedora Core 1
and various others running on virtual machines via \verb+vmware+.
\item Solaris on a sparc Ultra-60. Tested a while ago, but not recently.
\item NetBSD 1.5 on a distinctly elderly DEC Alpha 3000. I have to
fetch, build an install Posix Threads as well as FOX, and
I needed to set \verb+LD_LIBRARY_PATH+ so it scanned both
\verb+/usr/X11R6/lib+ and \verb+/usr/local/lib+. Otherwise things looked
pretty happy on the FOX build front, and fwin, Lisp and REDUCE build OK.
No further testing on that platform is expected.
\end{enumerate}
\end{document}