Fossil

Using TLS-Encrypted Communications with Fossil

If you are storing sensitive information in a repository accessible over a network whose integrity you do not fully trust, you should use TLS to encrypt all communications with it. This is most true for repositories accessed over the Internet, especially if they will be accessed from edge networks you do not control, since that admits of various forms of man in the middle attack.

TLS protects the credentials used to access the server, prevents eavesdropping, prevents in-flight data modification, prevents server identify spoofing, and more.

There are two major aspects to this, both of which have to be addressed in different ways. Those are the subjects of the next two major sections.

Client-Side Configuration

You can build Fossil against OpenSSL to allow it to clone and sync with a remote Fossil repository via https URIs.

Building Against OpenSSL Automatically

The configure script will attempt to find OpenSSL on your system automatically. It first tries asking the pkg-config system where the OpenSSL development files are, and if that fails, it falls back to looking through a list of likely directories.

If it can't find the files it needs, the most common solution is to install the OpenSSL development package on your system via your OS's package manager. Examples:

• RHEL & Fedora: sudo dnf install openssl-devel
• Debian & Ubuntu: sudo apt install libssl-dev
• FreeBSD: su -c 'pkg install openssl'
• macOS: sudo brew install openssl
• Cygwin: Install openssl-devel via Cygwin's setup-*.exe program

The macOS case requires explanation. Apple last shipped OpenSSL develpoment files in OS X 10.6 (Snow Leopard), choosing to deprecate it from that point forward. (Apple wants you to use their proprietary platform-specific encryption methods instead.) Since macOS has no built-in package manager, a number have sprung up out of the FOSS world. It is not known to this author whether Fossil's current build system can find OpenSSL as installed with any of these other package managers, so unless you have a particular reason to avoid it, we recomend that you use Homebrew on macOS to install OpenSSL as above. Fossil's build system will seek it out and use it automatically.

Building Against a Non-Platform Version of OpenSSL

The Fossil build system has a few other methods for finding OpenSSL when the automatic methods fail or when you'd prefer that Fossil use a different version of OpenSSL than the one Fossil's build system picks on its own.

A good reason to do this is when the Fossil build system finds a functioning version of OpenSSL which is nevertheless unsuitable. One common case is that your OS is sufficiently outdated that the platform version of OpenSSL can no longer communicate with remote systems adhering to the latest advice on secure communications. An old OpenSSL might not support any of the cipher suites the remote Fossil repository's HTTPS proxy is willing to offer, for example, so that even though both sides are speaking a variant of TLS/SSL, the peers cannot come to an agreement on the cryptography.

If you've installed the OpenSSL development files somewhere that Fossil's build system cannot find on its own, you can clue it in by passing the --with-openssl option to the configure script. Type ./configure --help for details.

Another option is to download the source code to OpenSSL and build Fossil against that private version of OpenSSL:

    cd compat             # relative to the Fossil source tree root
    tar xf /path/to/openssl-*.tar.gz
    ln -fs openssl-x.y.z openssl
    cd openssl
    ./config              # or, e.g. ./Configure darwin64-x86_64-cc
    make -j11
    cd ../..
    ./configure --with-openssl=tree
    make -j11

That will get you a Fossil binary statically linked to this in-tree version of OpenSSL.

Beware, taking this path typically opens you up to new problems, which are conveniently covered in the next section!

Certificates

To verify the identify of a server, TLS uses X.509 certificates, a scheme that depends on a trust hierarchy of so-called Certificate Authorities. The tree of trust relationships ultimately ends in the CA roots, which are considered the ultimate arbiters of who to trust in this scheme.

The question then is, what CA roots does Fossil trust?

If you are using a self-signed certificate, Fossil will initially not know that it can trust your certificate, so you'll be asked if you want to accept the certificate the first time you communicate with the server. Verify the certificate fingerprint is correct, then answer "always" if you want Fossil to remember your decision.

If you are cloning from or syncing to Fossil servers that use a certificate signed by a well-known CA or one of its delegates, Fossil still has to know which CA roots to trust. When this fails, you get an error message that looks like this in Fossil 2.11 and newer:

    Unable to verify SSL cert from fossil-scm.org
      subject: CN = sqlite.org
      issuer:  C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
      sha256:  bf26092dd97df6e4f7bf1926072e7e8d200129e1ffb8ef5276c1e5dd9bc95d52
    accept this cert and continue (y/N)?

In older versions, the message was much longer and began with this line:

    SSL verification failed: unable to get local issuer certificate

Fossil relies on the OpenSSL library to have some way to check a trusted list of CA signing keys. There are two common ways this fails:

1. The OpenSSL library Fossil is linked to doesn't have a CA signing key set at all, so that it initially trusts no certificates at all.

1. The OpenSSL library does have a CA cert set, but your Fossil server's TLS certificate was signed by a CA that isn't in that set.

A common reason to fall into the second trap is that you're using certificates signed by a local private CA, as often happens in large enterprises. You can solve this sort of problem by getting your local CA's signing certificate in PEM format and pointing OpenSSL at it:

     fossil set --global ssl-ca-location /path/to/local-ca.pem

The use of --global with this option is common, since you may have multiple repositories served under certificates signed by that same CA. However, if you have a mix of publicly-signed and locally-signed certificates, you might want to drop the --global flag and set this option on a per-repository basis instead.

A common way to run into the broader first problem is that you're on FreeBSD, which does not install a CA certificate set by default, even as a dependency of the OpenSSL library. If you're using a certificate signed by one of the major public CAs, you can solve this by installing the ca_root_nss package. That package contains the Mozilla NSS certificate bundle, which gets installed in a location that OpenSSL checks by default, so you don't need to change any Fossil settings. (This is the same certificate set that ships with Firefox, by the way.)

The same sort of thing happens with the Windows build of OpenSSL, but for a different core reason: Windows does ship with a stock CA certificate set, but it's not in a format that OpenSSL understands how to use. Rather than try to find a way to convert the data format, you may find it acceptable to use the same Mozilla NSS cert set. I do not know of a way to easily get this from Mozilla themselves, but I did find a third party source for the cacert.pem file. I suggest placing the file into your Windows user home directory so that you can then point Fossil at it like so:

     fossil set --global ssl-ca-location %userprofile%\cacert.pem

This can also happen if you've linked Fossil to a version of OpenSSL built from source. That same cacert.pem fix can work in that case, too.

When you build Fossil on Linux platforms against the binary OpenSSL package provided with the OS, you typically get a root cert store along with the platform OpenSSL package, either built-in or as a hard dependency.

Client-Side Certificates

You can also use client side certificates to add an extra layer of authentication, over and above Fossil's built in user management. If you are particularly paranoid, you'll want to use this to remove the ability of anyone on the internet from making any request to Fossil. Without presenting a valid client side certificate, the web server won't invoke the Fossil CGI handler.

Configure your server to request a client side certificate, and set up a certificate authority to sign your client certificates. For each person who needs to access the repository, create a private key and certificate signed with that CA.

The PEM encoded private key and certificate should be stored in a single file, simply by concatenating the key and certificate files. Specify the location of this file with the ssl-identity setting, or the --ssl-identity option to the clone command.

If you've password protected the private key, the password will be requested every time you connect to the server. This password is not stored by fossil, as doing so would defeat the purpose of having a password.

If you attempt to connect to a server which requests a client certificate, but don't provide one, fossil will show an error message which explains what to do to authenticate with the server.

Server-Side Configuration

Fossil's built-in HTTP server got TLS support in December 2021, released as version 2.18 in early 2022. Prior to that, system administrators that wanted to add TLS support to a Fossil server had to put it behind a reverse proxy that would do the translation. Since advantages remain for delegating TLS to another layer in the stack, instructions for doing so continue to be included in our documentation, such as:

• Serving via stunnel
• Serving via stunnel + althttpd
• Serving via SCGI with nginx on Debian

Enforcing TLS Access

To use TLS encryption in cloning and syncing to a remote Fossil repository, be sure to use the https: URI scheme in clone and sync commands. If your server is configured to serve the repository via both HTTP and HTTPS, it's easy to accidentally use unencrypted HTTP if you forget the all-important 's'.

As of Fossil 2.9, using an http:// URI with fossil clone or sync on a site that forwards to HTTPS will cause Fossil to remember the secure URL. However, there's a TOFU problem with this: it's still better to use https:// from the start.

As of Fossil 2.8, there is a setting in the Fossil UI under Admin → Access called "Redirect to HTTPS," which is set to "Off" by default. Changing this only affects web UI access to the Fossil repository. It doesn't affect clones and syncs done via the http URI scheme.

In Fossil 2.7 and earlier, there was a much weaker form of this setting affecting the /login page only. If you're using this setting, you should migrate to the new setting as soon as possible, because the old setting allows multiple ways of defeating it.

WARNING: Enabling HTTPS redirects at the Fossil repo level while running Fossil behind an HTTPS proxy can result in an infinite redirect loop. It happens when the proxy mechanism presents "http" URIs to Fossil, so Fossil issues a redirect, so the browser fetches the page again, causing Fossil to see an "http" URI again, so it issues a redirect...'round and 'round it goes until the web browser detects it's in a redirect loop and gives up. This problem prevents you from getting back into the Admin UI to fix it, but there are several ways to fix it:

1. Reset via CLI. You can turn the setting back off from the CLI with the command "fossil -R /path/to/repo.fossil set redirect-to-https 0". (Currently doesn't work.)

1. Backup first. This setting is stored in the Fossil repository, so if you make a backup first on the server, you can restore the repo file if enabling this feature creates a redirect loop.

1. Download, fix, and restore. You can copy the remote repository file down to a local machine, use fossil ui to fix the setting, and then upload it to the repository server again.

It's best to enforce TLS-only access at the front-end proxy level anyway. It not only avoids the problem entirely, it can be significantly more secure. The nginx-on-Debian proxy guide shows one way to achieve this.

Terminology Note

This document is called ssl.wiki for historical reasons. The TLS protocol was originally called SSL, and it went through several revisions before being replaced by TLS. Years before this writing, SSL finally became entirely obsolete due to weaknesses in the protocol fixed in the later TLS series of protocols.

Some people still use the term "SSL" when they actually mean "TLS," but in the Fossil project, we always use "TLS" except when we must preserve some sort of historical compatibility, as with this document's name in order to avoid broken external URLs. The Fossil TLS-related settings also often use "ssl" in their names for the same reason.

This series of protocols is also called "HTTPS" after the URI scheme used to specify "HTTP over TLS."