Fossil

### Administrivia

It is perfectly fine for a Fossil repository to only have Setup users,
no Admin users. The smaller the repository, the more likely the
repository has no Admin-only users. If the Setup user neither needs nor
wants to grant Admin power to others, there is no requirement in Fossil
to do so. [Setup capability is a pure superset of Admin capability.][sia]

As the number of users on a Fossil repository grows, the value in
delegating administrivia also grows, because the Setup user typically
has other time sinks they consider more important.

Admin users can take over the following routine tasks on behalf of the
Setup user:

    only allows arbitrary ability to modify the repository blockchain
    and its backing data tables, it can probably also be used to damage
    the host such as via `PRAGMA temp_store = FILE`.

*   **TH1**: The [TH1 language][TH1] is quite restricted relative to the
    Tcl language it descends from, so this author does not believe there
    is a way to damage the Fossil repository or its host via the Admin →
    TH1 feature, which allows execution of arbitrary TH1 code within the
    repository's execution context. Nevertheless, interpreters are a
    well-known source of security problems, so it seems best to restrict
    this feature to Setup-only users as long as we lack a good reason
    for Admin-only users to have access to it.


[fcp]:   https://fossil-scm.org/fossil/help?cmd=configuration
[forum]: https://fossil-scm.org/forum/
[rs]:    https://www.fossil-scm.org/index.html/doc/trunk/www/settings.wiki
[sia]:   https://fossil-scm.org/fossil/artifact?udc=1&ln=1259-1260&name=0fda31b6683c206a
[th1]:   https://www.fossil-scm.org/index.html/doc/trunk/www/th1.md
[tt]:    https://en.wikipedia.org/wiki/Tiger_team#Security
[ucap]:  https://fossil-scm.org/fossil/setup_ucap_list

`chroot` feature to wall Fossil off from the rest of the machine, it's
fairly simple to set up email alerts.

(Otherwise, skip [ahead](#advanced) to the sections on advanced email
service setup.)

This is our "quick setup" option even though setting up an SMTP mail
server is not trivial, because there are many other reasons to have such
a server set up already: internal project email service, `cron`
notifications, server status monitoring notifications...

With that out of the way, the Fossil-specific steps are easy:

1.  Go to [Admin → Notification](/setup_notification) and fill out all
    of the **Required** fields:

and via the default skin's hamburger menu (☰).


<a id="unsub" name="unsubscribe"></a>
### Unsubscribing

To unsubscribe from alerts, visit the `/alerts` page on the repository,
click the "Unsubscribe" button, then check the "Unsubscribe" checkbox to
verify your action and press the "Unsubscribe" button a second time.

This interlock is intended to prevent accidental unsubscription.


<a id="test"></a>
### Test Email Service

   *  The [`/subscribers`](/help?cmd=/subscribers) page


<a id="design"></a>
## Design of Email Alerts

This section describes the low-level design of the email alert system in
Fossil.  This expands on the high-level administration focused material
above with minimal repetition.

This section assumes expert-level systems knowledge. If the material
above sufficed for your purposes, feel free to skip this section, which
runs to the end of this document.

-------------------------------

The automatic backoffice runs are sufficient for most installations.
However, the daily digest of email notifications is handled by the
backoffice.  If a Fossil server can sometimes go more than a day without
being accessed, then the automatic backoffice will never run, and the
daily digest might not go out until somebody does visit a webpage.
If this is a problem, an administrator can set up a cron job to
periodically run:

>   fossil backoffice _REPOSITORY_

That command will cause backoffice processing to occur immediately.
Note that this is almost never necessary for an internet-facing
Fossil repository, since most repositories will get multiple accesses

   cryptography. Each block contains a cryptographic hash of the previous
   block, a timestamp, and transaction data..." [(1)][]


By that definition, Fossil is clearly an implementation of blockchain.
The blocks are ["manifests" artifacts](./fileformat.wiki#manifest).
Each manifest has a SHA1 or SHA3 hash of its parent or parents,
a timestamp, and other transactional data.  The repository grows by
add new manifests onto the list.

Some people have come to associate blockchain with cryptocurrency, however,
and since Fossil has nothing to do with cryptocurrency, the claim that
Fossil is build around blockchain is met with skepticism.  The key thing
to note here is that cryptocurrency implementations like BitCoin are
built around blockchain, but they are not synonymous with blockchain.

    the user doing the check-in has no network connection at the moment
    of the commit, Fossil has no way of knowing that it is creating a
    fork until the two repositories are later sync'd.</p></li>

    <li><p id="dist-clone">By Fossil when the cloning hierarchy is more
    than 2 levels deep.
    <br><br>
    [./sync.wiki|Fossil's synchronization protocol] is a two-party
    negotiation; syncs don't automatically propagate up the clone tree
    beyond that. Because of that, if you have a master repository and
    Alice clones it, then Bobby clones from Alice's repository, a
    check-in by Bobby that autosyncs with Alice's repo will <i>not</i>
    also autosync with the master repo. The master doesn't get a copy of
    Bobby's checkin until Alice <i>separately</i> syncs with the master.
    If Carol cloned from the master repo and checks something in that

<li><p><i>Unix without running "configure"</i> → if you prefer to avoid
running configure, you can also use: <b>make -f Makefile.classic</b>.  You may
want to make minor edits to Makefile.classic to configure the build for your
system.

<li><p><i>MinGW 3.x (<u>not</u> 4.x) / MinGW-w64</i> → Use the MinGW makefile:
"<b>make -f win/Makefile.mingw</b>".  On a Windows box you will need either
Cygwin or MSYS as build environment. On Cygwin, Linux or Darwin you may want
to make minor edits to win/Makefile.mingw to configure the cross-compile
environment.

To enable the native [./th1.md#tclEval | Tcl integration feature], use a
command line like the following (all on one line):

<b>make -f win/Makefile.mingw FOSSIL_ENABLE_TCL=1 FOSSIL_ENABLE_TCL_STUBS=1 FOSSIL_ENABLE_TCL_PRIVATE_STUBS=1</b>

animation is disabled by setting the attribute to `"0"`.

TH1 Variables
-------------

Before expanding the TH1 within the header and footer, Fossil first
initializes a number of TH1 variables to values that depend on
repository settings and the specific page being generated.

   *   **project_name** - The project_name variable is filled with the
       name of the project as configured under the Admin/Configuration
       menu.

   *   **project_description** - The project_description variable is
       filled with the description of the project as configured under

</pre></blockquote>
A setting of 1 or greater
prevents fossil from trying to remember the previous sync password.
<blockquote><pre>
export FOSSIL_SECURITY_LEVEL=2
</pre></blockquote>
A setting of 2 or greater
causes all password prompts to be preceded by a random translation matrix similar
to the following:
<blockquote><pre>
abcde fghij klmno pqrst uvwyz
qresw gjymu dpcoa fhkzv inlbt
</pre></blockquote>
When entering the password, the user must substitute the letter on the second
line that corresponds to the letter on the first line.  Uppercase substitutes

variable found in the environment from the list `FOSSIL_HOME`,
`LOCALAPPDATA` (Windows), `APPDATA` (Windows), `HOMEDRIVE` and
`HOMEPATH` (Windows, used together), and `HOME` is used as the
location of the `~/.fossil` file.


`FOSSIL_USE_SEE_TEXTKEY`: If set, treat the encryption key string for
SEE as text to be hashed into the actual encryption key.  This has no
effect if Fossil was not compiled with SEE support enabled.


`FOSSIL_USER`: Name of the default user account if the checkout, local
or global `default-user` setting is not present. The first environment
variable found in the environment from the list `FOSSIL_USER`, `USER`,
`LOGNAME`, and `USERNAME` is the user name. If none of those are set,

are looking at some historical check-in then you cannot ask "What came
next?" or "What are the children of this check-in?"

Fossil, on the other hand, parses essential information about check-ins
(parents, children, committers, comments, files changed, etc.)
into a relational database that can be easily
queried using concise SQL statements to find both ancestors and
descendants of a check-in.

Leaf check-ins in Git that lack a "ref" become "detached," making them
difficult to locate and subject to garbage collection. This
[http://gitfaq.org/articles/what-is-a-detached-head.html|detached head
state] problem has caused untold grief for countless Git users. With
Fossil, all check-ins are easily located via multiple possible paths,
so that detached heads are simply not possible in Fossil.

    of a guy in a room]."</p></li>

    <li><p><b>Branch names sync:</b> Unlike in Git, branch names in
    Fossil are not purely local labels. They sync along with everything
    else, so everyone sees the same set of branch names. Fossil's design
    choice here is a direct reflection of the Linux vs. SQLite project
    outlook: SQLite's developers collaborate closely on a single
    coherent project, whereas Linux's developers go off on tangents and
    occasionally sync changes up with each other.</p></li>

    <li><p><b>Private branches are rare:</b>
    [/doc/trunk/www/private.wiki|Private branches exist in Fossil], but
    they're normally used to handle rare exception cases, whereas in
    many Git projects, they're part of the straight-line development
    process.</p></li>

For example, the default "sync" behavior in Git is to only sync
a single branch, whereas with Fossil the only sync option it to
sync the entire DAG.  Git commands,
GitHub, and GitLab tend to show only a single branch at
a time, whereas Fossil usually shows all parallel branches at
once.  Git has commands like "rebase" that help keep all relevant
changes on a single branch, whereas Fossil encourages a style of
many concurrent branches constantly springing into existence,
undergoing active development in parallel for a few days or weeks, then
merging back into the main line and disappearing.

This difference in emphasis arises from the different purposes of
the two systems.  Git focuses on individual branches, because that
is exactly what you want for a highly-distributed bazaar-style project
such as Linux.  Linus Torvalds does not want to see every check-in

[./hashpolicy.wiki|full backwards compatibility] to old SHA-1 based
repositories.

Here in mid-2019, that feature is now in every OS and package repository
known to include Fossil so that the next release as of this writing
(Fossil 2.10) will default to enforcing SHA-3 hashes by default. This
not only solves the SHAttered problem, it should prevent a reoccurrence
for the foreseeable future. Only repositories created before the
transition to Fossil 2 are still using SHA-1, and then only if the
repository's maintainer chose not to switch them into SHA-3 mode some
time over the past 2 years.

Meanwhile, the Git community took until August 2018 to announce
[https://git-scm.com/docs/hash-function-transition/2.18.0|their plan]
for solving the same problem by moving to SHA-256 (a variant of the
[https://en.wikipedia.org/wiki/SHA-2|older SHA-2 algorithm]) and until
February 2019 to release a version containing the change. It's looking
like this will take years more to percolate through the community.

The practical impact of SHAttered on structured data stores like the one
in Git and Fossil isn't clear, but you want to have your repositories
moved over to a stronger hash algorithm before someone figures out how
to make use of the weaknesses in the old one.  Fossil's developers moved
on this problem quickly and had a widely-deployed solution to it years
ago.

<h3 id="missing-in-fossil">3.2 Features found in Git but missing from Fossil</h3>

  *  <b>Rebase</b>

   Because of its emphasis on recording history exactly as it happened,
   rather than as we would have liked it to happen, Fossil deliberately
   does not provide a "rebase" command.  One can rebase manually in Fossil,
   with sufficient perseverance, but it is not something that can be done with
   a single command.

  *  <b>Push or pull a single branch</b>

   The [/help?cmd=push|fossil push], [/help?cmd=pull|fossil pull], and
   [/help?cmd=sync|fossil sync] commands do not provide the capability to
   push or pull individual branches.  Pushing and pulling in Fossil is

    [https://en.wikipedia.org/wiki/Z/OS|z/OS], for example, though it
    should be quite possible.

    <li><p>Both Fossil and Git support
    [https://en.wikipedia.org/wiki/Patch_(Unix)|<tt>patch(1)</tt>
    files], a common way to allow drive-by contributions, but it's a
    lossy contribution path for both systems. Unlike Git PRs and Fossil
    bundles, patch files collapse multiple checkins together, they don't
    include check-in comments, and they cannot encode changes made above
    the individual file content layer: you lose branching decisions,
    tag changes, file renames, and more when using patch files.</p></li>
</ol></i></small>

name, then continue to use the older SHA1 name.  Use SHA3 for new
artifacts that have never before been encountered.</td>
</tr>
<tr>
<td valign='top'>sha3-only</td>
<td>Name new artifacts using the SHA3 hash algorithm even if the artifact
already has a SHA1 name.  In other words, force the use of SHA3.  This can
cause some artifacts to be added to the repository twice, once under their
SHA1 name and again under their SHA3 name.  But delta compression will
prevent that from causing repository size problems.</td>
</tr>
<tr>
<td valign='top'>shun-sha1</td>
<td>Like "sha3-only" but at this level do not accept a push of SHA1-named
artifacts.  If another Fossil instance tries to push a SHA1-named artifact,

We chose to use Jupyter for this because it makes it easy for you to
modify the notebook to try different things.  Want to see how the
results change with a different image size?  Easy, change the `size`
value in the second cell of the notebook.  Want to try more image
formats?  You can put anything ImageMagick can recognize into the
`formats` list. Want to find the break-even point for images like those
in your own repository?  Easily done with a small amount of code.

[im]:  https://www.imagemagick.org/
[jp]:  https://jupyter.org/
[nbd]: ./image-format-vs-repo-size.ipynb
[nbp]: https://nbviewer.jupyter.org/urls/fossil-scm.org/fossil/doc/trunk/www/image-format-vs-repo-size.ipynb
[wp]:  http://wand-py.org/

build system might include some kind of bootstrapping or
auto-configuration step that you could attach this to, so that it
doesn’t need to be run by hand.

This `Makefile` illustrates two primary strategies:


### Input and Output File Formats Differ by Extension

In the case of SVG and the bitmap image formats, the file name extension
differs between the cases, so we can use `make` suffix rules to get the
behavior we want.  The top half of the `Makefile` just tells `make` how
to map from `*.svg` to `*.svgz` and vice versa, and the same for `*.bmp`
to/from `*.png`.

3.  We're using *uncompressed* TIFF here, not [LZW][lzw]- or
    Zip-compressed TIFF, either of which would give similar results to
    PNG, which is always zlib-compressed.

4.  The raw data changes somewhat from one run to the next due to the
    use of random noise in the image to make the zlib/PNG compression
    more difficult, and the random pixel changes.  Those test design
    choices make this a [Monte Carlo experiment][mce].  We’ve found that
    the overall character of the results doesn’t change from one run to
    the next.

    The code in the notebook’s third cell drops the first three columns
    of data because the first column (the empty repository size) is
    boring, and the subsequent two checkins show the SQLite DB file
    format settling in with its first few checkins. There’s a single

Git only supports version control. The additional features of Fossil such
as Wiki, Tickets, Technotes, and the Forum are not supported in Git and
so those features are not included in an export.

## (2) Cherrypick Merges

The Git client supports cherrypick merges but does not remember them.
In other words, Git does not record a history of cherrypick merges
in its blockchain.

Fossil tracks cherrypick merges in its blockchain and display cherrypicks
(as dashed lines) in its timeline ([example](/timeline?c=0a9f12ce6655b7a5)).
But history information of cherrypicks cannot be exported to Git because
there is no way to represent it in the Git.

## (3) Named Branches

Git has only limited support for named branches.  Git identifies the head
check-in of each branch.  Depending on the check-in graph topology, this
is sufficient to infer the branch for many historical check-ins as well.
However, complex histories with lots of cross-merging
can lead to ambiguities.  Fossil keeps
track of historical branch names unambiguously. 
But the extra details about branch names that Fossil keeps
at hand cannot be exported to Git.

## (4) Non-unique Tags

Git requires tags to be unique.  Each tag must refer to exactly one
check-in.  Fossil does not have this restriction, and so it is common

hashes the password and compares it against the value stored in USER.PW.
If they match, the server sets a cookie on the client to record the
login.  This cookie contains a large amount of high-quality randomness
and is thus intractable to guess.  The value of the cookie and the IP
address of the client is stored in the USER.COOKIE and USER.IPADDR fields
of the USER table on the server.
The USER.CEXPIRE field holds an expiration date
for the cookie, encoded as a Julian day number.  On all subsequent
HTTP requests, the cookie value is matched against the USER table to
enable access to the repository.

A login cookie will only work if the IP address matches.  This feature
is designed to make it more difficult for an attacker to sniff the cookie
and take over the connection.  A cookie-sniffing attack will only work
if the attacker is able to send and receive from the same IP address as

that responds.  Nothing more.</p>

<h4>2.0.1 Encrypted Transport</h4>

<p>In the current implementation of Fossil, the server only
understands HTTP requests.  The client can send either
clear-text HTTP requests or encrypted HTTPS requests.  But when
HTTPS requests are sent, they first must be decrypted by a web server
or proxy before being passed to the Fossil server.  This limitation
may be relaxed in a future release.</p>

<h3>2.1 Server Identification</h3>

<p>The server is identified by a URL argument that accompanies the
push, pull, or sync command on the client.  (As a convenience to

holds.  The client will use this information to figure out which
unversioned files need to be synchronized.
The server might also send a uvigot card when it receives a uvgimme card
but its reply message size is already oversized and hence unable to hold
the usual uvfile reply.

<p>When a client receives a "uvigot" card, it checks to see if the
file needs to be transferred from client to server or from server to client.
If a client-to-server transmission is needed, the client schedules that
transfer to occur on a subsequent HTTP request.  If a server-to-client
transfer is needed, then the client sends a "uvgimme" card back to the
server to request the file content.

<h3>3.7 Gimme Cards</h3>

content on the server exactly matches the content on the client and
no further synchronization is required.

<li><p><b>uv-pull-only</b></i>
<p>A server sends the uv-pull-only pragma to the client in response
to a uv-hash pragma with a mismatched content hash argument.  This
pragma indicates that there are differences in unversioned content
between the client and server but that content can only be transferred
from server to client.  The server is unwilling to accept content from
the client because the client login lacks the "write-unversioned"
permission.</p>

<li><p><b>uv-push-ok</b></i>
<p>A server sends the uv-push-ok pragma to the client in response
to a uv-hash pragma with a mismatched content hash argument.  This
pragma indicates that there are differences in unversioned content
between the client and server and that content can only be transferred
in either direction.  The server is willing to accept content from
the client because the client login has the "write-unversioned"
permission.</p>

<li><p><b>ci-lock</b> <i>CHECKIN-HASH CLIENT-ID</i></p>
<p>A client sends the "ci-lock" pragma to the server to indicate
that it is about to add a new check-in as a child of the

    challenge used by the Let’s Encrypt service only runs over HTTP. This is
    not only because it has to work before HTTPS is first configured,
    but also because it might need to work after a certificate is
    accidentally allowed to lapse, to get that server back into a state
    where it can speak HTTPS safely again.

So, from the second `service { }` block, we include this file to set up
the minimal HTTP service we require, `local/http-certbot-only`:

      listen 80;
      listen [::]:80;

      # This is expressed as a rewrite rule instead of an "if" because
      # http://wiki.nginx.org/IfIsEvil
      #rewrite ^(/.well-known/acme-challenge/.*) $1 break;

Then in `local/http-certbot-only` say:

      root /var/www/$host;
      access_log /var/log/nginx/$host-http-access.log;
       error_log /var/log/nginx/$host-http-error.log;

Sadly, nginx doesn’t allow variable substitution into these particular
directives. As I understand it, allowing that would make nginx slower,
so we must largely repeat these directives in each HTTP `server { }`
block.

These configurations are, as shown, as small as I know how to get them.
If you know of a way to reduce some of this repetition, [I solicit your
advice][fd].


## Step 3: Dry Run

We want to first request a dry run, because Let’s Encrypt puts some
rather low limits on how often you’re allowed to request an actual

<title>Unversioned Content</title>
<h1 align="center">Unversioned Content</h1>

"Unversioned content" or "unversioned files" are
files stored in a Fossil repository without history.
Only the newest version of each unversioned file is retained.

Though history is omitted, unversioned content is synced between
repositories.  In the event of a conflict during a sync, the most recent
version of each unversioned file is retained and older versions are discarded.

Unversioned files are useful for storing ephemeral content such as builds
or frequently changing web pages.
The [https://www.fossil-scm.org/fossil/uv/download.html|download] page of
the self-hosting Fossil repository is stored as unversioned
content, for example.

<h2>Accessing Unversioned Files</h2>

Unversioned files are <u>not</u> a part of a check-out.
Unversioned files are intended to be accessible as web pages using
URLs of the form:  "http://domain/cgi-script/<b>uv</b>/<i>FILENAME</i>".
In other words, the URI method "<b>uv</b>" (short for "unversioned")
followed by the name of the unversioned file will retrieve the content
of the file.  The MIME type is inferred from the filename suffix.

The content of unversioned files can also be retrieved using the
[/help?cmd=unversioned|fossil unvers cat <i>FILENAME</i>] command.

A list of all unversioned files on a server can be seen using
the [/help?cmd=/uvlist|/uvlist] URL.  ([/uvlist|example]).

<h2>Syncing Unversioned Files</h2>

Unversioned content is synced between repositories, though not by default.
Special commands or command-line options are required.
Unversioned content can be synced using the following commands:

<blockquote><pre>
fossil sync <b>-u</b>
fossil clone <b>-u</b> <i>URL local-repo-name</i>
fossil unversioned sync