Fossil

<title>How CGI Works In Fossil</title>
<h2>Introduction</h2><blockquote>
<p>CGI or "Common Gateway Interface" is a venerable yet reliable technique for
generating dynamic web content.  This article gives a quick background on how
CGI works and describes how Fossil can act as a CGI service.

<p>This is a "how it works" guide.  This document provides background
information on the CGI protocol so that you can better understand what
is going on behind the scenes.  If you just want to set up Fossil
as a CGI server, see the [./server/ | Fossil Server Setup] page.  Or
if you want to development CGI-based extensions to Fossil, see
the [./serverext.wiki|CGI Server Extensions] page.
</blockquote>
<h2>A Quick Review Of CGI</h2><blockquote>
<p>
An HTTP request is a block of text that is sent by a client application
(usually a web browser) and arrives at the web server over a network
connection.  The HTTP request contains a URL that describes the information
being requested.  The URL in the HTTP request is typically the same URL
that appears in the URL bar at the top of the web browser that is making
the request.  The URL might contain a "?" character followed
query parameters.  The HTTP will usually also contain other information
such as the name of the application that made the request, whether or
not the requesting application can accept a compressed reply, POST
parameters from forms, and so forth.
<p>
The job of the web server is to interpret the HTTP request and formulate
an appropriate reply.
The web server is free to interpret the HTTP request in any way it wants.
But most web servers follow a similar pattern, described below.
(Note: details may vary from one web server to another.)
<p>
Suppose the filename component of the URL in the HTTP request looks like this:
<blockquote><b>/one/two/timeline/four</b></blockquote>
Most web servers will search their content area for files that match
some prefix of the URL.  The search starts with <b>/one</b>, then goes to
<b>/one/two</b>, then <b>/one/two/timeline</b>, and finally
<b>/one/two/timeline/four</b> is checked.  The search stops at the first
match.
<p>
Suppose the first match is <b>/one/two</b>.  If <b>/one/two</b> is an
ordinary file in the content area, then that file is returned as static
content.  The "<b>/timeline/four</b>" suffix is silently ignored.
<p>
If <b>/one/two</b> is a CGI script (or program), then the web server
executes the <b>/one/two</b> script.  The output generated by
the script is collected and repackaged as the HTTP reply.
<p>
Before executing the CGI script, the web server will set up various
environment variables with information useful to the CGI script:
<table border=1 cellpadding=5>
<tr><th>Environment<br>Variable<th>Meaning
<tr><td>GATEWAY_INTERFACE<td>Always set to "CGI/1.0"
<tr><td>REQUEST_URI
    <td>The input URL from the HTTP request.
<tr><td>SCRIPT_NAME
    <td>The prefix of the input URL that matches the CGI script name.
    In this example: "/one/two".
<tr><td>PATH_INFO
    <td>The suffix of the URL beyond the name of the CGI script.
    In this example: "timeline/four".
<tr><td>QUERY_STRING
    <td>The query string that follows the "?" in the URL, if there is one.
</table>
<p>
There are other CGI environment variables beyond those listed above.
Many Fossil servers implement the
[https://fossil-scm.org/home/test_env/two/three?abc=xyz|test_env]
webpage that shows some of the CGI environment
variables that Fossil pays attention to.
<p>
In addition to setting various CGI environment variables, if the HTTP
request contains POST content, then the web server relays the POST content
to standard input of the CGI script.
<p>
In summary, the task of the
CGI script is to read the various CGI environment variables and
the POST content on standard input (if any), figure out an appropriate
reply, then write that reply on standard output.
The web server will read the output from the CGI script, reformat it
into an appropriate HTTP reply, and relay the result back to the
requesting application.
The CGI script exits as soon as it generates a single reply.
The web server will (usually) persist and handle multiple HTTP requests,
but a CGI script handles just one HTTP request and then exits.
<p>
The above is a rough outline of how CGI works.
There are many details omitted from this brief discussion.
See other on-line CGI tutorials for further information.
</blockquote>
<h2>How Fossil Acts As A CGI Program</h2>
<blockquote>
An appropriate CGI script for running Fossil will look something

for this script.  On unix, when you execute a script that starts with
a shebang, the operating system runs the program identified by the
shebang with a single argument that is the full pathname of the script
itself.
In our example, the interpreter is Fossil, and the argument might
be something like "/var/www/cgi-bin/one/two" (depending on how your
particular web server is configured).
<p>
The Fossil program that is run as the script interpreter
is the same Fossil that runs when
you type ordinary Fossil commands like "fossil sync" or "fossil commit".
But in this case, as soon as it launches, the Fossil program
recognizes that the GATEWAY_INTERFACE environment variable is
set to "CGI/1.0" and it therefore knows that it is being used as
CGI rather than as an ordinary command-line tool, and behaves accordingly.
<p>
When Fossil recognizes that it is being run as CGI, it opens and reads
the file identified by its sole argument (the file named by
<code>argv&#91;1&#93;</code>).  In our example, the second line of that file
tells Fossil the location of the repository it will be serving.
Fossil then starts looking at the CGI environment variables to figure
out what web page is being requested, generates that one web page,
then exits.
<p>
Usually, the webpage being requested is the first term of the
PATH_INFO environment variable.  (Exceptions to this rule are noted
in the sequel.)  For our example, the first term of PATH_INFO
is "timeline", which means that Fossil will generate
the [/help?cmd=/timeline|/timeline] webpage.
<p>
With Fossil, terms of PATH_INFO beyond the webpage name are converted into
the "name" query parameter.  Hence, the following two URLs mean
exactly the same thing to Fossil:
<ol type='A'>
<li> [https://fossil-scm.org/home/info/c14ecc43]
<li> [https://fossil-scm.org/home/info?name=c14ecc43]
</ol>

<blockquote>
The previous example showed how to serve a single Fossil repository
using a single CGI script.
On a website that wants to serve multiple repositories, one could
simply create multiple CGI scripts, one script for each repository.
But it is also possible to serve multiple Fossil repositories from
a single CGI script.
<p>
If the CGI script for Fossil contains a "directory:" line instead of
a "repository:" line, then the argument to "directory:" is the name
of a directory that contains multiple repository files, each ending
with ".fossil".  For example:
<blockquote><pre>
#!/usr/bin/fossil
directory: /home/www/repos

             \_________/\____________/\____________________/ \______/
                  |            |                 |               |
              HTTP_HOST   SCRIPT_NAME        PATH_INFO      QUERY_STRING
</pre>
</blockquote>
<h2>Additional CGI Script Options</h2>
<blockquote>
<p>
The CGI script can have additional options used to fine-tune
Fossil's behavior.  See the [./cgi.wiki|CGI script documentation]
for details.
</p>
</blockquote>
<h2>Additional Observations</h2>
<blockquote><ol type="I">
<li><p>
Fossil does not distinguish between the various HTTP methods (GET, PUT,
DELETE, etc).  Fossil figures out what it needs to do purely from the
webpage term of the URI.
<li><p>
Fossil does not distinguish between query parameters that are part of the
URI, application/x-www-form-urlencoded or multipart/form-data encoded
parameter that are part of the POST content, and cookies.  Each information
source is seen as a space of key/value pairs which are loaded into an
internal property hash table.  The code that runs to generate the reply
can then reference various properties values.
Fossil does not care where the value of each property comes from (POST
content, cookies, or query parameters) only that the property exists
and has a value.
<li><p>
The "[/help?cmd=ui|fossil ui]" and "[/help?cmd=server|fossil server]" commands
are implemented using a simple built-in web server that accepts incoming HTTP
requests, translates each request into a CGI invocation, then creates a
separate child Fossil process to handle each request.  In other words, CGI
is used internally to implement "fossil ui/server".
<p>
SCGI is processed using the same built-in web server, just modified
to parse SCGI requests instead of HTTP requests.  Each SCGI request is
converted into CGI, then Fossil creates a separate child Fossil
process to handle each CGI request.
<li><p>
Fossil is itself often launched using CGI.  But Fossil can also then
turn around and launch [./serverext.wiki|sub-CGI scripts to implement
extensions].
</ol>
</blockquote>

    claims to be a $100 bill.

*   **Type 2** is creation of new fraudulent currency that will pass
    in commerce.  To extend our analogy, it is the creation of new
    US $10 bills. There are two sub-types to this fraud. In terms of
    our analogy, they are:

    *  **Type 2a**: copying an existing legitimate $10 bill<p>

    *  **Type 2b**: printing a new $10 bill that is unlike an existing
       legitimate one, yet which will still pass in commerce

*   **Type 3** is double-spending existing legitimate cryptocurrency.
    There is no analogy in paper money due to its physical form; it is a
    problem unique to digital currency due to its infinitely-copyable

<title>Compiling and Installing Fossil</title>

<h2>0.0 Using A Pre-compiled Binary</h2>

<p>[/uv/download.html|Pre-compiled binaries] are available for recent
releases. Just download
the appropriate executable for your platform
and put it on your $PATH.
To uninstall, simply delete the executable.
To upgrade from an older release, just overwrite the older binary with
the newer one.</p>

<p>For details about how those binaries are built, see
[/wiki?name=Release+Build+How-To | the Release Build How-To wiki page].</p>


<h2>0.1 Executive Summary</h2>

<p>Building and installing is very simple.  Three steps:</p>

<ol>
<li> Download and unpack a source tarball or ZIP.
<li> <b>./configure; make</b>
<li> Move the resulting "fossil" or "fossil.exe" executable to someplace on
your $PATH.
</ol>

<p><hr>

<h2>1.0 Obtaining The Source Code</h2>

<p>Fossil is self-hosting, so you can obtain a ZIP archive or tarball
containing a snapshot of the <em>latest</em> version directly from
Fossil's own fossil repository. Additionally, source archives of
<em>released</em> versions of
fossil are available from the [/uv/download.html|downloads page].
To obtain a development version of fossil, follow these steps:</p>

<ol>
<li><p>Point your web browser to [https://fossil-scm.org/]</li>

<li><p>Click on the [/timeline|Timeline]
link at the top of the page.</p></li>

<li><p>Select a version of of Fossil you want to download.  The latest
version on the trunk branch is usually a good choice.  Click on its
link.</p></li>

<li><p>Finally, click on one of the
"Zip Archive" or "Tarball" links, according to your preference.
These link will build a ZIP archive or a gzip-compressed tarball of the
complete source code and download it to your computer.
</ol>

<h2>Aside: Is it really safe to use an unreleased development version of
the Fossil source code?</h2>

Yes!  Any check-in on the
[/timeline?t=trunk | trunk branch] of the Fossil

rendering this page.  It is always safe to use whatever version
of the Fossil code you find running on the main Fossil website.

<h2>2.0 Compiling</h2>

<ol>
<li value="5">
<p>Unpack the ZIP or tarball you downloaded then
<b>cd</b> into the directory created.</p></li>

<li><i>(Optional, Debian-compatible Linux only)</i>
Make sure you have all the necessary tools and libraries at hand by running:
<b>sudo apt install tcl-dev tk libssl-dev zlib1g-dev</b>.

<li><i>(Optional, Unix only)</i>
Run <b>./configure</b> to construct a makefile.

<ol type="a">
<li><p>
The build system for Fossil on Unix-like systems assumes that the
OpenSSL development and runtime files are available on your system,
because unprotected repositories are trivial to attack otherwise.
Indeed, some public Fossil repositories — including Fossil's own — today
run in an HTTPS-only mode, so that you can't even do an anonymous clone
from them without using the TLS features added to Fossil by OpenSSL. To
weaken that stance could allow a
[https://en.wikipedia.org/wiki/Man-in-the-middle_attack|man in the
middle attack], such as one that substitutes malicious code into your
Fossil repository clone.</p>

<p>You can force the Fossil build system to avoid searching for, building
against, and linking to the OpenSSL library by passing
<b>--with-openssl=none</b> to the <tt>configure</tt> script.</p>

<p>If you do not have the OpenSSL development libraries on your system,
we recommend that you install them, typically via your OS's package
manager. The Fossil build system goes to a lot of effort to seek these
out wherever they may be found, so that is typically all you need to
do.</p>

<p>For more advanced use cases, see the [./ssl.wiki#openssl-bin|OpenSSL
discussion in the "TLS and Fossil" document].</p>


<li><p>
To build a statically linked binary, you can <i>try</i> adding
the <b>--static</b> option, but
[https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead
| it may well not work]. If your platform of choice is affected by this,
the simplest workaround we're aware of is to build a Fossil container,
then [./containers.md#static | extract the static executable from it].


<li><p>
To enable the native [./th1.md#tclEval | Tcl integration feature] feature,
add the <b>--with-tcl=1</b> and <b>--with-tcl-private-stubs=1</b> options.


<li><p>
Other configuration options can be seen by running
<b>./configure --help</b>

</ol>

<li><p>Run "<b>make</b>" to build the "fossil" or "fossil.exe" executable.
The details depend on your platform and compiler.

<ol type="a">
<li><p><i>Unix</i> → the configure-generated Makefile should work on
all Unix and Unix-like systems.  Simply type "<b>make</b>".

<li><p><i>Unix without running "configure"</i> → if you prefer to avoid
running configure, you can also use: <b>make -f Makefile.classic</b>.  You may
want to make minor edits to Makefile.classic to configure the build for your
system.

<li><p><i>MinGW / MinGW-w64</i> → The best-supported path is to build
via the MinGW specific Makefile under a POSIX build of GNU make:
"<b>make -f win/Makefile.mingw</b>".

There is limited support for building under MinGW's native Windows port
of GNU Make instead by defining the <tt>USE_WINDOWS=1</tt> variable, but
it's better to build under MSYS, Cygwin, or WSL on Windows since this
mode doesn't take care of cases such as the "openssl" target, which
depends on <tt>sed</tt>. We've gone as far down this path as is
practical short of breaking cross-compilation under Linux, macOS, and so

<b>make -f win/Makefile.mingw FOSSIL_ENABLE_TCL=1 FOSSIL_ENABLE_TCL_STUBS=1 FOSSIL_ENABLE_TCL_PRIVATE_STUBS=1</b>

Alternatively, running <b>./configure</b> under MSYS should give a
suitable top-level Makefile. However, options passed to configure that are
not applicable on Windows may cause the configuration or compilation to fail
(e.g. fusefs, internal-sqlite, etc).

<li><p><i>MSVC</i> → Use the MSVC makefile.  

Run all of the following from a "x64 Native Tools Command Prompt". 

First
change to the "win/" subdirectory ("<b>cd win</b>") then run
"<b>nmake /f Makefile.msc</b>".<br><br>Alternatively, the batch
file "<b>win\buildmsvc.bat</b>" may be used and it will attempt to

<blockquote><pre>
nmake /f Makefile.msc FOSSIL_ENABLE_TCL=1
</pre></blockquote>
<blockquote><pre>
buildmsvc.bat FOSSIL_ENABLE_TCL=1
</pre></blockquote>

<li><p><i>Cygwin</i> → The same as other Unix-like systems. It is
recommended to configure using: "<b>configure --disable-internal-sqlite</b>",
making sure you have the "libsqlite3-devel" , "zlib-devel" and
"openssl-devel" packages installed first.
</ol>
</ol>

<h2>3.0 Installing</h2>

<ol>
<li value="9">
<p>The finished binary is named "fossil" (or "fossil.exe" on Windows).
Put this binary in a
directory that is somewhere on your PATH environment variable.
It does not matter where.</p>


<li>
<p><b>(Optional:)</b>
To uninstall, just delete the binary.</p>

</ol>

<h2>4.0 Additional Considerations</h2>

<ul>
<li><p>
  If the makefiles that come with Fossil do not work for
  you, or for some other reason you want to know how to build
  Fossil manually, then refer to the
  [./makefile.wiki | Fossil Build Process] document which describes
  in detail what the makefiles do behind the scenes.


<li><p>
  The fossil executable is self-contained and stand-alone and usually
  requires no special libraries or other software to be installed.  However,
  the "--tk" option to the [/help/diff|diff command] requires that Tcl/Tk
  be installed on the local machine.  You can get Tcl/Tk from
  [http://www.activestate.com/activetcl|ActiveState].


<li><p>
  To build on older Macs (circa 2002, MacOS 10.2) edit the Makefile
  generated by configure to add the following lines:
  <blockquote><pre>
  TCC += -DSQLITE_WITHOUT_ZONEMALLOC
  TCC += -D_BSD_SOURCE
  TCC += -DWITHOUT_ICONV
  TCC += -Dsocketlen_t=int
  TCC += -DSQLITE_MAX_MMAP_SIZE=0
</pre></blockquote>

</ul>


<h2 id="docker" name="oci">5.0 Building a Docker Container</h2>

The information on building Fossil inside an
[https://opencontainers.org/ | OCI container] is now in

of the file and the name of the file as it appears on disk,
and thus serves as a mapping from artifact ID to disk name.  The artifact ID
of the manifest is the identifier for the entire check-in.  When
you look at a "timeline" of changes in Fossil, the ID associated
with each check-in or commit is really just the artifact ID of the
manifest for that check-in.

<p>The manifest file is not normally a real file on disk.  Instead,
the manifest is computed in memory by Fossil whenever it needs it.
However, the "fossil setting manifest on" command will cause the
manifest file to be materialized to disk, if desired.  Both Fossil
itself, and SQLite cause the manifest file to be materialized to disk
so that the makefiles for these project can read the manifest and
embed version information in generated binaries.

<p>Fossil automatically generates a manifest whenever you "commit"
a new check-in.  So this is not something that you, the developer,
need to worry with.  The format of a manifest is intentionally
designed to be simple to parse, so that if
you want to read and interpret a manifest, either by hand or
with a script, that is easy to do.  But you will probably never
need to do so.</p>

<p>In addition to identifying all files in the check-in, a
manifest also contains a check-in comment, the date and time
when the check-in was established, who created the check-in,
and links to other check-ins from which the current check-in
is derived.  There is also a couple of checksums used to verify
the integrity of the check-in.  And the whole manifest might
be PGP clearsigned.</p>

<h3 id="keyconc">2.3 Key concepts</h3>

<ul>
<li>A <b>check-in</b> is a set of files arranged
    in a hierarchy.</li>
<li>A <b>repository</b> keeps a record of historical check-ins.</li>

</ol>

<h2>5.0 Setting Up A Fossil Server</h2>

With other configuration management software, setting up a server is
a lot of work and normally takes time, patience, and a lot of system
knowledge.  Fossil is designed to avoid this frustration.  Setting up
a server with Fossil is ridiculously easy.  You have four options:</p>

<ol>
<li><p><b>Stand-alone server.</b>
Simply run the [/help?cmd=server|fossil server] or
[/help?cmd=ui|fossil ui] command from the command-line.

<li><p><b>CGI.</b>
Install a 2-line CGI script on a CGI-enabled web-server like Apache.

<li><p><b>SCGI.</b>
Start an SCGI server using the
[/help?cmd=server| fossil server --scgi] command for handling
SCGI requests from web-servers like Nginx.

<li><p><b>Inetd or Stunnel.</b>
Configure programs like inetd, xinetd, or stunnel to hand off HTTP requests
directly to the [/help?cmd=http|fossil http] command.
</ol>

See the [./server/ | How To Configure A Fossil Server] document
for details.

<h2>6.0 Review Of Key Concepts</h2>

<ul>

<title>Customizing The Ticket System</title>
<nowiki>
<h2>Introduction</h2>
<p>
This guide will explain how to add the "assigned_to" and "submitted_by" fields
to the ticket system in Fossil, as well as making the system more useful.  You
must have "admin" access to the repository to implement these instructions.
</p>

<h2>First modify the TICKET table</h2><blockquote>
<p>

Click on the "Admin" menu, then "Tickets", then "Table".  After the other fields
and before the final ")", insert:
<pre>
  ,
  assigned_to TEXT,
  opened_by TEXT
</pre>
And "Apply Changes".  You have just added two more fields to the ticket
database!  NOTE: I won't tell you to "Apply Changes" after each step from here
on out.  Now, how do you use these fields?
</p>
</blockquote>

<h2>Next add assignees</h2><blockquote>
<p>

Back to the "Tickets" admin page, and click "Common".  Add something like this:
<pre>
set assigned_choices {
  unassigned
  tom
  dick
  harriet
}
</pre>
Obviously, choose names corresponding to the logins on your system.  The
'unassigned' entry is important, as it prevents you from having a NULL in that
field (which causes problems later when editing).
</p>
</blockquote>

<h2>Now modify the 'new ticket' page</h2><blockquote>
<p>

Back to the "Tickets" admin page, and click "New Ticket Page".  This is a little
more tricky.  Edit the top part:
<pre>
  if {[info exists submit]} {
     set status Open
     set opened_by $login
	 set assigned_to "unassigned"

questions.</td>
</tr>
<th1>enable_output 1</th1>
</pre>
This bit of code will get rid of the "email" field entry for logged-in users.
Since we know the user's information, we don't have to ask for it. NOTE: it
might be good to automatically scoop up the user's email and put it here.
</p>
<p>
You might also want to enable people to actually assign the ticket to a specific
person during creation. For this to work, you need to add the code
for "assigned_to" as shown below under the heading "Modify the 'edit ticket' page".
This will give you an additional combobox where you can choose a person during
ticket creation.
</p>
</blockquote>

<h2>Modify the 'view ticket' page</h2><blockquote>
<p>

Look for the text "Contact:" (about halfway through).  Then insert these lines
after the closing tr tag and before the "enable_output" line:
<pre>
<tr>
  &lt;td align="right">Assigned to:&lt;/td>&lt;td bgcolor="#d0d0d0">
  $&lt;assigned_to>
  &lt;/td>
  &lt;td align="right">Opened by:&lt;/td>&lt;td bgcolor="#d0d0d0">
  $&lt;opened_by>
  &lt;/td>
</pre>
This will add a row which displays these two fields, in the event the user has
<a href="./caps/ref.html#w">ticket "edit" capability</a>.
</p>
</blockquote>

<h2>Modify the 'edit ticket' page</h2><blockquote>
<p>

Before the "Severity:" line, add this:
<pre>
&lt;tr>&lt;td align="right">Assigned to:&lt;/td>&lt;td>
&lt;th1>combobox assigned_to $assigned_choices 1&lt;/th1>
&lt;/td>&lt;/tr>
</pre>
That will give you a drop-down list of assignees. The first argument to the TH1
command 'combobox' is the database field which the combobox is associated to.
The next argument is the list of choices you want to show in the combobox (and
that you specified in the second step above. The last argument should be 1 for a
true combobox (see the <a href="th1.md#combobox">TH1 documentation</a> for
details).</p>

<p>Now, similar to the previous
section, look for "Contact:" and add this:
<pre>
  &lt;tr>&lt;td align="right">Reported by:&lt;/td>&lt;td>
  &lt;input type="text" name="opened_by" size="40"
   value="$&lt;opened_by>">
  &lt;/td>&lt;/tr>
</pre>
</p>
</blockquote>

<h2>What next?</h2><blockquote>
<p>

Now you can add custom reports which select based on the person to whom the
ticket is assigned.  For example, an "Assigned to me" report could be:
<pre>
SELECT
  CASE WHEN status IN ('Open','Verified') THEN '#f2dcdc'
       WHEN status='Review' THEN '#e8e8e8'
       WHEN status='Fixed' THEN '#cfe8bd'
       WHEN status='Tested' THEN '#bde5d6'
       WHEN status='Deferred' THEN '#cacae5'
       ELSE '#c8c8c8' END AS 'bgcolor',
  substr(tkt_uuid,1,10) AS '#',
  datetime(tkt_mtime) AS 'mtime',
  type,
  status,
  subsystem,
  title
FROM ticket
WHERE assigned_to=user()
</pre>
</p>
</blockquote>
</nowiki>

approach is to use one of the existing built-in skins as a baseline and
make incremental modifications, testing after each step, to obtain the
desired result.

The skin is controlled by five files:

<blockquote><dl>
<dt><b>css.txt</b></dt><dd>

<p>The css.txt file is the text of the CSS for Fossil.
Fossil might add additional CSS elements after
the css.txt file, if it sees that the css.txt omits some
CSS components that Fossil needs.  But for the most part,
the content of the css.txt is the CSS for the page.</dd>

<dt><b>details.txt</b><dt><dd>

<p>The details.txt file is short list of settings that control
the look and feel, mostly of the timeline.  The default
details.txt file looks like this:

<blockquote><pre>
pikchr-background:          ""
pikchr-fontscale:           ""
pikchr-foreground:          ""
pikchr-scale:               ""
timeline-arrowheads:        1
timeline-circle-nodes:      1
timeline-color-graph-lines: 1
white-foreground:           0
</pre></blockquote>

The three "timeline-" settings in details.txt control the appearance
of certain aspects of the timeline graph.  The number on the
right is a boolean - "1" to activate the feature and "0" to
disable it.  The "white-foreground:" setting should be set to
"1" if the page color has light-color text on a darker background,
and "0" if the page has dark text on a light-colored background.
<p>
If the "pikchr-foreground" setting (added in Fossil 2.14)
is defined and is not an empty string then it specifies a
foreground color to use for [pikchr diagrams](./pikchr.md).  The
default pikchr foreground color is black, or white if the
"white-foreground" boolean is set.  The "pikchr-background"
settings does the same for the pikchr diagram background color.
If the "pikchr-fontscale" and "pikchr-scale" values are not
empty strings, then they should be floating point values (close
to 1.0) that specify relative scaling of the fonts in pikchr
diagrams and other elements of the diagrams, respectively.
</dd>

<dt><b>footer.txt</b> and <b>header.txt</b></dt><dd>

<p>The footer.txt and header.txt files contain the Content Footer
and Content Header respectively.  Of these, the Content Header is
the most important, as it contains the markup used to generate
the banner and menu bar for each page.

<p>Both the footer.txt and header.txt file are 
[processed using TH1](#headfoot) prior to being output as 
part of the overall web page.</dd>

<dt><b>js.txt</b></dt><dd>

<p>The js.txt file is optional.  It is intended to be javascript.
The complete text of this javascript might be inserted into
the Content Footer, after being processed using TH1, using
code like the following in the "footer.txt" file:

<blockquote><pre>
&lt;script nonce="$nonce"&gt;
  &lt;th1&gt;styleScript&lt;/th1&gt;
&lt;/script&gt;
</pre></blockquote>

<p>The js.txt file was originally used to insert javascript
that controls the hamburger menu in the default skin.  More
recently, the javascript for the hamburger menu was moved into
a separate built-in file.  Skins that use the hamburger menu
typically cause the javascript to be loaded by including the
following TH1 code in the "header.txt" file:

<blockquote><pre>

<title>Fossil Delta Format</title>
<h1>1.0 Overview</h1>

<p>Fossil achieves efficient storage and low-bandwidth synchronization
through the use of delta-compression.  Instead of storing
or transmitting the complete content of an artifact, Fossil stores or
transmits only the changes relative to a related artifact.
</p>

<p>This document describes the delta-encoding format used by Fossil.
The intended audience is developers working on either
<a href="index.wiki">Fossil</a> itself, or on tools compatible with
Fossil. Understanding of this document is <em>not</em> required for
ordinary users of Fossil.  This document is an implementation detail.</p>

<p>This document only describes the delta file format.  A 
[./delta_encoder_algorithm.wiki|separate document] describes one possible
algorithm for generating deltas in this format.</p>

<h2>1.1 Sample Software And Analysis Tools</h2>

<p>The core routines used to generate and apply deltas in this format
are contained in the [../src/delta.c|delta.c] source file.  Interface
logic, including "test-*" commands to directly manipulate deltas are
contained in the [../src/deltacmd.c|deltacmd.c] source file.  SQL functions
to create, apply, and analyze deltas are implemented by code in the
[../src/deltafunc.c|deltafunc.c] source file.

<p>The following command-line tools are available to create and apply
deltas and to test the delta logic:

   *   [/help?cmd=test-delta|fossil test-delta] &rarr; Run self-tests of
       the delta logic

   *   [/help?cmd=test-delta-create|fossil test-delta-create X Y] &rarr; compute
       a delta that converts file X into file Y.  Output that delta.

   *   [/help?cmd=test-delta-apply|fossil test-delta-apply X D] &rarr; apply
       delta D to input file X and output the result.

   *   [/help?cmd=test-delta-analyze|fossil test-delta-analyze X Y] &rarr; compute
       and delta that converts file X into file Y but instead of writing the
       delta to output, write performance information about the delta.

<p>When running the [/help?cmd=sqlite3|fossil sql] command to get an
interactive SQL session connected to the repository, the following
additional SQL functions are provided:

   *   <b>delta_create(</b><i>X</i><b>,</b><i>Y</i><b>)</b> &rarr;
       Compute a data that carries blob X into blob Y and return that delta
       as a blob.

<verbatim type="pikchr">
    leftmargin = 0.1
    box height 50% "Header"
    box same "Segments"
    box same "Trailer"
</verbatim>

<p>A delta consists of three parts, a "header", a "trailer", and a
"segment-list" between them.</p>

<p>Both header and trailer provide information about the target
helping the decoder, and the segment-list describes how the target can
be constructed from the original.</p>

<h2 id="header">2.1 Header</h2>
<verbatim type="pikchr">
    leftmargin = 0.1
    box height 50% "Size"
    box same "\"\\n\""
</verbatim>

<p>The header consists of a single number followed by a newline
character (ASCII 0x0a). The number is the length of the target in
bytes.</p>

<p>This means that, given a delta, the decoder can compute the size of
the target (and allocate any necessary memory based on that) by simply
reading the first line of the delta and decoding the number found
there. In other words, before it has to decode everything else.</p>

<h2 id="trailer">2.2 Trailer</h2>
<verbatim type="pikchr">
    leftmargin = 0.1
    box height 50% "Checksum"
    box same "\";\""
</verbatim>

<p>The trailer consists of a single number followed by a semicolon (ASCII
0x3b). This number is a checksum of the target and can be used by a
decoder to verify that the delta applied correctly, reconstructing the
target from the original.</p>

<p>The checksum is computed by treating the target as a series of
32-bit integer numbers (MSB first), and summing these up, modulo
2^32-1. A target whose length is not a multiple of 4 is padded with
0-bytes (ASCII 0x00) at the end.</p>

<p>By putting this information at the end of the delta a decoder has
it available immediately after the target has been reconstructed
fully.</p>

<h2 id="slist">2.3 Segment-List</h2>
<verbatim type="pikchr">
    leftmargin = 0.1
    PART1: [
        B1: box height 50% width 15% ""
        B2: box same ""

        box "Length" height 50%
        right
        box "\":\"" same
        box "Bytes" same
    ] with .nw at previous.sw
</verbatim>

<p>The segment-list of a delta describes how to create the target from
the original by a combination of inserting literal byte-sequences and
copying ranges of bytes from the original. This is where the
compression takes place, by encoding the large common parts of
original and target in small copy instructions.</p>

<p>The target is constructed from beginning to end, with the data
generated by each instruction appended after the data of all previous
instructions, with no gaps.</p>

<h3 id="insertlit">2.3.1 Insert Literal</h3>

<p>A literal is specified by two elements, the size of the literal in
bytes, and the bytes of the literal itself.</p>

<verbatim type="pikchr">
    leftmargin = 0.1
    box "Length" height 50%
    box "\":\"" same
    box "Bytes" same
</verbatim>

<p>The length is written first, followed by a colon character (ASCII
0x3a), followed by the bytes of the literal.</p>

<h3 id="copyrange">2.3.2 Copy Range</h3>

<p>A range to copy is specified by two numbers, the offset of the
first byte in the original to copy, and the size of the range, in
bytes. The size zero is special, its usage indicates that the range
extends to the end of the original.</p>

<verbatim type="pikchr">
    leftmargin = 0.1
    box "Length" height 50%
    box "\"@\"" same
    box "Offset" same
    box "\",\"" same
</verbatim>


<p>The length is written first, followed by an "at" character (ASCII
0x40), then the offset, followed by a comma (ASCII 0x2c).</p>

<h1 id="intcoding">3.0 Encoding of integers</h1>

<p>
The format currently handles only 32 bit integer numbers. They are
written base-64 encoded, MSB first, and without leading
"0"-characters, except if they are significant (i.e. 0 => "0").
</p>

<p>
The base-64 encoding uses one character for each 6 bits of
the integer to be encoded.  The encoding characters are:
</p>

<blockquote><pre>
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~
</pre></blockquote>

<p>The least significant 6 bits of the integer are encoded by
the first character, followed by
the next 6 bits, and so on until all non-zero bits of the integer
are encoded.  The minimum number of encoding characters is used.
Note that for integers less than 10, the base-64 coding is a
ASCII decimal rendering of the number itself.
</p>

<h1 id="examples">4.0 Examples</h1>

<h2 id="examplesint">4.1 Integer encoding</h2>

<table border=1>
<tr>

<td>-1101438770</td>
<td>2zMM3E</td>
</tr>
</table>

<h2 id="examplesdelta">4.2 Delta encoding</h2>

<p>An example of a delta using the specified encoding is:</p>

<table border=1><tr><td><pre>
1Xb
4E@0,2:thFN@4C,6:scenda1B@Jd,6:scenda5x@Kt,6:pieces79@Qt,F: Example: eskil~E@Y0,2zMM3E;</pre>
</td></tr></table>

<p>This can be taken apart into the following parts:</p>

<table border=1>
<tr><th>What  </th> <th>Encoding         </th><th>Meaning </th><th>Details</th></tr>
<tr><td>Header</td> <td>1Xb              </td><td>Size    </td><td> 6246         </td></tr>
<tr><td>S-List</td> <td>4E@0,            </td><td>Copy    </td><td> 270 @ 0      </td></tr>
<tr><td>&nbsp;</td> <td>2:th             </td><td>Literal </td><td> 2 'th'       </td></tr>
<tr><td>&nbsp;</td> <td>FN@4C,           </td><td>Copy    </td><td> 983 @ 268        </td></tr>
<tr><td>&nbsp;</td> <td>6:scenda         </td><td>Literal </td><td> 6 'scenda'       </td></tr>
<tr><td>&nbsp;</td> <td>1B@Jd,           </td><td>Copy    </td><td> 75 @ 1256        </td></tr>
<tr><td>&nbsp;</td> <td>6:scenda         </td><td>Literal </td><td> 6 'scenda'       </td></tr>
<tr><td>&nbsp;</td> <td>5x@Kt,           </td><td>Copy    </td><td> 380 @ 1336       </td></tr>
<tr><td>&nbsp;</td> <td>6:pieces     </td><td>Literal </td><td> 6 'pieces'       </td></tr>
<tr><td>&nbsp;</td> <td>79@Qt,           </td><td>Copy    </td><td> 457 @ 1720     </td></tr>
<tr><td>&nbsp;</td> <td>F: Example: eskil</td><td>Literal </td><td> 15 ' Example: eskil'</td></tr>
<tr><td>&nbsp;</td> <td>~E@Y0,           </td><td>Copy    </td><td>  4046 @ 2176        </td></tr>
<tr><td>Trailer</td><td>2zMM3E           </td><td>Checksum</td><td> -1101438770         </td></tr>
</table>

<p>The unified diff behind the above delta is</p>

<table border=1><tr><td><pre>
bluepeak:(761) ~/Projects/Tcl/Fossil/Devel/devel > diff -u ../DELTA/old ../DELTA/new
--- ../DELTA/old        2007-08-23 21:14:40.000000000 -0700
+++ ../DELTA/new        2007-08-23 21:14:33.000000000 -0700
@@ -5,7 +5,7 @@

<title>How To Use Encrypted Repositories</title>
<h2>Introduction</h2><blockquote>
Fossil can be compiled so that it works with encrypted repositories using
the [https://www.sqlite.org/see/doc/trunk/www/readme.wiki|SQLite Encryption Extension].
This technical note explains the process.
</blockquote>
<h2>Building An Encryption-Enabled Fossil</h2><blockquote>
The SQLite Encryption Extension (SEE) is proprietary software and requires
[https://sqlite.org/purchase/see|purchasing a license].
<p>
Assuming you have an SEE license, the first step of compiling Fossil to
use SEE is to create an SEE-enabled version of the SQLite database source code.
This alternative SQLite database source file should be called "sqlite3-see.c"
and should be placed in the extsrc/ subfolder of the Fossil sources, right beside
the public-domain "sqlite3.c" source file.  Also make a copy of the SEE-enabled
"shell.c" file, renamed as "shell-see.c", and place it in the extsrc/ subfolder
beside the original "shell.c".
<p>
Add the --with-see command-line option to the configuration script to enable
the use of SEE on unix-like systems.
<blockquote><pre>
./configure --with-see; make
</pre></blockquote>

<p>To build for Windows using MSVC, add
the "USE_SEE=1" argument to the "nmake" command line.
<blockquote><pre>
nmake -f makefile.msc USE_SEE=1
</pre></blockquote>
</blockquote>
<h2>Using Encrypted Repositories</h2><blockquote>
Any Fossil repositories whose filename ends with ".efossil" is taken to be
an encrypted repository.  Fossil will prompt for the encryption password and
attempt to open the repository database using that password.
<p>
Every invocation of fossil on an encrypted repository requires retyping the
encryption password.
To avoid excess password typing, consider using the "fossil shell"
command which prompts for the password just once, then reuses it for each
subsequent Fossil command entered at the prompt.
<p>
On Windows, the "fossil server", "fossil ui", and "fossil shell" commands do not
(currently) work on an encrypted repository.
</blockquote>
<h2>Additional Security</h2><blockquote>
Use the FOSSIL_SECURITY_LEVEL environment for additional protection.
<blockquote><pre>
export FOSSIL_SECURITY_LEVEL=1

#############################################################################
# Code to actually generate the FAQ
#
puts "<title>Fossil FAQ</title>"
puts "<h1 align=\"center\">Frequently Asked Questions</h1>\n"
puts "<p>Note: See also <a href=\"qandc.wiki\">Questions and Criticisms</a>.\n"

puts {<ol>}
for {set i 1} {$i<$cnt} {incr i} {
  puts "<li><a href=\"#q$i\">[lindex $faq($i) 0]</a></li>"
}
puts {</ol>}
puts {<hr>}

<title>Fossil FAQ</title>
<h1 align="center">Frequently Asked Questions</h1>

<p>Note: See also <a href="qandc.wiki">Questions and Criticisms</a>.

<ol>
<li><a href="#q1">What GUIs are available for fossil?</a></li>
<li><a href="#q2">What is the difference between a "branch" and a "fork"?</a></li>
<li><a href="#q3">How do I create a new branch?</a></li>
<li><a href="#q4">How do I tag a check-in?</a></li>
<li><a href="#q5">How do I create a private branch that won't get pushed back to the

<title>Checklist For Successful Open-Source Projects</title>
<nowiki>

<p>This checklist is loosely derived from Tom "Spot" Callaway's Fail Score
blog post <a href="http://spot.livejournal.com/308370.html">
http://spot.livejournal.com/308370.html</a> (see also
<a href="http://www.theopensourceway.org/book/The_Open_Source_Way-How_to_tell_if_a_FLOSS_project_is_doomed_to_FAIL.html">[1]</a>).
Tom's original post assigned point scores to the various elements and
by adding together the individual points, the reader is supposed to be able
to judge the likelihood that the project will fail.
The point scores, and the items on the list, clearly reflect Tom's
biases and are not necessarily those of the larger open-source community.
Nevertheless, the policy of the Fossil shall be to strive for a perfect
score.</p>

<p>This checklist is an inversion of Tom's original post in that it strives to
say what the project should do in order to succeed rather than what it
should not do to avoid failure.  The point values are omitted.</p>

<p>See also:
<ul>
<li><a href="http://offog.org/articles/packaging/">
http://offog.org/articles/packaging/</a>
<li>
<a href="http://www.gnu.org/prep/standards/standards.html#Managing-Releases">
http://www.gnu.org/prep/standards/standards.html#Managing-Releases</a>
</ul>

<hr>

<ol>
<li><p>The source code size is less than 100 MB, uncompressed.

<li><p>The project uses a Version Control System (VCS).
<ol type="a">
<li>The VCS has a working web interface.
<li>There is documentation on how to use the VCS.
<li>The VCS is general-purpose, not something hacked together for this
    one project.
</ol>

<li><p>The project comes with documentation on how to build from source
and that documentation is lucid, correct, and up-to-date.

<li><p>The project is configurable using an autoconf-generated configure
script, or the equivalent, and does not require:
<ol type="a">
<li>Manually editing flat files
<li>Editing code header files
</ol>

<li><p>The project should be buildable using commonly available and
       standard tools like "make".

<li><p>The project does not depend on 3rd-party proprietary build tools.

<li><p>The project is able to dynamically link against standard libraries
      such as zlib and libjpeg.
<ol type="a">
<li>The project does not ship with the sources to standard libraries.
    <i>(On the Fossil project, we will allow the SQLite library sources
    to be included in the source tree as long as a system-installed
    SQLite library can be used in its stead.)</i>
<li>The project does not use slightly modified versions of standard
    libraries.  Any required bug fixes in standard libraries are pushed
    back upstream.
</ol>


<li><p>"make install" works and can be configured to target any
       directory of the installer's choosing.
<ol type="a">
<li>The project contains no unconfigurable hard-coded pathnames like
    "/opt" or "/usr/local".
<li>After installation, the source tree can be moved or deleted and
    the application will continue working.
</ol>


<li><p>The source code uses only \n for line endings, not \r\n.

<li><p>The code does not depend on any special compiler features or bugs.

<li><p>The project has a mailing list and announces releases on
       the mailing list.

<li><p>The project has a bug tracker.

<li><p>The project has a website.

<li><p>Release version numbers are in the traditional X.Y or X.Y.Z format.

<li><p>Releases can be downloaded as tarball using
       gzip or bzip2 compression.

<li><p>Releases unpack into a versioned top-level directory.
       (ex:  "projectname-1.2.3/").

<li><p>A statement of license appears at the top of every source code file
       and the complete text of the license is included in the source code
       tarball.

<li><p>There are no incompatible licenses in the code.

<li><p>The project has not been blithely proclaimed "public domain" without
having gone through the tedious and exacting legal steps to actually put it
in the public domain.

<li><p>There is an accurate change log in the code and on the website.

<li><p>There is documentation in the code and on the website.
</ol>

All of this is exactly what one wants when doing bazaar-style
development.

Fossil's normal mode of operation differs on every one of these points,
with the specific designed-in goal of promoting SQLite's cathedral
development model:

<ul>
    <li><p><b>Personal engagement:</b> SQLite's developers know each
    other by name and work together daily on the project.</p></li>

    <li><p><b>Trust over hierarchy:</b> SQLite's developers check
    changes into their local repository, and these are immediately and
    automatically synchronized up to the central repository; there is no
    "[https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows#_dictator_and_lieutenants_workflow|dictator
    and lieutenants]" hierarchy as with Linux kernel contributions.  D.
    Richard Hipp rarely overrides decisions made by those he has trusted
    with commit access on his repositories. Fossil allows you to give
    [./caps/admin-v-setup.md|some users] more power over what
    they can do with the repository, but Fossil [./caps/index.md#ucap |
    only loosely supports] the enforcement of a development organization's
    social and power hierarchies. Fossil is a great fit for
    [https://en.wikipedia.org/wiki/Flat_organization|flat
    organizations].</p></li>

    <li><p><b>No easy drive-by contributions:</b> Git
    [https://www.git-scm.com/docs/git-request-pull|pull requests] offer
    a low-friction path to accepting
    [https://www.jonobacon.com/2012/07/25/building-strong-community-structural-integrity/|drive-by
    contributions]. Fossil's closest equivalents are its unique
    [/help?cmd=bundle|bundle] and [/help?cmd=patch|patch] features, which require higher engagement
    than firing off a PR.⁷ This difference comes directly from the
    initial designed purpose for each tool: the SQLite project doesn't
    accept outside contributions from previously-unknown developers, but
    the Linux kernel does.</p></li>

    <li><p><b>No rebasing:</b> When your local repo clone syncs changes
    up to its parent, those changes are sent exactly as they were
    committed locally. [#history|There is no rebasing mechanism in
    Fossil, on purpose.]</p></li>

    <li><p><b>Sync over push:</b> Explicit pushes are uncommon in
    Fossil-based projects: the default is to rely on
    [/help?cmd=autosync|autosync mode] instead, in which each commit
    syncs immediately to its parent repository. This is a mode so you
    can turn it off temporarily when needed, such as when working
    offline. Fossil is still a truly distributed version control system;
    it's just that its starting default is to assume you're rarely out
    of communication with the parent repo.
    <br><br>
    This is not merely a reflection of modern always-connected computing
    environments. It is a conscious decision in direct support of
    SQLite's cathedral development model: we don't want developers going
    dark, then showing up weeks later with a massive bolus of changes
    for us to integrate all at once.
    [https://en.wikipedia.org/wiki/Jim_McCarthy_(author)|Jim McCarthy]
    put it well in his book on software project management,
    <i>[https://www.amazon.com/dp/0735623198/|Dynamics of Software
    Development]</i>: "[https://www.youtube.com/watch?v=oY6BCHqEbyc|Beware
    of a guy in a room]."</p></li>

    <li><p><b>Branch names sync:</b> Unlike in Git, branch names in
    Fossil are not purely local labels. They sync along with everything
    else, so everyone sees the same set of branch names. Fossil's design
    choice here is a direct reflection of the Linux vs. SQLite project
    outlook: SQLite's developers collaborate closely on a single
    coherent project, whereas Linux's developers go off on tangents and
    occasionally send selected change sets to each other.</p></li>

    <li><p><b>Private branches are rare:</b>
    [/doc/trunk/www/private.wiki|Private branches exist in Fossil], but
    they're normally used to handle rare exception cases, whereas in
    many Git projects, they're part of the straight-line development
    process.</p></li>

    <li><p><b>Identical clones:</b> Fossil's autosync system tries to
    keep each local clone identical to the repository it cloned
    from.</p></li>
</ul>

Where Git encourages siloed development, Fossil fights against it.
Fossil places a lot of emphasis on synchronizing everyone's work and on
reporting on the state of the project and the work of its developers, so
that everyone — especially the project leader — can maintain a better
mental picture of what is happening, leading to better situational
awareness.

<li> [./hacker-howto.wiki | Hacker How-To]
<li> [./fossil-v-git.wiki | Fossil vs. Git]
<li> [./permutedindex.html | Doc Index]
</ul>
<center><img src="fossil3.gif" align="center"></center>
</div>

<p>Fossil is a simple, high-reliability, distributed software configuration
management system with these advanced features:

  1.  <b>Project Management</b> -
      In addition to doing [./concepts.wiki | distributed version control]
      like Git and Mercurial,
      Fossil also supports [./bugtheory.wiki | bug tracking],
      [./wikitheory.wiki | wiki], [./forum.wiki | forum],
      [./alerts.md|email alerts], [./chat.md | chat], and
      [./event.wiki | technotes].

  2.  <b>Built-in Web Interface</b> -
      Fossil has a built-in, [/skins | themeable], [./serverext.wiki | extensible],
      and intuitive [./webui.wiki | web interface]
      with a rich variety of information pages
      ([./webpage-ex.md|examples]) promoting situational awareness.
      <p>
      This entire website is just a running instance of Fossil.
      The pages you see here are all [./wikitheory.wiki | wiki] or
      [./embeddeddoc.wiki | embedded documentation] or (in the case of
      the [/uv/download.html|download] page)
      [./unvers.wiki | unversioned files].
      When you clone Fossil from one of its
      [./selfhost.wiki | self-hosting repositories],

# How To Mirror A Fossil Repository On GitHub

Beginning with Fossil version 2.9, you can mirror a Fossil-based
project on GitHub (with [limitations](./mirrorlimitations.md))
by following these steps:

<ol>
<li><p>Create an account on GitHub if you do not have one already.  Log
    into that account.

<li><p>Create a new project.  GitHub will ask you if you want to prepopulate
    your project with various things like a README file.  Answer "no" to
    everything.  You want a completely blank project.  GitHub will then
    supply you with a URL for your project that will look something
    like this:

<blockquote>
https://github.com/username/project.git
</blockquote>

<li><p>Back on your workstation, move to a checkout for your Fossil
    project and type:

<blockquote>

<pre>$ fossil git export /path/to/git/repo --autopush \
  https://<font color="orange">username</font>:<font color="red">password</font>@github.com/username/project.git</pre>

</blockquote>

<p>   In place of the <code>/path/to...</code> argument above, put in
      some directory name that is <i>outside</i> of your Fossil checkout. If
      you keep multiple Fossil checkouts in a directory of their own,
      consider using <code>../git-mirror</code> to place the Git export
      mirror alongside them, for example.  Fossil will create this
      directory if necessary.  This directory will become a Git
      repository that holds a translation of your Fossil repository.

<p>   The <code>--autopush</code> option tells Fossil that you want to
      push the Git translation up to GitHub every time it is updated.
      
<p>   The URL parameter is the same as the one GitHub gave you, but with
      your GitHub <font color="orange">username</font> and <font
      color="red">password</font> added.
      
<p>   If your GitHub account uses two-factor authentication (2FA), you
      will have to <a href="https://github.com/settings/tokens">generate
      a personal access token</a> and use that in place of your actual
      password in the URL. This token should have “repo” scope enabled,
      only.

<p>   You can also run the command above outside of any open checkout of
      your project by supplying the “<code>-R&nbsp;repository</code>”
      option.

<li><p>Get some coffee.  Depending on the size of your project, the
       initial "<code>fossil git export</code>" command in the previous
       step might run for several minutes.

<li><p>And you are done!  Assuming everything worked, your project is now
    mirrored on GitHub.

<li><p>Whenever you update your project, simply run this command to update
    the mirror:

<blockquote>
<pre>$ fossil git export</pre>
</blockquote>


<p>   Unlike with the first time you ran that command, you don’t need
      the remaining arguments, because Fossil remembers those things.
      Subsequent mirror updates should usually happen in a fraction of
      a second.

<li><p>To see the status of your mirror, run:

<blockquote>
<pre>$ fossil git status</pre>
</blockquote>
</ol>

## Notes:

  *  Unless you specify --force, the mirroring only happens if the Fossil
     repo has changed, with Fossil reporting "no changes", because Fossil 
     does not care about the success or failure of the mirror run. If a mirror
     run failed (for example, due to an incorrect password, or a transient

<title>How To Create A New Fossil Repository</title>

<p> The [/doc/tip/www/quickstart.wiki|quickstart guide] explains how
to get up and running with fossil. But once you're running, what can
you do with it? This document will walk you through the process of
creating a fossil repository, populating it with files, and then
sharing it over the web.</p>

The first thing we need to do is create a fossil repository file:

<verbatim>
stephan@ludo:~/fossil$ fossil new demo.fossil
project-id: 9d8ccff5671796ee04e60af6932aa7788f0a990a
server-id:  145fe7d71e3b513ac37ac283979d73e12ca04bfe

<title>Principles Of Operation</title>
<h1 align="center">Principles Of Operation</h1>

<p>
This page attempts to define the foundational principals upon
which Fossil is built.
</p>

<ul>
<li><p>A project consists of source files, wiki pages, and
trouble tickets, and control files (collectively "artifacts").
All historical copies of all artifacts
are saved.  The project maintains an audit
trail.</p></li>

<li><p>A project resides in one or more repositories.  Each
repository is administered and operates independently
of the others.</p></li>

<li><p>Each repository has both global and local state.  The
global state is common to all repositories (or at least
has the potential to be shared in common when the
repositories are fully synchronized).  The local state
for each repository is private to that repository.
The global state represents the content of the project.
The local state identifies the authorized users and
access policies for a particular repository.</p></li>

<li><p>The global state of a repository is an unordered
collection of artifacts.  Each artifact is named by a
cryptographic hash (SHA1 or SHA3-256) encoded in
lowercase hexadecimal.
In many contexts, the name can be
abbreviated to a unique prefix.  A five- or six-character
prefix usually suffices to uniquely identify a file.</p></li>

<li><p>Because artifacts are named by a cryptographic hash, all artifacts
are immutable.  Any change to the content of an artifact also
changes the hash that forms the artifacts name, thus
creating a new artifact.  Both the old original version of the
artifact and the new change are preserved under different names.</p></li>

<li><p>It is theoretically possible for two artifacts with different
content to share the same hash.  But finding two such
artifacts is so incredibly difficult and unlikely that we
consider it to be an impossibility.</p></li>

<li><p>The signature of an artifact is the cryptographic hash of the
artifact itself, exactly as it would appear in a disk file.  No prefix
or meta-information about the artifact is added before computing
the hash.  So you can
always find the signature of a file by using the
"sha1sum" or "sha3sum" or similar command-line utilities.</p></li>

<li><p>The artifacts that comprise the global state of a repository
are the complete global state of that repository.  The SQLite
database that holds the repository contains additional information
about linkages between artifacts, but all of that added information
can be discarded and reconstructed by rescanning the content
artifacts.</p></li>

<li><p>Two repositories for the same project can synchronize
their global states simply by sharing artifacts.  The local
state of repositories is not normally synchronized or
shared.</p></li>

<li><p>Every check-in has a special file at the top-level
named "manifest" which is an index of all other files in
that check-in.  The manifest is automatically created and
maintained by the system.</p></li>

<li><p>The <a href="fileformat.wiki">file formats</a>
used by Fossil are all very simple so that with access
to the original content files, one can easily reconstruct
the content of a check-in without the need for any
special tools or software.</p></li>

<title>Questions And Criticisms</title>
<nowiki>
<h1 align="center">Questions And Criticisms</h1>

<p>This page is a collection of real questions and criticisms that were
raised against Fossil early in its history (circa 2008).
This page is old and has not been kept up-to-date.  See the
</nowiki>[/finfo?name=www/qandc.wiki|change history of this page]<nowiki>.</p>

<b>Fossil sounds like a lot of reinvention of the wheel.
Why create your own DVCS when you could have reused mercurial?</b>

<blockquote>
  <p>I wrote fossil because none of the
  other available DVCSes met my needs.  If the other DVCSes do
  meet your needs, then you might not need fossil.  But they
  don't meet mine, and so fossil is necessary for me.</p>

  <p>Features provided by fossil that one does not get with other
  DVCSes include:</p>

  <ol>
  <li> Integrated <a href="wikitheory.wiki">wiki</a>. </li>
  <li> Integrated <a href="bugtheory.wiki">bug tracking</a> </li>
  <li> Immutable artifacts </li>
  <li> Self-contained, stand-alone executable that can be run in
       a <a href="http://en.wikipedia.org/wiki/Chroot">chroot jail</a> </li>

the massive user base of git or mercurial.
</blockquote>

<b>Fossil looks like the bug tracker that would be in your
Linksys Router's administration screen.</b>

<blockquote>
<p>I take a pragmatic approach to software: form follows function.
To me, it is more important to have a reliable, fast, efficient,
enduring, and simple DVCS than one that looks pretty.</p>

<p>On the other hand, if you have patches that improve the appearance
of Fossil without seriously compromising its reliability, performance,
and/or maintainability, I will be happy to accept them.  Fossil is
self-hosting.  Send email to request a password that will let
you push to the main fossil repository.</p>
</blockquote>

<b>It would be useful to have a separate application that
keeps the bug-tracking database in a versioned file. That file can
then be pushed and pulled along with the rest repository.</b>

<blockquote>
<p>Fossil already <u>does</u> push and pull bugs along with the files
in your repository.
But fossil does <u>not</u> track bugs as files in the source tree.
That approach to bug tracking was rejected for three reasons:</p>

<ol>
<li>  Check-ins in fossil are immutable.  So if
      tickets were part of the check-in, then there would be no way to add
      new tickets to a check-in as new bugs are discovered.

<li>  Any project of reasonable size and complexity will generate thousands
      and thousands of tickets, and we do not want all those ticket files
      cluttering the source tree.

<li>  We want tickets to be managed from the web interface and to have a
      permission system that is distinct from check-in permissions.
      In other words, we do not want to restrict the creation and editing
      of tickets to developers with check-in privileges and an installed
      copy of the fossil executable.  Casual passers-by on the internet should
      be permitted to create tickets.
</ol>

<p>These points are reiterated in the opening paragraphs of
the <a href="bugtheory.wiki">Bug-Tracking In Fossil</a> document.</p>
</blockquote>

<b>Fossil is already the name of a plan9 versioned
append-only filesystem.</b>

<blockquote>
I did not know that.  Perhaps they selected the name for the same reason that

<b>I am dubious of the benefits of including wikis and bug trackers
directly in the VCS - either they are under-featured compared to full
software like Trac, or the VCS is massively bloated compared to
Subversion or Bazaar.</b>

<blockquote>
<p>I have no doubt that Trac has many features that fossil lacks.  But that
is not the point.  Fossil has several key features that Trac lacks and that
I need:  most notably the fact that
fossil supports disconnected operation.</p>

<p>As for bloat:  Fossil is a single self-contained executable.
You do not need any other packages
(diff, patch, merge, cvs, svn, rcs, git, python, perl, tcl, apache,
sqlite, and so forth)
in order to run fossil.  Fossil runs just fine in a chroot jail all
by itself.  And the self-contained fossil
executable is much less than 1MB in size.  (Update 2015-01-12: Fossil has
grown in the years since the previous sentence was written but is still
much less than 2MB according to "size" when compiled using -Os on x64 Linux.)
Fossil is the very opposite of bloat.</p>
</blockquote>


</nowiki>

<title>Fossil Quick Start Guide</title>
<h1 align="center">Fossil Quick Start</h1>

<p>This is a guide to help you get started using the Fossil [https://en.wikipedia.org/wiki/Distributed_version_control|Distributed Version Control System] quickly
and painlessly.</p>

<h2 id="install">Installing</h2>

<p>Fossil is a single self-contained C program.  You need to
either download a
[https://fossil-scm.org/home/uv/download.html|precompiled
binary]
or <a href="build.wiki">compile it yourself</a> from sources.
Install Fossil by putting the fossil binary
someplace on your $PATH.</p>

You can test that Fossil is present and working like this:

<blockquote>
<b>
fossil version<br>
<tt>This is fossil version 2.13 [309af345ab] 2020-09-28 04:02:55 UTC</tt><br>
</b>
</blockquote>

<h2 id="workflow" name="fslclone">General Work Flow</h2>

<p>Fossil works with repository files (a database in a single file with the project's
complete history) and with checked-out local trees (the working directory
you use to do your work). 
(See [./glossary.md | the glossary] for more background.)
The workflow looks like this:</p>

<ul>
    <li>Create or clone a repository file.  ([/help/init|fossil init] or
        [/help/clone | fossil clone])
    <li>Check out a local tree.  ([/help/open | fossil open])
    <li>Perform operations on the repository (including repository
        configuration).
</ul>

Fossil can be entirely driven from the command line. Many features
can also be conveniently accessed from the built-in web user interface.

<p>The following sections give a brief overview of these
operations.</p>

<h2 id="new">Starting A New Project</h2>

<p>To start a new project with fossil create a new empty repository
this way: ([/help/init | more info]) </p>

<blockquote>
<b>fossil init </b><i> repository-filename</i>
</blockquote>

You can name the database anything you like, and you can place it anywhere in the filesystem.
The <tt>.fossil</tt> extension is traditional but only required if you are going to use the 
<tt>[/help/server | fossil server DIRECTORY]</tt> feature.”

<h2 id="clone">Cloning An Existing Repository</h2>

<p>Most fossil operations interact with a repository that is on the
local disk drive, not on a remote system.  Hence, before accessing
a remote repository it is necessary to make a local copy of that
repository.  Making a local copy of a remote repository is called
"cloning".</p>

<p>Clone a remote repository as follows: ([/help/clone | more info])</p>

<blockquote>
<b>fossil clone</b> <i>URL  repository-filename</i>
</blockquote>

<p>The <i>URL</i> specifies the fossil repository
you want to clone.  The <i>repository-filename</i> is the new local
filename into which the cloned repository will be written.  For
example, to clone the source code of Fossil itself:

<blockquote>
<b>fossil clone https://fossil-scm.org/ myclone.fossil</b>
</blockquote>

        Vacuuming the database... <br>
        project-id: 94259BB9F186226D80E49D1FA2DB29F935CCA0333<br>
        server-id:  016595e9043054038a9ea9bc526d7f33f7ac0e42<br>
        admin-user: exampleuser (password is "yoWgDR42iv")><br>
</tt></b>
</blockquote>

<p>If the remote repository requires a login, include a
userid in the URL like this:

<blockquote>
<b>fossil clone https://</b><i>remoteuserid</i><b>@www.example.org/ myclone.fossil</b>
</blockquote>

<p>You will be prompted separately for the password.
 Use [https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters|"%HH"] escapes for special characters in the userid.
 For example "/" would be replaced by "%2F" meaning that a userid of "Projects/Budget" would become "Projects%2FBudget") </p>

<p>If you are behind a restrictive firewall, you might need
to <a href="#proxy">specify an HTTP proxy</a>.</p>

<p>A Fossil repository is a single disk file.  Instead of cloning,
you can just make a copy of the repository file (for example, using
"scp").  Note, however, that the repository file contains auxiliary
information above and beyond the versioned files, including some
sensitive information such as password hashes and email addresses.  If you
want to share Fossil repositories directly by copying, consider running the
[/help/scrub|fossil scrub] command to remove sensitive information
before transmitting the file.

<h2 id="import">Importing From Another Version Control System</h2>

<p>Rather than start a new project, or clone an existing Fossil project,
you might prefer to
<a href="./inout.wiki">import an existing Git project</a>
into Fossil using the [/help/import | fossil import] command. 

You can even decide to export your project back into git using the
[/help/git | fossil git] command, which is how the Fossil project maintains
[https://github.com/drhsqlite/fossil-mirror | its public GitHub mirror]. There
is no limit to the number of times a tree can be imported and exported between
Fossil and git.

The [https://git-scm.com/docs/git-fast-export|Git fast-export format] has become
a popular way to move files between version management systems, including from
[https://www.mercurial-scm.org/|Mercurial].
Fossil can also import [https://subversion.apache.org/|Subversion projects] directly.

<h2 id="checkout">Checking Out A Local Tree</h2>

<p>To work on a project in fossil, you need to check out a local
copy of the source tree.  Create the directory you want to be
the root of your tree and cd into that directory.  Then
do this: ([/help/open | more info])</p>

<blockquote>
<b>fossil open </b><i> repository-filename</i>
</blockquote>

for example:

<blockquote>
<b><tt>
    fossil open ../myclone.fossil<br>
    BUILD.txt<br>
    COPYRIGHT-BSD2.txt<br>
    README.md<br>
      ︙<br>
</tt></b>
</blockquote>

(or "fossil open ..\myclone.fossil" on Windows).

<p>This leaves you with the newest version of the tree
checked out.
From anywhere underneath the root of your local tree, you
can type commands like the following to find out the status of
your local tree:</p>

<blockquote>
<b>[/help/info | fossil info]</b><br>
<b>[/help/status | fossil status]</b><br>
<b>[/help/changes | fossil changes]</b><br>
<b>[/help/diff | fossil diff]</b><br>
<b>[/help/timeline | fossil timeline]</b><br>
<b>[/help/ls | fossil ls]</b><br>
<b>[/help/branch | fossil branch]</b><br>
</blockquote>

<p>If you created a new repository using "fossil init" some commands will not
produce much output.</p>

<p>Note that Fossil allows you to make multiple check-outs in
separate directories from the same repository.  This enables you,
for example, to do builds from multiple branches or versions at
the same time without having to generate extra clones.</p>

<p>To switch a checkout between different versions and branches,
use:</p>

<blockquote>
<b>[/help/update | fossil update]</b><br>
<b>[/help/checkout | fossil checkout]</b><br>
</blockquote>

<p>[/help/update | update] honors the "autosync" option and
does a "soft" switch, merging any local changes into the target
version, whereas [/help/checkout | checkout] does not
automatically sync and does a "hard" switch, overwriting local
changes if told to do so.</p>

<h2 id="changes">Making and Committing Changes</h2>

<p>To add new files to your project or remove existing ones, use these
commands:</p>

<blockquote>
<b>[/help/add | fossil add]</b> <i>file...</i><br>
<b>[/help/rm | fossil rm]</b> <i>file...</i><br>
<b>[/help/addremove | fossil addremove]</b> <i>file...</i><br>
</blockquote>

<p>The command:</p>

<blockquote>
<b>
        [/help/changes | fossil changes]</b>
</blockquote>

<p>lists files that have changed since the last commit to the repository. For
example, if you edit the file "README.md":</p>

<blockquote>
<b>
        fossil changes<br>
        EDITED     README.md
</b>
</blockquote>

<p>To see exactly what change was made you can use the command</p>
[/help/diff | fossil diff]:

<blockquote>
<b>
        fossil diff <br><tt>
        Index: README.md<br>
        ============================================================<br>
        --- README.md<br>
        +++ README.md<br>
        @@ -1,5 +1,6 @@<br>
        +Made some changes to the project<br>
        # Original text<br>
 </tt></b>
</blockquote>

<p>"fossil diff" is the difference between your tree on disk now and as the tree was
when you did "fossil open". An open is the first checkout from a repository 
into a new directory. </p>

<p>To see the most recent changes made to the repository by other users, use "fossil timeline" to
find out the most recent commit, and then "fossil diff" between that commit and the
current tree: </p>
<blockquote>
<b>
        fossil timeline <br><tt>
        === 2021-03-28 === <br>
        03:18:54 [ad75dfa4a0] *CURRENT* Added details to frobnicate command (user: user-one tags: trunk) <br>
        === 2021-03-27 === <br>
        23:58:05 [ab975c6632] Update README.md. (user: user-two tags: trunk) <br>

        # Original text<br>
</tt></b>
</blockquote>

"current" is an alias for the checkout version, so the command
"fossil diff --from ad75dfa4a0 --to ab975c6632" gives identical results.

<p>To commit your changes to a local-only repository:</p>
<blockquote>
<b>
fossil commit     </b><i>(... Fossil will start your editor, if defined)</i><b><br><tt>
# Enter a commit message for this check-in. Lines beginning with # are ignored.<br>
#<br>
# user: exampleuser<br>
# tags: trunk<br>
#<br>
# EDITED     README.md<br>
Edited file to add description of code changes<br>
New_Version: 7b9a416ced4a69a60589dde1aedd1a30fde8eec3528d265dbeed5135530440ab<br>
</tt></b>
</blockquote>

<p>You will be prompted for check-in comments using whatever editor
is specified by your VISUAL or EDITOR environment variable. If none is
specified Fossil uses line-editing in the terminal.</p>

<p>To commit your changes to a repository that was cloned from remote you 
perform the same actions but the results are different. Fossil
defaults to 'autosync' mode, a single-stage commit that sends all changes 
committed to the local repository immediately on to the remote parent repository. This
only works if you have write permission to the remote respository.</p>

<h2 id="naming">Naming of Files, Checkins, and Branches</h2>

<p>Fossil deals with information artifacts. This Quickstart document only deals
with files and collections of files, but be aware there are also tickets, wiki pages and more. 
Every artifact in Fossil has a universally-unique hash id, and may also have a
human-readable name.</p>

<p>The following are all equivalent ways of identifying a Fossil file,
checkin or branch artifact:</p>

<ul>
<li> the full unique SHA-256 hash, such as be836de35a821523beac2e53168e135d5ebd725d7af421e5f736a28e8034673a
<li> an abbreviated hash prefix, such as the first ten characters: be836de35a . This won't be universally unique, but it is usually unique within any one repository. As an example, the [https://fossil-scm.org/home/hash-collisions|Fossil project hash collisions] showed at the time of writing that there are no artifacts with identical first 8 characters
<li> a branch name, such as "special-features" or "juliet-testing". Each branch also has a unique SHA-256 hash
</ul>

<p>A special convenience branch is "trunk", which is Fossil's default branch name for
the first checkin, and the default for any time a branch name is needed but not
specified.</p>

This will get you started on identifying checkins. The
<a href="./checkin_names.wiki">Checkin Names document</a> is a complete reference, including
how timestamps can also be used.

<h2 id="config">Configuring Your Local Repository</h2>

<p>When you create a new repository, either by cloning an existing
project or create a new project of your own, you usually want to do some
local configuration.  This is easily accomplished using the web-server
that is built into fossil.  Start the fossil web server like this:
([/help/ui | more info])</p>

<blockquote>
<b>fossil ui </b><i> repository-filename</i>
</blockquote>

<p>You can omit the <i>repository-filename</i> from the command above
if you are inside a checked-out local tree.</p>

<p>This starts a web server then automatically launches your
web browser and makes it point to this web server.  If your system
has an unusual configuration, fossil might not be able to figure out
how to start your web browser.  In that case, first tell fossil
where to find your web browser using a command like this:</p>

<blockquote>
<b>fossil setting web-browser </b><i>  path-to-web-browser</i>
</blockquote>

<p>By default, fossil does not require a login for HTTP connections
coming in from the IP loopback address 127.0.0.1.  You can, and perhaps
should, change this after you create a few users.</p>

<p>When you are finished configuring, just press Control-C or use
the <b>kill</b> command to shut down the mini-server.</p>

<h2 id="changes">Making Changes</h2>

<p>To add new files to your project, or remove old files, use these
commands:</p>

<blockquote>
<b>[/help/add | fossil add]</b> <i>file...</i><br>
<b>[/help/rm | fossil rm]</b> <i>file...</i><br>
<b>[/help/addremove | fossil addremove]</b> <i>file...</i><br>
</blockquote>

<p>You can also edit files freely.  Once you are ready to commit
your changes, type:</p>

<blockquote>
<b>[/help/commit | fossil commit]</b>
</blockquote>

<p>You will be prompted for check-in comments using whatever editor
is specified by your VISUAL or EDITOR environment variable.</p>

In the default configuration, the [/help/commit|commit]
command will also automatically [/help/push|push] your changes, but that
feature can be disabled.  (More information about
[./concepts.wiki#workflow|autosync] and how to disable it.)
Remember that your coworkers can not see your changes until you
commit and push them.</p>

<h2 id="sharing">Sharing Changes</h2>

<p>When [./concepts.wiki#workflow|autosync] is turned off,
the changes you [/help/commit | commit] are only
on your local repository.
To share those changes with other repositories, do:</p>

<blockquote>
<b>[/help/push | fossil push]</b> <i>URL</i>
</blockquote>

<p>Where <i>URL</i> is the http: URL of the server repository you
want to share your changes with.  If you omit the <i>URL</i> argument,
fossil will use whatever server you most recently synced with.</p>

<p>The [/help/push | push] command only sends your changes to others.  To
Receive changes from others, use [/help/pull | pull].  Or go both ways at
once using [/help/sync | sync]:</p>

<blockquote>
<b>[/help/pull | fossil pull]</b> <i>URL</i><br>
<b>[/help/sync | fossil sync]</b> <i>URL</i>
</blockquote>

<p>When you pull in changes from others, they go into your repository,
not into your checked-out local tree.  To get the changes into your
local tree, use [/help/update | update]:</p>

<blockquote>
<b>[/help/update | fossil update]</b> <i>VERSION</i>
</blockquote>

<p>The <i>VERSION</i> can be the name of a branch or tag or any
abbreviation to the 40-character
artifact identifier for a particular check-in, or it can be a
date/time stamp.  ([./checkin_names.wiki | more info])
If you omit
the <i>VERSION</i>, then fossil moves you to the
latest version of the branch your are currently on.</p>

<p>The default behavior is for [./concepts.wiki#workflow|autosync] to
be turned on.  That means that a [/help/pull|pull] automatically occurs
when you run [/help/update|update] and a [/help/push|push] happens
automatically after you [/help/commit|commit].  So in normal practice,
the push, pull, and sync commands are rarely used.  But it is important
to know about them, all the same.</p>

<blockquote>
<b>[/help/checkout | fossil checkout]</b> <i>VERSION</i>
</blockquote>

<p>Is similar to update except that it does not honor the autosync
setting, nor does it merge in local changes - it prefers to overwrite
them and fails if local changes exist unless the <tt>--force</tt>
flag is used.</p>

<h2 id="branch" name="merge">Branching And Merging</h2>

<p>Use the --branch option to the [/help/commit | commit] command
to start a new branch.  Note that in Fossil, branches are normally
created when you commit, not before you start editing.  You can
use the [/help/branch | branch new] command to create a new branch
before you start editing, if you want, but most people just wait
until they are ready to commit.

To merge two branches back together, first
[/help/update | update] to the branch you want to merge into.
Then do a [/help/merge|merge] of the other branch that you want to incorporate
the changes from.  For example, to merge "featureX" changes into "trunk"
do this:</p>

<blockquote>
<b>fossil [/help/update|update] trunk</b><br>
<b>fossil [/help/merge|merge] featureX</b><br>
<i># make sure the merge didn't break anything...</i><br>
<b>fossil [/help/commit|commit]
</blockquote>

<p>The argument to the [/help/merge|merge] command can be any of the
version identifier forms that work for [/help/update|update].
([./checkin_names.wiki|more info].)
The merge command has options to cherry-pick individual
changes, or to back out individual changes, if you don't want to
do a full merge.</p>

The merge command puts all changes in your working check-out.
No changes are made to the repository.
You must run [/help/commit|commit] separately
to add the merge changes into your repository to make them persistent
and so that your coworkers can see them.
But before you do that, you will normally want to run a few tests
to verify that the merge didn't cause logic breaks in your code.

The same branch can be merged multiple times without trouble. Fossil
automatically keeps up with things and avoids conflicts when doing
multiple merges.  So even if you have merged the featureX branch
into trunk previously, you can do so again and Fossil will automatically
know to pull in only those changes that have occurred since the previous
merge.

<p>If a merge or update doesn't work out (perhaps something breaks or
there are many merge conflicts) then you back up using:</p>

<blockquote>
<b>[/help/undo | fossil undo]</b>
</blockquote>

<p>This will back out the changes that the merge or update made to the
working checkout.  There is also a [/help/redo|redo] command if you undo by
mistake.  Undo and redo only work for changes that have
not yet been checked in using commit and there is only a single
level of undo/redo.</p>


<h2 id="server">Setting Up A Server</h2>

<p>Fossil can act as a stand-alone web server using one of these
commands:</p>

<blockquote>
<b>[/help/server | fossil server]</b> <i>repository-filename</i><br>
<b>[/help/ui | fossil ui]</b> <i>repository-filename</i>
</blockquote>

<p>The <i>repository-filename</i> can be omitted when these commands
are run from within an open check-out, which is a particularly useful
shortcut with the <b>fossil ui</b> command.

<p>The <b>ui</b> command is intended for accessing the web user interface
from a local desktop. (We sometimes call this mode "Fossil UI.")
The <b>ui</b> command differs from the
<b>server</b> command by binding to the loopback IP
address only (thus making the web UI visible only on the
local machine) and by automatically starting your default web browser,
pointing it at the running UI
server. The localhost restriction exists because it also gives anyone
who can access the resulting web UI full control over the
repository. (This is the [./caps/admin-v-setup.md#apsu | all-powerful
Setup capabliity].)</p>

<p>For cross-machine collaboration, use the <b>server</b> command instead,
which binds on all IP addresses, does not try to start a web browser,
and enforces [./caps/ | Fossil's role-based access control system].</p>

<p>Servers are also easily configured as:

<ul>
<li>[./server/any/inetd.md|inetd]
<li>[./server/debian/service.md|systemd]
<li>[./server/any/cgi.md|CGI]
<li>[./server/any/scgi.md|SCGI]
</ul>

<p>…along with [./server/#matrix | several other options].</p>

<p>The [./selfhost.wiki | self-hosting fossil repositories] use
CGI.

<p>You might <i>need</i> to set up a server, whether you know it yet or
not.  See the [./server/whyuseaserver.wiki | Benefits of a Fossil Server]
article for details.</p>

<h2 id="proxy">HTTP Proxies</h2>

<p>If you are behind a restrictive firewall that requires you to use
an HTTP proxy to reach the internet, then you can configure the proxy
in three different ways.  You can tell fossil about your proxy using
a command-line option on commands that use the network,
<b>sync</b>, <b>clone</b>, <b>push</b>, and <b>pull</b>.</p>

<blockquote>
<b>fossil clone </b><i>URL</i>  <b>--proxy</b> <i>Proxy-URL</i>
</blockquote>

<p>It is annoying to have to type in the proxy URL every time you
sync your project, though, so you can make the proxy configuration
persistent using the [/help/setting | setting] command:</p>

<blockquote>
<b>fossil setting proxy </b><i>Proxy-URL</i>
</blockquote>

<p>Or, you can set the "<b>http_proxy</b>" environment variable:</p>

<blockquote>
<b>export http_proxy=</b><i>Proxy-URL</i>
</blockquote>

<p>To stop using the proxy, do:</p>

<blockquote>
<b>fossil setting proxy off</b>
</blockquote>

<p>Or unset the environment variable.  The fossil setting for the
HTTP proxy takes precedence over the environment variable and the
command-line option overrides both.  If you have a persistent
proxy setting that you want to override for a one-time sync, that
is easily done on the command-line.  For example, to sync with
a co-worker's repository on your LAN, you might type:</p>

<blockquote>
<b>fossil sync http://192.168.1.36:8080/ --proxy off</b>
</blockquote>

<h2 id="links">Other Resources</h2>

<blockquote>
<i>viablepanic at [https://www.reddit.com/r/programming/comments/bxcto/why_not_fossil_scm/c0p30b4?utm_source=share&utm_medium=web2x&context=3]</i>
</blockquote>

<li>In the fossil community - and hence in fossil itself - development history
is pretty much sacrosanct. The very name "fossil" was to chosen to
reflect the unchanging nature of things in that history.

<p>In git (or rather, the git community), the development history is part of
the published aspect of the project, so it provides tools for rearranging
that history so you can present what you "should" have done rather
than what you actually did.

<blockquote>
<i>Mike Meyer on the Fossil mailing list, 2011-10-04</i>
</blockquote>

# Serving via althttpd

[Althttpd][althttpd]
is a light-weight web server that has been used to implement the SQLite and
Fossil websites for well over a decade. Althttpd strives for simplicity,
security, ease of configuration, and low resource usage.

To set up a Fossil server as CGI on a host running the althttpd web
server, follow these steps.
<ol>
<li<p>Get the althttpd webserver running on the host.  This is easily
done by following the [althttpd documentation][althttpd].

<li><p>Create a CGI script for your Fossil repository.  The script will
be typically be two lines of code that look something like this:

~~~
    #!/usr/bin/fossil
    repository: /home/yourlogin/fossils/project.fossil
~~~

Modify the filenames to conform to your system, of course.  The
CGI script accepts [other options][cgi] besides the
repository:" line.  You can add in other options as you desire,
but the single "repository:" line is normally all that is needed
to get started.

<li><p>Make the CGI script executable.

<li><p>Verify that the fossil repository file and the directory that contains
the repository are both writable by whatever user the web server is
running and.
</ol>

And you are done.  Visit the URL that corresponds to the CGI script
you created to start using your Fossil server.

to ensure that (in the limit) all participating peers see the same content.

<h2>But, a Server Can Be Useful</h2>

Fossil does not require a server, but a server can be very useful.
Here are a few reasons to set up a Fossil server for your project:

  1.  <b>A server works as a complete project website.</b><p>

      Fossil does more than just version control.  It also supports
      [../tickets.wiki|trouble-tickets], 
      [../wikitheory.wiki|wiki], 
      a [../chat.md|developer chat room], and a [../forum.wiki|forum].
      The [../embeddeddoc.wiki|embedded documentation]
      feature provides  a great mechanism for providing project documentation.
      The [../unvers.wiki|unversioned files] feature is a convenient way
      to host builds and downloads on the project website.

  2.  <b>A server gives developers a common point of rendezvous for
      syncing their work.</b><p>

      It is possible for developers to synchronize peer-to-peer but
      that requires the developers coordinate the sync, which in turn
      requires that the developers both want to sync at the same moment.
      A server alleviates this time dependency by allowing each developer
      to sync whenever it is convenient. For example, a developer may
      choose to automatically sync
      after each commit and before each update.  Developers all stay
      in sync with each other without having to interrupt each other
      constantly to set up a peer-to-peer sync.

  3.  <b>A server provides project leaders with up-to-date status.</b><p>

      Project coordinators and BDFLs can click on a link or two at the
      central Fossil server for a project and quickly tell what is
      going on.  They can do this from anywhere — even from their phones
      — without needing to actually sync to the device they are using.

  4.  <b>A server provides automatic off-site backups.</b><p>

      A Fossil server is an automatic remote backup for all the work
      going into a project. ([../backup.md | Within limits].)
      You can even set up multiple servers at
      multiple sites with automatic synchronization between them for
      added redundancy.  Such a setup means that no work is lost due
      to a single machine failure.

  5.  <b>A server consolidates [https://www.sqlite.org/howtocorrupt.html
      | SQLite corruption risk mitigation] to a single point.</b><p>

      The concerns in section 1 of that document assume you have direct
      access to the central DB files, which isn't the case when the
      server is remote and secure against tampering.<p>

      Section 2 is about file locking, which concerns disappear when Fossil's
      on the other side of an HTTP boundary and your server is set up
      properly.<p>

      Sections 3.1, 4 thru 6, and 8 apply to all Fossil configurations,
      but setting up a server lets you address the risks
      in a single place. Once a given commit is
      sync'd to the server, you can be reasonably sure any client-side
      corruption can be fixed with a fresh clone. Ultimately, this
      is an argument for off-machine backups, which returns us to reason
      #4 above.<p>

      Sections 3.2 and the entirety of section 7 are no concern with
      Fossil at all, since it's primarily written by the creator and
      primary maintainer of SQLite, so you can be certain Fossil doesn't
      actively pursue coding strategies known to risk database corruption.<p>

      <p>For another take on this topic, see the article
      "[https://sqlite.org/useovernet.html | SQLite Over a Network,
      Caveats and Considerations]". Fossil runs in rollback mode by
      default per recommendation #3 at the end of that article, and a
      Fossil server operates as a network proxy for the underlying
      SQLite repository DB per recommendation #2. This <i>may</i> permit
      you to safely switch it into WAL mode (<b>fossil rebuild --wal</b>)
      depending on the underlying storage used by the server itself.</p>

  6.  <b>A server allows [../caps/ | Fossil's RBAC system] to work.</b><p>

      The role-based access control (RBAC) system in Fossil only works
      when the remote system is on the other side of an HTTP barrier.
      ([../caps/#webonly | Details].) If you want its benefits, you need
      a Fossil server setup of some kind.

foil spammers up front], legally problematic check-ins should range from
rare to nonexistent, and you have to go way out of your way to force
Fossil to insert bad control artifacts. Therefore, before we get to
methods of permanently deleting content from a Fossil repos, let's give
some alternatives that usually suffice, which don't damage the project's
fossil record:

<ul>
    <li><p>When a forum post or wiki article is "deleted," what actually
    happens is that a new empty version is added to the Fossil repository.
    The web interface interprets this
    as "deleted," but the prior version remains available if you go
    digging for it.</p></li>

    <li><p>When you close a ticket, it's marked in a way that causes it
    to not show up in the normal ticket reports. You usually want to
    give it a Resolution such as "Rejected" when this happens, plus
    possibly a comment explaining why you're closing it. This is all new
    information added to the ticket, not deletion.</p></li>

    <li><p>When you <tt>fossil rm</tt> a file, a new manifest is
    checked into the repository with the same file list as for the prior
    version minus the "removed" file. The file is still present in the
    repository; it just isn't part of that version forward on that
    branch.</p></li>

    <li><p>If you make a bad check-in, you can shunt it off to the side
    by amending it to put it on a different branch, then continuing
    development on the prior branch:
    <p>
    <tt>$ fossil amend abcd1234 --branch BOGUS --hide<br>
    $ fossil up trunk</tt>
    <p>
    The first command moves check-in ID <tt>abcd1234</tt> (and any
    subsequent check-ins on that branch!) to a branch called
    <tt>BOGUS</tt>, then hides it so it doesn't show up on the
    timeline. You can call this branch anything you like, and you can
    re-use the same name as many times as you like. No content is
    actually deleted: it's just shunted off to the side and hidden away.
    You might find it easier to do this from the Fossil web UI in
    the "edit" function for a check-in.
    <p>
    The second command returns to the last good check-in on that branch
    so you can continue work from that point.</p></li>

    <li><p>When the check-in you want to remove is followed by good
    check-ins on the same branch, you can't use the previous method,
    because it will move the good check-ins, too. The solution is:
    <p>
    <tt>$ fossil merge --backout abcd1234</tt>

    <p>That creates a diff in the check-out directory that backs out the
    bad check-in <tt>abcd1234</tt>. You then fix up any merge conflicts,
    build, test, etc., then check the reverting change into the
    repository. Again, nothing is actually deleted; you're just adding
    more information to the repository which corrects a prior
    check-in.</p></li>
</ul>

<h2>Exception: Non-versioned Content</h2>

It is normal and expected to delete data which is not versioned, such as
usernames and passwords in the user table. The [/help/scrub|fossil scrub]
command will remove all sensitive non-versioned data from a repository.

<pre>
    SSL verification failed: unable to get local issuer certificate
</pre>

Fossil relies on the OpenSSL library to have some way to check a trusted
list of CA signing keys. There are two common ways this fails:

  #  <p>The OpenSSL library Fossil is linked to doesn't have a CA
     signing key set at all, so that it initially trusts no certificates
     at all.</p>

  #  <p>The OpenSSL library does have a CA cert set, but your Fossil server's
     TLS certificate was signed by a CA that isn't in that set.</p>

A common reason to fall into the second trap is that you're using
certificates signed by a local private CA, as often happens in large
enterprises.  You can solve this sort of problem by getting your local
CA's signing certificate in PEM format and pointing OpenSSL at it:

<pre>

"<tt>http</tt>" URIs to Fossil, so Fossil issues a redirect, so the browser
fetches the page again, causing Fossil to see an "<tt>http</tt>" URI again, so
it issues a redirect...'round and 'round it goes until the web browser
detects it's in a redirect loop and gives up. This problem prevents you
from getting back into the Admin UI to fix it, but there are several
ways to fix it:

  #  <p><b>Reset via CLI.</b> You can turn the setting back off from the
     CLI with the command "<tt>fossil -R /path/to/repo.fossil set
     redirect-to-https 0</tt>". (Currently doesn't work.)</p>

  #  <p><b>Backup first.</b> This setting is stored in the Fossil
     repository, so if you make a backup first <i>on the server</i>, you
     can restore the repo file if enabling this feature creates a
     redirect loop.</p>

  #  <p><b>Download, fix, and restore.</b> You can copy the remote
     repository file down to a local machine, use <tt>fossil ui</tt> to
     fix the setting, and then upload it to the repository server
     again.</p>

It's best to enforce TLS-only access at the front-end proxy level
anyway. It not only avoids the problem entirely, it can be significantly
more secure.  The [./server/debian/nginx.md#tls | nginx-on-Debian proxy guide] shows one way
to achieve this.</p>


<h2>Terminology Note</h2>

This document is called <tt>ssl.wiki</tt> for historical reasons. The
TLS protocol was originally called SSL, and it went through several
revisions before being replaced by TLS. Years before this writing, SSL

<title>The Fossil Sync Protocol</title>

<p>This document describes the wire protocol used to synchronize
content between two Fossil repositories.</p>

<h2>1.0 Overview</h2>

<p>The global state of a fossil repository consists of an unordered
collection of artifacts.  Each artifact is identified by a cryptographic
hash of its content, expressed as a lower-case hexadecimal string.
Synchronization is the process of sharing artifacts between
repositories so that all repositories have copies of all artifacts.  Because
artifacts are unordered, the order in which artifacts are received
is unimportant.  It is assumed that the hash names
of artifacts are unique - that every artifact has a different hash.
To a first approximation, synchronization proceeds by sharing lists
of hashes for available artifacts, then sharing the content of artifacts
whose names are missing from one side or the other of the connection.
In practice, a repository might contain millions of artifacts.  The list of
hash names for this many artifacts can be large.  So optimizations are
employed that usually reduce the number of hashes that need to be
shared to a few hundred.</p>

<p>Each repository also has local state.  The local state determines
the web-page formatting preferences, authorized users, ticket formats,
and similar information that varies from one repository to another.
The local state is not usually transferred during a sync.  Except,
some local state is transferred during a [/help?cmd=clone|clone]
in order to initialize the local state of the new repository.  Also,
an administrator can sync local state using
the [/help?cmd=configuration|config push] and
[/help?cmd=configuration|config pull]
commands.

<h3 id="crdt">1.1 Conflict-Free Replicated Datatypes</h3>

<p>The "bag of artifacts" data model used by Fossil is apparently an
implementation of a particular
[https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type|Conflict-Free
Replicated Datatype (CRDT)] called a "G-Set" or "Grow-only Set".  The
academic literature on CRDTs only began to appear in about 2011, and
Fossil predates that research by at least 4 years.  But it is nice to
know that theorists have now proven that the underlying data model of
Fossil can provide strongly-consistent replicas using only
peer-to-peer communication and without any kind of central
authority.</p>

<p>If you are already familiar with CRDTs and were wondering if Fossil
used them, the answer is "yes".  We just don't call them by that name.</p>


<h2>2.0 Transport</h2>

<p>All communication between client and server is via HTTP requests.
The server is listening for incoming HTTP requests.  The client
issues one or more HTTP requests and receives replies for each
request.</p>

<p>The server might be running as an independent server
using the <b>server</b> command, or it might be launched from
inetd or xinetd using the <b>http</b> command.  Or the server might
be launched from CGI.
(See "[./server/|How To Configure A Fossil Server]" for details.)
The specifics of how the server listens
for incoming HTTP requests is immaterial to this protocol.
The important point is that the server is listening for requests and
the client is the issuer of the requests.</p>

<p>A single push, pull, or sync might involve multiple HTTP requests.
The client maintains state between all requests.  But on the server
side, each request is independent.  The server does not preserve
any information about the client from one request to the next.</p>

<p>Note: Throughout this article, we use the terms "server" and "client"
to represent the listener and initiator of the interaction, respectively.
Nothing in this protocol requires that the server actually be a back-room
processor housed in a datacenter, nor does the client need to be a desktop
or handheld device.  For the purposes of this article "client" simply means
the repository that initiates the conversation and "server" is the repository
that responds.  Nothing more.</p>

<h4>2.0.1 HTTPS Transport</h4>

<p>In the current implementation of Fossil, the server only
understands HTTP requests.  The client can send either
clear-text HTTP requests or encrypted HTTPS requests.  But when
HTTPS requests are sent, they first must be decrypted by a web server
or proxy before being passed to the Fossil server.  This limitation
may be relaxed in a future release.</p>

<h4>2.0.2 SSH Transport</h4>

<p>When doing a sync using an "ssh:..." URL, the same HTTP transport protocol
is used.  Fossil simply uses [https://en.wikipedia.org/wiki/Secure_Shell|ssh]
to start an instance of the [/help?cmd=test-http|fossil test-http] command
running on the remote machine.  It then sends HTTP requests and gets back HTTP
replies over the SSH connection, rather than sending and receiving over an
internet socket.  To see the specific "ssh" command that the Fossil client
runs in order to set up a connection, add either of the the "--httptrace" or
"--sshtrace" options to the "fossil sync" command line.

<h4>2.0.3 FILE Transport</h4>

<p>When doing a sync using a "file:..." URL, the same HTTP protocol is
still used.  But instead of sending each HTTP request over a socket or
via SSH, the HTTP request is written into a temporary file.  The client
then invokes the [/help?cmd=http|fossil http] command in a subprocess
to process the request and and generate a reply.  The client then reads
the HTTP reply out of a temporary file on disk, and deletes the two
temporary files.  To see the specific "fossil http" command that is run
in order to implement the "file:" transport, add the "--httptrace"
option to the "fossil sync" command.

<h3>2.1 Server Identification</h3>

<p>The server is identified by a URL argument that accompanies the
push, pull, or sync command on the client.  (As a convenience to
users, the URL can be omitted on the client command and the same URL
from the most recent push, pull, or sync will be reused.  This saves
typing in the common case where the client does multiple syncs to
the same server.)</p>

<p>The client modifies the URL by appending the method name "<b>/xfer</b>"
to the end.  For example, if the URL specified on the client command
line is</p>

<blockquote>
https://fossil-scm.org/fossil
</blockquote>

<p>Then the URL that is really used to do the synchronization will
be:</p>

<blockquote>
https://fossil-scm.org/fossil/xfer
</blockquote>

<h3>2.2 HTTP Request Format</h3>

<p>The client always sends a POST request to the server.  The
general format of the POST request is as follows:</p>

<blockquote><pre>
POST /fossil/xfer HTTP/1.0
Host: fossil-scm.hwaci.com:80
Content-Type: application/x-fossil
Content-Length: 4216

<i>content...</i>
</pre></blockquote>

<p>In the example above, the pathname given after the POST keyword
on the first line is a copy of the URL pathname.  The Host: parameter
is also taken from the URL.  The content type is always either
"application/x-fossil" or "application/x-fossil-debug".  The "x-fossil"
content type is the default.  The only difference is that "x-fossil"
content is compressed using zlib whereas "x-fossil-debug" is sent
uncompressed.</p>

<p>A typical reply from the server might look something like this:</p>

<blockquote><pre>
HTTP/1.0 200 OK
Date: Mon, 10 Sep 2007 12:21:01 GMT
Connection: close
Cache-control: private
Content-Type: application/x-fossil; charset=US-ASCII
Content-Length: 265

<i>content...</i>
</pre></blockquote>

<p>The content type of the reply is always the same as the content type
of the request.</p>

<h2>3.0 Fossil Synchronization Content</h2>

<p>A synchronization request between a client and server consists of
one or more HTTP requests as described in the previous section.  This
section details the "x-fossil" content type.</p>

<h3>3.1 Line-oriented Format</h3>

<p>The x-fossil content type consists of zero or more "cards".  Cards
are separated by the newline character ("\n").  Leading and trailing
whitespace on a card is ignored.  Blank cards are ignored.</p>

<p>Each card is divided into zero or more space separated tokens.
The first token on each card is the operator.  Subsequent tokens
are arguments.  The set of operators understood by servers is slightly
different from the operators understood by clients, though the two
are very similar.</p>

<h3>3.2 Login Cards</h3>

<p>Every message from client to server begins with one or more login
cards.  Each login card has the following format:</p>

<blockquote>
<b>login</b>  <i>userid  nonce  signature</i>
</blockquote>

<p>The userid is the name of the user that is requesting service
from the server.  The nonce is the SHA1 hash of the remainder of
the message - all text that follows the newline character that
terminates the login card.  The signature is the SHA1 hash of
the concatenation of the nonce and the users password.</p>

<p>For each login card, the server looks up the user and verifies
that the nonce matches the SHA1 hash of the remainder of the
message.  It then checks the signature hash to make sure the
signature matches.  If everything
checks out, then the client is granted all privileges of the
specified user.</p>

<p>Privileges are cumulative.  There can be multiple successful
login cards.  The session privileges are the bit-wise OR of the
privileges of each individual login.</p>

<h3>3.3 File Cards</h3>

<p>Artifacts are transferred using either "file" cards, or "cfile"
or "uvfile" cards.
The name "file" card comes from the fact that most artifacts correspond to
files that are under version control.
The "cfile" name is an abbreviation for "compressed file".
The "uvfile" name is an abbreviation for "unversioned file".
</p>

<h4>3.3.1 Ordinary File Cards</h4>

<p>For sync protocols, artifacts are transferred using "file"
cards.  File cards come in two different formats depending
on whether the artifact is sent directly or as a delta from some
other artifact.</p>

<blockquote>
<b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i><br>
<b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
</blockquote>

<p>File cards are followed by in-line "payload" data.
The content of the artifact
or the artifact delta is the first <i>size</i> bytes of the
x-fossil content that immediately follow the newline that
terminates the file card.
</p>

<p>The first argument of a file card is the ID of the artifact that
is being transferred.  The artifact ID is the lower-case hexadecimal
representation of the name hash for the artifact.
The last argument of the file card is the number of bytes of
payload that immediately follow the file card.  If the file
card has only two arguments, that means the payload is the
complete content of the artifact.  If the file card has three
arguments, then the payload is a delta and second argument is
the ID of another artifact that is the source of the delta.</p>

<p>File cards are sent in both directions: client to server and
server to client.  A delta might be sent before the source of
the delta, so both client and server should remember deltas
and be able to apply them when their source arrives.</p>

<h4>3.3.2 Compressed File Cards</h4>

<p>A client that sends a clone protocol version "3" or greater will
receive artifacts as "cfile" cards while cloning.  This card was
introduced to improve the speed of the transfer of content by sending the
compressed artifact directly from the server database to the client.</p>

<p>Compressed File cards are similar to File cards, sharing the same
in-line "payload" data characteristics and also the same treatment of
direct content or delta content.  Cfile cards come in two different formats
depending on whether the artifact is sent directly or as a delta from
some other artifact.</p>

<blockquote>
<b>cfile</b> <i>artifact-id usize csize</i> <b>\n</b> <i>content</i><br>
<b>cfile</b> <i>artifact-id delta-artifact-id usize csize</i> <b>\n</b> <i>content</i><br>
</blockquote>

<p>The first argument of the cfile card is the ID of the artifact that
is being transferred.  The artifact ID is the lower-case hexadecimal
representation of the name hash for the artifact.  The second argument of
the cfile card is the original size in bytes of the artifact.  The last
argument of the cfile card is the number of compressed bytes of payload
that immediately follow the cfile card.  If the cfile card has only
three arguments, that means the payload is the complete content of the
artifact.  If the cfile card has four arguments, then the payload is a
delta and the second argument is the ID of another artifact that is the
source of the delta and the third argument is the original size of the
delta artifact.</p>

<p>Unlike file cards, cfile cards are only sent in one direction during a
clone from server to client for clone protocol version "3" or greater.</p>

<h4>3.3.3 Private artifacts</h4>

<p>"Private" content consist of artifacts that are not normally synced.
However, private content will be synced when the
the [/help?cmd=sync|fossil sync] command includes the "--private" option.
</p>

<p>Private content is marked by a "private" card:

<blockquote>
<b>private</b>
</blockquote>

<p>The private card has no arguments and must directly precede a
file card that contains the private content.</p>

<h4>3.3.4 Unversioned File Cards</h4>

<p>Unversioned content is sent in both directions (client to server and
server to client) using "uvfile" cards in the following format:

<blockquote>
<b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i>
</blockquote>

<p>The <i>name</i> field is the name of the unversioned file.  The
<i>mtime</i> is the last modification time of the file in seconds
since 1970.  The <i>hash</i> field is the hash of the content
for the unversioned file, or "<b>-</b>" for deleted content.
The <i>size</i> field is the (uncompressed) size of the content
in bytes.  The <i>flags</i> field is an integer which is interpreted
as an array of bits.  The 0x0004 bit of <i>flags</i> indicates that
the <i>content</i> is to be omitted.  The content might be omitted if
it is too large to transmit, or if the sender merely wants to update the
modification time of the file without changing the files content.
The <i>content</i> is the (uncompressed) content of the file.

<p>The receiver should only accept the uvfile card if the hash and
size match the content and if the mtime is newer than any existing
instance of the same file held by the receiver.  The sender will not
normally transmit a uvfile card unless all these constraints are true,
but the receiver should double-check.

<p>A server should only accept uvfile cards if the login user has
the "y" write-unversioned permission.

<p>Servers send uvfile cards in response to uvgimme cards received from
the client.  Clients send uvfile cards when they determine that the server
needs the content based on uvigot cards previously received from the server.

<h3>3.4 Push and Pull Cards</h3>

<p>Among the first cards in a client-to-server message are
the push and pull cards.  The push card tells the server that
the client is pushing content.  The pull card tells the server
that the client wants to pull content.  In the event of a sync,
both cards are sent.  The format is as follows:</p>

<blockquote>
<b>push</b> <i>servercode projectcode</i><br>
<b>pull</b> <i>servercode projectcode</i>
</blockquote>

<p>The <i>servercode</i> argument is the repository ID for the
client.  The <i>projectcode</i> is the identifier
of the software project that the client repository contains.
The projectcode for the client and server must match in order
for the transaction to proceed.</p>

<p>The server will also send a push card back to the client
during a clone.  This is how the client determines what project
code to put in the new repository it is constructing.</p>

<p>The <i>servercode</i> argument is currently unused.

<h3>3.5 Clone Cards</h3>

<p>A clone card works like a pull card in that it is sent from
client to server in order to tell the server that the client
wants to pull content.  The clone card comes in two formats.  Older
clients use the no-argument format and newer clients use the
two-argument format.</p>

<blockquote>
<b>clone</b><br>
<b>clone</b> <i>protocol-version sequence-number</i>
</blockquote>

<h4>3.5.1 Protocol 3</h4>

<p>The latest clients send a two-argument clone message with a
protocol version of "3".   (Future versions of Fossil might use larger
protocol version numbers.)  Version "3" of the protocol enhanced version
"2" by introducing the "cfile" card which is intended to speed up clone
operations.  Instead of sending "file" cards, the server will send "cfile"
cards</p>

<h4>3.5.2 Protocol 2</h4>

<p>The sequence-number sent is the number
of artifacts received so far.  For the first clone message, the
sequence number is 0.  The server will respond by sending file
cards for some number of artifacts up to the maximum message size.

<p>The server will also send a single "clone_seqno" card to the client
so that the client can know where the server left off.

<blockquote>
<b>clone_seqno</b>  <i>sequence-number</i>
</blockquote>

<p>The clone message in subsequent HTTP requests for the same clone
operation will use the sequence-number from the
clone_seqno of the previous reply.</p>

<p>In response to an initial clone message, the server also sends the client
a push message so that the client can discover the projectcode for
this project.</p>

<h4>3.5.3 Legacy Protocol</h4>

<p>Older clients send a clone card with no argument.  The server responds
to a blank clone card by sending an "igot" card for every artifact in the
repository.  The client will then issue "gimme" cards to pull down all the
content it needs.

<p>The legacy protocol works well for smaller repositories (50MB with 50,000
artifacts) but is too slow and unwieldy for larger repositories.
The version 2 protocol is an effort to improve performance.  Further
performance improvements with higher-numbered clone protocols are
possible in future versions of Fossil.

<h3>3.6 Igot Cards</h3>

<p>An igot card can be sent from either client to server or from
server to client in order to indicate that the sender holds a copy
of a particular artifact.  The format is:</p>

<blockquote>
<b>igot</b> <i>artifact-id</i> ?<i>flag</i>?
</blockquote>

<p>The first argument of the igot card is the ID of the artifact that
the sender possesses.
The receiver of an igot card will typically check to see if
it also holds the same artifact and if not it will request the artifact
using a gimme card in either the reply or in the next message.</p>

<p>If the second argument exists and is "1", then the artifact
identified by the first argument is private on the sender and should
be ignored unless a "--private" [/help?cmd=sync|sync] is occurring.

<p>The name "igot" comes from the English slang expression "I got" meaning
"I have".

<h4>3.6.1 Unversioned Igot Cards</h4>

<p>Zero or more "uvigot" cards are sent from server to client when
synchronizing unversioned content.  The format of a uvigot card is
as follows:

<blockquote>
<b>uvigot</b> <i>name mtime hash size</i>
</blockquote>

<p>The <i>name</i> argument is the name of an unversioned file.
The <i>mtime</i> is the last modification time of the unversioned file
in seconds since 1970.
The <i>hash</i> is the SHA1 or SHA3-256 hash of the unversioned file
content, or "<b>-</b>" if the file has been deleted.
The <i>size</i> is the uncompressed size of the file in bytes.

<p>When the server sees a "pragma uv-hash" card for which the hash
does not match, it sends uvigot cards for every unversioned file that it
holds.  The client will use this information to figure out which
unversioned files need to be synchronized.
The server might also send a uvigot card when it receives a uvgimme card
but its reply message size is already oversized and hence unable to hold
the usual uvfile reply.

<p>When a client receives a "uvigot" card, it checks to see if the
file needs to be transferred from client to server or from server to client.
If a client-to-server transmission is needed, the client schedules that
transfer to occur on a subsequent HTTP request.  If a server-to-client
transfer is needed, then the client sends a "uvgimme" card back to the
server to request the file content.

<h3>3.7 Gimme Cards</h3>

<p>A gimme card is sent from either client to server or from server
to client.  The gimme card asks the receiver to send a particular
artifact back to the sender.  The format of a gimme card is this:</p>

<blockquote>
<b>gimme</b> <i>artifact-id</i>
</blockquote>

<p>The argument to the gimme card is the ID of the artifact that
the sender wants.  The receiver will typically respond to a
gimme card by sending a file card in its reply or in the next
message.</p>

<p>The "gimme" name means "give me".  The imperative "give me" is
pronounced as if it were a single word "gimme" in some dialects of 
English (including the dialect spoken by the original author of Fossil).

<h4>3.7.1 Unversioned Gimme Cards</h4>

<p>Sync synchronizing unversioned content, the client may send "uvgimme"
cards to the server.  A uvgimme card requests that the server send
unversioned content to the client.  The format of a uvgimme card is
as follows:

<blockquote>
<b>uvgimme</b> <i>name</i>
</blockquote>

<p>The <i>name</i> is the name of the unversioned file found on the
server that the client would like to have.  When a server sees a
uvgimme card, it normally responses with a uvfile card, though it might
also send another uvigot card if the HTTP reply is already oversized.

<h3>3.8 Cookie Cards</h3>

<p>A cookie card can be used by a server to record a small amount
of state information on a client.  The server sends a cookie to the
client.  The client sends the same cookie back to the server on
its next request.  The cookie card has a single argument which
is its payload.</p>

<blockquote>
<b>cookie</b> <i>payload</i>
</blockquote>

<p>The client is not required to return the cookie to the server on
its next request.  Or the client might send a cookie from a different
server on the next request.  So the server must not depend on the
cookie and the server must structure the cookie payload in such
a way that it can tell if the cookie it sees is its own cookie or
a cookie from another server.  (Typically the server will embed
its servercode as part of the cookie.)</p>

<h3>3.9 Request-Configuration Cards</h3>

<p>A request-configuration or "reqconfig" card is sent from client to
server in order to request that the server send back "configuration"
data.  "Configuration" data is information about users or website
appearance or other administrative details which are not part of the
persistent and versioned state of the project.  For example, the "name"
of the project, the default Cascading Style Sheet (CSS) for the web-interface,
and the project logo displayed on the web-interface are all configuration
data elements.

<p>The reqconfig card is normally sent in response to the
"fossil configuration pull" command.  The format is as follows:

<blockquote>
<b>reqconfig</b> <i>configuration-name</i>
</blockquote>

<p>As of 2018-06-04, the configuration-name must be one of the
following values:

<table border=0 align="center">
<tr><td valign="top">
<ul>
<li> css
<li> header

<li> @reportfmt
<li> @user
<li> @concealed
<li> @shun
</ul></td></tr>
</table>

<p>New configuration-names are likely to be added in future releases of
Fossil.  If the server receives a configuration-name that it does not
understand, the entire reqconfig card is silently ignored.  The reqconfig
card might also be ignored if the user lacks sufficient privilege to
access the requested information.

<p>The configuration-names that begin with an alphabetic character refer
to values in the "config" table of the server database.  For example,
the "logo-image" configuration item refers to the project logo image
that is configured on the Admin page of the [./webui.wiki | web-interface].
The value of the configuration item is returned to the client using a
"config" card.

<p>If the configuration-name begins with "@", that refers to a class of
values instead of a single value.  The content of these configuration items
is returned in a "config" card that contains pure SQL text that is
intended to be evaluated by the client.

<p>The @user and @concealed configuration items contain sensitive information
and are ignored for clients without sufficient privilege.

<h3>3.10 Configuration Cards</h3>

<p>A "config" card is used to send configuration information from client
to server (in response to a "fossil configuration push" command) or
from server to client (in response to a "fossil configuration pull" or
"fossil clone" command).  The format is as follows:

<blockquote>
<b>config</b> <i>configuration-name size</i> <b>\n</b> <i>content</i>
</blockquote>

<p>The server will only accept a config card if the user has
"Admin" privilege.  A client will only accept a config card if
it had sent a corresponding reqconfig card in its request.

<p>The content of the configuration item is used to overwrite the
corresponding configuration data in the receiver.

<h3>3.11 Pragma Cards</h3>

<p>The client may try to influence the behavior of the server by
issuing a pragma card:

<blockquote>
<b>pragma</i> <i>name value...</i>
</blockquote>

<p>The "pragma" card has at least one argument which is the pragma name.
The pragma name defines what the pragma does.
A pragma might have zero or more "value" arguments
depending on the pragma name.

<p>New pragma names may be added to the protocol from time to time
in order to enhance the capabilities of Fossil.
Unknown pragmas are silently ignored, for backwards compatibility.

<p>The following are the known pragma names as of 2019-06-30:

<ol>
<li><p><b>send-private</b>
<p>The send-private pragma instructs the server to send all of its
private artifacts to the client.  The server will only obey this
request if the user has the "x" or "Private" privilege.

<li><p><b>send-catalog</b>
<p>The send-catalog pragma instructs the server to transmit igot
cards for every known artifact.  This can help the client and server
to get back in synchronization after a prior protocol error.  The
"--verily" option to the [/help?cmd=sync|fossil sync] command causes
the send-catalog pragma to be transmitted.</p>

<li><p><b>uv-hash</b> <i>HASH</i>
<p>The uv-hash pragma is sent from client to server to provoke a
synchronization of unversioned content.  The <i>HASH</i> is a SHA1
hash of the names, modification times, and individual hashes of all
unversioned files on the client.  If the unversioned content hash
from the client does not match the unversioned content hash on the
server, then the server will reply with either a "pragma uv-push-ok"
or "pragma uv-pull-only" card followed by one "uvigot" card for
each unversioned file currently held on the server.  The collection
of "uvigot" cards sent in response to a "uv-hash" pragma is called
the "unversioned catalog".  The client will used the unversioned
catalog to figure out which files (if any) need to be synchronized
between client and server and send appropriate "uvfile" or "uvgimme"
cards on the next HTTP request.</p>

<p>If a client sends a uv-hash pragma and does not receive back
either a uv-pull-only or uv-push-ok pragma, that means that the
content on the server exactly matches the content on the client and
no further synchronization is required.

<li><p><b>uv-pull-only</b></i>
<p>A server sends the uv-pull-only pragma to the client in response
to a uv-hash pragma with a mismatched content hash argument.  This
pragma indicates that there are differences in unversioned content
between the client and server but that content can only be transferred
from server to client.  The server is unwilling to accept content from
the client because the client login lacks the "write-unversioned"
permission.</p>

<li><p><b>uv-push-ok</b></i>
<p>A server sends the uv-push-ok pragma to the client in response
to a uv-hash pragma with a mismatched content hash argument.  This
pragma indicates that there are differences in unversioned content
between the client and server and that content can be transferred
in either direction.  The server is willing to accept content from
the client because the client login has the "write-unversioned"
permission.</p>

<li><p><b>ci-lock</b> <i>CHECKIN-HASH CLIENT-ID</i></p>
<p>A client sends the "ci-lock" pragma to the server to indicate
that it is about to add a new check-in as a child of the
CHECKIN-HASH check-in and on the same branch as CHECKIN-HASH.
If some other client has already indicated that it was also
trying to commit against CHECKIN-HASH, that indicates that a
fork is about to occur, and the server will reply with
a "ci-lock-fail" pragma (see below).  Check-in locks
automatically expire when the check-in actually occurs, or
after a timeout (currently one minute but subject to change).

<li><p><b>ci-lock-fail</b> <i>LOGIN MTIME</i></p>
<p>When a server receives two or more "ci-lock" pragma messages
for the same check-in but from different clients, the second a
subsequent ci-lock will provoke a ci-lock-fail pragma in the
reply to let the client know that it if continues with the
check-in it will likely generate a fork.  The LOGIN and MTIME
arguments are intended to provide information to the client to
help it generate a more useful error message.
</p>

<li><p><b>ci-unlock</b> <i>CLIENT-ID</i></p>
<p>A client sends the "ci-unlock" pragma to the server after
a successful commit.  This instructs the server to release
any lock on any check-in previously held by that client.
The ci-unlock pragma helps to avoid false-positive lock warnings
that might arise if a check-in is aborted and then restarted
on a branch.
</ol>

<h3>3.12 Comment Cards</h3>

<p>Any card that begins with "#" (ASCII 0x23) is a comment card and
is silently ignored.</p>

<h3>3.13 Message and Error Cards</h3>

<p>If the server discovers anything wrong with a request, it generates
an error card in its reply.  When the client sees the error card,
it displays an error message to the user and aborts the sync
operation.  An error card looks like this:</p>

<blockquote>
<b>error</b> <i>error-message</i>
</blockquote>

<p>The error message is English text that is encoded in order to
be a single token.
A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73).  A
newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E).  A backslash
(ASCII 0x5C) is represented as two backslashes "\\".  Apart from
space and newline, no other whitespace characters nor any
unprintable characters are allowed in
the error message.</p>

<p>The server can also send a message card that also prints a
message on the client console, but which is not an error:

<blockquote>
<b>message</b> <i>message-text</i>
</blockquote>

<p>The message-text uses the same format as an error message.

<h3>3.14 Unknown Cards</h3>

<p>If either the client or the server sees a card that is not
described above, then it generates an error and aborts.</p>

<h2>4.0 Phantoms And Clusters</h2>

<p>When a repository knows that an artifact exists and knows the ID of
that artifact, but it does not know the artifact content, then it stores that
artifact as a "phantom".  A repository will typically create a phantom when
it receives an igot card for an artifact that it does not hold or when it
receives a file card that references a delta source that it does not
hold.  When a server is generating its reply or when a client is
generating a new request, it will usually send gimme cards for every
phantom that it holds.</p>

<p>A cluster is a special artifact that tells of the existence of other
artifacts.  Any artifact in the repository that follows the syntactic rules
of a cluster is considered a cluster.</p>

<p>A cluster is line oriented.  Each line of a cluster
is a card.  The cards are separated by the newline ("\n") character.
Each card consists of a single character card type, a space, and a
single argument.  No extra whitespace and no trailing or leading
whitespace is allowed.  All cards in the cluster must occur in
strict lexicographical order.</p>

<p>A cluster consists of one or more "M" cards followed by a single
"Z" card.  Each M card holds an argument which is an artifact ID for an
artifact in the repository.  The Z card has a single argument which is the
lower-case hexadecimal representation of the MD5 checksum of all
preceding M cards up to and included the newline character that
occurred just before the Z that starts the Z card.</p>

<p>Any artifact that does not match the specifications of a cluster
exactly is not a cluster.  There must be no extra whitespace in
the artifact.  There must be one or more M cards.  There must be a
single Z card with a correct MD5 checksum.  And all cards must
be in strict lexicographical order.</p>

<h3>4.1 The Unclustered Table</h3>

<p>Every repository maintains a table named "<b>unclustered</b>"
which records the identity of every artifact and phantom it holds that is not
mentioned in a cluster.  The entries in the unclustered table can
be thought of as leaves on a tree of artifacts.  Some of the unclustered
artifacts will be other clusters.  Those clusters may contain other clusters,
which might contain still more clusters, and so forth.  Beginning
with the artifacts in the unclustered table, one can follow the chain
of clusters to find every artifact in the repository.</p>

<h2>5.0 Synchronization Strategies</h2>

<h3>5.1 Pull</h3>

<p>A typical pull operation proceeds as shown below.  Details
of the actual implementation may very slightly but the gist of
a pull is captured in the following steps:</p>

<ol>
<li>The client sends login and pull cards.
<li>The client sends a cookie card if it has previously received a cookie.
<li>The client sends gimme cards for every phantom that it holds.
<hr>
<li>The server checks the login password and rejects the session if

<li>The client adds the content of file cards to its repository.
<li>The client creates a phantom for every igot card in the server reply
that mentions an artifact that the client does not possess.
<li>The client creates a phantom for the delta source of file cards when
the delta source is an artifact that the client does not possess.
</ol>

<p>These ten steps represent a single HTTP round-trip request.
The first three steps are the processing that occurs on the client
to generate the request.  The middle four steps are processing
that occurs on the server to interpret the request and generate a
reply.  And the last three steps are the processing that the
client does to interpret the reply.</p>

<p>During a pull, the client will keep sending HTTP requests
until it holds all artifacts that exist on the server.</p>

<p>Note that the server tries
to limit the size of its reply message to something reasonable
(usually about 1MB) so that it might stop sending file cards as
described in step (6) if the reply becomes too large.</p>

<p>Step (5) is the only way in which new clusters can be created.
By only creating clusters on the server, we hope to minimize the
amount of overlap between clusters in the common configuration where
there is a single server and many clients.  The same synchronization
protocol will continue to work even if there are multiple servers
or if servers and clients sometimes change roles.  The only negative
effects of these unusual arrangements is that more than the minimum
number of clusters might be generated.</p>

<h3>5.2 Push</h3>

<p>A typical push operation proceeds roughly as shown below.  As
with a pull, the actual implementation may vary slightly.</p>

<ol>
<li>The client sends login and push cards.
<li>The client sends file cards for any artifacts that it holds that have
never before been pushed - artifacts that come from local check-ins.
<li>If this is the second or later cycle in a push, then the
client sends file cards for any gimme cards that the server sent

it does not possess.
<li>The server issues gimme cards for all phantoms.
<hr>
<li>The client remembers the gimme cards from the server so that it
can generate file cards in reply on the next cycle.
</ol>

<p>As with a pull, the steps of a push operation repeat until the
server knows all artifacts that exist on the client.  Also, as with
pull, the client attempts to keep the size of the request from
growing too large by suppressing file cards once the
size of the request reaches 1MB.</p>

<h3 id="sync">5.3 Sync</h3>

<p>A sync is just a pull and a push that happen at the same time.
The first three steps of a pull are combined with the first five steps
of a push.  Steps (4) through (7) of a pull are combined with steps
(5) through (8) of a push.  And steps (8) through (10) of a pull
are combined with step (9) of a push.</p>

<h3>5.4 Unversioned File Sync</h3>

<p>"Unversioned files" are files held in the repository
where only the most recent version of the file is kept rather than
the entire change history.  Unversioned files are intended to be
used to store ephemeral content, such as compiled binaries of the
most recent release.

<p>Unversioned files are identified by name and timestamp (mtime).
Only the most recent version of each file (the version with
the largest mtime value) is retained.

<p>Unversioned files are synchronized using the
[/help?cmd=unversioned|fossil unversioned sync] command.

<p>A schematic of an unversioned file synchronization is as follows:

<ol>
<li>The client sends a "pragma uv-hash" card to the server.  The argument
    to the uv-hash pragma is a hash of all filesnames, mtimes, and
    content hashes for the unversioned files held by the client.
    <hr>
<li>If the unversioned content hash from the client matches the unversioned

    then sends appropriate "uvgimme" or "uvfile" cards back to the
    server.
    <hr>
<li>The server updates its unversioned file store with received "uvfile"
    cards and answers "uvgimme" cards with "uvfile" cards in its reply.
</ol>

<p>The last two steps might be repeated multiple
times if there is more unversioned content to be transferred than will
fit comfortably in a single HTTP request.

<h2>6.0 Summary</h2>

<p>Here are the key points of the synchronization protocol:</p>

<ol>
<li>The client sends one or more PUSH HTTP requests to the server.
    The request and reply content type is "application/x-fossil".
<li>HTTP request content is compressed using zlib.
<li>The content of request and reply consists of cards with one
    card per line.

cluster and send igot messages for those artifacts.
<li>Repositories keep track of all the phantoms they hold and send
gimme messages for those artifacts.
</ol>

<h2>7.0 Troubleshooting And Debugging Hints</h2>

<p>
If you run the [/help?cmd=sync|fossil sync] command
(or [/help?cmd=pull|pull] or [/help?cmd=push|push] or
[/help?cmd=clone|clone]) with the --httptrace option, Fossil
will keep a copy of each HTTP request and reply in files
named:
<ul>
<li> <tt>http-request-</tt><i>N</i><tt>.txt</tt>
<li> <tt>http-reply-</tt><i>N</i><tt>.txt</tt>
</ul>

<p>In the above, <i>N</i> is an integer that increments with each
round-trip.  If you are having trouble on the server side,
you can run the "[/help?cmd=test-http|fossil test-http]" command in a
debugger using one the "http-request-N.txt" files as input and
single step through the processing performed by the server.

<p>The "--transport-command CMD" option on [/help?cmd=sync|fossil sync]
(and similar) causes the external program "CMD" to be used to move
the sync message to the server and retrieve the sync reply.  The
CMD is given three arguments:
<ol>
<li> The URL of the server
<li> The name of a temporary file that contains the output-bound sync
     protocol text, with the HTTP headers
<li> The name of a temporary file into which the CMD should write the
     reply sync protocol text, again without any HTTP headers
</ol>

<p>In a complex debugging situation, you can run the command
"fossil sync --transport-command ./debugging_script" where
"debugging_script" is some script of your own that invokes
the anomolous behavior your are trying to debug.