Fossil: TH1 Scripts

TH1 is a very small scripting language used to help generate web-page content in Fossil.

Origins

TH1 began as a minimalist re-implementation of the Tcl scripting language. There was a need to test the SQLite library on Symbian phones, but at that time all of the test cases for SQLite were written in Tcl and Tcl could not be easily compiled on the SymbianOS. So TH1 was developed as a cut-down version of Tcl that would facilitate running the SQLite test scripts on SymbianOS.

Fossil was first being designed at about the same time that TH1 was being developed for testing SQLite on SymbianOS. Early prototypes of Fossil were written in pure Tcl. But as the development shifted toward the use of C-code, the need arose to have a Tcl-like scripting language to help with code generation. TH1 was small and light-weight and used minimal resources and seemed ideally suited for the task.

The name "TH1" stands "Test Harness 1", since that was its original purpose.

Overview

TH1 is a string-processing language. All values are strings. Any numerical operations are accomplished by converting from string to numeric, performing the computation, then converting the result back into a string. (This might seem inefficient, but it is faster than people imagine, and numeric computations do not come up very often for the kinds of work that TH1 does, so it has never been a factor.)

A TH1 script consists of a sequence of commands. Each command is terminated by the first unescaped newline or ";" character. The text of the command (excluding the newline or semicolon terminator) is broken into space-separated tokens. The first token is the command name and subsequent tokens are the arguments. In this sense, TH1 syntax is similar to the familiar command-line shell syntax.

A token is any sequence of characters other than whitespace and semicolons. Text inside double-quotes is a single token even if it includes whitespace and semicolons. Text within {...} pairs is also a single token, which is useful because curly braces are easier to “pair” and nest properly than double-quotes.

The nested {...} form of tokens is important because it allows TH1 commands to have an appearance similar to C/C++. It is important to remember, though, that a TH1 script is really just a list of text commands, not a context-free language with a grammar like C/C++. This can be confusing to long-time C/C++ programmers because TH1 does look a lot like C/C++, but the semantics of TH1 are closer to FORTH or Lisp than they are to C.

Consider the if command in TH1.

if {$current eq "dev"} {
  puts "hello"
} else {
  puts "world"
}

The example above is a single command. The first token, and the name of the command, is if. The second token is $current eq "dev" - an expression. (The outer {...} are removed from each token by the command parser.) The third token is the puts "hello", with its whitespace and newlines. The fourth token is else and the fifth and last token is puts "world".

The if command evaluates its first argument (the second token) as an expression, and if that expression is true, evaluates its second argument (the third token) as a TH1 script. If the expression is false and the third argument is else, then the fourth argument is evaluated as a TH1 expression.

So, you see, even though the example above spans five lines, it is really just a single command.

All of this also explains the emphasis on unescaped characters above: the curly braces { } are string quoting characters in Tcl/TH1, not block delimiters as in C. This is how we can have a command that extends over multiple lines. It is also why the else keyword must be cuddled up with the closing brace for the if clause's scriptlet. The following is invalid Tcl/TH1:

if {$current eq "dev"} {
  puts "hello"
}
else {
  puts "world"
}

If you try to run this under either Tcl or TH1, the interpreter will tell you that there is no else command, because with the newline on the third line, you terminated the if command.

Occasionally in Tcl/TH1 scripts, you may need to use a backslash at the end of a line to allow a command to extend over multiple lines without being considered two separate commands. Here's an example from one of Fossil's test scripts:

return [lindex [regexp -line -inline -nocase -- \
    {^uuid:\s+([0-9A-F]{40}) } [eval [getFossilCommand \
    $repository "" info trunk]]] end]

Those backslashes allow the command to wrap nicely within a standard terminal width while telling the interpreter to consider those three lines as a single command.

Summary of Core TH1 Commands

The original Tcl language (after which TH1 is modeled) has a very rich repertoire of commands. TH1, as it is designed to be minimalist and embedded has a greatly reduced command set. The following bullets summarize the commands available in TH1:

array exists VARNAME
array names VARNAME
break
catch SCRIPT ?VARIABLE?
continue
error ?STRING?
expr EXPR
for INIT-SCRIPT TEST-EXPR NEXT-SCRIPT BODY-SCRIPT
foreach VARIABLE-LIST VALUE-LIST BODY-SCRIPT
if EXPR SCRIPT (elseif EXPR SCRIPT)* ?else SCRIPT?
info commands
info exists VARNAME
info vars
lappend VARIABLE TERM ...
lindex LIST INDEX
list ARG ...
llength LIST
lsearch LIST STRING
proc NAME ARG-LIST BODY-SCRIPT
rename OLD NEW
return ?-code CODE? ?VALUE?
set VARNAME VALUE
string compare STR1 STR2
string first NEEDLE HAYSTACK ?START-INDEX?
string index STRING INDEX
string is CLASS STRING
string last NEEDLE HAYSTACK ?START-INDEX?
string match PATTERN STRING
string length STRING
string range STRING FIRST LAST
string repeat STRING COUNT
unset VARNAME
uplevel ?LEVEL? SCRIPT
upvar ?FRAME? OTHERVAR MYVAR ?OTHERVAR MYVAR?

All of the above commands work as in the original Tcl. Refer to the Tcl documentation for details.

Summary of Core TH1 Variables

tcl_platform(engine) -- This will always have the value "TH1".
tcl_platform(platform) -- This will have the value "windows" or "unix".
th_stack_trace -- This will contain error stack information.

TH1 Extended Commands

There are many new commands added to TH1 and used to access the special features of Fossil. The following is a summary of the extended commands:

anoncap
anycap
artifact
builtin_request_js
capexpr
captureTh1
cgiHeaderLine
checkout
combobox
copybtn
date
decorate
defHeader
dir
enable_htmlify
enable_output
encode64
getParameter
glob_match
globalState
hascap
hasfeature
html
htmlize
http
httpize
insertCsrf
linecount
markdown
nonce
puts
query
randhex
redirect
regexp
reinitialize
render
repository
searchable
setParameter
setting
stime
styleHeader
styleFooter
styleScript
submenu
tclEval
tclExpr
tclInvoke
tclIsSafe
tclMakeSafe
tclReady
trace
unversioned content
unversioned list
utime
verifyCsrf
verifyLogin
wiki

Each of the commands above is documented by a block comment above their implementation in the th_main.c or th_tcl.c source files.

All commands starting with "tcl", with the exception of "tclReady", require the Tcl integration subsystem be included at compile-time. Additionally, the "tcl" repository setting must be enabled at runtime in order to successfully make use of these commands.

TH1 anoncap Command

Deprecated: prefer capexpr instead.

anoncap STRING...

Returns true if the anonymous user has all of the capabilities listed in STRING.

TH1 anycap Command

Deprecated: prefer capexpr instead.

anycap STRING

Returns true if the current user user has any one of the capabilities listed in STRING.

TH1 artifact Command

artifact ID ?FILENAME?

Attempts to locate the specified artifact and return its contents. An error is generated if the repository is not open or the artifact cannot be found.

TH1 builtin_request_js Command

builtin_request_js NAME

NAME must be the name of one of the built-in javascript source files. This command causes that javascript file to be appended to the delivered document.

TH1 capexpr Command

capexpr CAPABILITY-EXPR

The capability expression is a list. Each term of the list is a cluster of capability letters. The overall expression is true if any one term is true. A single term is true if all letters within that term are true. Or, if the term begins with "!", then the term is true if none of the terms are true. Or, if the term begins with "@" then the term is true if all of the capability letters in that term are available to the "anonymous" user. Or, if the term is "*" then it is always true.

Examples:

capexpr {j o r}               True if any one of j, o, or r are available
capexpr {oh}                  True if both o and h are available
capexpr {@2 @3 4 5 6}         2 or 3 available for anonymous or one of
                              4, 5 or 6 is available for the user
capexpr L                     True if the user is logged in
capexpr !L                    True if the user is not logged in

The L pseudo-capability is intended only to be used on its own or with the ! prefix for implementing login/logout menus via the mainmenu site configuration option:

Login     /login        !L  {}
Logout    /logout        L  {}

i.e. if the user is logged in, show the "Logout" link, else show the "Login" link.

TH1 captureTh1 Command

captureTh1 STRING

Executes its single argument as TH1 code and captures any TH1-generated output as a string, which becomes the result of the function call. e.g. any puts calls made from that block will not generate any output, and instead their output will become part of the result string.

TH1 cgiHeaderLine Command

cgiHeaderLine line

Adds the specified line to the CGI header.

TH1 checkout Command

checkout ?BOOLEAN?

Return the fully qualified directory name of the current checkout or an empty string if it is not available. Optionally, it will attempt to find the current checkout, opening the configuration ("user") database and the repository as necessary, if the boolean argument is non-zero.

TH1 combobox Command

combobox NAME TEXT-LIST NUMLINES

Generates and emits an HTML combobox. NAME is both the name of the CGI parameter and the name of a variable that contains the currently selected value. TEXT-LIST is a list of possible values for the combobox. NUMLINES is 1 for a true combobox. If NUMLINES is greater than one then the display is a listbox with the number of lines given.

TH1 copybtn Command

copybtn TARGETID FLIPPED TEXT ?COPYLENGTH?

Output TEXT with a click-to-copy button next to it. Loads the copybtn.js Javascript module, and generates HTML elements with the following IDs:

TARGETID: The <span> wrapper around TEXT.
copy-TARGETID: The <span> for the copy button.

If the FLIPPED argument is non-zero, the copy button is displayed after TEXT.

The optional COPYLENGTH argument defines the length of the substring of TEXT copied to clipboard:

<= 0: No limit (default if the argument is omitted).
>= 3: Truncate TEXT after COPYLENGTH (single-byte) characters.
1: Use the "hash-digits" setting as the limit.
2: Use the length appropriate for URLs as the limit (defined at compile-time by FOSSIL_HASH_DIGITS_URL, defaults to 16).

TH1 date Command

date ?-local?

Return a string which is the current time and date. If the -local option is used, the date appears using localtime instead of UTC.

TH1 decorate Command

decorate STRING

Renders STRING as wiki content; however, only links are handled. No other markup is processed.

TH1 defHeader Command

defHeader

Returns the default page header.

TH1 dir Command

dir CHECKIN ?GLOB? ?DETAILS?

Returns a list containing all files in CHECKIN. If GLOB is given only the files matching the pattern GLOB within CHECKIN will be returned. If DETAILS is non-zero, the result will be a list-of-lists, with each element containing at least three elements: the file name, the file size (in bytes), and the file last modification time (relative to the time zone configured for the repository).

TH1 enable_htmlify Command

enable_htmlify
enable_htmlify ?TRACE-LABEL? BOOLEAN

By default, certain output from puts and similar commands is escaped for HTML. The first call form returns the current state of that feature: 1 for on and 0 for off. The second call form enables (non-0) or disables (0) that feature and returns the pre-call state of that feature (so that a second call can pass that value to restore it to its previous state). The optional TRACE-LABEL argument causes the TH1 tracing output (if enabled) to add a marker when the second form of this command is invoked, and includes that label and the boolean argument's value in the trace. If tracing is disabled, that argument has no effect.

TH1 enable_output Command

enable_output BOOLEAN

Enable or disable sending output when the combobox, copybtn, puts, or wiki commands are used.

TH1 encode64 Command

encode64 STRING

Encode the specified string using Base64 and return the result.

TH1 getParameter Command

getParameter NAME ?DEFAULT?

Returns the value of the specified query parameter or the specified default value when there is no matching query parameter.

TH1 glob_match Command

glob_match ?-one? ?--? patternList string

Checks the string against the specified glob pattern -OR- list of glob patterns and returns non-zero if there is a match.

TH1 globalState Command

globalState NAME ?DEFAULT?

Returns a string containing the value of the specified global state variable -OR- the specified default value. The supported items are:

checkout -- Active local checkout directory, if any.
configuration -- Active configuration database file name, if any.
executable -- Fully qualified executable file name.
flags -- TH1 initialization flags.
log -- Error log file name, if any.
repository -- Active local repository file name, if any.
top -- Base path for the active server instance, if applicable.
user -- Active user name, if any.
vfs -- SQLite VFS in use, if overridden.

Attempts to query for unsupported global state variables will result in a script error. Additional global state variables may be exposed in the future.

TH1 hascap Command

Deprecated: prefer capexpr instead.

hascap STRING...

Returns true if the current user has all of the capabilities listed in STRING.

TH1 hasfeature Command

hasfeature STRING

Returns true if the binary has the given compile-time feature enabled. The possible features are:

ssl -- Support for the HTTPS transport.
legacyMvRm -- Support for legacy mv/rm command behavior.
execRelPaths -- Use relative paths with external diff/gdiff.
th1Docs -- Support for TH1 in embedded documentation.
th1Hooks -- Support for TH1 command and web page hooks.
tcl -- Support for Tcl integration.
useTclStubs -- Tcl stubs enabled in the Tcl headers.
tclStubs -- Uses Tcl stubs (i.e. linking with stubs library).
tclPrivateStubs -- Uses Tcl private stubs (i.e. header-only).
json -- Support for the JSON APIs.
markdown -- Support for Markdown documentation format.
unicodeCmdLine -- The command line arguments are Unicode.
dynamicBuild -- Dynamically linked to libraries.
mman -- Uses POSIX memory APIs from "sys/mman.h".
see -- Uses the SQLite Encryption Extension.

Specifying an unknown feature will return a value of false, it will not raise a script error.

TH1 html Command

html STRING

Outputs the STRING escaped for HTML.

TH1 htmlize Command

htmlize STRING

Escape all characters of STRING which have special meaning in HTML. Returns the escaped string.

TH1 http Command

http ?-asynchronous? ?--? url ?payload?

Performs an HTTP or HTTPS request for the specified URL. If a payload is present, it will be interpreted as text/plain and the POST method will be used; otherwise, the GET method will be used. Upon success, if the -asynchronous option is used, an empty string is returned as the result; otherwise, the response from the server is returned as the result. Synchronous requests are not currently implemented.

TH1 httpize Command

httpize STRING

Escape all characters of STRING which have special meaning in URI components. Returns the escaped string.

TH1 insertCsrf Command

insertCsrf

While rendering a form, call this command to add the Anti-CSRF token as a hidden element of the form.

TH1 linecount Command

linecount STRING MAX MIN

Returns one more than the number of \n characters in STRING. But never returns less than MIN or more than MAX.

TH1 markdown Command

markdown STRING

Renders the input string as markdown. The result is a two-element list. The first element contains the body, rendered as HTML. The second element is the text-only title string.

TH1 nonce Command

nonce

Returns the value of the cryptographic nonce for the request being processed.

TH1 puts Command

puts STRING

Outputs the STRING unchanged, where "unchanged" might, depending on the context, mean "with some characters escaped for HTML."

TH1 query Command

query ?-nocomplain? SQL CODE

Runs the SQL query given by the SQL argument. For each row in the result set, run CODE.

In SQL, parameters such as $var are filled in using the value of variable "var". Result values are stored in variables with the column name prior to each invocation of CODE.

TH1 randhex Command

randhex N

Returns a string of N*2 random hexadecimal digits with N<50. If N is omitted, use a value of 10.

TH1 redirect Command

redirect URL ?withMethod?

Issues an HTTP redirect to the specified URL and then exits the process. By default, an HTTP status code of 302 is used. If the optional withMethod argument is present and non-zero, an HTTP status code of 307 is used, which should force the user agent to preserve the original method for the request (e.g. GET, POST) instead of (possibly) forcing the user agent to change the method to GET.

TH1 regexp Command

regexp ?-nocase? ?--? exp string

Checks the string against the specified regular expression and returns non-zero if it matches. If the regular expression is invalid or cannot be compiled, an error will be generated.

TH1 reinitialize Command

reinitialize ?FLAGS?

Reinitializes the TH1 interpreter using the specified flags.

TH1 render Command

render STRING

Renders the TH1 template and writes the results.

TH1 repository Command

repository ?BOOLEAN?

Returns the fully qualified file name of the open repository or an empty string if one is not currently open. Optionally, it will attempt to open the repository if the boolean argument is non-zero.

TH1 searchable Command

searchable STRING...

Return true if searching in any of the document classes identified by STRING is enabled for the repository and user has the necessary capabilities to perform the search. The possible document classes are:

c -- Check-in comments
d -- Embedded documentation
t -- Tickets
w -- Wiki

To be clear, only one of the document classes identified by each STRING needs to be searchable in order for that argument to be true. But all arguments must be true for this routine to return true. Hence, to see if ALL document classes are searchable:

if {[searchable c d t w]} {...}

But to see if ANY document class is searchable:

if {[searchable cdtw]} {...}

This command is useful for enabling or disabling a "Search" entry on the menu bar.

TH1 setParameter Command

setParameter NAME VALUE

Sets the value of the specified query parameter.

TH1 setting Command

setting name

Gets and returns the value of the specified setting.

TH1 stime Command

stime

Returns the number of microseconds of CPU time consumed by the current process in system space.

TH1 styleHeader Command

styleHeader TITLE

Render the configured style header for the selected skin.

TH1 styleFooter Command

styleFooter

Render the configured style footer for the selected skin.

TH1 styleScript Command

styleScript

Render the configured JavaScript for the selected skin.

TH1 submenu Command

submenu link LABEL URL

Add hyperlink to the submenu of the current page.

TH1 tclEval Command

This command requires the Tcl integration feature.

tclEval arg ?arg ...?

Evaluates the Tcl script and returns its result verbatim. If a Tcl script error is generated, it will be transformed into a TH1 script error. The Tcl interpreter will be created automatically if it has not been already.

TH1 tclExpr Command

This command requires the Tcl integration feature.

tclExpr arg ?arg ...?

Evaluates the Tcl expression and returns its result verbatim. If a Tcl script error is generated, it will be transformed into a TH1 script error. The Tcl interpreter will be created automatically if it has not been already.

TH1 tclInvoke Command

This command requires the Tcl integration feature.

tclInvoke command ?arg ...?

Invokes the Tcl command using the supplied arguments. No additional substitutions are performed on the arguments. The Tcl interpreter will be created automatically if it has not been already.

TH1 tclIsSafe Command

This command requires the Tcl integration feature.

tclIsSafe

Returns non-zero if the Tcl interpreter is "safe". The Tcl interpreter will be created automatically if it has not been already.

TH1 tclMakeSafe Command

This command requires the Tcl integration feature.

tclMakeSafe

Forces the Tcl interpreter into "safe" mode by removing all "unsafe" commands and variables. This operation cannot be undone. The Tcl interpreter will remain "safe" until the process terminates. The Tcl interpreter will be created automatically if it has not been already.

TH1 tclReady Command

tclReady

Returns true if the binary has the Tcl integration feature enabled and it is currently available for use by TH1 scripts.

TH1 trace Command

trace STRING

Generates a TH1 trace message if TH1 tracing is enabled.

TH1 unversioned content Command

unversioned content FILENAME

Attempts to locate the specified unversioned file and return its contents. An error is generated if the repository is not open or the unversioned file cannot be found.

TH1 unversioned list Command

unversioned list

Returns a list of the names of all unversioned files held in the local repository. An error is generated if the repository is not open.

TH1 utime Command

utime

Returns the number of microseconds of CPU time consumed by the current process in user space.

TH1 verifyCsrf Command

verifyCsrf

Before using the results of a form, first call this command to verify that the Anti-CSRF token is present and is valid. If the Anti-CSRF token is missing or is incorrect, that indicates a cross-site scripting attack. If the event of an attack is detected, an error message is generated and all further processing is aborted.

TH1 verifyLogin Command

verifyLogin

Returns non-zero if the specified user name and password represent a valid login for the repository.

TH1 wiki Command

wiki STRING

Renders STRING as wiki content.

Tcl Integration Commands

When the Tcl integration subsystem is enabled, several commands are added to the Tcl interpreter. They are used to allow Tcl scripts access to the Fossil functionality provided via TH1. The following is a summary of the Tcl commands:

th1Eval
th1Expr

Tcl th1Eval Command

This command requires the Tcl integration feature.

th1Eval arg

Evaluates the TH1 script and returns its result verbatim. If a TH1 script error is generated, it will be transformed into a Tcl script error.

Tcl th1Expr Command

This command requires the Tcl integration feature.

th1Expr arg

Evaluates the TH1 expression and returns its result verbatim. If a TH1 script error is generated, it will be transformed into a Tcl script error.