tmdoc::tmdoc 0.3 - Tcl Markdown processor tutorial

Detlef Groth, Caputh-Schwielowsee, Germany

2020-02-23

Abstract

In the last years reproducible research has gained focus from the research community. It is a type of literate programming (introduced by Donald Knuth in 1984, where code of a programming language is embedded with standard text documents, like LaTeX or Markdown documents. The code is evaluated and replaces it’s own code with it’s output. Alternatively the code chunks as well as the output result can be also hidden if needed. There exist several tools which allow this type of programming/documenting, for instance knitr or Sweave. Knitr is an engine for dynamic report generation especially for the statistical language R but as well supports other programming languages such as for instance Python 3, Perl, Haskell, Ruby and more than 20 others. Embedding of Tcl in the knitr framework is somehow difficult as it has no command line option to evaluate Tcl code on the fly. There might be workarounds by creating a Tcl command line application which allows this. I thought however, that the Tcl programming as a mature highly dynamic scripting language invented by John Ousterhout in 1988, does not need a huge framework to support literate programming. Therefore I decided to implement a literate programming package for Tcl just based on standard Tcl which is called tmdoc::tmdoc , short just tmdoc. In it’s current state tmdoc supports standard code chunks with options to display and hide code and code output, to embed inside text code evaluations and image creation. This basic subset of a literate programming environment should be sufficient to create nice technical and statistical reports, tutorials about Tcl packages and other types of documentation.

Table of Contents

Introduction

The Tcl package tmdoc::tmdoc which is both a standalone command line application and as well a Tcl package is a report generator and a literate programming tool for Tcl. It can capture the results and plots or images from data analysis and works well with other packages like Tk, or gdtcl. The current text is a short tutorial on how to do literate programming with Tcl using the package. Literate programming is a style of progamming where in a text document, shorter or longer code fragments of a programming language are embedded. The document is then “weaved”, the programming code is replaced with it’s output. The produced document can thereafter be further processed to other formats.The tmdoc::tmdoc package supports Markdown as documentation language. Using Markdown processors like pandoc or the Tcl package Markdown the output of the tmdoc processor can be converted finally to other document formats like pdf or html.

To give an example, below is the pipleine which was used to create the HTML document you are currently viewing:

tclsh tmdoc.tcl tmdoc-example.tmd > tmd.md
pandoc -i tmd.md -s --highlight-style pygments -c dgw.css  -o tmd.html
# example for a pdf generation
pandoc -i tmd.md -s --highlight-style pygments -c -o tmd.pdf

So basically the pipeline is the following:

      tmdoc        pandoc
TMD --------> MD ----------> HTML/PDF/DOCX

Now the mandatory “Hello World!” example. If you embed the following text into your Markdown document, tmdoc will produce, after converting the document from Markdown to HTML, the following output:

This is the input (without the leading four spaces, nessessary here to avoid interpretation by tmdoc):

    ```{tcl}
    puts "Hello World!"
    ```

Below is the output (creme, the Tcl code within the code chunk, light blue it’s output), the forward arrow indicates the return value of the current code block:

puts "Hello World!"
Hello World!

Code block and/or the output can be as well hidden using chunk arguments, that will be described later in more detail. Short code fragments can be directly embedded into the text. Here an example for this approach:

The curent date and time is: `tcl clock format [clock seconds] -format "%Y-%m-%d %H:%M"`

This is the output:

The curent date and time is: 2020-02-23 18:47

Short Tutorial

Now follows a short tutorial for the features of the tmdoc Markdown processor. For the code blocks the following color codes are used, grey is example Tcl code which is not evaluated by Tcl, it shows mainly how the code chunks looks in the Markdown document, in creme is Tcl code which is evaluated, in blue is the output of the creme tcl code before. Red code lines indicate code errors.

So here again the color codes:

gray: code examples which are not evaluated within this document
creme: code examples which are evaluated by Tcl
blue: result output of evaluated Tcl code
red: errors resulting from Tcl code evaluations

Standard Code Block

Standard code block are started with three tickmarts at the beginning of a line and in curly brace the string tcl. Separated by a space might follow several evaluation options which will be described later in more detail.

Here the Tcl-Markdown input (without the leading four spaces):

    ```{tcl echo=true}
    set x 1
    set a 1
    set a
    set x
    ```

Please note that the indentation is neccessary only for the example here, as tmdoc is processing tmdoc inside tmdoc. So in fact there should be no four spaces at the beginning in your code.

And here the resulting output:

set x 1
set a 1
set a
set x
==> 1

Only the last statement is returned. You should use puts if you need to output more than the last statement.

The puts Statement

You should can have several puts statements in your code which are printed before the final return value. The channel stdout is displayed as well in the results part. Also the -nonewline option can be used:

puts "Hello World 1"
puts "Hello World 2"
set x 3
puts "Hello World $x"
puts stdout "Hello World [incr x]"
puts -nonewline "Hello"
puts " World [incr x]"
puts -nonewline stdout "Hello"
puts stdout " World [incr x]"
# let's reset x to one
set x 1
Hello World 1
Hello World 2
Hello World 3
Hello World 4
Hello World 5
Hello World 6
==> 1

You can as well write into file channels as usually:

set out [open file.ext w 0600]
puts $out "Hello File World!"
close $out
# now display the content
set filename file.ext
if [catch {open $filename r} infh] {
    puts stderr "Cannot open $filename: $infh"
    exit
} else {
    while {[gets $infh line] >= 0} {
        puts stdout "file: $line"
    }   
}
file: Hello File World!

To supress any output, your last statment should return an empty string:

set a 5
set a ""

Alternatively you can supress the output at all by giving an option results=hide to the code chunk

    ```{tcl results=hide}
    set a 1
    ```
set a 1

There is now no output with 1 here to see as we used chunk option results=hide.

You can as well supress both showing the Tcl code and the output of the Tcl code by specifying results=hide and echo=false as shown in the following example:

    ```{tcl results=hide,echo=false}
    set a 2
    ```
    
   Our variable a has now a value of `tcl set a` although we don't know why as the programming code is hidden.

Our variable a has now a value of 2 although we don’t know why.

This feature can be useful if you don’t want to show in a final version of a text document how the computation was done, but only the results of this computation within the text, for instance to display date and used Tcl packages as can be seen at the very end of this document.

Inline Code

Let's look what is the value of x. The value of x is `tcl set x`. Is it one??

Let’s look what is the value of x. The value of x is 1. Is it one??

Yes it is!

Now let’s try two times inline code on one line:

The value of x is: `tcl incr x` and now `tcl incr x`!.

The value of x is: 2 and now 3!. So also this works. Don’t write too long Tcl statements directly in the text. You should perform your major computations in the code blocks. Please note as well, that the inline code chunks can’t wrap over multiple lines. So opening and closing backtick must be on the same line.

Error Handling

Let’s display errors of code blocks:

set y 2
set z
can't read "z": no such variable

Should given an error … It does!

Now check for inline code errors in text:

This line contains an inline code error `tcl set z`!!

This line contains an inline code error ??can’t read “z”: no such variable??!!

Writing images

Displaying images generated by Tcl code is as well possible. Here an example using the tclgd package which is available from the flightaware github page

We compile and the then try to load the package:

lappend auto_path ../libs
package require tclgd
package provide tclgd
==> 1.2

Ok, this works on my machine, let’s now create an image using gdtcl:

GD create gd 200 200
set blue [gd allocate_color 50 50 100]
set white [gd allocate_color 190 190 210]
gd filled_rectangle 20 20 180 180 $blue
gd filled_rectangle 40 40 160 160 $white
set out [open out.png w 0600]
fconfigure $out -translation binary -encoding binary
gd write_png $out 7
close $out

After creating the image we can display it inline using standard Markdown syntax:

  ![](out.png)

Shows the image:

The last lines in the code are slightly ugly if you need to repeat them again. So there is an alternative. Obviously you can create a figure procedure which gets one argument, a filename which is the file where the image should be written in.

proc figure {filename} {
    set out [open $filename w 0600]
    fconfigure $out -translation binary -encoding binary
    gd write_png $out 7
    close $out
}
gd filled_rectangle 60 60 140 140 $blue
figure out2.png

That’s better, but you can even omit the figure call by specifiying fig=true as an other code cunk option. Please note, that this still requires your implementation of the figure procedure with the filename argument. The following code block with automatically create the figure and embed it afterwards, the code itself (in the creme block) will be not shown as we specified echo=false:

    ```{tcl fig=true,echo=false}
    gd filled_rectangle 80 80 120 120 $white
    ```

Sometimes you don’t want this automatic embedding because you need to have more control over the placement of the image. To achieve this, you can specify the option include=false and include the image later. The problem is, that you are can’t be sure what the filename will be. Per default all code chunks and images will be named using a counter automatically for you. So your filename of the image which is derived from the code chunk name might later change if you add some code before of the current code. To fix this you have to add your own label to the code chunk as can be seen in the following example. This label will then be part of the image filename.

    ```{tcl label=myfig,fig=true,echo=false,include=false}
    gd filled_rectangle 90 90 110 110 $blue
    ```

We can now place the image where we like and we can place it also several times. The name of the file consists of the basename of the Markdown file and the label with the png extension. It is basename-label.png. So:

    ![](tmdoc-example-myfig.png) ![](tmdoc-example-myfig.png)

gives us:

Tk samples with screenshots

We can also use screenshot applications to make screenshots of running Tk applications. So for instance on Unix systems the following code might work.

    ```{tcl}
    package require Tk
    toplevel .top
    wm title .top DGApp
    pack [button .top.btn -text "Canvas Demo 1a" -command { exit 0 }] \
       -padx 10 -pady 10 -fill x -expand false
    pack [canvas .top.c -background beige] \
       -padx 5 -pady 5 -side top -fill both -expand yes
    .top.c create rectangle 50 50 100 80  -fill yellow
    .top.c postscript -file canvas.ps
    update idletasks
    after 200
    exec import -window DGApp -delay 200 screenshot-1.png
    ```

And you get a screenshot.

You can wrap the screenshot part as explained as well again in a figure procedure and use the approach with fig=true etc as chunk options.

Canvas Images

As this screenshot procedure is very platform specific, I prefere to demonstrate making figures with a canvas widget. Tcl will probably have a canvas snap or tk snap command in the new Tcl/Tk 8.7 release. In Tcl/Tk 8.6 we can use the tklib package canvas::snap for this purpose which needs again a installation of the tkimg package.

    ```{tcl}
    package require canvas::snap
    pack [canvas .c -background beige] \
        -padx 5 -pady 5 -side top -fill both -expand yes
    .c create rectangle 50 50 100 80  -fill yellow
    set img [canvas::snap .c]
    $img write canvas.png -format png
    ```

produces:

package require canvas::snap
pack [canvas .c -background beige] \
    -padx 5 -pady 5 -side top -fill both -expand yes
.c create rectangle 50 50 100 80  -fill yellow
set img [canvas::snap .c]
$img write canvas.png -format png
    ![](canvas.png)

shows the figure:

Ok, as this basically works let’s create as well a figure procedure which does the snap on the canvas.

proc figure {filename} {
    set img [canvas::snap .c]
    $img write $filename -format png
}

Now our code chunk options for the figure setting should work on the canvas as well:

     ```{tcl,fig=true,results=hide}
     .c create rectangle 60 60 90 70  -fill blue
     ```

This produces with the figure placed automatically:

.c create rectangle 60 60 90 70  -fill blue

We used results=hide to avoid showing the return value of the rectangle function.

Let’s at last in this section test the include=false option:

    ```{tcl label=rect,fig=true,results=hide,include=false}
    .c create rectangle 65 65 85 70  -fill red
    ```

    ![](tmdoc-example-rect.png) ![](tmdoc-example-rect.png)

produces:

.c create rectangle 65 65 85 70  -fill red

Images with ukaz

Let’s take an other example using the ukaz widget from Christian Gollwitzer available at: https://github.com/auriocus/ukaz. Please note, that I had to add the following delegations to the graph widgetadaptor inside the source code of ukaz.tcl to get canvas::snap working:

delegate option * to hull
delegate method * to hull

The project page has a readme, we now just convert a part of this readme to a Tcl Markdown:

Text from the Readme with some small modifications from my side:

The simplest usage looks like this:

destroy .c ;# destroy old image example
package require ukaz
pack [ukaz::graph .c -xrange {0.8 4.2} -width 300 -height 300] -expand yes -fill both
set data {1 4.5 2 7.3 3 1.8 4 6.5}
.c plot $data

This displays a resizable plot of the data, which is expected as a list of alternating x-y coordinates. The range of the axis, the number and placement of the the tic marks and the plot style is automatically chosen. The data can be zoomed in by dragging the left mouse button over the canvas, and zoomed out by clicking the right mouse button.

The style of the plot can be adjusted by passing more options to the plot command, and by setting global options using either gnuplot-style commands or Tcl-style configure commands:

destroy .c
pack [ukaz::graph .c -xrange {0.8 4.2}] -expand yes -fill both
set data {1 4.5 2 7.3 3 1.8 4 6.5}
.c plot $data with points pointtype filled-squares color "#A0FFA0" 
.c set log y

This displays the same data with light-green squares on a logarithmic y-axis.

End of the readme part of the ukaz package. This example should show how easy Tcl Markdown can be used to document functions and usage of Tcl packages.

Summary

The tmdoc::tmdoc package provides a literate programming proccessor which extracts and evaluates Tcl code within code chunks of a Markdown documents. The code chunks are evaluated by the tmdoc processor and replaced with their ouptut. Tcl code and the output can be shown or hidden due to specific configuration options for the code chunk. There is as well the possibility to display images created by Tcl within the document and inline evaluation of short Tcl commands within backtick where the first backtick is followed by the character sequence tcl like here `tcl set x 1` .

Code chunks start on lines with three backticks and the character sequence tcl after a curly brace. They end at the next line which has again three backticks.

    ```{tcl}
    set x 1
    ```

The following options for code chunks are currently available:

Installation

There is currently no prepared zip file. Just go to the dgroth tclcode project releases site: https://chiselapp.com/user/dgroth/repository/tclcode/wiki?name=releases and download the latest tmdoc zip file.

Unzip this file into a folder which is in your Tcl auto_path variable. For installing tmdoc as standalone application just copy the file tmdoc.tcl as tmdoc in a directory which belongs to your executable PATH directory and make it executable, on Unix for instance using the command chmod.

Processing Information

For generating the document the following Tcl version and Tcl packages were used:

Document was processed at: 2020-02-23 18:47

Document processing was done within: 886 milliseconds.

Let’s destroy the Tk application at the end.

TODO: Do this below automatically at the end !? Might be not good if it is loaded as package inside an application as this destroy probably as well the main application, not only the toplevel of the slave interpreter.

after 10
destroy .

We need to write something …

indentation of code?
indentation of code?