Home

Files in the top-level directory in any check-in

  • CharisSIL-plot.png
  • README.md

March 27, 2024

I was looking in my Lattice: Multivariate Data Visualization with R book and re-learned about the page argument to Lattice plots and how it is a good way to add annotations at the whole-page level. This function has as its coordinate system the fule display area scaled to [0,1] by [0,1].

The page argument takes a function which has a single argument which corresponds to the page number. (Lattice graphics can be multiple pages.) In the example below, I ignore the n argument and draw text with panel.text and grid.text.

library(lattice)
library(grid)
xyplot(dist ~ speed,
       data = cars,
       page = function(n) {
         panel.text(labels = "A.", x = 0.01, y = 0.99, adj = c(0, 1), cex = 3)
         grid.text(label = "B", x = 0.99, y = 0.99, just = c("right", "top"),
                   gp = gpar(col = "firebrick"))

       })

I need to learn more about complicated legends because they may be a good way to add even more plot annotations but this will do for simple text annotations.

March 6, 2024

Using R's lattice and latticeExtra here is a scatter plot smoother with intervals by group

d <- data.frame(x = rnorm(100), y = rnorm(100), g = rep(c("A", "B"), c(25, 75)))
xyplot(y ~ x, d, groups = g,
       panel = panel.superpose,
       panel.groups = function(x, y, ...) {
         panel.points(x, y, ...)
         panel.smoother(x, y, method = "loess", span = 0.9, ...)
       })

January 13

In Org, I can use Babel to generate Org. The code block below sets the results to value and raw. (Using output rather than raw doesn't work.) The raw says to insert the results of running the code block as-is rather than wrapping it in any way.

#+begin_src elisp :results value raw :exports none
(concat "** Level 2 headline\n"
        "*** Level 3 headline\n"
        "A short sentence.\n")
#+end_src

And then if I do C-c C-c in the block above, the buffer looks like this:

#+begin_src elisp :results value raw :exports none
(concat "** Level 2 headline\n"
        "*** Level 3 headline\n"
        "A short sentence.\n")
#+end_src

#+RESULTS:
** Level 2 headline
*** Level 3 headline
A short sentence.

This is great but a limitation is that if I do C-c C-c again in the code block, then the RESULTS are inserted again rather than replaced. I think this is because once the Org text is in the buffer, Org has no way to know it's a code block's output. This makes sense if I think of it as raw Org; it's not a block to be "updated" via Babel.

I originally looked into this to use code to autogenerate a statistical report. I think it would be useful as "one-time autogeneration". For example, if I had a number of pies and I wanted to show one figure per pie and separate them by headlines, I could have this code block:

#+begin_src R :results value raw :exports none
headlines <-
    c(`Pumkin pie` = "./figures/pumpkin.png",
      `Applie pie` = "./figures/apple.png")
sprintf("** %s\n[[file:%s]]\n",
        names(headlines),
        headlines)
#+end_src

And do C-c C-c and it generates this.

#+RESULTS:
** Pumkin pie
[[file:./figures/pumpkin.png]]

** Applie pie
[[file:./figures/apple.png]]

I would run it once interactively to fill in the report.

I don't know if this is particularly useful. I could write a script to generate this Org text outside of Org but now that I think about it, it's much more maintainable to have the code block in the file. If I want to change what is output, I have to remove it from the Org file and regenerate it with C-c C-c but at least the report Org file shows what will be in the exported document.

November 1

In R, I can use read.csv to read a particular CSV in a Zip archive

sl <- read.csv(unzip("../data/file.zip",
                     files = "folder_unzips_to/file.csv")

October 27

In Emacs I am trying the cyberpunk theme because it works well with the rainbow delimeters package but cyberpunk was giving a light background to Org source blocks. I learned how to customize it by going to an empty part of the source block and typing describe-face and seeing that the face was org-block. I changed the background color to black (#000000) and pressed Save and Apply.

October 25, 2023

In Emacs, I sometimes type C-x C-- by accident which calls text-scale-adjust to shrink the text size of the window. And C-x C-+ makes it bigger. After either C-x C-- or C-x C-+ then - makes text smaller, + makes it bigger, and 0 returns it to the default.

October 24, 2023

Using Lattice graphics in R, I needed an axis on the square root scale for a scatter plot matrix generated with lattice::splom. The scales are controlled by the pscales argument to splom (but really pscales is an argument to panel.pairs).

If a single number, the pscales argument controls how many tick marks are desired on the axes. But it can also be a list with one component for each column being plotted. The list should have these elements:

‘at’: a numeric vector specifying tick locations

‘labels’: character vector labels to go with at

‘limits’: numeric 2-vector specifying axis limits (should be
made more flexible at some point to handle factors)

These are specifications on a per-variable basis, and used on
all four sides in the diagonal cells used for labelling.
Factor variables are labelled with the factor names.  Use
‘pscales=0’ to supress the axes entirely.

One thing I found is that the list's elements should be at, lab, and lim to avoid partial match warnings since panel.pairs uses pscales[[i]]$lab and pscales[[i]]$lim.

Here's an example. It's not ideal.

library(lattice)
d <- data.frame(x = exp(rnorm(100, 1)),
                y = exp(rnorm(100, 1)))

calc.pscales <- function(x.untransformed) {
    list(at = sqrt(pretty(x.untransformed)),
         lab = pretty(x.untransformed),
         lim = extendrange(sqrt(x.untransformed), f = 0.1))
}
lapply(d, calc.pscales)

splom(sqrt(d), pscales = lapply(d, calc.pscales))

October 16, 2023

I like text-based help in R and so in my ~/.Rprofile I had

options(help_type = "text")

That was working on Linux but not on Windows. It turns out adding this to ~/.Rprofile in Emacs on Windows does not help. To find out where R is looking for .Rprofile I read the manual and found:

R> Sys.getenv("R_USER")

This is c:\Users\WEIGAND\Documents.

So I needed to edit c:\Users\WEIGAND\Documents\.Rprofile.

October 15, 2023

Some variables in Emacs are buffer variables. For example setting show-trailing-whitespace to something non-nil means highlight trailing whitespace in the buffer. But the help for this variable says

Automatically becomes buffer-local when set.

I learned that in order to set this variable in my init.el file, I need to use

;; (setq show-trailing-whitespace t) <-- Won't do what I want
(setq-default show-trailing-whitespace t)

September 22, 2023

This is how I made a scatter plot in Times New Roman using R's lattice graphics package on Windows:

library(lattice)
library(grid)

windowsFonts(TNR = "Times New Roman")

xyplot(dist ~ speed,
       cars,
       main = "A scatter plot in Times New Roman",
       par.settings = list(grid.pars = gpar(fontfamily = "TNR"),
                           fontsize = list(text = 24)))

On Linux it was more direct:

library(lattice)

xyplot(dist ~ speed,
       cars,
       main = "A scatter plot in Times New Roman",
       par.settings = list(grid.pars = gpar(fontfamily = "Times New Roman"),
                           fontsize = list(text = 24)))

September 18, 2023

Here is a strip plot in lattice that shifts points by group to reduce overlap:

stripplot(wool ~ breaks,
          data = warpbreaks,
          groups = tension,
          panel = panel.superpose,
          panel.groups = function(x, y, group.value, ...) {
              dots <- list(...) # Print this to see what else is in `...`
              panel.xyplot(x, y + c(-0.1, 0, 0.1)[match(group.value, c("L", "M", "H"))], ...)
          })

September 18, 2023

R's lattice package makes it easy to add a regression line by a grouping factor but I did not know how to add confidence intervals to the regression line. This uses the latticeExtra package to "add" a regression line with 95% CI but it does not do it by group:

library(lattice)
library(latticeExtra)

n <- 100
test <- data.frame(x = rnorm(n),
                   y = rnorm(n),
                   g = sample(letters[1:3], size = n, replace = TRUE))

xyplot(y ~ x, test, groups = g) + layer_(panel.smoother(x, y, method = "lm"))

The solution I came up with is to create a simple panel function that plots points and adds a regression line with a 95% CI. It uses panel.points from lattice and panel.smoother from latticeExtra. Then I created a simplified version of panel.xyplot which uses panel.superpose to handle grouping.

## A simple panel function with a scatter plot and a smoother
panel.simple <- function(x, y, ...) {
    panel.points(x, y, ...)
    panel.smoother(x, y, method = "lm", ...) 
}

## A "groups-aware" panel function the uses `panel.superpose`
panel.xyplot.simple <- function(x, y, groups = NULL, ...) {
    if (is.null(groups)) stop("We need 'groups' to not be NULL")
    panel.superpose(x, y, groups = groups, panel.groups = panel.simple, ...)
}                                        # ^^^^^^^^^^^^   ^^^^^^^^^^^^

## A plot with group-specific regression lines and 95% CIs.
xyplot(y ~ x, test, groups = g, panel = panel.xyplot.simple)

And here is a better way that only requires me to write panel.simple and which is documented in the manual page for xyplot:

xyplot(y ~ x, test, groups = g, panel = panel.superpose, panel.groups = panel.simple)

August 6, 2023

I learned how to publish a project using Emacs Org mode. The manual explains it well (see Publishing).

This is my alist with one project (www) but there can be multiple projects.

  (setq org-publish-project-alist
	'(("www"
	   :base-directory "~/fossils/www/"
	   :publishing-function org-html-publish-to-html
	   :publishing-directory "~/fossils/www"
	   :section-numbers nil
	   :with-toc nil
	   :html-doctype  "<!DOCTYPE html>"
	   :html-head "<link href=\"normalize.css\"                      rel=\"stylesheet\" type=\"text/css\">
		       <link href=\"sakura-vader.css\" id=\"sakura-css\" rel=\"stylesheet\" type=\"text/css\">")))

Then, to publish, it's org-publish-project and when prompted say which project. Putting the parameters in the org-publish-project-alist reduces boilerplate in the Org file. I was doing this at the top of files I was planning to export to HTML.

#+TITLE: My Title
#+OPTIONS: toc:nil num:nil html-style:nil html-scripts:nil html-postamble:nil html5-fancy:t
#+HTML_HEAD: <link href="normalize.css" rel="stylesheet" type="text/css">
#+HTML_HEAD: <link href='sakura-vader.css' id="sakura-css" rel='stylesheet' type='text/css'>
#+HTML_DOCTYPE: html5

My list above is not consistent with some of the lines immediately above which I will try to fix. To see what options can be specified in the list, go to Publishing options.

As an example, the option :html-preamble is controlled by the variable org-html-preamble. It is by default t and non-nil means insert an HTML preamble. If t it inserts a string defined by org-html-preamble-format. But if set to a string it uses the string. And if it is set to a function, apply the function.

The variable org-html-preamble-format is by default (("en" "")). The second element of this enclosed list is a format string as follows:

  %t stands for the title.
  %s stands for the subtitle.
  %a stands for the author’s name.
  %e stands for the author’s email.
  %d stands for the date.
  %c will be replaced by ‘org-html-creator-string’.
  %v will be replaced by ‘org-html-validation-link’.
  %T will be replaced by the export time.
  %C will be replaced by the last modification time.

If this is part of the org-publish-project-alist:

:html-preamble "written by %a")))

The exported HTML will have this div:

<head>
...
</head>
<body>
  <div id="preamble" class="status">
  written by Stephen Weigand
  </div>
  <div id="content">

  <h1 class="title"> ... etc.

August 2, 2023

I'm not sure why using Fossil via ssh started requiring my passphrase. I did the following:

PS> ssh-keygen.exe -t ed25519
PS> cd ~/.ssh
PS> cat id_ed25519.pub | ssh weigand@myserver.com "cat >> ~/.ssh/authorized_keys"

But then on myserver.com I had to edit authorized_keys and take out the Windows stuff like a carriage return before the newline at the end of the file. And while I was there I deleted a previous entry from this Windows machine.

July 11, 2023

Using R's lattice package I learned or remembered one can give a vector of x-axis, y-axis, and main labels. Here is a 2 by 2 figure with two x-axis labels and two y-axis labels

library(lattice)
n <- 100
d <- data.frame(x = rnorm(n),
                y = rnorm(n),
                g = sample(letters[1:4], size = n, replace = TRUE))
xyplot(y ~ x | g,
       d,
       xlab = c("Flower", "Tree"),
       ylab = c("Shrub", "Fruit"))

Another thing us I remembered to use main as a list. This puts the main title at the left of the viewport. See the main entry in the help file for xyplot.

xyplot(dist ~ speed,
       data = cars,
       main = list(label = "A. Scatter plots", x = unit(0, "npc"), just = "left")

July 1, 2023

The Feb. 9, 2023 entry shows how to delete files in Windows even if they don't exist. In searching for a platform independent way to remove files I remembered fossil clean. See https://fossil-scm.org/home/help?cmd=clean.

Here's how to remove all *.Rout and *.log files which may be in the current or any other directory.

fossil clean --clean *.Rout,*.log

It also works with

fossil clean --clean "*.Rout,*.log"

One can also ask Fossil to remove empty directories as part of the operation:

fossil clean --clean "*.Rout,*.log" --emptydirs

Adding the --force option to the above command will prevent Fossil from asking before removing each empty directory.

May 4, 2023

In R's base graphics, coloring box plots separately by group is trivial:

boxplot(breaks ~ interaction(tension, wool), warpbreaks, col = six.nice.colors)

Using Lattice graphics I learned it can be done by specifying a vector of fill colors as part of the box.rectangle element of the par.settings list.

p.s <- list(box.umbrella = list(col = "black", lty = 1),
            box.rectangle = list(col = "black", fill = six.nice.colors))
bwplot(breaks ~ interaction(tension, wool), warpbreaks, par.settings = p.s, pch = "|")

More general Lattice note

When generating a Lattice plot, creating a plot object doesn't generate any calls to grid::viewport. The viewpart function is only called when one plots the object. The reason I am exploring how Lattice works is I would like to make the plots more "table like" so that a stripplot of the form

stripplot(level ~ y | group, layout = c(1,2), as.table = TRUE)

looks like this

Group 1
  Level 1   *  *  *      *
  Level 2    *  * * 
  Level 3      **       *   **

Group 2
  Level 1   * * * *
  Level 2  * *   *  **
  Level 3  *   *   **   *
  
          |---+---+---+---+---|

To customize the strips, strip.default is (probably) not going to help since the strips are drawn to fill up the parent viewport. lattice:::plot.trellis is giving the strip a narrow viewport spanning the panel which strip.default fills up with a background, text, and border.

I think I need to examine lattice:::plot.trellis to see where the viewport for the strip is being created. It might be here:

pushViewport(viewport(layout.pos.row = pos.row - 1,
                      layout.pos.col = pos.col,
                      xscale = xscale, 
                      clip = "off",
                      name = trellis.vpname("strip",
                                            column = column,
                                            row = row,
                                            prefix = prefix,
                                            clip.off = TRUE)))

File this under "Today I Don't Know" because I don't have any good sense of where to implement my idea of moving strips left. I think I could say, "Hey, don't generate a default viewport for the strip but a wider version expanding off to the left." But then which function is doing the calculations to make it "fit". (I think it is lattice:::plot.trellis which is huge and gnarly.)

A hack may be to say the text in the strips will be adjusted left to the extent of the existing plot. We don't need (much?) more space on the left margin so long as we are just a little to the left of the y-axis labels (which are serving as subgroup names).

That is, we don't want this which takes a lot of new spacing calculations:

Group 1
        Level 1   *  *  *      *

but rather an "indented look" like this which can use existing or slightly modified padding:

Group 1
  Level 1   *  *  *      *

Maybe the simplest thing is along the lines of this:

myplot <- stripplot(...)
myplot <- add_left_padding_as_needed_based_on_strip_text(myplot)
justify_strip(myplot, just = "left") # Uses grid::grid.edit iterating over panels

Another possible name/metaphor is exdent_strip(myplot). (Or exdentStrip since Lattice often uses camel case.)

April 28, 2023

I remembered to try using Emacs file-local variables to "compile" R files.

In a plain R file I might have this at the top:

## -*- compile-command: "Rscript foo.R" -*-

And in an Rmd file I might have this:

<!-- -*- compile-command: "Rscript -e 'library(\"knitr\"); knit(\"foo.Rmd\")'" -*- -->

April 24, 2023

Pipx

I previously played with pipx to manage python command line applications. I couldn't remember if I used Scoop but it doesn't look like it. Today I'm using Python 3.11 via Scoop and I did:

python -m pip install --user pipx

I then got a warning about c:\users\weigand\appdata\roaming\python\python311\scripts\pipx.EXE not being on my path so I fixed that (via typing "path" in the Windows search bar and clicking on Environmental Variables).

I then followed the pipx suggestion and typed: pipx ensurepath.

Then I did pipx install csvkit and it worked.

in2csv from csvkit

This worked great to convert an Excel XLSX sheet into a CSV.

 in2csv.exe --sheet "Sheet XYZ" myfile.xlsx > sheet_xyz.csv

March 29, 2023

Passing arbitrary arguments to Rscript (or Rscript.exe on Windows) is easy. If a script called test-rscript.R exists and has this one line:

print(commandArgs())

then a command like this:

Rscript --vanilla test-rscript.R --never always -often --

Produces this:

 [1] "C:\\scoop\\apps\\r\\current\\bin\\x64\\Rterm.exe"
 [2] "--no-echo"
 [3] "--no-restore"
 [4] "--vanilla"
 [5] "--file=test-rscript.R"
 [6] "--args"
 [7] "--never"
 [8] "always"
 [9] "-often"
[10] "--"

This shows me that it easy to parse arbitrary command line arguments without having to rely on an external package.

March 28, 2023

I'm trying to write a shell command to export an Org file. I'm using

 emacs test.org --batch --funcall org-html-export-to-html --kill

but it doesn't evaluate the code blocks to generate figures using R. My true aim is RTF export using a local exporter I created called ox-rtf.el. This creates an RTF document but again, no figures.

 emacs test.org --batch --load ox-rtf.el --funcall org-rtf-export-to-rtf --kill

I'm using Batch export of org-mode files from the command line from Stack Overflow. (The answers are nine years old.) There is also this suite of command line tools https://github.com/fniessen/orgmk.

Additional info

The command line above "works" in that it exports the file as it is on disk. This means that if the code blocks are all evaluated then the exporting will show the figures. When editing the file and typing C-c C-e (which calls org-export-dispatch and invokes the menu) the exporting process includes the additional step of evaluating the code blocks. (It may be something like it calls org-babel-execute-buffer then does the exporting.)

The manual has this about

The sample script shows batch processing of multiple files using org-babel-tangle.

#!/bin/sh
# Tangle files with Org mode
#
emacs -Q --batch --eval "
    (progn
      (require 'ob-tangle)
      (dolist (file command-line-args-left)
        (with-current-buffer (find-file-noselect file)
          (org-babel-tangle))))
  " "$@"

So maybe the solution is something like

emacs -Q --visit test.org --batch --load ox-rtf.el --eval "(org-babel-execute-buffer) (org-rtf-export-to-rtf)" --kill

The problem with the above is Org Babel didn't know what to do with R code blocks. So I tried this and it seems to work.

emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el --eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill

If I am using the CMD shell, I can continue lines with ^

emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el ^
  --eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill

One problem is that I am loading my own init.el file and so this command will only work for me. I am using the Org (version 9.5.5) that came with my Emacs (version 28.2).

March 28, 2023

I'm trying to customize my ~eshell~ prompt. That is not working right now but this Emacs lisp snippet was helpful to learn:

(substring (shell-command-to-string "fossil status") 0 -1)

It captures the results of the fossil status command and trims the last character which is a newline.

March 21, 2023

From https://www.masteringemacs.org/article/executing-shell-commands-emacs I read about shell-command for one-liners.

March 21, 2023

A couple of times recently I've been typing madly in Emacs and my buffer's font size gets smaller. After a web search I want to remember these keys:

  • C-x C-+ will increase the buffer font size (as will C-x C-= which saves a shift key)
  • C-x C-- will decrease the buffer font size

These seem to invoke text-scale-adjust and I also wan to remember

  • C-x C-0 will reset to the default size

March 13, 2023

After updating Emacs to 28.2 using Scoop on Windows, my usual way of starting Emacs via the Start Menu search bar starts Emacs with a CMD window behind it. From this:

https://stackoverflow.com/questions/41300/emacs-in-windows

I think the problem is this is starting the executable program emacs.exe but I really want to be starting runemacs.exe. I don't know how to get the old behavior but I found that I could create a "Desktop Shortcut" as follows:

  • I right-clicked on runemacs.exe and created a shortcut
  • I moved this shortcut to the Desktop folder
  • I right-clicked on the shortcut and selected properties
  • In the Shortcut key text box I typed "e" so that my shortcut is "Ctrl + Alt + E". (It's a weird box because it is not editable; you just have to type a single character in there and the short cut is "Ctrl + Alt + X".)

While I was doing this I also changed the "double click action" (my term) for files I want to open in Emacs to be opened with runemacs.exe.

A follow-up note is that I originally tried having my shortcut in a Shortcut folder within Desktop but that didn't seem to work. I guess the term "desktop shortcut" means a shortcut on the desktop and not anywhere else.

March 7, 2023

Using Scoop on Windows I updated R and Rtools and then was unable to use the brms package. I used this to see what version I had and what versions were installed.

PS> scoop info r


Name        : r
Description : A free software environment for statistical computing and graphics.
Version     : 4.2.2
Bucket      : main
Website     : https://www.r-project.org
Updated at  : 11/2/2022 1:22:47 AM
Updated by  : Hsiao-nan Cheung
Installed   : 4.1.0
              4.1.1
              4.1.3
              4.2.2
Binaries    : bin\x64\R.exe | bin\x64\Rcmd.exe | bin\x64\Rgui.exe | bin\x64\Rscript.exe | bin\x64\Rterm.exe
Shortcuts   : R
...

I used this to "reset" to 4.1.3.

PS> scoop reset r@4.1.3
Resetting r (4.1.3).
Linking C:\scoop\apps\r\current => C:\scoop\apps\r\4.1.3
Creating shim for 'R'.
Creating shim for 'Rcmd'.
Creating shim for 'Rgui'.
Creating shim for 'Rscript'.
Creating shim for 'Rterm'.
Creating shortcut for R (Rgui.exe)
C:\Users\WEIGAND

This was very, very nice.

Feb. 14, 2023

In Emacs I decided to always have line numbers on and Web searches suggested as of Emacs 26 use (global-display-line-numbers-mode) in one's initialization file. I then customized the line-number face and the line-number-current-line face to both have a height of 80%.

Feb. 9, 2023

I learned how to remove files in Windows CMD even if such files aren't present. Here is an example from a Makefile:

.PHONY: clean
clean:
        cd code && if exist *.png (del *.png)

Jan. 6, 2023

Through trial and error I now understand colors in RTF as implemented by Microsoft Word (365 on the desktop) a little better.

  • Highlighting is done via \highlightN. For example, {\highlight5 This text is highlighted} will be highlighted using the fifth color in the color table.

  • But value for N has to correspond to a row in the color table.

{\colortbl;
\red0\green70\blue173;     N=1
\red182\green0\blue20;     N=2
\red169\green169\blue169;  ...
\red222\green235\blue247;
\red204\green121\blue167;
\red255\green255\blue0;    N=6 which here is yellow
\red204\green121\blue167;
\red0\green255\blue0;      N=8 which here is green
\red139\green0\blue0;      N=9 which is what R gives for col2rgb("darkred")
}

Value Description
1 Black
2 Blue
3 Cyan
4 Green
5 Magenta
6 Red
7 Yellow
8 Unused
9 Dark Blue
10 Dark Cyan
11 Dark Green
12 Dark Magenta
13 Dark Red
14 Dark Yellow
15 Dark Gray
16 Light Gray
  • The secret recipe is then \highlightN will highlight text using the Nth color in the color table so long as the Nth color in the color table corresponds to the RGB of one of the colors above.

Jan 3, 2023

I am having all kinds of difficulties with a Makefile on Windows. I have this:

# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe
ifeq ($(OS), Windows_NT)
        R_CMD := rcmd.exe
        R_SCRIPT := rscript.exe
        SHELL = c:\scoop\shims\bash.exe
        SCPEXE := c:\scoop\apps\gow\current\bin\pscp.exe # Requests password :(
        SSHEXE := c:\scoop\apps\gow\current\bin\plink.exe # Requests password :(

and this target:

.PHONY: model
model: /path/to/data.rds
/path/to/data.rds: src/a.R data/b.rds
     $(SCPEXE) src/a.R weigand@server:/path/to/putit
	 $(SSHEXE) weigand@server 'cd /path/to/putit/; R CMD BATCH a.R'

The problems with the above is that the call to SCP requires a password even though I otherwise have passwordless login setup. And the call to SSH needs single quotes as far as I can tell and it doesn't understand R CMD BATCH even though my intent is that command is run on the server.

Part of the problem may be the clashing between Scoop's use of pscp.exe and plink.exe versus OpenSSH's scp.exe and ssh.exe in c:/Windows/System32/OpenSSH.

Dec. 28, 2022

I searched for how to type accented characters in Emacs and got https://irreal.org/blog/?p=4945 and https://irreal.org/blog/?p=4254.

One general way to enter accented characters is C-x 8 then a one- or two-character sequence. For example the character "ñ" is entered with C-x 8 ~ n. The character "á" is entered with C-x 8 ' a. There are also special characters like "÷" which is C-x 8 / /. To see what is available, do C-x 8 C-h.

The blog author prefers "switch[ing] into TeX input mode and then using the TeX code for the character." This can be done by typing C-\ to run the command toggle-input-method. It is described as:

Enable or disable multilingual text input method for the current buffer.

If one sets their default input method to TeX (see below) then the character "β" can be entered simply with C-\ \beta. To turn off TeX input mode one enters C-\ again. As far as I can tell, the only way to know if you are in TeX input mode is the buffer mode line starts with \. (I suppose it desponds on how the mode line is set up.)

I haven't experimented with this but I imagine in many buffers one can stay in TeX input mode rather then toggling it on or off.

Here is a link to a 22 page PDF of LaTeX symbols: https://www3.nd.edu/~nmark/UsefulFacts/LaTeX_symbols.pdf

As an example, it looks like the section symbol "§" is just \S.

(One can customize default-input-method to TeX with Alt-x customize-variable default-input-method. I am curious about other input modes but for now I'll let it go...)

Oct. 14, 2022

On Windows I miss the Bash equivalent of emacs myfile &. In CMD.exe I can do

start /b c:\scoop\shims\emacs.exe myfile

Oct. 12, 2022

I re-found a video on YouTube (https://youtu.be/D8hEirGijmo) by "tdhock" (Toby Hocking?) that describes how to set the Emacs Speaks Statistics (ESS) variable ess-r-package-auto-enable-namespaced-evaluation. I customized this to nil so that evaluation of a function, paragraph, buffer etc. is not into the package namespace when ESS detects you are in a package. I think this is more straightforward since I'm used to evaluating an R buffer or parts of an R buffer into the global environment during testing or development.

Oct. 6, 2022

SSH file and directory permissions

I was trying to work out file sharing on a Linux machine that a colleague and I both have access to and I changed my home directory permissions there to 777 to see if she could write a file in one of my folders. (The server was protected by the corporate firewall and users were trused so security wasn't a concern.)

A few days later I noticed that passwordless login to the serverwasn't working. I didn't make the connection because it had been a few days and I spent a lot of time trying to understand the problem, check permissions on files in my varoius .ssh directories, generate new keys, etc.

It turns out that the fix was to change permissions on that server to 705 (i.e., read and execute for others).

Apparently, for passwordless SSH to work, the user's home directory can't be writable by world. (One server I use a lot has 755 for my home directory.)

ASCII and Emacs

I had a file on Linux that had some non-ASCII characters. (Windows was involved.)

In R one can do:

library(tools)
showNonASCIIfile("filename.txt")

This gave me a printout of

2: <ef><bb><bf>and this and that...

So R showed me the non-ASCII but I couldn't delete those characters in Emacs.

From a Web search and landing on https://www.gnu.org/software/emacs/manual/html_node/emacs/Coding-Systems.html, I learned that I could call ~revert-buffer-with-coding-system~ and choose ~raw-text~. This displayed

\357\273\277and this and that...

I could then delete these three characters/entities/things and the file passes the showNonASCIIfile test in R.

Sept. 21, 2022

In Emacs on Windows the command Alt-x shell starts a CMD.EXE shell. This is fine/nice. I learned from http://www.hanselman.com/blog/a-better-prompt-for-cmdexe-or-cool-prompt-environment-variables-and-a-nice-transparent-multiprompt that I can customize my prompt. I'm now using

set prompt=$P$_$+cmd$G$S

which gives the drive and path ($P), a newline and carriage return ($_), the pushd/popd depth (+), the text cmd (cmd), then ">" ($G), then a space ($S).

I set the prompt variable by typing "environmental variables" in the search bar and then added a new user variable.

Also, thanks to tips in https://stackoverflow.com/questions/20530996/aliases-in-windows-command-prompt, I have now pseudo aliases in the form of batch files that are saved in a directory that is already on my path. An example is:

@echo off
echo.
cd %USERPROFILE%\Fossils
dir /ad /b /n /w

This takes me to my Fossils directory and lists the directories therein. The first option of /ad is the most useful because it lists directories only. (Running fossils.bat in PowerShell works but the dir command there is an alias for Get-ChildItem and my options are ignored.

June 16, 2022

The generic preprocessor called GPP http://files.nothingisreal.com/software/gpp/gpp.html allows users to define macros in various syntaxes including something similar to the C preprocessor or to LaTeX macros. As I use it, the command is

gpp -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile

or on Windows:

gpp.exe -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile

I could run this within a Makefile on Windows and Linux but not from Windows Terminal. I realized it works from the CMD shell (and GNU Make is using CMD by default) but not when using PowerShell which must require escaping or quoting of the arguments. (The arguments are probably very, very PowerShell unfriendly with double quotes and double backslashes.)

I don't have a work around but at least now I know why there was a problem running it from the (Windows Terminal) command line.

May 18, 2022

This is how I made something of a platform independent Makefile to run either rcmd.exe on Windows or R CMD on Linux.

# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe

ifeq ($(OS), Windows_NT)
        R_CMD := rcmd.exe
        SHELL = c:\scoop\shims\bash.exe
else
        R_CMD := R CMD
endif

May 6, 2022

This is how I used scp to copy a directory from Linux to Windows:

scp -r username@server.com:/file/to/syncdir .
                                            ^ syncdir created locally here

May 5, 2022

How I updated my Pi to Python 3.10.4: https://stackoverflow.com/questions/64718274/how-to-update-python-in-raspberry-pi

sudo apt-get install -y build-essential tk-dev \
  libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev \
  libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev \
  liblzma-dev zlib1g-dev libffi-dev

wget https://www.python.org/ftp/python/3.10.4/Python-3.10.4.tgz
sudo tar zxf Python-3.10.4.tgz
cd Python-3.10.4.tgz
sudo ./configure --enable-optimizations
sudo make -j4
sudo make altinstall
  

April 12, 2022

I had to work with text files with a non-ASCII character representing a superscript 2. When checking these into Fossil, Fossil warned of invalid UTF-8. I think the problem was that the file had the superscript 2 in LATIN-1. I told Fossil to convert the file and opened the converted file and original file in Emacs and turned on hexl-mode. I forgot to finish this post but I think the difference was between b2 for LATIN-1 and 00b2 for UTF-8.

March 7, 2022

I like the "generic preprocessor" program called gpp (https://github.com/logological/gpp/) for use as a general macro processor. (See https://en.wikipedia.org/wiki/Preprocessor and some links therein.) On Linux gpp is easy to install on with the usual ./configure, make, etc.

Below is how I got it working on Windows without using ./configure or make. (I have Make but I don't think I have sh needed by configure. I'm not sure, but it wasn't hard to compile "by hand".)

These are the preliminary steps:

/* Header created by Stephen Weigand to avoid configure on Windows */
#define PACKAGE "gpp"
#define PACKAGE_BUGREPORT "tristan@logological.org"
#define PACKAGE_NAME "GPP"
#define PACKAGE_STRING "GPP 2.27"
#define PACKAGE_TARNAME "gpp"
#define PACKAGE_URL ""
#define PACKAGE_VERSION "2.27"


#define HAVE_STRDUP 1
#define HAVE_STRCASECMP 1
#define HAVE_FNMATCH_H 0

The HAVE_<something> reflects that my GCC can find the C functions strdup and strcasecp but not fnmatch. Now edit gpp.c as follows.

Add the include for stephen.h near the top. Note the quotes around stephen.h which says look in the current directory for the include file.

/* Added by Stephen Weigand March 7, 2022 */
# include "stephen.h"

Now make a few changes based on not having fnmatch. Lines 1880-1902 need to be like this:

    if (SpliceInfix(buf, pos1, pos2, "=~", &spl1, &spl2)) {
#if ! HAVE_FNMATCH_H
        bug("globbing support has not been compiled in");
#else <--- I had to convert this from #endif to #else
        if (!DoArithmEval(buf, pos1, spl1, &result1)
                || !DoArithmEval(buf, spl2, pos2, &result2)) {
            char *str1, *str2;

            /* revert to string comparison */
            while ((pos1 < spl1) && isWhite(buf[spl1 - 1]))
                spl1--;
            while ((pos2 > spl2) && isWhite(buf[spl2]))
                spl2++;
            str1 = strdup(buf + pos1);
            str1[spl1 - pos1] = '\0';
            str2 = strdup(buf + spl2);
            str2[pos2 - spl2] = '\0';
            *result = (fnmatch(str2, str1, 0) == 0);
            free(str1);
            free(str2);
        } else
            *result = (result1 == result2);
        return 1;
#endif <--- This was moved down to here.
    }

With this change (which may be a bug fix, really) I could then do the old-fashioned way of compiling the program.

gcc.exe -Wall -Os -o gpp.exe gpp.c

Then I moved it to somewhere in my path.

February 22, 2022

Commands for setting up a new Fossil

This is my set of commands for starting a Fossil project. I create the repository, set my password, add another user (otheruser) and set their password, and then give them administrator (a) capabilities.

cd /path/to/fossils
fossil new myrepo.fossil
fossil user password myname mypAssW0rd -R myrepo.fossil
fossil user new otheruser otheruser@example.com the1rPasSw0rd -R myrepo.fossil
fossil user capabilities otheruser a -R myrepo.fossil

Then I start the UI and change a few settings:

fossil ui myrepo.fossil

Once the UI opens, click on Admin then Configuration and set the following:

  • Project name. I keep this short like a filename since it displays in my bash prompt.

  • Project description. This is longer.

  • Index page. I use /dir?ci=tip so that when I click on Home in the UI it takes me to the list of files.

I also add a deposit target in a Makefile which will "deposit" and unpack the repository to a location on disk that colleagues who don't use Fossil can look at.

MYPATH := /path/to/shared/directory
.PHONY: deposit
deposit:
        fossil zip trunk source-code-repository.zip --name source-code-repository
        mv source-code-repository.zip $(MYPATH)
        cd $(MYPATH)/; unzip -o source-code-repository.zip
        rm -f $(MYPATH)/source-code-repository.zip
        echo "WARNING: This is a read-only copy of files under Fossil version control for this project." > $(MYPATH)/source-code-repository/WARNING.txt

February 14, 2022

Building R packages

I'm building my R package on my Raspberry Pi and did this

cd ~/Subversions/myproject-branches/
R CMD build mydir-wip-branch

But I needed texi2dvi and so did:

sudo apt install texinfo

But I needed texlive so I did:

sudo apt-install texlive

But I needed inconsolata.sty so I did:

sudo apt-install texlive-fonts-extra

And then I got R CMD build and R CMD check to both be happy and the latter command even got my vignette built.

Subversion branches

I have a checkout in ~/Subversions/myproject and made a branch like this.

First I used the svn copy command

svn copy ^/mypkg/mydir ^/mypkg/mydir-wip-branch -m "Making a work in progress branch of mydir"

Here the ^ means the "root" part of the repository. Then I did

mkdir ~/Subversions/myproject-branches
cd ~/Subversions/myproject-branches
svn checkout svn+ssh://username@example.com/svnroot/myproject/mypkg/mydir-wip-branch

I now have a work-in-progress branch of a subdirectory of the repository. I will need to merge it someday.

January 28, 2022

Background

This is from https://everyday.codes/linux/how-passwordless-ssh-login-works/

  • I generate a private key which is a long sequence of bits/bytes/characters.
  • A public is another sequence that is derived from this.
  • I can get the public from the private but with the public I can't go back to the private.
  • My public key is available for everyone and stored on my PC and the server

Using the public key, you can encrypt (or sign) any message, and it will only be possible to decrypt it using the private key. In other words, anyone with your public key can send you encrypted messages that only you will be able to read.

The server can authenticate me like this:

  1. Encrypt a message using the public key I've stored on the server and send the message to my PC
  2. My PC decrypts the message using my private key and sends it back
  3. If the server sees I got it right then I am authenticated.

I'm live tiloring my steps to set up passwordless login for R-forge.

This is a good resource it seeks https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement

I will have a private key and a public key. I place my public key on the R-forge server. When I ssh to r-forge Microsoft says:

When using key authentication with an SSH server, the SSH server and client compare the public key for a user name provided against the private key. If the server-side public key cannot be validated against the client-side private key, authentication fails.

PS> ssh-keygen.exe -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\USERNAME/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\USERNAME/.ssh/id_ed25519.
Your public key has been saved in C:\Users\USERNAME/.ssh/id_ed25519.pub.

And:

Now you have a public/private Ed25519 key pair in the location specified.

    Directory: C:\Users\WEIGAND\.ssh

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        1/28/2022   9:38 PM            464 id_ed25519
-a----        1/28/2022   9:38 PM            104 id_ed25519.pub

Remember that private key files are the equivalent of a password should be protected the same way you protect your password. To help with that, use ssh-agent to securely store the private keys within a Windows security context, associated with your Windows login. To do that, start the ssh-agent service as Administrator and use ssh-add to store the private key.

December 16, 2021

I forgot how to control what applications start up automatically in Windows 10. A while back I set it so PowerShell started up automatically and disabled the same for Skype. But I needed to disable Microsoft Teams from starting up automatically. Here's how I did it:

I typed "Startup" in the search bar. This opened the Settings app and took me to the Startup setting. Then I disabled Teams.

Obviously this was trivial but I lived with Teams starting up automatically for a couple weeks so I added a note about it to cement it in my memory. (I was remembering it as complicated and involving a start-up folder and shortcuts. I guess not.)

December 1, 2021

To fully detach a package in R I do this:

R> detach("package:mypackage", character.only = TRUE, unload = TRUE)

One thing that seems to be a pain is if you "attach" something like ggplot2 (i.e., do any of library(ggplot2), library("ggplot2"), require(ggplot2), etc.) then your session will get a lot of packages loaded "via a namespace" but not "attached"):

> sessionInfo() # after library(ggplot2)
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
<snip>
attached base packages:
[1] stats     graphics  utils     datasets  grDevices methods   base     

other attached packages:
[1] ggplot2_3.3.5

loaded via a namespace (and not attached):
[1] fansi_0.4.2      withr_2.4.2      assertthat_0.2.1 dplyr_1.0.7     
 [5] crayon_1.4.1     utf8_1.2.2       grid_4.0.3       R6_2.5.1        
 [9] DBI_1.1.1        lifecycle_1.0.1  gtable_0.3.0     magrittr_2.0.1  
[13] scales_1.1.1     pillar_1.6.3     rlang_0.4.11     generics_0.1.0  
[17] vctrs_0.3.8      ellipsis_0.3.2   splines_4.0.3    tools_4.0.3     
[21] glue_1.4.2       purrr_0.3.4      munsell_0.5.0    compiler_4.0.3  
[25] pkgconfig_2.0.3  colorspace_2.0-2 tidyselect_1.1.1 tibble_3.1.2    

There are still 28 packages "loaded via a namespace (and not attached)" after I detach ggplot2. (But I'm not sure what that means exactly.) The way to get rid of these hangers-on is via the unloadNamespace() function.

But they seem to need to be detached based on what's at the top of the dependency tree. Here is an illustration of the problem

> unloadNamespace("fansi") # try to unload first in the list
Error in unloadNamespace("fansi") : 
  namespace 'fansi' is imported by 'pillar' so cannot be unloaded

> unloadNamespace("pillar") # OK, I'll unload 'pillar' first.
Error in unloadNamespace("pillar") : 
  namespace 'pillar' is imported by 'tibble', 'dplyr' so cannot be unloaded

> unloadNamespace("tibble") # OK, unload 'tibble'.
Error in unloadNamespace("tibble") : 
  namespace 'tibble' is imported by 'dplyr' so cannot be unloaded

> unloadNamespace("dplyr") # OK, here's an order that works.
> unloadNamespace("tibble")
> unloadNamespace("pillar")
> unloadNamespace("fansi") 

I think there are packages that help you detach and unload packages but I am liking the idea of not attaching ggplot2 in the first place.

November 29, 2021

UTF-8 in C

Here is short C program that works for me on Linux and Windows to print out Hello Aβ42:

#include <stdio.h>

int main(){
  printf("Hello, A\u03b242\n");
  return 0;
}

On Windows I compile it with gcc.exe -Wall -o hello.exe .\hello.c using GCC 8.3.0 and get no errors or warnings. On Linux I am using GCC 4.8.5 and get

hello.c: In function 'main':
hello.c:4:10: warning: universal character names are only valid in C++ and C99 [enabled by default]
   printf("Hello, A\u03b242\n");
          ^

I can use GCC 8.3.1 after typing scl enable devtoolset-8 bash and then the warning goes away. But if I add the GCC flag -std=c89 then the warning comes back (which makes sense).

UTF-8 in Windows Terminal

When using Windows Terminal, I clicked on the down arrow in the tab bar and clicked on Settings which opened up a JSON settings file in Notepad. Then I added two arguments to the PowerShell command to tell the terminal to output in UTF-8.

 // Make changes here to the powershell.exe profile.
 "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
 "name": "Windows PowerShell",
 "commandline": "powershell.exe -Noexit -Command chcp 65001",
 "hidden": false                ^^^^^^^^^^^^^^^^^^^^^^^^^^^

July 22, 2021

From my Windows 10 desktop I can clone a repo on a Linux machine with

fossil set ssh-command "ssh"
fossil clone -v ssh://weigand@server.com/./fossils/filename.fossil filename.fossil

This is good except I should be able to specify the SSH command using -c ssh. Also, I wonder why it doesn't work with plink.exe which allows commands like this

plink.exe -ssh weigand@server.com ls

July 21, 2021

When exporting in Org mode to prevent underscores from being treated as subscripts, I did this:

#+OPTIONS: ^:{}

July 1, 2021

From https://endlessparentheses.com/debugging-emacs-lisp-part-1-earn-your-independence.html I learned several things which may help with my RTF backend for Org mode.

  1. The command find-function will take me to the definition of a function. If I am inside a function, I can invoke it and it will ask me what function to go to with the default being the one I'm in. It will then take me to where in the source that function is defined and put point at the start of the function. By default it doesn't look like their are key bindings. I might want to make an alias so I only need to do M-x ff.

  2. The command describe-function is bound to C-h f or <f1> f and shows the full documentation of the function. If point is in (or next two?) a function, issuing the command will set the default to the function point is in/near and so just press enter to see the help for the function.

  3. I just realized that <f1> is the help function. So naturally <f1> f will give the help for a function, <f1> v the help for a variable, etc.

  4. The Edebug part is that I either find or describe the function and then do C-u C-M-x. This is eval-defun with a prefix according to the manual. My only problem is I can't get this to work except for a function that is buggy.

June 15, 2021

In R on Windows and Linux, source("clipboard") will source the contents of the clipboard. On MacOS the equivalent is source(pipe("pbpaste")).

June 13, 2021

From https://devblogs.microsoft.com/scripting/powertip-use-powershell-to-display-windows-path/ I can get my path on PowerShell via

$env:path -split ";"

I turned this into a function:

function path() {
  $env:path -split ";"
}

June 12, 2021

In Emacs I can get an interactive SQLite session via M-x sql-sqlite and then I just have to specify a data base filename.

May 3, 2021

An easy way to get semi-transparent colors in R is to use adjustcolor:

barplot(c(1, 2, 3), col = adjustcolor("firebrick", 0.5),
        main = "Firebrick bar plot with alpha=0.5")

April 26, 2021

Create a simple SQLite data base:

sqlite3 test.sqlite3
sqlite> .databases -- Show the databases

  main: /home/weigand/fossils/tilor/test.sqlite3

ATTACH DATABASE 'test.sqlite3' as 'project';

.databases

main: /home/weigand/fossils/tilor/test.sqlite3
   project: /home/weigand/fossils/tilor/test.sqlite3

DROP TABLE IF EXISTS person;
CREATE TABLE IF NOT EXISTS project.person(
  id INT PRIMARY KEY NOT NULL,
  date_birth CHAR(10),
  sex CHAR(2) CHECK(sex == 'M' OR sex == 'F')
);

INSERT INTO project.person VALUES(1, '2010-01-01', 'M');
INSERT INTO project.person VALUES(2, '2011-11-11', 'F');

SELECT * FROM main.person;

   1|2010-01-01|M
   2|2011-11-11|F

April 14, 2021

When using smbclient on Linux to put a file on a Windows share I wasn't specifying the local file path correctly. I learned about the lcd command in smbclient to set the local current directory and then I could put the filename without a full or relative path. Like this:

PASSWD=`grep -woi -m 1 '^machine mymachine login .* password .*$' ~/.netrc | cut -d ' ' -f 6`
smbclient //share.example.com/DirectoryX \
  -W MYWORKGROUP  \
  -E       \
  -U $USER $PASSWD \
  --command 'cd DirA/DirB ;
             lcd /local/working/directory ;
             put localfile.csv ;
             exit'

March 17, 2021

This is how to get interval estimates for random effects from a linear mixed effects model fit with lme4::lmer in R using the unexported lme4:::asDf0 function.

library(lme4)
library(lattice)
fit <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)

dotplot(ranef(fit)) # Caterpillar plot shows intervals

out <- lme4:::asDf0(ranef(fit), "Subject")
out$lcl <- out$values - 1.96 * out$se
out$ucl <- out$values + 1.96 * out$se

out[c(1:2, 35:36), ]

values         ind .nn   se
1    2.259 (Intercept) 308 12.1
2  -40.399 (Intercept) 309 12.1
35  -0.988        Days 371  2.3
36   1.284        Days 372  2.3

March 10, 2021

On Linux & runs a command in the background. For example:

emacs README.md &

From PowerShell, the equivalent seems to be:

Start-Process -NoNewWindow emacs.exe README.md

The above is from a Web search where I landed on https://ariefbayu.xyz/run-background-command-in-powershell-8ea86436684e. That post shows how to make a general run-it-in-the-background command.

function bg() {
     Start-Process -NoNewWindow @args
}

And then I might use:

bg emacs.exe README.md

March 3, 2021

The Journal NeuroImage uses an open source font called Charis SIL. This can be downloaded and on my department's Linux system, I just have to put the *.ttf files in ~/.fonts and then I can create figures in "native" NeuroImage style:

library(ggplot2)

ggplot(cars, aes(x = speed, y = dist)) +
    geom_point() +
    labs(title = "Scatter plot in Charis SIL font used by NeuroImage",
         x = "Speed (miles per hour)",
         y = "Distance (feet)") +
    theme_light(base_family = "CharisSIL")

Scatter plot of distance versus speed demonstrating CharisSIL font

December 12, 2020

This is how I used Dropbox's Python API to upload my son's piano recital video. (My daughter's video was smaller and uploaded fine through the Dropbox web interface before timing out.)

My guide was Dropbox for Python Developers.

  1. Set-up a Python virtual environment and install dropbox

    • python3.7 -m venv ~/venvs/recital
    • source ~/venvs/recital/bin/activate
    • pip install --upgrade pip
    • pip install dropbox
  2. Linked my account in Dropbox to an app

    • Go to the app console
    • Created and named the app "recital". To keep it simple I gave the app read and write permissions across my whole Dropbox account.
    • Generated a short-lived and quite long access token (which lasts for four hours)
  3. Wrote Python code to create a dropbox instance, read the file into an object, and upload the object

import dropbox
from dropbox.files import WriteMode


dbx = dropbox.Dropbox('AVerySpecial138CharacterAccessToken')

with open('/home/weigand/recital/piano.m4v', 'rb') as f:
        data = f.read()

dbx.files_upload(data,                         # My file
                 '/recital-folder/piano.m4v',  # Full path of destination
                 mode=WriteMode('overwrite'))  # Want to overwrite

This obviously isn't "professional grade" Python. It works but doesn't handle errors and hard codes the filename.

December 11, 2020

This is how I managed to print to a WiFi-enabled printer from my Raspberry Pi.

I first went to the support page the Brother MFC-J870DW printer and selected Downloads and then did the following:

  1. Select OS Family as Linux
  2. Select OS Version as Linux (deb)
  3. Clicked on the Driver Install Tool
  4. Agreed to the End User Licence Agreement (EULA) and clicked Download
  5. Saved the file linux-brprinter-installer-2.2.2-1.gz to my ~/Downloads directory

In the shell I did:

cd ~/Downloads
gunzip linux-brprinter-installer-2.2.2-1.gz
sudo bash linux-brprinter-installer-2.2.2-1

  Input model name ->MFC-J870DW

  You are going to install following packages.
     mfcj870dwlpr-3.0.0-1.i386.deb
     mfcj870dwcupswrapper-3.0.0-1.i386.deb
     brscan4-0.4.9-1.i386.deb
     brscan-skey-0.3.1-1.i386.deb
  OK? [y/N] ->
  ... <say yes to a few additional things>
  Will you specify the Device URI? [Y/n] ->y

  select the number of destination Device URI. ->



December 10, 2020

CSVY format

The data.table package in R has a fast way to read CSV files in the form of data.table::fread and as of version 1.12.4 it has support for reading CSVY files. The CSVY format is a text format where there is a YAML header at the top of the file which defines the data "schema" and below the header are the data lines. Here is the general idea of a CSVY file.

schema:
  fields:
  - name: name
    type: string
  - name: age
    type: integer
  - name: date
    type: number
name, age, date
Maria, 44, 2010-01-01
Roberto, 43, 2009-01-01

Tukey HSD

A colleague likes using Tukey's honest significant difference for pairwise comparisons. I never use it but maybe I should. The example for stats::TukeyHSD in R is as follows.

R> summary(fm1 <- aov(breaks ~ wool + tension, data = warpbreaks))
            Df Sum Sq Mean Sq F value Pr(>F)
wool         1    451     451    3.34 0.0736 .
tension      2   2034    1017    7.54 0.0014 **
Residuals   50   6748     135
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

R> TukeyHSD(fm1, "tension", ordered = TRUE) # Still need to understand 'ordered'
  Tukey multiple comparisons of means
    95% family-wise confidence level
    factor levels have been ordered

Fit: aov(formula = breaks ~ wool + tension, data = warpbreaks)

$tension
    diff   lwr upr p adj
M-H  4.7 -4.63  14  0.45
L-H 14.7  5.37  24  0.00
L-M 10.0  0.65  19  0.03

I understand penalization as a better way to handle multiple comparisons but this is quick and easy. It would be interesting to better understand Tukey HSD and perhaps see if I can borrow the automatically generated pairwise comparisons code.

December 4, 2020

Indent a region five spaces in Emacs

C-u 5 C-x TAB is what I wanted.

R's base and grid graphics in one plot

This is one way to combine base and grid graphics in one plot:

 library(ggplot2)
 library(ggplotify)
 library(gridExtra)

 ## Using `::` in a few places to indicate the package
 p1 <- ggplotify::as.grob(~plot(dist ~ speed,  # Notice the unusual `~`
								data = cars,
								main = "Base graphics scatter plot"))

 p2 <- ggplot(cars) +
	 aes(x = speed, y = dist) +
	 geom_point() +
	 ggtitle("ggplot scatter plot")

 gridExtra::grid.arrange(p1, p2, nrow = 1)

Org and TODO states

I had to remind myself how to change the TODO states in an Org file. Here is an example Org file with the information.

 #+TODO: ASAP SOON WAITING SOMEDAY | DONE CANCELED

 * ASAP Learn about Org TODO states
 - It seems easiest to put the TODO state sequence in the file itself
   by having a ~#+TODO:#~ line at the top of the file.
 - See [[https://orgmode.org/org.html#Per_002dfile-keywords][per-file TODO states]]
 - The vertical bar separates not-done versus done states.
 - If you change your TODO states in a file go to the line with the states
   and do ~C-c C-c~ and you'll see "Local setup has been refreshed."
 - As a reminder, the TODO states can be advanced via ~C-c C-t~.

November 28, 2020

I know of two disk space programs on Linux:

  1. du says it is to estimate file space usage
  2. df says it is to report file system disk space usage but I found it's better to think of it as reporting free space.

I think df is more what I want since it shows

If no file name is given, the space available on all currently mounted file systems is shown

On this Raspberry Pi, df -h gives the following:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        14G  5.4G  7.1G  43% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M   60M  404M  13% /dev/shm
tmpfs           464M  6.3M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p6   71M   23M   49M  32% /boot
tmpfs            93M  4.0K   93M   1% /run/user/1000
/dev/mmcblk0p5   30M  398K   28M   2% /media/pi/SETTINGS

A Web search gives me this nice link from Adafruit:

https://learn.adafruit.com/an-illustrated-shell-command-primer/checking-file-space-usage-du-and-df

This led to free for memory usage. On the Pi, free -h gives (with some whitespace removed):

         total    used    free  shared  buff/cache   available
Mem:      926M    392M     47M     67M        486M        487M
Swap:      99M    5.8M     94M

November 25, 2020

From the Linux Documentation Project (via Stack Exchange):

/etc/skel/ The default files for each new user are stored in this directory. Each time a new user is added, these skeleton files are copied into their home directory. An average system would have: .alias, .bash_profile, .bashrc and .cshrc files. Other files are left up to the system administrator.

On the Raspberry Pi that I am working on righ now, I see .bash_logout, .bashrc, and .profile.

And per a comment in this .profile, Bash reads ~/.profile if there is no ~/.bash_profile and no ~/.bash_login so I don't want to have the latter two files. At least on this Pi the process is:

  • Read ~/.profile which has my environmental variables

  • ~/.profile sources ~/.bashrc if it exists. The latter has local Bash configurations.

  • ~/.bashrc sources ~/.bash_aliases if it exists. Having aliases in a separate file is a "separation of concerns" idea.

November 5, 2020

Here are two LaTeX-like macros for gpp that define two levels of RTF list.

This macro has the bullet flush left with the margin and the text indented as a block by 360/1440 twips or 1/4 inch.

\define{\bulletpoint}{
\paragraph{\fi-360\li360\bullet\tab #1}
}

This second-level bullet has a "white bullet" (specified in unicode decimal format) indented 1/4 inch and the rest of the body indented 1/4 inch more.

\define{\subbulletpoint}{
\paragraph{\fi-360\li720{\u9702-}\tab #1}
}

Both macros depend on \paragraph{} so that the bullet points inherit the paragraph spacing and other formatting. An example paragraph macro is:

\define{\paragraph}{{\pard \ql \sa60 \sb60 \f0 \fs24 \kerning12
#1
\par}}

October 16, 2020

SAS has special missing data codes which represent

a type of numeric missing value that enables you to represent different categories of missing data by using the letters A-Z or an underscore.

R doesn't have this built in. But here is prototype of how to have missing data codes. The data frame has three rows and two variables. Then first variable is an integer ID variable and the second is trails_a which holds the time in seconds a person takes on part A of a test called Trails Making Test. If the person does not have a score on the test, we can record the reason.

I don't think data.frame can generate this object so I use structure with row.names and class attributes. The trails_a variable itself is a list with two elements each of length 3. In a way this is like an embedded data frame.

d <- structure(list(id = c(11L, 22L, 33L),
                    trails_a = structure(list(score = c(50L,
                                                        NA_integer_,
                                                        NA_integer_),
                                              reason = c(NA_character_,
                                                         "Exceeded time limit",
                                                         "Too severe to test")),
                                         class = "trailsList"),
                    random = runif(3)),
               row.names = c(NA, -3L),
               class = "data.frame")

This doesn't print properly (and gives a "corrupt data frame" warning) unless we create a format method for our trailsList class.

format.trailsList <- function(x, ...){
    ifelse(is.na(x$score),
           sprintf("NA (%s)", x$reason),
           sprintf("%ds", x$score))
}

But with a nice format method we get this:

  id                 trails_a random
1 11                      50s  0.061
2 22 NA (Exceeded time limit)  0.664
3 33  NA (Too severe to test)  0.826

To use a data frame with a trailsList object we need to define a number of other methods including operations like == for subsetting. I don't know all that is involved. But an example to look at would be the methods for Surv in the survival package.

Conclusion

I don't think most analysts would want to deal with this complexity. It's more straightforward and understandable to just use two columns in the data frame: the score and if no score, the reason for NA.

Postscript

Having a variable in a data frame that is of POSIXlt class is an "established" case that is very similar. This is a data frame with two observations and three variables clinic, date, and ldate.

d <- data.frame(clinic = 1:2,
                date = as.Date(c("2010-01-01", "2020-01-05")))
d$ldate <- as.POSIXlt(d$date)
dput(d)
structure(list(id = 1:2,
               date = structure(c(14610, 18266), class = "Date"),
               ldate = structure(list(sec = c(0, 0),
                                      min = c(0L, 0L),
                                      hour = c(0L, 0L),
                                      mday = c(1L, 5L),
                                      mon = c(0L, 0L),
                                      year = c(110L, 120L),
                                      wday = c(5L, 0L),
                                      yday = c(0L, 4L),
                                      isdst = c(0L, 0L)),
                                 class = c("POSIXlt", "POSIXt"),
                                 tzone = "UTC"),
               row.names = c(NA, -2L),
               class = "data.frame")

September 21, 2020

PowerShell

I used to look down my nose at PowerShell but now I realize there are lots of shells that do a good job. I need to learn the equivalent of aliases in PowerShell to make navigating the Windows filesystem faster but for now if I navigate to a directory with a file that I want to open (say a Word document) then I can open quickly with

 ii myfile.docx

Here, ii is a shortening of Invoke-Item.

September 15, 2020

Lattice graphics

A StackOverflow question asked about removing tickmarks on the right and top margins of a plot.

https://stackoverflow.com/questions/62945745/removing-the-ticks-on-the-3rd-and-4th-axes-in-plots-from-library-effects-in-r

The context was

library(effects)
m <- lm(Fertility ~ ., data = swiss)
plot(allEffects(m), rug = FALSE)

And plot from the effects package didn't allow the user much control. For example, the usual scales argument wasn't handled. This was my solution:

trellis.par.set(axis.components = list(top = list(tck = 0),
                                       right = list(tck = 0))
plot(allEffects(m), rug = FALSE)

Another thing I learned was to to control the font family easily. Using trellis.par.set outside of a plot cal or using par.settings inside a plot call, one of the elements is grid.pars. See ?gpar.

str(get.gpar())
xyplot(dist ~ speed, data = cars,
       par.settings = list(grid.pars = list(fontfamily = "Open Sans")))

August 27, 2020

Polymode

In a RMarkdown file edited with Emacs and Polymode, the command polymode-eval-buffer-from-beg-to-point is bound to both M-n v <up> and M-n v u.

The command polymode-eval-buffer is bound to M-n v b.

The command polymode-eval-region-or-chunk is bound to M-n v v.

August 26, 2020

Lattice graphics

  • Axis tick marks on log scale with "normal" numbers: list(log = 2, equispaced.log = FALSE)
  • Gridlines to match tick marks: panel.grid(h = -1, v = -1)

August 20, 2020

The data.table package has fread which is great. One difference between fread and read.csv is how a field like ,"", is handled. Here is an example file:

 "id","date1","date2","code"
 "1","2019-01-01","","3"
 "1","2019-01-01","2018-03-18","3"

Using read.csv with defaults gives:

 R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE))
 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: chr  "2019-01-01" "2019-01-01"
 $ date2: chr  "" "2018-03-18"   <---- Empty string
 $ code : int  3 3

Using na.strings = "" gives

 R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: chr  "2019-01-01" "2019-01-01"
 $ date2: chr  NA "2018-03-18"   <---- Beter
 $ code : int  3 3

But I do not know a way to get the same from fread:

 R> str(fread("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
 Classes 'data.table' and 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: chr  "2019-01-01" "2019-01-01"
 $ date2: chr  "" "2018-03-18"
 $ code : int  3 3
 - attr(*, ".internal.selfref")=<externalptr>

By design, fread wants to be smart about allowing an NA field to be data (e.g., the string "NA". It also wants to allow zero length strings. I don't know if there is a way around this.

For example, this doesn't work:

 R> str(fread("test-fread.csv", colClasses = c("date1" = "Date", "date2" = "Date")))
 Classes 'data.table' and 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: Date, format: "2019-01-01" "2019-01-01"
 $ date2: chr  "" "2018-03-18"
 $ code : int  3 3
 - attr(*, ".internal.selfref")=<externalptr>
 Warning message:
 Column 'date2' was requested to be 'Date' but fread encountered the following error:
 character string is not in a standard unambiguous format
 so the column has been left as type 'character'

This comes up with ADNI data and the PTDEMOG.csv. Maybe the root of the problem is having everything quoted in the source CSV.