Nov. 17, 2024

I installed Proton VPN on Lubuntu using these two links:

• https://protonvpn.com/support/official-linux-vpn-ubuntu

• https://protonvpn.com/support/linux-vpn-lubuntu

  wget https://repo.protonvpn.com/debian/dists/stable/main/binary-all/protonvpn-stable-release_1.0.6_all.deb sudo dpkg -i ./protonvpn-stable-release_1.0.6_all.deb && sudo apt update echo "e5e03976d0980bafdf07da2f71b14fbc883c091e72b16772199742c98473002f protonvpn-stable-release_1.0.6_all.deb" | sha256sum --check - sudo apt install proton-vpn-gnome-desktop sudo apt install libayatana-appindicator3-1 gir1.2-ayatanaappindicator3-0.1 gnome-shell-extension-appindicator

Our app relies on the nm-applet NetworkManager applet to work, but Lubuntu uses nm-tray by default. To fix this, go to Menu → Preferences → LXQt settings → LXQt Configuration Center.

Then click on Session Settings and

Go to the Autostart tab → Global Autostart and: Check the Network box Uncheck the nm-tray box

Then restart the machine. Click on the main menu and search for ProtonVPN, enter credentials. I do not know if one has to enter their Proton credentials after every reboot. That would be a minor pain but it makes sense.

Nov. 15, 2024

I am trying Proton Bridge which promises to allow me to use a local mail client. I started here: https://proton.me/support/installing-bridge-linux-deb-file.

cd ~/Downloads
wget https://proton.me/download/bridge/protonmail-bridge_3.13.0-1_amd64.deb
sudo apt install ./protonmail-bridge_3.13.0-1_amd64.deb

Oct. 29, 2024

I figured out how to have Fossil ignore a virtual environment directory. In the checkout directory I did this:

mkdir -p .fossil-settings
echo "venv" > .fossil-settings/ignore-glob

Then fossil add didn't show the venv directory. So easy.

See:

• https://fossil-scm.org/home/doc/trunk/www/globs.md#settings
• https://fossil-scm.org/home/doc/trunk/www/settings.wiki

Oct. 29, 2024

I logged into my Unix account, I did

cd ~/Fossils
fossil new myproject.fossil
mkdir myproject
cd myproject
fossil open ../myproject.fossil

Then from my local computer I cloned the repo via this:

fossil clone ssh://myname@example.com//a/b/c/myname/Fossils/myproject.fossil myproject.fossil

And it worked. And I could commit to the server.

Oct. 27, 2024

To prevent a future web search, I used a Digital Ocean article to add a user to the sudo group:

sudo usermod -aG sudo <username>

And to show whitespace in Nano it is Meta-P.

Oct. 24, 2024

This is how I zipped up my homed directory while excluding a Python virtual environment directory (myvenv)

cd ~/
zip -r backup.zip ./ -x ./project/myvenv/*

I don't think the quotes are needed.

Oct. 21, 2024

Here is a sketch of how to use R's Lattice graphics to show group-wise estimates in panel A and their difference in panel B keeping things lined up nicely.

library(lattice)
library(data.table)

d <- data.table(expand.grid(letters = LETTERS[1:11],
                            percentile = c("P25", "P75")))
d[, mean := rnorm(nrow(d))]

## duplicate the differences so the y-axs scales are the same for both plots
d[, difference := mean[percentile == "P25"] - mean[percentile == "P75"], letters]
d[letters == "A"]

#    letters percentile    mean difference
#     <fctr>     <fctr>   <num>      <num>
# 1:       A        P25  0.0746      0.128
# 2:       A        P75 -0.0538      0.128

p1 <- dotplot(factor(paste(letters, percentile)) ~ mean, data = d, main = "Estimates")

p2 <- dotplot(factor(paste(letters, percentile)) ~ difference,
              data = d,
              main = "Difference",
              panel = function(x, y) {
                ## If y is odd plot y + 0.5, otherwise NA
                panel.xyplot(x, ifelse(unclass(y) %% 2 == 1, unclass(y) + 0.5, NA))
                panel.refline(h = 1:22)
              },
              scales = list(y = list(draw = FALSE)))
graphics.off()
plot(p1, more = TRUE, newpage = TRUE, split = c(1, 1, 2, 1))
plot(p2, more = FALSE, newpage = FALSE, split = c(2, 1, 2, 1))

October 12, 2024

Today I wrote this function to insert the current date into the buffer. At one time I knew about format-time-string but had to search the Web for how to insert a formated date. I also was tripped up by my function initially not returning anything but then I remembered insert is what I want. I don't know if interactive is used correctly here.

(defun stephen-insert-date ()
  "Insert current date.
Replace two spaces with one space if needed."
  (interactive)
  (insert
   (replace-regexp-in-string "  " " " (format-time-string "%B %d, %Y"))))

(global-set-key (kbd "C-c d") 'stephen-insert-date)

Oct. 9, 2024

On Windows, I used Scoop to install coreutils and have Grep in the form of grep.exe. I can run a command like this:

grep 'Emacs' README.md           Doesn't work with CMD
grep Emacs README.md
grep "Emacs" README.md

and get nice results. But if I have two Markdown files to search, CMD and PowerShell do not allow me to use file globbing.

grep "Emacs" *.md

One has to use some kind of for loop to collect the files into a variable.

In Emacs, I learned about file-expand-wildcards:

(file-expand-wildcards "*.org")   ;; relative paths
(file-expand-wildcards "*.org" t) ;; Full paths

Maybe I can create a function that allows me to use Emacs's grep but rather than having the OS do the globbing, I can use this function.

Here is a first attempt:

(defun stephen-grep ()
  (interactive)
  (let ((command "grep --color=always -nH --null -e")
	(pattern (read-string "grep pattern: "))
	(file-list
	 (mapconcat
	  #'format
	  (file-expand-wildcards (read-string "file pattern: "))
	  " ")))
    (grep (format "%s \"%s\" %s" command pattern file-list))))

I looked at my Emacs Lisp: an Introduction by Chassell and think I abused interactive so here is a better way:

(defun stephen-grep (pattern file-list)
  (interactive (list (read-string "grep pattern: ")
		     (mapconcat
		      #'format
		      (file-expand-wildcards (read-string "file pattern: "))
		      " ")))
  (let ((command "grep --color=always -nH --null -e"))
    (grep (format "%s \"%s\" %s" command pattern file-list))))

This is even better.

(defun stephen-grep (command pattern file-list)
  (interactive
   (list (read-string
	  (format "grep command (defaults to \"%s\"): " grep-command)
	  nil nil grep-command)
	 (read-string "grep pattern: ")
	 (mapconcat
	  #'format
	  (file-expand-wildcards (read-string "file pattern: "))
	  " ")))
  (grep (format "%s \"%s\" %s" command pattern file-list)))

Sept. 23, 2024

In R's lattice graphics, most extra arguments are passed on to the panel function but today I learned that p won't work and it's not clear why.

This works and prints out letters:

xyplot(dist ~ speed,
       data = test,
       myletters = letters,
       panel = function(x, y, ...) {
           dots <- list(...)
           print(dots)
           panel.xyplot(x, y)
       })

This does not work.

xyplot(dist ~ speed,
       data = test,
       p = letters,
       panel = function(x, y, ...) {
           dots <- list(...)
           print(dots)
           panel.xyplot(x, y)
       })

It prints out

Error in get(FUN) : first argument has length > 1

So if I want to pass additional arguments I should use longer argument names that won't cause a collision. (For example pvalue rather than p.)

Aug. 21, 2024

I was confused about Make and .PHONY. Let's say I have this makefile. The idea is I have a data set named data.txt and the file report.txt depends on it.

This is how I was using .PHONY:

.PHONY: clean                      # .PHONY is correct here
clean:
	rm -f data.txt report.txt

.PHONY: data                       # .PHONY not helpful
data: data.txt
data.txt:
	echo "This is data" > data.txt

.PHONY: report                     # .PHONY not helpful
report: report.txt
report.txt: data 	# always runs; could replace with `data.txt`
	cat data.txt > report.txt
	echo "This is the report" >> report.txt

The .PHONY for clean is correct. clean is not a file but if there were a file named clean then clean wouldn't run since clean (the file) would never be out of date. That makes sense.

But data isn't file so shouldn't it be phony? The GNU Make documentation says this:

A phony target should not be a prerequisite of a real target file; if it is, its recipe will be run every time make considers that file. As long as a phony target is never a prerequisite of a real target, the phony target recipe will be executed only when the phony target is a specified goal (see Arguments to Specify the Goals).

Since the prerequisite for report.txt is data, which is a phony target, I do in fact have the problem that the recipe is run every time Make considers report.

So take away .PHONY data. But when report.txt depends on multiple data files how do I specify them without listing each file? I guess it could be

data = data1.txt data2.txt

I have to think I'm using Make wrong and should just re-read the manual.

One postscript is that I learned about make -d to print debugging information but what makes it useful is make -d -r which is debugging info but disable the build-in implicit rules which will make the output much shorter.

Aug. 10, 2024

This GNU Make snippet uses the dir and notdir functions with $< which is the first prerequisite.

output/plot.pdf: scripts/program.R
    cd $(dir $<); Rscript $(notdir $<)

This can be almost cryptic with what follows which is equivalent.

output/plot.pdf: scripts/program.R
    cd $(<D); Rscript $(<F)

On the other hand, maybe once one learns <D and <F as the directory part and filename part of the first prerequisite it is elegant.

Aug. 8, 2024

Here is how I could take all files of the from *table.org, use Org to format them via org-table-align, and output *table-fmt.org files.

.PHONY: org-tables
org-tables: $(patsubst %table.org, %table-fmt.org, $(wildcard *table.org))

%table-fmt.org : %table.org
	emacs --quick --batch --eval "
               (with-temp-buffer
                   (insert-file \"$<\")
                   (org-table-align)
                   (write-file \"$@\"))"

Aug. 7, 2024

On Windows, from PowerShell or Command Prompt, and using Emacs Speaks Statistics within Emacs, R knows about

c:\Users\MYNAME\Documents\.Rprofile

But using GNU make (installed via Scoop), R looks for .Rprofile in

c:\Users\MYNAME

I tried to set HOME in the Make file but I couldn't get R to know about the .Rprofile in my Documents directory. The simplest thing that works is now I have this in c:\Users\MYNAME\.Rrofile:

message("This file is 'C:/Users/WEIGAND/.Rprofile'")
message("We will source 'C:/Users/WEIGAND/Documents/.Rprofile'")
source("C:/Users/WEIGAND/Documents/.Rprofile")

Aug. 4, 2024

It looks like the key required to type F4 is also dead. I only wanted this in emacs for keyboard macros but kmacro.el says C-x ( and C-x ) and C-x e are usable.

It looks like C-x e is kmacro-end-and-call-macro; C-x ) is kmacro-end and C-x ( is kmacro-start-macro.

If I create a macro to insert the letter x, this is the output of M-x insert-kbd-macro:

(setq last-kbd-macro "x")

But I couldn't replay that macro in the minibuffer.

C-x C-k n will name the keyboard macro. I just called it my-r.

C-x C-k b will bind the most recent keyboard macro to a key sequence.

Finally

To avoid problems caused by overriding existing bindings, the key sequences C-x C-k 0 through C-x C-k 9 and C-x C-k A through C-x C-k Z are reserved for your own keyboard macro bindings.

And to bind a keyboard macro to, say, A, one does C-x C-k b A.

This makes me wonder if a keychord with xk standing in for C-x C-k would make sense.

Aug. 3, 2024

On my 2008 MacBook I have two keys that are not working. The laptop has a top row of keys with escape, then F1 though F12 and then eject. Can I use F5 and F6 as substitutes for my two bad keys?

These are my problem keys:

• r is ASCII 114
• R is ASCII 82
• 4 is ASCII 52
• $ is ASCII 36

I wrote a series of functions and keyboard shortcuts in Emacs lisp as workarounds.

But then I found F5 and F6 were "available" and wrote these:

(defun myfix-r ()
  "Insert an 'r'."
  (interactive)
  (insert-char 114))
(global-set-key (kbd "<XF86KbdBrightnessDown>") 'myfix-r)

(defun myfix-R ()
  "Insert an 'R'."
  (interactive)
  (insert-char 82))
(global-set-key (kbd "<XF86KbdBrightnessUp>") 'myfix-R)

But this is wrong-headed; I found out when attempting an incremental search in the *Packages* buffer that I can't do C-s and then spell out "markdown-mode" since my fix inserts an "r" rather than types one. I don't know if this has something to do with how self-inset-command works. I think I need a system-level fix.

Previously, since this MacBook has a single, difficult to reach control key (on the left), I edited /etc/default/keyboard as follows:

# I saw this on Mastodon. It makes capslock an additional control
XKBOPTIONS="ctrl:nocaps"

My thinking was that I can do what I want using this file.

The first thing I leaned was that xmodmap -g gives the grammar.

xmodmap accepts the following input expressions:

    pointer = default              reset pointer buttons to default
    pointer = NUMBER ...           set pointer button codes
    keycode NUMBER = [KEYSYM ...]  map keycode to given keysyms
    keysym KEYSYM = [KEYSYM ...]   look up keysym and do a keycode operation
    clear MODIFIER                 remove all keys for this modifier
    add MODIFIER = KEYSYM ...      add the keysyms to the modifier
    remove MODIFIER = KEYSYM ...   remove the keysyms from the modifier

where NUMBER is a decimal, octal, or hex constant; KEYSYM is a valid
Key Symbol name; and MODIFIER is one of the eight modifier names:  Shift,
Lock, Control, Mod1, Mod2, Mod3, Mod4, or Mod5.  Lines beginning with
an exclamation mark (!) are taken as comments.  Case is significant except
for MODIFIER names.

Keysyms on the left hand side of the = sign are looked up before any changes
are made; keysyms on the right are looked up after all of those on the left
have been resolved.  This makes it possible to swap modifier keys.

This grammar is for the xmodmap command but it gives a hint of the format of the text I'll enter in /etc/default/keyboard.

I think an important command in Linux is xev -event keyboard. I do this and then press first "q" and then "s" and this tells me the keycode for "q" is 24 and that for "s" is 39. The output below shows press events and release events.

KeyPress event, serial 25, synthetic NO, window 0x2800001,
    root 0x22f, subw 0x0, time 13104639, (760,452), root:(762,482),
    state 0x0, keycode 24 (keysym 0x71, q), same_screen YES,
    XLookupString gives 1 bytes: (71) "q"
    XmbLookupString gives 1 bytes: (71) "q"
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2800001,
    root 0x22f, subw 0x0, time 13104751, (760,452), root:(762,482),
    state 0x0, keycode 24 (keysym 0x71, q), same_screen YES,
    XLookupString gives 1 bytes: (71) "q"
    XFilterEvent returns: False

KeyPress event, serial 28, synthetic NO, window 0x2800001,
    root 0x22f, subw 0x0, time 13111655, (760,452), root:(762,482),
    state 0x0, keycode 39 (keysym 0x73, s), same_screen YES,
    XLookupString gives 1 bytes: (73) "s"
    XmbLookupString gives 1 bytes: (73) "s"
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2800001,
    root 0x22f, subw 0x0, time 13111735, (760,452), root:(762,482),
    state 0x0, keycode 39 (keysym 0x73, s), same_screen YES,
    XLookupString gives 1 bytes: (73) "s"
    XFilterEvent returns: False

But I found out the keycode for "e" is 26 and "t" is 28 so for "r" it is probably 27. But that is not important since th R key is dead. Maybe I need to know this which is my brightness down key (but it has a little F5 on the key):

KeyPress event, serial 25, synthetic NO, window 0x2800001,
    root 0x22f, subw 0x0, time 13474566, (229,174), root:(231,204),
    state 0x0, keycode 237 (keysym 0x1008ff06, XF86KbdBrightnessDown), same_screen YES,
    XLookupString gives 0 bytes:
    XmbLookupString gives 0 bytes:
    XFilterEvent returns: False

Here is what may work using the format keycode <KEYCODE> = <KEYSYM>

xmodmap -e "keycode 237 = 72"

Well, that gave me an h but though trial and error this gives me an "r":

xmodmap -e "keycode 237 = 82"

Is it chance that ASCII 82 is a capital "R"? I don't know but after running that, shift with brightness down gives me a capital "R". Yay.

OK, now I see that the keycode is for, well, a key, and a capital letter has a shift event and a key event which gives a capital character.

Now can I get a "4"?

xmodmap -e "keycode 238 = 52"

That works but with shift I do not get the dollar sign.

xmodmap -e "keycode 173 = 36"

OK, now all good. Actually, better is http://xahlee.info/linux/linux_xmodmap_tutorial.html.

keycode 238 = 4 dollar 4 dollar

Finally, this is what my ~/.Xmodmap looks like:

! Added by Stephen Aug. 3, 2024
! See http://xahlee.info/linux/linux_xmodmap_tutorial.html


! Key with small F5 on it gives me an r/R.
keycode 237 = r R r R

! Key with small F6 on it gives me a 4/$
keycode 238 = 4 dollar 4 dollar

But I had a hard time getting this to autostart. Since I am using Lubuntu which uses LXQt, I found I could do this: Menu -> Preferences -> LXQt Settings -> Session Settings and then click on Autostart and then New. I named an item "Xmodmap" and had the command be xmodmap ~/.Xmodmap. That works.

That works and does so by creating a file called ~/.config/autostat/Xmodmap.desktop which is this ini-style file:

[Desktop Entry]
Exec=xmodmap ~/.Xmodmap
Name=Xmodmap
OnlyShowIn=LXQt;
Type=Application
Version=1.0

From man startx:

Note that in the Debian system, what many people traditionally put in the .xinitrc file should go in .xsession instead;

But the LXQt way is different.

This took most of a Saturday but it made the laptop usable and I needed to be supine for most of the day to rest my back.

Aug. 2, 2024

I forget what Org entities are available so I need to remember the command M-x org-entities-help.

Aug. 2, 2024

In Emacs, to widen or narrow the frame, I learned about this

M-x set-frame-width

And programmatically I can use this

(set-frame-width (selected-frame) 125)

which could be helpful if I wanted to write functions to set the frame to, say, three different widths.

Aug. 2, 2024

Using R on a Windows machine, I had trouble installing the R package deldir using

install.packages("deldir", repos = "https://mirror.las.iastate.edu/CRAN/")

I think it was because this package needed to be compiled and maybe the zip file comes with suspicious binaries. (It includes a file called deldir/libs/x64/deldir.dll which a human could read as being related to deleting a directory.)

I don't know why a package like data.table which needs compilation doesn't have any DLL or EXE files and whether that is related to why I can install it.

All this comes because (I think) the latticeExtra package can't be loaded without the deldir package even though the latter is only suggested for latticeExtra.

July 31, 2024

I had trouble updating Scoop, getting this message:

fatal: unable to access 'https://github.com/lukesampson/scoop/': SSL certificate problem: unable to get local issuer certificate

I know nothing about certificates but searching for the problem led me to GitHub advice to the effect of:

git config --global http.sslVerify "false"

or

git config --global http.sslBackend openssl

Turning off SSL verification works but seems bad. Setting the backend to schannel which is "the native Windows Secure Channel library" seems to work.

I also learned that using --global means the file C:\Users\myname\.gitconfig is updated. Here's what mine looks like now:

[credential]
	helper = manager
[credential "helperselector"]
	selected = <no helper>
[http]
	sslBackend = schannel
	sslVerify = true

July 12, 2024

I wanted to use Make to process one type of file to create another without me having to specify each file to process. That is, I want n.foo to be converted to n.bar automatically.

An example "pattern rule" to do this in Make is of the following form:

%.bar : %.foo
      myconverter $< $@

This generates n.bar from n.foo using myconverter. (I always forget the "automatic variables" but $< is the first prerequisite and $@ is the target.)

The problem is that even if n.foo exists and the Makefile is only the above two lines, the command make will give the following:

make: *** No targets.  Stop.

I didn't understand this. I found that if the file n.foo exists I can enter make n.bar and I will get n.bar. But I don't want to list out each I want on the command line.

The documentation pointed out my problem:

The order of rules is not significant, except for determining the default goal: the target for make to consider, if you do not otherwise specify one. The default goal is the first target of the first rule in the first makefile. There are two exceptions: a target starting with a period is not a default unless it also contains one or more slashes, ‘/’; and, a target that defines a pattern rule has no effect on the default goal.

I take the problem to be answered by the last sentence. I have a pattern rule but it has no effect on the default goal (or targets). My single line file doesn't have any goals.

But I can add the "all" goal which creates a list of the .bar files I want.

all: $(patsubst %.foo, %.bar, $(wildcard *.foo))

%.bar : %.foo
      myconverter $< $@

This may not be a perfect explanation but it's close enough and my problem is solved.

May 15, 2024

I don't understand formulas and their environments in R. I wanted this to work:

library(data.table)
library(lme4)

## Example of model to fit
fit <- lmer(log(y) ~ x1 + x2 + (1 | unit),
            data = mydata,
            subset = region == 1)


## This doesn't work
out <- mydata[, coef(summary(lmer(formula(fit)))), region]

But this works of course:

out <- mydata[, coef(summary(lmer(log(y) ~ x1 + x2 + (1 | unit)))), region]

A solution is to use .SD so

out <- mydata[, coef(summary(lmer(formula(fit), data = .SD))), region]

In the manual page for formula it says:

A formula object has an associated environment, and this environment (rather than the parent environment) is used by model.frame to evaluate variables that are not found in the supplied data argument.

I take this to mean that when I did this:

out <- mydata[, coef(summary(lmer(formula(fit), data = .SD))), region]

then lmer was using the environment for formula(fit) to look for variables since I didn't supply a data argument. But if I supply a data argument via .SD I am OK.

May 7, 2034

From https://opensource.com/article/19/8/dig-binary-files-hexdump I learned I can customize the output for hexdump:

hexdump -e'60/1 "%02x""\n"' myfigure.png

Will output the file as "pure" hexadecimal, kind of like what Emacs's hexlify-buffer does except without the address information and ASCII representation.

I also learned that hexlify-buffer relies on a binary called hexl on my system. On my system it is found at

/usr/libexec/emacs/29.1/x86_64-linux-gnu/hexl

The source is https://git.savannah.gnu.org/cgit/emacs.git/tree/lib-src/hexl.c and I don't think there are any meaningful formatting options.

I think I can customize the Emacs variables hexl-program and hexl-options to use hexdump if available.

I could also use hexlify-region and then clean up the non-hex columns.

May 5, 2024

• I went to https://zoom.us/download?os=linux
• I selected Ubuntu 64 bit and clicked download
• Then in the directory where the DEB file ended up: sudo apt install ./zoom_amd64.deb

April 26, 2024

Here is an Emacs lisp file that could form the basis of a command line script.

#!/usr/bin/emacs -x
(print command-line-args)

March 27, 2024

I was looking in my Lattice: Multivariate Data Visualization with R book and re-learned about the page argument to Lattice plots and how it is a good way to add annotations at the whole-page level. This function has as its coordinate system the full display area scaled to [0,1] by [0,1].

The page argument takes a function which has a single argument which corresponds to the page number. (Lattice graphics can be multiple pages.) In the example below, I ignore the n argument and draw text with panel.text and grid.text.

library(lattice)
library(grid)
xyplot(dist ~ speed,
       data = cars,
       page = function(n) {
         panel.text(labels = "A.", x = 0.01, y = 0.99, adj = c(0, 1), cex = 3)
         grid.text(label = "B", x = 0.99, y = 0.99, just = c("right", "top"),
                   gp = gpar(col = "firebrick"))

       })

I need to learn more about complicated legends because they may be a good way to add even more plot annotations but this will do for simple text annotations.

March 6, 2024

Using R's lattice and latticeExtra here is a scatter plot smoother with intervals by group

d <- data.frame(x = rnorm(100), y = rnorm(100), g = rep(c("A", "B"), c(25, 75)))
xyplot(y ~ x, d, groups = g,
       panel = panel.superpose,
       panel.groups = function(x, y, ...) {
         panel.points(x, y, ...)
         panel.smoother(x, y, method = "loess", span = 0.9, ...)
       })

January 13

In Org, I can use Babel to generate Org. The code block below sets the results to value and raw. (Using output rather than raw doesn't work.) The raw says to insert the results of running the code block as-is rather than wrapping it in any way.

#+begin_src elisp :results value raw :exports none
(concat "** Level 2 headline\n"
        "*** Level 3 headline\n"
        "A short sentence.\n")
#+end_src

And then if I do C-c C-c in the block above, the buffer looks like this:

#+begin_src elisp :results value raw :exports none
(concat "** Level 2 headline\n"
        "*** Level 3 headline\n"
        "A short sentence.\n")
#+end_src

#+RESULTS:
** Level 2 headline
*** Level 3 headline
A short sentence.

This is great but a limitation is that if I do C-c C-c again in the code block, then the RESULTS are inserted again rather than replaced. I think this is because once the Org text is in the buffer, Org has no way to know it's a code block's output. This makes sense if I think of it as raw Org; it's not a block to be "updated" via Babel.

I originally looked into this to use code to auto-generate a statistical report. I think it would be useful as "one-time auto-generation". For example, if I had a number of pies and I wanted to show one figure per pie and separate them by headlines, I could have this code block:

#+begin_src R :results value raw :exports none
headlines <-
    c(`Pumpkin pie` = "./figures/pumpkin.png",
      `Apple pie` = "./figures/apple.png")
sprintf("** %s\n[[file:%s]]\n",
        names(headlines),
        headlines)
#+end_src

And do C-c C-c and it generates this.

#+RESULTS:
** Pumpkin pie
[[file:./figures/pumpkin.png]]

** Apple pie
[[file:./figures/apple.png]]

I would run it once interactively to fill in the report.

I don't know if this is particularly useful. I could write a script to generate this Org text outside of Org but now that I think about it, it's much more maintainable to have the code block in the file. If I want to change what is output, I have to remove it from the Org file and regenerate it with C-c C-c but at least the report Org file shows what will be in the exported document.

November 1

In R, I can use read.csv to read a particular CSV in a Zip archive

sl <- read.csv(unzip("../data/file.zip",
                     files = "folder_unzips_to/file.csv")

October 27

In Emacs I am trying the cyberpunk theme because it works well with the rainbow delimiters package but cyberpunk was giving a light background to Org source blocks. I learned how to customize it by going to an empty part of the source block and typing describe-face and seeing that the face was org-block. I changed the background color to black (#000000) and pressed Save and Apply.

October 25, 2023

In Emacs, I sometimes type C-x C-- by accident which calls text-scale-adjust to shrink the text size of the window. And C-x C-+ makes it bigger. After either C-x C-- or C-x C-+ then - makes text smaller, + makes it bigger, and 0 returns it to the default.

October 24, 2023

Using Lattice graphics in R, I needed an axis on the square root scale for a scatter plot matrix generated with lattice::splom. The scales are controlled by the pscales argument to splom (but really pscales is an argument to panel.pairs).

If a single number, the pscales argument controls how many tick marks are desired on the axes. But it can also be a list with one component for each column being plotted. The list should have these elements:

‘at’: a numeric vector specifying tick locations

‘labels’: character vector labels to go with at

‘limits’: numeric 2-vector specifying axis limits (should be
made more flexible at some point to handle factors)

These are specifications on a per-variable basis, and used on
all four sides in the diagonal cells used for labeling.
Factor variables are labeled with the factor names.  Use
‘pscales=0’ to suppress the axes entirely.

One thing I found is that the list's elements should be at, lab, and lim to avoid partial match warnings since panel.pairs uses pscales[[i]]$lab and pscales[[i]]$lim.

Here's an example. It's not ideal.

library(lattice)
d <- data.frame(x = exp(rnorm(100, 1)),
                y = exp(rnorm(100, 1)))

calc.pscales <- function(x.untransformed) {
    list(at = sqrt(pretty(x.untransformed)),
         lab = pretty(x.untransformed),
         lim = extendrange(sqrt(x.untransformed), f = 0.1))
}
lapply(d, calc.pscales)

splom(sqrt(d), pscales = lapply(d, calc.pscales))

October 16, 2023

I like text-based help in R and so in my ~/.Rprofile I had

options(help_type = "text")

That was working on Linux but not on Windows. It turns out adding this to ~/.Rprofile in Emacs on Windows does not help. To find out where R is looking for .Rprofile I read the manual and found:

R> Sys.getenv("R_USER")

This is c:\Users\WEIGAND\Documents.

So I needed to edit c:\Users\WEIGAND\Documents\.Rprofile.

October 15, 2023

Some variables in Emacs are buffer variables. For example setting show-trailing-whitespace to something non-nil means highlight trailing whitespace in the buffer. But the help for this variable says

Automatically becomes buffer-local when set.

I learned that in order to set this variable in my init.el file, I need to use

;; (setq show-trailing-whitespace t) <-- Won't do what I want
(setq-default show-trailing-whitespace t)

September 22, 2023

This is how I made a scatter plot in Times New Roman using R's lattice graphics package on Windows:

library(lattice)
library(grid)

windowsFonts(TNR = "Times New Roman")

xyplot(dist ~ speed,
       cars,
       main = "A scatter plot in Times New Roman",
       par.settings = list(grid.pars = gpar(fontfamily = "TNR"),
                           fontsize = list(text = 24)))

On Linux it was more direct:

library(lattice)

xyplot(dist ~ speed,
       cars,
       main = "A scatter plot in Times New Roman",
       par.settings = list(grid.pars = gpar(fontfamily = "Times New Roman"),
                           fontsize = list(text = 24)))

September 18, 2023

Here is a strip plot in lattice that shifts points by group to reduce overlap:

stripplot(wool ~ breaks,
          data = warpbreaks,
          groups = tension,
          panel = panel.superpose,
          panel.groups = function(x, y, group.value, ...) {
              dots <- list(...) # Print this to see what else is in `...`
              panel.xyplot(x, y + c(-0.1, 0, 0.1)[match(group.value, c("L", "M", "H"))], ...)
          })

September 18, 2023

R's lattice package makes it easy to add a regression line by a grouping factor but I did not know how to add confidence intervals to the regression line. This uses the latticeExtra package to "add" a regression line with 95% CI but it does not do it by group:

library(lattice)
library(latticeExtra)

n <- 100
test <- data.frame(x = rnorm(n),
                   y = rnorm(n),
                   g = sample(letters[1:3], size = n, replace = TRUE))

xyplot(y ~ x, test, groups = g) + layer_(panel.smoother(x, y, method = "lm"))

The solution I came up with is to create a simple panel function that plots points and adds a regression line with a 95% CI. It uses panel.points from lattice and panel.smoother from latticeExtra. Then I created a simplified version of panel.xyplot which uses panel.superpose to handle grouping.

## A simple panel function with a scatter plot and a smoother
panel.simple <- function(x, y, ...) {
    panel.points(x, y, ...)
    panel.smoother(x, y, method = "lm", ...)
}

## A "groups-aware" panel function the uses `panel.superpose`
panel.xyplot.simple <- function(x, y, groups = NULL, ...) {
    if (is.null(groups)) stop("We need 'groups' to not be NULL")
    panel.superpose(x, y, groups = groups, panel.groups = panel.simple, ...)
}                                        # ^^^^^^^^^^^^   ^^^^^^^^^^^^

## A plot with group-specific regression lines and 95% CIs.
xyplot(y ~ x, test, groups = g, panel = panel.xyplot.simple)

And here is a better way that only requires me to write panel.simple and which is documented in the manual page for xyplot:

xyplot(y ~ x, test, groups = g, panel = panel.superpose, panel.groups = panel.simple)

August 6, 2023

I learned how to publish a project using Emacs Org mode. The manual explains it well (see Publishing).

This is my alist with one project (www) but there can be multiple projects.

  (setq org-publish-project-alist
	'(("www"
	   :base-directory "~/fossils/www/"
	   :publishing-function org-html-publish-to-html
	   :publishing-directory "~/fossils/www"
	   :section-numbers nil
	   :with-toc nil
	   :html-doctype  "<!DOCTYPE html>"
	   :html-head "<link href=\"normalize.css\"                      rel=\"stylesheet\" type=\"text/css\">
		       <link href=\"sakura-vader.css\" id=\"sakura-css\" rel=\"stylesheet\" type=\"text/css\">")))

Then, to publish, it's org-publish-project and when prompted say which project. Putting the parameters in the org-publish-project-alist reduces boilerplate in the Org file. I was doing this at the top of files I was planning to export to HTML.

#+TITLE: My Title
#+OPTIONS: toc:nil num:nil html-style:nil html-scripts:nil html-postamble:nil html5-fancy:t
#+HTML_HEAD: <link href="normalize.css" rel="stylesheet" type="text/css">
#+HTML_HEAD: <link href='sakura-vader.css' id="sakura-css" rel='stylesheet' type='text/css'>
#+HTML_DOCTYPE: html5

My list above is not consistent with some of the lines immediately above which I will try to fix. To see what options can be specified in the list, go to Publishing options.

As an example, the option :html-preamble is controlled by the variable org-html-preamble. It is by default t and non-nil means insert an HTML preamble. If t it inserts a string defined by org-html-preamble-format. But if set to a string it uses the string. And if it is set to a function, apply the function.

The variable org-html-preamble-format is by default (("en" "")). The second element of this enclosed list is a format string as follows:

  %t stands for the title.
  %s stands for the subtitle.
  %a stands for the author’s name.
  %e stands for the author’s email.
  %d stands for the date.
  %c will be replaced by ‘org-html-creator-string’.
  %v will be replaced by ‘org-html-validation-link’.
  %T will be replaced by the export time.
  %C will be replaced by the last modification time.

If this is part of the org-publish-project-alist:

:html-preamble "written by %a")))

The exported HTML will have this div:

<head>
...
</head>
<body>
  <div id="preamble" class="status">
  written by Stephen Weigand
  </div>
  <div id="content">

  <h1 class="title"> ... etc.

August 2, 2023

I'm not sure why using Fossil via ssh started requiring my pass-phrase. I did the following:

PS> ssh-keygen.exe -t ed25519
PS> cd ~/.ssh
PS> cat id_ed25519.pub | ssh weigand@myserver.com "cat >> ~/.ssh/authorized_keys"
                   ^^^
				   IMPORTANT!!!

But then on myserver.com I had to edit authorized_keys and take out the Windows stuff like a carriage return before the newline at the end of the file. And while I was there I deleted a previous entry from this Windows machine.

July 11, 2023

Using R's lattice package I learned or remembered one can give a vector of x-axis, y-axis, and main labels. Here is a 2 by 2 figure with two x-axis labels and two y-axis labels

library(lattice)
n <- 100
d <- data.frame(x = rnorm(n),
                y = rnorm(n),
                g = sample(letters[1:4], size = n, replace = TRUE))
xyplot(y ~ x | g,
       d,
       xlab = c("Flower", "Tree"),
       ylab = c("Shrub", "Fruit"))

Another thing us I remembered to use main as a list. This puts the main title at the left of the viewport. See the main entry in the help file for xyplot.

xyplot(dist ~ speed,
       data = cars,
       main = list(label = "A. Scatter plots", x = unit(0, "npc"), just = "left")

July 1, 2023

The Feb. 9, 2023 entry shows how to delete files in Windows even if they don't exist. In searching for a platform independent way to remove files I remembered fossil clean. See https://fossil-scm.org/home/help?cmd=clean.

Here's how to remove all *.Rout and *.log files which may be in the current or any other directory.

fossil clean --clean *.Rout,*.log

It also works with

fossil clean --clean "*.Rout,*.log"

One can also ask Fossil to remove empty directories as part of the operation:

fossil clean --clean "*.Rout,*.log" --emptydirs

Adding the --force option to the above command will prevent Fossil from asking before removing each empty directory.

May 4, 2023

In R's base graphics, coloring box plots separately by group is trivial:

boxplot(breaks ~ interaction(tension, wool), warpbreaks, col = six.nice.colors)

Using Lattice graphics I learned it can be done by specifying a vector of fill colors as part of the box.rectangle element of the par.settings list.

p.s <- list(box.umbrella = list(col = "black", lty = 1),
            box.rectangle = list(col = "black", fill = six.nice.colors))
bwplot(breaks ~ interaction(tension, wool), warpbreaks, par.settings = p.s, pch = "|")

More general Lattice note

When generating a Lattice plot, creating a plot object doesn't generate any calls to grid::viewport. The viewport function is only called when one plots the object. The reason I am exploring how Lattice works is I would like to make the plots more "table like" so that a stripplot of the form

stripplot(level ~ y | group, layout = c(1,2), as.table = TRUE)

looks like this

Group 1
  Level 1   *  *  *      *
  Level 2    *  * *
  Level 3      **       *   **

Group 2
  Level 1   * * * *
  Level 2  * *   *  **
  Level 3  *   *   **   *

          |---+---+---+---+---|

To customize the strips, strip.default is (probably) not going to help since the strips are drawn to fill up the parent viewport. lattice:::plot.trellis is giving the strip a narrow viewport spanning the panel which strip.default fills up with a background, text, and border.

I think I need to examine lattice:::plot.trellis to see where the viewport for the strip is being created. It might be here:

pushViewport(viewport(layout.pos.row = pos.row - 1,
                      layout.pos.col = pos.col,
                      xscale = xscale,
                      clip = "off",
                      name = trellis.vpname("strip",
                                            column = column,
                                            row = row,
                                            prefix = prefix,
                                            clip.off = TRUE)))

File this under "Today I Don't Know" because I don't have any good sense of where to implement my idea of moving strips left. I think I could say, "Hey, don't generate a default viewport for the strip but a wider version expanding off to the left." But then which function is doing the calculations to make it "fit". (I think it is lattice:::plot.trellis which is huge and gnarly.)

A hack may be to say the text in the strips will be adjusted left to the extent of the existing plot. We don't need (much?) more space on the left margin so long as we are just a little to the left of the y-axis labels (which are serving as subgroup names).

That is, we don't want this which takes a lot of new spacing calculations:

Group 1
        Level 1   *  *  *      *

but rather an "indented look" like this which can use existing or slightly modified padding:

Group 1
  Level 1   *  *  *      *

Maybe the simplest thing is along the lines of this:

myplot <- stripplot(...)
myplot <- add_left_padding_as_needed_based_on_strip_text(myplot)
justify_strip(myplot, just = "left") # Uses grid::grid.edit iterating over panels

Another possible name/metaphor is exdent_strip(myplot). (Or exdentStrip since Lattice often uses camel case.)

April 28, 2023

I remembered to try using Emacs file-local variables to "compile" R files.

In a plain R file I might have this at the top:

## -*- compile-command: "Rscript foo.R" -*-

And in an Rmd file I might have this:

<!-- -*- compile-command: "Rscript -e 'library(\"knitr\"); knit(\"foo.Rmd\")'" -*- -->

April 24, 2023

Pipx

I previously played with pipx to manage python command line applications. I couldn't remember if I used Scoop but it doesn't look like it. Today I'm using Python 3.11 via Scoop and I did:

python -m pip install --user pipx

I then got a warning about c:\users\weigand\appdata\roaming\python\python311\scripts\pipx.EXE not being on my path so I fixed that (via typing "path" in the Windows search bar and clicking on Environmental Variables).

I then followed the pipx suggestion and typed: pipx ensurepath.

Then I did pipx install csvkit and it worked.

in2csv from csvkit

This worked great to convert an Excel XLSX sheet into a CSV.

 in2csv.exe --sheet "Sheet XYZ" myfile.xlsx > sheet_xyz.csv

March 29, 2023

Passing arbitrary arguments to Rscript (or Rscript.exe on Windows) is easy. If a script called test-rscript.R exists and has this one line:

print(commandArgs())

then a command like this:

Rscript --vanilla test-rscript.R --never always -often --

Produces this:

 [1] "C:\\scoop\\apps\\r\\current\\bin\\x64\\Rterm.exe"
 [2] "--no-echo"
 [3] "--no-restore"
 [4] "--vanilla"
 [5] "--file=test-rscript.R"
 [6] "--args"
 [7] "--never"
 [8] "always"
 [9] "-often"
[10] "--"

This shows me that it easy to parse arbitrary command line arguments without having to rely on an external package.

March 28, 2023

I'm trying to write a shell command to export an Org file. I'm using

 emacs test.org --batch --funcall org-html-export-to-html --kill

but it doesn't evaluate the code blocks to generate figures using R. My true aim is RTF export using a local exporter I created called ox-rtf.el. This creates an RTF document but again, no figures.

 emacs test.org --batch --load ox-rtf.el --funcall org-rtf-export-to-rtf --kill

I'm using Batch export of org-mode files from the command line from Stack Overflow. (The answers are nine years old.) There is also this suite of command line tools https://github.com/fniessen/orgmk.

Additional info

The command line above "works" in that it exports the file as it is on disk. This means that if the code blocks are all evaluated then the exporting will show the figures. When editing the file and typing C-c C-e (which calls org-export-dispatch and invokes the menu) the exporting process includes the additional step of evaluating the code blocks. (It may be something like it calls org-babel-execute-buffer then does the exporting.)

The manual has this about

The sample script shows batch processing of multiple files using org-babel-tangle.

#!/bin/sh
# Tangle files with Org mode
#
emacs -Q --batch --eval "
    (progn
      (require 'ob-tangle)
      (dolist (file command-line-args-left)
        (with-current-buffer (find-file-noselect file)
          (org-babel-tangle))))
  " "$@"

So maybe the solution is something like

emacs -Q --visit test.org --batch --load ox-rtf.el --eval "(org-babel-execute-buffer) (org-rtf-export-to-rtf)" --kill

The problem with the above is Org Babel didn't know what to do with R code blocks. So I tried this and it seems to work.

emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el --eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill

If I am using the CMD shell, I can continue lines with ^

emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el ^
  --eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill

One problem is that I am loading my own init.el file and so this command will only work for me. I am using the Org (version 9.5.5) that came with my Emacs (version 28.2).

March 28, 2023

I'm trying to customize my ~eshell~ prompt. That is not working right now but this Emacs lisp snippet was helpful to learn:

(substring (shell-command-to-string "fossil status") 0 -1)

It captures the results of the fossil status command and trims the last character which is a newline.

March 21, 2023

From https://www.masteringemacs.org/article/executing-shell-commands-emacs I read about shell-command for one-liners.

March 21, 2023

A couple of times recently I've been typing madly in Emacs and my buffer's font size gets smaller. After a web search I want to remember these keys:

• C-x C-+ will increase the buffer font size (as will C-x C-= which saves a shift key)
• C-x C-- will decrease the buffer font size

These seem to invoke text-scale-adjust and I also wan to remember

• C-x C-0 will reset to the default size

March 13, 2023

After updating Emacs to 28.2 using Scoop on Windows, my usual way of starting Emacs via the Start Menu search bar starts Emacs with a CMD window behind it. From this:

https://stackoverflow.com/questions/41300/emacs-in-windows

I think the problem is this is starting the executable program emacs.exe but I really want to be starting runemacs.exe. I don't know how to get the old behavior but I found that I could create a "Desktop Shortcut" as follows:

• I right-clicked on runemacs.exe and created a shortcut
• I moved this shortcut to the Desktop folder
• I right-clicked on the shortcut and selected properties
• In the Shortcut key text box I typed "e" so that my shortcut is "Ctrl + Alt + E". (It's a weird box because it is not editable; you just have to type a single character in there and the short cut is "Ctrl + Alt + X".)

While I was doing this I also changed the "double click action" (my term) for files I want to open in Emacs to be opened with runemacs.exe.

A follow-up note is that I originally tried having my shortcut in a Shortcut folder within Desktop but that didn't seem to work. I guess the term "desktop shortcut" means a shortcut on the desktop and not anywhere else.

March 7, 2023

Using Scoop on Windows I updated R and Rtools and then was unable to use the brms package. I used this to see what version I had and what versions were installed.

PS> scoop info r


Name        : r
Description : A free software environment for statistical computing and graphics.
Version     : 4.2.2
Bucket      : main
Website     : https://www.r-project.org
Updated at  : 11/2/2022 1:22:47 AM
Updated by  : Hsiao-nan Cheung
Installed   : 4.1.0
              4.1.1
              4.1.3
              4.2.2
Binaries    : bin\x64\R.exe | bin\x64\Rcmd.exe | bin\x64\Rgui.exe | bin\x64\Rscript.exe | bin\x64\Rterm.exe
Shortcuts   : R
...

I used this to "reset" to 4.1.3.

PS> scoop reset r@4.1.3
Resetting r (4.1.3).
Linking C:\scoop\apps\r\current => C:\scoop\apps\r\4.1.3
Creating shim for 'R'.
Creating shim for 'Rcmd'.
Creating shim for 'Rgui'.
Creating shim for 'Rscript'.
Creating shim for 'Rterm'.
Creating shortcut for R (Rgui.exe)
C:\Users\WEIGAND

This was very, very nice.

Feb. 14, 2023

In Emacs I decided to always have line numbers on and Web searches suggested as of Emacs 26 use (global-display-line-numbers-mode) in one's initialization file. I then customized the line-number face and the line-number-current-line face to both have a height of 80%.

Feb. 9, 2023

I learned how to remove files in Windows CMD even if such files aren't present. Here is an example from a Makefile:

.PHONY: clean
clean:
        cd code && if exist *.png (del *.png)

Jan. 6, 2023

Through trial and error I now understand colors in RTF as implemented by Microsoft Word (365 on the desktop) a little better.

• Highlighting is done via \highlightN. For example, {\highlight5 This text is highlighted} will be highlighted using the fifth color in the color table.

• But value for N has to correspond to a row in the color table.

{\colortbl;
\red0\green70\blue173;     N=1
\red182\green0\blue20;     N=2
\red169\green169\blue169;  ...
\red222\green235\blue247;
\red204\green121\blue167;
\red255\green255\blue0;    N=6 which here is yellow
\red204\green121\blue167;
\red0\green255\blue0;      N=8 which here is green
\red139\green0\blue0;      N=9 which is what R gives for col2rgb("darkred")
}

• And you can't specify an arbitrary color to use for highlighting. Based on http://www.biblioscape.com/rtf15_spec.htm, the colors are:

Value	Description
1	Black
2	Blue
3	Cyan
4	Green
5	Magenta
6	Red
7	Yellow
8	Unused
9	Dark Blue
10	Dark Cyan
11	Dark Green
12	Dark Magenta
13	Dark Red
14	Dark Yellow
15	Dark Gray
16	Light Gray

• The secret recipe is then \highlightN will highlight text using the Nth color in the color table so long as the Nth color in the color table corresponds to the RGB of one of the colors above.

Jan 3, 2023

I am having all kinds of difficulties with a Makefile on Windows. I have this:

# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe
ifeq ($(OS), Windows_NT)
        R_CMD := rcmd.exe
        R_SCRIPT := rscript.exe
        SHELL = c:\scoop\shims\bash.exe
        SCPEXE := c:\scoop\apps\gow\current\bin\pscp.exe # Requests password :(
        SSHEXE := c:\scoop\apps\gow\current\bin\plink.exe # Requests password :(
endif

and this target:

.PHONY: model
model: /path/to/data.rds
/path/to/data.rds: src/a.R data/b.rds
     $(SCPEXE) src/a.R weigand@server:/path/to/putit
	 $(SSHEXE) weigand@server 'cd /path/to/putit/; R CMD BATCH a.R'

The problems with the above is that the call to SCP requires a password even though I otherwise have password-less login setup. And the call to SSH needs single quotes as far as I can tell and it doesn't understand R CMD BATCH even though my intent is that command is run on the server.

Part of the problem may be the clashing between Scoop's use of pscp.exe and plink.exe versus OpenSSH's scp.exe and ssh.exe in c:/Windows/System32/OpenSSH.

Dec. 28, 2022

I searched for how to type accented characters in Emacs and got https://irreal.org/blog/?p=4945 and https://irreal.org/blog/?p=4254.

One general way to enter accented characters is C-x 8 then a one- or two-character sequence. For example the character "ñ" is entered with C-x 8 ~ n. The character "á" is entered with C-x 8 ' a. There are also special characters like "÷" which is C-x 8 / /. To see what is available, do C-x 8 C-h.

The blog author prefers "switch[ing] into TeX input mode and then using the TeX code for the character." This can be done by typing C-\ to run the command toggle-input-method. It is described as:

Enable or disable multilingual text input method for the current buffer.

If one sets their default input method to TeX (see below) then the character "β" can be entered simply with C-\ \beta. To turn off TeX input mode one enters C-\ again. As far as I can tell, the only way to know if you are in TeX input mode is the buffer mode line starts with \. (I suppose it depends on how the mode line is set up.)

I haven't experimented with this but I imagine in many buffers one can stay in TeX input mode rather then toggling it on or off.

Here is a link to a 22 page PDF of LaTeX symbols: https://www3.nd.edu/~nmark/UsefulFacts/LaTeX_symbols.pdf

As an example, it looks like the section symbol "§" is just \S.

(One can customize default-input-method to TeX with Alt-x customize-variable default-input-method. I am curious about other input modes but for now I'll let it go...)

Oct. 14, 2022

On Windows I miss the Bash equivalent of emacs myfile &. In CMD.exe I can do

start /b c:\scoop\shims\emacs.exe myfile

Oct. 12, 2022

I re-found a video on YouTube (https://youtu.be/D8hEirGijmo) by "tdhock" (Toby Hocking?) that describes how to set the Emacs Speaks Statistics (ESS) variable ess-r-package-auto-enable-namespaced-evaluation. I customized this to nil so that evaluation of a function, paragraph, buffer etc. is not into the package namespace when ESS detects you are in a package. I think this is more straightforward since I'm used to evaluating an R buffer or parts of an R buffer into the global environment during testing or development.

Oct. 6, 2022

SSH file and directory permissions

I was trying to work out file sharing on a Linux machine that a colleague and I both have access to and I changed my home directory permissions there to 777 to see if she could write a file in one of my folders. (The server was protected by the corporate firewall and users were trused so security wasn't a concern.)

A few days later I noticed that passwordless login to the serverwasn't working. I didn't make the connection because it had been a few days and I spent a lot of time trying to understand the problem, check permissions on files in my varoius .ssh directories, generate new keys, etc.

It turns out that the fix was to change permissions on that server to 705 (i.e., read and execute for others).

Apparently, for passwordless SSH to work, the user's home directory can't be writable by world. (One server I use a lot has 755 for my home directory.)

ASCII and Emacs

I had a file on Linux that had some non-ASCII characters. (Windows was involved.)

In R one can do:

library(tools)
showNonASCIIfile("filename.txt")

This gave me a printout of

2: <ef><bb><bf>and this and that...

So R showed me the non-ASCII but I couldn't delete those characters in Emacs.

From a Web search and landing on https://www.gnu.org/software/emacs/manual/html_node/emacs/Coding-Systems.html, I learned that I could call ~revert-buffer-with-coding-system~ and choose ~raw-text~. This displayed

\357\273\277and this and that...

I could then delete these three characters/entities/things and the file passes the showNonASCIIfile test in R.

Sept. 21, 2022

In Emacs on Windows the command Alt-x shell starts a CMD.EXE shell. This is fine/nice. I learned from http://www.hanselman.com/blog/a-better-prompt-for-cmdexe-or-cool-prompt-environment-variables-and-a-nice-transparent-multiprompt that I can customize my prompt. I'm now using

set prompt=$P$_$+cmd$G$S

which gives the drive and path ($P), a newline and carriage return ($_), the pushd/popd depth (+), the text cmd (cmd), then ">" ($G), then a space ($S).

I set the prompt variable by typing "environmental variables" in the search bar and then added a new user variable.

Also, thanks to tips in https://stackoverflow.com/questions/20530996/aliases-in-windows-command-prompt, I have now pseudo aliases in the form of batch files that are saved in a directory that is already on my path. An example is:

@echo off
echo.
cd %USERPROFILE%\Fossils
dir /ad /b /n /w

This takes me to my Fossils directory and lists the directories therein. The first option of /ad is the most useful because it lists directories only. (Running fossils.bat in PowerShell works but the dir command there is an alias for Get-ChildItem and my options are ignored.

June 16, 2022

The generic preprocessor called GPP http://files.nothingisreal.com/software/gpp/gpp.html allows users to define macros in various syntaxes including something similar to the C preprocessor or to LaTeX macros. As I use it, the command is

gpp -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile

or on Windows:

gpp.exe -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile

I could run this within a Makefile on Windows and Linux but not from Windows Terminal. I realized it works from the CMD shell (and GNU Make is using CMD by default) but not when using PowerShell which must require escaping or quoting of the arguments. (The arguments are probably very, very PowerShell unfriendly with double quotes and double backslashes.)

I don't have a work around but at least now I know why there was a problem running it from the (Windows Terminal) command line.

May 18, 2022

This is how I made something of a platform independent Makefile to run either rcmd.exe on Windows or R CMD on Linux.

# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe

ifeq ($(OS), Windows_NT)
        R_CMD := rcmd.exe
        SHELL = c:\scoop\shims\bash.exe
else
        R_CMD := R CMD
endif

May 6, 2022

This is how I used scp to copy a directory from Linux to Windows:

scp -r username@server.com:/file/to/syncdir .
                                            ^ syncdir created locally here

May 5, 2022

How I updated my Pi to Python 3.10.4: https://stackoverflow.com/questions/64718274/how-to-update-python-in-raspberry-pi

sudo apt-get install -y build-essential tk-dev \
  libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev \
  libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev \
  liblzma-dev zlib1g-dev libffi-dev

wget https://www.python.org/ftp/python/3.10.4/Python-3.10.4.tgz
sudo tar zxf Python-3.10.4.tgz
cd Python-3.10.4.tgz
sudo ./configure --enable-optimizations
sudo make -j4
sudo make altinstall

April 12, 2022

I had to work with text files with a non-ASCII character representing a superscript 2. When checking these into Fossil, Fossil warned of invalid UTF-8. I think the problem was that the file had the superscript 2 in LATIN-1. I told Fossil to convert the file and opened the converted file and original file in Emacs and turned on hexl-mode. I forgot to finish this post but I think the difference was between b2 for LATIN-1 and 00b2 for UTF-8.

March 7, 2022

I like the "generic preprocessor" program called gpp (https://github.com/logological/gpp/) for use as a general macro processor. (See https://en.wikipedia.org/wiki/Preprocessor and some links therein.) On Linux gpp is easy to install on with the usual ./configure, make, etc.

Below is how I got it working on Windows without using ./configure or make. (I have Make but I don't think I have sh needed by configure. I'm not sure, but it wasn't hard to compile "by hand".)

These are the preliminary steps:

• Download the file (https://github.com/logological/gpp/releases/download/2.27/gpp-2.27.tar.bz2)
• Type bunzip2.exe ./gpp-2.27.tar.bz2
• Type tar.exe -xvf gpp-2.27.tar and change directories into gpp-2.27/src
• Create a file called stephen.h with the following:

/* Header created by Stephen Weigand to avoid configure on Windows */
#define PACKAGE "gpp"
#define PACKAGE_BUGREPORT "tristan@logological.org"
#define PACKAGE_NAME "GPP"
#define PACKAGE_STRING "GPP 2.27"
#define PACKAGE_TARNAME "gpp"
#define PACKAGE_URL ""
#define PACKAGE_VERSION "2.27"


#define HAVE_STRDUP 1
#define HAVE_STRCASECMP 1
#define HAVE_FNMATCH_H 0

The HAVE_<something> reflects that my GCC can find the C functions strdup and strcasecp but not fnmatch. Now edit gpp.c as follows.

Add the include for stephen.h near the top. Note the quotes around stephen.h which says look in the current directory for the include file.

/* Added by Stephen Weigand March 7, 2022 */
# include "stephen.h"

Now make a few changes based on not having fnmatch. Lines 1880-1902 need to be like this:

    if (SpliceInfix(buf, pos1, pos2, "=~", &spl1, &spl2)) {
#if ! HAVE_FNMATCH_H
        bug("globbing support has not been compiled in");
#else <--- I had to convert this from #endif to #else
        if (!DoArithmEval(buf, pos1, spl1, &result1)
                || !DoArithmEval(buf, spl2, pos2, &result2)) {
            char *str1, *str2;

            /* revert to string comparison */
            while ((pos1 < spl1) && isWhite(buf[spl1 - 1]))
                spl1--;
            while ((pos2 > spl2) && isWhite(buf[spl2]))
                spl2++;
            str1 = strdup(buf + pos1);
            str1[spl1 - pos1] = '\0';
            str2 = strdup(buf + spl2);
            str2[pos2 - spl2] = '\0';
            *result = (fnmatch(str2, str1, 0) == 0);
            free(str1);
            free(str2);
        } else
            *result = (result1 == result2);
        return 1;
#endif <--- This was moved down to here.
    }

With this change (which may be a bug fix, really) I could then do the old-fashioned way of compiling the program.

gcc.exe -Wall -Os -o gpp.exe gpp.c

Then I moved it to somewhere in my path.

February 22, 2022

Commands for setting up a new Fossil

This is my set of commands for starting a Fossil project. I create the repository, set my password, add another user (otheruser) and set their password, and then give them administrator (a) capabilities.

cd /path/to/fossils
fossil new myrepo.fossil
fossil user password myname mypAssW0rd -R myrepo.fossil
fossil user new otheruser otheruser@example.com the1rPasSw0rd -R myrepo.fossil
fossil user capabilities otheruser a -R myrepo.fossil

Then I start the UI and change a few settings:

fossil ui myrepo.fossil

Once the UI opens, click on Admin then Configuration and set the following:

• Project name. I keep this short like a filename since it displays in my bash prompt.

• Project description. This is longer.

• Index page. I use /dir?ci=tip so that when I click on Home in the UI it takes me to the list of files.

I also add a deposit target in a Makefile which will "deposit" and unpack the repository to a location on disk that colleagues who don't use Fossil can look at.

MYPATH := /path/to/shared/directory
.PHONY: deposit
deposit:
        fossil zip trunk source-code-repository.zip --name source-code-repository
        mv source-code-repository.zip $(MYPATH)
        cd $(MYPATH)/; unzip -o source-code-repository.zip
        rm -f $(MYPATH)/source-code-repository.zip
        echo "WARNING: This is a read-only copy of files under Fossil version control for this project." > $(MYPATH)/source-code-repository/WARNING.txt

February 14, 2022

Building R packages

I'm building my R package on my Raspberry Pi and did this

cd ~/Subversions/myproject-branches/
R CMD build mydir-wip-branch

But I needed texi2dvi and so did:

sudo apt install texinfo

But I needed texlive so I did:

sudo apt-install texlive

But I needed inconsolata.sty so I did:

sudo apt-install texlive-fonts-extra

And then I got R CMD build and R CMD check to both be happy and the latter command even got my vignette built.

Subversion branches

I have a checkout in ~/Subversions/myproject and made a branch like this.

First I used the svn copy command

svn copy ^/mypkg/mydir ^/mypkg/mydir-wip-branch -m "Making a work in progress branch of mydir"

Here the ^ means the "root" part of the repository. Then I did

mkdir ~/Subversions/myproject-branches
cd ~/Subversions/myproject-branches
svn checkout svn+ssh://username@example.com/svnroot/myproject/mypkg/mydir-wip-branch

I now have a work-in-progress branch of a subdirectory of the repository. I will need to merge it someday.

January 28, 2022

Background

This is from https://everyday.codes/linux/how-passwordless-ssh-login-works/

• I generate a private key which is a long sequence of bits/bytes/characters.
• A public is another sequence that is derived from this.
• I can get the public from the private but with the public I can't go back to the private.
• My public key is available for everyone and stored on my PC and the server

Using the public key, you can encrypt (or sign) any message, and it will only be possible to decrypt it using the private key. In other words, anyone with your public key can send you encrypted messages that only you will be able to read.

The server can authenticate me like this:

1. Encrypt a message using the public key I've stored on the server and send the message to my PC
2. My PC decrypts the message using my private key and sends it back
3. If the server sees I got it right then I am authenticated.

I'm live tiloring my steps to set up passwordless login for R-forge.

This is a good resource it seeks https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement

I will have a private key and a public key. I place my public key on the R-forge server. When I ssh to r-forge Microsoft says:

When using key authentication with an SSH server, the SSH server and client compare the public key for a user name provided against the private key. If the server-side public key cannot be validated against the client-side private key, authentication fails.

PS> ssh-keygen.exe -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\USERNAME/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\USERNAME/.ssh/id_ed25519.
Your public key has been saved in C:\Users\USERNAME/.ssh/id_ed25519.pub.

And:

Now you have a public/private Ed25519 key pair in the location specified.

    Directory: C:\Users\WEIGAND\.ssh

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        1/28/2022   9:38 PM            464 id_ed25519
-a----        1/28/2022   9:38 PM            104 id_ed25519.pub

Remember that private key files are the equivalent of a password should be protected the same way you protect your password. To help with that, use ssh-agent to securely store the private keys within a Windows security context, associated with your Windows login. To do that, start the ssh-agent service as Administrator and use ssh-add to store the private key.

December 16, 2021

I forgot how to control what applications start up automatically in Windows 10. A while back I set it so PowerShell started up automatically and disabled the same for Skype. But I needed to disable Microsoft Teams from starting up automatically. Here's how I did it:

I typed "Startup" in the search bar. This opened the Settings app and took me to the Startup setting. Then I disabled Teams.

Obviously this was trivial but I lived with Teams starting up automatically for a couple weeks so I added a note about it to cement it in my memory. (I was remembering it as complicated and involving a start-up folder and shortcuts. I guess not.)

December 1, 2021

To fully detach a package in R I do this:

R> detach("package:mypackage", character.only = TRUE, unload = TRUE)

One thing that seems to be a pain is if you "attach" something like ggplot2 (i.e., do any of library(ggplot2), library("ggplot2"), require(ggplot2), etc.) then your session will get a lot of packages loaded "via a namespace" but not "attached"):

> sessionInfo() # after library(ggplot2)
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
<snip>
attached base packages:
[1] stats     graphics  utils     datasets  grDevices methods   base

other attached packages:
[1] ggplot2_3.3.5

loaded via a namespace (and not attached):
[1] fansi_0.4.2      withr_2.4.2      assertthat_0.2.1 dplyr_1.0.7
 [5] crayon_1.4.1     utf8_1.2.2       grid_4.0.3       R6_2.5.1
 [9] DBI_1.1.1        lifecycle_1.0.1  gtable_0.3.0     magrittr_2.0.1
[13] scales_1.1.1     pillar_1.6.3     rlang_0.4.11     generics_0.1.0
[17] vctrs_0.3.8      ellipsis_0.3.2   splines_4.0.3    tools_4.0.3
[21] glue_1.4.2       purrr_0.3.4      munsell_0.5.0    compiler_4.0.3
[25] pkgconfig_2.0.3  colorspace_2.0-2 tidyselect_1.1.1 tibble_3.1.2

There are still 28 packages "loaded via a namespace (and not attached)" after I detach ggplot2. (But I'm not sure what that means exactly.) The way to get rid of these hangers-on is via the unloadNamespace() function.

But they seem to need to be detached based on what's at the top of the dependency tree. Here is an illustration of the problem

> unloadNamespace("fansi") # try to unload first in the list
Error in unloadNamespace("fansi") :
  namespace 'fansi' is imported by 'pillar' so cannot be unloaded

> unloadNamespace("pillar") # OK, I'll unload 'pillar' first.
Error in unloadNamespace("pillar") :
  namespace 'pillar' is imported by 'tibble', 'dplyr' so cannot be unloaded

> unloadNamespace("tibble") # OK, unload 'tibble'.
Error in unloadNamespace("tibble") :
  namespace 'tibble' is imported by 'dplyr' so cannot be unloaded

> unloadNamespace("dplyr") # OK, here's an order that works.
> unloadNamespace("tibble")
> unloadNamespace("pillar")
> unloadNamespace("fansi")

I think there are packages that help you detach and unload packages but I am liking the idea of not attaching ggplot2 in the first place.

November 29, 2021

UTF-8 in C

Here is short C program that works for me on Linux and Windows to print out Hello Aβ42:

#include <stdio.h>

int main(){
  printf("Hello, A\u03b242\n");
  return 0;
}

On Windows I compile it with gcc.exe -Wall -o hello.exe .\hello.c using GCC 8.3.0 and get no errors or warnings. On Linux I am using GCC 4.8.5 and get

hello.c: In function 'main':
hello.c:4:10: warning: universal character names are only valid in C++ and C99 [enabled by default]
   printf("Hello, A\u03b242\n");
          ^

I can use GCC 8.3.1 after typing scl enable devtoolset-8 bash and then the warning goes away. But if I add the GCC flag -std=c89 then the warning comes back (which makes sense).

UTF-8 in Windows Terminal

When using Windows Terminal, I clicked on the down arrow in the tab bar and clicked on Settings which opened up a JSON settings file in Notepad. Then I added two arguments to the PowerShell command to tell the terminal to output in UTF-8.

 // Make changes here to the powershell.exe profile.
 "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
 "name": "Windows PowerShell",
 "commandline": "powershell.exe -Noexit -Command chcp 65001",
 "hidden": false                ^^^^^^^^^^^^^^^^^^^^^^^^^^^

July 22, 2021

From my Windows 10 desktop I can clone a repo on a Linux machine with

fossil set ssh-command "ssh"
fossil clone -v ssh://weigand@server.com/./fossils/filename.fossil filename.fossil

This is good except I should be able to specify the SSH command using -c ssh. Also, I wonder why it doesn't work with plink.exe which allows commands like this

plink.exe -ssh weigand@server.com ls

July 21, 2021

When exporting in Org mode to prevent underscores from being treated as subscripts, I did this:

#+OPTIONS: ^:{}

July 1, 2021

From https://endlessparentheses.com/debugging-emacs-lisp-part-1-earn-your-independence.html I learned several things which may help with my RTF backend for Org mode.

1. The command find-function will take me to the definition of a function. If I am inside a function, I can invoke it and it will ask me what function to go to with the default being the one I'm in. It will then take me to where in the source that function is defined and put point at the start of the function. By default it doesn't look like their are key bindings. I might want to make an alias so I only need to do M-x ff.

2. The command describe-function is bound to C-h f or <f1> f and shows the full documentation of the function. If point is in (or next two?) a function, issuing the command will set the default to the function point is in/near and so just press enter to see the help for the function.

3. I just realized that <f1> is the help function. So naturally <f1> f will give the help for a function, <f1> v the help for a variable, etc.

4. The Edebug part is that I either find or describe the function and then do C-u C-M-x. This is eval-defun with a prefix according to the manual. My only problem is I can't get this to work except for a function that is buggy.

June 15, 2021

In R on Windows and Linux, source("clipboard") will source the contents of the clipboard. On MacOS the equivalent is source(pipe("pbpaste")).

June 13, 2021

From https://devblogs.microsoft.com/scripting/powertip-use-powershell-to-display-windows-path/ I can get my path on PowerShell via

$env:path -split ";"

I turned this into a function:

function path() {
  $env:path -split ";"
}

June 12, 2021

In Emacs I can get an interactive SQLite session via M-x sql-sqlite and then I just have to specify a data base filename.

May 3, 2021

An easy way to get semi-transparent colors in R is to use adjustcolor:

barplot(c(1, 2, 3), col = adjustcolor("firebrick", 0.5),
        main = "Firebrick bar plot with alpha=0.5")

April 26, 2021

Create a simple SQLite data base:

sqlite3 test.sqlite3
sqlite> .databases -- Show the databases

  main: /home/weigand/fossils/tilor/test.sqlite3

ATTACH DATABASE 'test.sqlite3' as 'project';

.databases

main: /home/weigand/fossils/tilor/test.sqlite3
   project: /home/weigand/fossils/tilor/test.sqlite3

DROP TABLE IF EXISTS person;
CREATE TABLE IF NOT EXISTS project.person(
  id INT PRIMARY KEY NOT NULL,
  date_birth CHAR(10),
  sex CHAR(2) CHECK(sex == 'M' OR sex == 'F')
);

INSERT INTO project.person VALUES(1, '2010-01-01', 'M');
INSERT INTO project.person VALUES(2, '2011-11-11', 'F');

SELECT * FROM main.person;

   1|2010-01-01|M
   2|2011-11-11|F

April 14, 2021

When using smbclient on Linux to put a file on a Windows share I wasn't specifying the local file path correctly. I learned about the lcd command in smbclient to set the local current directory and then I could put the filename without a full or relative path. Like this:

PASSWD=`grep -woi -m 1 '^machine mymachine login .* password .*$' ~/.netrc | cut -d ' ' -f 6`
smbclient //share.example.com/DirectoryX \
  -W MYWORKGROUP  \
  -E       \
  -U $USER $PASSWD \
  --command 'cd DirA/DirB ;
             lcd /local/working/directory ;
             put localfile.csv ;
             exit'

March 17, 2021

This is how to get interval estimates for random effects from a linear mixed effects model fit with lme4::lmer in R using the unexported lme4:::asDf0 function.

library(lme4)
library(lattice)
fit <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)

dotplot(ranef(fit)) # Caterpillar plot shows intervals

out <- lme4:::asDf0(ranef(fit), "Subject")
out$lcl <- out$values - 1.96 * out$se
out$ucl <- out$values + 1.96 * out$se

out[c(1:2, 35:36), ]

values         ind .nn   se
1    2.259 (Intercept) 308 12.1
2  -40.399 (Intercept) 309 12.1
35  -0.988        Days 371  2.3
36   1.284        Days 372  2.3

March 10, 2021

On Linux & runs a command in the background. For example:

emacs README.md &

From PowerShell, the equivalent seems to be:

Start-Process -NoNewWindow emacs.exe README.md

The above is from a Web search where I landed on https://ariefbayu.xyz/run-background-command-in-powershell-8ea86436684e. That post shows how to make a general run-it-in-the-background command.

function bg() {
     Start-Process -NoNewWindow @args
}

And then I might use:

bg emacs.exe README.md

March 3, 2021

The Journal NeuroImage uses an open source font called Charis SIL. This can be downloaded and on my department's Linux system, I just have to put the *.ttf files in ~/.fonts and then I can create figures in "native" NeuroImage style:

library(ggplot2)

ggplot(cars, aes(x = speed, y = dist)) +
    geom_point() +
    labs(title = "Scatter plot in Charis SIL font used by NeuroImage",
         x = "Speed (miles per hour)",
         y = "Distance (feet)") +
    theme_light(base_family = "CharisSIL")

December 12, 2020

This is how I used Dropbox's Python API to upload my son's piano recital video. (My daughter's video was smaller and uploaded fine through the Dropbox web interface before timing out.)

My guide was Dropbox for Python Developers.

1. Set-up a Python virtual environment and install dropbox

   • python3.7 -m venv ~/venvs/recital
   • source ~/venvs/recital/bin/activate
   • pip install --upgrade pip
   • pip install dropbox

2. Linked my account in Dropbox to an app

   • Go to the app console
   • Created and named the app "recital". To keep it simple I gave the app read and write permissions across my whole Dropbox account.
   • Generated a short-lived and quite long access token (which lasts for four hours)

3. Wrote Python code to create a dropbox instance, read the file into an object, and upload the object

import dropbox
from dropbox.files import WriteMode


dbx = dropbox.Dropbox('AVerySpecial138CharacterAccessToken')

with open('/home/weigand/recital/piano.m4v', 'rb') as f:
        data = f.read()

dbx.files_upload(data,                         # My file
                 '/recital-folder/piano.m4v',  # Full path of destination
                 mode=WriteMode('overwrite'))  # Want to overwrite

This obviously isn't "professional grade" Python. It works but doesn't handle errors and hard codes the filename.

December 11, 2020

This is how I managed to print to a WiFi-enabled printer from my Raspberry Pi.

I first went to the support page the Brother MFC-J870DW printer and selected Downloads and then did the following:

1. Select OS Family as Linux
2. Select OS Version as Linux (deb)
3. Clicked on the Driver Install Tool
4. Agreed to the End User Licence Agreement (EULA) and clicked Download
5. Saved the file linux-brprinter-installer-2.2.2-1.gz to my ~/Downloads directory

In the shell I did:

cd ~/Downloads
gunzip linux-brprinter-installer-2.2.2-1.gz
sudo bash linux-brprinter-installer-2.2.2-1

  Input model name ->MFC-J870DW

  You are going to install following packages.
     mfcj870dwlpr-3.0.0-1.i386.deb
     mfcj870dwcupswrapper-3.0.0-1.i386.deb
     brscan4-0.4.9-1.i386.deb
     brscan-skey-0.3.1-1.i386.deb
  OK? [y/N] ->
  ... <say yes to a few additional things>
  Will you specify the Device URI? [Y/n] ->y

  select the number of destination Device URI. ->

December 10, 2020

CSVY format

The data.table package in R has a fast way to read CSV files in the form of data.table::fread and as of version 1.12.4 it has support for reading CSVY files. The CSVY format is a text format where there is a YAML header at the top of the file which defines the data "schema" and below the header are the data lines. Here is the general idea of a CSVY file.

schema:
  fields:
  - name: name
    type: string
  - name: age
    type: integer
  - name: date
    type: number
name, age, date
Maria, 44, 2010-01-01
Roberto, 43, 2009-01-01

Tukey HSD

A colleague likes using Tukey's honest significant difference for pairwise comparisons. I never use it but maybe I should. The example for stats::TukeyHSD in R is as follows.

R> summary(fm1 <- aov(breaks ~ wool + tension, data = warpbreaks))
            Df Sum Sq Mean Sq F value Pr(>F)
wool         1    451     451    3.34 0.0736 .
tension      2   2034    1017    7.54 0.0014 **
Residuals   50   6748     135
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

R> TukeyHSD(fm1, "tension", ordered = TRUE) # Still need to understand 'ordered'
  Tukey multiple comparisons of means
    95% family-wise confidence level
    factor levels have been ordered

Fit: aov(formula = breaks ~ wool + tension, data = warpbreaks)

$tension
    diff   lwr upr p adj
M-H  4.7 -4.63  14  0.45
L-H 14.7  5.37  24  0.00
L-M 10.0  0.65  19  0.03

I understand penalization as a better way to handle multiple comparisons but this is quick and easy. It would be interesting to better understand Tukey HSD and perhaps see if I can borrow the automatically generated pairwise comparisons code.

December 4, 2020

Indent a region five spaces in Emacs

C-u 5 C-x TAB is what I wanted.

R's base and grid graphics in one plot

This is one way to combine base and grid graphics in one plot:

 library(ggplot2)
 library(ggplotify)
 library(gridExtra)

 ## Using `::` in a few places to indicate the package
 p1 <- ggplotify::as.grob(~plot(dist ~ speed,  # Notice the unusual `~`
								data = cars,
								main = "Base graphics scatter plot"))

 p2 <- ggplot(cars) +
	 aes(x = speed, y = dist) +
	 geom_point() +
	 ggtitle("ggplot scatter plot")

 gridExtra::grid.arrange(p1, p2, nrow = 1)

Org and TODO states

I had to remind myself how to change the TODO states in an Org file. Here is an example Org file with the information.

 #+TODO: ASAP SOON WAITING SOMEDAY | DONE CANCELED

 * ASAP Learn about Org TODO states
 - It seems easiest to put the TODO state sequence in the file itself
   by having a ~#+TODO:#~ line at the top of the file.
 - See [[https://orgmode.org/org.html#Per_002dfile-keywords][per-file TODO states]]
 - The vertical bar separates not-done versus done states.
 - If you change your TODO states in a file go to the line with the states
   and do ~C-c C-c~ and you'll see "Local setup has been refreshed."
 - As a reminder, the TODO states can be advanced via ~C-c C-t~.

November 28, 2020

I know of two disk space programs on Linux:

1. du says it is to estimate file space usage
2. df says it is to report file system disk space usage but I found it's better to think of it as reporting free space.

I think df is more what I want since it shows

If no file name is given, the space available on all currently mounted file systems is shown

On this Raspberry Pi, df -h gives the following:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        14G  5.4G  7.1G  43% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M   60M  404M  13% /dev/shm
tmpfs           464M  6.3M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p6   71M   23M   49M  32% /boot
tmpfs            93M  4.0K   93M   1% /run/user/1000
/dev/mmcblk0p5   30M  398K   28M   2% /media/pi/SETTINGS

A Web search gives me this nice link from Adafruit:

https://learn.adafruit.com/an-illustrated-shell-command-primer/checking-file-space-usage-du-and-df

This led to free for memory usage. On the Pi, free -h gives (with some whitespace removed):

         total    used    free  shared  buff/cache   available
Mem:      926M    392M     47M     67M        486M        487M
Swap:      99M    5.8M     94M

November 25, 2020

From the Linux Documentation Project (via Stack Exchange):

/etc/skel/ The default files for each new user are stored in this directory. Each time a new user is added, these skeleton files are copied into their home directory. An average system would have: .alias, .bash_profile, .bashrc and .cshrc files. Other files are left up to the system administrator.

On the Raspberry Pi that I am working on righ now, I see .bash_logout, .bashrc, and .profile.

And per a comment in this .profile, Bash reads ~/.profile if there is no ~/.bash_profile and no ~/.bash_login so I don't want to have the latter two files. At least on this Pi the process is:

• Read ~/.profile which has my environmental variables

• ~/.profile sources ~/.bashrc if it exists. The latter has local Bash configurations.

• ~/.bashrc sources ~/.bash_aliases if it exists. Having aliases in a separate file is a "separation of concerns" idea.

November 5, 2020

Here are two LaTeX-like macros for gpp that define two levels of RTF list.

This macro has the bullet flush left with the margin and the text indented as a block by 360/1440 twips or 1/4 inch.

\define{\bulletpoint}{
\paragraph{\fi-360\li360\bullet\tab #1}
}

This second-level bullet has a "white bullet" (specified in unicode decimal format) indented 1/4 inch and the rest of the body indented 1/4 inch more.

\define{\subbulletpoint}{
\paragraph{\fi-360\li720{\u9702-}\tab #1}
}

Both macros depend on \paragraph{} so that the bullet points inherit the paragraph spacing and other formatting. An example paragraph macro is:

\define{\paragraph}{{\pard \ql \sa60 \sb60 \f0 \fs24 \kerning12
#1
\par}}

October 16, 2020

SAS has special missing data codes which represent

a type of numeric missing value that enables you to represent different categories of missing data by using the letters A-Z or an underscore.

R doesn't have this built in. But here is prototype of how to have missing data codes. The data frame has three rows and two variables. Then first variable is an integer ID variable and the second is trails_a which holds the time in seconds a person takes on part A of a test called Trails Making Test. If the person does not have a score on the test, we can record the reason.

I don't think data.frame can generate this object so I use structure with row.names and class attributes. The trails_a variable itself is a list with two elements each of length 3. In a way this is like an embedded data frame.

d <- structure(list(id = c(11L, 22L, 33L),
                    trails_a = structure(list(score = c(50L,
                                                        NA_integer_,
                                                        NA_integer_),
                                              reason = c(NA_character_,
                                                         "Exceeded time limit",
                                                         "Too severe to test")),
                                         class = "trailsList"),
                    random = runif(3)),
               row.names = c(NA, -3L),
               class = "data.frame")

This doesn't print properly (and gives a "corrupt data frame" warning) unless we create a format method for our trailsList class.

format.trailsList <- function(x, ...){
    ifelse(is.na(x$score),
           sprintf("NA (%s)", x$reason),
           sprintf("%ds", x$score))
}

But with a nice format method we get this:

  id                 trails_a random
1 11                      50s  0.061
2 22 NA (Exceeded time limit)  0.664
3 33  NA (Too severe to test)  0.826

To use a data frame with a trailsList object we need to define a number of other methods including operations like == for subsetting. I don't know all that is involved. But an example to look at would be the methods for Surv in the survival package.

Conclusion

I don't think most analysts would want to deal with this complexity. It's more straightforward and understandable to just use two columns in the data frame: the score and if no score, the reason for NA.

Postscript

Having a variable in a data frame that is of POSIXlt class is an "established" case that is very similar. This is a data frame with two observations and three variables clinic, date, and ldate.

d <- data.frame(clinic = 1:2,
                date = as.Date(c("2010-01-01", "2020-01-05")))
d$ldate <- as.POSIXlt(d$date)
dput(d)
structure(list(id = 1:2,
               date = structure(c(14610, 18266), class = "Date"),
               ldate = structure(list(sec = c(0, 0),
                                      min = c(0L, 0L),
                                      hour = c(0L, 0L),
                                      mday = c(1L, 5L),
                                      mon = c(0L, 0L),
                                      year = c(110L, 120L),
                                      wday = c(5L, 0L),
                                      yday = c(0L, 4L),
                                      isdst = c(0L, 0L)),
                                 class = c("POSIXlt", "POSIXt"),
                                 tzone = "UTC"),
               row.names = c(NA, -2L),
               class = "data.frame")

September 21, 2020

PowerShell

I used to look down my nose at PowerShell but now I realize there are lots of shells that do a good job. I need to learn the equivalent of aliases in PowerShell to make navigating the Windows filesystem faster but for now if I navigate to a directory with a file that I want to open (say a Word document) then I can open quickly with

 ii myfile.docx

Here, ii is a shortening of Invoke-Item.

September 15, 2020

Lattice graphics

A StackOverflow question asked about removing tickmarks on the right and top margins of a plot.

https://stackoverflow.com/questions/62945745/removing-the-ticks-on-the-3rd-and-4th-axes-in-plots-from-library-effects-in-r

The context was

library(effects)
m <- lm(Fertility ~ ., data = swiss)
plot(allEffects(m), rug = FALSE)

And plot from the effects package didn't allow the user much control. For example, the usual scales argument wasn't handled. This was my solution:

trellis.par.set(axis.components = list(top = list(tck = 0),
                                       right = list(tck = 0))
plot(allEffects(m), rug = FALSE)

Another thing I learned was to to control the font family easily. Using trellis.par.set outside of a plot cal or using par.settings inside a plot call, one of the elements is grid.pars. See ?gpar.

str(get.gpar())
xyplot(dist ~ speed, data = cars,
       par.settings = list(grid.pars = list(fontfamily = "Open Sans")))

August 27, 2020

Polymode

In a RMarkdown file edited with Emacs and Polymode, the command polymode-eval-buffer-from-beg-to-point is bound to both M-n v <up> and M-n v u.

The command polymode-eval-buffer is bound to M-n v b.

The command polymode-eval-region-or-chunk is bound to M-n v v.

August 26, 2020

Lattice graphics

• Axis tick marks on log scale with "normal" numbers: list(log = 2, equispaced.log = FALSE)
• Gridlines to match tick marks: panel.grid(h = -1, v = -1)

August 20, 2020

The data.table package has fread which is great. One difference between fread and read.csv is how a field like ,"", is handled. Here is an example file:

 "id","date1","date2","code"
 "1","2019-01-01","","3"
 "1","2019-01-01","2018-03-18","3"

Using read.csv with defaults gives:

 R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE))
 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: chr  "2019-01-01" "2019-01-01"
 $ date2: chr  "" "2018-03-18"   <---- Empty string
 $ code : int  3 3

Using na.strings = "" gives

 R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: chr  "2019-01-01" "2019-01-01"
 $ date2: chr  NA "2018-03-18"   <---- Beter
 $ code : int  3 3

But I do not know a way to get the same from fread:

 R> str(fread("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
 Classes 'data.table' and 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: chr  "2019-01-01" "2019-01-01"
 $ date2: chr  "" "2018-03-18"
 $ code : int  3 3
 - attr(*, ".internal.selfref")=<externalptr>

By design, fread wants to be smart about allowing an NA field to be data (e.g., the string "NA". It also wants to allow zero length strings. I don't know if there is a way around this.

For example, this doesn't work:

 R> str(fread("test-fread.csv", colClasses = c("date1" = "Date", "date2" = "Date")))
 Classes 'data.table' and 'data.frame':	2 obs. of  4 variables:
 $ id   : int  1 1
 $ date1: Date, format: "2019-01-01" "2019-01-01"
 $ date2: chr  "" "2018-03-18"
 $ code : int  3 3
 - attr(*, ".internal.selfref")=<externalptr>
 Warning message:
 Column 'date2' was requested to be 'Date' but fread encountered the following error:
 character string is not in a standard unambiguous format
 so the column has been left as type 'character'

This comes up with ADNI data and the PTDEMOG.csv. Maybe the root of the problem is having everything quoted in the source CSV.