Files in the top-level directory in any check-in
- CharisSIL-plot.png
- README.md
November 1
In R, I can use read.csv to read a particular CSV in a Zip archive
sl <- read.csv(unzip("../data/file.zip",
files = "folder_unzips_to/file.csv")
October 27
In Emacs I am trying the cyberpunk theme because it works well with
the rainbow delimeters package but cyberpunk was giving a light
background to Org source blocks. I learned how to customize it by
going to an empty part of the source block and typing describe-face
and seeing that the face was org-block. I changed the background
color to black (#000000) and pressed Save and Apply.
October 25, 2023
In Emacs, I sometimes type C-x C-- by accident which calls
text-scale-adjust to shrink the text size of the window.
And C-x C-+ makes it bigger. After either C-x C-- or
C-x C-+ then - makes text smaller, + makes it bigger,
and 0 returns it to the default.
October 24, 2023
Using Lattice graphics in R, I needed an axis on the square root scale
for a scatter plot matrix generated with lattice::splom. The scales
are controlled by the pscales argument to splom (but really
pscales is an argument to panel.pairs).
If a single number, the pscales argument controls how many tick
marks are desired on the axes. But it can also be a list with one
component for each column being plotted. The list should have these
elements:
‘at’: a numeric vector specifying tick locations
‘labels’: character vector labels to go with at
‘limits’: numeric 2-vector specifying axis limits (should be
made more flexible at some point to handle factors)
These are specifications on a per-variable basis, and used on
all four sides in the diagonal cells used for labelling.
Factor variables are labelled with the factor names. Use
‘pscales=0’ to supress the axes entirely.
One thing I found is that the list's elements should be at,
lab, and lim to avoid partial match warnings since
panel.pairs uses pscales[[i]]$lab and pscales[[i]]$lim.
Here's an example. It's not ideal.
library(lattice)
d <- data.frame(x = exp(rnorm(100, 1)),
y = exp(rnorm(100, 1)))
calc.pscales <- function(x.untransformed) {
list(at = sqrt(pretty(x.untransformed)),
lab = pretty(x.untransformed),
lim = extendrange(sqrt(x.untransformed), f = 0.1))
}
lapply(d, calc.pscales)
splom(sqrt(d), pscales = lapply(d, calc.pscales))
October 16, 2023
I like text-based help in R and so in my ~/.Rprofile I had
options(help_type = "text")
That was working on Linux but not on Windows. It turns out adding this to
~/.Rprofile in Emacs on Windows does not help. To find out where
R is looking for .Rprofile I read the manual and found:
R> Sys.getenv("R_USER")
This is c:\Users\WEIGAND\Documents.
So I needed to edit c:\Users\WEIGAND\Documents\.Rprofile.
October 15, 2023
Some variables in Emacs are buffer variables. For example setting
show-trailing-whitespace to something non-nil means highlight
trailing whitespace in the buffer. But the help for this variable says
Automatically becomes buffer-local when set.
I learned that in order to set this variable in my init.el file,
I need to use
;; (setq show-trailing-whitespace t) <-- Won't do what I want
(setq-default show-trailing-whitespace t)
September 22, 2023
This is how I made a scatter plot in Times New Roman using R's lattice graphics package on Windows:
library(lattice)
library(grid)
windowsFonts(TNR = "Times New Roman")
xyplot(dist ~ speed,
cars,
main = "A scatter plot in Times New Roman",
par.settings = list(grid.pars = gpar(fontfamily = "TNR"),
fontsize = list(text = 24)))
On Linux it was more direct:
library(lattice)
xyplot(dist ~ speed,
cars,
main = "A scatter plot in Times New Roman",
par.settings = list(grid.pars = gpar(fontfamily = "Times New Roman"),
fontsize = list(text = 24)))
September 18, 2023
Here is a strip plot in lattice that shifts points by group to reduce overlap:
stripplot(wool ~ breaks,
warpbreaks,
jitter = TRUE,
groups = tension,
panel = panel.superpose,
panel.groups = function(x, y, group.value, ...) {
dots <- list(...) # Print this to see what else is in `...`
panel.xyplot(x, y + c(-0.1, 0, 0.1)[match(group.value, c("L", "M", "H"))], ...)
})
September 18, 2023
R's lattice package makes it easy to add a regression line by a grouping factor but I did not know how to add confidence intervals to the regression line. This uses the latticeExtra package to "add" a regression line with 95% CI but it does not do it by group:
library(lattice)
library(latticeExtra)
n <- 100
test <- data.frame(x = rnorm(n),
y = rnorm(n),
g = sample(letters[1:3], size = n, replace = TRUE))
xyplot(y ~ x, test, groups = g) + layer_(panel.smoother(x, y, method = "lm"))
The solution I came up with is to create a simple panel function that
plots points and adds a regression line with a 95% CI. It uses
panel.points from lattice and panel.smoother from
latticeExtra. Then I created a simplified version of panel.xyplot
which uses panel.superpose to handle grouping.
## A simple panel function with a scatter plot and a smoother
panel.simple <- function(x, y, ...) {
panel.points(x, y, ...)
panel.smoother(x, y, method = "lm", ...)
}
## A "groups-aware" panel function the uses `panel.superpose`
panel.xyplot.simple <- function(x, y, groups = NULL, ...) {
if (is.null(groups)) stop("We need 'groups' to not be NULL")
panel.superpose(x, y, groups = groups, panel.groups = panel.simple, ...)
} # ^^^^^^^^^^^^ ^^^^^^^^^^^^
## A plot with group-specific regression lines and 95% CIs.
xyplot(y ~ x, test, groups = g, panel = panel.xyplot.simple)
And here is a better way that only requires me to write panel.simple
and which is documented in the manual page for xyplot:
xyplot(y ~ x, test, groups = g, panel = panel.superpose, panel.groups = panel.simple)
August 6, 2023
I learned how to publish a project using Emacs Org mode. The manual explains it well (see Publishing).
This is my alist with one project (www) but there can be multiple
projects.
(setq org-publish-project-alist
'(("www"
:base-directory "~/fossils/www/"
:publishing-function org-html-publish-to-html
:publishing-directory "~/fossils/www"
:section-numbers nil
:with-toc nil
:html-doctype "<!DOCTYPE html>"
:html-head "<link href=\"normalize.css\" rel=\"stylesheet\" type=\"text/css\">
<link href=\"sakura-vader.css\" id=\"sakura-css\" rel=\"stylesheet\" type=\"text/css\">")))
Then, to publish, it's org-publish-project and when prompted say
which project. Putting the parameters in the
org-publish-project-alist reduces boilerplate in the Org file. I was
doing this at the top of files I was planning to export to HTML.
#+TITLE: My Title
#+OPTIONS: toc:nil num:nil html-style:nil html-scripts:nil html-postamble:nil html5-fancy:t
#+HTML_HEAD: <link href="normalize.css" rel="stylesheet" type="text/css">
#+HTML_HEAD: <link href='sakura-vader.css' id="sakura-css" rel='stylesheet' type='text/css'>
#+HTML_DOCTYPE: html5
My list above is not consistent with some of the lines immediately above which I will try to fix. To see what options can be specified in the list, go to Publishing options.
As an example, the option :html-preamble is controlled by the
variable org-html-preamble. It is by default t and non-nil
means insert an HTML preamble. If t it inserts a string
defined by org-html-preamble-format. But if set to a string
it uses the string. And if it is set to a function, apply
the function.
The variable org-html-preamble-format is by default (("en" "")).
The second element of this enclosed list is a format string as follows:
%t stands for the title.
%s stands for the subtitle.
%a stands for the author’s name.
%e stands for the author’s email.
%d stands for the date.
%c will be replaced by ‘org-html-creator-string’.
%v will be replaced by ‘org-html-validation-link’.
%T will be replaced by the export time.
%C will be replaced by the last modification time.
If this is part of the org-publish-project-alist:
:html-preamble "written by %a")))
The exported HTML will have this div:
<head>
...
</head>
<body>
<div id="preamble" class="status">
written by Stephen Weigand
</div>
<div id="content">
<h1 class="title"> ... etc.
August 2, 2023
I'm not sure why using Fossil via ssh started requiring my passphrase. I did the following:
PS> ssh-keygen.exe -t ed25519
PS> cd ~/.ssh
PS> cat id_ed25519.pub | ssh weigand@myserver.com "cat >> ~/.ssh/authorized_keys"
But then on myserver.com I had to edit authorized_keys and take
out the Windows stuff like a carriage return before the newline at the
end of the file. And while I was there I deleted a previous entry from
this Windows machine.
July 11, 2023
Using R's lattice package I learned or remembered one can give a
vector of x-axis, y-axis, and main labels. Here is a 2 by 2 figure
with two x-axis labels and two y-axis labels
library(lattice)
n <- 100
d <- data.frame(x = rnorm(n),
y = rnorm(n),
g = sample(letters[1:4], size = n, replace = TRUE))
xyplot(y ~ x | g,
d,
xlab = c("Flower", "Tree"),
ylab = c("Shrub", "Fruit"))
Another thing us I remembered to use main as a list. This puts
the main title at the left of the viewport. See the main
entry in the help file for xyplot.
xyplot(dist ~ speed,
data = cars,
main = list(label = "A. Scatter plots", x = unit(0, "npc"), just = "left")
July 1, 2023
The Feb. 9, 2023 entry shows how to delete files in Windows even if they don't
exist. In searching for a platform independent way to remove files I remembered
fossil clean. See https://fossil-scm.org/home/help?cmd=clean.
Here's how to remove all *.Rout and *.log files which may be in the current
or any other directory.
fossil clean --clean *.Rout,*.log
It also works with
fossil clean --clean "*.Rout,*.log"
One can also ask Fossil to remove empty directories as part of the operation:
fossil clean --clean "*.Rout,*.log" --emptydirs
Adding the --force option to the above command will prevent Fossil from asking
before removing each empty directory.
May 4, 2023
In R's base graphics, coloring box plots separately by group is trivial:
boxplot(breaks ~ interaction(tension, wool), warpbreaks, col = six.nice.colors)
Using Lattice graphics I learned it can be done by specifying a vector of fill
colors as part of the box.rectangle element of the par.settings list.
p.s <- list(box.umbrella = list(col = "black", lty = 1),
box.rectangle = list(col = "black", fill = six.nice.colors))
bwplot(breaks ~ interaction(tension, wool), warpbreaks, par.settings = p.s, pch = "|")
More general Lattice note
When generating a Lattice plot, creating a plot object doesn't
generate any calls to grid::viewport. The viewpart function is
only called when one plots the object. The reason I am exploring how
Lattice works is I would like to make the plots more "table like" so
that a stripplot of the form
stripplot(level ~ y | group, layout = c(1,2), as.table = TRUE)
looks like this
Group 1
Level 1 * * * *
Level 2 * * *
Level 3 ** * **
Group 2
Level 1 * * * *
Level 2 * * * **
Level 3 * * ** *
|---+---+---+---+---|
To customize the strips, strip.default is (probably) not going to
help since the strips are drawn to fill up the parent
viewport. lattice:::plot.trellis is giving the strip a narrow
viewport spanning the panel which strip.default fills up with a
background, text, and border.
I think I need to examine lattice:::plot.trellis to see where the
viewport for the strip is being created. It might be here:
pushViewport(viewport(layout.pos.row = pos.row - 1,
layout.pos.col = pos.col,
xscale = xscale,
clip = "off",
name = trellis.vpname("strip",
column = column,
row = row,
prefix = prefix,
clip.off = TRUE)))
File this under "Today I Don't Know" because I don't have any good sense of
where to implement my idea of moving strips left. I think I could say, "Hey,
don't generate a default viewport for the strip but a wider version expanding
off to the left." But then which function is doing the calculations to make
it "fit". (I think it is lattice:::plot.trellis which is huge and gnarly.)
A hack may be to say the text in the strips will be adjusted left to the extent of the existing plot. We don't need (much?) more space on the left margin so long as we are just a little to the left of the y-axis labels (which are serving as subgroup names).
That is, we don't want this which takes a lot of new spacing calculations:
Group 1
Level 1 * * * *
but rather an "indented look" like this which can use existing or slightly modified padding:
Group 1
Level 1 * * * *
Maybe the simplest thing is along the lines of this:
myplot <- stripplot(...)
myplot <- add_left_padding_as_needed_based_on_strip_text(myplot)
justify_strip(myplot, just = "left") # Uses grid::grid.edit iterating over panels
Another possible name/metaphor is exdent_strip(myplot). (Or exdentStrip since Lattice
often uses camel case.)
April 28, 2023
I remembered to try using Emacs file-local variables to "compile" R files.
In a plain R file I might have this at the top:
## -*- compile-command: "Rscript foo.R" -*-
And in an Rmd file I might have this:
<!-- -*- compile-command: "Rscript -e 'library(\"knitr\"); knit(\"foo.Rmd\")'" -*- -->
April 24, 2023
Pipx
I previously played with pipx to manage python command line
applications. I couldn't remember if I used Scoop but it doesn't look
like it. Today I'm using Python 3.11 via Scoop and I did:
python -m pip install --user pipx
I then got a warning about
c:\users\weigand\appdata\roaming\python\python311\scripts\pipx.EXE
not being on my path so I fixed that (via typing "path" in the Windows
search bar and clicking on Environmental Variables).
I then followed the pipx suggestion and typed: pipx ensurepath.
Then I did pipx install csvkit and it worked.
in2csv from csvkit
This worked great to convert an Excel XLSX sheet into a CSV.
in2csv.exe --sheet "Sheet XYZ" myfile.xlsx > sheet_xyz.csv
March 29, 2023
Passing arbitrary arguments to Rscript (or Rscript.exe on Windows) is easy.
If a script called test-rscript.R exists and has this one line:
print(commandArgs())
then a command like this:
Rscript --vanilla test-rscript.R --never always -often --
Produces this:
[1] "C:\\scoop\\apps\\r\\current\\bin\\x64\\Rterm.exe"
[2] "--no-echo"
[3] "--no-restore"
[4] "--vanilla"
[5] "--file=test-rscript.R"
[6] "--args"
[7] "--never"
[8] "always"
[9] "-often"
[10] "--"
This shows me that it easy to parse arbitrary command line arguments without having to rely on an external package.
March 28, 2023
I'm trying to write a shell command to export an Org file. I'm using
emacs test.org --batch --funcall org-html-export-to-html --kill
but it doesn't evaluate the code blocks to generate figures using R.
My true aim is RTF export using a local exporter I created called
ox-rtf.el. This creates an RTF document but again, no figures.
emacs test.org --batch --load ox-rtf.el --funcall org-rtf-export-to-rtf --kill
I'm using Batch export of org-mode files from the command line from Stack Overflow. (The answers are nine years old.) There is also this suite of command line tools https://github.com/fniessen/orgmk.
Additional info
The command line above "works" in that it exports the file as it is on
disk. This means that if the code blocks are all evaluated then the
exporting will show the figures. When editing the file and typing C-c
C-e (which calls org-export-dispatch and invokes the menu) the exporting
process includes the additional step of evaluating the code blocks.
(It may be something like it calls org-babel-execute-buffer then
does the exporting.)
The manual has this about
The sample script shows batch processing of multiple files using
org-babel-tangle.
#!/bin/sh
# Tangle files with Org mode
#
emacs -Q --batch --eval "
(progn
(require 'ob-tangle)
(dolist (file command-line-args-left)
(with-current-buffer (find-file-noselect file)
(org-babel-tangle))))
" "$@"
So maybe the solution is something like
emacs -Q --visit test.org --batch --load ox-rtf.el --eval "(org-babel-execute-buffer) (org-rtf-export-to-rtf)" --kill
The problem with the above is Org Babel didn't know what to do with R code blocks. So I tried this and it seems to work.
emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el --eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill
If I am using the CMD shell, I can continue lines with ^
emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el ^
--eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill
One problem is that I am loading my own init.el file and so this
command will only work for me. I am using the Org (version 9.5.5) that
came with my Emacs (version 28.2).
March 28, 2023
I'm trying to customize my ~eshell~ prompt. That is not working right now but this Emacs lisp snippet was helpful to learn:
(substring (shell-command-to-string "fossil status") 0 -1)
It captures the results of the fossil status command and trims the last
character which is a newline.
March 21, 2023
From
https://www.masteringemacs.org/article/executing-shell-commands-emacs
I read about shell-command for one-liners.
March 21, 2023
A couple of times recently I've been typing madly in Emacs and my buffer's font size gets smaller. After a web search I want to remember these keys:
C-x C-+will increase the buffer font size (as willC-x C-=which saves a shift key)C-x C--will decrease the buffer font size
These seem to invoke text-scale-adjust and I also wan to remember
C-x C-0will reset to the default size
March 13, 2023
After updating Emacs to 28.2 using Scoop on Windows, my usual way of starting Emacs via the Start Menu search bar starts Emacs with a CMD window behind it. From this:
https://stackoverflow.com/questions/41300/emacs-in-windows
I think the problem is this is starting the executable program
emacs.exe but I really want to be starting runemacs.exe. I don't
know how to get the old behavior but I found that I could create a
"Desktop Shortcut" as follows:
- I right-clicked on
runemacs.exeand created a shortcut - I moved this shortcut to the Desktop folder
- I right-clicked on the shortcut and selected properties
- In the Shortcut key text box I typed "e" so that my shortcut is "Ctrl + Alt + E". (It's a weird box because it is not editable; you just have to type a single character in there and the short cut is "Ctrl + Alt + X".)
While I was doing this I also changed the "double click action" (my term)
for files I want to open in Emacs to be opened with runemacs.exe.
A follow-up note is that I originally tried having my shortcut in a Shortcut folder within Desktop but that didn't seem to work. I guess the term "desktop shortcut" means a shortcut on the desktop and not anywhere else.
March 7, 2023
Using Scoop on Windows I updated R and Rtools and then was unable to use the brms package. I used this to see what version I had and what versions were installed.
PS> scoop info r
Name : r
Description : A free software environment for statistical computing and graphics.
Version : 4.2.2
Bucket : main
Website : https://www.r-project.org
Updated at : 11/2/2022 1:22:47 AM
Updated by : Hsiao-nan Cheung
Installed : 4.1.0
4.1.1
4.1.3
4.2.2
Binaries : bin\x64\R.exe | bin\x64\Rcmd.exe | bin\x64\Rgui.exe | bin\x64\Rscript.exe | bin\x64\Rterm.exe
Shortcuts : R
...
I used this to "reset" to 4.1.3.
PS> scoop reset r@4.1.3
Resetting r (4.1.3).
Linking C:\scoop\apps\r\current => C:\scoop\apps\r\4.1.3
Creating shim for 'R'.
Creating shim for 'Rcmd'.
Creating shim for 'Rgui'.
Creating shim for 'Rscript'.
Creating shim for 'Rterm'.
Creating shortcut for R (Rgui.exe)
C:\Users\WEIGAND
This was very, very nice.
Feb. 14, 2023
In Emacs I decided to always have line numbers on and Web searches
suggested as of Emacs 26 use (global-display-line-numbers-mode)
in one's initialization file. I then customized the line-number face
and the line-number-current-line face to both have a height of 80%.
Feb. 9, 2023
I learned how to remove files in Windows CMD even if such files aren't present. Here is an example from a Makefile:
.PHONY: clean
clean:
cd code && if exist *.png (del *.png)
Jan. 6, 2023
Through trial and error I now understand colors in RTF as implemented by Microsoft Word (365 on the desktop) a little better.
Highlighting is done via
\highlightN. For example,{\highlight5 This text is highlighted}will be highlighted using the fifth color in the color table.But value for
Nhas to correspond to a row in the color table.
{\colortbl;
\red0\green70\blue173; N=1
\red182\green0\blue20; N=2
\red169\green169\blue169; ...
\red222\green235\blue247;
\red204\green121\blue167;
\red255\green255\blue0; N=6 which here is yellow
\red204\green121\blue167;
\red0\green255\blue0; N=8 which here is green
\red139\green0\blue0; N=9 which is what R gives for col2rgb("darkred")
}
- And you can't specify an arbitrary color to use for highlighting. Based on http://www.biblioscape.com/rtf15_spec.htm, the colors are:
| Value | Description |
| 1 | Black |
| 2 | Blue |
| 3 | Cyan |
| 4 | Green |
| 5 | Magenta |
| 6 | Red |
| 7 | Yellow |
| 8 | Unused |
| 9 | Dark Blue |
| 10 | Dark Cyan |
| 11 | Dark Green |
| 12 | Dark Magenta |
| 13 | Dark Red |
| 14 | Dark Yellow |
| 15 | Dark Gray |
| 16 | Light Gray |
- The secret recipe is then
\highlightNwill highlight text using the Nth color in the color table so long as the Nth color in the color table corresponds to the RGB of one of the colors above.
Jan 3, 2023
I am having all kinds of difficulties with a Makefile on Windows. I have this:
# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe
ifeq ($(OS), Windows_NT)
R_CMD := rcmd.exe
R_SCRIPT := rscript.exe
SHELL = c:\scoop\shims\bash.exe
SCPEXE := c:\scoop\apps\gow\current\bin\pscp.exe # Requests password :(
SSHEXE := c:\scoop\apps\gow\current\bin\plink.exe # Requests password :(
and this target:
.PHONY: model
model: /path/to/data.rds
/path/to/data.rds: src/a.R data/b.rds
$(SCPEXE) src/a.R weigand@server:/path/to/putit
$(SSHEXE) weigand@server 'cd /path/to/putit/; R CMD BATCH a.R'
The problems with the above is that the call to SCP requires a password even though I
otherwise have passwordless login setup. And the call to SSH needs single quotes
as far as I can tell and it doesn't understand R CMD BATCH even though my intent
is that command is run on the server.
Part of the problem may be the clashing between Scoop's use of pscp.exe and
plink.exe versus OpenSSH's scp.exe and ssh.exe in c:/Windows/System32/OpenSSH.
Dec. 28, 2022
I searched for how to type accented characters in Emacs and got https://irreal.org/blog/?p=4945 and https://irreal.org/blog/?p=4254.
One general way to enter accented characters is C-x 8 then a one- or
two-character sequence. For example the character "ñ" is entered with
C-x 8 ~ n. The character "á" is entered with C-x 8 ' a. There are
also special characters like "÷" which is C-x 8 / /. To see what is
available, do C-x 8 C-h.
The blog author prefers "switch[ing] into TeX input mode and then
using the TeX code for the character." This can be done by typing
C-\ to run the command toggle-input-method. It is described as:
Enable or disable multilingual text input method for the current buffer.
If one sets their default input method to TeX (see below) then the
character "β" can be entered simply with C-\ \beta. To turn off TeX
input mode one enters C-\ again. As far as I can tell, the only way
to know if you are in TeX input mode is the buffer mode line starts
with \. (I suppose it desponds on how the mode line is set up.)
I haven't experimented with this but I imagine in many buffers one can stay in TeX input mode rather then toggling it on or off.
Here is a link to a 22 page PDF of LaTeX symbols: https://www3.nd.edu/~nmark/UsefulFacts/LaTeX_symbols.pdf
As an example, it looks like the section symbol "§" is just \S.
(One can customize default-input-method to TeX with Alt-x
customize-variable default-input-method. I am curious about other
input modes but for now I'll let it go...)
Oct. 14, 2022
On Windows I miss the Bash equivalent of emacs myfile &. In CMD.exe I can do
start /b c:\scoop\shims\emacs.exe myfile
Oct. 12, 2022
I re-found a video on YouTube (https://youtu.be/D8hEirGijmo) by
"tdhock" (Toby Hocking?) that describes how to set the Emacs Speaks
Statistics (ESS) variable
ess-r-package-auto-enable-namespaced-evaluation. I customized this
to nil so that evaluation of a function, paragraph, buffer etc. is
not into the package namespace when ESS detects you are in a
package. I think this is more straightforward since I'm used to
evaluating an R buffer or parts of an R buffer into the global
environment during testing or development.
Oct. 6, 2022
SSH file and directory permissions
I was trying to work out file sharing on a Linux machine that a colleague and I both have access to and I changed my home directory permissions there to 777 to see if she could write a file in one of my folders. (The server was protected by the corporate firewall and users were trused so security wasn't a concern.)
A few days later I noticed that passwordless login to the serverwasn't
working. I didn't make the connection because it had been a few days
and I spent a lot of time trying to understand the problem, check
permissions on files in my varoius .ssh directories, generate new
keys, etc.
It turns out that the fix was to change permissions on that server to 705 (i.e., read and execute for others).
Apparently, for passwordless SSH to work, the user's home directory can't be writable by world. (One server I use a lot has 755 for my home directory.)
ASCII and Emacs
I had a file on Linux that had some non-ASCII characters. (Windows was involved.)
In R one can do:
library(tools)
showNonASCIIfile("filename.txt")
This gave me a printout of
2: <ef><bb><bf>and this and that...
So R showed me the non-ASCII but I couldn't delete those characters in Emacs.
From a Web search and landing on https://www.gnu.org/software/emacs/manual/html_node/emacs/Coding-Systems.html, I learned that I could call ~revert-buffer-with-coding-system~ and choose ~raw-text~. This displayed
\357\273\277and this and that...
I could then delete these three characters/entities/things and the file
passes the showNonASCIIfile test in R.
Sept. 21, 2022
In Emacs on Windows the command Alt-x shell starts a CMD.EXE
shell. This is fine/nice. I learned from
http://www.hanselman.com/blog/a-better-prompt-for-cmdexe-or-cool-prompt-environment-variables-and-a-nice-transparent-multiprompt
that I can customize my prompt. I'm now using
set prompt=$P$_$+cmd$G$S
which gives the drive and path ($P), a newline and carriage return
($_), the pushd/popd depth (+), the text cmd (cmd), then ">"
($G), then a space ($S).
I set the prompt variable by typing "environmental variables" in the search bar and then added a new user variable.
Also, thanks to tips in https://stackoverflow.com/questions/20530996/aliases-in-windows-command-prompt, I have now pseudo aliases in the form of batch files that are saved in a directory that is already on my path. An example is:
@echo off
echo.
cd %USERPROFILE%\Fossils
dir /ad /b /n /w
This takes me to my Fossils directory and lists the directories
therein. The first option of /ad is the most useful because it lists
directories only. (Running fossils.bat in PowerShell works but the
dir command there is an alias for Get-ChildItem and my options are
ignored.
June 16, 2022
The generic preprocessor called GPP http://files.nothingisreal.com/software/gpp/gpp.html allows users to define macros in various syntaxes including something similar to the C preprocessor or to LaTeX macros. As I use it, the command is
gpp -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile
or on Windows:
gpp.exe -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile
I could run this within a Makefile on Windows and Linux but not from Windows Terminal. I realized it works from the CMD shell (and GNU Make is using CMD by default) but not when using PowerShell which must require escaping or quoting of the arguments. (The arguments are probably very, very PowerShell unfriendly with double quotes and double backslashes.)
I don't have a work around but at least now I know why there was a problem running it from the (Windows Terminal) command line.
May 18, 2022
This is how I made something of a platform independent Makefile
to run either rcmd.exe on Windows or R CMD on Linux.
# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe
ifeq ($(OS), Windows_NT)
R_CMD := rcmd.exe
SHELL = c:\scoop\shims\bash.exe
else
R_CMD := R CMD
endif
May 6, 2022
This is how I used scp to copy a directory from Linux to Windows:
scp -r username@server.com:/file/to/syncdir .
^ syncdir created locally here
May 5, 2022
How I updated my Pi to Python 3.10.4: https://stackoverflow.com/questions/64718274/how-to-update-python-in-raspberry-pi
sudo apt-get install -y build-essential tk-dev \
libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev \
libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev \
liblzma-dev zlib1g-dev libffi-dev
wget https://www.python.org/ftp/python/3.10.4/Python-3.10.4.tgz
sudo tar zxf Python-3.10.4.tgz
cd Python-3.10.4.tgz
sudo ./configure --enable-optimizations
sudo make -j4
sudo make altinstall
April 12, 2022
I had to work with text files with a non-ASCII character representing
a superscript 2. When checking these into Fossil,
Fossil warned of invalid UTF-8. I think the problem was that the file
had the superscript 2 in LATIN-1. I told Fossil to convert the file and
opened the converted file and original file in Emacs and turned on
hexl-mode. I forgot to finish this post but I think the difference was between
b2 for LATIN-1 and 00b2 for UTF-8.
March 7, 2022
I like the "generic preprocessor" program called gpp
(https://github.com/logological/gpp/) for use as a general macro
processor. (See https://en.wikipedia.org/wiki/Preprocessor and some
links therein.) On Linux gpp is easy to install on with the usual
./configure, make, etc.
Below is how I got it working on Windows without using ./configure or
make. (I have Make but I don't think I have sh needed by configure. I'm
not sure, but it wasn't hard to compile "by hand".)
These are the preliminary steps:
- Download the file (https://github.com/logological/gpp/releases/download/2.27/gpp-2.27.tar.bz2)
- Type
bunzip2.exe ./gpp-2.27.tar.bz2 - Type
tar.exe -xvf gpp-2.27.tarand change directories intogpp-2.27/src - Create a file called
stephen.hwith the following:
/* Header created by Stephen Weigand to avoid configure on Windows */
#define PACKAGE "gpp"
#define PACKAGE_BUGREPORT "tristan@logological.org"
#define PACKAGE_NAME "GPP"
#define PACKAGE_STRING "GPP 2.27"
#define PACKAGE_TARNAME "gpp"
#define PACKAGE_URL ""
#define PACKAGE_VERSION "2.27"
#define HAVE_STRDUP 1
#define HAVE_STRCASECMP 1
#define HAVE_FNMATCH_H 0
The HAVE_<something> reflects that my GCC can find the C functions
strdup and strcasecp but not fnmatch. Now edit gpp.c as
follows.
Add the include for stephen.h near the top. Note the quotes around
stephen.h which says look in the current directory for the include
file.
/* Added by Stephen Weigand March 7, 2022 */
# include "stephen.h"
Now make a few changes based on not having fnmatch. Lines 1880-1902
need to be like this:
if (SpliceInfix(buf, pos1, pos2, "=~", &spl1, &spl2)) {
#if ! HAVE_FNMATCH_H
bug("globbing support has not been compiled in");
#else <--- I had to convert this from #endif to #else
if (!DoArithmEval(buf, pos1, spl1, &result1)
|| !DoArithmEval(buf, spl2, pos2, &result2)) {
char *str1, *str2;
/* revert to string comparison */
while ((pos1 < spl1) && isWhite(buf[spl1 - 1]))
spl1--;
while ((pos2 > spl2) && isWhite(buf[spl2]))
spl2++;
str1 = strdup(buf + pos1);
str1[spl1 - pos1] = '\0';
str2 = strdup(buf + spl2);
str2[pos2 - spl2] = '\0';
*result = (fnmatch(str2, str1, 0) == 0);
free(str1);
free(str2);
} else
*result = (result1 == result2);
return 1;
#endif <--- This was moved down to here.
}
With this change (which may be a bug fix, really) I could then do the old-fashioned way of compiling the program.
gcc.exe -Wall -Os -o gpp.exe gpp.c
Then I moved it to somewhere in my path.
February 22, 2022
Commands for setting up a new Fossil
This is my set of commands for starting a Fossil project. I create the
repository, set my password, add another user (otheruser) and set
their password, and then give them administrator (a) capabilities.
cd /path/to/fossils
fossil new myrepo.fossil
fossil user password myname mypAssW0rd -R myrepo.fossil
fossil user new otheruser otheruser@example.com the1rPasSw0rd -R myrepo.fossil
fossil user capabilities otheruser a -R myrepo.fossil
Then I start the UI and change a few settings:
fossil ui myrepo.fossil
Once the UI opens, click on Admin then Configuration and set the following:
Project name. I keep this short like a filename since it displays in my bash prompt.
Project description. This is longer.
Index page. I use
/dir?ci=tipso that when I click on Home in the UI it takes me to the list of files.
I also add a deposit target in a Makefile which will "deposit" and unpack
the repository to a location on disk that colleagues who don't use Fossil
can look at.
MYPATH := /path/to/shared/directory
.PHONY: deposit
deposit:
fossil zip trunk source-code-repository.zip --name source-code-repository
mv source-code-repository.zip $(MYPATH)
cd $(MYPATH)/; unzip -o source-code-repository.zip
rm -f $(MYPATH)/source-code-repository.zip
echo "WARNING: This is a read-only copy of files under Fossil version control for this project." > $(MYPATH)/source-code-repository/WARNING.txt
February 14, 2022
Building R packages
I'm building my R package on my Raspberry Pi and did this
cd ~/Subversions/myproject-branches/
R CMD build mydir-wip-branch
But I needed texi2dvi and so did:
sudo apt install texinfo
But I needed texlive so I did:
sudo apt-install texlive
But I needed inconsolata.sty so I did:
sudo apt-install texlive-fonts-extra
And then I got R CMD build and R CMD check to both be happy and
the latter command even got my vignette built.
Subversion branches
I have a checkout in ~/Subversions/myproject and made a branch like this.
First I used the svn copy command
svn copy ^/mypkg/mydir ^/mypkg/mydir-wip-branch -m "Making a work in progress branch of mydir"
Here the ^ means the "root" part of the repository. Then I did
mkdir ~/Subversions/myproject-branches
cd ~/Subversions/myproject-branches
svn checkout svn+ssh://username@example.com/svnroot/myproject/mypkg/mydir-wip-branch
I now have a work-in-progress branch of a subdirectory of the repository. I will need to merge it someday.
January 28, 2022
Background
This is from https://everyday.codes/linux/how-passwordless-ssh-login-works/
- I generate a private key which is a long sequence of bits/bytes/characters.
- A public is another sequence that is derived from this.
- I can get the public from the private but with the public I can't go back to the private.
- My public key is available for everyone and stored on my PC and the server
Using the public key, you can encrypt (or sign) any message, and it will only be possible to decrypt it using the private key. In other words, anyone with your public key can send you encrypted messages that only you will be able to read.
The server can authenticate me like this:
- Encrypt a message using the public key I've stored on the server and send the message to my PC
- My PC decrypts the message using my private key and sends it back
- If the server sees I got it right then I am authenticated.
I'm live tiloring my steps to set up passwordless login for R-forge.
This is a good resource it seeks https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement
I will have a private key and a public key. I place my public key on the R-forge server. When I ssh to r-forge Microsoft says:
When using key authentication with an SSH server, the SSH server and client compare the public key for a user name provided against the private key. If the server-side public key cannot be validated against the client-side private key, authentication fails.
PS> ssh-keygen.exe -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\USERNAME/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\USERNAME/.ssh/id_ed25519.
Your public key has been saved in C:\Users\USERNAME/.ssh/id_ed25519.pub.
And:
Now you have a public/private Ed25519 key pair in the location specified.
Directory: C:\Users\WEIGAND\.ssh
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 1/28/2022 9:38 PM 464 id_ed25519
-a---- 1/28/2022 9:38 PM 104 id_ed25519.pub
Remember that private key files are the equivalent of a password should be protected the same way you protect your password. To help with that, use ssh-agent to securely store the private keys within a Windows security context, associated with your Windows login. To do that, start the ssh-agent service as Administrator and use ssh-add to store the private key.
December 16, 2021
I forgot how to control what applications start up automatically in Windows 10. A while back I set it so PowerShell started up automatically and disabled the same for Skype. But I needed to disable Microsoft Teams from starting up automatically. Here's how I did it:
I typed "Startup" in the search bar. This opened the Settings app and took me to the Startup setting. Then I disabled Teams.
Obviously this was trivial but I lived with Teams starting up automatically for a couple weeks so I added a note about it to cement it in my memory. (I was remembering it as complicated and involving a start-up folder and shortcuts. I guess not.)
December 1, 2021
To fully detach a package in R I do this:
R> detach("package:mypackage", character.only = TRUE, unload = TRUE)
One thing that seems to be a pain is if you "attach" something like
ggplot2 (i.e., do any of library(ggplot2), library("ggplot2"),
require(ggplot2), etc.) then your session will get a lot of packages
loaded "via a namespace" but not "attached"):
> sessionInfo() # after library(ggplot2)
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
<snip>
attached base packages:
[1] stats graphics utils datasets grDevices methods base
other attached packages:
[1] ggplot2_3.3.5
loaded via a namespace (and not attached):
[1] fansi_0.4.2 withr_2.4.2 assertthat_0.2.1 dplyr_1.0.7
[5] crayon_1.4.1 utf8_1.2.2 grid_4.0.3 R6_2.5.1
[9] DBI_1.1.1 lifecycle_1.0.1 gtable_0.3.0 magrittr_2.0.1
[13] scales_1.1.1 pillar_1.6.3 rlang_0.4.11 generics_0.1.0
[17] vctrs_0.3.8 ellipsis_0.3.2 splines_4.0.3 tools_4.0.3
[21] glue_1.4.2 purrr_0.3.4 munsell_0.5.0 compiler_4.0.3
[25] pkgconfig_2.0.3 colorspace_2.0-2 tidyselect_1.1.1 tibble_3.1.2
There are still 28 packages "loaded via a namespace (and not
attached)" after I detach ggplot2. (But I'm not sure what that means
exactly.) The way to get rid of these hangers-on is via the
unloadNamespace() function.
But they seem to need to be detached based on what's at the top of the dependency tree. Here is an illustration of the problem
> unloadNamespace("fansi") # try to unload first in the list
Error in unloadNamespace("fansi") :
namespace 'fansi' is imported by 'pillar' so cannot be unloaded
> unloadNamespace("pillar") # OK, I'll unload 'pillar' first.
Error in unloadNamespace("pillar") :
namespace 'pillar' is imported by 'tibble', 'dplyr' so cannot be unloaded
> unloadNamespace("tibble") # OK, unload 'tibble'.
Error in unloadNamespace("tibble") :
namespace 'tibble' is imported by 'dplyr' so cannot be unloaded
> unloadNamespace("dplyr") # OK, here's an order that works.
> unloadNamespace("tibble")
> unloadNamespace("pillar")
> unloadNamespace("fansi")
I think there are packages that help you detach and unload packages but I am liking the idea of not attaching ggplot2 in the first place.
November 29, 2021
UTF-8 in C
Here is short C program that works for me on Linux and Windows to print out
Hello Aβ42:
#include <stdio.h>
int main(){
printf("Hello, A\u03b242\n");
return 0;
}
On Windows I compile it with gcc.exe -Wall -o hello.exe .\hello.c
using GCC 8.3.0 and get no errors or warnings. On Linux I am using
GCC 4.8.5 and get
hello.c: In function 'main':
hello.c:4:10: warning: universal character names are only valid in C++ and C99 [enabled by default]
printf("Hello, A\u03b242\n");
^
I can use GCC 8.3.1 after typing scl enable devtoolset-8 bash and then
the warning goes away. But if I add the GCC flag -std=c89 then the warning
comes back (which makes sense).
UTF-8 in Windows Terminal
When using Windows Terminal, I clicked on the down arrow in the tab bar and clicked on Settings which opened up a JSON settings file in Notepad. Then I added two arguments to the PowerShell command to tell the terminal to output in UTF-8.
// Make changes here to the powershell.exe profile.
"guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
"name": "Windows PowerShell",
"commandline": "powershell.exe -Noexit -Command chcp 65001",
"hidden": false ^^^^^^^^^^^^^^^^^^^^^^^^^^^
July 22, 2021
From my Windows 10 desktop I can clone a repo on a Linux machine with
fossil set ssh-command "ssh"
fossil clone -v ssh://weigand@server.com/./fossils/filename.fossil filename.fossil
This is good except I should be able to specify the SSH command using -c ssh. Also, I wonder
why it doesn't work with plink.exe which allows commands like this
plink.exe -ssh weigand@server.com ls
July 21, 2021
When exporting in Org mode to prevent underscores from being treated as subscripts, I did this:
#+OPTIONS: ^:{}
July 1, 2021
From https://endlessparentheses.com/debugging-emacs-lisp-part-1-earn-your-independence.html I learned several things which may help with my RTF backend for Org mode.
The command
find-functionwill take me to the definition of a function. If I am inside a function, I can invoke it and it will ask me what function to go to with the default being the one I'm in. It will then take me to where in the source that function is defined and put point at the start of the function. By default it doesn't look like their are key bindings. I might want to make an alias so I only need to doM-x ff.The command
describe-functionis bound toC-h for<f1> fand shows the full documentation of the function. If point is in (or next two?) a function, issuing the command will set the default to the function point is in/near and so just press enter to see the help for the function.I just realized that
<f1>is the help function. So naturally<f1> fwill give the help for a function,<f1> vthe help for a variable, etc.The Edebug part is that I either find or describe the function and then do
C-u C-M-x. This iseval-defunwith a prefix according to the manual. My only problem is I can't get this to work except for a function that is buggy.
June 15, 2021
In R on Windows and Linux, source("clipboard") will source the
contents of the clipboard. On MacOS the equivalent is
source(pipe("pbpaste")).
June 13, 2021
From https://devblogs.microsoft.com/scripting/powertip-use-powershell-to-display-windows-path/ I can get my path on PowerShell via
$env:path -split ";"
I turned this into a function:
function path() {
$env:path -split ";"
}
June 12, 2021
In Emacs I can get an interactive SQLite session via M-x sql-sqlite and
then I just have to specify a data base filename.
May 3, 2021
An easy way to get semi-transparent colors in R is to use adjustcolor:
barplot(c(1, 2, 3), col = adjustcolor("firebrick", 0.5),
main = "Firebrick bar plot with alpha=0.5")
April 26, 2021
Create a simple SQLite data base:
sqlite3 test.sqlite3
sqlite> .databases -- Show the databases
main: /home/weigand/fossils/tilor/test.sqlite3
ATTACH DATABASE 'test.sqlite3' as 'project';
.databases
main: /home/weigand/fossils/tilor/test.sqlite3
project: /home/weigand/fossils/tilor/test.sqlite3
DROP TABLE IF EXISTS person;
CREATE TABLE IF NOT EXISTS project.person(
id INT PRIMARY KEY NOT NULL,
date_birth CHAR(10),
sex CHAR(2) CHECK(sex == 'M' OR sex == 'F')
);
INSERT INTO project.person VALUES(1, '2010-01-01', 'M');
INSERT INTO project.person VALUES(2, '2011-11-11', 'F');
SELECT * FROM main.person;
1|2010-01-01|M
2|2011-11-11|F
April 14, 2021
When using smbclient on Linux to put a file on a Windows share I wasn't
specifying the local file path correctly. I learned about the lcd command
in smbclient to set the local current directory and then I could put
the filename without a full or relative path. Like this:
PASSWD=`grep -woi -m 1 '^machine mymachine login .* password .*$' ~/.netrc | cut -d ' ' -f 6`
smbclient //share.example.com/DirectoryX \
-W MYWORKGROUP \
-E \
-U $USER $PASSWD \
--command 'cd DirA/DirB ;
lcd /local/working/directory ;
put localfile.csv ;
exit'
March 17, 2021
This is how to get interval estimates for random effects from a linear
mixed effects model fit with lme4::lmer in R using the unexported
lme4:::asDf0 function.
library(lme4)
library(lattice)
fit <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)
dotplot(ranef(fit)) # Caterpillar plot shows intervals
out <- lme4:::asDf0(ranef(fit), "Subject")
out$lcl <- out$values - 1.96 * out$se
out$ucl <- out$values + 1.96 * out$se
out[c(1:2, 35:36), ]
values ind .nn se
1 2.259 (Intercept) 308 12.1
2 -40.399 (Intercept) 309 12.1
35 -0.988 Days 371 2.3
36 1.284 Days 372 2.3
March 10, 2021
On Linux & runs a command in the background. For example:
emacs README.md &
From PowerShell, the equivalent seems to be:
Start-Process -NoNewWindow emacs.exe README.md
The above is from a Web search where I landed on https://ariefbayu.xyz/run-background-command-in-powershell-8ea86436684e. That post shows how to make a general run-it-in-the-background command.
function bg() {
Start-Process -NoNewWindow @args
}
And then I might use:
bg emacs.exe README.md
March 3, 2021
The Journal NeuroImage uses an open source font called Charis
SIL. This can be downloaded and on
my department's Linux system, I just have to put the *.ttf files in
~/.fonts and then I can create figures in "native" NeuroImage style:
library(ggplot2)
ggplot(cars, aes(x = speed, y = dist)) +
geom_point() +
labs(title = "Scatter plot in Charis SIL font used by NeuroImage",
x = "Speed (miles per hour)",
y = "Distance (feet)") +
theme_light(base_family = "CharisSIL")
December 12, 2020
This is how I used Dropbox's Python API to upload my son's piano recital video. (My daughter's video was smaller and uploaded fine through the Dropbox web interface before timing out.)
My guide was Dropbox for Python Developers.
Set-up a Python virtual environment and install
dropboxpython3.7 -m venv ~/venvs/recitalsource ~/venvs/recital/bin/activatepip install --upgrade pippip install dropbox
Linked my account in Dropbox to an app
- Go to the app console
- Created and named the app "recital". To keep it simple I gave the app read and write permissions across my whole Dropbox account.
- Generated a short-lived and quite long access token (which lasts for four hours)
Wrote Python code to create a dropbox instance, read the file into an object, and upload the object
import dropbox
from dropbox.files import WriteMode
dbx = dropbox.Dropbox('AVerySpecial138CharacterAccessToken')
with open('/home/weigand/recital/piano.m4v', 'rb') as f:
data = f.read()
dbx.files_upload(data, # My file
'/recital-folder/piano.m4v', # Full path of destination
mode=WriteMode('overwrite')) # Want to overwrite
This obviously isn't "professional grade" Python. It works but doesn't handle errors and hard codes the filename.
December 11, 2020
This is how I managed to print to a WiFi-enabled printer from my Raspberry Pi.
I first went to the support page the Brother MFC-J870DW printer and selected Downloads and then did the following:
- Select OS Family as Linux
- Select OS Version as Linux (deb)
- Clicked on the Driver Install Tool
- Agreed to the End User Licence Agreement (EULA) and clicked Download
- Saved the file
linux-brprinter-installer-2.2.2-1.gzto my~/Downloadsdirectory
In the shell I did:
cd ~/Downloads
gunzip linux-brprinter-installer-2.2.2-1.gz
sudo bash linux-brprinter-installer-2.2.2-1
Input model name ->MFC-J870DW
You are going to install following packages.
mfcj870dwlpr-3.0.0-1.i386.deb
mfcj870dwcupswrapper-3.0.0-1.i386.deb
brscan4-0.4.9-1.i386.deb
brscan-skey-0.3.1-1.i386.deb
OK? [y/N] ->
... <say yes to a few additional things>
Will you specify the Device URI? [Y/n] ->y
select the number of destination Device URI. ->
December 10, 2020
CSVY format
The data.table package in R has a fast way to read CSV files in the
form of data.table::fread and as of version 1.12.4 it has support
for reading CSVY files. The CSVY format is a text format where there
is a YAML header at the top of the file which defines the data
"schema" and below the header are the data lines. Here is the general
idea of a CSVY file.
schema:
fields:
- name: name
type: string
- name: age
type: integer
- name: date
type: number
name, age, date
Maria, 44, 2010-01-01
Roberto, 43, 2009-01-01
Tukey HSD
A colleague likes using Tukey's honest significant difference for pairwise
comparisons. I never use it but maybe I should. The example for stats::TukeyHSD
in R is as follows.
R> summary(fm1 <- aov(breaks ~ wool + tension, data = warpbreaks))
Df Sum Sq Mean Sq F value Pr(>F)
wool 1 451 451 3.34 0.0736 .
tension 2 2034 1017 7.54 0.0014 **
Residuals 50 6748 135
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R> TukeyHSD(fm1, "tension", ordered = TRUE) # Still need to understand 'ordered'
Tukey multiple comparisons of means
95% family-wise confidence level
factor levels have been ordered
Fit: aov(formula = breaks ~ wool + tension, data = warpbreaks)
$tension
diff lwr upr p adj
M-H 4.7 -4.63 14 0.45
L-H 14.7 5.37 24 0.00
L-M 10.0 0.65 19 0.03
I understand penalization as a better way to handle multiple comparisons but this is quick and easy. It would be interesting to better understand Tukey HSD and perhaps see if I can borrow the automatically generated pairwise comparisons code.
December 4, 2020
Indent a region five spaces in Emacs
C-u 5 C-x TAB is what I wanted.
R's base and grid graphics in one plot
This is one way to combine base and grid graphics in one plot:
library(ggplot2)
library(ggplotify)
library(gridExtra)
## Using `::` in a few places to indicate the package
p1 <- ggplotify::as.grob(~plot(dist ~ speed, # Notice the unusual `~`
data = cars,
main = "Base graphics scatter plot"))
p2 <- ggplot(cars) +
aes(x = speed, y = dist) +
geom_point() +
ggtitle("ggplot scatter plot")
gridExtra::grid.arrange(p1, p2, nrow = 1)
Org and TODO states
I had to remind myself how to change the TODO states in an Org file. Here is an example Org file with the information.
#+TODO: ASAP SOON WAITING SOMEDAY | DONE CANCELED
* ASAP Learn about Org TODO states
- It seems easiest to put the TODO state sequence in the file itself
by having a ~#+TODO:#~ line at the top of the file.
- See [[https://orgmode.org/org.html#Per_002dfile-keywords][per-file TODO states]]
- The vertical bar separates not-done versus done states.
- If you change your TODO states in a file go to the line with the states
and do ~C-c C-c~ and you'll see "Local setup has been refreshed."
- As a reminder, the TODO states can be advanced via ~C-c C-t~.
November 28, 2020
I know of two disk space programs on Linux:
dusays it is to estimate file space usagedfsays it is to report file system disk space usage but I found it's better to think of it as reporting free space.
I think df is more what I want since it shows
If no file name is given, the space available on all currently mounted file systems is shown
On this Raspberry Pi, df -h gives the following:
Filesystem Size Used Avail Use% Mounted on
/dev/root 14G 5.4G 7.1G 43% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 464M 60M 404M 13% /dev/shm
tmpfs 464M 6.3M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
/dev/mmcblk0p6 71M 23M 49M 32% /boot
tmpfs 93M 4.0K 93M 1% /run/user/1000
/dev/mmcblk0p5 30M 398K 28M 2% /media/pi/SETTINGS
A Web search gives me this nice link from Adafruit:
https://learn.adafruit.com/an-illustrated-shell-command-primer/checking-file-space-usage-du-and-df
This led to free for memory usage. On the Pi, free -h gives (with
some whitespace removed):
total used free shared buff/cache available
Mem: 926M 392M 47M 67M 486M 487M
Swap: 99M 5.8M 94M
November 25, 2020
From the Linux Documentation Project (via Stack Exchange):
/etc/skel/ The default files for each new user are stored in this directory. Each time a new user is added, these skeleton files are copied into their home directory. An average system would have: .alias, .bash_profile, .bashrc and .cshrc files. Other files are left up to the system administrator.
On the Raspberry Pi that I am working on righ now, I see
.bash_logout, .bashrc, and .profile.
And per a comment in this .profile, Bash reads ~/.profile if there
is no ~/.bash_profile and no ~/.bash_login so I don't want to have
the latter two files. At least on this Pi the process is:
Read
~/.profilewhich has my environmental variables~/.profilesources~/.bashrcif it exists. The latter has local Bash configurations.~/.bashrcsources~/.bash_aliasesif it exists. Having aliases in a separate file is a "separation of concerns" idea.
November 5, 2020
Here are two LaTeX-like macros for gpp that define two levels of
RTF list.
This macro has the bullet flush left with the margin and the text indented as a block by 360/1440 twips or 1/4 inch.
\define{\bulletpoint}{
\paragraph{\fi-360\li360\bullet\tab #1}
}
This second-level bullet has a "white bullet" (specified in unicode decimal format) indented 1/4 inch and the rest of the body indented 1/4 inch more.
\define{\subbulletpoint}{
\paragraph{\fi-360\li720{\u9702-}\tab #1}
}
Both macros depend on \paragraph{} so that the bullet points
inherit the paragraph spacing and other formatting. An example
paragraph macro is:
\define{\paragraph}{{\pard \ql \sa60 \sb60 \f0 \fs24 \kerning12
#1
\par}}
October 16, 2020
SAS has special missing data codes which represent
a type of numeric missing value that enables you to represent different categories of missing data by using the letters A-Z or an underscore.
R doesn't have this built in. But here is prototype of how to have
missing data codes. The data frame has three rows and two
variables. Then first variable is an integer ID variable and the
second is trails_a which holds the time in seconds a person takes on
part A of a test called Trails Making Test. If the person does not
have a score on the test, we can record the reason.
I don't think data.frame can generate this object so I use
structure with row.names and class attributes. The trails_a variable
itself is a list with two elements each of length 3. In a way this is
like an embedded data frame.
d <- structure(list(id = c(11L, 22L, 33L),
trails_a = structure(list(score = c(50L,
NA_integer_,
NA_integer_),
reason = c(NA_character_,
"Exceeded time limit",
"Too severe to test")),
class = "trailsList"),
random = runif(3)),
row.names = c(NA, -3L),
class = "data.frame")
This doesn't print properly (and gives a "corrupt data frame" warning) unless we
create a format method for our trailsList class.
format.trailsList <- function(x, ...){
ifelse(is.na(x$score),
sprintf("NA (%s)", x$reason),
sprintf("%ds", x$score))
}
But with a nice format method we get this:
id trails_a random
1 11 50s 0.061
2 22 NA (Exceeded time limit) 0.664
3 33 NA (Too severe to test) 0.826
To use a data frame with a trailsList object we need to define a
number of other methods including operations like == for subsetting.
I don't know all that is involved. But an example to look at would be
the methods for Surv in the survival package.
Conclusion
I don't think most analysts would want to deal with this complexity.
It's more straightforward and understandable to just use two columns
in the data frame: the score and if no score, the reason for NA.
Postscript
Having a variable in a data frame that is of POSIXlt class is
an "established" case that is very similar. This is a data frame
with two observations and three variables clinic, date, and
ldate.
d <- data.frame(clinic = 1:2,
date = as.Date(c("2010-01-01", "2020-01-05")))
d$ldate <- as.POSIXlt(d$date)
dput(d)
structure(list(id = 1:2,
date = structure(c(14610, 18266), class = "Date"),
ldate = structure(list(sec = c(0, 0),
min = c(0L, 0L),
hour = c(0L, 0L),
mday = c(1L, 5L),
mon = c(0L, 0L),
year = c(110L, 120L),
wday = c(5L, 0L),
yday = c(0L, 4L),
isdst = c(0L, 0L)),
class = c("POSIXlt", "POSIXt"),
tzone = "UTC"),
row.names = c(NA, -2L),
class = "data.frame")
September 21, 2020
PowerShell
I used to look down my nose at PowerShell but now I realize there are lots of shells that do a good job. I need to learn the equivalent of aliases in PowerShell to make navigating the Windows filesystem faster but for now if I navigate to a directory with a file that I want to open (say a Word document) then I can open quickly with
ii myfile.docx
Here, ii is a shortening of Invoke-Item.
September 15, 2020
Lattice graphics
A StackOverflow question asked about removing tickmarks on the right and top margins of a plot.
The context was
library(effects)
m <- lm(Fertility ~ ., data = swiss)
plot(allEffects(m), rug = FALSE)
And plot from the effects package didn't allow the user much
control. For example, the usual scales argument wasn't handled. This
was my solution:
trellis.par.set(axis.components = list(top = list(tck = 0),
right = list(tck = 0))
plot(allEffects(m), rug = FALSE)
Another thing I learned was to to control the font family easily.
Using trellis.par.set outside of a plot cal or using
par.settings inside a plot call, one of the elements is grid.pars.
See ?gpar.
str(get.gpar())
xyplot(dist ~ speed, data = cars,
par.settings = list(grid.pars = list(fontfamily = "Open Sans")))
August 27, 2020
Polymode
In a RMarkdown file edited with Emacs and Polymode, the command
polymode-eval-buffer-from-beg-to-point is bound to both M-n v <up> and
M-n v u.
The command polymode-eval-buffer is bound to M-n v b.
The command polymode-eval-region-or-chunk is bound to M-n v v.
August 26, 2020
Lattice graphics
- Axis tick marks on log scale with "normal" numbers:
list(log = 2, equispaced.log = FALSE) - Gridlines to match tick marks:
panel.grid(h = -1, v = -1)
August 20, 2020
The data.table package has fread which is great. One difference between
fread and read.csv is how a field like ,"", is handled. Here is an example
file:
"id","date1","date2","code"
"1","2019-01-01","","3"
"1","2019-01-01","2018-03-18","3"
Using read.csv with defaults gives:
R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE))
'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: chr "2019-01-01" "2019-01-01"
$ date2: chr "" "2018-03-18" <---- Empty string
$ code : int 3 3
Using na.strings = "" gives
R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: chr "2019-01-01" "2019-01-01"
$ date2: chr NA "2018-03-18" <---- Beter
$ code : int 3 3
But I do not know a way to get the same from fread:
R> str(fread("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
Classes 'data.table' and 'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: chr "2019-01-01" "2019-01-01"
$ date2: chr "" "2018-03-18"
$ code : int 3 3
- attr(*, ".internal.selfref")=<externalptr>
By design, fread wants to be smart about allowing an NA field to
be data (e.g., the string "NA". It also wants to allow zero length strings.
I don't know if there is a way around this.
For example, this doesn't work:
R> str(fread("test-fread.csv", colClasses = c("date1" = "Date", "date2" = "Date")))
Classes 'data.table' and 'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: Date, format: "2019-01-01" "2019-01-01"
$ date2: chr "" "2018-03-18"
$ code : int 3 3
- attr(*, ".internal.selfref")=<externalptr>
Warning message:
Column 'date2' was requested to be 'Date' but fread encountered the following error:
character string is not in a standard unambiguous format
so the column has been left as type 'character'
This comes up with ADNI data and the PTDEMOG.csv. Maybe the root of the problem
is having everything quoted in the source CSV.