Files in the top-level directory in any check-in
- ar1.org
- CharisSIL-plot.png
- Makefile
- method-comparison.org
- prev-and-measurement.org
- README.md
- test_mapping.jag
- test_mapping.R
December 16, 2024
Lattice graphics don't have a built-in plot type for plotting point
and interval estimates. The latticeExtras package has segplot
which builds on levelplot and requires draw.bands = FALSE. But
segplot does not have a groups argument which is helpful to
shift estimates by group.
To make a high-level plot called compatplot for compatibility
interval plot, I will probably have to borrow from bwplot.formula
but tweak it so that I can use a formula like
group ~ lower + estimate + upper | sex
which I think will require using lattice:::latticeParseFormula. Here
If I call this:
str(x <- lattice:::latticeParseFormula(wool ~ breaks + log(breaks) + sqrt(breaks),
warpbreaks,
groups = warpbreaks$tension,
subset = TRUE,
multiple = TRUE,
outer = TRUE, subscripts = TRUE))
Then I get this. The data has 54 rows but that is expanded three times
to get 162 rows so the function has "stacked" my right hand side
variables. Note also that groups was a vector of 54 elements but
it is now 162. The list element called condition indicates the
RHS variable.
List of 7
$ left : Factor w/ 2 levels "A","B": 1 1 1 1 1 1 1 1 1 1 ...
$ right : num [1:162] 26 30 54 25 70 52 51 26 67 18 ...
$ condition :List of 1
..$ : Factor w/ 3 levels "breaks","log(breaks)",..: 1 1 1 1 1 1 1 1 1 1 ...
$ left.name : chr "wool"
$ right.name: chr "breaks + log(breaks) + sqrt(breaks)"
$ groups : Factor w/ 3 levels "L","M","H": 1 1 1 1 1 1 1 1 1 2 ...
$ subscr : int [1:162] 1 2 3 4 5 6 7 8 9 10 ...
That makes sense but I wonder if I can also have a conditioning
variable in the formula? It turns out I can. The list element
condition will have the unnamed vector indicating the
right-hand-side variables and then will also incude whatever variables
are conditioned on using the | in the formula.
Interestingly, this generates the call I want to latticeParseFormula
(and I need outer = TRUE):
bwplot(wool ~ breaks1 + breaks2 | group, dd, groups = tension, outer = TRUE)
The plot will give a panel for low tension and high tension for each
group level so the plot itself isn't "on the right track" but the
formula is parsed OK with the last element of condition being
unnamed and a factor which gives "breaks" or "breaks2".
The resulting trellis object will have panel.args as a length 4
list with each list having elements x, y, and subscripts each
54 elemnts long. (panel.args.common has a groups vector
which is 216 elements.)
But this won't work because I need a high-level function that
generates trellis object x with a panel.args that has elements
like y, lower, estimate, upper and that is more similar to
levelplot which has elements panel.args named x, y, z, and
centers (among others).
But levelplot doesn't want a formula with a RHS of lower + estimate + upper
and so if that is important I have a lot of work. Can I give that up and
expand levelplot/segplot to use groups? Maybe I could adopt the otherwise
unused level argument since segplot.formula has
level <- eval(substitute(level), data, parent.frame())
centers <- eval(substitute(centers), data, parent.frame())
Who knows.
December 12, 2024
I used
https://www.raspberrypi.com/documentation/computers/remote-access.html
to test syncing a directory between a Linux laptop and a Raspberry
Pi. After getting passwordless login via SSH setup using
ssh-copy-id weigand@pi from my laptop, I did:
rsync -avz -e ssh /home/weigand/Test weigand@pi:/home/weigand/Backups/laptop
This copies/syncs the contents (inclusing subdirectories) of the
laptop's Test directory to the Pi. The Pi needed the directory
/home/weigand/Backups/laptop to exist first but after that command
the Pi had /home/weigand/Backups/laptop/Test with all the files and
subdirectories.
December 3, 2024
In Org files, I find file- or buffer-local variables confusing. One
approach is with BIND. In the example that follows, I set a
variable named org-html-foo to a list with three elements. (Note the
list is not quoted.)
#+BIND: org-html-foo ("Happy" "Sad" "Medium)
The manual says this:
y
> If org-export-allow-bind-keywords is non-nil, Emacs variables can
> become buffer-local during export by using the BIND keyword. Its
> syntax is #+BIND: variable value. This is particularly useful for
> in-buffer settings that cannot be changed using keywords.
Nov. 17, 2024
I installed Proton VPN on Lubuntu using these two links:
- https://protonvpn.com/support/official-linux-vpn-ubuntu
https://protonvpn.com/support/linux-vpn-lubuntu
wget https://repo.protonvpn.com/debian/dists/stable/main/binary-all/protonvpn-stable-release_1.0.6_all.deb sudo dpkg -i ./protonvpn-stable-release_1.0.6_all.deb && sudo apt update echo "e5e03976d0980bafdf07da2f71b14fbc883c091e72b16772199742c98473002f protonvpn-stable-release_1.0.6_all.deb" | sha256sum --check - sudo apt install proton-vpn-gnome-desktop sudo apt install libayatana-appindicator3-1 gir1.2-ayatanaappindicator3-0.1 gnome-shell-extension-appindicator
Our app relies on the nm-applet NetworkManager applet to work, but Lubuntu uses nm-tray by default. To fix this, go to Menu → Preferences → LXQt settings → LXQt Configuration Center.
Then click on Session Settings and
Go to the Autostart tab → Global Autostart and: Check the Network box Uncheck the nm-tray box
Then restart the machine. Click on the main menu and search for ProtonVPN, enter credentials. I do not know if one has to enter their Proton credentials after every reboot. That would be a minor pain but it makes sense.
Nov. 15, 2024
I am trying Proton Bridge which promises to allow me to use a local mail client. I started here: https://proton.me/support/installing-bridge-linux-deb-file.
cd ~/Downloads
wget https://proton.me/download/bridge/protonmail-bridge_3.13.0-1_amd64.deb
sudo apt install ./protonmail-bridge_3.13.0-1_amd64.deb
Oct. 29, 2024
I figured out how to have Fossil ignore a virtual environment directory. In the checkout directory I did this:
mkdir -p .fossil-settings
echo "venv" > .fossil-settings/ignore-glob
Then fossil add didn't show the venv directory. So easy.
See:
- https://fossil-scm.org/home/doc/trunk/www/globs.md#settings
- https://fossil-scm.org/home/doc/trunk/www/settings.wiki
Oct. 29, 2024
I logged into my Unix account, I did
cd ~/Fossils
fossil new myproject.fossil
mkdir myproject
cd myproject
fossil open ../myproject.fossil
Then from my local computer I cloned the repo via this:
fossil clone ssh://myname@example.com//a/b/c/myname/Fossils/myproject.fossil myproject.fossil
And it worked. And I could commit to the server.
Oct. 27, 2024
To prevent a future web search, I used a Digital Ocean
article
to add a user to the sudo group:
sudo usermod -aG sudo <username>
And to show whitespace in Nano it is Meta-P.
Oct. 24, 2024
This is how I zipped up my homed directory while excluding a Python
virtual environment directory (myvenv)
cd ~/
zip -r backup.zip ./ -x ./project/myvenv/*
I don't think the quotes are needed.
Oct. 21, 2024
Here is a sketch of how to use R's Lattice graphics to show group-wise estimates in panel A and their difference in panel B keeping things lined up nicely.
library(lattice)
library(data.table)
d <- data.table(expand.grid(letters = LETTERS[1:11],
percentile = c("P25", "P75")))
d[, mean := rnorm(nrow(d))]
## duplicate the differences so the y-axs scales are the same for both plots
d[, difference := mean[percentile == "P25"] - mean[percentile == "P75"], letters]
d[letters == "A"]
# letters percentile mean difference
# <fctr> <fctr> <num> <num>
# 1: A P25 0.0746 0.128
# 2: A P75 -0.0538 0.128
p1 <- dotplot(factor(paste(letters, percentile)) ~ mean, data = d, main = "Estimates")
p2 <- dotplot(factor(paste(letters, percentile)) ~ difference,
data = d,
main = "Difference",
panel = function(x, y) {
## If y is odd plot y + 0.5, otherwise NA
panel.xyplot(x, ifelse(unclass(y) %% 2 == 1, unclass(y) + 0.5, NA))
panel.refline(h = 1:22)
},
scales = list(y = list(draw = FALSE)))
graphics.off()
plot(p1, more = TRUE, newpage = TRUE, split = c(1, 1, 2, 1))
plot(p2, more = FALSE, newpage = FALSE, split = c(2, 1, 2, 1))
October 12, 2024
Today I wrote this function to insert the current date into the
buffer. At one time I knew about format-time-string but had to
search the Web for how to insert a formated date. I also was tripped
up by my function initially not returning anything but then I
remembered insert is what I want. I don't know if interactive is
used correctly here.
(defun stephen-insert-date ()
"Insert current date.
Replace two spaces with one space if needed."
(interactive)
(insert
(replace-regexp-in-string " " " " (format-time-string "%B %d, %Y"))))
(global-set-key (kbd "C-c d") 'stephen-insert-date)
Oct. 9, 2024
On Windows, I used Scoop to install coreutils and have Grep in the form of grep.exe.
I can run a command like this:
grep 'Emacs' README.md Doesn't work with CMD
grep Emacs README.md
grep "Emacs" README.md
and get nice results. But if I have two Markdown files to search, CMD and PowerShell do not allow me to use file globbing.
grep "Emacs" *.md
One has to use some kind of for loop to collect the files into a variable.
In Emacs, I learned about file-expand-wildcards:
(file-expand-wildcards "*.org") ;; relative paths
(file-expand-wildcards "*.org" t) ;; Full paths
Maybe I can create a function that allows me to use Emacs's grep but
rather than having the OS do the globbing, I can use this function.
Here is a first attempt:
(defun stephen-grep ()
(interactive)
(let ((command "grep --color=always -nH --null -e")
(pattern (read-string "grep pattern: "))
(file-list
(mapconcat
#'format
(file-expand-wildcards (read-string "file pattern: "))
" ")))
(grep (format "%s \"%s\" %s" command pattern file-list))))
I looked at my Emacs Lisp: an Introduction by Chassell and think I
abused interactive so here is a better way:
(defun stephen-grep (pattern file-list)
(interactive (list (read-string "grep pattern: ")
(mapconcat
#'format
(file-expand-wildcards (read-string "file pattern: "))
" ")))
(let ((command "grep --color=always -nH --null -e"))
(grep (format "%s \"%s\" %s" command pattern file-list))))
This is even better.
(defun stephen-grep (command pattern file-list)
(interactive
(list (read-string
(format "grep command (defaults to \"%s\"): " grep-command)
nil nil grep-command)
(read-string "grep pattern: ")
(mapconcat
#'format
(file-expand-wildcards (read-string "file pattern: "))
" ")))
(grep (format "%s \"%s\" %s" command pattern file-list)))
Sept. 23, 2024
In R's lattice graphics, most extra arguments are passed on to the
panel function but today I learned that p won't work and it's not
clear why.
This works and prints out letters:
xyplot(dist ~ speed,
data = test,
myletters = letters,
panel = function(x, y, ...) {
dots <- list(...)
print(dots)
panel.xyplot(x, y)
})
This does not work.
xyplot(dist ~ speed,
data = test,
p = letters,
panel = function(x, y, ...) {
dots <- list(...)
print(dots)
panel.xyplot(x, y)
})
It prints out
Error in get(FUN) : first argument has length > 1
So if I want to pass additional arguments I should use longer argument
names that won't cause a collision. (For example pvalue rather than
p.)
Aug. 21, 2024
I was confused about Make and .PHONY. Let's say I have this
makefile. The idea is I have a data set named data.txt and the file
report.txt depends on it.
This is how I was using .PHONY:
.PHONY: clean # .PHONY is correct here
clean:
rm -f data.txt report.txt
.PHONY: data # .PHONY not helpful
data: data.txt
data.txt:
echo "This is data" > data.txt
.PHONY: report # .PHONY not helpful
report: report.txt
report.txt: data # always runs; could replace with `data.txt`
cat data.txt > report.txt
echo "This is the report" >> report.txt
The .PHONY for clean is correct. clean is not a file but if
there were a file named clean then clean wouldn't run since
clean (the file) would never be out of date. That makes sense.
But data isn't file so shouldn't it be phony? The GNU Make documentation
says this:
A phony target should not be a prerequisite of a real target file; if it is, its recipe will be run every time
makeconsiders that file. As long as a phony target is never a prerequisite of a real target, the phony target recipe will be executed only when the phony target is a specified goal (see Arguments to Specify the Goals).
Since the prerequisite for report.txt is data, which is a phony target,
I do in fact have the problem that the recipe is run every time Make considers
report.
So take away .PHONY data. But when report.txt depends on multiple data files
how do I specify them without listing each file? I guess it could be
data = data1.txt data2.txt
I have to think I'm using Make wrong and should just re-read the manual.
One postscript is that I learned about make -d to print debugging
information but what makes it useful is make -d -r which is
debugging info but disable the build-in implicit rules which will make
the output much shorter.
Aug. 10, 2024
This GNU Make snippet uses the dir and notdir functions with $<
which is the first prerequisite.
output/plot.pdf: scripts/program.R
cd $(dir $<); Rscript $(notdir $<)
This can be almost cryptic with what follows which is equivalent.
output/plot.pdf: scripts/program.R
cd $(<D); Rscript $(<F)
On the other hand, maybe once one learns <D and <F as the
directory part and filename part of the first prerequisite it is
elegant.
Aug. 8, 2024
Here is how I could take all files of the from *table.org, use Org to format
them via org-table-align, and output *table-fmt.org files.
.PHONY: org-tables
org-tables: $(patsubst %table.org, %table-fmt.org, $(wildcard *table.org))
%table-fmt.org : %table.org
emacs --quick --batch --eval "
(with-temp-buffer
(insert-file \"$<\")
(org-table-align)
(write-file \"$@\"))"
Aug. 7, 2024
On Windows, from PowerShell or Command Prompt, and using Emacs Speaks Statistics within Emacs, R knows about
c:\Users\MYNAME\Documents\.Rprofile
But using GNU make (installed via Scoop), R looks for .Rprofile in
c:\Users\MYNAME
I tried to set HOME in the Make file but I couldn't get R to know
about the .Rprofile in my Documents directory. The simplest thing that works
is now I have this in c:\Users\MYNAME\.Rrofile:
message("This file is 'C:/Users/WEIGAND/.Rprofile'")
message("We will source 'C:/Users/WEIGAND/Documents/.Rprofile'")
source("C:/Users/WEIGAND/Documents/.Rprofile")
Aug. 4, 2024
It looks like the key required to type F4 is also dead. I only wanted
this in emacs for keyboard macros but kmacro.el says C-x ( and C-x )
and C-x e are usable.
It looks like C-x e is kmacro-end-and-call-macro; C-x ) is kmacro-end
and C-x ( is kmacro-start-macro.
If I create a macro to insert the letter x, this is the output of M-x
insert-kbd-macro:
(setq last-kbd-macro "x")
But I couldn't replay that macro in the minibuffer.
C-x C-k n will name the keyboard macro. I just called it my-r.
C-x C-k b will bind the most recent keyboard macro to a key sequence.
Finally
To avoid problems caused by overriding existing bindings, the key sequences C-x C-k 0 through C-x C-k 9 and C-x C-k A through C-x C-k Z are reserved for your own keyboard macro bindings.
And to bind a keyboard macro to, say, A, one does C-x C-k b A.
This makes me wonder if a keychord with xk standing in for C-x C-k
would make sense.
Aug. 3, 2024
On my 2008 MacBook I have two keys that are not working. The laptop has a top row of keys with escape, then F1 though F12 and then eject. Can I use F5 and F6 as substitutes for my two bad keys?
These are my problem keys:
- r is ASCII 114
- R is ASCII 82
- 4 is ASCII 52
- $ is ASCII 36
I wrote a series of functions and keyboard shortcuts in Emacs lisp as workarounds.
But then I found F5 and F6 were "available" and wrote these:
(defun myfix-r ()
"Insert an 'r'."
(interactive)
(insert-char 114))
(global-set-key (kbd "<XF86KbdBrightnessDown>") 'myfix-r)
(defun myfix-R ()
"Insert an 'R'."
(interactive)
(insert-char 82))
(global-set-key (kbd "<XF86KbdBrightnessUp>") 'myfix-R)
But this is wrong-headed; I found out when attempting an incremental
search in the *Packages* buffer that I can't do C-s and then spell
out "markdown-mode" since my fix inserts an "r" rather than types
one. I don't know if this has something to do with how
self-inset-command works. I think I need a system-level fix.
Previously, since this MacBook has a single, difficult to reach
control key (on the left), I edited /etc/default/keyboard as follows:
# I saw this on Mastodon. It makes capslock an additional control
XKBOPTIONS="ctrl:nocaps"
My thinking was that I can do what I want using this file.
The first thing I leaned was that xmodmap -g gives the grammar.
xmodmap accepts the following input expressions:
pointer = default reset pointer buttons to default
pointer = NUMBER ... set pointer button codes
keycode NUMBER = [KEYSYM ...] map keycode to given keysyms
keysym KEYSYM = [KEYSYM ...] look up keysym and do a keycode operation
clear MODIFIER remove all keys for this modifier
add MODIFIER = KEYSYM ... add the keysyms to the modifier
remove MODIFIER = KEYSYM ... remove the keysyms from the modifier
where NUMBER is a decimal, octal, or hex constant; KEYSYM is a valid
Key Symbol name; and MODIFIER is one of the eight modifier names: Shift,
Lock, Control, Mod1, Mod2, Mod3, Mod4, or Mod5. Lines beginning with
an exclamation mark (!) are taken as comments. Case is significant except
for MODIFIER names.
Keysyms on the left hand side of the = sign are looked up before any changes
are made; keysyms on the right are looked up after all of those on the left
have been resolved. This makes it possible to swap modifier keys.
This grammar is for the xmodmap command but it gives a hint of the
format of the text I'll enter in /etc/default/keyboard.
I think an important command in Linux is xev -event keyboard. I do
this and then press first "q" and then "s" and this tells me the
keycode for "q" is 24 and that for "s" is 39. The output below shows
press events and release events.
KeyPress event, serial 25, synthetic NO, window 0x2800001,
root 0x22f, subw 0x0, time 13104639, (760,452), root:(762,482),
state 0x0, keycode 24 (keysym 0x71, q), same_screen YES,
XLookupString gives 1 bytes: (71) "q"
XmbLookupString gives 1 bytes: (71) "q"
XFilterEvent returns: False
KeyRelease event, serial 28, synthetic NO, window 0x2800001,
root 0x22f, subw 0x0, time 13104751, (760,452), root:(762,482),
state 0x0, keycode 24 (keysym 0x71, q), same_screen YES,
XLookupString gives 1 bytes: (71) "q"
XFilterEvent returns: False
KeyPress event, serial 28, synthetic NO, window 0x2800001,
root 0x22f, subw 0x0, time 13111655, (760,452), root:(762,482),
state 0x0, keycode 39 (keysym 0x73, s), same_screen YES,
XLookupString gives 1 bytes: (73) "s"
XmbLookupString gives 1 bytes: (73) "s"
XFilterEvent returns: False
KeyRelease event, serial 28, synthetic NO, window 0x2800001,
root 0x22f, subw 0x0, time 13111735, (760,452), root:(762,482),
state 0x0, keycode 39 (keysym 0x73, s), same_screen YES,
XLookupString gives 1 bytes: (73) "s"
XFilterEvent returns: False
But I found out the keycode for "e" is 26 and "t" is 28 so for "r" it is probably 27. But that is not important since th R key is dead. Maybe I need to know this which is my brightness down key (but it has a little F5 on the key):
KeyPress event, serial 25, synthetic NO, window 0x2800001,
root 0x22f, subw 0x0, time 13474566, (229,174), root:(231,204),
state 0x0, keycode 237 (keysym 0x1008ff06, XF86KbdBrightnessDown), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 0 bytes:
XFilterEvent returns: False
Here is what may work using the format keycode <KEYCODE> = <KEYSYM>
xmodmap -e "keycode 237 = 72"
Well, that gave me an h but though trial and error this gives me an "r":
xmodmap -e "keycode 237 = 82"
Is it chance that ASCII 82 is a capital "R"? I don't know but after running that, shift with brightness down gives me a capital "R". Yay.
OK, now I see that the keycode is for, well, a key, and a capital letter has a shift event and a key event which gives a capital character.
Now can I get a "4"?
xmodmap -e "keycode 238 = 52"
That works but with shift I do not get the dollar sign.
xmodmap -e "keycode 173 = 36"
OK, now all good. Actually, better is http://xahlee.info/linux/linux_xmodmap_tutorial.html.
keycode 238 = 4 dollar 4 dollar
Finally, this is what my ~/.Xmodmap looks like:
! Added by Stephen Aug. 3, 2024
! See http://xahlee.info/linux/linux_xmodmap_tutorial.html
! Key with small F5 on it gives me an r/R.
keycode 237 = r R r R
! Key with small F6 on it gives me a 4/$
keycode 238 = 4 dollar 4 dollar
But I had a hard time getting this to autostart. Since I am using
Lubuntu which uses LXQt, I found I could do this: Menu -> Preferences
-> LXQt Settings -> Session Settings and then click on Autostart and
then New. I named an item "Xmodmap" and had the command be xmodmap
~/.Xmodmap. That works.
That works and does so by creating a file called
~/.config/autostat/Xmodmap.desktop which is this ini-style file:
[Desktop Entry]
Exec=xmodmap ~/.Xmodmap
Name=Xmodmap
OnlyShowIn=LXQt;
Type=Application
Version=1.0
From man startx:
Note that in the Debian system, what many people traditionally put in the .xinitrc file should go in .xsession instead;
But the LXQt way is different.
This took most of a Saturday but it made the laptop usable and I needed to be supine for most of the day to rest my back.
Aug. 2, 2024
I forget what Org entities are available so I need to remember the command M-x org-entities-help.
Aug. 2, 2024
In Emacs, to widen or narrow the frame, I learned about this
M-x set-frame-width
And programmatically I can use this
(set-frame-width (selected-frame) 125)
which could be helpful if I wanted to write functions to set the frame to, say, three different widths.
Aug. 2, 2024
Using R on a Windows machine, I had trouble installing the R package deldir using
install.packages("deldir", repos = "https://mirror.las.iastate.edu/CRAN/")
I think it was because this package needed to be compiled and maybe the zip file
comes with suspicious binaries. (It includes a file called deldir/libs/x64/deldir.dll
which a human could read as being related to deleting a directory.)
I don't know why a package like data.table which needs compilation
doesn't have any DLL or EXE files and whether that is related to why I can
install it.
All this comes because (I think) the latticeExtra package can't be loaded
without the deldir package even though the latter is only suggested for latticeExtra.
July 31, 2024
I had trouble updating Scoop, getting this message:
fatal: unable to access 'https://github.com/lukesampson/scoop/': SSL certificate problem: unable to get local issuer certificate
I know nothing about certificates but searching for the problem led me to GitHub advice to the effect of:
git config --global http.sslVerify "false"
or
git config --global http.sslBackend openssl
Turning off SSL verification works but seems bad. Setting the backend
to schannel which is "the native Windows Secure Channel library"
seems to work.
I also learned that using --global means the file C:\Users\myname\.gitconfig is updated. Here's what mine
looks like now:
[credential]
helper = manager
[credential "helperselector"]
selected = <no helper>
[http]
sslBackend = schannel
sslVerify = true
July 12, 2024
I wanted to use Make to process one type of file to create another
without me having to specify each file to process. That is, I want
n.foo to be converted to n.bar automatically.
An example "pattern rule" to do this in Make is of the following form:
%.bar : %.foo
myconverter $< $@
This generates n.bar from n.foo using myconverter.
(I always forget the "automatic variables" but $< is the first
prerequisite and $@ is the target.)
The problem is that even if n.foo exists and the Makefile
is only the above two lines, the command make will give the
following:
make: *** No targets. Stop.
I didn't understand this. I found that if the file n.foo exists I
can enter make n.bar and I will get n.bar. But I don't want to
list out each I want on the command line.
The documentation pointed out my problem:
The order of rules is not significant, except for determining the default goal: the target for make to consider, if you do not otherwise specify one. The default goal is the first target of the first rule in the first makefile. There are two exceptions: a target starting with a period is not a default unless it also contains one or more slashes, ‘/’; and, a target that defines a pattern rule has no effect on the default goal.
I take the problem to be answered by the last sentence. I have a pattern rule but it has no effect on the default goal (or targets). My single line file doesn't have any goals.
But I can add the "all" goal which creates a list of the .bar
files I want.
all: $(patsubst %.foo, %.bar, $(wildcard *.foo))
%.bar : %.foo
myconverter $< $@
This may not be a perfect explanation but it's close enough and my problem is solved.
May 15, 2024
I don't understand formulas and their environments in R. I wanted this to work:
library(data.table)
library(lme4)
## Example of model to fit
fit <- lmer(log(y) ~ x1 + x2 + (1 | unit),
data = mydata,
subset = region == 1)
## This doesn't work
out <- mydata[, coef(summary(lmer(formula(fit)))), region]
But this works of course:
out <- mydata[, coef(summary(lmer(log(y) ~ x1 + x2 + (1 | unit)))), region]
A solution is to use .SD so
out <- mydata[, coef(summary(lmer(formula(fit), data = .SD))), region]
In the manual page for formula it says:
A formula object has an associated environment, and this environment (rather than the parent environment) is used by
model.frameto evaluate variables that are not found in the supplieddataargument.
I take this to mean that when I did this:
out <- mydata[, coef(summary(lmer(formula(fit), data = .SD))), region]
then lmer was using the environment for formula(fit) to look for
variables since I didn't supply a data argument. But if I supply
a data argument via .SD I am OK.
May 7, 2034
From https://opensource.com/article/19/8/dig-binary-files-hexdump
I learned I can customize the output for hexdump:
hexdump -e'60/1 "%02x""\n"' myfigure.png
Will output the file as "pure" hexadecimal, kind of like
what Emacs's hexlify-buffer does except without the
address information and ASCII representation.
I also learned that hexlify-buffer relies on a binary called
hexl on my system. On my system it is found at
/usr/libexec/emacs/29.1/x86_64-linux-gnu/hexl
The source is https://git.savannah.gnu.org/cgit/emacs.git/tree/lib-src/hexl.c and I don't think there are any meaningful formatting options.
I think I can customize the Emacs variables hexl-program and
hexl-options to use hexdump if available.
I could also use hexlify-region and then clean up the non-hex
columns.
May 5, 2024
- I went to https://zoom.us/download?os=linux
- I selected Ubuntu 64 bit and clicked download
- Then in the directory where the DEB file ended up:
sudo apt install ./zoom_amd64.deb
April 26, 2024
Here is an Emacs lisp file that could form the basis of a command line script.
#!/usr/bin/emacs -x
(print command-line-args)
March 27, 2024
I was looking in my Lattice: Multivariate Data Visualization with R
book and re-learned about the page argument to Lattice plots and how
it is a good way to add annotations at the whole-page level. This
function has as its coordinate system the full display area scaled to
[0,1] by [0,1].
The page argument takes a function which has a single argument
which corresponds to the page number. (Lattice graphics can be
multiple pages.) In the example below, I ignore the n argument
and draw text with panel.text and grid.text.
library(lattice)
library(grid)
xyplot(dist ~ speed,
data = cars,
page = function(n) {
panel.text(labels = "A.", x = 0.01, y = 0.99, adj = c(0, 1), cex = 3)
grid.text(label = "B", x = 0.99, y = 0.99, just = c("right", "top"),
gp = gpar(col = "firebrick"))
})
I need to learn more about complicated legends because they may be a good way to add even more plot annotations but this will do for simple text annotations.
March 6, 2024
Using R's lattice and latticeExtra here is a scatter plot
smoother with intervals by group
d <- data.frame(x = rnorm(100), y = rnorm(100), g = rep(c("A", "B"), c(25, 75)))
xyplot(y ~ x, d, groups = g,
panel = panel.superpose,
panel.groups = function(x, y, ...) {
panel.points(x, y, ...)
panel.smoother(x, y, method = "loess", span = 0.9, ...)
})
January 13
In Org, I can use Babel to generate Org. The code block below sets the
results to value and raw. (Using output rather than raw
doesn't work.) The raw says to insert the results of running the
code block as-is rather than wrapping it in any way.
#+begin_src elisp :results value raw :exports none
(concat "** Level 2 headline\n"
"*** Level 3 headline\n"
"A short sentence.\n")
#+end_src
And then if I do C-c C-c in the block above, the buffer looks like this:
#+begin_src elisp :results value raw :exports none
(concat "** Level 2 headline\n"
"*** Level 3 headline\n"
"A short sentence.\n")
#+end_src
#+RESULTS:
** Level 2 headline
*** Level 3 headline
A short sentence.
This is great but a limitation is that if I do C-c C-c again in the
code block, then the RESULTS are inserted again rather than
replaced. I think this is because once the Org text is in the buffer,
Org has no way to know it's a code block's output. This makes sense if
I think of it as raw Org; it's not a block to be "updated" via Babel.
I originally looked into this to use code to auto-generate a statistical report. I think it would be useful as "one-time auto-generation". For example, if I had a number of pies and I wanted to show one figure per pie and separate them by headlines, I could have this code block:
#+begin_src R :results value raw :exports none
headlines <-
c(`Pumpkin pie` = "./figures/pumpkin.png",
`Apple pie` = "./figures/apple.png")
sprintf("** %s\n[[file:%s]]\n",
names(headlines),
headlines)
#+end_src
And do C-c C-c and it generates this.
#+RESULTS:
** Pumpkin pie
[[file:./figures/pumpkin.png]]
** Apple pie
[[file:./figures/apple.png]]
I would run it once interactively to fill in the report.
I don't know if this is particularly useful. I could write a script to
generate this Org text outside of Org but now that I think about it,
it's much more maintainable to have the code block in the file. If I
want to change what is output, I have to remove it from the Org file
and regenerate it with C-c C-c but at least the report Org file
shows what will be in the exported document.
November 1
In R, I can use read.csv to read a particular CSV in a Zip archive
sl <- read.csv(unzip("../data/file.zip",
files = "folder_unzips_to/file.csv")
October 27
In Emacs I am trying the cyberpunk theme because it works well with
the rainbow delimiters package but cyberpunk was giving a light
background to Org source blocks. I learned how to customize it by
going to an empty part of the source block and typing describe-face
and seeing that the face was org-block. I changed the background
color to black (#000000) and pressed Save and Apply.
October 25, 2023
In Emacs, I sometimes type C-x C-- by accident which calls
text-scale-adjust to shrink the text size of the window.
And C-x C-+ makes it bigger. After either C-x C-- or
C-x C-+ then - makes text smaller, + makes it bigger,
and 0 returns it to the default.
October 24, 2023
Using Lattice graphics in R, I needed an axis on the square root scale
for a scatter plot matrix generated with lattice::splom. The scales
are controlled by the pscales argument to splom (but really
pscales is an argument to panel.pairs).
If a single number, the pscales argument controls how many tick
marks are desired on the axes. But it can also be a list with one
component for each column being plotted. The list should have these
elements:
‘at’: a numeric vector specifying tick locations
‘labels’: character vector labels to go with at
‘limits’: numeric 2-vector specifying axis limits (should be
made more flexible at some point to handle factors)
These are specifications on a per-variable basis, and used on
all four sides in the diagonal cells used for labeling.
Factor variables are labeled with the factor names. Use
‘pscales=0’ to suppress the axes entirely.
One thing I found is that the list's elements should be at,
lab, and lim to avoid partial match warnings since
panel.pairs uses pscales[[i]]$lab and pscales[[i]]$lim.
Here's an example. It's not ideal.
library(lattice)
d <- data.frame(x = exp(rnorm(100, 1)),
y = exp(rnorm(100, 1)))
calc.pscales <- function(x.untransformed) {
list(at = sqrt(pretty(x.untransformed)),
lab = pretty(x.untransformed),
lim = extendrange(sqrt(x.untransformed), f = 0.1))
}
lapply(d, calc.pscales)
splom(sqrt(d), pscales = lapply(d, calc.pscales))
October 16, 2023
I like text-based help in R and so in my ~/.Rprofile I had
options(help_type = "text")
That was working on Linux but not on Windows. It turns out adding this to
~/.Rprofile in Emacs on Windows does not help. To find out where
R is looking for .Rprofile I read the manual and found:
R> Sys.getenv("R_USER")
This is c:\Users\WEIGAND\Documents.
So I needed to edit c:\Users\WEIGAND\Documents\.Rprofile.
October 15, 2023
Some variables in Emacs are buffer variables. For example setting
show-trailing-whitespace to something non-nil means highlight
trailing whitespace in the buffer. But the help for this variable says
Automatically becomes buffer-local when set.
I learned that in order to set this variable in my init.el file,
I need to use
;; (setq show-trailing-whitespace t) <-- Won't do what I want
(setq-default show-trailing-whitespace t)
September 22, 2023
This is how I made a scatter plot in Times New Roman using R's lattice graphics package on Windows:
library(lattice)
library(grid)
windowsFonts(TNR = "Times New Roman")
xyplot(dist ~ speed,
cars,
main = "A scatter plot in Times New Roman",
par.settings = list(grid.pars = gpar(fontfamily = "TNR"),
fontsize = list(text = 24)))
On Linux it was more direct:
library(lattice)
xyplot(dist ~ speed,
cars,
main = "A scatter plot in Times New Roman",
par.settings = list(grid.pars = gpar(fontfamily = "Times New Roman"),
fontsize = list(text = 24)))
September 18, 2023
Here is a strip plot in lattice that shifts points by group to reduce overlap:
stripplot(wool ~ breaks,
data = warpbreaks,
groups = tension,
panel = panel.superpose,
panel.groups = function(x, y, group.value, ...) {
dots <- list(...) # Print this to see what else is in `...`
panel.xyplot(x, y + c(-0.1, 0, 0.1)[match(group.value, c("L", "M", "H"))], ...)
})
I didn't undertand this before but if the panel had only ... as the argument list, Lattice
would call the function for each level of groups and dots would be this:
str(dots)
List of 21
$ x : num [1:18] 26 30 54 25 70 52 51 26 67 27 ...
$ subscripts : int [1:18] 1 2 3 4 5 6 7 8 9 28 ...
$ pch : num 1
[... other par.settings stuff ...]
$ alpha : num 1
$ type : chr "p"
$ group.number: int 1 <--- first time it's 1, then 2, etc.
$ group.value : chr "L" <--- "L", then "M", etc.
$ grid : logi FALSE
$ box.ratio : num 1
$ horizontal : logi TRUE
$ y : num [1:18] 1 1 1 1 1 1 1 1 1 2 ...
So I can count on group.number and group.value being meaningful
explicit arguments or part of ....
September 18, 2023
R's lattice package makes it easy to add a regression line by a grouping factor but I did not know how to add confidence intervals to the regression line. This uses the latticeExtra package to "add" a regression line with 95% CI but it does not do it by group:
library(lattice)
library(latticeExtra)
n <- 100
test <- data.frame(x = rnorm(n),
y = rnorm(n),
g = sample(letters[1:3], size = n, replace = TRUE))
xyplot(y ~ x, test, groups = g) + layer_(panel.smoother(x, y, method = "lm"))
The solution I came up with is to create a simple panel function that
plots points and adds a regression line with a 95% CI. It uses
panel.points from lattice and panel.smoother from
latticeExtra. Then I created a simplified version of panel.xyplot
which uses panel.superpose to handle grouping.
## A simple panel function with a scatter plot and a smoother
panel.simple <- function(x, y, ...) {
panel.points(x, y, ...)
panel.smoother(x, y, method = "lm", ...)
}
## A "groups-aware" panel function the uses `panel.superpose`
panel.xyplot.simple <- function(x, y, groups = NULL, ...) {
if (is.null(groups)) stop("We need 'groups' to not be NULL")
panel.superpose(x, y, groups = groups, panel.groups = panel.simple, ...)
} # ^^^^^^^^^^^^ ^^^^^^^^^^^^
## A plot with group-specific regression lines and 95% CIs.
xyplot(y ~ x, test, groups = g, panel = panel.xyplot.simple)
And here is a better way that only requires me to write panel.simple
and which is documented in the manual page for xyplot:
xyplot(y ~ x, test, groups = g, panel = panel.superpose, panel.groups = panel.simple)
August 6, 2023
I learned how to publish a project using Emacs Org mode. The manual explains it well (see Publishing).
This is my alist with one project (www) but there can be multiple
projects.
(setq org-publish-project-alist
'(("www"
:base-directory "~/fossils/www/"
:publishing-function org-html-publish-to-html
:publishing-directory "~/fossils/www"
:section-numbers nil
:with-toc nil
:html-doctype "<!DOCTYPE html>"
:html-head "<link href=\"normalize.css\" rel=\"stylesheet\" type=\"text/css\">
<link href=\"sakura-vader.css\" id=\"sakura-css\" rel=\"stylesheet\" type=\"text/css\">")))
Then, to publish, it's org-publish-project and when prompted say
which project. Putting the parameters in the
org-publish-project-alist reduces boilerplate in the Org file. I was
doing this at the top of files I was planning to export to HTML.
#+TITLE: My Title
#+OPTIONS: toc:nil num:nil html-style:nil html-scripts:nil html-postamble:nil html5-fancy:t
#+HTML_HEAD: <link href="normalize.css" rel="stylesheet" type="text/css">
#+HTML_HEAD: <link href='sakura-vader.css' id="sakura-css" rel='stylesheet' type='text/css'>
#+HTML_DOCTYPE: html5
My list above is not consistent with some of the lines immediately above which I will try to fix. To see what options can be specified in the list, go to Publishing options.
As an example, the option :html-preamble is controlled by the
variable org-html-preamble. It is by default t and non-nil
means insert an HTML preamble. If t it inserts a string
defined by org-html-preamble-format. But if set to a string
it uses the string. And if it is set to a function, apply
the function.
The variable org-html-preamble-format is by default (("en" "")).
The second element of this enclosed list is a format string as follows:
%t stands for the title.
%s stands for the subtitle.
%a stands for the author’s name.
%e stands for the author’s email.
%d stands for the date.
%c will be replaced by ‘org-html-creator-string’.
%v will be replaced by ‘org-html-validation-link’.
%T will be replaced by the export time.
%C will be replaced by the last modification time.
If this is part of the org-publish-project-alist:
:html-preamble "written by %a")))
The exported HTML will have this div:
<head>
...
</head>
<body>
<div id="preamble" class="status">
written by Stephen Weigand
</div>
<div id="content">
<h1 class="title"> ... etc.
August 2, 2023
I'm not sure why using Fossil via ssh started requiring my pass-phrase. I did the following:
PS> ssh-keygen.exe -t ed25519
PS> cd ~/.ssh
PS> cat id_ed25519.pub | ssh weigand@myserver.com "cat >> ~/.ssh/authorized_keys"
^^^
IMPORTANT!!!
But then on myserver.com I had to edit authorized_keys and take
out the Windows stuff like a carriage return before the newline at the
end of the file. And while I was there I deleted a previous entry from
this Windows machine.
July 11, 2023
Using R's lattice package I learned or remembered one can give a
vector of x-axis, y-axis, and main labels. Here is a 2 by 2 figure
with two x-axis labels and two y-axis labels
library(lattice)
n <- 100
d <- data.frame(x = rnorm(n),
y = rnorm(n),
g = sample(letters[1:4], size = n, replace = TRUE))
xyplot(y ~ x | g,
d,
xlab = c("Flower", "Tree"),
ylab = c("Shrub", "Fruit"))
Another thing us I remembered to use main as a list. This puts
the main title at the left of the viewport. See the main
entry in the help file for xyplot.
xyplot(dist ~ speed,
data = cars,
main = list(label = "A. Scatter plots", x = unit(0, "npc"), just = "left")
July 1, 2023
The Feb. 9, 2023 entry shows how to delete files in Windows even if they don't
exist. In searching for a platform independent way to remove files I remembered
fossil clean. See https://fossil-scm.org/home/help?cmd=clean.
Here's how to remove all *.Rout and *.log files which may be in the current
or any other directory.
fossil clean --clean *.Rout,*.log
It also works with
fossil clean --clean "*.Rout,*.log"
One can also ask Fossil to remove empty directories as part of the operation:
fossil clean --clean "*.Rout,*.log" --emptydirs
Adding the --force option to the above command will prevent Fossil from asking
before removing each empty directory.
May 4, 2023
In R's base graphics, coloring box plots separately by group is trivial:
boxplot(breaks ~ interaction(tension, wool), warpbreaks, col = six.nice.colors)
Using Lattice graphics I learned it can be done by specifying a vector of fill
colors as part of the box.rectangle element of the par.settings list.
p.s <- list(box.umbrella = list(col = "black", lty = 1),
box.rectangle = list(col = "black", fill = six.nice.colors))
bwplot(breaks ~ interaction(tension, wool), warpbreaks, par.settings = p.s, pch = "|")
More general Lattice note
When generating a Lattice plot, creating a plot object doesn't
generate any calls to grid::viewport. The viewport function is
only called when one plots the object. The reason I am exploring how
Lattice works is I would like to make the plots more "table like" so
that a stripplot of the form
stripplot(level ~ y | group, layout = c(1,2), as.table = TRUE)
looks like this
Group 1
Level 1 * * * *
Level 2 * * *
Level 3 ** * **
Group 2
Level 1 * * * *
Level 2 * * * **
Level 3 * * ** *
|---+---+---+---+---|
To customize the strips, strip.default is (probably) not going to
help since the strips are drawn to fill up the parent
viewport. lattice:::plot.trellis is giving the strip a narrow
viewport spanning the panel which strip.default fills up with a
background, text, and border.
I think I need to examine lattice:::plot.trellis to see where the
viewport for the strip is being created. It might be here:
pushViewport(viewport(layout.pos.row = pos.row - 1,
layout.pos.col = pos.col,
xscale = xscale,
clip = "off",
name = trellis.vpname("strip",
column = column,
row = row,
prefix = prefix,
clip.off = TRUE)))
File this under "Today I Don't Know" because I don't have any good sense of
where to implement my idea of moving strips left. I think I could say, "Hey,
don't generate a default viewport for the strip but a wider version expanding
off to the left." But then which function is doing the calculations to make
it "fit". (I think it is lattice:::plot.trellis which is huge and gnarly.)
A hack may be to say the text in the strips will be adjusted left to the extent of the existing plot. We don't need (much?) more space on the left margin so long as we are just a little to the left of the y-axis labels (which are serving as subgroup names).
That is, we don't want this which takes a lot of new spacing calculations:
Group 1
Level 1 * * * *
but rather an "indented look" like this which can use existing or slightly modified padding:
Group 1
Level 1 * * * *
Maybe the simplest thing is along the lines of this:
myplot <- stripplot(...)
myplot <- add_left_padding_as_needed_based_on_strip_text(myplot)
justify_strip(myplot, just = "left") # Uses grid::grid.edit iterating over panels
Another possible name/metaphor is exdent_strip(myplot). (Or exdentStrip since Lattice
often uses camel case.)
April 28, 2023
I remembered to try using Emacs file-local variables to "compile" R files.
In a plain R file I might have this at the top:
## -*- compile-command: "Rscript foo.R" -*-
And in an Rmd file I might have this:
<!-- -*- compile-command: "Rscript -e 'library(\"knitr\"); knit(\"foo.Rmd\")'" -*- -->
April 24, 2023
Pipx
I previously played with pipx to manage python command line
applications. I couldn't remember if I used Scoop but it doesn't look
like it. Today I'm using Python 3.11 via Scoop and I did:
python -m pip install --user pipx
I then got a warning about
c:\users\weigand\appdata\roaming\python\python311\scripts\pipx.EXE
not being on my path so I fixed that (via typing "path" in the Windows
search bar and clicking on Environmental Variables).
I then followed the pipx suggestion and typed: pipx ensurepath.
Then I did pipx install csvkit and it worked.
in2csv from csvkit
This worked great to convert an Excel XLSX sheet into a CSV.
in2csv.exe --sheet "Sheet XYZ" myfile.xlsx > sheet_xyz.csv
March 29, 2023
Passing arbitrary arguments to Rscript (or Rscript.exe on Windows) is easy.
If a script called test-rscript.R exists and has this one line:
print(commandArgs())
then a command like this:
Rscript --vanilla test-rscript.R --never always -often --
Produces this:
[1] "C:\\scoop\\apps\\r\\current\\bin\\x64\\Rterm.exe"
[2] "--no-echo"
[3] "--no-restore"
[4] "--vanilla"
[5] "--file=test-rscript.R"
[6] "--args"
[7] "--never"
[8] "always"
[9] "-often"
[10] "--"
This shows me that it easy to parse arbitrary command line arguments without having to rely on an external package.
March 28, 2023
I'm trying to write a shell command to export an Org file. I'm using
emacs test.org --batch --funcall org-html-export-to-html --kill
but it doesn't evaluate the code blocks to generate figures using R.
My true aim is RTF export using a local exporter I created called
ox-rtf.el. This creates an RTF document but again, no figures.
emacs test.org --batch --load ox-rtf.el --funcall org-rtf-export-to-rtf --kill
I'm using Batch export of org-mode files from the command line from Stack Overflow. (The answers are nine years old.) There is also this suite of command line tools https://github.com/fniessen/orgmk.
Additional info
The command line above "works" in that it exports the file as it is on
disk. This means that if the code blocks are all evaluated then the
exporting will show the figures. When editing the file and typing C-c
C-e (which calls org-export-dispatch and invokes the menu) the exporting
process includes the additional step of evaluating the code blocks.
(It may be something like it calls org-babel-execute-buffer then
does the exporting.)
The manual has this about
The sample script shows batch processing of multiple files using
org-babel-tangle.
#!/bin/sh
# Tangle files with Org mode
#
emacs -Q --batch --eval "
(progn
(require 'ob-tangle)
(dolist (file command-line-args-left)
(with-current-buffer (find-file-noselect file)
(org-babel-tangle))))
" "$@"
So maybe the solution is something like
emacs -Q --visit test.org --batch --load ox-rtf.el --eval "(org-babel-execute-buffer) (org-rtf-export-to-rtf)" --kill
The problem with the above is Org Babel didn't know what to do with R code blocks. So I tried this and it seems to work.
emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el --eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill
If I am using the CMD shell, I can continue lines with ^
emacs -Q --visit test.org --batch --load ox-rtf.el --load ~/.emacs.d/init.el ^
--eval "(progn (setq ess-ask-for-ess-directory nil) (org-babel-execute-buffer) (org-rtf-export-to-rtf))" --kill
One problem is that I am loading my own init.el file and so this
command will only work for me. I am using the Org (version 9.5.5) that
came with my Emacs (version 28.2).
March 28, 2023
I'm trying to customize my ~eshell~ prompt. That is not working right now but this Emacs lisp snippet was helpful to learn:
(substring (shell-command-to-string "fossil status") 0 -1)
It captures the results of the fossil status command and trims the last
character which is a newline.
March 21, 2023
From
https://www.masteringemacs.org/article/executing-shell-commands-emacs
I read about shell-command for one-liners.
March 21, 2023
A couple of times recently I've been typing madly in Emacs and my buffer's font size gets smaller. After a web search I want to remember these keys:
C-x C-+will increase the buffer font size (as willC-x C-=which saves a shift key)C-x C--will decrease the buffer font size
These seem to invoke text-scale-adjust and I also wan to remember
C-x C-0will reset to the default size
March 13, 2023
After updating Emacs to 28.2 using Scoop on Windows, my usual way of starting Emacs via the Start Menu search bar starts Emacs with a CMD window behind it. From this:
https://stackoverflow.com/questions/41300/emacs-in-windows
I think the problem is this is starting the executable program
emacs.exe but I really want to be starting runemacs.exe. I don't
know how to get the old behavior but I found that I could create a
"Desktop Shortcut" as follows:
- I right-clicked on
runemacs.exeand created a shortcut - I moved this shortcut to the Desktop folder
- I right-clicked on the shortcut and selected properties
- In the Shortcut key text box I typed "e" so that my shortcut is "Ctrl + Alt + E". (It's a weird box because it is not editable; you just have to type a single character in there and the short cut is "Ctrl + Alt + X".)
While I was doing this I also changed the "double click action" (my term)
for files I want to open in Emacs to be opened with runemacs.exe.
A follow-up note is that I originally tried having my shortcut in a Shortcut folder within Desktop but that didn't seem to work. I guess the term "desktop shortcut" means a shortcut on the desktop and not anywhere else.
March 7, 2023
Using Scoop on Windows I updated R and Rtools and then was unable to use the brms package. I used this to see what version I had and what versions were installed.
PS> scoop info r
Name : r
Description : A free software environment for statistical computing and graphics.
Version : 4.2.2
Bucket : main
Website : https://www.r-project.org
Updated at : 11/2/2022 1:22:47 AM
Updated by : Hsiao-nan Cheung
Installed : 4.1.0
4.1.1
4.1.3
4.2.2
Binaries : bin\x64\R.exe | bin\x64\Rcmd.exe | bin\x64\Rgui.exe | bin\x64\Rscript.exe | bin\x64\Rterm.exe
Shortcuts : R
...
I used this to "reset" to 4.1.3.
PS> scoop reset r@4.1.3
Resetting r (4.1.3).
Linking C:\scoop\apps\r\current => C:\scoop\apps\r\4.1.3
Creating shim for 'R'.
Creating shim for 'Rcmd'.
Creating shim for 'Rgui'.
Creating shim for 'Rscript'.
Creating shim for 'Rterm'.
Creating shortcut for R (Rgui.exe)
C:\Users\WEIGAND
This was very, very nice.
Feb. 14, 2023
In Emacs I decided to always have line numbers on and Web searches
suggested as of Emacs 26 use (global-display-line-numbers-mode)
in one's initialization file. I then customized the line-number face
and the line-number-current-line face to both have a height of 80%.
Feb. 9, 2023
I learned how to remove files in Windows CMD even if such files aren't present. Here is an example from a Makefile:
.PHONY: clean
clean:
cd code && if exist *.png (del *.png)
Jan. 6, 2023
Through trial and error I now understand colors in RTF as implemented by Microsoft Word (365 on the desktop) a little better.
Highlighting is done via
\highlightN. For example,{\highlight5 This text is highlighted}will be highlighted using the fifth color in the color table.But value for
Nhas to correspond to a row in the color table.
{\colortbl;
\red0\green70\blue173; N=1
\red182\green0\blue20; N=2
\red169\green169\blue169; ...
\red222\green235\blue247;
\red204\green121\blue167;
\red255\green255\blue0; N=6 which here is yellow
\red204\green121\blue167;
\red0\green255\blue0; N=8 which here is green
\red139\green0\blue0; N=9 which is what R gives for col2rgb("darkred")
}
- And you can't specify an arbitrary color to use for highlighting. Based on http://www.biblioscape.com/rtf15_spec.htm, the colors are:
| Value | Description |
| 1 | Black |
| 2 | Blue |
| 3 | Cyan |
| 4 | Green |
| 5 | Magenta |
| 6 | Red |
| 7 | Yellow |
| 8 | Unused |
| 9 | Dark Blue |
| 10 | Dark Cyan |
| 11 | Dark Green |
| 12 | Dark Magenta |
| 13 | Dark Red |
| 14 | Dark Yellow |
| 15 | Dark Gray |
| 16 | Light Gray |
- The secret recipe is then
\highlightNwill highlight text using the Nth color in the color table so long as the Nth color in the color table corresponds to the RGB of one of the colors above.
Jan 3, 2023
I am having all kinds of difficulties with a Makefile on Windows. I have this:
# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe
ifeq ($(OS), Windows_NT)
R_CMD := rcmd.exe
R_SCRIPT := rscript.exe
SHELL = c:\scoop\shims\bash.exe
SCPEXE := c:\scoop\apps\gow\current\bin\pscp.exe # Requests password :(
SSHEXE := c:\scoop\apps\gow\current\bin\plink.exe # Requests password :(
endif
and this target:
.PHONY: model
model: /path/to/data.rds
/path/to/data.rds: src/a.R data/b.rds
$(SCPEXE) src/a.R weigand@server:/path/to/putit
$(SSHEXE) weigand@server 'cd /path/to/putit/; R CMD BATCH a.R'
The problems with the above is that the call to SCP requires a password even though I
otherwise have password-less login setup. And the call to SSH needs single quotes
as far as I can tell and it doesn't understand R CMD BATCH even though my intent
is that command is run on the server.
Part of the problem may be the clashing between Scoop's use of pscp.exe and
plink.exe versus OpenSSH's scp.exe and ssh.exe in c:/Windows/System32/OpenSSH.
Dec. 28, 2022
I searched for how to type accented characters in Emacs and got https://irreal.org/blog/?p=4945 and https://irreal.org/blog/?p=4254.
One general way to enter accented characters is C-x 8 then a one- or
two-character sequence. For example the character "ñ" is entered with
C-x 8 ~ n. The character "á" is entered with C-x 8 ' a. There are
also special characters like "÷" which is C-x 8 / /. To see what is
available, do C-x 8 C-h.
The blog author prefers "switch[ing] into TeX input mode and then
using the TeX code for the character." This can be done by typing
C-\ to run the command toggle-input-method. It is described as:
Enable or disable multilingual text input method for the current buffer.
If one sets their default input method to TeX (see below) then the
character "β" can be entered simply with C-\ \beta. To turn off TeX
input mode one enters C-\ again. As far as I can tell, the only way
to know if you are in TeX input mode is the buffer mode line starts
with \. (I suppose it depends on how the mode line is set up.)
I haven't experimented with this but I imagine in many buffers one can stay in TeX input mode rather then toggling it on or off.
Here is a link to a 22 page PDF of LaTeX symbols: https://www3.nd.edu/~nmark/UsefulFacts/LaTeX_symbols.pdf
As an example, it looks like the section symbol "§" is just \S.
(One can customize default-input-method to TeX with Alt-x
customize-variable default-input-method. I am curious about other
input modes but for now I'll let it go...)
Oct. 14, 2022
On Windows I miss the Bash equivalent of emacs myfile &. In CMD.exe I can do
start /b c:\scoop\shims\emacs.exe myfile
Oct. 12, 2022
I re-found a video on YouTube (https://youtu.be/D8hEirGijmo) by
"tdhock" (Toby Hocking?) that describes how to set the Emacs Speaks
Statistics (ESS) variable
ess-r-package-auto-enable-namespaced-evaluation. I customized this
to nil so that evaluation of a function, paragraph, buffer etc. is
not into the package namespace when ESS detects you are in a
package. I think this is more straightforward since I'm used to
evaluating an R buffer or parts of an R buffer into the global
environment during testing or development.
Oct. 6, 2022
SSH file and directory permissions
I was trying to work out file sharing on a Linux machine that a colleague and I both have access to and I changed my home directory permissions there to 777 to see if she could write a file in one of my folders. (The server was protected by the corporate firewall and users were trused so security wasn't a concern.)
A few days later I noticed that passwordless login to the serverwasn't
working. I didn't make the connection because it had been a few days
and I spent a lot of time trying to understand the problem, check
permissions on files in my varoius .ssh directories, generate new
keys, etc.
It turns out that the fix was to change permissions on that server to 705 (i.e., read and execute for others).
Apparently, for passwordless SSH to work, the user's home directory can't be writable by world. (One server I use a lot has 755 for my home directory.)
ASCII and Emacs
I had a file on Linux that had some non-ASCII characters. (Windows was involved.)
In R one can do:
library(tools)
showNonASCIIfile("filename.txt")
This gave me a printout of
2: <ef><bb><bf>and this and that...
So R showed me the non-ASCII but I couldn't delete those characters in Emacs.
From a Web search and landing on https://www.gnu.org/software/emacs/manual/html_node/emacs/Coding-Systems.html, I learned that I could call ~revert-buffer-with-coding-system~ and choose ~raw-text~. This displayed
\357\273\277and this and that...
I could then delete these three characters/entities/things and the file
passes the showNonASCIIfile test in R.
Sept. 21, 2022
In Emacs on Windows the command Alt-x shell starts a CMD.EXE
shell. This is fine/nice. I learned from
http://www.hanselman.com/blog/a-better-prompt-for-cmdexe-or-cool-prompt-environment-variables-and-a-nice-transparent-multiprompt
that I can customize my prompt. I'm now using
set prompt=$P$_$+cmd$G$S
which gives the drive and path ($P), a newline and carriage return
($_), the pushd/popd depth (+), the text cmd (cmd), then ">"
($G), then a space ($S).
I set the prompt variable by typing "environmental variables" in the search bar and then added a new user variable.
Also, thanks to tips in https://stackoverflow.com/questions/20530996/aliases-in-windows-command-prompt, I have now pseudo aliases in the form of batch files that are saved in a directory that is already on my path. An example is:
@echo off
echo.
cd %USERPROFILE%\Fossils
dir /ad /b /n /w
This takes me to my Fossils directory and lists the directories
therein. The first option of /ad is the most useful because it lists
directories only. (Running fossils.bat in PowerShell works but the
dir command there is an alias for Get-ChildItem and my options are
ignored.
June 16, 2022
The generic preprocessor called GPP http://files.nothingisreal.com/software/gpp/gpp.html allows users to define macros in various syntaxes including something similar to the C preprocessor or to LaTeX macros. As I use it, the command is
gpp -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile
or on Windows:
gpp.exe -o outfile +n -U "\\" "" "{" "}{" "}" "{" "}" "#" "" +c "\n%" "\n" infile
I could run this within a Makefile on Windows and Linux but not from Windows Terminal. I realized it works from the CMD shell (and GNU Make is using CMD by default) but not when using PowerShell which must require escaping or quoting of the arguments. (The arguments are probably very, very PowerShell unfriendly with double quotes and double backslashes.)
I don't have a work around but at least now I know why there was a problem running it from the (Windows Terminal) command line.
May 18, 2022
This is how I made something of a platform independent Makefile
to run either rcmd.exe on Windows or R CMD on Linux.
# This file assumes that when called from Windows
# there will be a bash.exe at c:\scoop\shims\bash.exe
ifeq ($(OS), Windows_NT)
R_CMD := rcmd.exe
SHELL = c:\scoop\shims\bash.exe
else
R_CMD := R CMD
endif
May 6, 2022
This is how I used scp to copy a directory from Linux to Windows:
scp -r username@server.com:/file/to/syncdir .
^ syncdir created locally here
May 5, 2022
How I updated my Pi to Python 3.10.4: https://stackoverflow.com/questions/64718274/how-to-update-python-in-raspberry-pi
sudo apt-get install -y build-essential tk-dev \
libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev \
libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev \
liblzma-dev zlib1g-dev libffi-dev
wget https://www.python.org/ftp/python/3.10.4/Python-3.10.4.tgz
sudo tar zxf Python-3.10.4.tgz
cd Python-3.10.4.tgz
sudo ./configure --enable-optimizations
sudo make -j4
sudo make altinstall
April 12, 2022
I had to work with text files with a non-ASCII character representing
a superscript 2. When checking these into Fossil,
Fossil warned of invalid UTF-8. I think the problem was that the file
had the superscript 2 in LATIN-1. I told Fossil to convert the file and
opened the converted file and original file in Emacs and turned on
hexl-mode. I forgot to finish this post but I think the difference was between
b2 for LATIN-1 and 00b2 for UTF-8.
March 7, 2022
I like the "generic preprocessor" program called gpp
(https://github.com/logological/gpp/) for use as a general macro
processor. (See https://en.wikipedia.org/wiki/Preprocessor and some
links therein.) On Linux gpp is easy to install on with the usual
./configure, make, etc.
Below is how I got it working on Windows without using ./configure or
make. (I have Make but I don't think I have sh needed by configure. I'm
not sure, but it wasn't hard to compile "by hand".)
These are the preliminary steps:
- Download the file (https://github.com/logological/gpp/releases/download/2.27/gpp-2.27.tar.bz2)
- Type
bunzip2.exe ./gpp-2.27.tar.bz2 - Type
tar.exe -xvf gpp-2.27.tarand change directories intogpp-2.27/src - Create a file called
stephen.hwith the following:
/* Header created by Stephen Weigand to avoid configure on Windows */
#define PACKAGE "gpp"
#define PACKAGE_BUGREPORT "tristan@logological.org"
#define PACKAGE_NAME "GPP"
#define PACKAGE_STRING "GPP 2.27"
#define PACKAGE_TARNAME "gpp"
#define PACKAGE_URL ""
#define PACKAGE_VERSION "2.27"
#define HAVE_STRDUP 1
#define HAVE_STRCASECMP 1
#define HAVE_FNMATCH_H 0
The HAVE_<something> reflects that my GCC can find the C functions
strdup and strcasecp but not fnmatch. Now edit gpp.c as
follows.
Add the include for stephen.h near the top. Note the quotes around
stephen.h which says look in the current directory for the include
file.
/* Added by Stephen Weigand March 7, 2022 */
# include "stephen.h"
Now make a few changes based on not having fnmatch. Lines 1880-1902
need to be like this:
if (SpliceInfix(buf, pos1, pos2, "=~", &spl1, &spl2)) {
#if ! HAVE_FNMATCH_H
bug("globbing support has not been compiled in");
#else <--- I had to convert this from #endif to #else
if (!DoArithmEval(buf, pos1, spl1, &result1)
|| !DoArithmEval(buf, spl2, pos2, &result2)) {
char *str1, *str2;
/* revert to string comparison */
while ((pos1 < spl1) && isWhite(buf[spl1 - 1]))
spl1--;
while ((pos2 > spl2) && isWhite(buf[spl2]))
spl2++;
str1 = strdup(buf + pos1);
str1[spl1 - pos1] = '\0';
str2 = strdup(buf + spl2);
str2[pos2 - spl2] = '\0';
*result = (fnmatch(str2, str1, 0) == 0);
free(str1);
free(str2);
} else
*result = (result1 == result2);
return 1;
#endif <--- This was moved down to here.
}
With this change (which may be a bug fix, really) I could then do the old-fashioned way of compiling the program.
gcc.exe -Wall -Os -o gpp.exe gpp.c
Then I moved it to somewhere in my path.
February 22, 2022
Commands for setting up a new Fossil
This is my set of commands for starting a Fossil project. I create the
repository, set my password, add another user (otheruser) and set
their password, and then give them administrator (a) capabilities.
cd /path/to/fossils
fossil new myrepo.fossil
fossil user password myname mypAssW0rd -R myrepo.fossil
fossil user new otheruser otheruser@example.com the1rPasSw0rd -R myrepo.fossil
fossil user capabilities otheruser a -R myrepo.fossil
Then I start the UI and change a few settings:
fossil ui myrepo.fossil
Once the UI opens, click on Admin then Configuration and set the following:
Project name. I keep this short like a filename since it displays in my bash prompt.
Project description. This is longer.
Index page. I use
/dir?ci=tipso that when I click on Home in the UI it takes me to the list of files.
I also add a deposit target in a Makefile which will "deposit" and unpack
the repository to a location on disk that colleagues who don't use Fossil
can look at.
MYPATH := /path/to/shared/directory
.PHONY: deposit
deposit:
fossil zip trunk source-code-repository.zip --name source-code-repository
mv source-code-repository.zip $(MYPATH)
cd $(MYPATH)/; unzip -o source-code-repository.zip
rm -f $(MYPATH)/source-code-repository.zip
echo "WARNING: This is a read-only copy of files under Fossil version control for this project." > $(MYPATH)/source-code-repository/WARNING.txt
February 14, 2022
Building R packages
I'm building my R package on my Raspberry Pi and did this
cd ~/Subversions/myproject-branches/
R CMD build mydir-wip-branch
But I needed texi2dvi and so did:
sudo apt install texinfo
But I needed texlive so I did:
sudo apt-install texlive
But I needed inconsolata.sty so I did:
sudo apt-install texlive-fonts-extra
And then I got R CMD build and R CMD check to both be happy and
the latter command even got my vignette built.
Subversion branches
I have a checkout in ~/Subversions/myproject and made a branch like this.
First I used the svn copy command
svn copy ^/mypkg/mydir ^/mypkg/mydir-wip-branch -m "Making a work in progress branch of mydir"
Here the ^ means the "root" part of the repository. Then I did
mkdir ~/Subversions/myproject-branches
cd ~/Subversions/myproject-branches
svn checkout svn+ssh://username@example.com/svnroot/myproject/mypkg/mydir-wip-branch
I now have a work-in-progress branch of a subdirectory of the repository. I will need to merge it someday.
January 28, 2022
Background
This is from https://everyday.codes/linux/how-passwordless-ssh-login-works/
- I generate a private key which is a long sequence of bits/bytes/characters.
- A public is another sequence that is derived from this.
- I can get the public from the private but with the public I can't go back to the private.
- My public key is available for everyone and stored on my PC and the server
Using the public key, you can encrypt (or sign) any message, and it will only be possible to decrypt it using the private key. In other words, anyone with your public key can send you encrypted messages that only you will be able to read.
The server can authenticate me like this:
- Encrypt a message using the public key I've stored on the server and send the message to my PC
- My PC decrypts the message using my private key and sends it back
- If the server sees I got it right then I am authenticated.
I'm live tiloring my steps to set up passwordless login for R-forge.
This is a good resource it seeks https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement
I will have a private key and a public key. I place my public key on the R-forge server. When I ssh to r-forge Microsoft says:
When using key authentication with an SSH server, the SSH server and client compare the public key for a user name provided against the private key. If the server-side public key cannot be validated against the client-side private key, authentication fails.
PS> ssh-keygen.exe -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\USERNAME/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\USERNAME/.ssh/id_ed25519.
Your public key has been saved in C:\Users\USERNAME/.ssh/id_ed25519.pub.
And:
Now you have a public/private Ed25519 key pair in the location specified.
Directory: C:\Users\WEIGAND\.ssh
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 1/28/2022 9:38 PM 464 id_ed25519
-a---- 1/28/2022 9:38 PM 104 id_ed25519.pub
Remember that private key files are the equivalent of a password should be protected the same way you protect your password. To help with that, use ssh-agent to securely store the private keys within a Windows security context, associated with your Windows login. To do that, start the ssh-agent service as Administrator and use ssh-add to store the private key.
December 16, 2021
I forgot how to control what applications start up automatically in Windows 10. A while back I set it so PowerShell started up automatically and disabled the same for Skype. But I needed to disable Microsoft Teams from starting up automatically. Here's how I did it:
I typed "Startup" in the search bar. This opened the Settings app and took me to the Startup setting. Then I disabled Teams.
Obviously this was trivial but I lived with Teams starting up automatically for a couple weeks so I added a note about it to cement it in my memory. (I was remembering it as complicated and involving a start-up folder and shortcuts. I guess not.)
December 1, 2021
To fully detach a package in R I do this:
R> detach("package:mypackage", character.only = TRUE, unload = TRUE)
One thing that seems to be a pain is if you "attach" something like
ggplot2 (i.e., do any of library(ggplot2), library("ggplot2"),
require(ggplot2), etc.) then your session will get a lot of packages
loaded "via a namespace" but not "attached"):
> sessionInfo() # after library(ggplot2)
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
<snip>
attached base packages:
[1] stats graphics utils datasets grDevices methods base
other attached packages:
[1] ggplot2_3.3.5
loaded via a namespace (and not attached):
[1] fansi_0.4.2 withr_2.4.2 assertthat_0.2.1 dplyr_1.0.7
[5] crayon_1.4.1 utf8_1.2.2 grid_4.0.3 R6_2.5.1
[9] DBI_1.1.1 lifecycle_1.0.1 gtable_0.3.0 magrittr_2.0.1
[13] scales_1.1.1 pillar_1.6.3 rlang_0.4.11 generics_0.1.0
[17] vctrs_0.3.8 ellipsis_0.3.2 splines_4.0.3 tools_4.0.3
[21] glue_1.4.2 purrr_0.3.4 munsell_0.5.0 compiler_4.0.3
[25] pkgconfig_2.0.3 colorspace_2.0-2 tidyselect_1.1.1 tibble_3.1.2
There are still 28 packages "loaded via a namespace (and not
attached)" after I detach ggplot2. (But I'm not sure what that means
exactly.) The way to get rid of these hangers-on is via the
unloadNamespace() function.
But they seem to need to be detached based on what's at the top of the dependency tree. Here is an illustration of the problem
> unloadNamespace("fansi") # try to unload first in the list
Error in unloadNamespace("fansi") :
namespace 'fansi' is imported by 'pillar' so cannot be unloaded
> unloadNamespace("pillar") # OK, I'll unload 'pillar' first.
Error in unloadNamespace("pillar") :
namespace 'pillar' is imported by 'tibble', 'dplyr' so cannot be unloaded
> unloadNamespace("tibble") # OK, unload 'tibble'.
Error in unloadNamespace("tibble") :
namespace 'tibble' is imported by 'dplyr' so cannot be unloaded
> unloadNamespace("dplyr") # OK, here's an order that works.
> unloadNamespace("tibble")
> unloadNamespace("pillar")
> unloadNamespace("fansi")
I think there are packages that help you detach and unload packages but I am liking the idea of not attaching ggplot2 in the first place.
November 29, 2021
UTF-8 in C
Here is short C program that works for me on Linux and Windows to print out
Hello Aβ42:
#include <stdio.h>
int main(){
printf("Hello, A\u03b242\n");
return 0;
}
On Windows I compile it with gcc.exe -Wall -o hello.exe .\hello.c
using GCC 8.3.0 and get no errors or warnings. On Linux I am using
GCC 4.8.5 and get
hello.c: In function 'main':
hello.c:4:10: warning: universal character names are only valid in C++ and C99 [enabled by default]
printf("Hello, A\u03b242\n");
^
I can use GCC 8.3.1 after typing scl enable devtoolset-8 bash and then
the warning goes away. But if I add the GCC flag -std=c89 then the warning
comes back (which makes sense).
UTF-8 in Windows Terminal
When using Windows Terminal, I clicked on the down arrow in the tab bar and clicked on Settings which opened up a JSON settings file in Notepad. Then I added two arguments to the PowerShell command to tell the terminal to output in UTF-8.
// Make changes here to the powershell.exe profile.
"guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
"name": "Windows PowerShell",
"commandline": "powershell.exe -Noexit -Command chcp 65001",
"hidden": false ^^^^^^^^^^^^^^^^^^^^^^^^^^^
July 22, 2021
From my Windows 10 desktop I can clone a repo on a Linux machine with
fossil set ssh-command "ssh"
fossil clone -v ssh://weigand@server.com/./fossils/filename.fossil filename.fossil
This is good except I should be able to specify the SSH command using -c ssh. Also, I wonder
why it doesn't work with plink.exe which allows commands like this
plink.exe -ssh weigand@server.com ls
July 21, 2021
When exporting in Org mode to prevent underscores from being treated as subscripts, I did this:
#+OPTIONS: ^:{}
July 1, 2021
From https://endlessparentheses.com/debugging-emacs-lisp-part-1-earn-your-independence.html I learned several things which may help with my RTF backend for Org mode.
The command
find-functionwill take me to the definition of a function. If I am inside a function, I can invoke it and it will ask me what function to go to with the default being the one I'm in. It will then take me to where in the source that function is defined and put point at the start of the function. By default it doesn't look like their are key bindings. I might want to make an alias so I only need to doM-x ff.The command
describe-functionis bound toC-h for<f1> fand shows the full documentation of the function. If point is in (or next two?) a function, issuing the command will set the default to the function point is in/near and so just press enter to see the help for the function.I just realized that
<f1>is the help function. So naturally<f1> fwill give the help for a function,<f1> vthe help for a variable, etc.The Edebug part is that I either find or describe the function and then do
C-u C-M-x. This iseval-defunwith a prefix according to the manual. My only problem is I can't get this to work except for a function that is buggy.
June 15, 2021
In R on Windows and Linux, source("clipboard") will source the
contents of the clipboard. On MacOS the equivalent is
source(pipe("pbpaste")).
June 13, 2021
From https://devblogs.microsoft.com/scripting/powertip-use-powershell-to-display-windows-path/ I can get my path on PowerShell via
$env:path -split ";"
I turned this into a function:
function path() {
$env:path -split ";"
}
June 12, 2021
In Emacs I can get an interactive SQLite session via M-x sql-sqlite and
then I just have to specify a data base filename.
May 3, 2021
An easy way to get semi-transparent colors in R is to use adjustcolor:
barplot(c(1, 2, 3), col = adjustcolor("firebrick", 0.5),
main = "Firebrick bar plot with alpha=0.5")
April 26, 2021
Create a simple SQLite data base:
sqlite3 test.sqlite3
sqlite> .databases -- Show the databases
main: /home/weigand/fossils/tilor/test.sqlite3
ATTACH DATABASE 'test.sqlite3' as 'project';
.databases
main: /home/weigand/fossils/tilor/test.sqlite3
project: /home/weigand/fossils/tilor/test.sqlite3
DROP TABLE IF EXISTS person;
CREATE TABLE IF NOT EXISTS project.person(
id INT PRIMARY KEY NOT NULL,
date_birth CHAR(10),
sex CHAR(2) CHECK(sex == 'M' OR sex == 'F')
);
INSERT INTO project.person VALUES(1, '2010-01-01', 'M');
INSERT INTO project.person VALUES(2, '2011-11-11', 'F');
SELECT * FROM main.person;
1|2010-01-01|M
2|2011-11-11|F
April 14, 2021
When using smbclient on Linux to put a file on a Windows share I wasn't
specifying the local file path correctly. I learned about the lcd command
in smbclient to set the local current directory and then I could put
the filename without a full or relative path. Like this:
PASSWD=`grep -woi -m 1 '^machine mymachine login .* password .*$' ~/.netrc | cut -d ' ' -f 6`
smbclient //share.example.com/DirectoryX \
-W MYWORKGROUP \
-E \
-U $USER $PASSWD \
--command 'cd DirA/DirB ;
lcd /local/working/directory ;
put localfile.csv ;
exit'
March 17, 2021
This is how to get interval estimates for random effects from a linear
mixed effects model fit with lme4::lmer in R using the unexported
lme4:::asDf0 function.
library(lme4)
library(lattice)
fit <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)
dotplot(ranef(fit)) # Caterpillar plot shows intervals
out <- lme4:::asDf0(ranef(fit), "Subject")
out$lcl <- out$values - 1.96 * out$se
out$ucl <- out$values + 1.96 * out$se
out[c(1:2, 35:36), ]
values ind .nn se
1 2.259 (Intercept) 308 12.1
2 -40.399 (Intercept) 309 12.1
35 -0.988 Days 371 2.3
36 1.284 Days 372 2.3
March 10, 2021
On Linux & runs a command in the background. For example:
emacs README.md &
From PowerShell, the equivalent seems to be:
Start-Process -NoNewWindow emacs.exe README.md
The above is from a Web search where I landed on https://ariefbayu.xyz/run-background-command-in-powershell-8ea86436684e. That post shows how to make a general run-it-in-the-background command.
function bg() {
Start-Process -NoNewWindow @args
}
And then I might use:
bg emacs.exe README.md
March 3, 2021
The Journal NeuroImage uses an open source font called Charis
SIL. This can be downloaded and on
my department's Linux system, I just have to put the *.ttf files in
~/.fonts and then I can create figures in "native" NeuroImage style:
library(ggplot2)
ggplot(cars, aes(x = speed, y = dist)) +
geom_point() +
labs(title = "Scatter plot in Charis SIL font used by NeuroImage",
x = "Speed (miles per hour)",
y = "Distance (feet)") +
theme_light(base_family = "CharisSIL")
December 12, 2020
This is how I used Dropbox's Python API to upload my son's piano recital video. (My daughter's video was smaller and uploaded fine through the Dropbox web interface before timing out.)
My guide was Dropbox for Python Developers.
Set-up a Python virtual environment and install
dropboxpython3.7 -m venv ~/venvs/recitalsource ~/venvs/recital/bin/activatepip install --upgrade pippip install dropbox
Linked my account in Dropbox to an app
- Go to the app console
- Created and named the app "recital". To keep it simple I gave the app read and write permissions across my whole Dropbox account.
- Generated a short-lived and quite long access token (which lasts for four hours)
Wrote Python code to create a dropbox instance, read the file into an object, and upload the object
import dropbox
from dropbox.files import WriteMode
dbx = dropbox.Dropbox('AVerySpecial138CharacterAccessToken')
with open('/home/weigand/recital/piano.m4v', 'rb') as f:
data = f.read()
dbx.files_upload(data, # My file
'/recital-folder/piano.m4v', # Full path of destination
mode=WriteMode('overwrite')) # Want to overwrite
This obviously isn't "professional grade" Python. It works but doesn't handle errors and hard codes the filename.
December 11, 2020
This is how I managed to print to a WiFi-enabled printer from my Raspberry Pi.
I first went to the support page the Brother MFC-J870DW printer and selected Downloads and then did the following:
- Select OS Family as Linux
- Select OS Version as Linux (deb)
- Clicked on the Driver Install Tool
- Agreed to the End User Licence Agreement (EULA) and clicked Download
- Saved the file
linux-brprinter-installer-2.2.2-1.gzto my~/Downloadsdirectory
In the shell I did:
cd ~/Downloads
gunzip linux-brprinter-installer-2.2.2-1.gz
sudo bash linux-brprinter-installer-2.2.2-1
Input model name ->MFC-J870DW
You are going to install following packages.
mfcj870dwlpr-3.0.0-1.i386.deb
mfcj870dwcupswrapper-3.0.0-1.i386.deb
brscan4-0.4.9-1.i386.deb
brscan-skey-0.3.1-1.i386.deb
OK? [y/N] ->
... <say yes to a few additional things>
Will you specify the Device URI? [Y/n] ->y
select the number of destination Device URI. ->
December 10, 2020
CSVY format
The data.table package in R has a fast way to read CSV files in the
form of data.table::fread and as of version 1.12.4 it has support
for reading CSVY files. The CSVY format is a text format where there
is a YAML header at the top of the file which defines the data
"schema" and below the header are the data lines. Here is the general
idea of a CSVY file.
schema:
fields:
- name: name
type: string
- name: age
type: integer
- name: date
type: number
name, age, date
Maria, 44, 2010-01-01
Roberto, 43, 2009-01-01
Tukey HSD
A colleague likes using Tukey's honest significant difference for pairwise
comparisons. I never use it but maybe I should. The example for stats::TukeyHSD
in R is as follows.
R> summary(fm1 <- aov(breaks ~ wool + tension, data = warpbreaks))
Df Sum Sq Mean Sq F value Pr(>F)
wool 1 451 451 3.34 0.0736 .
tension 2 2034 1017 7.54 0.0014 **
Residuals 50 6748 135
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
R> TukeyHSD(fm1, "tension", ordered = TRUE) # Still need to understand 'ordered'
Tukey multiple comparisons of means
95% family-wise confidence level
factor levels have been ordered
Fit: aov(formula = breaks ~ wool + tension, data = warpbreaks)
$tension
diff lwr upr p adj
M-H 4.7 -4.63 14 0.45
L-H 14.7 5.37 24 0.00
L-M 10.0 0.65 19 0.03
I understand penalization as a better way to handle multiple comparisons but this is quick and easy. It would be interesting to better understand Tukey HSD and perhaps see if I can borrow the automatically generated pairwise comparisons code.
December 4, 2020
Indent a region five spaces in Emacs
C-u 5 C-x TAB is what I wanted.
R's base and grid graphics in one plot
This is one way to combine base and grid graphics in one plot:
library(ggplot2)
library(ggplotify)
library(gridExtra)
## Using `::` in a few places to indicate the package
p1 <- ggplotify::as.grob(~plot(dist ~ speed, # Notice the unusual `~`
data = cars,
main = "Base graphics scatter plot"))
p2 <- ggplot(cars) +
aes(x = speed, y = dist) +
geom_point() +
ggtitle("ggplot scatter plot")
gridExtra::grid.arrange(p1, p2, nrow = 1)
Org and TODO states
I had to remind myself how to change the TODO states in an Org file. Here is an example Org file with the information.
#+TODO: ASAP SOON WAITING SOMEDAY | DONE CANCELED
* ASAP Learn about Org TODO states
- It seems easiest to put the TODO state sequence in the file itself
by having a ~#+TODO:#~ line at the top of the file.
- See [[https://orgmode.org/org.html#Per_002dfile-keywords][per-file TODO states]]
- The vertical bar separates not-done versus done states.
- If you change your TODO states in a file go to the line with the states
and do ~C-c C-c~ and you'll see "Local setup has been refreshed."
- As a reminder, the TODO states can be advanced via ~C-c C-t~.
November 28, 2020
I know of two disk space programs on Linux:
dusays it is to estimate file space usagedfsays it is to report file system disk space usage but I found it's better to think of it as reporting free space.
I think df is more what I want since it shows
If no file name is given, the space available on all currently mounted file systems is shown
On this Raspberry Pi, df -h gives the following:
Filesystem Size Used Avail Use% Mounted on
/dev/root 14G 5.4G 7.1G 43% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 464M 60M 404M 13% /dev/shm
tmpfs 464M 6.3M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
/dev/mmcblk0p6 71M 23M 49M 32% /boot
tmpfs 93M 4.0K 93M 1% /run/user/1000
/dev/mmcblk0p5 30M 398K 28M 2% /media/pi/SETTINGS
A Web search gives me this nice link from Adafruit:
https://learn.adafruit.com/an-illustrated-shell-command-primer/checking-file-space-usage-du-and-df
This led to free for memory usage. On the Pi, free -h gives (with
some whitespace removed):
total used free shared buff/cache available
Mem: 926M 392M 47M 67M 486M 487M
Swap: 99M 5.8M 94M
November 25, 2020
From the Linux Documentation Project (via Stack Exchange):
/etc/skel/ The default files for each new user are stored in this directory. Each time a new user is added, these skeleton files are copied into their home directory. An average system would have: .alias, .bash_profile, .bashrc and .cshrc files. Other files are left up to the system administrator.
On the Raspberry Pi that I am working on righ now, I see
.bash_logout, .bashrc, and .profile.
And per a comment in this .profile, Bash reads ~/.profile if there
is no ~/.bash_profile and no ~/.bash_login so I don't want to have
the latter two files. At least on this Pi the process is:
Read
~/.profilewhich has my environmental variables~/.profilesources~/.bashrcif it exists. The latter has local Bash configurations.~/.bashrcsources~/.bash_aliasesif it exists. Having aliases in a separate file is a "separation of concerns" idea.
November 5, 2020
Here are two LaTeX-like macros for gpp that define two levels of
RTF list.
This macro has the bullet flush left with the margin and the text indented as a block by 360/1440 twips or 1/4 inch.
\define{\bulletpoint}{
\paragraph{\fi-360\li360\bullet\tab #1}
}
This second-level bullet has a "white bullet" (specified in unicode decimal format) indented 1/4 inch and the rest of the body indented 1/4 inch more.
\define{\subbulletpoint}{
\paragraph{\fi-360\li720{\u9702-}\tab #1}
}
Both macros depend on \paragraph{} so that the bullet points
inherit the paragraph spacing and other formatting. An example
paragraph macro is:
\define{\paragraph}{{\pard \ql \sa60 \sb60 \f0 \fs24 \kerning12
#1
\par}}
October 16, 2020
SAS has special missing data codes which represent
a type of numeric missing value that enables you to represent different categories of missing data by using the letters A-Z or an underscore.
R doesn't have this built in. But here is prototype of how to have
missing data codes. The data frame has three rows and two
variables. Then first variable is an integer ID variable and the
second is trails_a which holds the time in seconds a person takes on
part A of a test called Trails Making Test. If the person does not
have a score on the test, we can record the reason.
I don't think data.frame can generate this object so I use
structure with row.names and class attributes. The trails_a variable
itself is a list with two elements each of length 3. In a way this is
like an embedded data frame.
d <- structure(list(id = c(11L, 22L, 33L),
trails_a = structure(list(score = c(50L,
NA_integer_,
NA_integer_),
reason = c(NA_character_,
"Exceeded time limit",
"Too severe to test")),
class = "trailsList"),
random = runif(3)),
row.names = c(NA, -3L),
class = "data.frame")
This doesn't print properly (and gives a "corrupt data frame" warning) unless we
create a format method for our trailsList class.
format.trailsList <- function(x, ...){
ifelse(is.na(x$score),
sprintf("NA (%s)", x$reason),
sprintf("%ds", x$score))
}
But with a nice format method we get this:
id trails_a random
1 11 50s 0.061
2 22 NA (Exceeded time limit) 0.664
3 33 NA (Too severe to test) 0.826
To use a data frame with a trailsList object we need to define a
number of other methods including operations like == for subsetting.
I don't know all that is involved. But an example to look at would be
the methods for Surv in the survival package.
Conclusion
I don't think most analysts would want to deal with this complexity.
It's more straightforward and understandable to just use two columns
in the data frame: the score and if no score, the reason for NA.
Postscript
Having a variable in a data frame that is of POSIXlt class is
an "established" case that is very similar. This is a data frame
with two observations and three variables clinic, date, and
ldate.
d <- data.frame(clinic = 1:2,
date = as.Date(c("2010-01-01", "2020-01-05")))
d$ldate <- as.POSIXlt(d$date)
dput(d)
structure(list(id = 1:2,
date = structure(c(14610, 18266), class = "Date"),
ldate = structure(list(sec = c(0, 0),
min = c(0L, 0L),
hour = c(0L, 0L),
mday = c(1L, 5L),
mon = c(0L, 0L),
year = c(110L, 120L),
wday = c(5L, 0L),
yday = c(0L, 4L),
isdst = c(0L, 0L)),
class = c("POSIXlt", "POSIXt"),
tzone = "UTC"),
row.names = c(NA, -2L),
class = "data.frame")
September 21, 2020
PowerShell
I used to look down my nose at PowerShell but now I realize there are lots of shells that do a good job. I need to learn the equivalent of aliases in PowerShell to make navigating the Windows filesystem faster but for now if I navigate to a directory with a file that I want to open (say a Word document) then I can open quickly with
ii myfile.docx
Here, ii is a shortening of Invoke-Item.
September 15, 2020
Lattice graphics
A StackOverflow question asked about removing tickmarks on the right and top margins of a plot.
The context was
library(effects)
m <- lm(Fertility ~ ., data = swiss)
plot(allEffects(m), rug = FALSE)
And plot from the effects package didn't allow the user much
control. For example, the usual scales argument wasn't handled. This
was my solution:
trellis.par.set(axis.components = list(top = list(tck = 0),
right = list(tck = 0))
plot(allEffects(m), rug = FALSE)
Another thing I learned was to to control the font family easily.
Using trellis.par.set outside of a plot cal or using
par.settings inside a plot call, one of the elements is grid.pars.
See ?gpar.
str(get.gpar())
xyplot(dist ~ speed, data = cars,
par.settings = list(grid.pars = list(fontfamily = "Open Sans")))
August 27, 2020
Polymode
In a RMarkdown file edited with Emacs and Polymode, the command
polymode-eval-buffer-from-beg-to-point is bound to both M-n v <up> and
M-n v u.
The command polymode-eval-buffer is bound to M-n v b.
The command polymode-eval-region-or-chunk is bound to M-n v v.
August 26, 2020
Lattice graphics
- Axis tick marks on log scale with "normal" numbers:
list(log = 2, equispaced.log = FALSE) - Gridlines to match tick marks:
panel.grid(h = -1, v = -1)
August 20, 2020
The data.table package has fread which is great. One difference between
fread and read.csv is how a field like ,"", is handled. Here is an example
file:
"id","date1","date2","code"
"1","2019-01-01","","3"
"1","2019-01-01","2018-03-18","3"
Using read.csv with defaults gives:
R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE))
'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: chr "2019-01-01" "2019-01-01"
$ date2: chr "" "2018-03-18" <---- Empty string
$ code : int 3 3
Using na.strings = "" gives
R> str(read.csv("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: chr "2019-01-01" "2019-01-01"
$ date2: chr NA "2018-03-18" <---- Beter
$ code : int 3 3
But I do not know a way to get the same from fread:
R> str(fread("test-fread.csv", stringsAsFactors = FALSE, na.strings = ""))
Classes 'data.table' and 'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: chr "2019-01-01" "2019-01-01"
$ date2: chr "" "2018-03-18"
$ code : int 3 3
- attr(*, ".internal.selfref")=<externalptr>
By design, fread wants to be smart about allowing an NA field to
be data (e.g., the string "NA". It also wants to allow zero length strings.
I don't know if there is a way around this.
For example, this doesn't work:
R> str(fread("test-fread.csv", colClasses = c("date1" = "Date", "date2" = "Date")))
Classes 'data.table' and 'data.frame': 2 obs. of 4 variables:
$ id : int 1 1
$ date1: Date, format: "2019-01-01" "2019-01-01"
$ date2: chr "" "2018-03-18"
$ code : int 3 3
- attr(*, ".internal.selfref")=<externalptr>
Warning message:
Column 'date2' was requested to be 'Date' but fread encountered the following error:
character string is not in a standard unambiguous format
so the column has been left as type 'character'
This comes up with ADNI data and the PTDEMOG.csv. Maybe the root of the problem
is having everything quoted in the source CSV.