Shared Items folder

There is a folder you see primarily in OS/X Server called « /Shared Items ». It is the default location for sharepoints. If you create a subfolder (e.g., « SharedData ») there and share it, then when it mounts on the client, it will appear by default as « /Volumes/SharedData ». Furthermore, when you specify what to mount, you can use « afp://server.domain/SharedData ».

What I wanted to do was to place a lab data folder within a user's folder. For years, I had had it under /Shared Items, but it turns out that it is simply easier to deal with things in OS/X when they are in user folders than when they are elsewhere in the directory hierarchy.

I created a “data” user and moved by data hierarchy into its home folder: « ~data/Data ». I marked this folder as shared, and then tried mounting it remotely. When I tried « afp://server.domain/Data », it failed. When I tried « afp://server.domain/Users/data/Data », it mounted the server's /Users folder (but at least it allowed access to the Data folder as « /Volumes/Users/data/Data ». But this is far from satisfactory.

It took me quite a lot of searching online, but finally a tangential remark led me to success.

It turns out that each user is also allowed to have a « Shared Items » folder that operates more or less the same as the one in the root directory of the system. This is parallel to the user-level Applications folder that can be used to install apps privately (but very few apps actually use this for some reason). Another example is that there are both user-level and root-level Library folders: these are both in heavy and regular use.

I now made a folder « ~data/Shared Items » and moved Data under it. I also made a symlink at « ~data/Data » pointing at « ~data/Shared Items/Data », mostly so scripts could avoid the space in the name. I then shared « /Users/data/Shared Items/Data ». Now it worked the same as it used to when it was in the system « /Shared Items » folder, yet it would now allow me to deal with « data » as a user with a big home folder, which in certain situations is very handy.

This is a very useful trick, I believe.

As for the space in « Shared Items », I find it unbelievable that Apple would do such a thing. I suppose it's not terrible that users are allowed to have spaces in filenames, but for a major system component to have a space, well, that's just unbelievable. SharedItems would have been fine, or even Shared_Items, but « Shared Items »? Pfui. And the same for things like « Application Support » and « Contextual Menu Items » (and yet, they got it right for tons of other ones: ColorSync, PrivilegedHelperTools, QuickTimeStreaming, ...).

Waste less time (OS/X)

There is a very interesting application called WasteNoTime that is available for Safari and Chrome. This app allows you to create a schedule during which time limited or no access is allowed to certain web sites. I've found this to be a very valuable tool.
However, there are other web browsers around, and sometimes, even though it's time to get to work, the temptation to make just one more comment or to finish reading a very interesting article is too strong, and I end up firing up OmniWeb or Opera (which do not support WasteNoTime) in order to waste just a little more time. But this is pernicious, because “just a little more” can soon become hours. Oh ye of little will power.
Well, it would be great to have some kind of impediment that would at least make it less convenient to use those browsers (or certain other applications) during work hours.
So, here's how I decided to do it.
There is a command called “chmod” that can change permissions on files or folders. If certain permissions are not set on an application (e.g., /Applications/Opera.app), it cannot be executed. So my very simple idea is based on this script:

#!/bin/ksh
# Usage: lockBrowsers.ksh lock|unlock
# very simple locker for all web browsers that do not have any means of
# restricting time-wasting activities; to be called by cron. (Note that both
# Safari and Chrome have the WasteNoTime app, which is better than this.) We do
# allow some time wasting a couple of days per week (but not on weekends).

# Current schedule:
# Unlocked every day at 7PM
# Locked every day at 10PM
# Unlocked M F at 6AM

APP=/Applications
set -A Waster FireFox OmniWeb Opera

case "$1" in
lock ) m="a-x" ;;
unlock ) m="a+x" ;;
ls ) m=ls ;;
* ) print -u2 "Usage: $(basename $0) lock|unlock|ls" ; exit 1 ;;
esac

for (( i=0 ; i<${#Waster[*]} ; i++ )) ; do
if [[ $m == ls ]] ; then
ls -ld $APP/${Waster[i]}.app
else
chmod $m $APP/${Waster[i]}.app
fi
done

exit 0

This very simple script contains a list of applications, and if called with an argument “lock”, locks them by removing execute/search permission at the top level; if called with an argument “unlock”, unlocks them by restoring execute/search permission. NOTE: you must own the applications you want to use this with. For example, Safari and other system commands might be owned by root, and in that case, this method will fail.

I installed the script, called “lockBroswers.ksh” in the libexec folder in my home directory.

Next, using the command “crontab -e”, I set up the schedule I wanted:

0 19 * * * ~/libexec/lockBrowsers.ksh unlock

0 22 * * * ~/libexec/lockBrowsers.ksh lock

This lets me use those applications in the evening every day. You can read the crontab documentation if you want some other schedule, as you probably will.

BOX — A way to manage the Mountain Lion Inbox

This method involves creating one ordinary mailbox, called Hold, in the IMAP account of your choice, and a smart mailbox. The smart mailbox, called “Box” contains messages matching “any” of the following conditions: “Message is Unread”; “Message Has Flag Red/Orange/Yellow”; or “Message is in Mailbox “Inbox”. Do not include either Trash or Sent. Drag both Hold and Box to the bookmark bar. In View>Message Attributes, make sure that “Flags”, “Mailbox”, and “Date Received” are selected. Above the message list display, select “Sort by Flags ▾” and “Ascending”.

At this point, instead of viewing your Inbox, you should view Box. You will see unread messages and Inbox messages at the top, followed by all red-flag messages, then orange-flag messages, then yellow-flag messages. Within each level, you'll get the same default sorting by priority and date, oldest first. Note that messages flagged at a lower priority (Green-Grey) can be viewed via the standard Flagged bookmark (this bookmark should also be set up to sort by Flags  ▾, ascending). Normally, all messages in Hold will be flagged, but the Hold bookmark can be selected to see all current contents, flagged or not.

In addition to this, you should drag the root of your current mailbox archive hierarchy to the bookmark bar (in my case, this is currently named y2012). This should allow you to keep the mailboxes sidebar hidden most of the time.

When a message arrives, it will be at the top of the column. Here is a suggested sequence for dealing with a message in your Inbox.

• Delete it if you can, possibly after jotting down a quick reply (the rule is, less than two minutes).
• If you need to act on it, but it will take longer than two minutes, set a flag according to its priority. Then...
• If you're not sure where or whether you want to save it after you've dealt with it, drag it onto the Hold bookmark.
• If you already know where to save it, drag it onto your archive bookmark and down to the appropriate mailbox.

Once your Inbox has been cleared, then at your leisure, you can deal with your flagged messages. When you have finally dealt with one, set its flag to “None”. If the message is located in Hold, just delete it (or you can decide to archive it).

If you need to deal with a message on a certain date, drag it to iCal on that day, and consider setting an alert. If you tend not to look at iCal regularly, I recommend a utility called EmailMyCal from the App store: it can email you your agenda, including a mention of pending email, each day. You will also receive (OS/X ≥ Mountain Lion) alerts about that message, depending on your configuration.

OK, that's the system. You really need to keep unread/Inbox messages low, or you won't see the flagged messages.

BOX -- [NOT] a simple inbox management scheme

****IGNORE THIS****
Due to an extremely annoying lack in Mail.app, the critical “Date last viewed/Not in the last...” filter simply doesn't work. Sorry.

The Box: How to keep a clean Inbox (OSX Lion)

It is very common to use the Inbox as a To Do list or reminder system. Once a message has been dealt with, it is either deleted or archived into a folder somewhere. The problem is that sometimes you can get behind--sometimes very behind--and the Inbox gets filled with messages that you promise yourself that you will deal with.

Here is an approach to dealing with this problem using Smart Mailboxes and the seven Message Flags. I implemented it on Lion, but it will probably work on most OS/X versions.

HOW TO SET IT UP

First, although the underlying interpreter is apparently capable of dealing with complicated rule systems for Smart Mailboxes, the user interface is not. In particular, there is no direct way to deal with situations like (X and Y) or (A and B). The indirect way of dealing with this is to create a Smart Mailbox XY that contains all messages meeting criteria X and Y and a second one AB with messages meeting criteria A and B. Then the whole thing can be represented by a third Smart Mailbox XYAB meeting XY or AB. In order to make this less busy, we will create a Smart Mailbox Folder and call it “zzz” so it will sleep peacefully at the end of the list of Smart Mailboxes.

Second, we will create a whole set of individual Smart Mailboxes and move them into zzz. The list is:

• BoxToday (red)
• Box1day (orange)
• Box2day (yellow)
• Box3day (green)
• Box1week (blue)
• Box2week (purple)
• BoxAnyFlag (gray)
• BoxUnread
• BoxOldTMP

These will contain rules such that the message will be in the Smart Mailbox if it has been flagged with the color indicated and if has not been read in the number of days or weeks indicated (BoxToday just depends on their being a red flag). For example Box2day has two conditions: Date Last viewed is not in the last 2 days; Message has Flag Yellow. BoxAnyFlag will always apply if any flag is set and if the message has not been read in a month. BoxUnread applies to all unread messages. Note that I chose the lags (0-1-2-3 day, 1-2 week, 1 month) arbitrarily, based on the notion that you'll want to push things back a short time more often than a long time, and that a monthly review of pushed-away mails is about right. But these can be set to whatever you want—for example, 0-1-2-3-4-5-6 days plus weekly reviews of unread and _TMP would also be perfectly logical, and there are other possibilities as well. Note that the colors are assigned an increasing lag as a function of their position in the flag menu button (red to gray).

Next, create a Smart Mailbox called Box. Unlike the others, it will apply if *any* rule applies. The rules it contains are “Message is in Mailbox X”, where X is all of the above Smart Mailboxes, plus Inbox (the common inbox for all accounts). Do not move this Smart Mailbox into the zzz folder. However, you should drag it up into the shortcut row along with Inbox, Drafts, Sent, and so on. It will stick up there for easy access.

Create a regular mailbox called _TMP and put it somewhere convenient. This will be used to contain miscellaneous messages that will not be archived but that you must deal with in the future. It may be convenient to add this as a shortcut next to Box. Be careful not to let the _TMP folder contain messages without flags. If you delete the flag, also delete (or refile) the message. Note that the BoxOldTMP Smart Mailbox will include any message in _TMP that hasn't been read in a month or more.

Optionally go into the View>Message Attributes menubar item and make sure that Mailbox is checked. This is very useful when looking in the Box because it will show you where each message is currently stored. You might also check either Date Sent or Date Received; this will help you distinguish old, recycled messages from new ones. Also, it is critical that Flags is checked here.

HOW TO USE IT

Once you have the infrastructure set up, you should click on the Box shortcut. You will see all of your current Inbox contents plus probably a bunch of old unread messages, and perhaps some previously flagged message. You *must* go through these, weeding out stuff you don't want, filing things that have been dealt with, and flagging and filing everything else. At the end, you will have an empty Inbox and no unread messages anywhere. If you come across a message you want to deal with but not file, put it into the _TMP folder, that's what it's for.

Now, instead of looking in the Inbox for new mail, look at Box. When a message arrives in the Box, either:

• Deal with it, remove its flag if any, and delete it
• Deal with it, remove its flag if any, and file it
• Flag it with a (different) priority and file it

But never leave it in the Inbox. Note that when you flag a message with a Red flag, it will always show in the Box, regardless of whether it was filed. Similarly, after a day has passed, a message flagged with an Orange flag will show again, but will go away once you have looked at it again. Any message with a flag will show back up at least once a month.
It is also useful to add a few other shortcuts to the top of the window, for example, the current year's receipts folder (i.e., receipts2012). The shortcuts are especially useful in that in Lion, the mailbox hierarchy can remain hidden most of the time.

THE BUILT-IN FLAGGED FOLDER

There is a standard “Flagged” smart folder that you can use to look at all flagged messages if you forget where you put one. Note that this will show flagged messages that are in the trash, which is why you should remove the flag before deleting a previously flagged message.

AUTOMATIC FILING

In some cases, it is possible to file messages from certain recipients automatically with an incoming mail rule. This approach meshes extremely well with the Box approach. If you simply file such messages, then they will show up in the Box because they have not been read. However, if you “accidentally” read them (while browsing in Box) then they will not show up again. To help with this, you could also have the rule give them a flag, for example the Red or Orange flag, to make sure you see it.
USING iCal
In some cases, you don't want to just push back an email to some rough time in the future, you must deal with it by a specific but far-off deadline. The Box method is not for that. Instead, file the message and then drag the message to a date and time in iCal. This will make the reminder part of your regular calendar system.

Post-iDisk backups

Apple supplies a program called Backup.app, formerly available on to those who had mac.com memberships, but later available generally. This program was intended for backing up relatively small but critical information to the iDisk. It could also be used to back up to network drives on the LAN, and to drives attached directly to the Mac. However, the iDisk is now going away, to be replaced by iCloud. But there is a difference between the kind of backup done via iCloud and the kind done by Backup to the iDisk, in that the new iCloud backups are intended to include a very wide range of things, where Backup could be used to make very specific, possibly redundant backups of selected things only. I wanted to have something that could be used to replace the functionality of the iDisk as a place to store selected files, and as a destination for a Backup-like automated backup of selected elements.
I selected the free service offered by CloudSafe GmbH as the replacement iDisk. They offer 2 GB for free. Their site is very secure in that all access is via https, and all data stored there is highly encrypted and must be decrypted through the use of a lengthy key. Also, they offer WebDAV over https to the data.
The free CloudSafe accounts can have up to three WebDAV mountable remote drives, called “safes”, each with its own encryption key and access rules. For the purposes of backup, I created a safe called “Backup”.
In order to use the remote drive, you first have to use CloudSafe's dashboard to enable WebDAV on the safe. When you do this, the system will display two critical codes. The first code is part of the address used to access the drive, and is a 10-digit number, like « https://0123456789.webdav.cloudsafe.com/ ». The second code is used, along with the e-mail address you use to access your CloudSafe data online, to get access (i.e., decrypt) the data. The other code consists of four six-character alphanumeric strings, like ACB123-DEF456-GHI789-JKLMN0.
When you have received those codes, the first thing to do is to use Finder's CMD-K option to open the safe. It may be necessary to have some content in the safe for it to open correctly. In my case, I created a folder called Daily there. When you go through Finder's authentication protocol, enter the full https address as the device, the email address as the login name, and the decryption string as the password. IMPORTANT: save this in your login keychain.
Now, some of what follows can be done differently if you prefer, but this is what I did.
I have a miniature partial unix-style file system called “usr” under Documents in my home directory. I put it there to keep it relatively unobtrusive and to avoid cluttering the main file system. In what follows, it is assumed that the folder “~/Documents/usr/libexec” exists to contain the script.
Next, the script itself:

#!/bin/ksh
# backs up a list of folders or files to the CloudSafe Daily folder.
# The backups are done in subfolders of Daily as follows: there is a
# folder for every month (%m; 01-12) in every year (%Y). The backup is
# done there whenever the corresponding folder (%Y%m) doesn't exist.  On
# all other days, the backup is done in a 7-day cycle based on the day
# of the week (%u; 1-7; Monday = 1). All previous contents (if any) are
# removed before each backup.

# NOTE: the CloudSafe file system is very simple and does not support
# links and so on, so nothing complicated should be backed up here. all
# are below $HOME. If it becomes necessary to backup more complicated
# filesystem structures, maybe we can backup using tar or a disk image

Me=`basename "$0" .ksh`

# server info
SAFE=0123456789 # REPLACE THIS WITH YOUR SAFE'S INFORMATION
SERVER=webdav.cloudsafe.com
URL="https://$SAFE.$SERVER/Daily"
# mountpoint info
MNT=/Volumes
DEST="$MNT/Daily"

Year=`date +%Y`
Month=`date +%m`
Day=`date +%u`

# try a command n times or until success
function tryrep {
 typeset i ntry=$1 ; shift ; typeset cmd="$@"
 for (( i=0 ; i<$ntry ; i++ )) ; do
  if $cmd ; then return 0 ; fi
  sleep 10
 done
 return 1
}

log(){
 print -- "$Me: $*" | logger -s
}
err(){
 log "$*"
 exit 1
}
errum(){
 if tryrep 100 umount "$DEST" ; then
  sleep 5
  if [[ -d "$DEST" ]] ; then
   rmdir "$DEST"
  fi
 fi
 err "$*"
}

# the list of assets
set -A Src \
 Library/Keychains/personal.keychain \
 Library/Keychains/login.keychain

# mount volume
if ! mkdir "$DEST" ; then
 err "Mountpoint '$DEST' is in use or $MNT is unwritable"
fi
# assumes that authentication is in user's keychain & mount_webdav has access
if ! tryrep 10 /sbin/mount_webdav "$URL" "$DEST" ; then
 rmdir "$DEST"
 err "Failed to mount '$DEST'"
fi

log "Mounted '$URL' at '$DEST'"

# establish and zero the destination folder
if [[ ! -d "$DEST/$Year$Month" ]] ; then
 Dest="$DEST/$Year$Month"
else
 Dest="$DEST/$Day"
fi
rm -rf "$Dest"
mkdir "$Dest"

for (( i=0 ; i<${#Src[*]} ; i++ )) ; do
 where=$(dirname "${Src[i]}")
 mkdir -p "$Dest/$where"
 if ! cp -Rp "$HOME"/"${Src[i]}" "$Dest/$where" ; then
  errum "Copy returned an error (${Src[i]})"
 fi
 log "Copied '${Src[i]}' to '$Dest/$where'"
done

log "Backup complete"

if tryrep 100 umount "$DEST" ; then
 sleep 5
 if [[ -d "$DEST" ]] ; then
  rmdir "$DEST"
 fi
else
 err "Problem unmounting $DEST"
fi
log "Unmounted '$DEST', exiting"
exit 0

This script should be copied and pasted into a file (look it over for random HTML character entities that might get inserted), and saved as something like « cloudSafeDaily.ksh » in ~/Documents/usr/libexec. Use the « chmod +x » command to make it executable. Note that you must REPLACE the 0123456789 with YOUR SAFE's 10-DIGIT CODE.
The version of the script above backs up only your main login keychain plus a “personal” keychain, but you can alter the « Src » array to contain what you want to include. These can be either files or folders. Note that they shouldn't include symlinks or Finder aliases, because those aren't supported in the CloudSafe filesystem.
Next, use the crontab -e command to create an entry in your personal crontab like this:

30 2 * * * ~/Documents/usr/libexec/cloudSafeDaily.ksh

In the example, this  will run the above script at 2:30 AM every day. Take a look at the documentation in crontab(1) and crontab(5) for more information about how you can set this up to run.
Basically what it does is to try (heroically) to mount your Backup safe at the indicated time. It figures out the year, month, and the day of the week by using the date(1) command. It looks to see if there is a long-term backup already for the year and month (for example, /Volumes/Daily/201203) and if there isn't, it will use that as the destination; otherwise, it will use the day of the week (for example, /Volumes/Daily/1) as the destination. Then it copies the indicated data into the destination (after first removing whatever was there before), creating all folders in the paths as needed. For example, in the example it will create (e.g.) /Volumes/Daily/1/Library/Keychains/login.keychain along with the Library and Keychains folders. This folder-creation is necessary in order to prevent files of the same name in different folders overwriting each other.
This will allow you always to go back 7 days, plus it will keep one backup per month as long as you let it run.
It does not check for space, because the WebDAV filesystem doesn't support that feature correctly. So, it will keep going until you get an error, which shouldn't be a problem if you use this only for smallish files. If the script works normally, there will be a few lines of information written to the system log; if there are errors, a descriptive log entry will be made to help you try to pinpoint the problem.

Why did I make the login and personal keychains the default items to backup?
There is a bunch of critical information in the login keychain, plus, you can store texts in there as encrypted secure notes. You can use this for all of my password information and various other important, secret information.
Note that secure notes do not unlock automatically by default, but some passwords do. Also note that the password for the login keychain is normally the same as your login password and some feel that this is a security problem. If you think this, then my advice is to create a second keychain file, which I call « personal.keychain », for example. Put things that are unlikely to be needed by programs, such as your secure notes and certain passwords and certificates, and give it its own, different password. I added this to the nightly backup on a line before « Library/Keychains/login.keychain » that says « Library/Keychains/personal.keychain \ ». They will both be backed up. Note the backslash at the end of the non-final line: this is critical. Another option would be to remove the final « /login.keychain » from the existing line; this will cause the entire Keychains folder to be backed up, no matter how many keychains you have in there (I didn't do that by default because sometimes a lot of useless files can accumulate in the Keychains folder).
UPDATE: It turns out that in order for the crontab process to get access to the information in the keychain, it must be added to the System keychain, and access must not be restricted. This doesn't seem acceptable to me.

/etc/profile

The system-wide sh-class shell initialization file can be very useful, but there are some potentially confusing aspects of how it is used in different shells. The goal is to have a reasonable version of /etc/profile that can be used for all users.

Classic sh shell.

The Bourne shell as described in the BSD 4.4 User's Reference Manual distinguishes between "interactive shells" (stdin is a terminal or -i flag was used); "login shells" (0th argument begins with '-' (e.g., "-sh"); and other invocations. Login shells evaluate /etc/profile and .profile if they exist, non-login shells skip this step. Then for every shell invocation, if the environment ENV is set, its contents are interpreted as a path that is then evaluated. Note that for non-login shells, ENV must already be in the environment; for login shells, it may be set in one of the profiles. Interactive shells can be identified by using case $- in *i* ) ... ;; ... esac.

Bash shell.

This is the default OS/X shell and is the most widely used descendant of the Bourne shell. It behaves differently when it is invoked as "sh" or "bash". In the former case, its startup is intended to emulate that of the classic sh (note that this mode is used in single-user mode and in many shell scripts intended to be widely compatible). For bash, an interactive shell is one whose stdin and stdout are connected to terminal, or if the -i flag was used. A login shell is one whose arg0 starts with - ("-bash", "-sh"), or where the --login (or -l) flag was used. When bash is invoked as "sh", it first evaluates /etc/profile and then ~/.profile unless --noprofile is given. Note that the --login can be used even with "sh" invocation. At this point, "sh"-invoked bash enters "posix mode" (the --posix flag can also be used for this purpose). In posix mode, ENV is handled as with classic sh. When bash is invoked as "bash", it also evaluates /etc/profile, then the first existing file in the set ~/.bash_profile, ~/.bash_login, and ~/.profile (unless --noprofile was given). Interactive, non-login bash evaluates ~/.bashrc, unless --norc is given. Non-interactive bash evaluates $BASH_ENV if defined. Note that for interactive bash shells, $- will include i and PS1 will be set. In bash, the following variables will be set by the shell: BASH, BASH_VERSINFO (array), BASH_VERSION.

Korn shell.

This is an excellent extended version of sh which differs from bash in various ways. Ksh defines interactive the same as bash, but it has no effect on the startup files used. Login shells are defined as for sh: arg0 must begin with '-' (e.g., "-ksh"). Login shells evaluates /etc/profile if it exists, and then .profile or $HOME/.profile, if either exists. As for ENV, it is handled the same as classic sh, except if it is not set, $HOME/..kshrc will be evaluated if it exists. If the real and effective uid or gid do not match, /etc/suid_profile will be used instead of ENV or HOME/.profile (interactive shells). Also, in ksh $- contains i. In ksh, the variable KSH_VERSION will be set by the shell.

Single user mode

On standard UNIX-style systems, either /bin/csh or /bin/sh are used in single-user mode. If /bin/csh, we are already forked, but if /bin/sh, then there are consequences for /etc/profile, because it will generally by evaluated in single-user mode (in the current launchd under OS/X, it is invoked as /bin/bash, with arg0 set to "-sh"). Functionally similar invocations are probably the norm.

Some conclusions

Basically, /etc/profile will be evaluated for all logins. If ENV is set, then in some cases but not all, it will be evaluated, and of course there are some other shell-specific files that also will be evaluated in some cases, that we aren't concerned with here. There is no simple test to detect the currently running shell. One can use $0, but that doesn't distinguish true sh from one of the others masquerading as sh. However, that may not matter in many cases. Therefore, a simple case statement on $0 will work in most cases in /etc/profile. The situation in ENV is more complicated, because there could be an unknown amount of environment setting (e.g., for PS1) before ENV is run. In one case I know of, the login shell is ksh, and it is detected correctly, and then ksh-specific material is placed in PS1. If bash is then run interactively from the ksh login session, it *inherits* PS1. The fix is to put stuff in ~/.bashrc to set up PS1, or to do other things where bash and ksh differ.

Scripting single-user mode

As I have written earlier, it was possible to add commands to /etc/rc.server, and they would be executed in a context very similar to single-user mode. However, with the the 10.5.7 upgrade, /etc/rc.server was moved to a later point in the boot sequence, to an environment more similar to ordinary multi-user mode. So not only is the context different, but this indicates how fragile the whole /etc/rc* vestige is in OS/X. A new method is required.

The best alternative I've come up with is to use actual single-user mode. It is possible to get into single-user mode from a script via this sequence (executed as root): « nvram boot-args=-s » ; reboot. At some point once single-user mode is entered, the command « nvram boot-args= » must be run in order to re-eneable multi-user mode.

There is a script that is executed by the shell, and that can be hooked for the purpose of scripting maintenance in single-user mode: /etc/profile, the shared, system-wide start-up file for all shells in the sh family. However, since this location can (and should) be used to customize the shell environment at the system level for all users, it should be changed as "invisibly" as possible.

I prefer to deal with these issues as follows: I'll put one line at the top of /etc/profile that contains some fast heuristics and slower deterministic tests for single-user mode which if passed result in a call to jidaemon (which is the script I want to run in single-user mode). The presence of this line at the top of /etc/profile is required. It can be checked by comparing [[ "$THELINE" == `head -1 < /etc/profile` ]]. The heuristics should all be based on the shell's internal environment, and should be as fast as possible, because /etc/profile is called every time the shell starts up. The heuristics are UID=0, HOME=""; if those are true, the deterministic tests are `sysctl -n kern.singleuser`=1 and -x /var/root/jidaemon. If those are true, run /var/root/jidaemon. Within jidaemon, all those tests are repeated, and some additional tests are run: nvram boot.args == *-s*, read-only root, -f /tmp/just.imagine and so on. Also, if -s is set in nvram boot.args, jidaemon must clear it while preserving any other flags. If any of these tests fail, then jidaemon returns to caller and the only result (beyond clearing the boot.args -s flag) is a slight delay--the shell will continue and an interactive single-user mode session will begin. If jidaemon runs normally, it will restart the system when complete.

Importing SSL certificates on OS/X leopard server

I'm not going to go through the whole process, which is well-documented elsewhere. Basically, you buy & download the ssl.crt (certificate/public key), ssl.key (private key--I go passwordless, but YMMV), and ca.pem (certificate authority) files and then click on "Certificates" in Server Admin, browse to their locations, and install them. My problem was related to the fact that last year, when I first got certs from startcom, their master ca was not listed in the standard list of signing authorities on the server. I tried a lot of ways to get around that, and eventually got it working without really understanding why. My trick was to install the certs manually into /etc/certificates and use "custom configurations" in each ssl service. Recently when I had to renew the certificate, I had to revisit the whole mess. When I tried to import the renewed certificates, I put them into /etc/certificates as before, but after each reboot, the old ones would keep getting written on top of them. This undoubtedly was happing last year, but I didn't realize it because I only had one set of certs. I eventually decided that the only place the old ones could be coming from was the system keychain.

I looked in the system keychain and tried to install the new ones there, but kept getting an error saying the the identity already existed.

It turns out that in fact, the server copies certificates from the keychain into /etc/certificates at boot time. I hadn't known this. When I deleted the certificates from the keychain, everything "just worked" after I installed the new certs into /etc/certificates. The missing piece of the puzzle was the server scribbling in /etc/certificates.

Chapter three of this (in progress) is that now the startcom signing authority cert is in the server's default list. I verified this on a new install of the server software--on that system, the standard Server Admin approach works flawlessly, no direct access to /etc/certificates is needed at all. So, the next step on the older system is to turn off all ssl services (at least iCal, iChat, Mail, OD, RADIUS, VPN, and Web), clean out /etc/certificates, and install the up-to-date certs into Server Admin. Then, go through each service and ditch the custom configurations, replacing them with the standard wildcard cert installed normally.

OK, I think I've done this successfully: it seems to be working. So, the comment about the server scribbling in /etc/certificates no longer is relevant to my particular configuration, but it is very relevant to someone who has a custom configuration. My advice: go ahead and put the certs in /etc/certificates, but (1) don't name them either Default or the address certified (e.g., *.domain.net), and (2) make sure they are NOT entered in the keychain as well. One or the other, please.

Note on using webdav idisk for experiment data

One of the problems I had before with setting up a script to manage the experimental scenario and data on a webdav idisk was that I hadn't stumbled across how to automatically mount & unmount the filesystem. It turns out that this is very easy.

First, create a directory to use as the mountpoint. For example, assuming you are in a writable directory, use something like this:

mkdir mnt

Next, use a command of this form:

/sbin/mount_webdav -s http://idisk.mac.com/groups.labname mnt

("groups.labname" should be replaced with the actual name of your idisk)

If the login info us not in your keychain, it will ask you for it. You might consider putting in the keychain for convenience. Note that any user (i.e., any RA who will access the database) must have access to the idisk.

Note that it is possible for a subdirectory to mounted directly with a command like this:

/sbin/mount_webdav -s http://idisk.mac.com/groups.labname/Databases/Thisdatabase mnt

However, in this case, a separate keychain entry will be needed to log in. It would normally be simpler to mount the root of the idisk and navigate its hierarchy programatically.

When complete, a simple "umount mnt" will be required.

Note that this method will work even if the idisk had already been mounted in the standard location or somewhere else.

Combining image and rsync backups

This is from a note on the Apple server list.

The procedure is to start out with an asr image of the root volume using the -erase flag. This is now a bootable volume.

Then, each night, perform an rsync on this volume, using -delete and other flags so that all changes are written to the volume. This is now an updated, but still fully bootable volume.

This is almost equivalent to doing a nightly clone. It is food for thought. Here are some related issues:

• Unless you want to bring the system down to single user mode every night, you are still going to have to deal with open database files during the rsync.
  
• Assuming that those files have been dealth with, would it be faster just to do the image dump every night?
• Is the target system more likely to be bootable in the even of a catastrophe mid-copy if rsync is used?
• One of the biggest advantages of using rsync is the ability to do snapshots. Clearly, the original image could be used as the starting point for nightly snapshots, but then the image itself would never be updated. Is there some way for the root hierarchy to be the most recent snapshot, with previous versions stored elsewhere? (See below)

It seems to me that what might be needed here is for there to be two rsyncs each night. The first one is to update the bootable image from the working image, as in the original hint. The second one, to be done when the first one is finished, is to make a snapshot of the bootable image.

In other words, the backup would go like this:

1. Before any backup, go through the process of dumping all system databases.
   
2. For the first (labor intensive) backup, use asr to make a complete copy of the base system. The system should be as quiesent as possible for this, and the system database dumps should contain the current database contents. The backup volume should be considerably larger than the working volume.
3. Dump the databases and use rsync to update the smaller backup volume. Be careful to exclude the /snapshots directory on the backup volume; this should be untouched by this run of rsync.
4. Then use rsync again to create a new snapshot of the non-snapshot regions of the backup volume (there is no need for an additional database dump). This will be stored in /snapshots on that volume (and so /snapshots will again be excluded).

Both runs of rsync could be run at fairly low priorities.

Note that if this were to be done over the network via an rsync server, the second (snapshot) rsync run could be done locally on the server. Not sure if that would be worth it, though. For now, I'm assuming an attached drive.

Also, it would be possible for /snapshots to be a different drive. For example, with two internal drives, the second drive could be the mirror of the first, with a larger external firewire drive for the snapshots. In a fairly non-intensive application like in our lab, this use of the second internal drive would probably be better than mirroring RAID, which is how it is being used now.

In any case, we would want to avoid spotlight on the backup drives, and also we would want them to be mounted read-only except when actually being written during backups.

A variation of the above would be to split drive 1 between the backup-able part and a large non-backupable part, so that less space would be needed on drive 2.

OS/X single-user-mode backups

I've been trying to take advantage of the script /etc/rc.server to do full backups of the boot drive. This file is present in OS/X server, and it can be added to non-server systems. Basically, the script is run via /bin/sh early in the boot process, at a time similar to single-user mode. Only kernel drivers are present, which means that the internal harddrive and firewire drives are available, but not (at least not yet) USB drives. The boot drive is semi-mounted, in read-only mode. Semi-mounted means that its device is listed as "root_device", not as an actual drive.

The advantage of bringing a server down on a regular schedule for backups is that there are no open files, and the entire system drive is unwritable. This maximized the thoroughness of the backup. Furthermore, there are some cases where backup programs such as asr(1) can use more efficient techniques for read-only drives than for read-write drives.

The are huge disadvantages, though. First and foremost is that while the server is down for backups, it can't provide whatever services it is responsible for. In the case of our servers, that's at least DHCP, DNS, OD, and file and web services. However, in a situation such as in our lab, where the amount of data is relatively small (probably less than 20-25 GB), the down-time will not be excessive, probably an hour or less.

The second class of disadvantages is almost a deal-breaker. OS/X has chosen to implement so much of its device and file-system interface code in terms of user-mode "frameworks" rather than kernel-mode drivers, hardly any even of the command-line utilities are available for use in single-user mode! For example, diskutil, the main filesystem tool, is unavailable. Disktool doesn't work any better. Hdiutil will do some things, but cannot attach images or use the -srcfolder mode. It turns out that the best tool of the ones that will actually work in single-user mode is asr.

Asr works far faster when it is in "device" mode. In order to enter device mode, the target drive must be greater than or equal to the size of the source drive, the "-erase" flag must be used, the source drive must be mounted in read-only mode. Asr is only for hfs-format drives. There is also "copy" mode, which is also fairly fast; it is used when device mode's criteria are not met.

The biggest problem with asr is that it copies everything, including the volume label, and there is no way to avoid this. Therefore, the copy ends up as a filesystem named the same as the boot drive. This can cause confusion. One would think that the solution would be to simply change the volume name after the clone operation. However, due to the impoverished runtime environment of single-user mode, there is no tool available to do it. The pdisk(1) program has a partition-labeling option that does work in single-user mode, but it turns out that this is not the same thing as the hfs volume label. When the system comes up, the clone will be mounted under the same name as the main drive with a " 1" suffix, like "Macintosh HD 1". If there are several backup partitions and nothing is done, they will all end up with the same name, like "... 1", "... 2", etc. (See update below.)

This can cause considerable confusion. The only "solution" I've been able to come up with is to write some information into the /tmp directory regarding the backup itself, and then once the system comes back up, use diskutil to rename the volume accordingly. A good way to do this is in crontab, with the "@reboot" time indicator.

As for a general strategy, it is important to have at least 2 backup partitions (2 drives would be better), so in case something bad happens, the previous backup would be available. Also, the backup partitions should be larger than the system disk.

At present, my servers both have 250GB Raid mirrors, and I have a 300GB firewire drive for each of them. As an initial test, I will simply do a single asr backup of the main drive onto the firewire drive--this is nearly as safe as having two partition on the firewire drive. Later, I'll get another firewire drive for each one, and swap the drives.

UPDATE

Here's a kind of strange way to get around the problem of the volume name. Instead of trying to change, ex post facto, the name of the clone, why not change the volume name of the system disk? This can be run very early in the launchd process, and all it takes is "[sudo] diskutil rename / newname". Obviously, the name will have the boot time in it, or, more difficult, a sequence number related to the backup system. I think a timestamp of the form: 200901171402 would be good. As for the rest of the string, why not simply look for the current name (you can get this from the "list -plist" diskutil command). If the last 12 digits of the current name (which will be identical to that of the most recent backup) are digits, then they will be replaced, otherwise nothing will happen. If there is a reasonably short system name, like "Lab 13", then why not name the root drive something like "Lab 13 System Disk " ? Each time the system is rebooted, the timestamp will be updated. The volume names of the clones will contain the timestamp of the previous boot. Since the root volume, unlike other mounted volumes, doesn't get mounted as /Volumes/VolName, the change of name will have no effect on paths, etc.

Plan 2: A slightly more satisfactory but more risky way to do a similar thing would be to change the name of the root volume (e.g., to "Lab 13 Clone 200901171402") before restarting the system for an automated backup. The name would be changed always to a constant (e.g., "Lab 13" or to the part before " Clone...") during the early the boot process.

In either case, there would need to be three scripts:

1. The setup-and-reboot script, which would search for the target volumes, decide which one to write to next, and if there is one, to store that information in /tmp/rc.autoclone or whatever. This script will change the name of the root volume just before rebooting under plan 2.
   
2. The clone script, which would expect information in /tmp/autoclone that if it checked out will control the clone operation.
3. The cleanup script, to be run very early in the boot process, to change the name of the root volume back to its normal name. In plan 1, the name will contain a time stamp, in plan 2, it will be an ordinary name like "Lab 13".

The first script can be run either from the command line or from a launchd/crontab entry. The second script must be run at the end of/etc/rc.server, and it must contain safety checks to prevent writing into the wrong medium. The third script would ordinarily be run very early in the boot sequence from launchd/crontab.

As always, the most dangerous possibility is that the clone will be written into the wrong place. Since diskutil doesn't work right in single-user mode, probably the safest way to handle this is to write a check string into the destination volume, for example, a file in its root directory called rc.autoclone identical to the one in /tmp. Also, there must be a directory created in /tmp called "mnt" as a place to mount the destination volume, so that /tmp/mnt/rc.autoclone can be compared to /tmp/rc.autoclone.

So in rc.server,

1. Check for /tmp/rc.autoclone
2. Check that the timeout interval has not passed (e.g., 5 minutes)
3. Check for /tmp/mnt
4. Get info on all current drives
5. Search for the drive indicated in /tmp/rc.autoclone
6. Mount the drive on /tmp/mnt and compare its version of rc.autoclone
7. Unmount the target drive
8. Perform the clone operation
9. Done

In the after-boot crontab (plan 2),

1. Get the current / volume name
2. If it needs to be changed, change it

The setup script does:

1. Do general checking to prevent being called at the wrong time (e.g., too soon)
2. Make sure that at least one target volume is available
3. Look through all possible target volumes to find the one with the oldest timestamp.
4. Compute its ID string
5. Create an rc.autoboot containing the ID string
6. Write the rc.autoboot into the root directory of the target volume
7. Reboot

An example experimental project

This project, which we call "picolf6" consists of six experiments, four of which use Sensonics odor labels as stimuli. In addition, the counterbalancing and randomization is within each of the two sets of three experiments. Therefore, this was done by scripts rather than within Superlab.

One discovery we made about Superlab in the course of setting up picolf6 was that any time that a file system path is stored within Superlab, all symbolic links within the path are expanded and the path converted to an absolute path. This complicated several aspects of the process.

There are three types of stimuli used: pictures (stored as .jpg files), words (stored as .png files) and ticket numbers (stored as .png files). If you will refer to my recent blog entry on the sl4am hierarchy, you will see the layout we used here. Under the project directory, there is a Shared directory containing all of the .sl4 Superlab scenarios, and folders for all stimuli. The general idea is that the folder for each experiment (with subjects and groups) will have a link back to the scenario for that experiment in Shared, plus a "stim" folder containing links back to specific stimulus files within Shared.

The odor labels we used were mounted on standard 1"x2" event tickets and torn off one at a time to be "scratched and sniffed" and given to the subject. So, one of the things required is for there to be an inobtrusive ticket number displayed on the screen at the beginning of each trial. In superlab, the only way to do that is to make a graphic with the numbers in the corners and then specify that superlab scale it to fill the screen. We used the least significant digits of the tickets for this, so experiment 1 used tickets 1-30, exp 4 31-42, exp 5 43-54, and exp 6 55-66. Since this was always the same for all subjects, we set up folders in Shared named ticket[1456] and stored the .png files with the images there, named, e.g., 23.png.

One thing we learned the hard way was that it is best to set Superlab up so that the scenario is in a file hierarchy identical to where the experiment will be run. So for example, we simply specified /Users/.../Shared/ticket1 and Superlab could use the same relative path at runtime and find the stimulus folders.

The other stimuli were more complicated, because they had to be different for each subject. Superlab always accesses stimulus list folder contents in alphabetical order, so while setting up the experiment, we set up dummy folders containing files with names like w/img34.png (for word image event #34). Later, when setting up the hierarchy, we put links with those names to image files in Shared. So if for example on trial #34, a certain subject needs to see the word "alligator", stim/w/img34.png would be a link to ../../../../../Shared/words/alligator.png (for example).

Now, Superlab doesn't print the name of the file in a stimulus list folder, only its sequence number. So in order to figure out which stimuli were presented to a give subject without going back to the setup data, we inserted a dummy text event into each trial. These events were simply numbered like "@=1-34=@" for experiment 1, trial 34. As part of the setup sequence, we created a sed script that we placed into each experiment folder that translates that dummy code into a condition code for that trial, for example "alligator:old". This allows the actual stimuli that were used to be determined.

We also placed a link to a superlab scenario file in Shared into each experiment subfolder. However, we were not able to use symbolic links for this, since superlab then assumes that the scenario file is actually in the Shared folder and all of the subject-specific links fail to work. As a work-around, we just used hard links for it, since it is a file rather than a directory (file system rules disallow hard-linking to directories).

Here is a tar archive (tgz) of the Korn shell scripts I used to set this experiment up. Note that you also will need to install some packages to generate the graphics versions of the textual stimuli. I also didn't include the picture stimuli we used.

Multiple superlab stimulus folders

Apparently when you set up an experiment in Superlab that uses external stimulus files or folders, it saves two pointers to the files: the absolute path at the time the file or folder was specified, and the file or folder relative to the scenario file when it was saved. In fact, it appears that sometimeis if you use "save as" within superlab to save a scenario, it recomputes the relative address(es) using the new scenario location. Therefore, as long as the relative path points to folders and/or files that exist, and that the absolute path does NOT exist, things will work. However, this latter approach will not always work

However, if the absolute path is valid, then those files will be used instead of the local relative path, and the wrong stimuli will be presented. What is needed is some way to relocate the scenario file, relative to a certain version of the stimulus files.

Say for example you have 25 subject folders named 01-25, and they are all "sister" folders, that is, subfolders of the same superordinate folder. You might think that if you set up the scenario and stored the scenario in subject 01's folder, you could then copy the scenario into each of the others and use their stimulus set-up. But since the absolute address of subject 01's stimuli would still exist, I strongly suspect that they would be used instead of the relative address. What is needed is some way to relocate the paths in the scenario file so that they point to the new location.

If superlab can't find the stimuli, it asks the experimenter where they are. This could cause some fairly minor problems, but the larger issue is when it does find them, in the wrong place.

It turns out that it is possible simply to overlay the absolute paths (or as many of them as you want to relativize) with a sequence of X's of the same length, using some method such as a binary editor like bbe(1). Superlab is quite content to use the relative address. Note that if you should happen to save the scenario, new absolute addresses will be written in there.

It would be better if there were some way to identify the absolute addresses automatically, but since they can be located almost anywhere, that's a bit tricky. There are some other paths in there, such as the default logfile location. Maybe the best way is simply to specify them as part of the setup, since they will be known then, and do the overlay as part of building the project.

UPDATE

One very important lesson: take the time to do all development of superlab scenarios in a directory hierarchy that matches how it will eventually be installed. This is a little less convenient in the beginning, but pays off hugely later on.

Converting text to graphic files for Superlab

Since graphics files usually exist outside of the immediate Superlab scenario folder, and are not loaded until the experiment runs (and if the appropriate option is selected, not until just before a trial runs), it is possible to use an external script to re-randomize graphics files before running a particular subject.

The way it works is, you set up Superlab to use generic names for files, for example, "trial001.png", "trial002.png", and so on. You have your real stimuli in a different folder, with names like "elephant.png" and "COW.png". Then, when you are setting up for a particular experiment, you get rid of the dummy files and put in links in to the real stimuli, in the appropriate order, for example, trial001.png --> elephant.png ; trial002.png --> COW.png. Obviously, you have to keep the mapping around so that the logfile can be patched up after the run, by replacing instances of "trial001.png" with "elephant.png" and so on.

Obviously, you can't do this with text stimuli because text stimuli are internal to Superlab. I have written a utility to patch an existing scenario file to contain a different order of stimuli, but this method is very risky and isn't flexible enough. So the correct solution is to convert the text stimuli you want to use into graphics files.

There are many ways to do this, but one of the easiest to use from a scripting standpoint is called "a2png". This is a utility that can be downloaded from sourceforge. It needs either the cairo or the gdlib graphics libraries; one or the other also has to be installed for a2png to build. Once it has been installed, you can use .ttf font files to create .png files from text strings. By default, the image is cropped to the font's cell size, and has a black background. There are a number of options to change the background, foreground, size, font, spacing, and so on. The .png files can be used directly by Superlab on both Windows and Mac systems, or they can be converted to jpeg or some other supported graphics format.

By default, if you give a2png the name of a .txt file containing a stimulus, it will create a .png file in the same folder with the same basename. So for example, "a2png ... elephant.txt" should result in an output file "elephant.png". If you don't want all those *.txt files, a2png will also accept standard input if the file is "-", and will write to X in "--output=X". So, an alternative way to create elephant.png is "print elephant | a2png ... --output=elephant.png -".

How to specify the right font

Now, at least on my system, a2png doesn't want to find the ttf fonts. The built-in font folder list is a poor match for the fonts I have installed on my system. So, what I do is to give the whole path to the .ttf file I want to use. You can find all the appropriate fonts on your system with "locate .ttf | grep ttf$". Also, there are thousands of truetype fonts (ttf) out there on the internet. It's probably better to give the whole path anyway. I like a sans-serif font for displaying stimuli, such as free-sans, which can be readily downloaded.

You can also mix text and pictures, and randomly switch, for example, which kind is on the left or the right of the display, just by setting the link appropriately before running.

UPDATE

There appears to be some kind of glitch in a2png such that the cropping that it does removes the bottom of each character on the last (i.e., only) line. There is a workaround of suffixing a '\n', but this adds too much space.

There is a completely different approach available with the classic netpbm package. This command line:

print JjKg_Xq \

| pbmtext -font ~/Downloads/bdffont/100dpi/helvR24.bdf \

| pnmcrop | ppmchange white black black white \

| pnmtopng > foo.png

isn't too bad. An alternative is:

pbmtextps -font=Helvetica JJKG_XQ \

| pnmcrop | ppmchange white black black white \

| pnmtopng > foo.png

So, one way or another, there will be a way to do this. Frankly, a2png produces prettier output. The pbmtextps output is quite fuzzy, while the pbmtext output depends on having the bdf fonts available, and in turn, they have limitations on size. Since a2png uses the Cairo graphics library, it can use ttf fonts and scale them, etc., very prettily. Hopefully I will find a fix for a2png.

UPDATE2

It is true that a2png produces more attractive lettering, but as it turns out, there is a very real application for the netpbm package here: setting up the experiment template. The scheme I am trying to use involves setting up a single experiment with all of the trials indexing external event files, usually images or images of text. By changing the names of these external files, you can change the stimuli presented to subjects with none of the limitations imposed by superlab. So, what I've been doing is to generate dummy stimuli to be used while testing. These are graphics containing text strings that make it easy to identify the order and type of the stimuli for debugging purposes.

In one of the experiments I'm setting up now, there are 30 640x480 pictures. Here is the shell function I'm using to create jpeg dummy files for them:

jpg640x480(){
ppmmake lightblue 640 480 \
    | ppmlabel -x $((320-(5*${#1}))) -y 240 -size 10 -background lightblue -color black -text "$1" \
    | pnmtojpeg > $2 2>/dev/null
}

I chose black over lightblue so they would be very contrastive with the white over black text stimuli.

UPDATE3

Well, there is a fairly easy way to get images cropped correctly with a2png that will work until the program is fixed somehow: use the --no-crop option in a2png, and crop the result using netpbm. For example:

print Somejunque \

| a2png --no-crop -s --overwrite --font-size-0.1 --output=uncropped.png \

; pngtopnm < uncropped.png

| pnmcrop \

| pnmtopng > cropped.png

This yields the best of both worlds: flexible, high-quality text rendering plus correct cropping of the result.

How to set up an sl4am project

Once the basic SuperLab experiments are running, the next step is to set up the hierarchy of files and folders. This can be done with a fairly simple script, given the names of the population and condition groups and the initial number of subjects in each cell. Options include setting up an empty Shared folder and links to it in each experiment folder. Empty rc files and flag.free files are created everywhere. Another option is to clone a new population group from an existing one; another is to look for empty rc files (this would be the sign that an external setup script didn't do its job completely).

However, once the hierarchy is all set up -- and it probably is a good idea to set up only the Try population group first, and clone the other groups from it -- the next step is to populate all of the experiment folders and to put actual code into the rc files. The best way to do this is to write a custom script. This script could use find(1) and be driven by the existing factors as an organizational approach.

Update:

Just stumbled across the automator(1) command. This should be very useful for running experiments, since it is a way to invoke Automator workflows from the command line. There are options to set variables and to pass input, including standard input, to the workflow. It is less clear to to take output from the workflow, probably temporary files will be needed.

A Useful Experimental Data Hierarchy

This is a scheme for storing experimental data, primarily with SuperLab, that should be easy to setup, easy to use, and easy to automate.

At several specific points throughout the hierarchy there are rc files: rc.sl4am, rc.proj, rc.subj, and rc.exp. These can contain arbitrary ksh code and can change the operation of sl4am, but are intended to customize the operation of sl4am by changing specific parameters or by altering the setup, runtime, and/or cleanup phases of a SuperLab scenario.

The SL4AM subfolder within the SuperLab Experiments folder is the default sl4am home folder. If an rc.sl4am file is present, it can redefine the variable SL4AMHOME in order to use a different home folder, for example, in a shared location on the local machine, like "/Users/Shared/Documents/SuperLab Experiments/SL4AM", or on a remote location, such as "/Volumes/groups.ourlab/Documents/SuperLab Experiments/SL4AM". If there is a separate root for the status flags and datafiles, that can be defined in rc.sl4am as "DATAROOT" (see below). If DATAROOT is defined, it should give the path to a folder that will be organized in parallel to the SL4AM folder.

Note that spaces in folder and file names should avoided if possible below SL4AM. I'm trying to make the script immune to problems resulting from spaces in filenames, but it's safer (and much wiser) to avoid them.

Project roots are top-level subfolders in SL4AM. There can also be a top-level subfolder in SL4AM called Archival that is intended to store completed or inactive projects.

Except for some code at the beginning to select a project, sl4am is concerned only with the world within a single project root, and in fact, it uses cd to go there as soon as possible so that most paths within a project are relative to the project's root. While running SuperLab, sl4am changes temporarily into the folder of the specified scenario (see below). At the top level of a project root, there must exist a file called rc.proj that is sourced when sl4am starts a session in that project. A subfolder X of SL4AMROOT is a project root iff X/rc.proj exists.

Each project contains a subject hierarchy. The top level is a dot-separated list of population.group folders, for example, Healthy.Elderly, Healthy.Young, Schiz.Elderly, Schiz.Young. Below each population.group folder is one or more condition.group folders, such as Set1.VisualFirst, Set1.AuditoryFirst, Set2.VisualFirst, Set2.AuditoryFirst. Below each condition-group folder is one or more numeric subject folders, for example 1, 2, 3. There is one subject folder for each subject to be tested in the project. Each subject folder must contain a flag file. It may be desirable for sl4am to create new subject folders automatically as needed; in any case, the names of these folders are simply numbers starting at 1. There also needs to be a way to insure that sl4am will test the first subject in each condition before starting the numerically next subjects, and so on.

Note that even though some labs never test any population group other than college students, both group levels are required. The best way to handle this situation is to use two population group identifiers, one named "Try" and the other something like "YN" (young normal). The Try group is for testing the experimental setup, while the YN group is a reasonable label for college students. If at some point you need to add another population group, it will be very easy to do it. The presence of the "YN" level won't interfere with anything, and the "Try" pseudogroup is very useful for development and for training RAs.

The flag file is a simple advisory access-control mechanism. When an experiment is first set up, each slot's flag file is named "flag.free", indicating that any computer can run that subject slot. The file is renamed to "flag.user@en0", where "user" is the current user and en0 identifies the current host. (The figure says "fern", which is the name of one of our computers; this method is too variable.) This "checks out" the subject slot to the named individual. The parameter "en0" is set to the ethernet address of en0 (ifconfig en0 | grep ether | read junque ether junque) in order to identify the machine in a somewhat unambiguous way. An individual may relinquish a subject slot by renaming the flag file back to "flag.free", but the presence of datafiles will cause a multi-experiment sequence to pick up where it left off before. When a subject slot is complete and all data has been saved, the flag file must be renamed to "flag.completed".

Note that if DATAROOT is defined, then sl4am will search there for flag.free slots, change to flag.user@en0 there, and will upload data there. The structure under DATAROOT is identical to the SL4AMHOME hierarchy, but it need not have any Shared folder, rc files, or scenarioes or stimuli. In the event that subjects must be tested offline, one or more subject slots should be checked out in advance. When sl4am starts up, it will sweep all experiments looking for the flag "SYNC" in a subject-number folder. If it finds any, and if DATAROOT is accessible, it will synchronize the parallel structure there by uploading any data not already present, and renaming the flag file if necessary. Any time sl4am tries to upload data or change the flag file but DATAROOT is inaccessible, SYNC is created. After a successful upload, it is removed.

There also must be an rc.subj file in a subject-number folder, primarily designed to control running multi-experiment projects (for example, the order in which to run the experiments). This can be a zero-length file.

For each experiment, there is an experiment root under the subject-number folder. This folder contains all that is needed to run one SuperLab experiment: the scenario (.sl4) file; all stimuli needed (these will generally be links elsewhere, or in the case of all-text experiments, missing altogether); and the datafile (.txt) once the experiment has been run. There is also a mandatory rc.exp file, primarily intended to help customizing the SuperLab run or the datafile processing afterwards. For example, this could give the subject some feedback about his performance. Sl4am cd's into the experiment root before running rc.exp and SuperLab.

Under each project folder, there can be an optional Shared subfolder. This contains all shared stimuli and/or sl4 files. For convenience, the setup script will install a Shared symbolic link in each experiment root that points back to the project-level Shared folder (if it exists). To link to it from a stimulus folder in an experiment root under a subject file, just use

    ln -s ../Shared/SomeFolder/somefile.xxx stimset/somename.xxx

To link a scenario file in the experiment root to one in the Shared folder, use

    ln -s ./Shared/some-exp.sl4 some-exp.sl4

It is possible to fully populate this tree before beginning to run, but it is equally possible to use the rc scripts to populate fill things out at run time.

UPDATE

It really isn't hard to link relatively back to the Shared folder without the klutzy locate Shared link. Here's how to do it. You create a dummy tree all the way down to the stim/stimset folder inside an experiment. Also, create a Shared folder with a stimset/xxx.png in it. Then cd down there and do "ls ..", "ls ../.." and so on until you get to (e.g.) "ls ../../../../../../Shared/stimset". The test it with ln -s ../../../../../../Shared/stimset/xxx.png yyy.png" or whatever. Once you've done this and gotten it to work, just make sure that your script does two things: use the right number of dotdots, and actually cd into the stimset folder to do the linkage:

(cd downpath; ln -s uppath/Shared/stimset/file.suff link.suff)

Note that assuming the script is running in the project root, downpath will be like "pgrp/cgrp/subj/exp/stim/stimset", and uppath will be like "../../../../../..". As for why this is worth doing instead of just using absolute links, it's to allow experiments to be installed using simple methods like tar. With relative links, no adjustment is required, with absolute links, basically there would need to be a script run to adjust links after installation in new locations.

Subject checkout on shared volume

In our lab, we have five macbook pros that could theoretically be used all at once to test subjects in a single experiment. In the past, we have gotten into trouble when, due to experimenter error, a certain subject slot has been run on more than one computer. To get past that problem, we want to use a shared volume to contain the experiment setup hierarchy, and come up with some way for all of the computers to share that hierarchy. Obviously, there must be some method to prevent two computers from trying to use the same resources. The simplest way is to set a lock at the filesystem level, marking the subject as "taken", and releasing the lock. However, the most straightforward way to do the sharing, using one of Apple's group iDisks, has no locking mechanism. You can't even make files read-only. The lockfile(1) program, when asked to create a lockfile on an iDisk, gives up and suggests praying instead.

I did come up with a locking mechanism. What you do is to use a reserved folder. A computer that wants to lock the resource waits until the folder is empty, then writes its ID into the folder (the ID could be, for example, the ethernet address of en0). After a short delay, the computer then checks to see if there is exactly one file in the folder, namely, it's own ID. If so, then it has the lock. If there is more than one, then it removes its ID, waits a short but random period, and tries again. The only problem with this mechanism is that it is very slow, on an already slow filesystem like the iDisk.

After pondering this for a while, I thought of another approach. Instead of setting a lock before accessing the subject slot, you randomly choose the "next" subject to test, and then rename it to a name with your ID. For example, if the subject is called "12", and if your ID is aa.bb.cc.dd, then you would simply "mv 12 12-incomplete-aa.bb.cc.dd". Then wait a short time and see if "12-incomplete-aa.bb.cc.dd" exists. If it does, you now own subject 12; if not, try again. (If the locked name doesn't exist, it means that a race occurred and another computer locked it between the time you found it and the time you did the mv command.)

The random selection is somewhat important, but not critical. If you just go in a fixed order, all it means is that there is slightly greater probability that a given computer will have to try more than once to get a subject.

Once the subject is locked, testing proceds. When it is complete, the name is changed again to, e.g., "12-complete-aa.bb.cc.dd". Note that it is still locked, in a sense, since it will not appear in the list for testing.

One other brief note: it might make sense for each subject on the remote volume to be an archive, for example tar.gz format. This would facilitate copying it onto the macbook pro. A question to be resolved is whether data is place into the archive or somewhere else on the remote volume.

SuperLabAutoMator: superlab + automator

We use Superlab 4 for some experiments we do in the lab, but it almost always seems that we need fancier randomization/counterbalancing than the program provides out of the box. Also, the dialog that the RAs must go through, to deal with subject and group IDs, different scenario files for different conditions, and the right name to use for the logfile, have resulted in errors and lost data in our lab. The traditional solution for this is scripting, and in the Macintosh world, many user-oriented scripts make use of the Automator utility. I'm currently setting up an experiment that requires a specific randomization and counterbalancing across three different procedures for 24 subjects. What I intend to do is to make a shell script embedded in an automator script called "SuperLabAutoMator" (or "slam" for short) that will do this in a generalized way. What superlabautomator does is to pop up a window asking to select from an experiment (all must be a subfolder in a standard folder, or if not there, in the same folder as the script). It then follows the instructions in the experiment subfolder, by running "prescript", "midscript",  and "postscript", which are functions defined in the script.

In the experiment subfolder, there is optionally a file called "rc.slam" that can define the following objects:

• name=xxx -- the subject ID to use (default = null)
• group=xxx -- the subject's group (default = null)
• scenario=ppp -- default is "scenario.sl4" in the experiment subfolder
• logfile=ppp -- default is "logfile.txt" in the experiment subfolder
• fifofile=ppp -- use as Superlab's logfile; filtered data should be written to $logfile by midscript
• prescript() -- set up to run
• midscript() -- interact with Superlab while running
• postscript() -- run after midscript and Superlab have finished

All of these have default values. The rc.slam file will be sourced early, before asking for the subject ID, for example. This list could expand or shrink.

Midscript is to be run while Superlab is running, and it can either fiddle with the logfile, or prescript could create a named pipe to be used by Superlab as the logfile and open it for reading by midscript. The purpose of midscript is to handle cases where the actual stimuli to be presented must be changed as a function of performance. In most cases, it will not be needed. Note that if the midscript is reading the logfile from a named pipe, it should also save the raw contents. One way to do this might be to use the tee(1) command; alternatively, midscript can simply save each line it reads. One unresolved problem is that Superlab brings up a user confirmation window if the logfile already exists, as it must for a named pipe. It would be nice to override that somehow.

The core of Superlabautomator will run Superlab from the executable file rather from the GUI, so that command-line options can be specified. After changing into the experiment subfolder, it will source "rc.slam".  Next, it will call prescript, and wait for it to finish. It will then set Superlab to run in the background, bring Superlab into the GUI foreground, and call midscript. After midscript completes and Superlab exits, SuperLabAutoMator will call postscript before exiting.

In general, it is a good idea for Superlab at a minumum to wait either for several seconds before the first trial, or more typically, to display an instructions screen and wait for a response.

From the RA's point of view, all that will be required is to start SuperLabAutoMator, choose the correct experiment (only active ones should be available, and if only one is available, only a confirmation window comes up); choose the subject group from a short list (if more than one); choose the subject (only untested subject numbers will be available). The script will take care of setting things up for Superlab, running it, and dealing with the data, including filtering it, giving some feedback to the subject, and possibly storing it away in a centralized database.

When I get this running with the first experiment (no runtime interaction will be used, btw), I will post the SuperLabAutoMator app and the setup of the first experiment.

Note: while slam is a great name, it is already in use with a couple of different programs/utilities, so we will go with the GUI name SuperLabAutoMator and use slam as an internal shortcut (as in the rc.slam filename), and we can also pronounce the name optionally as slam.

Sending email to root

It is pretty important that the root get asynchronous notification of problems detected by the maintenance system. However, it is obviously impossible to sent email in single-user mode (sendmail not running, boot drive write-protected). Therefore, there are two methods that could be used to notify root of errors or just of system status. The basic idea is that the message be written someplace other than the boot drive and then mailed in single-user mode via a launch daemon.

First, when a backup is done, it will be done to a writable medium, namely a firewire drive. This drive will be mounted when the system is running, so a message posted in /Volumes/Snapshots can easily be sent on to root. This will be the main method of notifying root about snapshots and clones.

Second, there are times when Snapshots isn't available. In fact, one of the critical messages might be that is wasn't available so no snapshot was made. In this case, the best method is to use /var/log/system.log. The system startup saves the standard output and standard error from rc.server (and other boot programs) in system.log. This is in fact a more reliable way to send notifications.

The approach will be simply to write out a banner lines like "org.bogs.rootmail begin" and "org.bogs.rootmail end" so that the daemon can simply extract the intermediate lines (if any) and mail them to root. In general, this should be limited to the bare minimum of lines. By default, system.log is cycled at midnight, and eight gzipped copies are kept around, which should be more than ample.

One thing this does is to complicate the daemon. Not only must it write out org.bogs.maintenance-mode, but it now has to send mail to root. This will require something to be run immediately after entering multi-user mode as well as something when it is time for more maintenance. Also, some kind of flag must be used to prevent multiple mailings, but I don't know what it should be. Maybe just a file ~root/.org.bogs.rootmail containing the timestamp of the last "org.bogs.rootmail end" line that was mailed out would be sufficient. That is, when the mailing script is run, it will send only more recent segments of system.log, and it will update the flag file.

Where to put maintenance scripts

This is slightly complicated, because system areas are of course sometimes overwritten by software updates. Here are some ideas.

I want to keep the changes to /etc/rc.server as minimal as possible. So I will add two lines at the very end of the file that will do two things in a conditional:

# org.bogs.maintenance

if [ -e /private/tmp/org.bogs.maintenance-mode ] ; then source /var/root/Scripts/maintenance.sh ; fi

The file /tmp/org.bogs.maintenance-mode is used to pass parameters to ~root/Scripts/maintenance.sh. If it is not present, then this is not a maintenance boot. If it is present, but if the maintenance.sh script is missing, an error message "no such file or directory" will be written to the log. Note that maintenance.sh runs in the same bash environment as rc.server. Also note that ~root is already a protected place, and Scripts should also be protected (mode 700).

Also note that /tmp will be erased at some point after the maintenance has completed and the system comes up multiuser.

The file maintenance.sh should not do very much, its role is to call other scripts or programs located in the same directory based on the contents of org.bogs.maintenance-mode.

All of the maintenance code will be maintained elsewhere, on another system, and be copied into a directory on the server and installed from there into /var/root and /etc/rc.server. In addition, the installation script will add or replace the last two lines of rc.server, and will also add the appropriate material to /Library/LaunchDaemons (which is where "system-wide daemons provided by the administrator" are supposed to go).