Beat to the punch..

I was looking to start hacking away at a doing lossless CD backup today, and I came across flactag .

Which of course is pretty much everything that I wanted to implement. Woo!

jack | posted Tue Nov 7 16:58:28 2006 | #
category: projects/abcdefgm

Masking out bad GPS points with R

I've been playing around with R to correlate GPS and heart rate monitor logs from when we were cycling around in Italy.

One of the problems that I've run into is that the GPS position accuracy degrades significantly when less than 4 satellites are used, altitude data likewise degrades significantly when less than 6 satellites are used.

These high error data points add noise to the data set and complicate analysis. One option would be to find a way to calculate the error in position for each point, and move that data around with the position data.

That's hard. An easier way is to simply convert the inaccurate points to NULL (NA in R).

To import the data from my heart rate monitor, I used s710. To import the data from the GPS NMEA logs, I created a gpsbabel filter:


DESCRIPTION      GRASS GIS v.in.ascii
SHORTLEN  8
EXTENSION grass
# FILE LAYOUT DEFINITIIONS:
FIELD_DELIMITER         PIPE
RECORD_DELIMITER        NEWLINE
BADCHARS                ,"

IFIELD LON_DECIMAL,"","%f"
IFIELD LAT_DECIMAL,"","%f"
IFIELD ALT_METERS,"", "%f"
IFIELD GMT_TIME,"","%m/%d/%Y %H:%M:%S"
IFIELD PATH_SPEED,"","%f"
IFIELD PATH_COURSE,"","%f"
IFIELD GPS_HDOP,"","%f"
IFIELD GPS_VDOP,"","%f"
IFIELD GPS_PDOP,"","%f"
IFIELD GPS_SAT,"","%d"
IFIELD GPS_FIX,"","%s"

I had originally created this format for easy import into GRASS GIS, but it works nicely for importing into R :


Lat|Long|Altitude|timestamp|speed|bearing|hdop|vdop|pdop|satellites|fix
11.128023|43.430142|143.300000|09/26/2006 07:42:34|0.092600|11.690000|1.600000|0.000000|0.000000|6|3d
11.128025|43.430142|140.400000|09/26/2006 07:42:35|0.174911|177.690002|1.600000|0.000000|0.000000|6|3d
...

Read the GPS and heart rate data into an R frame with read.table.


raw_gps<-read.table("060926_bike_colle_valdelsa_siena.grass", sep="|", header=TRUE)
raw_hr1<-read.table("060926.0947.txt", header=TRUE)
raw_hr2<-read.table("060926.1447.txt", header=TRUE)

Create vectors to mask off the bad data.


mask_4sat <- !((!(raw_gps$satellites > 4)) & NA)
mask_6sat <- !((!(raw_gps$satellites > 6)) & NA)

This uses the identity that FALSE & NA FALSE, but TRUE & NA NA.

More data massaging is in order

I don't have HR data for the entire time interval that I have GPS data for
I need to line up the data sets
the GPS data is sampled with a period of 1s, and the HR data is sampled with a period of 5s

The way I choose to deal with that is to create time series using ts(), and extend and resample the time series with window().


speed <- ts( raw_gps$speed    * mask_4sat, start=c(7, 2554), frequency=3600)
alt   <- ts( raw_gps$Altitude * mask_6sat, start=c(7, 2554), frequency=3600)
hr1   <- ts(data=raw_hr1$HR, start=c(7,564), frequency=720)
hr2   <- ts(data=raw_hr2$HR, start=c(12,567), frequency=720)

I've chose my unit of time to be the hour. The start argument to ts is particularly important, to get all of the data to line up. The easiest way to get all of this is to ensure that the clock on the heart rate monitor is set to UTC, and the GPS logs are in UTC. The frequency argument reflects the sampling rate of the GPS and heart rate monitor.

Now, mash all of this data together into a time series matrix, padding and resampling the constituent vectors to be uniform.


bike_cole_valdelsa_to_sienna <- ts(
    matrix( 
	c(
	    window(alt,   start=7.5, end=13.5, frequency=720, extend=TRUE), 
	    window(speed, start=7.5, end=13.5, frequency=720, extend=TRUE), 
	    window(hr1,   start=7.5, end=13.5, frequency=720, extend=TRUE), 
	    window(hr2,   start=7.5, end=13.5, frequency=720, extend=TRUE)
	),
	ncol=4),
    start=7.5, end=13.5, frequency=720)
colnames(bike_cole_valdelsa_to_sienna) <- c("altitude", "speed", "HR1", "HR2")

Now the data is all referenced together, and ready for some analysis.

jack | posted Mon Oct 30 13:15:44 2006 | #
category: projects/R

Go greylisting!

About a year ago I set up greylisting (using postfix and postgrey) on my mail servers to help filter spam. Except that I didn't validate that it worked on the mail server that I
use.

I fixed that. Guess when.

jack | posted Mon Sep 11 13:59:40 2006 | #
category: projects/count_spam

So I was wrong about unstowing..

It turns out that I do need the ability to unstow things.

This feature wasn't too hard to cram in, although there is no simple inverse of the split rule, so we may leave empty dirs.

jack | posted Wed Aug 30 23:34:38 2006 | updated Sat Sep 9 00:43:36 2006 | #
category: projects/sstow

sstow, the simple stow replacement

I put my nose to the grindstone and got sstow done (go there if you want to take a look at it). I think it's rather clever program for being 149 lines long.

Now let's see how well it does when I try and use it..

jack | posted Wed Aug 30 22:12:04 2006 | #
category: projects/sstow

Automation for running Monotone servers

Historically, I've kept a lot of data for quite a few projects in monotone. I kept it all in one big database. This worked well, until it didn't .

I hit the the magic 2GiB mark where offset_t wraps unless _FILE_OFFSET_BITS=64 was defined at compile time.
One of my boxes decided to start swizzling bits to one of the disks in the raid1

So, I decided to start keeping all of the different branches in their own databases, and using monotone usher to proxy for them.

However, with 40 branches this is a bit of a maintenance problem. So I created some automation to template out runit runfiles and an usher config file. These run files include running usher, doing period db_checks, and syncing to other servers for a particular branch.

The scripts can be found in the org.mudshark.jack.proj.mtn-metaserver branch on the mudshark.org monotone server. The README file contains setup information.

Note: the db_check runfiles currently don't work, because I'm having trouble authenticating to the usher admin interface in order to shut
down the servers so they can be fscked.

jack | posted Wed Aug 30 15:50:51 2006 | updated Mon Jan 29 22:53:35 2007 | #
category: projects/mtn-metaserver

Backup Maildirs with monotone

On September 19 2005, I posted a couple of scripts to the monotone-devel mailing list for backing up a Maildir with monotone. It only took 11.5 months,
but I now have a public monotone repository up, so I can actually
share the script in a sane manner.

The scripts can be found on the org.mudshark.jack.proj.backup_mail branch on the mudshark.org monotone server. See here for more information.

jack | posted Wed Aug 30 15:17:58 2006 | updated Wed Aug 30 15:18:39 2006 | #
category: projects/backup_mail

Measuring spam filter effectiveness

I wrote a little set of scripts that store spam filtering stats (mail, spam, false negatives, and false positives) in a round
robin database, so I can look at trends over time.

The scripts are intended to work with procmail, mutt, and bogofilter.

Information on the package can be found here.

jack | posted Wed Aug 30 14:35:40 2006 | updated Wed Aug 30 15:01:46 2006 | #
category: projects/count_spam

Single command to pull/checkout with monotone

If you want to do an anonymous pull of something with monotone, a few steps are required:

mtn --db=... db init
mtn --db=... pull ... ...
mtn --db=... co .. ..

I created a little script (apull.sh) that automates this:

Creates the database in a temporary directory
does an anonymous pull
checks out the branch that you pulled
moves the database from the temporary directory into the monotone metadata directory _MTN

You can download a version of it here.

You can also get it from the monotone server on mudshark.org on the org.mudshark.jack.proj.mtn-apull branch.

jack | posted Wed Aug 30 14:25:24 2006 | updated Wed Aug 30 14:28:34 2006 | #
category: projects/mtn-apull

GNU stow and friends

I've been using GNU stow, and lately xstow to manage package
installation on my myriad linux systems:

I like the idea of being able to find out what a package a file comes from simply by doing a ls -l on it.
I like only having to look in one place for libraries, and binaries, includes and man pages.
I like having a sane $PATH.
I like a packaging system to use the filesystem to store information about packages.
I like to be able to fix the system with ln when I mangle it.
I also don't have the guts to go to a full on djb style directory layout.

Today I got fed up. GNU stow is nice, but it depends on perl. xstow is nice, because it can be made into a static binary, but it seems a wee bit bloatish. I
can't seem to get either one to support multiple stow directories for a single
target directory, so directory splitting doesn't work properly. So, I'm going
to write my own:

It will be in bash.
It will not be guaranteed to be stow compatible.
It will support n:1 source to target mappings.

jack | posted Thu Aug 17 16:59:56 2006 | updated Thu Aug 17 17:11:16 2006 | #
category: projects/sstow