I recently found this article: The Techno Maestro’s Amazing Machine, which seems like the coolest thing since sliced bread. A quick summary:

A fellow in Japan named Minato has invented an electric motor that is 80% efficient. As a demonstration of what this means, he can run a standard washer/drier that normally needs a 220 watt outlet can run on 16 watts (that’s a 35kg motor).

Yes, I know it’s impossible. It’s still cool. :)

Everybody knows that Macs are more expensive than PCs. Except Paul Murphy, who begs to differ. He says:

At the low end, therefore, the PC desktops are marginally less expensive than the Macs—if you can do without their connectivity and multimedia capabilities—and considerably more expensive if you can’t. At the very high end, however, all of the design focus is on multimedia processing and the PCs simply aren’t competitive from either hardware or cost perspectives.

But, as he says, the mitigating factor that comes up in most people’s minds is:

The PC community response is, first, that the multimedia features distinguishing the Mac aren’t necessary and, secondly, that the PC is so far ahead of the Mac on speed that the comparisons are pointless anyway.

Paul Murphy discusses this thing as well. (Hint: Macs Kick PC Ass)

I just found this link off of slashdot. The summary is this: a scientist has developed a machine for generating random numbers. This machine appears to be able to be influenced by human thought, and appears to be able to predict (when several are aggregated from around the world) catastrophes several hours before they happen.

I, for one, would love to know how these things work. Random number generation is kinda difficult, and the first of these devices supposedly was developed back in the 70’s. I say: show me the money. But still, the article (mostly fluff, really) claims they have the power of statistics and a lot of respected mathematicians and other scientists behind them. I’m skeptical, but… what a fascinating concept, if true.

Factoids from the history channel:

• 3rd most common element
• usually found as aluminum oxide, and very very VERY hard to separate from the oxygen, thus:
• used to be more expensive than gold
• the Washington Monument was capped with aluminum, because at the time, it was one of the most expensive metals we could lay hands on, but not any more
• how to separate from oxide: mix with molten metal, and run GIGANTIC amounts of electricity through it

I just thought this take on childhood was fascinating. It talks about the historical context of the concept of childhood. For example:

It is around this same time that another change was occurring - this one a shift in attitude. In the early seventeenth century, there did not seem to be the same emphasis on sexual decency and propriety that we see in later years (Aries, 1962). People were not shocked by coarse jokes and sexual games with children (e.g., attempting to grab a child’s genitals). Children under the age of puberty were simply not considered aware of sexual matters. Nobody thought that sexual references could corrupt a child’s innocence because the idea of childish innocence did not yet exist (Aries, 1962).

Something I’ve been nothing, and this obviously isn’t a serious problem on a system as small as the one I administer, is that spam bounces frequently fill up the queue.

What happens is that some spammer sends spam to one of my legitimate users (illegitimate users are rejected right away, which may make gathering legitimate names easy, but avoids illegitimate bounces). Some of my legitimate users have their mail forwarded to verizon.com, which enjoys refusing to accept email. Of course, my policy on MY email server is to accept all mail: my spam filters (e.g. spamassassin) can be wrong and therefore are only advisory. When verizon refuses to accept the mail, my mail server is stuck holding the bag: I can’t deliver it, and I usually can’t bounce it either. I’d like to just dispose of it, but my queue lifetime is 7 days, so I have to keep it in the queue for 7 days while qmail realizes that verizon is never going to accept it, and the return address doesn’t exist.

So here’s my thought: have two qmail installs, one with a queue lifetime of 7 days, one with a queue lifetime of 1 day. Then, put a script in the QMAILQUEUE chain that decides which qmail-queue to use (the 7-day one or the 1-day one) based on whatever I want (i.e. the X-Spam-Score header, or the $RECIPIENT envariable).

Probably won’t happen on the WOPR here, because, of course, people are skittish about that kind of fiddling (read: it’s a production machine). which means it’ll basically never get done. Oh well - but it was a thought.

She Said YES!

or, to put it another way…

Emily and I Are Getting Married!

So, last Friday (the 29th of July), at 11pm, I said to myself “why am I still in South Bend? I could be proposing to Emily right now.” Being unable to come up with a satisfactory answer, I threw a few things in a bag, hopped in my car, and drove through the night, stopping only at McDonald’s and gas stations. (Oh, and I stopped at Meijer’s to pick up 2 dozen roses).

I arrived at Emily’s front door at 1:30 (2:30 her time) the next day (Saturday). She was planning on heading out to the library and then off to her friend Kate’s house to accompany her to a different friend’s birthday party. To keep her in her apartment until I could arrive, I enlisted the help of Rich to keep her online! I arrived seconds before she was going to leave. I knocked on the door, and presented a thoroughly flabbergasted Emily with 2 dozen roses. When we got inside, in the front hallway, I kneeled down, presented the ring, and asked her to marry me. She cried, and said yes! Then we went to her friend’s house.

The night I left, she had said over instant messenger that when I came and visited next (which was planned for two weeks time), we would need to “talk about where this relationship is headed”. After she said yes, I asked her if she was okay with where the relationship was headed. (She just smiled and cried a little more.)

So far, we’re thinking May of 2007. I’m leaning towards SB, but that far out, it may have to move.

I found some ancient fonts I created once. Be a shame for them to be forgotten in the mists of time. So, here they are:

(Mac Win)

(Mac Win)

(Mac Win)

(Mac Win)

(Mac Win)

(Mac Win)

I also found my old “Poetry & Prose Archive”… it’s childish in places, but still was something I invested a lot of time in back in the day. Rather than let it fade, I’m making it live forever, here, unaltered from exactly how it was back in 1998. Let’s pretend it’s not embarrassing. :) By the way, I highly doubt any of the email addresses on that site still work.

Finally, an old MOD site I put together. No comment.

Well, it’s official, I am now a published book author! Of what do I speak? Qmail Quickstarter Yep, seriously, that’s me. It’s even on Amazon!

This has been a kinda under-ground writing project for me so far. I haven’t told a lot of people (including most of my friends) what I have been up to. I think originally I had some reasons for that; stuff like I was unsure whether it would actually pan out (several other qmail-related books have started and died in recent years, and I didn’t want to jinx this one), and I didn’t want to bother with answering questions like “How will you have time for that between writing your proposal, getting married, and doing work for Sandia?” (because of course the answer would have been “I haven’t a clue, but it’s an opportunity I can’t pass up!” — which isn’t a particularly good answer). In retrospect, though, I guess mostly it just felt like bragging.

This project got started back in… wow, apparently, back in August. I got an email out of the blue from a fellow named Nanda Padmanabhan, a Development Editor for Packt Publishing (a company I hadn’t heard of before) in England. He had apparently been subscribed to the qmail mailing list for a while, and wanted to know whether I’d be interested in contributing to a qmail book project that their company was planning on embarking on. Riiiiiiight, I thought. The Nigerian scammers have thought up a new one.

Doing a little research, it turns out that Packt is a real company (surprise!) and has been at least somewhat active in the nerdly fields before: they were mentioned on Slashdot for some coding competitions with cash prizes that they did.

So I exchanged a few emails with Nanda. They were interested in putting together a qmail book that would get a person (of unknown experience) up and running quickly (i.e. in around 150 pages or so). I suggested a few other folks I know that are very qmail-savvy, but was curious why they were interested. After all, there’s several other books out there, including the exhaustive The qmail Handbook by Dave Sill, O’Reilly’s book on the subject by John Levine, and even another (less well-regarded by the qmail community) book by Richard Blum. In fact, on the qmail mailing list, people are often referred to Dave Sill’s book and online reference, http://www.lifewithqmail.org, because they are so thorough and well-done. Who is not being served by these books? Like any good psychologist, Nanda turned that question to me: what do I think might be missing?

My answer was a book with a better sense of the “why” of qmail (i.e. what is qmail’s internal architecture, why is it that way, and how can you use this knowledge?), and examples of the cool tricks you can do with qmail once you understand what’s going on. I had the thought that maybe a book titled “Things I Wish I’d Known About Qmail” or something like that. Dave’s book gives you all you ever wanted to know about the “what” and the “how” of qmail, and is more like a book from the “missing manual” series of books (by Pogue). Nanda also said that people are a bit turned off by the sheer volume of some of these books. They want something quick and light and useful, rather than an exhaustive catalog of every config file and option that qmail has.

This, of course, then spiraled into me brainstorming an outline for what I wished a book would look like, which turned into an official contract shockingly fast (well, shocking for me; it was in October when I got the official contract).

And thus, I began writing. Mostly I packed writing into the weekends, when I could spend all day writing and editing and not be bothered by anyone or anything. This made for a pretty tight schedule. The publisher wanted chapters ASAP (of course), but in general, I got about three weeks for each chapter (according to the contract)… which, because of my goofy schedule of visiting Emily, going to weddings, and holidays, typically meant I got a single weekend to pound out each chapter. Thankfully that was just for the first draft; I got a little more time to race through doing all the editing… but not a lot!

Finally, now, it is done! And you can buy it! (Please do! Buy several!)

I really could not be prouder.

Time to go get married! :D

I’ve been stumbling across a few reviews of my book! Here they are:

Free Software Magazine

I liked many parts of the book: firstly, the diagrams dividing down the underlying responsibilities in a crisp no nonsense approach as exampled by the queuing diagram on page 39. Secondly, the book is not verbose and does what it has to do with no extra embellishments. For a busy system administrator the book is thus more viable than a 500-page manual. Thirdly, I enjoyed the discussion of the Sender Policy Framework (SPF) and also DomainKeys contained within the pages 93-97. Finally, the mentioning of Silly Qmail Syndrome (page 132) and a patch solution should for some of you short cut a degree of pain and potential embarrassment.

AdminSpotting

qmail is a great email system. It’s been around forever — the latest stable version is almost 10 years old. I considered The qmail Handbook to be the only quality book on qmail available. However, it’s now going to have to share the spotlight with the release of Qmail Quickstarter.

As mentioned before, the latest qmail release is almost 10 years old. A lot has happened with email administration since then — for instance with security and filtering. Not only has the qmail community adapted to these changes, but they’re also all covered in Qmail Quickstarter. This book also does a great job at describing the unique architecture to qmail. qmail takes the age-old Unix approach of using small programs to do one thing well. Each of the pieces that make up qmail are explained and illustrated here.

Though I haven’t used qmail in years, I’m glad to see that it hasn’t fallen behind in the world of email servers. On the contrary, it’s alive and well and this book does a great job showing that off. I give it a 10/10.

There are others… one was submitted to slashdot but appears to have been rejected (I may post it in its entirety later; the author sent me a copy).

I just got a link to another review of my book! This time, by a fellow named James Craig Burley, who also frequents the qmail mailing list. Here’s his intro and conclusion:

Email servers, also called Mail Transfer Agents (MTAs), today do much of the heavy lifting required to transport email from sender to recipient, ideally without the sender or recipient being particularly aware of them. They store (queue) incoming email, then forward it to user’s mailboxes, sometimes via other email servers, while often trying to avoid accepting or sending out spam or viruses. They also allow users to read email waiting for them in their mailboxes. qmail is a popular email server for Unix-based systems; “Qmail Quickstarter: Install, Set Up, and Run your own Email Server” introduces the reader to qmail as an email-delivery architecture that provides the building blocks for an email server.

“Qmail Quickstarter” is a good book for anyone wanting to come up to speed on qmail, whether as their email server of choice or as a means to better choose among the many email servers available. I give this book a 7 out of 10.

I recently stumbled across a google tech talk video by a man named Cliff Click Jr. He works for a company named Azul, and he has a blog here This tech talk was all about a lock-free hash table that he’d invented for Azul. The video is here

Lock free hash table? Heck yeah I want a lock free hash table!

Sadly, Cliff’s implementation is Java-only, and relies on some Java memory semantics, but it’s on sourceforge if anyone’s interested.

So, as I began reading up on the subject, I discovered that he’s not the only one interested. In fact, there’s another fellow who has a C-based library here. Only problem? IT’S NOT ACTUALLY LOCK FREE!!! At least, not yet. At the moment it’s a pthread/mutex-based hash table that happens to have all the pthreads stuff ifdef’d out (joy). There are other people out there who talk about it. A fellow from IBM named Maged M. Michael has a paper about how to do lock-free hash tables, and he even has a patent on his particular method, but no implementations appear to be available. Chris Purcell wrote a paper on the topic, which contains pseudocode, but yet again, no implementation.

So it would appear that if I want a lock-free hash table, I’m going to have to implement it myself. But boy, it gets me giddy just thinking about it. :) Pthreads, you’re going down!

There are few things that, over my time using Unix-like systems, I have put more cumulative effort into than into my configuration files. I’ve been tweaking them since the day I discovered them, attempting to make my environment more and more to my liking. I have posted them on my other website (here), but it occurred to me that they’ve gotten sufficiently hoary and complex that a walkthrough might help someone other than myself.

Anyway, my bashrc is first on the list.

The file is divided into several (kinda fuzzy) sections:
- Initialization & Other Setup
- Useful Functions
- Loading System-wide Bashrc
- Behavioral Settings
- Environment Variables
- Character Set Detection
- Aliases
- Tab-completion Options
- Machine-local settings
- Auto-logout

Let’s take them one at a time.

Initialization & Other Setup

Throughout my bashrc, I use a function I define here ( dprint ) to allow me to quickly turn on debugging information, which includes printing the seconds-since-bash-started variable ( SECONDS ) in case something is taking too long and you want to find the culprit. Yes, my bashrc has a debug mode. This is essentially controlled by the KBWDEBUG environment variable. Then, because this has come in useful once or twice, I allow myself to optionally create a ~/.bashrc.local.preload file which is sourced now, before anything else. Here’s the code:

KBWDEBUG=${KBWDEBUG:-no}

function dprint {
if [[ "$KBWDEBUG"  "yes" && "$-"  *i* ]]; then
    #date "+%H:%M:%S $*"
    echo $SECONDS $*
fi
}
dprint alive
if [ -r "${HOME}/.bashrc.local.preload" ]; then
    dprint "Loading bashrc preload"
    source "${HOME}/.bashrc.local.preload"
fi

Useful Functions

This section started with some simple functions for PATH manipulation. Then those functions got a little more complicated, then I wanted some extra functions for keeping track of my config files (which were now in CVS), and then they got more complicated…

You’ll notice something about these functions. Bash (these days) will accept function declarations in this form:

function fname()
{
    do stuff
}

But that wasn’t always the case. To maintain compatability with older bash versions, I avoid using the uselessly cosmetic parens and I make sure that the curly-braces are on the same line, like so:

function fname \
{
    do stuff
}

Anyway, the path manipulation functions are pretty typical — they’re similar to the ones that Fink uses, but slightly more elegant. The idea is based on these rules of PATH variables:

1. Paths must not have duplicate entries
2. Paths are faster if they don’t have symlinks in them
3. Paths must not have “.” in them
4. All entries in a path must exist (usually)

There are two basic path manipulation functions: add_to_path and add_to_path_first. They do predictable things — the former appends something to a given path variable (e.g. PATH or MANPATH or LD_LIBRARY_PATH ) unless it’s already in that path, and the latter function prepends something to the given PATH variable (or, if it’s already in there, moves it to the beginning). Before they add a value to a path, they first check it to make sure it exists, is readable, that I can execute things that are inside it, and they resolve any symlinks in that path (more on that in a moment). Here’s the code (ignore the reference to add_to_path_force in add_to_path for now; I’ll explain shortly):

function add_to_path \
{
    local folder="${2%%/}"
    [ -d "$folder" -a -x "$folder" ] || return
    folder=`( cd "$folder" ; \pwd -P )`
    add_to_path_force "$1" "$folder"
}

function add_to_path_first \
{
    local folder="${2%%/}"
    [ -d "$folder" -a -x "$folder" ] || return
    folder=`( cd "$folder" ; \pwd -P )`
    # in the middle, move to front
    if eval '[[' -z "\"\${$1##*:$folder:*}\"" ']]'; then
        eval "$1=\"$folder:\${$1//:\$folder:/:}\""
        # at the end
    elif eval '[[' -z "\"\${$1%%*:\$folder}\"" ']]'; then
        eval "$1=\"$folder:\${$1%%:\$folder}\""
        # no path
    elif eval '[[' -z "\"\$$1\"" ']]'; then
        eval "$1=\"$folder\""
        # not in the path
    elif ! eval '[[' -z "\"\${$1##\$folder:*}\"" '||' \
      "\"\$$1\"" '==' "\"$folder\"" ']]'; then
        eval "export $1=\"$folder:\$$1\""
    fi
}

Then, because I was often logging into big multi-user Unix systems (particularly Solaris systems) with really UGLY PATH settings that had duplicate entries, often included “.”, not to mention directories that either didn’t exist or that I didn’t have sufficient permissions to read, I added the function verify_path. All this function does is separates a path variable into its component pieces, eliminates “.”, and then reconstructs the path using add_to_path, which handily takes care of duplicate and inaccessible entries. Here’s that function:

function verify_path \
{
    # separating cmd out is stupid, but is compatible
    # with older, buggy, bash versions (2.05b.0(1)-release)
    local cmd="echo \$$1"
    local arg="`eval $cmd`"
    eval "$1=\"\""
    while [[ $arg == *:* ]] ; do
        dir="${arg%:${arg#*:}}"
        arg="${arg#*:}"
        if [ "$dir" != "." -a -d "$dir" -a \
          -x "$dir" -a -r "$dir" ] ; then
            dir=`( \cd "$dir" ; \pwd -P )`
            add_to_path "$1" "$dir"
        fi
    done
    if [ "$arg" != "." -a -d "$arg" -a -x "$arg" -a -r "$arg" ] ;
    then
        arg=`( cd "$arg" ; \pwd -P )`
        add_to_path "$1" "$arg"
    fi
}

Finally, I discovered XFILESEARCHPATH — a path variable that requires a strange sort of markup (it’s for defining where your app-defaults files are for X applications). This wouldn’t work for add_to_path, so I created add_to_path_force that still did duplicate checking but didn’t do any verification of the things added to the path.

function add_to_path_force \
{
    if eval '[[' -z "\$$1" ']]'; then
        eval "export $1='$2'"
    elif ! eval '[[' \
        -z "\"\${$1##*:\$2:*}\"" '||' \
        -z "\"\${$1%%*:\$2}\"" '||' \
        -z "\"\${$1##\$2:*}\"" '||' \
        "\"\${$1}\"" '==' "\"$2\"" ']]'; then
        eval "export $1=\"\$$1:$2\""
    fi
}

I mentioned that I resolved symlinks before adding directories to path variables. This is a neat trick I discovered due to the existence of pwd -P and subshells. pwd -P will return the “real” path to the folder you’re in, with all symlinks resolved. And it does so very efficiently (without actually resolving symlinks — it just follows all the “..” records). Since you can change directories in a subshell (i.e. between parentheses) without affecting the parent shell, a quick way to transform a folder’s path into a resolved path is this: ( \cd "$folder"; pwd -P). I put the backslash in there to use the shell’s builtin cd, just in case I’d somehow lost my mind and aliased cd to something else.

And then, just because it was convenient, I added another function: have, which detects whether a binary is accessible or not:

function have { type "$1" &>/dev/null ; }

Then I had to confront file paths, such as the MAILCAP variable. A lot of the same logic (i.e. add_to_path_force), but entry validation is different:

function add_to_path_file \
{
    local file="${2}"
    [ -f "$file" -a -r "$file" ] || return
    # realpath alias may not be set up yet
    file=`realpath_func "$file"`
    add_to_path_force "$1" "$file"
}

You’ll note the realpath_func line in there. realpath is a program that takes a filename or directory name and resolves the symlinks in it. Unfortunately, realpath is a slightly unusual program; I’ve only ever found it on OSX (it may be on other BSDs). But, with the power of my pwd -P trick, I can fake most of it. The last little piece (resolving a file symlink) relies on a tool called readlink … but I can fake that too. Here are the two functions:

function readlink_func \
{
    if have readlink ; then
        readlink "$1"
    #elif have perl ; then # seems slower than alternative
    #    perl -e 'print readlink("'"$1"'") . "\n"'
    else
        \ls -l "$1" | sed 's/[^>]*-> //'
    fi
}

function realpath_func \
{
    local input="${1}"
    local output="/"
    if [ -d "$input" -a -x "$input" ] ; then
        # All too easy...
        output=`( cd "$input"; \pwd -P )`
    else
        # sane-itize the input to the containing folder
        input="${input%%/}"
        local fname="${input##*/}"
        input="${input%/*}"
        if [ ! -d "$input" -o ! -x "$input" ] ; then
            echo "$input is not an accessible directory" >&2
            return
        fi
        output="`( cd "$input" ; \pwd -P )`/"
        input="$fname"
        # output is now the realpath of the containing folder
        # so all we have to do is handle the fname (aka "input)
        if [ ! -L "$output$input" ] ; then
            output="$output$input"
        else
            input="`readlink_func "$output$input"`"
            while [ "$input" ] ; do
                if [[ $input  /* ]] ; then
                    output="$input"
                    input=""
                elif [[ $input  ../* ]] ; then
                    output="${output%/*/}/"
                    input="${input#../}"
                elif [[ $input  ./* ]] ; then
                    input="${input#./}"
                elif [[ $input  */* ]] ; then
                    output="$output${input%${input#*/}}"
                    input="${input#*/}"
                else
                    output="$output$input"
                    input=""
                fi
                if [ -L "${output%%/}" ] ; then
                    if [ "$input" ] ; then
                        input="`readlink_func "${output%%/}"`/$input"
                    else
                        input="`readlink_func "${output%%/}"`"
                    fi
                    output="${output%%/}"
                    output="${output%/*}/"
                fi
            done
        fi
    fi
    echo "${output%%/}"
}

Loading System-wide Bashrc

This section isn’t too exciting. According to the man page:

When bash is invoked as an interactive login shell, or as a non-interactive shell with the —login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable.

SOME systems have a version of bash that appears not to obey this rule. And some systems put crucial configuration settings in /etc/bashrc (why?!?). And some systems even do something silly like use /etc/bashrc to source ~/.bashrc (I did this myself, once upon a time, when I knew not-so-much). I’ve decided that this behavior cannot be relied upon, so I explicitly source these files myself. The only interesting bit is that I added a workaround so that systems that use /etc/bashrc to source ~/.bashrc won’t get into an infinite loop. There’s probably a lot more potential trouble here that I’m ignoring. But here’s the code:

if [[ -r /etc/bashrc && $SYSTEM_BASHRC != 1 ]]; then
    dprint " - loading /etc/bashrc"
    . /etc/bashrc
    export SYSTEM_BASHRC=1
fi

Behavioral Settings

This is basic stuff, but after you get used to certain behaviors (such as whether * matches . and ..), you often get surprised when they don’t work that way on other systems. Some of this is because I found a system that did it another way by default; some is because I decided I like my defaults and I don’t want to be surprised in the future.

The interactive-shell-detection here is nice. $- is a variable set by bash containing a set of letters indicating certain settings. It always contains the letter i if bash is running interactively. So far, this has been quite backwards-compatible.

shopt -s extglob # Fancy patterns, e.g. +()
# only interactive
if [[ $- == *i* ]]; then
    dprint setting the really spiffy stuff
    shopt -s checkwinsize # don't get confused by resizing
    shopt -s checkhash # if hash is broken, doublecheck it
    shopt -s cdspell # be tolerant of cd spelling mistakes
fi

Environment Variables

There are a slew of standard environment variables that bash defines for you (such as HOSTNAME). There are even more standard environment variables that various programs pay attention to (such as EDITOR and PAGER). And there are a few others that are program-specific (such as PARINIT and CVSROOT).

Before I get going, though, let me show you a secret. Ssh doesn’t like transmitting information from client to server shell… the only reliable way to do it that I’ve found is the TERM variable. So… I smuggle info through that way, delimited by colons. Before I set any other environment variables, first, I find my smuggled information:

if [[ $TERM == *:* && ( $SSH_CLIENT || $SSH_TTY || $SSH_CLIENT2 ) ]] ; then
    dprint "Smuggled information through the TERM variable!"
    term_smuggling=( ${TERM//:/ } )
    export SSH_LANG=${term_smuggling[1]}
    TERM=${term_smuggling[0]}
    unset term_smuggling
fi

I begin by setting GROUPNAME and USER in a standard way:

if [[ $OSTYPE  solaris* ]] ; then
    idout=(`/bin/id -a`)
    USER="${idout[0]%%\)*}"
    USER="${USER##*\(}"
    [[ $USER  ${idout[0]} ]] && USER="UnknownUser"
    GROUPNAME="UnknownGroup"
    unset idout
else
    [[ -z $GROUPNAME ]] && GROUPNAME="`id -gn`"
    [[ -z $USER ]] && USER="`id -un`"
fi

Then some standard things (MAILPATH is used by bash to check for mail, that kind of thing), including creating OS_VER and HOST to allow me to identify the system I’m running on:

# I tote my own terminfo files around with me
[ -d ~/.terminfo ] && export TERMINFO=~/.terminfo/
[ "$TERM_PROGRAM" == "Apple_Terminal" ] && \
    export TERM=nsterm-16color

MAILPATH=""
MAILCHECK=30
add_to_path_file MAILPATH /var/spool/mail/$USER
add_to_path MAILPATH $HOME/Maildir/
[[ -z $MAILPATH ]] && unset MAILCHECK
[[ -z $HOSTNAME ]] && \
    export HOSTNAME=`/bin/hostname` && echo 'Fake Bash!'
HISTSIZE=1000
HOST=${OSTYPE%%[[:digit:]]*}
OS_VER=${OSTYPE#$HOST}
[ -z "$OS_VER" ] && OS_VER=$( uname -r )
OS_VER=(${OS_VER//./ })
TTY=`tty`
PARINIT="rTbgq B=.,?_A_a P=_s Q=>|}+"

export USER GROUPNAME MAILPATH HISTSIZE OS_VER HOST TTY PARINIT

I’ve also gotten myself into trouble in the past with UMASK being set improperly, so it’s worth setting manually. Additionally, to head off trouble, I make it hard to leave myself logged in as root on other people’s systems accidentally:

if [[ $GROUPNAME == $USER && $UID -gt 99 ]]; then
    umask 002
else
    umask 022
fi

if [[ $USER == root ]] ; then
    [[ $SSH_CLIENT || $SSH_TTY || $SSH_CLIENT2  ]] && \
        export TMOUT=600 || export TMOUT=3600
fi

if [[ -z $INPUTRC && ! -r $HOME/.inputrc && -r /etc/inputrc ]];
then
    export INPUTRC=/etc/inputrc
fi

It is at this point that we should pause and load anything that was in /etc/profile, just in case it was left out (and, if its in there, maybe it should override what we’ve done so far):

export BASHRCREAD=1

if [[ -r /etc/profile && -z $SYSTEM_PROFILE ]]; then
    dprint "- loading /etc/profile ... "
    . /etc/profile
    export SYSTEM_PROFILE=1
fi

Now I set my prompt (but only if this is an interactive shell). The idea is that, if I’m logged into another system, I want to see how long I’ve been idle. This works out well:

if [[ $- == *i* ]]; then
    if [[ $SSH_CLIENT || $SSH_TTY || $SSH_CLIENT2 ]] ; then
        PS1='(\d \T)\n[\u@\h \W]\$ '
    else
        PS1='[\u@\h \W]\$ '
    fi
fi

Now I set up the various paths. Note that it doesn’t matter if these paths don’t exist; they’ll be checked and ignored if they don’t exist:

verify_path PATH
add_to_path PATH "/usr/local/sbin"
add_to_path PATH "/usr/local/teTeX/bin"
add_to_path PATH "/usr/X11R6/bin"
add_to_path PATH "$HOME/bin"
add_to_path_first PATH "/sbin"

add_to_path_first PATH "/bin"
add_to_path_first PATH "/usr/sbin"
add_to_path_first PATH "/opt/local/bin"
add_to_path_first PATH "/usr/local/bin"

if [[ $OSTYPE == darwin* ]] ; then
    add_to_path PATH "$HOME/.conf/darwincmds"

    # The XFILESEARCHPATH (for app-defaults and such)
    # is a wonky kind of path
    [ -d /opt/local/lib/X11/app-defaults/ ] && \
        add_to_path_force XFILESEARCHPATH \
            /opt/local/lib/X11/%T/%N
    [ -d /sw/etc/app-defaults/ ] && \
        add_to_path_force XFILESEARCHPATH /sw/etc/%T/%N
    add_to_path_force XFILESEARCHPATH /private/etc/X11/%T/%N
fi

verify_path MANPATH
add_to_path MANPATH "/usr/man"
add_to_path MANPATH "/usr/share/man"
add_to_path MANPATH "/usr/X11R6/man"
add_to_path_first MANPATH "/opt/local/share/man"
add_to_path_first MANPATH "/opt/local/man"
add_to_path_first MANPATH "/usr/local/man"
add_to_path_first MANPATH "/usr/local/share/man"

verify_path INFOPATH
add_to_path INFOPATH "/usr/share/info"
add_to_path INFOPATH "/opt/local/share/info"

And now there are STILL MORE environment variables to set. This final group may rely on some of the previous paths being set (most notably, PATH).

export PAGER='less'
have vim && export EDITOR='vim' || export EDITOR='vi'
if [[ -z $DISPLAY && $OSTYPE  darwin* ]]; then
    processes=`ps ax`
    # there are double-equals here, even though they don't show
    # on the webpage
    if [[ $processes  *xinit* || $processes  *quartz-wm* ]]; then
        export DISPLAY=:0
    else
        unset DISPLAY
    fi
fi
if [[ $HOSTNAME  wizard ]] ; then
    dprint Wizards X forwarding is broken
    unset DISPLAY
fi
export TZ="US/Central"
if [ "${BASH_VERSINFO[0]}" -le 2 ]; then
    export HISTCONTROL=ignoreboth
else
    export HISTCONTROL="ignorespace:erasedups"
fi
export HISTIGNORE="&:ls:[bf]g:exit"
export GLOBIGNORE=".:.."
export CVSROOT=kyle@cvs.memoryhole.net:/home/kyle/cvsroot
export CVS_RSH=ssh
export BASH_ENV=$HOME/.bashrc
add_to_path_file MAILCAPS $HOME/.mailcap
add_to_path_file MAILCAPS /etc/mailcap
add_to_path_file MAILCAPS /usr/etc/mailcap
add_to_path_file MAILCAPS /usr/local/etc/mailcap
export EMAIL='kyle-envariable@memoryhole.net'
export GPG_TTY=$TTY
export RSYNC_RSH="ssh -2 -c arcfour -o Compression=no -x"
if [ -d /opt/local/include -a -d /opt/local/lib ] ; then
    export CPPFLAGS="-I/opt/local/include $CPPFLAGS"
    export LDFLAGS="-L/opt/local/lib $LDFLAGS"
fi
if have glibtoolize ; then
    have libtoolize || export LIBTOOLIZE=glibtoolize
fi

One little detail that I rather like is the fact that xterm’s window title often tells me exactly what user I am on what machine I am, particularly when I’m ssh’d into another host. This little bit of code ensures that this happens:

if [[ $TERM  xterm* || $OSTYPE  darwin* ]]; then
    export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME/.*/}: ${PWD/${HOME}/~}\007"'
else
    unset PROMPT_COMMAND
fi

Character Set Detection

I typically work in a UTF-8 environment. MacOS X (my preferred platform for day-to-day stuff) has made this pretty easy with really excellent UTF-8 support, and Linux has come a long way (to near-parity) in the last few years. Most of my computing is done via a uxterm (aka. xterm with UTF-8 capability turned on), but I also occasionally dabble in other terminals (sometimes without realizing it). Despite the progress made, however, not all systems support UTF-8, and neither do all terminals. Some systems, including certain servers I’ve used, simply don’t have UTF-8 support installed, even though they’re quite capable of it.

The idea is that the LANG environment variable is supposed to reflect the language and character set you’re using. So, this is where I try and figure out what LANG should be.

The nifty xprop trick here is from a vim hint I found. I haven’t tried it out for very long, but so far it seems to be a really slick way of finding out what sort of environment your term is doing, even if it hasn’t set the right environment variables (e.g. LANG).

One of the more annoying details of this stuff is that ssh doesn’t pass LANG (or any other locale information) along when you connect to a remote server. Granted, there are good reasons for this (just because my computer is happy when LANG=en_US.utf-8 doesn’t mean any server I connect to would be). But remember how I smuggled that info through and stuck it in the SSH_LANG variable? Here’s where it becomes important.

As a final note here, I discovered that less is capable of handling multibyte charsets (at least, recent versions of it are), but for whatever reason it doesn’t always support LANG and other associated envariables. It DOES however support LESSCHARSET…

Anyway, here’s the code:

if [[ -z $LC_ALL && -z $LC_CTYPE && -z $LANG ]] ; then
    dprint no LC_ALL or LC_CTYPE or LANG
    if [[ $WINDOWID ]] && have xprop ; then
        dprint querying xprop
        __bashrc__wmlocal=(`xprop -id $WINDOWID -f WM_LOCALE_NAME 8s ' $0' -notype WM_LOCALE_NAME`)
        export LANG=`eval echo ${__bashrc__wmlocal[1]}`
        unset __bashrc__wmlocal
    elif [[ $OSTYPE  darwin* ]] ; then
        dprint "I'm on Darwin"
        if [[ ( $SSH_LANG && \
            ( $SSH_LANG  *.UTF* || $SSH_LANG  *.utf* ) || \
            $TERM_PROGRAM  Apple_Terminal ) && \
            -d "/usr/share/locale/en_US.UTF-8" ]] ; then
            export LANG='en_US.UTF-8'
        elif [ -d "/usr/share/locale/en_US" ] ; then
            export LANG='en_US'
        else
            export LANG=C
        fi
    elif [[ $TERM  linux || $TERM_PROGRAM  GLterm ]] ; then
        if [ -d "/usr/share/locale/en_US" ] ; then
            export LANG='en_US'
        else
            export LANG=C # last resort
        fi
    else
        if have locale ; then
            locales=`locale -a`
            case "$locales" in
                *en_US.utf8[[:space:]]*|*en_US.utf8)
                export LANG='en_US.utf8'
                export LESSCHARSET=utf-8
                ;;
                *en_US.utf-8[[:space:]]*|*en_US.utf-8)
                export LANG='en_US.utf-8'
                export LESSCHARSET=utf-8
                ;;
                *en_US[[:space:]]*|*en_US)
                export LANG='en_US'
                ;;
                *)
                export LANG=C
                unset LESSCHARSET
                ;;
            esac
            unset locales
        fi
    fi
else
    dprint LANG IS ALREADY SET! $LANG
fi

Aliases

This is where a lot of the real action is, in terms of convenience settings. Like anyone who uses a computer every day, I type a lot; and if I can avoid it, so much the better. (I’m a lazy engineer.)

Sometimes I can’t quite get what I want out of an alias. In csh aliases can specify what to do with their arguments. In bash, aliases are really more just shorthand — “pretend I really typed this” kind of stuff. Instead, if you want to be more creative with argument handling, you have to use functions (it’s not a big deal, really). Here’s a few functions I added just because they’re occasionally handy to have the shell do for me:

function exec_cvim {
/Applications/Vim.app/Contents/MacOS/Vim -g "$@" &
}

function darwin_locate { mdfind "kMDItemDisplayName  '$@'wc"; }
if [[ $-  *i* && $OSTYPE == darwin* && ${OS_VER[0]} -ge 8 ]] ;
then
alias locate=darwin_locate
fi

function printargs { for F in "$@" ; do echo "$F" ; done ; }
function psq { ps ax | grep -i $@ | grep -v grep ; }
function printarray {
for ((i=0;$i<`eval 'echo ${#'$1'[*]}'`;i++)) ; do
    echo $1"[$i]" = `eval 'echo ${'$1'['$i']}'`
done
}
alias back='cd $OLDPWD'

There are often a lot of things that I just expect to work. For example, when I type “ls”, I want it to print out the contents of the current directory. In color if possible, without if necessary. It often annoys me, on Solaris systems, when the working version of ls is buried in the path, while a really lame version is up in /bin for me to find first. Here’s how I fix that problem:

# GNU ls check
if [[ $OSTYPE  darwin* ]]; then
    dprint "- DARWIN ls"
    alias ls='/bin/ls -FG'
    alias ll='/bin/ls -lhFG'
elif have colorls ; then
    dprint "- BSD colorls"
    alias ls='colorls -FG'
    alias ll='colorls -lhFG'
else
    __kbwbashrc__lsarray=(`\type -ap ls`)
    __kbwbashrc__lsfound=no
    for ((i=0;$i<${#__kbwbashrc__lsarray[*]};i=$i+1)) ; do
        if ${__kbwbashrc__lsarray[$i]} --version &>/dev/null ;
        then
            dprint "- found GNU ls: ${__kbwbashrc__lsarray[$i]}"
            alias ls="${__kbwbashrc__lsarray[$i]} --color -F"
            alias ll="${__kbwbashrc__lsarray[$i]} --color -F -lh"
            __kbwbashrc__lsfound=yes
            break
        fi
    done
    if [ "$__kbwbashrc__lsfound"  no ] ; then
        if ls -F &>/dev/null ; then
            dprint "- POSIX ls"
            alias ls='ls -F'
            alias ll='ls -lhF'
        else
            alias ll='ls -lh'
        fi
    fi
    unset __kbwbashrc__lsarray __kbwbashrc__lsfound
fi

Similar things are true of make and sed and such. I’ve gotten used to GNU’s version, and if they exist on the machine I’d much rather automatically use them than have to figure out whether it’s really called gnused or gsed or justtowasteyourtimesed all by myself:

if [[ $OSTYPE == linux* ]] ; then
    # actually, just Debian, but this works for now
    alias gv="gv --watch --antialias"
else
    alias gv="gv -watch -antialias"
fi
if have gsed ; then
    alias sed=gsed
elif have gnused ; then
    alias sed=gnused
fi
if have gmake ; then
    alias make=gmake
elif have gnumake ; then
    alias make=gnumake
fi

The rest of them are mostly boring, with one exception:

alias macfile="perl -e 'tr/\x0d/\x0a/'"
have tidy && alias tidy='tidy -m -c -i'
have vim && alias vi='vim'
alias vlock='vlock -a'
alias fastscp='scp -c arcfour -o Compression=no' # yay speed!
alias startx='nohup ssh-agent startx & exit'
alias whatlocale='printenv | grep ^LC_'
alias fixx='xauth generate $DISPLAY'
alias whatuses='fuser -v -n tcp'
alias which=type
alias ssh='env TERM="$TERM:$LANG" ssh'
have realpath || alias realpath=realpath_func
if have readlink ; then
    unset -f readlink_func
else
    alias readlink=readlink_func
fi
if [[ $OSTYPE == darwin* ]]; then
    alias top='top -R -F -ocpu -Otime'
    alias cvim='exec_cvim'
    alias gvim='exec_cvim'
fi

Did you note that ssh alias? Heh.

Tab-completion Options

Bash has had, for a little while at least, the ability to do custom tab-completion. This is really convenient (for example, when I’ve typed cvs commit and I hit tab, bash can know that I really just want to tab-complete files that have been changed). However, I won’t bore you with a long list of all the handy tab-completions that are out there. Most of mine are just copied from here anyway. But I often operate in places where that big ol’ bash-completion file can be in multiple places. Here’s the simple little loop I use. You’ll notice that it only does the loop after ensuring that bash is of recent-enough vintage:

completion_options=(
~/.conf/bash_completion
/etc/bash_completion
/opt/local/etc/bash_completion
)
if [[ $BASH_VERSION && -z $BASH_COMPLETION && $- == *i* ]] ;
then
    bash=${BASH_VERSION%.*}; bmajor=${bash%.*}; bminor=${bash#*.}
    if [ $bmajor -eq 2 -a $bminor '>' 04 ] || [ $bmajor -gt 2 ] ;
    then
        for bc in "${completion_options[@]}" ; do
            if [[ -r $bc ]] ; then
                dprint Loading the bash_completion file
                if [ "$BASH_COMPLETION" ] ; then
                    BASH_COMPLETION="$bc"
                fi
                #COMP_CVS_REMOTE=yes
                export COMP_CVS_ENTRIES=yes
                source "$bc"
                break
            fi
        done
    fi
    unset bash bminor bmajor
fi
unset completion_options

Machine-local settings

You’d be surprised how useful this can be sometimes. Sometimes I need machine-specific settings. For example, on some machines there’s a PGI compiler I want to use, and maybe it needs some environment variable set. Rather than put it in the main bashrc, I just put that stuff into ~/.bashrc.local and have it loaded:

dprint checking for bashrc.local in $HOME
if [ -r "${HOME}/.bashrc.local" ]; then
    dprint Loading local bashrc
    source "${HOME}/.bashrc.local"
fi

Auto-logout

Lastly, it is sometimes the case that the TMOUT variable has been set, either by myself, or by a sysadmin who doesn’t like idle users (on a popular system, too many idle users can unnecessarily run you out of ssh sockets, for example). In any case, when my time is limited, I like being aware of how much time I have left. So I have my bashrc detect the TMOUT variable and print out a big banner so that I know what’s up and how much time I have. Note that bash can do simple math all by itself with the $(( )) construction. Heheh. Anyway:

if [[ $TMOUT && "$-" == *i* ]]; then
    echo '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'
    echo You will be autologged out after:
    echo -e -n '\t'
    seconds=$TMOUT
    days=$((seconds/60/60/24))
    seconds=$((seconds-days*24*60*60))
    hours=$((seconds/60/60))
    seconds=$((seconds-hours*60*60))
    minutes=$((seconds/60))
    seconds=$((seconds-minutes*60))
    [[ $days != 0 ]] && echo -n "$days days "
    [[ $hours != 0 ]] && echo -n "$hours hours "
    [[ $minutes != 0 ]] && echo -n "$minutes minutes "
    [[ $seconds != 0 ]] && echo -n "$seconds seconds "
    echo
    echo ... of being idle.
    unset days hours minutes seconds
fi

dprint BASHRC_DONE

While I’m at it, I suppose I should point out that I also have a ~/.bash_logout file that’s got some niceness to it. If it’s the last shell, it clears sudo’s cache, empties the console’s scrollback buffer, and clears the screen. Note: DO NOT PUT THIS IN YOUR BASHRC You wouldn’t like it in there.

if [ "$SHLVL" -eq 1 ] ; then
    sudo -k
    type -P clear_console &>/dev/null && clear_console 2>/dev/null
    clear
fi

And that’s about it! Of course, I’m sure I’ll add little details here and there and this blog entry will become outdated. But hopefully someone finds my bashrc useful. I know I’ve put a lot of time and effort into it. :)

Branden and I had an idea to help with the spam problem on our system, and it’s proven particularly effective. How effective? Here’s the graphs from the last year of email on my system. Can you tell when I started using the system?

If you want to see the live images, check here.

The idea is based on the following observations: certain addresses on my domain ONLY get spam. This is generally because they either don’t exist or because I stopped using them; for example, spammers often send email to buy@memoryhole.net. Branden and I also both use the user-tag@domain scheme, so we get a lot of disposable addresses that way. These addresses are such that we know for certain that anyone sending email to them is a spammer. Some of these addresses were already being rejected as invalid; some we hadn’t gotten around to invalidating yet.

By simply rejecting emails sent to those addresses, we were able to reduce the spam load of our domains by a fair bit, and the false-positive rate is nil. But we took things a step further: since spammers rarely send only one message, often they will send spam to both invalid AND valid addresses.

If I view those known-bad addresses as, essentially, honeypots, I can say: aha! Any IP sending to a known-bad address is a spammer, and I can refuse (with a permanent fail) any email from that IP for some short time. I started with 5 minutes, but have moved to an exponentially increasing timeout system. Each additional spam increased the length of the timeout (5 minutes for the first spam, 6 for the second, 8 for the third, and so on). Longer-term bans, as a result of the exponentially increasing timeout, are made more efficient via the equivalent of /etc/hosts.deny. I haven’t gotten into the maintaining-my-spammer-database much yet, but I think this may not be terribly important (I’ll explain in a moment).

One of the best parts of the system is that it is fast: new spammers that identify themselves by sending to honeypot addresses get blocked quickly and without my intervention. So far this has been particularly helpful in eliminating spam spikes. Another feature that I originally thought would be useful, but hasn’t really appeared to be (yet) is that it allows our multiple domains to share information about spam sources. Thus far, however, our domains seem to be plagued by different spammers.

Now, interestingly, about a week after we started using the system, our database of known spammers was wiped out (it’s kept in /tmp, and we rebooted the system). Result? No noticeable change in effectiveness. How’s that for a result? And, as you can see from the graph above, there’s no obvious change in spam blocking over the course of a month that would indicate that the long-term history is particularly useful. So, it may be sufficient to keep a much shorter history. Maybe only a week is necessary, maybe two weeks, I haven’t decided yet (and, as there hasn’t yet been much of a speed penalty for it, there’s no pressure to establish a cutoff). But, given that most spam is sent from botnets with dynamic IPs, this isn’t a particularly surprising behavior.

Forkit.org and memoryhole.net have been using this filter for a month so far. The week before we started using this filter, memoryhole.net averaged around 262 emails per hour. The week after instituting this filter, the average was around 96 per hour (a 60+% reduction!). Before using the filter, forkit.org averaged 70 emails per hour; since starting to use the filter, that number is down to 27.4 per hour (also a 60+% reduction). We have recorded spams from over 33,000 IPs, most of which only ever sent one or two spams. We typically have between 100 and 150 IPs that are “in jail” at any one time (at this moment: 143), and most of those (at this moment 134) are blocked for sending more than ten spams (114 of them have a timeout measured in days rather than minutes).

Now, granted, I know that by simply dropping 60% of all connections we’d get approximately the same results. But I think our particular technique is superior to that because it’s based on known-bad addresses. Anyone who doesn’t send to invalid addresses will never notice the filter.

The biggest potential problem that I can see with this system is that of spammers who have taken over a normally friendly host, such as Gmail spam. I’ve waffled on this potential problem: on the one hand, Gmail has so many outbound servers that it’s unlikely to get caught (a couple bad emails won’t have much of a penalty). Thus far, I’ve seen a few yahoo servers in Japan sending us spam, but no Gmail servers. On the other hand, as long as I simply use temporary failures (at least for good addresses), and as long as ND doesn’t retry in the same order every time, messages will get through.

I’ve also begun testing a “restricted sender” feature to work with this. For example, I have the address kyle-slashdot@memoryhole.net that I use exclusively for my slashdot.org account. The only people who are allowed to send to that email address is slashdot.org (i.e. if I forget my password). If anyone from any other domain attempts that address, well, then I know that sending IP is a spammer and I can treat it as if it was a known-bad address. Not applicable to every email address, obviously, but it’s a start.

It’s been pointed out that this system is, in some respects, a variant on greylisting. The major difference is that it’s a penalty-based system, rather than a “prove yourself worthy by following the RFC” system, and I like that a bit better. I’m somewhat tempted to define some bogus address (bogus@memoryhole.net) and sign it up for spam (via spamyourenemies.com or something similar), but given that part of the benefit here is due to spammers trying both valid and invalid addresses, I think it would probably just generate lots of extra traffic and not achieve anything particularly useful.

Now, this technique is simply one of many; it’s not sufficient to guarantee a spam-free inbox. I use it in combination with several other antispam techniques, including a greet-delay system and a frequently updated SpamAssassin setup. But check out the difference it’s made in our CPU utilization:

Okay, so, grand scheme of things: knocking the CPU use down three percentage points isn’t huge, but knocking it down by 50%? That sounds better, anyway. And as long as it doesn’t cause problems by making valid email disappear (possible, but rather unlikely), it seems to me to be a great way to cut my spam load relatively easily.