Five years later, in 2007, the GNU libc folks said to themselves “dang, those BSD guys are smart, I wish we had qsort_r()”. Then some idiot said: WAIT! We cannot simply use the same prototype as the BSD folks; they use an evil license! We can’t put that into GPL’d code! So the GNU libc folks solved the problem by reordering the arguments.

And now we have two, incompatible, widely published versions of qsort_r(), which both do the exact same thing: crash horribly if you use the wrong argument order.

<sigh>

Okay, here’s an alternate history:

… Then some lazy idiot said: WAIT! The existing qsort_r() is a poor design that requires a second implementation of qsort()! If we throw out compatibility with existing qsort_r() code, we can implement qsort() as a call to qsort_r() and no one will ever know!

<sigh>

Either way, we all lose.

(I have no argument with the alternate history point… but why’d you have to call it the exact same thing??? Call it qsort_gnu() or something! Make it easy to detect the difference!)

Subtle Bugs in My Code

Let’s start with the minor mistakes it found in my code that other compilers hadn’t complained about:

static inline uint64_t qthread_incr64(
           volatile uint64_t *operand, const int incr)
{
  union {
    uint64_t i;
    struct {
      uint32_t l, h;
    } s;
  } oldval, newval;
  register char test;
  do {
    oldval.i = *operand;
    newval.i = oldval.i + incr;
    __asm__ __volatile__ ("lock; cmpxchg8b %1\n\t setne %0"
        :"=r"(test)
        :"m"(*operand),
         "a"(oldval.s.l),
         "d"(oldval.s.h),
         "b"(newval.s.l),
         "c"(newval.s.h)
        :"memory");
  } while (test);
  return oldval.i;
}

Seems fairly straightforward, right? Works fine on most compilers, but the PGI compiler complains that “%sli” is an invalid register. Really obvious error, right? Right? (I don’t really know what the %sli register is for either). Turns out that because setne requires a byte-sized register, I need to tell the compiler that I want a register that can be byte-sized. In other words, that "=r" needs to become "=q". Fair enough. It’s a confusing error, and thus annoying, but I am technically wrong (or at least I’m providing an incomplete description of my requirements) here so I concede the ground to PGI.

Unnecessary Pedantry

And then there are places where PGI is simply a bit more pedantic than it really needs to be. For example, it generates an error when you implicitly cast a volatile struct foo * into a void * when calling into a function. Okay, yes, the pointers are different, but… most compilers allow you to implicitly convert just about any pointer type into a void * without kvetching, because you aren’t allowed to dereference a void pointer (unless you cast again, and if you’re casting, all bets are off anyway), thus it’s a safe bet that you want to work on the pointer rather than what it points to. Yes, technically PGI has made a valid observation, but I disagree that their observation rises to the level of “warning-worthy” (I have no argument if they demote it to the sort of thing that shows up with the -Minform=inform flag).

Flat-out Broken

But there are other places where PGI is simply wrong/broken. For example, if I have (and use) a #define like this:

#define PARALLEL_FUNC(initials, type, shorttype, category) \
type qt_##shorttype##_##category (type *array, size_t length, int checkfeb) \
{ \
  struct qt##initials arg = { array, checkfeb }; \
  type ret; \
  qt_realfunc(0, length, sizeof(type), &ret, \
    qt##initials##_worker, \
    &arg, qt##initials##_acc, 0); \
  return ret; \
}
PARALLEL_FUNC(uis, aligned_t, uint, sum);

PGI will die! Specifically, it complains that struct qtuisarg does not exist, and that an identifier is missing. In other words, it blows away the whitespace following initials so that this line:

struct qt##initials arg = { array, checkfeb }; \

is interpreted as if it looked like this:

struct qt##initials##arg = { array, checkfeb }; \

But at least that’s easy to work around: rename the struct so that it has a _s at the end! Apparently PGI is okay with this:

struct qt##initials##_s arg = { array, checkfeb }; \

::sigh:: Stupid, stupid compiler. At least it can be worked around.

Thwarting The Debugger

PGI also bad at handling static inline functions in headers. How bad? Well, first of all, the DWARF2 symbols it generates (the default) are incorrect. It gets the line-numbers right but the file name wrong. For example, if I have an inline function in qthread_atomics.h on line 75, and include that header in qt_mpool.c, and then use that function on line 302, the DWARF2 symbols generated will claim that the function is on line 75 of qt_mpool.c (which isn’t even correct if we assume that it’s generating DWARF2 symbols based on the pre-processed source! and besides which, all the other line numbers are from non-pre-processed source). You CAN tell it to generate DWARF1 or DWARF3 symbols, but then it simply leaves out the line numbers and file names completely. Handy, no?

Everyone Else is Doing It…

Here’s another bug in PGI… though I suppose it’s my fault for outsmarting myself. So, once upon a time, I (think I) found that some compilers require assembly memory references to be within parentheses, while others require them to be within brackets. Unfortunately I didn’t write down which ones did what, so I don’t remember if I was merely being over-cautious in my code, or if it really was a compatibility problem. Nevertheless, I frequently do things like this:

atomic_incr(volatile uint32_t *op, const int incr) {
  uint32_t retval = incr;
  __asm__ __volatile__ ("lock; xaddl %0, %1"
    :"=r"(retval)
    :"m"(*op), "0"(retval)
    :"memory");
  return retval;
}

Note that weird "m"(*op) construction? That was my way of ensuring that the right memory reference syntax was automatically used, no matter what the compiler thought it was. So, what does PGI do in this instance? It actually performs the dereference! In other words, it behaves as if I had written:

atomic_incr(volatile uint32_t *op, const int incr) {
  uint32_t retval = incr;
  __asm__ __volatile__ ("lock; xaddl %0, (%1)"
    :"=r"(retval)
    :"r"(*op), "0"(retval)
    :"memory");
  return retval;
}

when what I really wanted was:

atomic_incr(volatile uint32_t *op, const int incr) {
  uint32_t retval = incr;
  __asm__ __volatile__ ("lock; xaddl %0, (%1)"
    :"=r"(retval)
    :"r"(op), "0"(retval)
    :"memory");
  return retval;
}

See the difference? Again, it’s not hard to fix so that PGI does the right thing. And maybe I was being too clever in the first place. But dagnabit, my trick should work! And, more pointedly, it DOES work on other compilers (gcc and icc at the bare minimum, and I’ve tested similar things with xlc).

Note that snopes.com has their own page discussing this video. They don’t address the accuracy of the facts presented, but its an interesting read nevertheless.

It is probably true that there is an aspect of evolution to religion. If we think of religion as a gene, the dominant religion will be defined (over the long run) by the extent to which it benefits or promotes reproduction, just as any piece of DNA does. Of course, given evangelism, it may be more apt to think of religion as similar to a virus that gets passed from person to person, but I doubt that’d be a popular viewpoint. In any case, of course, religion isn’t a gene, because we’d have to assume that people stick with the religion they’re born with. Christianity started with just 12 Christians, after all, and every last one of them was male (and thus couldn’t pass on any genes without help from an originally non-Christian woman).

That said, I’m skeptical of this video’s claims. For one thing, practically no sources are cited (2.11 is the bare minimum to maintain a society? Says who? How do we know?). For another, it’s interesting to note what gets left out. For example, the video says the Muslim population skyrocketed from 82,000 in the UK to several million, but according to the CIA World Factbook, Muslims are a whopping 2.7% of the UK population. In the Netherlands, the CIA World FactBook says that Muslims are a crushing 5.8% of the population. Not only that, but it says that the Netherlands have a fertility rate of 1.66. Woo! Scary!

Now that I come to think of it… let me look this up.

Yikes - they didn’t get a single one correct! The closest was England, and even there they rounded in the wrong direction.

So should we really believe that French Muslims have a birth rate of 8.1? Think about that, EIGHT kids on average, which means that for every childless Muslim woman, there’s another out there with SIXTEEN KIDS. That sounds totally plausible, right?

The Population Reference Bureau (who I’ve never heard of, but they were linked to by about.com) says that in Austria, Muslim women had a fertility rate of 3.1 in 1981, but that by 2001 the rate was a mere 2.3 (apparently they didn’t get the memo from their French kindred). That reflects the falling fertility rates in Muslim countries the immigrants came from. For example, in Turkey the fertility rate dropped from 3.3 in 1985 to 2.2 in 2003. According to the CIA World FactBook, Turkey’s current fertility rate is 2.21. In Morocco it fell from 4.5 to 2.5 in the same time period (CIA says 2.51). And get this: in Iran, it fell from 5.6 to 2.1 in 2003. The CIA World FactBook currently pegs the Iranian fertility rate at a paltry 1.71!

1.71! 1.71! The Iranian culture cannot survive! The US fertility rate is 2.05! We will crush them with our progeny! MUAHAHAHAHA!

Oh, wait, does that not serve the purpose of getting people riled up?

I’m thinking of a word… fearmongering! That’s the one.

It reminds me a lot of grade school. I remember playing with a bunch of computer “simulations” that showed that the world population was exploding and that we’d run out of food by the turn of the millenium. The reason they were wrong is that they made assumptions without realizing it. For example, they assumed that food production would stay constant, and that fertility rates would stay constant. Surprise! They didn’t.

And THAT reminds me of another quote:

Scientists have shown that the moon is moving away at a tiny, although measurable distance from the earth every year. If you do the math, you can calculate that 65 million years ago, the moon was orbiting at a distance of about 35 feet from the earth’s surface. This would explain the death of the dinosaurs … the tallest ones, anyway.

Don’t believe everything you see on YouTube. ;)

I’ve got a SlingLink Turbo that I use for connecting my Macs upstairs to my cable modem downstairs. I went with a network-over-powerline option, because I’ve been having all kinds of intermittent interference problems with my wireless connectivity. So, rather than running an extra-long patch cable up the stairs and taping it down to the carpet, I went the SlingLink route. It seems to be designed specifically for SlingBox applications, but it forwards plain ol’ ethernet signals, and it’s about $40 cheaper than the NetGear equivalents. Huzzah for getting a bargain!

First impression: fabulous! I went along happily for weeks, enjoying my newfound reliable network connection. Then I tried downloading the latest Ubuntu ISO images via BitTorrent, and within five to ten minutes, the internet connection went offline. If I went downstairs and turned the cable modem off-and-on again, the internet would come back. For five to ten minutes. Then it’d go down again.

Surely, I say to myself, that’s a cable-modem problem, right?

I had to have the tech guys from Time Warner’s Cable group come out (twice!) before I finally figured out that it wasn’t their fault (the first time they said they replaced the splitter, and presto, the network was fine! I didn’t go after the ISO again for a while so…). Turns out I didn’t need to restart the cable modem, all I had to do was restart the SlingLink node and I’d get another five to ten minutes out of it. But it ONLY happens when BitTorrent is running; otherwise, the network connection is rock solid!

Weird, no?

So, to experiment, I tried limiting the BitTorrent connections: no dice. Then I tried limiting the BitTorrent bandwidth and all of a sudden the network would stay up. Somewhere between 100Kb/s and 150Kb/s is the cutoff. Something about BitTorrent’s bandwidth seems to either confuse the SlingLink node OR triggers some sort of antiviral cutoff in the SlingLink hardware (either way is annoying). For the record, it’s not a pure bandwidth issue: I can transfer files over the SlingLink network at speeds of over 400Kb/s. As soon as I introduce BitTorrent, though… down she goes.

Maybe it’s a packet-size issue. Maybe it’s a connection-tracking issue. I have no idea. But at least now I know that SlingLink has its limitations. And now, so do you.

There would appear to be some truth to this idea, but not like you’d think (surprise! it’s a complicated subject). Congress has indeed encouraged banks to loan money to poor folks, starting with the Community Reinvestment Act (CRA), enacted in 1977, under Carter’s administration. The original idea was to ban a practice known as “redlining”, which essentially was a practice of defining financial no-fly-zones. Anyone who wanted a loan who happened to live within the redlined area wouldn’t be considered by any bank. This resulted in de-facto slums, among other things. Removing the problem is a touchy one, and you’d think would require very specific rules. But rather than be very specific, the law as enacted used very vague language (apparently to pacify critics of the bill who complained that it would be a train-wreck if it required banks to lend to people who couldn’t afford it). According to Wikipedia:

Congress included little prescriptive detail and simply directs the banking regulatory agencies to ensure that banks and savings associations serve the credit needs of their local communities in a safe and sound manner.

With the Financial Institutions Reform Recovery and Enforcement Act, in 1984, the credit ratings that banks used to evaluate people’s creditworthiness was made public (using a four-tier system) and was required to contain supporting details, so that their decisions could be audited if necessary. The Federal Housing Enterprises Financial Safety and Soundness Act of 1992 supposedly created the Housing and Urban Development (HUD) organization, and according to Fannie Mae’s website:

The Federal Housing Enterprises Financial Safety and Soundness Act (“FHEFSSA”) of 1992 modernized the regulatory oversight of Fannie Mae and Freddie Mac. It created the Office of Federal Housing Enterprise Oversight (“OFHEO”) as a new regulatory office within HUD with the responsibility to “ensure that Fannie Mae and Freddie Mac are adequately capitalized and operating safely.” OFHEO is funded by assessments on Fannie Mae and Freddie Mac and is authorized to act without HUD oversight on a range of regulatory issues enumerated in the statute. FHEFSSA established risk-based and minimum capital standards for Fannie Mae and Freddie Mac. And, it established HUD-imposed housing goals for financing of affordable housing and housing in central cities and other rural and underserved areas.

(emphasis mine) I couldn’t find all the specifics of those housing goals in a quick look through the text of the Act, but it’s long and complicated (maybe someone else will spend the time). Anyway, in an effort to finance these things effectively, Fannie and Freddie created and sold mortgage-backed securities. THIS is where the door was built (aka: the HUD), but the door isn’t yet “open.” More on this in a moment.

Because of the CRA requirements of transparency (imposed in 1984), banks were having trouble selling themselves, especially to out-of-state buyers. Not that the sales wouldn’t happen, but the sales would be slowed by CRA-based protests (I’m not sure what exactly they were protesting). So, Congress passed, in 1994, the Riegle-Neal Interstate Banking and Branching Efficiency Act, which removed the requirements that the banks had to publish the reasons why they made lending decisions under many circumstances, in order to facilitate sale of the banks. The publishing requirements were still in place if you weren’t selling your bank, though.

In 1995, Bill Clinton’s HUD made the policy decision to allow Fannie and Freddie to purchase subprime mortgage securities. Supposedly, HUD expected that Freddie and Fannie would impose their usual high standards upon subprime lenders. They did… mostly (hey, it’s still a subprime loan, after all), for a while, anyway. But this was still a bad idea (more on this in a moment).

In 1999, Congress passed the Gramm-Leach-Bliley Act, which repealed the Depression-era Glass-Steagall Act’s prohibition against banks acting as any combination of investment bank, commercial bank, and/or insurance company. Having more than one of those under one roof was a bad idea in 1929 because it creates conflict-of-interest on the part of the bankers/brokers, but apparently it’s a good idea in 1999. This paved the way for the formation of Citigroup and other investment/commercial/insurance banks. The GLB Act also reduced the number of banks to which the CRA applied, and altered their requirements so that they’d have to report on their CRA compliance less often.

In 2003, the Federal Reserve Bank of New York said:

Today, less than 30 percent of all home purchase loans are subject to intensive review under the CRA. In some metropolitan areas, this share is less than 10 percent.

This was considered bad news because it meant that the remaining 70 percent of loans were virtually un-auditable, and the reasons that loan decisions were made were untraceable. And Congressman Ron Paul made a speech predicting that these policies of increased lending to poor people would result in a credit collapse and a bailout. Spooky, eh?

What about predatory lending practices? In 2002, according to Wikipedia:

Kathleen C. Engel and Patricia A. McCoy published a study of the predatory lending implications of the CRA, noting that by the late 1990s, predatory high cost mortgages to “gullible borrowers” were leading to foreclosures against low-income people of color and the elderly.

As a result, the FDIC decided to strongly discourage the practice of “predatory lending”. And any bank found guilty of predatory lending was considered a higher risk, resulting in a lower CRA performance rating for that bank, both of which made FDIC insurance significantly more expensive. Interestingly, this move seems to have been quite effective. A study sponsored by a New York law firm (Traiger & Hinckley LLP, whoever they are), in January 2008 found:

Our study suggests that without the CRA, the subprime crisis and related spike in foreclosures might have negatively impacted even more borrowers and neighborhoods. Compared to other lenders in their assessment areas, CRA Banks were less likely to make a high cost loan, charged less for the high cost loans that were made, and were substantially more likely to eschew the secondary market and hold high cost and other loans in portfolio. Moreover, branch availability is a key element of CRA compliance, and foreclosure rates were lower in metropolitan areas with proportionately greater numbers of bank branches.

The FDIC’s measures to prevent irresponsible and predatory lending worked, and worked really well. So, if CRA wasn’t the problem, what was? What brought down Fannie Mae?

Remember the HUD? They expected that Fannie and Freddie would buy only the best quality subprime loans. But that backfired: what that ended up doing was making the market for these loans, including the even-crappier ones, bigger. According to Patricia McCoy, teacher of securities law at the University of Connecticut, cited in a Washington Post article, “That just pumped more capital into a very unregulated market that has turned out to be a disaster.” The HUD’s policies and encouragement are probably largely to blame for that market being particularly popular. It isn’t clear where the HUD’s policies started in 1992, but by 2000 the HUD required Fannie and Freddie to use 50% of their budget for “affordable” housing (sub-prime housing is one category that is considered “affordable”). Bush’s HUD upped that requirement to 56% in 2004. Clinton’s HUD opened the door by allowing subprime investing. But for some reason, it didn’t really take hold as a preferred strategy for investing in “affordable” housing.

Consider this: between 2005 and 2008, Fannie made loans to risky (sub-prime) borrowers of more than $270 billion dollars. That staggering sum is more than three times as much as all earlier years combined. But if it was all the HUD’s fault, then between 2005 and 2008, Fannie should have only increased subprime investment by about 6%, not 300%. And this would seem to have only a tangential relationship with the 30-year-old CRA. Something happened in 2004 or 2005! (One allegation I’ve heard is that Countrywide Mortgage started strong-arming Fannie into buying more of their riskiest loans at about that point.)

According to a New York Times article, Fannie Mae’s CEO, Daniel H. Mudd, under pressure from competing banks stealing his market and from Congress members demanding that they loan more to low-income folks, was between a rock and a hard place:

So Mr. Mudd made a fateful choice. Disregarding warnings from his managers that lenders were making too many loans that would never be repaid, he steered Fannie into more treacherous corners of the mortgage market, according to executives.

For a time, that decision proved profitable. In the end, it nearly destroyed the company and threatened to drag down the housing market and the economy.

Nevertheless, folks like the venerable Wall Street Journal blame the CRA for encouraging loose lending standards:

This 1977 law compels banks to make loans to poor borrowers who often cannot repay them.

And yet somehow, the CRA’s actual effect appears to have been the exact opposite: banks that were covered by the CRA were LESS likely to make sub-prime loans than banks that had were unaffected by the CRA. Those independent mortgage companies (unaffected by the CRA) made “high-priced loans” at more than twice the rate of banks governed by the CRA, according to Janet L. Yellen, President of the Federal Reserve Bank of San Francisco. According to the congressional testimony of Michael Barr (law professor at University of Michigan), 50% of the subprime loans were made by non-CRA mortgage companies, and another 25-30% came from mortgage companies only partially regulated by the CRA—-fully CRA-regulated institutions are responsible for “perhaps” one-in-four sub-prime loans. To quote the professor: “the worst and most widespread abuses occurred in the institutions with the least federal oversight.” And even more definitively, the Bank for International Settlements (an international organization of central banks) concluded:

Contrary to some media commentary, there is no evidence that the Community Reinvestment Act was responsible for encouraging the subprime lending boom and subsequent housing bust. This Act only applies to depositories, and did not cover most of the important subprime lenders. Depositories showed a lesser tendency to write subprime loans than lenders not subject to the Act (Yellen 2008).

Weird… you mean banks behave more responsibly when someone is peering over their shoulder? Say it ain’t so!

It seems to me that a large part of the Fannie problem was the HUD, enabling large scale sub-prime investing, and encouraging Fannie and Freddie to invest heavily in “affordable” housing. But that’s not the entirety of the problem; the HUD opened the door, but cannot have been responsible for the huge increase in sub-prime investment starting in 2005. However, even minimal investment in sub-prime mortgages (that HUD was trying to create) created demand in the sub-prime mortgage-backed securities market, which drove up the cost and encouraged other folks to invest as well.

I think the real lesson to be learned here is that there is no simple answer to questions like “what caused this economic mess?”, no matter how much politicians would love to say “it’s the other guy and his party’s fault”.

For example, consider Bloomberg’s take on the candidates’ reactions to the latest financial news (namely, the $700 billion proposed bailout of the entire financial sector). Here’s a representative snippet:

[Bill] Donaldson, who was tapped by Bush to head the SEC, says Obama called him last year about the financial-regulatory problems. He has never heard from McCain.

“Obama has been talking about the need for better financial regulation well before this crisis hit and has done some real thinking about it,” says Donaldson, a lifelong Republican. “McCain comes across as someone who suddenly realized changes have to be made.”

Additionally, they compare the people that Obama and McCain turned to for advice. Both of them turned to three people for advice. Obama called three former Treasury Secretaries. Two are described as particularly effective (Rubin and Summers), and the third became a well-regarded chairman of the Federal Reserve (Paul Volcker — described as “towering”). McCain, on the other hand, called a Reagan administration adviser (Martin Feldstein) who was regularly cut out of the decision making process by Reagan’s chief-of-staff, a former undersecretary of the Treasury not known for doing much (John Taylor), and the former CEO of HP who was fired rather publicly and received $21 million from the golden parachute clause in her contract (Carly Fiorina). Bloomberg clearly prefers Obama’s choices.

And just when you thought fiscal conservatives would be mad enough about the government buying AIG ($75 billion) and bailing out Fannie Mae and Freddie Mac ($25 billion) and bailing out Bear Stearns so that JPMorgan would buy it ($29 billion), now comes the Big Bailout. Let me reformat that: the BIG BAILOUT. An order of magnitude bigger. The taxpayer is now on the hook for an additional seven hundred billion dollars. Let’s put that in perspective: according to President Bush’s budget, we’re expected to spend $659 billion on the oh-so-odious Social Security in 2008. The US Government total 2008 budget tips the scale at 2.931 trillion dollars, according to the Heritage Foundation, which means this bailout would be TWENTY THREE PERCENT OF OUR ANNUAL BUDGET!!! Imagine if a guy that makes $100k/yr (which amounts to $77k/yr after taxes, i.e. $6,416/month) suddenly had to spend $17,710. That’s what we’re talking about. Are you screaming yet?

Jared Bernstein has an article in the Huffington Post. Here we go:

You hear that implosion reverberating through financial markets? It’s the sound of decades of conservative ideology collapsing. … The week that just ended revealed the myth of market fundamentalism: the notion associated with mainstream, Milton Freidman’esque economics, and amplified by anti-government conservatives that unfettered markets will provide society with the best outcomes. Such simplicity, such elegance…such nonsense.

He has a list of what he calls “glaringly obvious truths” that I think is very worth reading.

Since you probably won’t go read that article, here’s a few key ones (and my reactions):

1. Deregulated markets cannot police themselves; they tend toward speculation, vastly underpriced risk, and deeply damaging bubbles; (sounds accurate to me, given what we’ve seen; or perhaps it’s more accurate to say that deregulated markets DO NOT police themselves, even if they can)
2. An economy that generates growth while leaving most families behind is a broken economy (ok)
3. We can neither achieve broad prosperity nor compete globally without robust growth in key sectors which we have ignored or underfunded, including manufacturing, green production, and cradle-to-retirement public education; crafting evermore clever financial instruments will not pave the way to dependable, broadly shared growth (translation: creative accounting is no substitute for good fundamentals)
4. No private sector firm should be too big to fail; any firm of that magnitude must be nationalized (I would say, rather than nationalized, “broken up”, but the question becomes: at what point do we know that a company is too big? And what kind of a disincentive for smart business practices does this become? It’s fun to say that too big is bad, but is the cure worse than the disease?)
5. Capital markets are dysfunctional; borrowing and lending standards are ignored; lax capital requirements lead to constant over-leveraging; shadow accounts thwart transparency (that’s not a truism, that’s describing the current situation)
6. We apparently can quickly find (or borrow) the money to do the stuff the authorities deem necessary, be it war or bailout; thus, we can also find the money we need for investment in people, from health care to education to infrastructure, etc. (typical complaint… and saying “we don’t have the money” is obviously a poor excuse. These discussions should go back to “should we do this”)

There’s more, but those are the most central to his point.

Is he saying that capitalism can’t work? Maybe. I disagree with such an extremist statement. But consider this: the current Treasury Secretary Paulson (whatever you think of him) said about this crisis: Raw capitalism is dead.

Another fellow, named Robert Borosage, puts it slightly differently in his article:

Call it extortion. Every American is now being told to ante up $2000 — an estimated $700 billion in all — to bail out the banks from their bad bets, or they’ll bring down the entire economy.

In the speculative frenzy that allowed the Masters of the Universe to pocket millions personally, the banks filled their coffers with toxic paper that no one wants to buy. Now they sensibly don’t want to lend money to each other, since no one knows if the other is solvent. So they go on strike, and threaten to trigger a global depression, if they don’t get rescued.

He then lists seven of his own suggestions, which are also worth reading. Granted, I don’t see how his fourth suggestion (curb excessive CEO pay) would do much more than make us all feel better, but the heart of the matter is: these guys ran their companies, along with anyone who depended on them, into the ground. They shouldn’t be allowed to profit. The government’s bailout money should, if used at all, be only used to keep the economy afloat. CEO golden parachutes should be considered just as forfeit as if the company went bankrupt. Because that’s what these companies are: BANKRUPT.

Does nobody else in the world get chills down their spine from just uttering that word?

I still want to know just what would happen if these companies were simply forced to sleep in the beds they’ve made for themselves. If AIG did such a bad job, what would happen if they simply went belly-up? Wouldn’t other folks in the market be forced to realize that they really DO need to perform due diligence? That there’s a REASON things are called risky? So people get fearful, so credit starts to be harder to get, so people stop investing in things they don’t understand… isn’t that a good thing? The fundamental problem here, that started this whole mess, is that they were being too free with the credit and loaned money to anyone with a pulse and never bothered checking whether it was a good idea to loan them money! Credit should be harder to get! People who can’t afford a house shouldn’t be buying one! And if you don’t know how risky an investment is or if you don’t understand where the return comes from DON’T BUY IT! There’s a shocking idea for you…

These people deserve to lose lots of money for making stupid decisions, and anyone who loaned those folks money (aka investors) won’t be getting it back. But we don’t want some sort of cascade to happen where nobody invests in anybody ever again. How can we make people learn their lessons without taking the stock market down with them? There are a lot of ordinary folks in the stock market these days, including ones that may not even realize it, due to their retirement accounts: the day of the pension has passed, and 401k’s rule the land. When the market goes down, they take the little guys with them.

Buying all of these bad investments, such that they now become OUR bad investments, seems like a bad idea (no matter whether it’s the government doing it or our mutual fund managers). What do we-the-people want with bad investments? We don’t! And yet, this bailout seems more like, as Bernstein puts it: privatizing the profits and socializing the losses. Oh goody, sign me up for that! You notice all the pro-deregulation folks are lining up behind this bailout, don’t you? It’s called voting yourself money. It’s also called “saving your own hide”… at everyone else’s expense.

So, here’s the goal: atomically increment a variable. On a Sparc (specifically, SparcV9), the function looks something like this:

static inline int atomic_inc(int * operand)
{
    register uint32_t oldval, newval;
    newval = *operand;
    do {
        oldval = newval;
        newval++;
        __asm__ __volatile__ ("cas [%1], %2, %0"
            : "=&r" (newval)
            : "r" (operand), "r"(oldval)
            : "cc", "memory");
    } while (oldval != newval);
    return oldval+1;
}

Seems trivial, right? We use the CAS instruction (compare and swap). Conveniently, whenever the comparison fails, it stores the value of *operand in the second register (i.e. %0 aka newval), so there are no extraneous memory operations in this little loop. Right? Right. Does it work? NO.

Let’s take a look at the assembly that the compiler (gcc) generates with -O2 optimization:

save    %sp, -0x60, %sp
ld      [%i0], %i5      /* newval = *operand; */
mov     %i0, %o1        /* operand is copied into %o1 */
mov     %i5, %o2        /* oldval = newval; */
cas     [%o1], %o2, %o0 /* o1 = operand, o2 = newval, o0 = ? */
ret
restore %i5, 0x1, %o0

Say what? Does that have ANYTHING to do with what I told it? Nope! %o0 is never even initialized, but somehow it gets used anyway! What about the increment? Nope! It was optimized out, apparently (which, in fairness, is probably because we didn’t explicitly list it as an input). Of course, gcc is awful, you say! Use SUN’s compiler! Sorry, it produces the exact same output.

But let’s be a bit more explicit about the fact that the newval register is an input to the assembly block:

static inline int atomic_inc(int * operand)
{
    register uint32_t oldval, newval;
    newval = *operand;
    do {
        oldval = newval;
        newval++;
        __asm__ __volatile__ ("cas [%1], %2, %0"
            : "=&r" (newval)
            : "r" (operand), "r"(oldval), "0"(newval)
            : "cc", "memory");
    } while (oldval != newval);
    return oldval+1;
}

Now, Sun’s compiler complains: warning: parameter in inline asm statement unused: %3. Well gosh, isn’t that useful; way to recognize the fact that "0" declares the input to be an output! But at least, gcc leaves the add operation in:

save    %sp, -0x60, %sp
ld      [%i0], %i5      /* oldval = *operand; */
mov     %i0, %o1        /* operand is copied to %o1 */
add     %i5, 0x1, %o0   /* newval = oldval + 1; */
mov     %i5, %o2        /* oldval is copied to %o2 */
cas     [%o1], %o2, %o0
ret
restore %i5, 0x1, %o0

Yay! The increment made it in there, and %o0 is now initialized to something! But what happened to the do{ }while() loop? Sorry, that was optimized away, because gcc doesn’t recognize that newval can change values, despite the fact that it’s listed as an output!

Sun’s compiler will at least leave the while loop in, but will often use the WRONG REGISTER for comparison (such as %i2 instead of %o0).

But check out this minor change:

static inline int atomic_inc(int * operand)
{
    register uint32_t oldval, newval;
    do {
        newval = *operand;
        oldval = newval;
        newval++;
        __asm__ __volatile__ ("cas [%1], %2, %0"
            : "=&r" (newval)
            : "r" (operand), "r"(oldval), "0"(newval)
            : "cc", "memory");
    } while (oldval != newval);
    return oldval+1;
}

See the difference? Rather than using the output of the cas instruction (newval), we’re throwing it away and re-reading *operand no matter what. And guess what suddenly happens:

save     %sp, -0x60, %sp
ld       [%i0], %i5           /* oldval = *operand; */
add      %i5, 0x1, %o0        /* newval = oldval + 1; */
mov      %i0, %o1             /* operand is copied to %o1 */
mov      %i5, %o2             /* oldval is copied to %o2 */
cas      [%o1], %o2, %o0
cmp      %i5, %o0             /* if (oldval != newval) */
bne,a,pt %icc, atomic_inc+0x8 /* then go back and try again */
ld       [%i0], %i5
ret
restore  %i5, 0x1, %o0

AHA! The while loop returns! And best of all, both GCC and Sun’s compiler suddenly, magically, (and best of all, consistently) use the correct registers for the loop comparison! It’s amazing! For some reason this change reminds the compilers that newval is an output!

It’s completely idiotic. So, we can get it to work… but we have to be inefficient in order to do it, because otherwise (inexplicably) the compiler refuses to acknowledge that our output register can change.

In case you’re curious, the gcc version is:
sparc-sun-solaris2.10-gcc (GCC) 4.0.4 (gccfss)
and the Sun compiler is:
cc: Sun C 5.9 SunOS_sparc 2007/05/03

I am coming to this realization because I recently lost a BUNCH of messages because of a bad delivery path (I told procmail to pipe messages to a non-existent executable). So what did procmail do? According to its log:

/bin/sh: /tmp/dovecot11/libexec/dovecot/deliver: No such file or directory
procmail: Error while writing to "/tmp/dovecot11/libexec/dovecot/deliver"

Well, sure, that’s to be expected, right? So what happened to the email? VANISHED. Into the bloody ether.

Of course, determining that the message vanished is trickier than just saying “hey, it’s not in my mailbox.” Oh no, there’s a “feature”, called ORGMAIL. What is this? According to the procmailrc documentation (*that* collection of wisdom):

ORGMAIL     Usually the system  mailbox  (ORiGinal  MAIL‐
            box).   If,  for  some  obscure  reason (like
            ‘filesystem full’)  the  mail  could  not  be
            delivered, then this mailbox will be the last
            resort.  If procmail fails to save  the  mail
            in  here  (deep,  deep  trouble :-), then the
            mail will bounce back to the sender.

And so where is THAT? Why, /var/mail/$LOGNAME of course, where else? And if LOGNAME isn’t set for some reason? Or what if ORGMAIL is unset? Oh, well… nuts to you! Procmail will use $SENDMAIL to BOUNCE THE EMAIL rather than just try again later. That’s what they mean by “deep, deep trouble.” Notice the smiley face? Here’s why the manual has a smiley-face in it: to mock your pain.

But here’s the real crux of it: procmail doesn’t see delivery errors as FATAL. If one delivery instruction fails, it’ll just keep going through the procmailrc, looking for anything else that might match. In other words, the logic of your procmailrc has to take into account the fact that sometimes mail delivery can fail. If you fail to do this, your mail CAN end up in RANDOM LOCATIONS, depending on how messages that were supposed to match earlier rules fare against later rules.

If you want “first failure bail” behavior (which makes the most sense, in my mind), you have to add an extra rule after EVERY delivery instruction. For example:

:0 H
* ^From: .*fred@there\.com
./from_fred

:0 e # handle failure
{
    EXITCODE=75 # set a non-zero exit code
    HOST # This causes procmail to stop, obviously
}

You agree that HOST means “stop processing and exit”, right? Obviously. That’s procmail for you. Note that that second clause has gotta go after EVERY delivery instruction. I hope you enjoy copy-and-paste.

Another way to handle errors, since successful delivery does stop procmail, is to add something like that to the end of your procmailrc, like so:

:0 # catch-all default delivery
${DEFAULT}

 # If we get this far, there must have been an error
EXITCODE=75
HOST

Of course, you could also send the mail to /dev/null at that point, but unsetting the HOST variable (which is what listing it does) does the same thing faster. Intuitive, right? Here’s my smiley-face:

>:-P

If you want to see the live images, check here.

The idea is based on the following observations: certain addresses on my domain ONLY get spam. This is generally because they either don’t exist or because I stopped using them; for example, spammers often send email to buy@memoryhole.net. Branden and I also both use the user-tag@domain scheme, so we get a lot of disposable addresses that way. These addresses are such that we know for certain that anyone sending email to them is a spammer. Some of these addresses were already being rejected as invalid; some we hadn’t gotten around to invalidating yet.

By simply rejecting emails sent to those addresses, we were able to reduce the spam load of our domains by a fair bit, and the false-positive rate is nil. But we took things a step further: since spammers rarely send only one message, often they will send spam to both invalid AND valid addresses.

If I view those known-bad addresses as, essentially, honeypots, I can say: aha! Any IP sending to a known-bad address is a spammer, and I can refuse (with a permanent fail) any email from that IP for some short time. I started with 5 minutes, but have moved to an exponentially increasing timeout system. Each additional spam increased the length of the timeout (5 minutes for the first spam, 6 for the second, 8 for the third, and so on). Longer-term bans, as a result of the exponentially increasing timeout, are made more efficient via the equivalent of /etc/hosts.deny. I haven’t gotten into the maintaining-my-spammer-database much yet, but I think this may not be terribly important (I’ll explain in a moment).

One of the best parts of the system is that it is fast: new spammers that identify themselves by sending to honeypot addresses get blocked quickly and without my intervention. So far this has been particularly helpful in eliminating spam spikes. Another feature that I originally thought would be useful, but hasn’t really appeared to be (yet) is that it allows our multiple domains to share information about spam sources. Thus far, however, our domains seem to be plagued by different spammers.

Now, interestingly, about a week after we started using the system, our database of known spammers was wiped out (it’s kept in /tmp, and we rebooted the system). Result? No noticeable change in effectiveness. How’s that for a result? And, as you can see from the graph above, there’s no obvious change in spam blocking over the course of a month that would indicate that the long-term history is particularly useful. So, it may be sufficient to keep a much shorter history. Maybe only a week is necessary, maybe two weeks, I haven’t decided yet (and, as there hasn’t yet been much of a speed penalty for it, there’s no pressure to establish a cutoff). But, given that most spam is sent from botnets with dynamic IPs, this isn’t a particularly surprising behavior.

Forkit.org and memoryhole.net have been using this filter for a month so far. The week before we started using this filter, memoryhole.net averaged around 262 emails per hour. The week after instituting this filter, the average was around 96 per hour (a 60+% reduction!). Before using the filter, forkit.org averaged 70 emails per hour; since starting to use the filter, that number is down to 27.4 per hour (also a 60+% reduction). We have recorded spams from over 33,000 IPs, most of which only ever sent one or two spams. We typically have between 100 and 150 IPs that are “in jail” at any one time (at this moment: 143), and most of those (at this moment 134) are blocked for sending more than ten spams (114 of them have a timeout measured in days rather than minutes).

Now, granted, I know that by simply dropping 60% of all connections we’d get approximately the same results. But I think our particular technique is superior to that because it’s based on known-bad addresses. Anyone who doesn’t send to invalid addresses will never notice the filter.

The biggest potential problem that I can see with this system is that of spammers who have taken over a normally friendly host, such as Gmail spam. I’ve waffled on this potential problem: on the one hand, Gmail has so many outbound servers that it’s unlikely to get caught (a couple bad emails won’t have much of a penalty). Thus far, I’ve seen a few yahoo servers in Japan sending us spam, but no Gmail servers. On the other hand, as long as I simply use temporary failures (at least for good addresses), and as long as ND doesn’t retry in the same order every time, messages will get through.

I’ve also begun testing a “restricted sender” feature to work with this. For example, I have the address kyle-slashdot@memoryhole.net that I use exclusively for my slashdot.org account. The only people who are allowed to send to that email address is slashdot.org (i.e. if I forget my password). If anyone from any other domain attempts that address, well, then I know that sending IP is a spammer and I can treat it as if it was a known-bad address. Not applicable to every email address, obviously, but it’s a start.

It’s been pointed out that this system is, in some respects, a variant on greylisting. The major difference is that it’s a penalty-based system, rather than a “prove yourself worthy by following the RFC” system, and I like that a bit better. I’m somewhat tempted to define some bogus address (bogus@memoryhole.net) and sign it up for spam (via spamyourenemies.com or something similar), but given that part of the benefit here is due to spammers trying both valid and invalid addresses, I think it would probably just generate lots of extra traffic and not achieve anything particularly useful.

Now, this technique is simply one of many; it’s not sufficient to guarantee a spam-free inbox. I use it in combination with several other antispam techniques, including a greet-delay system and a frequently updated SpamAssassin setup. But check out the difference it’s made in our CPU utilization:

Okay, so, grand scheme of things: knocking the CPU use down three percentage points isn’t huge, but knocking it down by 50%? That sounds better, anyway. And as long as it doesn’t cause problems by making valid email disappear (possible, but rather unlikely), it seems to me to be a great way to cut my spam load relatively easily.

My impressions of the new OS are pretty favorable. I’ve read all the complaints about the UI changes, and they have some merit. By the time I upgraded, Apple had already released 10.5.2, which addressed many of the more unfortunate problems for people like me who put /Applications into the Dock.

I really DO like the “Fan” icon display, though, particularly for the new “Downloads” folder. Creating a folder just for downloads is something I could have done years ago, of course, but I hadn’t - everything downloaded to the Desktop, which inevitably became incredibly cluttered. But I love the new approach, and part of what makes it especially useful is that things in the “Fan” display can be dragged to the trash. HA! I love it! It’s the little things that make me happy. :)

The new X11 is a bit of a pain in the butt. I’d become very used to using xterm - or more precisely, uxterm - for all my terminal needs (which is to say, for 90% of what I do with my computer). That’s not so tenable now, particularly since Apple has apparently decided that uxterm was just too useful a shell script to let stand. I am keeping a copy of that shell script (which just runs xterm with all the necessary utf-8 flags and sets the LANG appropriately) handy, just in case, but for the time being, I’ve decided to migrate to using Apple’s Terminal full time now. Undoubtedly, it’s still not as fast as uxterm, but since getting an Intel iMac, I don’t really notice anymore (on the old dual 500Mhz G4, it was definitely noticable).

For migrating, I’ve had to create my own nsterm-16color termcap file (which I keep in ~/.terminfo/n/nsterm-16color ) in order to ensure that all the features I want work properly. I stole the file from ncurses 5.6, and modified it to add correct dual-mode swapping ( smcup=\E7\E[?47h, rmcup=\E[2J\E[?47l\E8 ) and then to support the home and end keys ( khome=\E[H, kend=\E[F ). These are things that the native OSX dtterm/xterm/xterm-color/whatever terminfo settings don’t do correctly. ( WHY???) …And then, of course, I had to fix the key mapping of pageup/shift-pageup and pagedown/shift-pagedown and all the relative keys, but that was easy to do in the Terminal.app’s preferences. The defaults are sensible, just not for folks who are used to xterm’s behavior. I also re-discovered that I hate Terminal.app’s default blue (a dark, almost-midnight blue), and much prefer having a lighter one. Thankfully I’m not the only one - Ciarán Walsh’s update to the TerminalColors plugin is solid and works well.

Other than that, things have been pretty smooth. I haven’t experienced any really strange compatibility problems — in large part, I think, because I keep my system pretty up-to-date, so I already had the “Leopard-compatible” versions of all the software I use (and all the Unix applications seem to work flawlessly without even needing a recompile - huzzah for that!).

The one application that needed SERIOUS fiddling is VirtualBox. They have an OSX version, but only in beta form. I use it mostly so I can provide sensible Windows XP support to relatives who have computer questions (and for doing browser compatibility tests). I had been using Beta 2 (1.4.6), which had worked flawlessly for my needs. Unfortunately, Beta 2 isn’t compatible with Leopard, so an upgrade to the latest (Beta 3) was necessary. THIS beta seems to have a few problems. For one thing, it can’t understand all the old machine definitions (so when upgrading, make sure you don’t have any important system snapshots or saved machine state that you need). However, it does understand the old disk files, so it’s a simple matter to create a new machine definition using the old disk. The new machine still won’t BOOT, though, and it took me an hour or so of fiddling to figure out how to fix it.

There are two major problems that crop up. First: they changed the default IDE controller for Windows XP guests. The old default was PIIX3; the new default is PIIX4. Either one will work, and if you install XP from scratch on a newly created XP host, it will work with the PIIX4 controller just fine. But if you’re booting from an XP that was created with Beta 2 (i.e. a WindowsXP installation that thinks you have a PIIX3 controller), it will blue-screen and reboot immediately after displaying the Microsoft logo: not good. Fixing it is easy, though: just change the IDE controller for your XP machine in the machine settings dialog.

The second problem is that the network doesn’t work. Actually, that’s not true, the network works just fine, it’s DNS resolution that doesn’t work (but one looks a lot like the other when you’re not paying close attention to error messages). For whatever reason, when your XP system uses DHCP to get its network information, the information it receives from VirtualBox is wrong. Specifically, VirtualBox tells it to resolve DNS names by contacting 10.0.2.3; it should be contacting 10.0.2.2 (i.e. the same as the router). Fixing this was just a matter of changing Windows’ network configuration to use a custom DNS server (10.0.2.2) rather than the one supplied by DHCP. Annoying, but nothing terrible.

The only other stumbling block in Leopard that I’ve come across is the iChat-vs-Internet-Sharing problem that other people have discovered. Essentially, if you have enabled Internet Sharing, iChat can’t do video conferencing. Something to do with being able to remap ports… the explanations I’ve read are rather vague. It’s not especially important to me, but came up when I was trying to demonstrate the virtues of Leopard to Emily.

Which reminds me: the new iChat is MUCH better for talking to multiple people at the same time. The “tabbed” chatting interface is terrific. The vaunted “Spaces” (virtual desktops) are nice, and implemented well, but I gotta say that I’ve gotten used to having just one desktop these days (I use Exposé a lot). Getting used to having the extra desktops will probably take a while.

Two more features I noticed were the Quick View (in Finder, press the space bar to quickly view something) and Web Clips (in Safari, you can take a snippet of a webpage and turn it into a Dashboard widget). Quick View is pretty great, especially for folders full of PDFs, because you can leave it up and keep navigating around the Finder (the contents of the Quick View window will track whatever you select in the Finder), but since I don’t spend much time in the Finder, it’s of limited use. If I could integrate it with my ~/.mailcap file, now THAT would be awesome. Web Clips are not quite as great as they could be. For one thing, they don’t refresh quickly (but they DO refresh—at first I didn’t think they did—and in the worst case, you can click on them and press Ctrl-R to force the issue), and for another, they can’t scale — many of the things I want to clip are large graphics that I wish to monitor. If OSX could scale clips down for me, that would make them much more useful.

Which reminds me — one new feature of Leopard that I adore is their new built-in VNC viewer. It may not actually be VNC, but that’s fine by me — it’s blazing fast, and best of all, it scales the screen down so that you can easily control a screen that’s larger than the one you have. Chicken of the VNC used to be a must-have application for me, but Leopard’s built-in screen viewer is much better for what I usually want to do (which is control the iMac upstairs from the laptop down on the couch).

Anyway, my bashrc is first on the list. (Or, if you like, the pure text version.)

The file is divided into several (kinda fuzzy) sections:
- Initialization & Other Setup
- Useful Functions
- Loading System-wide Bashrc
- Behavioral Settings
- Environment Variables
- Character Set Detection
- Aliases
- Tab-completion Options
- Machine-local settings
- Auto-logout

Let's take them one at a time.

Initialization & Other Setup

Throughout my bashrc, I use a function I define here ( dprint ) to allow me to quickly turn on debugging information, which includes printing the seconds-since-bash-started variable ( SECONDS ) in case something is taking too long and you want to find the culprit. Yes, my bashrc has a debug mode. This is essentially controlled by the KBWDEBUG environment variable. Then, because this has come in useful once or twice, I allow myself to optionally create a ~/.bashrc.local.preload file which is sourced now, before anything else. Here's the code:

KBWDEBUG=${KBWDEBUG:-no}

function dprint {
if [[ "$KBWDEBUG" == "yes" && "$-" == *i* ]]; then
    #date "+%H:%M:%S $*"
    echo $SECONDS $*
fi
}
dprint alive
if [ -r "${HOME}/.bashrc.local.preload" ]; then
    dprint "Loading bashrc preload"
    source "${HOME}/.bashrc.local.preload"
fi

Useful Functions

This section started with some simple functions for PATH manipulation. Then those functions got a little more complicated, then I wanted some extra functions for keeping track of my config files (which were now in CVS), and then they got more complicated...

You'll notice something about these functions. Bash (these days) will accept function declarations in this form:

function fname()
{
    do stuff
}

But that wasn't always the case. To maintain compatability with older bash versions, I avoid using the uselessly cosmetic parens and I make sure that the curly-braces are on the same line, like so:

function fname \
{
    do stuff
}

Anyway, the path manipulation functions are pretty typical — they're similar to the ones that Fink uses, but slightly more elegant. The idea is based on these rules of PATH variables:

1. Paths must not have duplicate entries
2. Paths are faster if they don't have symlinks in them
3. Paths must not have "." in them
4. All entries in a path must exist (usually)

There are two basic path manipulation functions: add_to_path and add_to_path_first. They do predictable things — the former appends something to a given path variable (e.g. PATH or MANPATH or LD_LIBRARY_PATH ) unless it's already in that path, and the latter function prepends something to the given PATH variable (or, if it's already in there, moves it to the beginning). Before they add a value to a path, they first check it to make sure it exists, is readable, that I can execute things that are inside it, and they resolve any symlinks in that path (more on that in a moment). Here's the code (ignore the reference to add_to_path_force in add_to_path for now; I'll explain shortly):

function add_to_path \
{
    local folder="${2%%/}"
    [ -d "$folder" -a -x "$folder" ] || return
    folder=`( cd "$folder" ; \pwd -P )`
    add_to_path_force "$1" "$folder"
}

function add_to_path_first \
{
    local folder="${2%%/}"
    [ -d "$folder" -a -x "$folder" ] || return
    folder=`( cd "$folder" ; \pwd -P )`
    # in the middle, move to front
    if eval '[[' -z "\"\${$1##*:$folder:*}\"" ']]'; then
        eval "$1=\"$folder:\${$1//:\$folder:/:}\""
        # at the end
    elif eval '[[' -z "\"\${$1%%*:\$folder}\"" ']]'; then
        eval "$1=\"$folder:\${$1%%:\$folder}\""
        # no path
    elif eval '[[' -z "\"\$$1\"" ']]'; then
        eval "$1=\"$folder\""
        # not in the path
    elif ! eval '[[' -z "\"\${$1##\$folder:*}\"" '||' \
      "\"\$$1\"" '==' "\"$folder\"" ']]'; then
        eval "export $1=\"$folder:\$$1\""
    fi
}

Then, because I was often logging into big multi-user Unix systems (particularly Solaris systems) with really UGLY PATH settings that had duplicate entries, often included ".", not to mention directories that either didn't exist or that I didn't have sufficient permissions to read, I added the function verify_path. All this function does is separates a path variable into its component pieces, eliminates ".", and then reconstructs the path using add_to_path, which handily takes care of duplicate and inaccessible entries. Here's that function:

function verify_path \
{
    # separating cmd out is stupid, but is compatible
    # with older, buggy, bash versions (2.05b.0(1)-release)
    local cmd="echo \$$1"
    local arg="`eval $cmd`"
    eval "$1=\"\""
    while [[ $arg == *:* ]] ; do
        dir="${arg%:${arg#*:}}"
        arg="${arg#*:}"
        if [ "$dir" != "." -a -d "$dir" -a \
          -x "$dir" -a -r "$dir" ] ; then
            dir=`( \cd "$dir" ; \pwd -P )`
            add_to_path "$1" "$dir"
        fi
    done
    if [ "$arg" != "." -a -d "$arg" -a -x "$arg" -a -r "$arg" ] ;
    then
        arg=`( cd "$arg" ; \pwd -P )`
        add_to_path "$1" "$arg"
    fi
}

Finally, I discovered XFILESEARCHPATH — a path variable that requires a strange sort of markup (it's for defining where your app-defaults files are for X applications). This wouldn't work for add_to_path, so I created add_to_path_force that still did duplicate checking but didn't do any verification of the things added to the path.

function add_to_path_force \
{
    if eval '[[' -z "\$$1" ']]'; then
        eval "export $1='$2'"
    elif ! eval '[[' \
        -z "\"\${$1##*:\$2:*}\"" '||' \
        -z "\"\${$1%%*:\$2}\"" '||' \
        -z "\"\${$1##\$2:*}\"" '||' \
        "\"\${$1}\"" '==' "\"$2\"" ']]'; then
        eval "export $1=\"\$$1:$2\""
    fi
}

I mentioned that I resolved symlinks before adding directories to path variables. This is a neat trick I discovered due to the existence of pwd -P and subshells. pwd -P will return the "real" path to the folder you're in, with all symlinks resolved. And it does so very efficiently (without actually resolving symlinks — it just follows all the ".." records). Since you can change directories in a subshell (i.e. between parentheses) without affecting the parent shell, a quick way to transform a folder's path into a resolved path is this: ( \cd "$folder"; pwd -P). I put the backslash in there to use the shell's builtin cd, just in case I'd somehow lost my mind and aliased cd to something else.

And then, just because it was convenient, I added another function: have, which detects whether a binary is accessible or not:

function have { type "$1" &>/dev/null ; }

Then I had to confront file paths, such as the MAILCAP variable. A lot of the same logic (i.e. add_to_path_force), but entry validation is different:

function add_to_path_file \
{
    local file="${2}"
    [ -f "$file" -a -r "$file" ] || return
    # realpath alias may not be set up yet
    file=`realpath_func "$file"`
    add_to_path_force "$1" "$file"
}

You'll note the realpath_func line in there. realpath is a program that takes a filename or directory name and resolves the symlinks in it. Unfortunately, realpath is a slightly unusual program; I've only ever found it on OSX (it may be on other BSDs). But, with the power of my pwd -P trick, I can fake most of it. The last little piece (resolving a file symlink) relies on a tool called readlink ... but I can fake that too. Here are the two functions:

function readlink_func \
{
    if have readlink ; then
        readlink "$1"
    #elif have perl ; then # seems slower than alternative
    #    perl -e 'print readlink("'"$1"'") . "\n"'
    else
        \ls -l "$1" | sed 's/[^>]*-> //'
    fi
}

function realpath_func \
{
    local input="${1}"
    local output="/"
    if [ -d "$input" -a -x "$input" ] ; then
        # All too easy...
        output=`( cd "$input"; \pwd -P )`
    else
        # sane-itize the input to the containing folder
        input="${input%%/}"
        local fname="${input##*/}"
        input="${input%/*}"
        if [ ! -d "$input" -o ! -x "$input" ] ; then
            echo "$input is not an accessible directory" >&2
            return
        fi
        output="`( cd "$input" ; \pwd -P )`/"
        input="$fname"
        # output is now the realpath of the containing folder
        # so all we have to do is handle the fname (aka "input)
        if [ ! -L "$output$input" ] ; then
            output="$output$input"
        else
            input="`readlink_func "$output$input"`"
            while [ "$input" ] ; do
                if [[ $input == /* ]] ; then
                    output="$input"
                    input=""
                elif [[ $input == ../* ]] ; then
                    output="${output%/*/}/"
                    input="${input#../}"
                elif [[ $input == ./* ]] ; then
                    input="${input#./}"
                elif [[ $input == */* ]] ; then
                    output="$output${input%${input#*/}}"
                    input="${input#*/}"
                else
                    output="$output$input"
                    input=""
                fi
                if [ -L "${output%%/}" ] ; then
                    if [ "$input" ] ; then
                        input="`readlink_func "${output%%/}"`/$input"
                    else
                        input="`readlink_func "${output%%/}"`"
                    fi
                    output="${output%%/}"
                    output="${output%/*}/"
                fi
            done
        fi
    fi
    echo "${output%%/}"
}

Loading System-wide Bashrc

This section isn't too exciting. According to the man page:

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable.

SOME systems have a version of bash that appears not to obey this rule. And some systems put crucial configuration settings in /etc/bashrc (why?!?). And some systems even do something silly like use /etc/bashrc to source ~/.bashrc (I did this myself, once upon a time, when I knew not-so-much). I've decided that this behavior cannot be relied upon, so I explicitly source these files myself. The only interesting bit is that I added a workaround so that systems that use /etc/bashrc to source ~/.bashrc won't get into an infinite loop. There's probably a lot more potential trouble here that I'm ignoring. But here's the code:

if [[ -r /etc/bashrc && $SYSTEM_BASHRC != 1 ]]; then
    dprint " - loading /etc/bashrc"
    . /etc/bashrc
    export SYSTEM_BASHRC=1
fi

Behavioral Settings

This is basic stuff, but after you get used to certain behaviors (such as whether * matches . and ..), you often get surprised when they don't work that way on other systems. Some of this is because I found a system that did it another way by default; some is because I decided I like my defaults and I don't want to be surprised in the future.

The interactive-shell-detection here is nice. $- is a variable set by bash containing a set of letters indicating certain settings. It always contains the letter i if bash is running interactively. So far, this has been quite backwards-compatible.

shopt -s extglob # Fancy patterns, e.g. +()
# only interactive
if [[ $- == *i* ]]; then
    dprint setting the really spiffy stuff
    shopt -s checkwinsize # don't get confused by resizing
    shopt -s checkhash # if hash is broken, doublecheck it
    shopt -s cdspell # be tolerant of cd spelling mistakes
fi

Environment Variables

There are a slew of standard environment variables that bash defines for you (such as HOSTNAME). There are even more standard environment variables that various programs pay attention to (such as EDITOR and PAGER). And there are a few others that are program-specific (such as PARINIT and CVSROOT).

Before I get going, though, let me show you a secret. Ssh doesn't like transmitting information from client to server shell... the only reliable way to do it that I've found is the TERM variable. So... I smuggle info through that way, delimited by colons. Before I set any other environment variables, first, I find my smuggled information:

if [[ $TERM == *:* && ( $SSH_CLIENT || $SSH_TTY || $SSH_CLIENT2 ) ]] ; then
    dprint "Smuggled information through the TERM variable!"
    term_smuggling=( ${TERM//:/ } )
    export SSH_LANG=${term_smuggling[1]}
    TERM=${term_smuggling[0]}
    unset term_smuggling
fi

I begin by setting GROUPNAME and USER in a standard way:

if [[ $OSTYPE == solaris* ]] ; then
    idout=(`/bin/id -a`)
    USER="${idout[0]%%\)*}"
    USER="${USER##*\(}"
    [[ $USER == ${idout[0]} ]] && USER="UnknownUser"
    GROUPNAME="UnknownGroup"
    unset idout
else
    [[ -z $GROUPNAME ]] && GROUPNAME="`id -gn`"
    [[ -z $USER ]] && USER="`id -un`"
fi

Then some standard things (MAILPATH is used by bash to check for mail, that kind of thing), including creating OS_VER and HOST to allow me to identify the system I'm running on:

# I tote my own terminfo files around with me
[ -d ~/.terminfo ] && export TERMINFO=~/.terminfo/
[ "$TERM_PROGRAM" == "Apple_Terminal" ] && \
    export TERM=nsterm-16color

MAILPATH=""
MAILCHECK=30
add_to_path_file MAILPATH /var/spool/mail/$USER
add_to_path MAILPATH $HOME/Maildir/
[[ -z $MAILPATH ]] && unset MAILCHECK
[[ -z $HOSTNAME ]] && \
    export HOSTNAME=`/bin/hostname` && echo 'Fake Bash!'
HISTSIZE=1000
HOST=${OSTYPE%%[[:digit:]]*}
OS_VER=${OSTYPE#$HOST}
[ -z "$OS_VER" ] && OS_VER=$( uname -r )
OS_VER=(${OS_VER//./ })
TTY=`tty`
PARINIT="rTbgq B=.,?_A_a P=_s Q=>|}+"

export USER GROUPNAME MAILPATH HISTSIZE OS_VER HOST TTY PARINIT

I've also gotten myself into trouble in the past with UMASK being set improperly, so it's worth setting manually. Additionally, to head off trouble, I make it hard to leave myself logged in as root on other people's systems accidentally:

if [[ $GROUPNAME == $USER && $UID -gt 99 ]]; then
    umask 002
else
    umask 022
fi

if [[ $USER == root ]] ; then
    [[ $SSH_CLIENT || $SSH_TTY || $SSH_CLIENT2  ]] && \
        export TMOUT=600 || export TMOUT=3600
fi

if [[ -z $INPUTRC && ! -r $HOME/.inputrc && -r /etc/inputrc ]];
then
    export INPUTRC=/etc/inputrc
fi

It is at this point that we should pause and load anything that was in /etc/profile, just in case it was left out (and, if its in there, maybe it should override what we've done so far):

export BASHRCREAD=1

if [[ -r /etc/profile && -z $SYSTEM_PROFILE ]]; then
    dprint "- loading /etc/profile ... "
    . /etc/profile
    export SYSTEM_PROFILE=1
fi

Now I set my prompt (but only if this is an interactive shell). There are several details here (obviously). The first is that, if I'm logged into another system, I want to see how long I've been idle. So I include a timestamp whenever I'm logged into a remote system. I also added color to my prompt in two ways, which has been very useful. First, it changes the color of the $ at the end of the prompt to red if the last command didn't exit cleanly. Second, remote systems have yellow prompts, whenever I'm root I have a red prompt, and I created commands to flip between a few other colors (blue, purple, cyan, green, etc.) in case I find that useful to quickly distinguish between terminals. Anyway, here's the code:

if [[ $- == *i* ]]; then
    if [[ $TERM == xterm* || $OSTYPE == darwin* ]]; then
        # This puts the term information into the title
        PSterminfo='\[\e]2;\u@\h: \w\a\]'
    fi
    PSparts[3]='(\d \T)\n'
    PSparts[2]='[\u@\h \W]'
    PSparts[1]='\$ '
    PScolors[2]='\[\e[34m\]' # Blue
    PScolors[3]='\[\e[35m\]' # Purple
    PScolors[4]='\[\e[36m\]' # Cyan
    PScolors[5]='\[\e[32m\]' # Green
    PScolors[6]='\[\e[33m\]' # Yellow
    PScolors[100]='\[\e[31m\]' # Badc
    PScolors[0]='\[\e[0m\]' # Reset
    if [[ $USER == root ]] ; then
        PScolors[1]='\[\e[31m\]' # Red
    elif [[ $SSH_CLIENT || $SSH_TTY || $SSH_CLIENT2 ]] ; then
        PScolors[1]="${PScolors[6]}" # yellow
        if [[ $HOSTNAME == marvin ]] ; then
            PScolors[1]="${PScolors[5]}" # green
        fi
    else
        unset PSparts[3]
        PScolors[1]=""
    fi
    function bashrc_genps {
        if [ "$1" -a "${PScolors[$1]}" ] ; then
            PSgood="$PSterminfo${PSparts[3]}${PScolors[$1]}${PSparts[2]}${PSparts[1]}${PScolors[0]}"
        else
            PSgood="$PSterminfo${PSparts[3]}${PSparts[2]}${PSparts[1]}"
        fi
        PSbad="$PSterminfo${PSparts[3]}${PScolors[$1]}${PSparts[2]}${PScolors[100]}${PSparts[1]}${PScolors[0]}"
    }
    bashrc_genps 1
    function safeprompt {
        export PS1='{\u@\h \W}\$ '
        unset PROMPT_COMMAND
    }
    alias stdprompt='bashrc_genps 1'
    alias blueprompt='bashrc_genps 2'
    alias purpleprompt='bashrc_genps 3'
    alias cyanprompt='bashrc_genps 4'
    alias greenprompt='bashrc_genps 5'
    alias whiteprompt='bashrc_genps'
    # this is executed before every prompt is displayed
    # it changes the prompt based on the preceeding command
    export PROMPT_COMMAND='[ $? = 0 ] && PS1=$PSgood || PS1=$PSbad'
fi

Now I set up the various paths. Note that it doesn't matter if these paths don't exist; they'll be checked and ignored if they don't exist:

verify_path PATH
add_to_path PATH "/usr/local/sbin"
add_to_path PATH "/usr/local/teTeX/bin"
add_to_path PATH "/usr/X11R6/bin"
add_to_path PATH "$HOME/bin"
add_to_path_first PATH "/sbin"

add_to_path_first PATH "/bin"
add_to_path_first PATH "/usr/sbin"
add_to_path_first PATH "/opt/local/bin"
add_to_path_first PATH "/usr/local/bin"

if [[ $OSTYPE == darwin* ]] ; then
    add_to_path PATH "$HOME/.conf/darwincmds"

    # The XFILESEARCHPATH (for app-defaults and such)
    # is a wonky kind of path
    [ -d /opt/local/lib/X11/app-defaults/ ] && \
        add_to_path_force XFILESEARCHPATH \
            /opt/local/lib/X11/%T/%N
    [ -d /sw/etc/app-defaults/ ] && \
        add_to_path_force XFILESEARCHPATH /sw/etc/%T/%N
    add_to_path_force XFILESEARCHPATH /private/etc/X11/%T/%N
fi

verify_path MANPATH
add_to_path MANPATH "/usr/man"
add_to_path MANPATH "/usr/share/man"
add_to_path MANPATH "/usr/X11R6/man"
add_to_path_first MANPATH "/opt/local/share/man"
add_to_path_first MANPATH "/opt/local/man"
add_to_path_first MANPATH "/usr/local/man"
add_to_path_first MANPATH "/usr/local/share/man"

verify_path INFOPATH
add_to_path INFOPATH "/usr/share/info"
add_to_path INFOPATH "/opt/local/share/info"

And now there are STILL MORE environment variables to set. This final group may rely on some of the previous paths being set (most notably, PATH).

export PAGER='less'
have vim && export EDITOR='vim' || export EDITOR='vi'
if [[ -z $DISPLAY && $OSTYPE == darwin* ]]; then
    processes=`ps ax`
    # there are double-equals here, even though they don't show
    # on the webpage
    if [[ $processes == *xinit* || $processes == *quartz-wm* ]]; then
        export DISPLAY=:0
    else
        unset DISPLAY
    fi
fi
if [[ $HOSTNAME == wizard ]] ; then
    dprint Wizards X forwarding is broken
    unset DISPLAY
fi
export TZ="US/Central"
if [ "${BASH_VERSINFO[0]}" -le 2 ]; then
    export HISTCONTROL=ignoreboth
else
    export HISTCONTROL="ignorespace:erasedups"
fi
export HISTIGNORE="&:ls:[bf]g:exit"
export GLOBIGNORE=".:.."
export CVSROOT=kyle@cvs.memoryhole.net:/home/kyle/cvsroot
export CVS_RSH=ssh
export BASH_ENV=$HOME/.bashrc
add_to_path_file MAILCAPS $HOME/.mailcap
add_to_path_file MAILCAPS /etc/mailcap
add_to_path_file MAILCAPS /usr/etc/mailcap
add_to_path_file MAILCAPS /usr/local/etc/mailcap
export EMAIL='kyle-envariable@memoryhole.net'
export GPG_TTY=$TTY
export RSYNC_RSH="ssh -2 -c arcfour -o Compression=no -x"
if [ -d /opt/local/include -a -d /opt/local/lib ] ; then
    export CPPFLAGS="-I/opt/local/include $CPPFLAGS"
    export LDFLAGS="-L/opt/local/lib $LDFLAGS"
fi
if have glibtoolize ; then
    have libtoolize || export LIBTOOLIZE=glibtoolize
fi

One little detail that I rather like is the fact that xterm's window title often tells me exactly what user I am on what machine I am, particularly when I'm ssh'd into another host. This little bit of code ensures that this happens:

if [[ $TERM == xterm* || $OSTYPE == darwin* ]]; then
    export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME/.*/}: ${PWD/${HOME}/~}\007"'
else
    unset PROMPT_COMMAND
fi

Character Set Detection

I typically work in a UTF-8 environment. MacOS X (my preferred platform for day-to-day stuff) has made this pretty easy with really excellent UTF-8 support, and Linux has come a long way (to parity, as far as I can tell) in the last few years. Most of my computing is done via a uxterm (aka. xterm with UTF-8 capability turned on), but I also occasionally dabble in other terminals (sometimes without realizing it). Despite the progress made, however, not all systems support UTF-8, and neither do all terminals. Some systems, including certain servers I've used, simply don't have UTF-8 support installed, even though they're quite capable of it.

if [[ -z $LANG ]] ; then
    dprint no LANG set
    if [[ $WINDOWID ]] && have xprop ; then
        dprint querying xprop
        __bashrc__wmlocal=(`xprop -id $WINDOWID -f WM_LOCALE_NAME 8s ' $0' -notype WM_LOCALE_NAME`)
        export LANG=`eval echo ${__bashrc__wmlocal[1]}`
        unset __bashrc__wmlocal
    elif [[ $OSTYPE == darwin* ]] ; then
        dprint "I'm on Darwin"
        if [[ ( $SSH_LANG && \
            ( $SSH_LANG == *.UTF* || $SSH_LANG == *.utf* ) || \
            $TERM_PROGRAM == Apple_Terminal ) && \
            -d "/usr/share/locale/en_US.UTF-8" ]] ; then
            export LANG='en_US.UTF-8'
        elif [ -d "/usr/share/locale/en_US" ] ; then
            export LANG='en_US'
        else
            export LANG=C
        fi
    elif [[ $TERM == linux || $TERM_PROGRAM == GLterm ]] ; then
        if [ -d "/usr/share/locale/en_US" ] ; then
            export LANG='en_US'
        else
            export LANG=C # last resort
        fi
    else
        if [[ $SSH_LANG == C ]] ; then
            export LANG=C
        elif have locale ; then
            dprint "checking locale from big list (A)"
            locales=`locale -a`
            locales="${locales//[[:space:]]/|}" # not +() because that's slow
            if [[ en_US.utf8 == @($locales) ]] ; then
                export LANG='en_US.utf8'
            elif [[ en_US.utf-8 == @($locales) ]] ; then
                export LANG='en_US.utf-8'
            elif [[ en_US.UTF8 == @($locales) ]] ; then
                export LANG='en_US.UTF8'
            elif [[ en_US.UTF-8 == @($locales) ]] ; then
                export LANG='en_US.UTF-8'
            elif [[ en_US == @($locales) ]] ; then
                export LANG='en_US'
            else
                export LANG=C
            fi
            unset locales
        fi
    fi
else
    dprint "- LANG IS ALREADY SET! ($LANG)"
    if [[ $SSH_LANG && $SSH_LANG != $LANG ]]; then
        if [[ $SSH_LANG == C ]] ; then
            export LANG=C
        else
            dprint "checking locale from big list (B)"
            locales=`locale -a`
            locales="${locales//[[:space:]]/|}" # not +() because that's slow
            if [[ $SSH_LANG == @(${locales}) ]] ; then
                dprint "- SSH_LANG is a valid locale, resetting LANG"
                LANG=$SSH_LANG
            else
                dprint "- SSH_LANG is NOT a valid locale"
                wantutf8=no
                if [[ $SSH_LANG == *.(u|U)(t|T)@(f|F)?(-)8 ]] ; then
                    wantutf8=yes
                    if [[ ! $LANG == *.(u|U)(t|T)@(f|F)?(-)8 ]] ; then
                        dprint "- want utf-8, but LANG is not utf8, unsetting"
                        unset LANG
                    fi
                else
                    dprint "- don't want utf-8"
                fi
                if [[ ! $LANG || ! $LANG == @($locales) ]] ; then
                    if [ "$wantutf8" = yes ] ; then
                        dprint "- finding a utf8 LANG"
                        if [[ en_US.utf8 == @($locales) ]] ; then
                            export LANG='en_US.utf8'
                        elif [[ en_US.utf-8 == @($locales) ]] ; then
                            export LANG='en_US.utf-8'
                        elif [[ en_US.UTF8 == @($locales) ]] ; then
                            export LANG='en_US.UTF8'
                        elif [[ en_US.UTF-8 == @($locales) ]] ; then
                            export LANG='en_US.UTF-8'
                        elif [[ en_US == @($locales) ]] ; then
                            export LANG='en_US'
                        else
                            export LANG=C
                        fi
                    else
                        dprint "- finding a basic LANG"
                        if [[ en_US == @($locales) ]] ; then
                            export LANG='en_US'
                        else
                            export LANG=C
                        fi
                    fi
                fi
                unset wantutf8
            fi
            unset locales
        fi
    else
        dprint "- ... without SSH_LANG, why mess with it?"
    fi
fi
dprint - LANG is $LANG
unset LESSCHARSET
if [[ $LANG == *.(u|U)(t|T)@(f|F)?(-)8 ]] ; then
    export LESSCHARSET=utf-8
fi

Sometimes I can't quite get what I want out of an alias. In csh aliases can specify what to do with their arguments. In bash, aliases are really more just shorthand — "pretend I really typed this" kind of stuff. Instead, if you want to be more creative with argument handling, you have to use functions (it's not a big deal, really). Here's a few functions I added just because they're occasionally handy to have the shell do for me:

function exec_cvim {
/Applications/Vim.app/Contents/MacOS/Vim -g "$@" &
}

function darwin_locate { mdfind "kMDItemDisplayName == '$@'wc"; }
if [[ $- == *i* && $OSTYPE == darwin* && ${OS_VER[0]} -ge 8 ]] ;
then
alias locate=darwin_locate
fi

function printargs { for F in "$@" ; do echo "$F" ; done ; }
function psq { ps ax | grep -i $@ | grep -v grep ; }
function printarray {
for ((i=0;$i<`eval 'echo ${#'$1'[*]}'`;i++)) ; do
    echo $1"[$i]" = `eval 'echo ${'$1'['$i']}'`
done
}
alias back='cd $OLDPWD'

There are often a lot of things that I just expect to work. For example, when I type "ls", I want it to print out the contents of the current directory. In color if possible, without if necessary. It often annoys me, on Solaris systems, when the working version of ls is buried in the path, while a really lame version is up in /bin for me to find first. Here's how I fix that problem:

# GNU ls check
if [[ $OSTYPE == darwin* ]]; then
    dprint "- DARWIN ls"
    alias ls='/bin/ls -FG'
    alias ll='/bin/ls -lhFG'
elif have colorls ; then
    dprint "- BSD colorls"
    alias ls='colorls -FG'
    alias ll='colorls -lhFG'
else
    __kbwbashrc__lsarray=(`\type -ap ls`)
    __kbwbashrc__lsfound=no
    for ((i=0;$i<${#__kbwbashrc__lsarray[*]};i=$i+1)) ; do
        if ${__kbwbashrc__lsarray[$i]} --version &>/dev/null ;
        then
            dprint "- found GNU ls: ${__kbwbashrc__lsarray[$i]}"
            alias ls="${__kbwbashrc__lsarray[$i]} --color -F"
            alias ll="${__kbwbashrc__lsarray[$i]} --color -F -lh"
            __kbwbashrc__lsfound=yes
            break
        fi
    done
    if [ "$__kbwbashrc__lsfound" == no ] ; then
        if ls -F &>/dev/null ; then
            dprint "- POSIX ls"
            alias ls='ls -F'
            alias ll='ls -lhF'
        else
            alias ll='ls -lh'
        fi
    fi
    unset __kbwbashrc__lsarray __kbwbashrc__lsfound
fi

Similar things are true of make and sed and such. I've gotten used to GNU's version, and if they exist on the machine I'd much rather automatically use them than have to figure out whether it's really called gnused or gsed or justtowasteyourtimesed all by myself:

if [[ $OSTYPE == linux* ]] ; then
    # actually, just Debian, but this works for now
    alias gv="gv --watch --antialias"
else
    alias gv="gv -watch -antialias"
fi
if have gsed ; then
    alias sed=gsed
elif have gnused ; then
    alias sed=gnused
fi
if have gmake ; then
    alias make=gmake
elif have gnumake ; then
    alias make=gnumake
fi

The rest of them are mostly boring, with one exception:

alias macfile="perl -e 'tr/\x0d/\x0a/'"
have tidy && alias tidy='tidy -m -c -i'
have vim && alias vi='vim'
alias vlock='vlock -a'
alias fastscp='scp -c arcfour -o Compression=no' # yay speed!
alias startx='nohup ssh-agent startx & exit'
alias whatlocale='printenv | grep ^LC_'
alias fixx='xauth generate $DISPLAY'
alias whatuses='fuser -v -n tcp'
alias which=type
alias ssh='env TERM="$TERM:$LANG" ssh'
have realpath || alias realpath=realpath_func
if have readlink ; then
    unset -f readlink_func
else
    alias readlink=readlink_func
fi
if [[ $OSTYPE == darwin* ]]; then
    alias top='top -R -F -ocpu -Otime'
    alias cvim='exec_cvim'
    alias gvim='exec_cvim'
fi

Did you note that ssh alias? Heh.

Tab-completion Options

Bash has had, for a little while at least, the ability to do custom tab-completion. This is really convenient (for example, when I've typed cvs commit and I hit tab, bash can know that I really just want to tab-complete files that have been changed). However, I won't bore you with a long list of all the handy tab-completions that are out there. Most of mine are just copied from here anyway. But I often operate in places where that big ol' bash-completion file can be in multiple places. Here's the simple little loop I use. You'll notice that it only does the loop after ensuring that bash is of recent-enough vintage:

completion_options=(
~/.conf/bash_completion
/etc/bash_completion
/opt/local/etc/bash_completion
)
if [[ $BASH_VERSION && -z $BASH_COMPLETION && $- == *i* ]] ;
then
    bash=${BASH_VERSION%.*}; bmajor=${bash%.*}; bminor=${bash#*.}
    if [ $bmajor -eq 2 -a $bminor '>' 04 ] || [ $bmajor -gt 2 ] ;
    then
        for bc in "${completion_options[@]}" ; do
            if [[ -r $bc ]] ; then
                dprint Loading the bash_completion file
                if [ "$BASH_COMPLETION" ] ; then
                    BASH_COMPLETION="$bc"
                fi
                #COMP_CVS_REMOTE=yes
                export COMP_CVS_ENTRIES=yes
                source "$bc"
                break
            fi
        done
    fi
    unset bash bminor bmajor
fi
unset completion_options

Machine-local settings

You'd be surprised how useful this can be sometimes. Sometimes I need machine-specific settings. For example, on some machines there's a PGI compiler I want to use, and maybe it needs some environment variable set. Rather than put it in the main bashrc, I just put that stuff into ~/.bashrc.local and have it loaded:

dprint checking for bashrc.local in $HOME
if [ -r "${HOME}/.bashrc.local" ]; then
    dprint Loading local bashrc
    source "${HOME}/.bashrc.local"
fi

Auto-logout

Lastly, it is sometimes the case that the TMOUT variable has been set, either by myself, or by a sysadmin who doesn't like idle users (on a popular system, too many idle users can unnecessarily run you out of ssh sockets, for example). In any case, when my time is limited, I like being aware of how much time I have left. So I have my bashrc detect the TMOUT variable and print out a big banner so that I know what's up and how much time I have. Note that bash can do simple math all by itself with the $(( )) construction. Heheh. Anyway:

if [[ $TMOUT && "$-" == *i* ]]; then
    echo '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'
    echo You will be autologged out after:
    echo -e -n '\t'
    seconds=$TMOUT
    days=$((seconds/60/60/24))
    seconds=$((seconds-days*24*60*60))
    hours=$((seconds/60/60))
    seconds=$((seconds-hours*60*60))
    minutes=$((seconds/60))
    seconds=$((seconds-minutes*60))
    [[ $days != 0 ]] && echo -n "$days days "
    [[ $hours != 0 ]] && echo -n "$hours hours "
    [[ $minutes != 0 ]] && echo -n "$minutes minutes "
    [[ $seconds != 0 ]] && echo -n "$seconds seconds "
    echo
    echo ... of being idle.
    unset days hours minutes seconds
fi

dprint BASHRC_DONE

While I'm at it, I suppose I should point out that I also have a ~/.bash_logout file that's got some niceness to it. If it's the last shell, it clears sudo's cache, empties the console's scrollback buffer, and clears the screen. Note: DO NOT PUT THIS IN YOUR BASHRC You wouldn't like it in there.

if [ "$SHLVL" -eq 1 ] ; then
    sudo -k
    type -P clear_console &>/dev/null && clear_console 2>/dev/null
    clear
fi

And that's about it! Of course, I'm sure I'll add little details here and there and this blog entry will become outdated. But hopefully someone finds my bashrc useful. I know I've put a lot of time and effort into it. :)

Also, I noticed that w3m isn’t marked as depending on gdk-pixbuf. Strictly speaking, it doesn’t, but it does if you have --enable-image=x11. :P Add this to your Portfile:

depends_lib lib:libgccpp.1:boehmgc bin:gdk-pixbuf-config:gdk-pixbuf

Also, it seems that either w3m or gdk-pixbuf-config appends an extra library to the config line for gdk-pixbuf-config (essentially, they specify -lgdk_pixbuf AND -lgdk_pixbuf_xlib). That extra library causes build problems for w3m; you can fix it by editing /opt/local/bin/gdk-pixbuf-config and removing the -lgdk_pixbuf from what it prints out (however, if you use other software that uses gdk-pixbuf-config, you may need to put it back once w3m has finished building).

-d Sort in `phone directory’ order: ignore all characters except letters, digits and blanks when sorting.

What does that suggest to you? Well, let’s compare it to the GNU “sort” man page:

-d, —dictionary-order
consider only blanks and alphanumeric characters

So you’d THINK, right, that sorting with these two options would be equivalent, right?

Nope!

Here’s a simple list:

- 192.168.2.4 foo
- 192.168.2.42 foo

How should these things be sorted when the -d option is in effect? You’ve got a conundrum: is a space sorted BEFORE a number or AFTER a number?

Curse you, alphabet! You’re never around when I need you!

And, of course, BSD and GNU answer that question differently. On GNU, the answer is AFTER, on BSD the answer is BEFORE! Oh goody.

Here’s a better way if you need the sorting results to be the same on both BSD and GNU: replace all spaces with something else non-alpha-numeric that isn’t used in the file (such as an underscore, or an ellipsis, or an em-dash). Then sort with -ds (no last-minute saving throws!), then replace the underscore (or whatever) with a space again.

And if you need it to be consistent on OSX platforms too, make it a -dfs sort (so that capitals and lower-case are considered the same).

I have an atomic-increment function. To make it actually atomic, it uses assembly. Here’s the PPC version:

static inline int atomic_inc(int * operand)
{
    int retval;
    register unsigned int incrd = incrd; // silence initialization complaints
    asm volatile ("1:\n\t"
                  "lwarx  %0,0,%1\n\t" /* reserve operand into retval */
                  "addi   %2,%0,1\n\t" /* increment */
                  "stwcx. %2,0,%1\n\t" /* un-reserve operand */
                  "bne-   1b\n\t" /* if it failed, try again */
                  "isync" /* make sure it wasn't all just a dream */
                  :"=&r" (retval)
                  :"r" (operand), "r" (incrd)
                  :"cc","memory");
    return retval;
}

Now, what exactly is wrong with that, eh? This works great on Linux. The general GCC compiles this just fine, as does the PGI compiler, IBM’s compiler, and Intel’s compiler.

Apple’s compiler? Here’s the error I get:

gcc -c test.c
/var/tmp/ccqu2RmV.s:5949:Parameter error: r0 not allowed for parameter 2 (code as 0 not r0)

Okay, so, some kind of monkey business is going on. What does this look like in the .S file?

1:
    lwarx r0,0,r2
    addi   r3,r0,1
    stwcx. r3,0,r2
    bne-   1b
    isync
    mr r3,r0

It decided (retval) was going to be r0! Even though that’s apparently not allowed! (FYI it’s the addi that generates the error).

The correct workaround is to use the barely documented “b” option, like this:

static inline int atomic_inc(int * operand)
{
    int retval;
    register unsigned int incrd = incrd; // silence initialization complaints
    asm volatile ("1:\n\t"
                  "lwarx  %0,0,%1\n\t" /* reserve operand into retval */
                  "addi   %2,%0,1\n\t" /* increment */
                  "stwcx. %2,0,%1\n\t" /* un-reserve operand */
                  "bne-   1b\n\t" /* if it failed, try again */
                  "isync" /* make sure it wasn't all just a dream */
                  :"=&b" (retval) /* note the b instead of the r */
                  :"r" (operand), "r" (incrd)
                  :"cc","memory");
    return retval;
}

That ensures, on PPC machines, that the value is a “base” register (aka not r0).

How gcc on Linux gets it right all the time, I have no idea. But it does.