Author: dberger

Let me explain. No, there is too much. Let me sum up.

It’s been quite a while (two years) since I’ve posted anything here. The last two years have flown past – work has been busy, we’ve done a bit of traveling (in hind sight, I sorta wish I posted about our trips here), and generally had more good days than bad days.

So why, you might ask, am I resurrecting this blog?

Well, ’cause Dawnise and I are moving in December.

To Luxembourg.

As in that (tiny) little duchy sandwiched between Germany, France, and Belgium.

Valve is starting the process of opening a small office there, and I’m going to help do that.

Dawnise and I have been talking about this for a couple months – since the Valve office plan started really gaining momentum. We spent this past week in Luxembourg looking for an apartment. We found one about 2km outside thecentre ville that will accommodate us, the cats, and some as-yet-to-be-determined subset of our stuff.

When I said small office, I do mean small – there will be three of us initially; one who’s already there, one who’s heading over at the start of November, and me – others may join, and we’ll start looking to hire.

This could very well be one of the craziest things we’ve ever done – neither of us speak french (yet), and my memory of High School German isn’t even enough to get me in trouble anymore. We’ll know a total of five other people in the country, and unlike when we moved to Washington, I won’t be surrounded by an office full of potential friends.

We’ve thought a bunch about this, and there are lots of questions – it’s unclear what my role needs to become in this new setting, and Valves unique structure brings a unique set of challenges to working with folks who are several timezones and thousands of miles away.

There are about two million things that can still go wrong with this stunt.

Dawnise and I have said, several times while traveling in other countries, that we’d both like to experience of living abroad – so when this opportunity came up, it was basically too good to pass by.

We’ve been comparing this to being at the top of a roller coaster – looking over the edge into the drop. Very soon, things are going to start moving very fast.

So if you’re in Seattle – or are going to be in Seattle before the beginning of December – and want to grab dinner or something before we vanish for a year or two, drop us a line.

Wheeeeeee!!!!!

Rest In Peace, Junior

It’s been a while since I’ve made the time to post anything here – not that there hasn’t been stuff going on, just that life’s been, well, life.

This afternoon, Junior left us after a rocky couple of months. His diabetes, which had been under control without insulin for a couple years, suddenly returned, along with a smorgasbord of other symptoms. This afternoon, I walked into the bathroom to find him in the midst of a violent seizure – and he never came back.

He was with us for 15 years, and ingratiated himself with even the most stalwart anti-cat people. We have lots of record of his time here – as Dawnise has pretty much always thought of cameras as existing solely to capture images of the cats.

He will be missed, and the world has within it a bit less cute. Looking out the window, I’m sure the sky looks just a bit more silvery-grey than normal…

Postmortem of a catastrophic RAID failure

Wednesday of last week, I came home to find my three new 1TB hard disks waiting for me, destined to upgrade our ReadyNAS NV+.

Being a hot-plug-online-upgradable-all-singing-all-dancing sort of widget, I followed the recommended upgrade procedure and popped out one of the current 500GB drives, waited a few seconds, slotted one of the new 1TB replacements, waited ’till it started resynchronizing the volume, and went down to make dinner.

And spent the next several days picking up the pieces…

One critical bit of background – the NAS had three disks in a single RAID-5 volume. RAID-5 can tolerate one disk failure without data loss, but if two disks fail (regardless of the number of disks in the volume), kiss your data good bye.

When I went back upstairs after dinner to check on progress I discovered that the NAS had locked up, and completely dropped off the network. Wouldn’t answer it’s web management UI, and wasn’t responding to pings.

Hesitantly, I power-cycled it. It started booting, and hung about a quarter of the way through checking the volume.

After several reboot attempts all locking up at the same place, I applied a bit of coercion and convinced the box to boot. I checked the system logs and found nothing telling, removed and re-seated the new 1TB drive, and watched it start the resync again.

A couple hours later, sync still proceeding, I went to bed.

And woke the next morning to find the unit again fallen off the network.

Buried in the log messages – which I’d left scrolling past over night – was a warning that disk 2 was reporting SMART warnings about having to relocate failing sectors.

In other words, one disk of the three was being rebuilt while another one was busy dying.

At this point it became a race – would the rebuild complete (leaving me with two good disks, and intact data) before the failing one died completely.

In order to try to buy some insurance, I shut down the NAS, transplanted the failing drive into a spare PC, and started a disk-to-disk copy of it’s data onto the working 500GB disk I had removed at the start of this mounting disaster.

Despite valiant attempts by both dd_rescue and myrescue, the disk was dying faster than data could be retrieved, and after a day and a half of effort, I had to face the fact that I wasn’t going to be able to save it.

Fortunately, I had setup off-site backups using CrashPlan, so I had Vince bring my backup drive to work, and retrieved it from him on Friday.

Saturday was spent restoring our photos, music, and email (more later) from the backup.

Unfortunately, despite claiming to have been backing up Dawnises inbox, it was nowhere to be found in the CrashPlan backup set, and the most recent “hand-made” backup I found was almost exactly a year old (from her PC to Mac conversion). Losing a year of email is better than losing everything, but that seems like meager consolation under the circumstances.

By Saturday night I had things mostly back to rights, and had a chance to reflect on what had gone wrong.

The highlights:

1. SMART, as google discovered (and published) is a terrible predictor of failure. The drive that failed (and is being RMAd under warranty, for all the good it’ll do me) had never issued a SMART error before catastrophically failing.

2. In retrospect, I should have rebooted the NAS and done a full volume scan before starting the upgrade. That might have put enough load on the failing drive to make it show itself before I had made the critical and irreversible decision to remove a drive from the array.

3. By failing to provide disk scrubbing (a process whereby the system periodically touches every bit of every hard disk) the ReadyNAS fails to detect failing drives early.

4. While I had done test restores during my evaluation of CrashPlan, I had never actually done a test restore to Dawnise’s Mac. Had I done so, I might have discovered the missing files and been able to avoid losing data.

I have a support ticket opened with the CrashPlan folks, as it seems there’s a bug of some kind here. At the very least, I would have expected a warning from CrashPlan that it was unable to backup all the files in it’s backup set.

5. In my effort to be frugal, I bought a 500GB external drive to use as my remote backup destination – the sweet spot in the capacity/cost curve at the time.

Since I had more than 500GB of data, that meant I had to pick and choose what data I did and didn’t backup. My choices were ok, but not perfect. There’s some data lost which should have been in the backup set, but wasn’t due to space limitations.

6. CrashPlan worked well – but not flawlessly – and without it, I’d have been in a world of hurt. Having an off-site backup means that I didn’t lose my 20GB worth of digital photos, or several hundred GB of ripped music.

Aside from digital purchases, the bulk of the music would have been recoverable from the source CDs, but at great time expense. The photos would have just been lost.

7. In this case, the off-site aspect of CrashPlan wasn’t critical, but it’s easy to imagine a scenario where it would have been.

8. The belief that RAID improves your chances of retaining data is built largely on what I’m going to refer henceforth to as “The RAID fallacy” – that failure modes of the drives in the array are completely independent events. The reality is that many (most?) RAID arrays are populated with near-identical drives. Same manufacturer, same capacity (and model) , and often the same or very similar vintage. So the drives age together under similar work loads, and any inherent defect (like, say, a firmware bug that causes the drives not to POST reliably) is likely to affect multiple drives, which spells disaster for the volume.

Loss

My uncle Alan died last week, on April 1st, after being removed from life support. I’ve sat down to write about it several times since we got home, maybe this time I’ll actually finish…

We got a call last Saturday that he was going to be unplugged on Sunday, with as many family members who could be, and wanted to be, present. I wanted to be there to say good bye, all accounts were that the man was already gone – only the shell remained.

After discovering that flights would have cost us the better part of $1100, we hit the road Sunday morning for the drive down. We got a call from my dad just as we were crossing the OR/CA border that Alan had passed peacefully and quickly after being removed from the respirator.

Alan was one of eight children – he was my mom’s baby brother – and the only one of my mom’s siblings I really know (knew?). Alan and Amy lived near us for most of my life, and I spent a lot of time with them – holidays, parties, and for no particular reason at all. I guess that’s what people mean by “family.”

When Dawnise and I were looking for our first place together, Chino Hills had the double advantage of being vaguely centrally located to our relative commutes, but more importantly – it was close to Amy and Alan. So we rented a condo, and became neighbors. When we bought our first house, it was literally around the corner from them – and we’d invite them to dinner at their place regularly : Amy’s cat allergies prevented us from actually hosting. Dawnise came to think of Amy and Alan as the aunt and uncle she never had.

I only wish that I had been able to tell him how much he meant to me while he was still around to hear it.

The service was well attended – with much laughter during the shared remembrance. As an ER nurse, Alan touched countless lives, and many of his co-workers – and at least one former patient – came to pay their respects. Many folks took the time to share stories and memories of Alan – starting with my mom, and including myself, my aunt Dawn, Dawnise and many others.

My brother couldn’t be at the service – he’s on a 40-odd day wilderness trek through Utah – and he (and my sister) are at least as close to Amy and Alan as I am. I said a few words on his behalf – ’cause he couldn’t be there to say them – and I suspect even had he been there, he’d have had a hard time doing so.

I was told afterwards that I spoke well – but to be honest, I didn’t know what I was going to say when I walked up there, and I had little recollection of what I said when I was finished.

The centerpiece of the service was a photo montage, assembled by Shawna, Laurie, Dawnise and Christopher. I’m working on making it available online for those who couldn’t be there to see it.

My aunt Amy, in the midst of all this, managed to throw a party Alan would have attended – which is exactly what he would have wanted.

One of the songs used in the montage was Tim McGraw’s Live Like You Were Dying. When it came on the radio late Thursday night on our drive home I finally lost it, and sat sobbing in the car while Dawnise drove.

Gone, but never forgotten.

What’s this “6th floor” business?

The short answer is that for just under the past month, Dawnise and I (and the cats) have been living in an one-bedroom apartment on the south east corner of the 6th floor of the south east tower of the Harbor Steps – a relatively upscale apartment building in downtown Seattle. We can just see the waterfront from the bedroom window, and the living room has windows all along the east-facing wall, with a view into the city. We’re not really high enough to get a great panorama.

My walk to work is about 15 minutes, along the water, through Pikes Place Market. It’s almost surreal – watching the grocers and fish-mongers setting up in the morning – seems like something out of a movie. We have one of the cars with us – and take it out occasionally (mostly for trips to the local Costco, and yesterday for a jaunt to Archie McPhee’s), but most everything we need is in walking distance.

I’ve never been a “city person”; always lived in the suburbs, figured that driving was pretty much built-in to getting anywhere. I’ve spent enough time in NY to have had a taste of city life, and it never really appealed to me. Which is why it was completely unexpected when I realized that when we ultimately buy a house up here (more on that later) I’ll miss living down-town.

[tap, tap] ‘s this thing on?

I had a strange experience the other day – I finally “got” LiveJournal. I had always thought of LiveJournal (and blogging in general) as the most egregious example of sharing your proverbial vacation photos with an uninterested world.

A year or so back, Titus tried to explain the value of LJ to me: how he and his classmates at Mudd had been inseparably close until graduation and then scattered to the wind, and that using LJ was a tool to keep in touch, after a fashion.

Time passes, and in the middle of October, Dawnise and I move to Seattle. The other day, I wanted to feel connected to friends in Southern California. And blogging made sense.

I tried signing up for Live Journal – and their registration insisted that all my proposed passwords were based on reversed dictionary words (which they aren’t – not even remotely) and I gave up after three attempts. Instead, I grabbed Movable Type and installed it on the webserver that hosts my domain. I’m not sure how often I’ll post here, but for whatever it’s worth, I have a blog…