We’re F****D, It’s Over: Coming Back from the Brink

In 1997, about a year after launch, Hotmail was growing exponentially, adding thousands of new users every day. We were on fire. And then one night, it all seemed to unravel. We had a program called the “janitor” that ran as an overnight batch process and it erased all of the email that users put in the “trash” folder. Except this night, a bug spawned an army of other janitors that cleaned out everyone’s inboxes, too. That’s right, deep-sixed their email. Here is what went through all our spinning heads: “We’re fucked, it’s over.”

It’s pronounced whiff-eee-o, that horrible, terrifying moment that nearly every entrepreneur goes through when they are certain that their company is dead. I’ve seen it happen in so many different ways: A legal ruling goes against you; Apple refuses to approve your app unless you change the feature that makes it special;  Google launches a competitive product, and aims right at you; A critical technology partner decides not to renew their contract. It’s that moment before you gather yourself to do battle, when all seems lost. I have struggled through that moment, first at Hotmail and again at IronPort. Coming out the other side, scars and all, there are a few critical things you take with you:

The Hotmail WFIO

Once we had pulled the plug on the extra “janitors”, it turned out that about 25% of our total users were affected. Not everyone, but holy shit, a quarter of our customers had lost everything. Understandably, they were pissed. CNET and ZDnet were both on the horn wanting to know what happened. Customer care was inundated with angry calls and (ironically) emails. We figured out how to restore a few thousand customers, but millions were completely unrecoverable. While the calls rolled in, we were trying to figure out how to fix things.

I remember Hotmail’s CEO Sabeer calling us all in a room. “Hey, Rex (the COO), how long will it take to restore the email from the tape backups?” And I’ll never forget his answer: “Um, those got really expensive, so we stopped doing them about a month ago.” Gulp. Long … painful … silence.

The higher-level problem was that people were just getting comfortable trusting us with hosting their email and now we had completely let them down. Their email was just gone. We did a lot of communicating to the users and promised them we would “grandfather” them in for some planned paid services for free. We apologized profusely and explained how the sun, the moon and the stars lined up against us for it to happen. We also clearly explained what steps we were going to take so that it would never happen again. Over time, they started receiving more email, their inboxes filled up, and we just rode it out.

The IronPort WFIO

At IronPort, we developed a super fast and scalable email gateway that ran roughly 10 times faster than any other alternatives. We were definitely in the right place and time when the spam flood came. Our gateway was the only one that could handle the load. Just as at Hotmail, we were adding customers as fast as we could get the company names jotted down, but we needed to add an anti-spam component to our offering.

We struck up a partnership with Brightmail, the leading anti-spam software company, and the joint product – an IronPort gateway with Brightmail – was unbeatable in the marketplace. The only problem was we were heavily dependent on each other with both companies scrambling to build what each other had. Over time we both knew there was going to be a day of reckoning.  We attempted to merge the two companies to solve the problem, but the VCs couldn’t agree on terms.  And then Symantec bought Brightmail.

We knew the clock was ticking and had most of the engineering team working on IronPort anti-spam. But it was way late and wasn’t working. Shortly after the Symantec acquisition, we started hearing reports from channel partners that they were planning to cancel our contract. Although we knew that day would eventually come, we were totally unprepared! WFIO!

As detailed in a prior post, my VP of engineering, Nawaf, “cracked the egg with a sledgehammer” and got our anti-spam product working just in time.  Instead of Symantec canceling the contract, we went on the offensive and faxed a letter to all of our customers cancelling the contract with THEM– a position of strength. We managed through the madness and got to the other side.

After the IronPort WFIO had receded in my rear-view, I realized that you can almost always get to the other side. You just need to keep in mind a few things, and have some emergency tools ready to pull out.

It’s never really as bad as it seems.

Companies are damn resilient. Although it certainly feels like death at the time, it rarely is and companies just keep on moving forward. Ingenuity and guts usually help you find your way out of the jam. In fact, much of a company’s value is usually created by figuring out a solution to the big obstacle. Certainly, that was the case at IronPort where our entire business was built on the back of our anti-spam product.

Get all the brains around the table.

Whenever we went through a WFIO, we’d get all of the smartest people in the room and work through every angle. This is a little counterintuitive because most leaders have the tendency to share very little with the extended team as they are worried about freaking them out. This is a mistake on a number of fronts – trusting your team in crisis brings the company together in amazing ways and their contributions may very well save your company. All of our serious issues resulted in an “Apollo 13” atmosphere where we’d bring together top engineers, architects, VPs – anybody that could materially contribute – and hash it out.  Fighter pilots, who are constantly in pursuit of perfection, have what they call a “rank-less” debrief after every mission where everyone involved, regardless of rank, speaks up to criticize what went wrong.

Lead from the front.

This is the time for leadership – you cannot punk out. I think about a ship captain sailing the Atlantic in the 1700’s and rolling into a huge storm: regardless of how fearful, doubtful, or just scared shitless you may be, there’s only one way to play it with your team – you are in total control. Think of how much it could affect the outcome?! The team needs you to lead them through the problem. Back to the ship captain, what if he grabbed a bottle of whiskey and holed-up under his bed? The ship would certainly be lost. However, if he calmly makes a pot of coffee, ties himself to the bridge, and starts shouting orders, then I believe the chances of the ship making it through go up dramatically. The leader needs to be the first one there, the last one to leave, and be willing to do anything it takes – like answer customer care calls or personally drive a replacement part to an irate customer. Nothing is beneath a leader in times of crisis.

In between Hotmail and Ironport, I started an ecommerce company that got up to about six people. I funded it myself. When I couldn’t get funding for it and I had burned through all of my Hotmail money, I shut it down.  Sometimes it really is over, and I believe that one of the clearest signs is when you are completely out of cash.  But right up until that point, when the music is still playing, when your team is still driving hard, you  have a shot – and usually a decent one.