On the afternoon of Thursday 12 November, half the world discovered at the same moment how much of its work runs on email. Some people found out that day that they didn’t have a phone number for half their clients.
Outlook stopped working, and the outage dragged on into Friday. Teams, SharePoint and OneDrive were also affected, to varying degrees, although Teams chats and calls largely kept working. At the peak, for Outlook alone, Downdetector logged close to 130,000 problem reports.
According to Microsoft, it was not an attack. The provisional explanation is that a process responsible for setting up secure connections became overloaded and stopped handling requests. The definitive cause will only be known when Microsoft publishes its incident report. Anyone who has ever worked late on a Friday afternoon knows the feeling.
Before going further, a declaration of interest: we have outages too. Anyone who runs infrastructure and says otherwise is lying or hasn’t been doing it long enough. Microsoft’s engineering is excellent and this could have happened to any provider, including us.
That said, there are useful lessons here.
The problem wasn’t the outage. It was having everything in one place.
For years we were sold the idea of having everything in one place. Fewer contracts, fewer invoices, fewer suppliers. All true.
What the sales pitch leaves out is the part where, when that one place has a bad day, the whole company has a bad day. Those with email in one place and files in another had an annoying afternoon. Those with everything together went home early.
Three things not to do while it’s happening
Don’t touch the configuration. It’s tempting, and it’s the expensive mistake. Reinstalling email clients, changing passwords or editing domain records during a global outage creates new problems that linger into the following week, long after the original one has been fixed.
Don’t go silent. A text message or a phone call saying that email is unstable avoids the most unfair interpretation of all: that nobody replied because nobody could be bothered.
Don’t rely on memory. Write down the time, what stopped and how long it lasted. In three months nobody will remember, and that note is what comes in handy when the contract is renegotiated.
Four questions for next week
Once the dust settles, it’s worth half an hour of conversation:
- If our main provider goes down for six hours, what still works here?
- If we wanted to switch email provider, could we take everything with us? How long would it take?
- Does our contract promise any level of availability? And does anything happen if it isn’t met, or is it just a nice-sounding line?
- Who tells our clients next time, and through which channel?
For public bodies, we’ve put together a few more in Eight questions a public body should ask its email provider.
And the moral of the story
It’s not that the big providers are bad. Their availability is better than what most companies could achieve on their own.
The real point is different: reliability is not measured only by how often something fails, but by what happens when it does. How much of the company stops at the same time, what information is available, and whether there is someone to call.
And it starts before that, with knowing who the company depends on and which law that provider answers to. We wrote about it in Who does your email answer to?
We will have bad days too. The difference is that, on those days, someone who knows your company’s name picks up. It is not a promise of perfection, just a phone that gets answered.
Sources
- Al Jazeera (archived copy)
- azureblog.co.uk, timeline (archived copy)
- Microsoft reference: incident EX1550543
