Last week, Google suffered several major outages. Naturally, it made the news. So many people rely on Google for search, Gmail, YouTube, and maps, not to mention Drive, which some companies use to prepare documents and presentations. We don’t fault Google for having an outage. We make software, so we know how difficult it can be, even for a company with Google’s resources, to avoid this situation. However, we do think the way the outage was handled for Gmail was unforgivable and shows a lack of respect for users of Google’s services.
What Happened Was…
On Tuesday, December 15th, 2020, Gmail stopped accepting email sent to Gmail. It lasted from approximately 1:30 to 4:15 PT—the middle of the afternoon on the West Coast, and toward the end of the workday on the East Coast. Bad times for an outage like this when you consider some businesses send their mailings out as soon as they are ready to go and, in the United States anyway, that is exactly when Google went dark.
This shouldn’t have been a problem for sending email. The protocol that defines how email is sent allows for a store-and-forward system where a mail server can hold email until a server is ready to accept it, making multiple retries over time to get the email delivered. If Gmail had gone down such that its servers simply didn’t respond, anyone attempting to send email to a Gmail user would have their email delayed, and that would be the only negative result. After the servers came back up the email would be delivered, and the system would catch up.
However, Google didn’t make the Gmail servers unavailable. Instead, they were left running but answered requests to receive email with a message that the email address was not valid. For a user sending an email to a friend or relative and getting a message back that the address is not valid might be confusing at best. Less sophisticated email recipients might assume that the invalid email message is correct and go ahead and remove that contact from their address book or, worse, assume that their own system was hacked when their messages to friends started returning invalid email notices, leading to a lot of wasted time on hold with their Internet provider’s tech support.
The Effect on ESPs
For the business of email marketing, the results are more serious. It’s common for Email Service Providers (ESPs) to set their software up to remove recipients when the destination email server says the recipient is not valid. We do it, as do many others. This is done to avoid potential future deliverability problems. Sending repeatedly to an invalid email address is a good way to ruin your reputation score.1 In the Google case, it meant that a large amount of email wasn’t delivered to recipients. Worse than this, it meant that huge numbers of valid Gmail users were needlessly removed from ESP databases all around the world.
Fortunately, we were made aware of the problem early (the only real advantage to it happening in the middle of the workday). We contacted our customers and reversed the on-hold status for the hosted customers and the on-premise customers for whom we manage deliverability; but if you are using a different ESP or managing deliverability issues yourself, you should check to make sure your email lists have been corrected after this problem. We assume most quality ESPs will be proactively resolving this issue, but it would be good to check. If you run your own email marketing program, you’ll need to work with your programmers and database administrators to handle this issue.
Google’s handling of this outage was disappointing. Leaving the Gmail servers up but responding to all email requests with “Unknown User” needlessly caused all kinds of problems and confusion for users. We would go so far as to say that it was a very rude thing to do. If Google had simply turned off the computers, there would be almost no negative impact from the outage beyond delayed delivery. Google employs dozens, if not hundreds, of people in its Gmail division. Was there no one there who could simply pull the plug on the Gmail servers?
Turning off a server is easy, but they were unwilling to take this step, apparently unconcerned about how it would impact their users and anyone who wanted to contact them. Is this a sign of how Google feels about its users? Google removed the “Don’t be evil” motto from their code of conduct several years ago. We now see why.
1. For more on what a Reputation Score is and why it’s important, see the Deliverability Enhanced white paper in the Resources section of the Goolara website.
© Goolara, LLC, 2020. Unauthorized use and/or duplication of this material without express and written permission from this site’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Goolara, LLC and the Goolara Blog with appropriate and specific directions (i.e., links) to the original content.