Between the hours of 5PM and 7:30PM Eastern Time on Tuesday October 25th, 2011, Tech Superpowers email hosting customers experienced an outage where emails which were being sent to all TSP email accounts was not delivered in a timely manner. During this period, the mail server was holding emails for delivery and not delivering them to user mailboxes, but no email was lost. No other services were affected outside of email.
The reason for this outage was a malfunction in our spam filters which resulted in them not processing emails. These issues were an unanticipated result of actions which an email administrator was taking to increase the efficacy of our spam filters, and unfortunately, a script was run which caused the databases which run the spam filters to seize up and not function. Since all email goes through our filters, that resulted in incoming emails being held by the email server until we resolved the issue.
Once we realized that this issue was impacting customer emails (including our own), we determined the cause of the problem and fixed the issue. However, by that point, email had already been queued up for quite some time, so it took until around 7:30PM ET for all queues to clear.
We take full responsibility for causing this outage and realize that any delay in email delivery - no matter how long - is unacceptable for our customers. In addition, our hosting status line was not updated in a timely manner for our customers, resulting in much less information being available to customers than in previous outages.
We have begun an internal review of our process for making minor adjustments to our mail services. While we have strict team-wide reviews of major maintenance work, including protocols for doing this work during non-impacting hours, informing customers and posting information to our status line and website, and reducing the impact that it has on customers, many smaller changes are made to our mail services as part of day to day activities. We will make sure that going forward that we eliminate any possibility that these smaller issues will cause service-wide disruption as it did today.
We fully admit that we fell short of our own expectations and those of our customers for our service and apologize for the delay in service.
During any outage, our Hosting Status Line (+1-617-778-1225) is always the first line of information - as sometimes network and server issues may affect other methods of communication (i.e. web or email). In this case, we did not meet our goal of updating this message within 15 minutes of becoming aware of an outage, but this line will be updated in the future with current information.
Please direct any follow up questions to hosting@techsuperpowers.com regarding this outage.
Sincerely,
Tech Superpowers Hosting Team
