We are aware that our service is currently unavailable and we have been actively working on the issue since it began at approximately 6am MST. We hope to have everything back online ASAP and will post updates here as they come in.
May '19
May '19
We're back online and we'll keep monitoring the servers to be sure nothing else goes wrong, but we're all clear now. We are very sorry for the trouble and thank you for your patience.
Could you please give use some additional details about the the problem was? That was some serious downtime and a little more information as to what went wrong would be appreciated.
It was a sudden, massive spike in DDoS (distributed denial of service) attacks against our system. Basically, instead of trying to hack into our system (which hasn't happened), they just hammer us with so many bogus connection requests that our system has trouble responding to regular user/visitor traffic -- like a "flash mob" packing a store with so many people at once that regular customers can't do business there.
By engaging a "botnet" of malware-infected PCs around the world to perform the attack, the bogus connection requests don't all originate with just one or a few IPs that can easily be blocked, so it took some analysis to figure out which types of connection were the bogus ones and shut them out. Worse, our usual uptime monitoring tools weren't detecting that our system was down -- as it technically wasn't, just veeerrry sluggish to respond -- so we weren't even aware of the problem until our morning tech support started their shift and alerted DevOps personnel.
Dealing with DDoS traffic is unfortunately a daily fact of life for any ecommerce service these days, so yesterday's event was just more and worse of that than usual, and for no apparent reason. We've added some new system monitoring tools to catch this sort of thing better and earlier in the future, and we're evaluating options to block DDoS traffic in a more automated and effective manner.
