Outage 16Apr2012 ~2:10-2:25pm MST/PDT

[E-junkie_Guru]

We suffered a brief (~15 min.) service outage today due to an apparent power failure at the datacenter where our servers are colocated. They had announced they'd be performing maintenance today to upgrade their backup-power system which failed on 11Apr2012 (see http://www.e-junkie.com/bb/topic/5848), but this should have been a non-interrupting maintenance event, so we're still awaiting a full report from them explaining just what happened this time. We seem to be back up now, will post further details here as they come to light. We do apologize for this unfortunate incident and thank you for your patience and understanding.

[E-junkie_Guru]

There appears to be some sort of intermittent after-effect causing sporadic lapses in cart availability. Development is already investigating the matter and should have it nailed down forthwith. Thanks again for your patience, with our apologies for the inconvenience and concern.

[E-junkie_Guru]

Development found the intermittent issue and corrected that, so we should be back to normal operation now. Thank you all once again for your patience and understanding as we recovered from this unexpected power outage. Suffice to say we'll be having stern words with our datacenter's management regarding what happened at their facility today and what they'll be doing to ensure such issues do not recur.

[E-junkie_Guru]

The latest status report from our datacenter regarding this issue:

While replacing dead batteries, expert technicians from two companies failed to spot a short circuit and popped a breaker.



They will try again at 5:00PM [PDT/MST], this time trying to test many more things before actually turning it on. We anticipate no outage but are letting you know so you're aware.

[cynthia]

I am still unable to access my transaction logs.

[ellen]

Sorry, transaction log queries continue to be down while I perform maintenance on the machine which serves them; it did not take having its power cut abruptly particularly well. I will post an update when it's back online.



--Ellen (this has not been my favorite day ever)

[shana]

Hi, I'm still unable to view my transaction logs... are you sure it's all resolved?

[ellen]

Ok, as of about 9:12 PM MST (4:12 AM UTC), the transaction log query server is back running and caught up with all current transactions.