This may be a case of "success breeding complacency", as it's been such a long time since our own systems were responsible for any outage that our vigilance may have become unduly relaxed. Rest assured this event has shaken us up, and we are reviewing our internal outage communication and remediation policies and systems accordingly to ensure we won't drop the ball again in the future.
We do have systems monitoring our system's uptime and availability 24/7, which should have notified our sysadmin of this outage immediately; we have identified the reason this notification was not received in this case and corrected that issue, so that lapse should not recur. One of our staff noticed the outage within about half an hour after it started, but they merely sent an email to notify the rest of us, when of course they should have contacted our sysadmin personally, directly and persistently by every means available until they acknowledged the outage was being attended to.
Investigating the actual cause of the outage, tests we performed last night confirmed that all of our own internal systems and hardware were and are properly configured and fully operational, yet the failover to a fallback uplink still could not be triggered successfully, all of which indicates some sort of networking issue(s) at our colocation datacenter outside of our direct control were likely responsible for the outage. We have filed a trouble ticket with them to get the matter investigated and resolved, as this is the second time their "redundant" networking provisions have actually caused an outage for us, which is two more times than we're aware of it ever actually having prevented an outage.
