There was an outage on May 9th affecting Canadian toll-free traffic into RingCentral's system from a large national telco.
Due to the way we configured our test of RingCentral's services (call forwarding all calls into our local DID to our main toll-free number, 844-825-5226), this affected all incoming calls to our organization. Anecdotally from my own testing during the outage, only 1 out of every 5 to 10 calls completed.
This only seemed to affect calls originating from the PSTN network, however: cell phones, landlines; even to RingCentral's own toll-free Canadian support number from PSTN sources. Calls originating with VoIP were unaffected. I could use Google Hangouts, my own private VoIP service, or a number of other alternatives to successfully call our company's number.
We lost a good deal of trust and goodwill, both internally towards RingCentral, and externally towards our organization, as a result of the outage. At least one donor threatened to cancel their support if they had any more problems getting through to us. Issues on this scale, lasting for as long as they did (18 hours), cannot happen again.
If it were to come to an internal vote at this moment, I would need to cancel our RingCentral service because of the mistrust with which it is viewed by our staff, who are justifiably concerned about our reputation, and their ability to do their jobs. Part of the mistrust is borne from other technical issues we have experienced, which I am not nearly as concerned about; they can be addressed as tickets elsewhere.
My main concern in this post is to set proper expectations for myself and for Catholic Christian Outreach going forward, so I have a series of questions related to the outage which I hope you can help me address. Marie, a support representative I spoke with yesterday, suggested this forum as the best place to find the answers I am seeking.
1. How often have outages of this nature occurred in the past?
2. Is the issue solely related to the telco's equipment, or are any of the factors that led to the outage within reach of RingCentral's ability to fix?
3. How likely is it, theoretically, that we should expect similar outages in the future?
4. If it is an issue of missing redundancy, can the problem be resolved far enough upstream, so that, by working with a second telco, problems like this can be addressed by making the necessary structural changes to call routing to and from Canada into RingCentral's systems?
Specifically for the last question, I raise it because every other cloud-based VoIP vendor I spoke with, when they found out we had chosen RingCentral for further testing and a new contract, voiced major concerns about your ability to handle the amount of traffic, or the complexity of the setup that we were proposing. I don't mind mentioning the companies by name: Vonage, Versature, and NuEdge Communications (reseller of Avaya solutions out of Ottawa).
Versature specifically differentiated themselves by stating they had redundant connections to telcos which, in my mind, would seem to avoid the problem that led to the outage in May that we suffered from.
I don't know enough of the technical details to say either way whether they are correct, so I am asking directly: is the issue leading to the outage of May 9th completely outside of your control, and common to all VoIP vendors based in the U.S. who would have to work with the one telco to receive toll-free traffic, or is it specific to RingCentral? Could it be solved through seeking a redundant contract with a second telco? If so, is this planned in the aftermath of the outage?
We will decide whether or not to renew our RingCentral service solely based on the answers I receive to my questions here, before ever dealing with the other technical issues we have encountered over two months of testing. Obviously, I am not asking for a level of detail that would compromise the security of your internal network, but rather some assurance through additional information that the problem is not impossible to solve.
I appreciate your assistance. Good alternate ways to reach me outside this post are by email at christopher.mcmullan@cco.ca, or by cell phone at 613-231-3408. My name is Chris McMullan, and I am the Systems Administrator for Catholic Christian Outreach.
Due to the way we configured our test of RingCentral's services (call forwarding all calls into our local DID to our main toll-free number, 844-825-5226), this affected all incoming calls to our organization. Anecdotally from my own testing during the outage, only 1 out of every 5 to 10 calls completed.
This only seemed to affect calls originating from the PSTN network, however: cell phones, landlines; even to RingCentral's own toll-free Canadian support number from PSTN sources. Calls originating with VoIP were unaffected. I could use Google Hangouts, my own private VoIP service, or a number of other alternatives to successfully call our company's number.
We lost a good deal of trust and goodwill, both internally towards RingCentral, and externally towards our organization, as a result of the outage. At least one donor threatened to cancel their support if they had any more problems getting through to us. Issues on this scale, lasting for as long as they did (18 hours), cannot happen again.
If it were to come to an internal vote at this moment, I would need to cancel our RingCentral service because of the mistrust with which it is viewed by our staff, who are justifiably concerned about our reputation, and their ability to do their jobs. Part of the mistrust is borne from other technical issues we have experienced, which I am not nearly as concerned about; they can be addressed as tickets elsewhere.
My main concern in this post is to set proper expectations for myself and for Catholic Christian Outreach going forward, so I have a series of questions related to the outage which I hope you can help me address. Marie, a support representative I spoke with yesterday, suggested this forum as the best place to find the answers I am seeking.
1. How often have outages of this nature occurred in the past?
2. Is the issue solely related to the telco's equipment, or are any of the factors that led to the outage within reach of RingCentral's ability to fix?
3. How likely is it, theoretically, that we should expect similar outages in the future?
4. If it is an issue of missing redundancy, can the problem be resolved far enough upstream, so that, by working with a second telco, problems like this can be addressed by making the necessary structural changes to call routing to and from Canada into RingCentral's systems?
Specifically for the last question, I raise it because every other cloud-based VoIP vendor I spoke with, when they found out we had chosen RingCentral for further testing and a new contract, voiced major concerns about your ability to handle the amount of traffic, or the complexity of the setup that we were proposing. I don't mind mentioning the companies by name: Vonage, Versature, and NuEdge Communications (reseller of Avaya solutions out of Ottawa).
Versature specifically differentiated themselves by stating they had redundant connections to telcos which, in my mind, would seem to avoid the problem that led to the outage in May that we suffered from.
I don't know enough of the technical details to say either way whether they are correct, so I am asking directly: is the issue leading to the outage of May 9th completely outside of your control, and common to all VoIP vendors based in the U.S. who would have to work with the one telco to receive toll-free traffic, or is it specific to RingCentral? Could it be solved through seeking a redundant contract with a second telco? If so, is this planned in the aftermath of the outage?
We will decide whether or not to renew our RingCentral service solely based on the answers I receive to my questions here, before ever dealing with the other technical issues we have encountered over two months of testing. Obviously, I am not asking for a level of detail that would compromise the security of your internal network, but rather some assurance through additional information that the problem is not impossible to solve.
I appreciate your assistance. Good alternate ways to reach me outside this post are by email at christopher.mcmullan@cco.ca, or by cell phone at 613-231-3408. My name is Chris McMullan, and I am the Systems Administrator for Catholic Christian Outreach.
- Account type
Canada - Related case number (if applicable)
N/A - Detailed description of problem
Described above. - Goal
Further information about the outage on May 9th, incident ID INC-17370 - Previous troubleshooting steps taken
Work during the outage to narrow down the source of affected calls (VoIP vs. PSTN), and two support calls to verify the incident ID and follow up with the questions I pose here. - Software version
N/A - If desk-phone related, have you rebooted phone?
N/A
