ACTIVE - Mimecast has reopened this as an Active Incident for the US-hosted customers. Below are updates directly from Mimecast. You can also view the status of the Incident anytime here, https://status.mimecast.com/.
Excel Micro is monitoring the issue closely and will provide updates as they become available.
---From Mimecast---
Incident Status: Degraded Performance
Components: United States Grid
Service: Email Processing, Administration Console, Search, Web Security, Awareness Training, URL Protect, Attachment Protect, Impersonation Protect, User Applications, CyberGraph
May 3, 2022 11:59PM UTC
[Resolved]We can confirm we experienced a major power outage in one of our US data centers that impacted all power supply sources, including the backup generators, which in turn caused a cascading issue and resulted in degraded performance for US-hosted customers. We now understand the impact and have addressed the underlying issues with our datacenter infrastructure. Services are now fully available and we are processing email normally. Some customers may still experience delays until the backlog clears. We sincerely apologize for the extended outage and the disruption caused.
[Identified] We have identified the root cause of the issue affecting mail flow and service availability on the North American grid. While we continue to see steady improvement, some customers may still experience delays in mail flow as the backlog clears. We will closely monitor application availability and performance and will provide an additional update when all services are functioning normally.
May 3, 2022 10:58PM UTC
[Identified]We have identified the root cause of the issue affecting mail flow and service availability on the North American grid. Customers may still experience some delays in mail flow as the backlog continues to clear. We will closely monitor application availability and performance and will provide an additional update when all services are functioning normally.
May 3, 2022 10:26PM UTC
[Investigating] Recovery is continuing for the incident affecting customers on our US Gird, please expect slight delays in mail flow processing as the backlog clears. We will continue to closely monitor application availability and performance and will provide additional updates here.
May 3, 2022 9:09PM UTC
[Investigating] We believe we have identified the root cause of the issue affecting mail flow and service availability on the North American grid. Recovery is continuing, but there may be some delays in mail flow as the backlog clears. We will continue to closely monitor application availability and performance and will provide an additional update when all services are functioning normally.
May 3, 2022 8:24PM UTC
[Investigating] Recovery is continuing for the incident affecting some US-hosted customers, but there may be some delays in mail flow as the backlog clears. We will continue to closely monitor application availability and performance and will provide an additional update when all services are functioning normally.
May 3, 2022 7:50PM UTC
[Investigating] While we continue to see an improvement on both mail flow and service availability and work to clear the backlog we are still investigating reports of service degradation and mail flow delays impacting some of our US-hosted customers. We will post further updates here as they become available.
May 3, 2022 7:06PM UTC
[Investigating] We are continuing to investigate reports of service degradation and mail flow delays impacting some of our US-hosted customers. We will post further updates here as they become available.
May 3, 2022 5:54PM UTC
[Investigating] We are aware that some customers may still be experiencing continued service degradation and mail flow delays. We are currently investigating and will provide updates here.
May 3, 2022 4:28PM UTC
[Monitoring] We can confirm that email processing and all Mimecast services have been restored for our US-hosted customers. All services should now be functioning normally and processing queues are approaching normal levels as email is retried on the normal SMTP schedules. We continue to monitor and will provide updates here.
May 3, 2022 3:58PM UTC
[Identified] We can confirm we experienced a major power outage in one of our US data centers that impacted all power supply sources, including the backup generators, which in turn caused a cascading issue and resulted in degraded performance. Midway through our Disaster Recovery Process, the power was restored which complicated our ability to recover. As of now, services are available and we are processing email but customers may still experience delays until the backlog clears. We sincerely apologize for the extended outage and the disruption caused.
May 3, 2022 3:03PM UTC
[Identified] We are seeing an improvement in service availability and stability. We are still seeing a reduced service as we work to mitigate these remaining issues. We apologize for the disruption caused and appreciate your patience. We will post further updates as soon as they become available.
[Identified] We are continuing to mitigate the issue faced by some customers hosted on our US grid. We are aware customers are still facing delays to our services as we work to remediate the issue. We apologize for the disruption caused and appreciate your patience. We will post further updates as soon as they become available.
[Identified] Following the earlier update, our mitigation is continuing and most customers hosted on our US grid should start seeing a return to normal service. Thank you for your patience, further updates will be posted here
May 3, 2022 12:49PM UTC
[Identified] We believe we have identified the issue and are taking steps to mitigate this behavior. Customers should start to see improved performance and application accessibility. Further updates will continue to be posted here.
May 3, 2022 12:31PM UTC
[Investigating] Mimecast is aware that some customers hosted on our North American grid may be experiencing issues with access to the Mimecast Administration Console, End User Applications, as well as delayed resolution of re-written URLs and/or mail flow delays. We are currently investigating. Further updates will continue to be posted here.
May 3, 2022 12:13PM UTC
[Investigating] We are aware that some customers hosted on our US grid may be experiencing a degraded service. We are currently investigating the issue and updates will be posted here as they become available.
Mike Amadio
Comments