Incident Report for Zix | AppRiver
Previously you were notified about a major incident we experienced on November 16, 2023, that impacted several of the Zix Services. All impacted services have been restored. The root cause for the issues impacting the Zix Services was a firmware bug affecting the SAN storage application utilized by Zix. Working with our SAN provider, we were able to identify the issue and ultimately replaced the faulty systems. Below is a summary of the services that were impacted and the resolution. Impact
· Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
· Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features.
· Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may have experienced a disruption in the archiving of their email communication and may have experienced intermittent issues accessing their archive.
Resolution
· Zix Email Encryption services have been fully restored.
· Zix Email Threat Protection services have been fully restored.
· Zix Archiving services have been fully restored.
More information In the following weeks we will create a detailed Root Cause Analysis (RCA) and make the available to customers. If you would like to receive a copy of the RCA, you can request it by sending an email to Zix Support.
Posted Nov 30, 2023 - 09:19 CST
Monitoring - Previously you were notified about a major incident we experienced on November 16, 2023, that impacted several of the Zix Services. All impacted services have been restored. The root cause for the issues impacting the Zix Services was a firmware bug affecting the SAN storage application utilized by Zix. Working with our SAN provider, we were able to identify the issue and ultimately replaced the faulty systems. Below is a summary of the services that were impacted and the resolution. Impact
· Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
· Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features.
· Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may have experienced a disruption in the archiving of their email communication and may have experienced intermittent issues accessing their archive.
Resolution
· Zix Email Encryption services have been fully restored.
· Zix Email Threat Protection services have been fully restored.
· Zix Archiving services have been fully restored.
More information In the following weeks we will create a detailed Root Cause Analysis (RCA) and make the available to customers. If you would like to receive a copy of the RCA, you can request it by sending an email to Zix Support.
Posted Nov 28, 2023 - 19:40 CST
Update - Our backend engineers continue in the recovery efforts of systems affected by the data center.
Threat Service - ZixProtect advanced feature (message retraction) is still being worked.
Additional status page updates will be provided in 24 hours or when new information becomes
Posted Nov 28, 2023 - 09:33 CST
Update - Our backend engineers continue in the recovery efforts of systems affected by the data center.
Threat Service - ZixProtect advanced features are still being recovered.
Additional status page updates will be provided in 24 hours or when new information becomes available.
Posted Nov 27, 2023 - 07:24 CST
Update - Service Incident Affecting Zix Services:
The teams continue to work to resolve outstanding issues.
Threat Protection service - ZixProtect advanced features are still being worked through.
The data center has made progress on restoring affected hardware and is in recovery efforts.
We apologize for any inconvenience this is currently causing.
Additional status page updates will be provided when any new information is available.
Posted Nov 25, 2023 - 09:09 CST
Update - Service incident affecting Zix Services.
While the team continues to work to resolve the outstanding issues, progress has been made:
ZixArchive service has been restored.
ZixProtect Link Protection restored.
SMB ZixDLP systems restored.
We apologize for any inconvenience this may currently be causing.
Additional updates to the status page will be provided when any new information is available.
Posted Nov 21, 2023 - 18:04 CST
Update - Service incident affecting Zix Services.
Team continues to work to resolve:
Legacy email threat protection in ZixCentral (console.zixcentral.com) and the intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction, and Attachment Quarantine features.
Archiving service in ZixCentral (console.zixcentral.com) experiencing a disruption in the archiving of email communication and intermittent issues accessing their archived emails.
We apologize for any inconvenience this may currently be causing.
Additional updates to the status page will be provided when any new information is available.
Posted Nov 20, 2023 - 16:49 CST
Update - Customer Advisory
Service incident affecting Zix Services.
On November 16, 2023, the major incident process was initiated to resolve issues affecting several of the Zix Services.
Impact
· Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
· Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features. · Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may experience a disruption in the archiving of their email communication and may have intermittent issues accessing their archive.
Resolution
· Zix Email Encryption services have been fully restored.
· Zix Email Threat Protection services have been fully restored for customers using AppRiver/Secure Cloud email threat protection. Customers using our legacy email threat protection in ZixCentral (console.zixcentral.com) may still be experiencing intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction, and Attachment Quarantine features.
· Zix Archiving services in AppRiver/Secure Cloud are fully functional. Customers using the archiving service in ZixCentral (console.zixcentral.com) are experiencing a disruption in the archiving of their email communication and may have intermittent issues accessing their archived emails.
More information
We are actively working to resolve all outstanding issues.
For more information, contact Zix Support.
Copyright © 2023 Open Text. All rights reserved. OpenText is a trademark or registered trademark of Open Text. The list of trademarks is not exhaustive of other trademarks. Registered trademarks, product names, company names, brands and service names mentioned herein are property of Open Text or other respective owners.
Nov 19, 2023 - 21:08 CST
Update - Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services - Starting to implement the workaround to restore mail flow.
Legacy ETP (ZixProtect) - not working for outbound scans
Legacy ZixArchive - still working on impact
ZixProtect ECS - still working on impact;
LinkProtection (ZixProtect) - still being worked to restore
- Inbound email after 11/15 will not have links wrapped
- Inbound email prior to 11/15 will have links wrapped but will come up with an error on the UI but still can proceed to the link (use at your discretion)
Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
Updates will be posted with new information.
Nov 19, 2023 - 14:39 CST
Update - Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services - Working on a workaround to restore mail flow
Legacy ETP (ZixProtect) - not working for outbound scans
Legacy ZixArchive - still working on impact
ZixProtect ECS - still working on impact;
LinkProtection (ZixProtect) - still being worked to restore
- Inbound email after 11/15 will not have links wrapped
- Inbound email prior to 11/15 will have links wrapped but will come up with an error on the UI but still can proceed to the link (use at your discretion)
Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
Nov 19, 2023 - 14:16 CST
Update - Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services - Working on a workaround to restore mail flow
Legacy ETP (ZixProtect) - not working for outbound scans
Legacy ZixArchive - still working on impact
ZixProtect ECS - still working on impact;
LinkProtection (ZixProtect) - still being worked to restore
- Inbound email after 11/15 will not have links wrapped
- Inbound email prior to 11/15 will have links wrapped but will come up with an error on the UI but still can proceed to the link (use at your discretion)
Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
Nov 19, 2023 - 13:49 CST
Update - Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services
Legacy ETP (ZixProtect)
Legacy ZixArchive
ZixProtect ECS
LinkProtection (ZixProtect)
Any SMB/Hosted Enterprise encryption service using outbound ETP scanning--(workaround is to point smart host directly to encryption servers)
We will update you as soon as information becomes available.
Nov 19, 2023 - 10:17 CST
Update - Hello All,
OpenText Engineers are still working to restore services. A majority of services have already been restored; however, the below services are still impacted:
Secure Cloud encryption services
Any SMB/Hosted Enterprise encryption service using outbound ETP scanning
Legacy ETP (ZixProtect)
Legacy ZixArchive
ZixProtect ECS
LinkProtection (ZixProtect)
We will update you as soon as information becomes available.
Nov 19, 2023 - 08:58 CST
Update - RSYNC of data continues to process as our engineers are working to restore the issues surrounding mail flow.
Our QA/DEV teams continue to test other services that have been impacted and are working to restore functionality.
The ZixPortal service (secure message centers) seems to have stabilized and appears to be operational at this point.
An update to the status page will be made when any new information is available.
Nov 19, 2023 - 07:51 CST
Update - RSYNCs of data continue to process.
Our QA/DEV teams continue to test other services that have been impacted by this disruption of services and are working to restore functionality.
The ZixPortal service seems to have stabilized and appears to be operational at this point.
Nov 18, 2023 - 19:11 CST
Update - RSYNCs of data to a new pool continue to process.
The ZixPortal service appears to be in a stable position and after testing all typical functions of users passed inspection.
Our QA/DEV teams continue to test other services that have been impacted by this disruption of services and are working to restore functionality to those services.
Nov 18, 2023 - 17:43 CST
Update - Rsyncs are still progressing. The engineering teams are still attempting to bring up certain VMs to verify data integrity. Other teams continue to work with Oracle and VMware resources.
The next update will be at 5:00 PM Central or when we receive a major update from our engineers.
Thank you
Nov 18, 2023 - 12:58 CST
Update - Our Engineers are still working with the vendor engineers on the rsyncs to move data off the bad drive pool.
Next update will be at 1:00 PM Central.
Nov 18, 2023 - 11:13 CST
Update - Rsyncs are still progressing and taking longer than expected, the engineering team is attempting to bring up certain VMs to verify data integrity. The team continues to work with Oracle and VMware resources.
The next update will be at 11:00AM Central
Nov 18, 2023 - 09:11 CST
Update - Rsync process is still underway and progressing. Some engineer teams remain engaged on a call with Oracle as extended teams perform integrity checks and other required VM functions.
The next update will be at 9:00AM Central; sooner if there is a major update to the progress.
Nov 18, 2023 - 07:03 CST
Update - Our engineers have reported that all remaining RSYNC processes are underway. These are expected to run for approximately the next 4 hours.
The next update will be at 7:30 AM Central.
Nov 18, 2023 - 02:05 CST
Update - Our engineers continue to run RSYNCs of data to the new drive pool. There is no ETA on when the RSYNCs are expected to be completed.
The next update will be at 2:00 AM Central.
Nov 18, 2023 - 00:04 CST
Update - We do apologize for the delays in the release of information as we progress through this issue. Our engineers continue to validate data moved in preparation for the new drive install on the SAN.
We will continue to update the status as we get information from the engineering team.
Nov 17, 2023 - 22:05 CST
Update - Hello,
Our engineers continue to work on the issue with vendor engineers. They currently are in a crucial part of the data copy to a new pool from the old pool of drives. Once this has been completed, the engineers will swap out damaged drives and finally start the copy back once the drives are tested.
There is no ETA for this action.
Nov 17, 2023 - 14:28 CST
Update - Hello Team,
We are aware of the issue of sending emails outbound from O365 to the encryption services. This issue is the result of the SAN service issue we have been experiencing. To restore service admins can access their O365 Exchange Admin Center to temporarily disable the outbound transport rule and send directly from O365. The rule can be enabled when the SAN service issue has been resolved.
This is at the sender's discretion and must be aware that emails with sensitive data will receive no encryption.
Our engineers are still working with the vendor engineers. We have had some success and now are able to move the data to a new pool to replace the hard drives.
Thank you
Nov 17, 2023 - 10:13 CST
Update - Our engineers are still working with our vendor engineers on the issue.
We will update the status page in 3 hours or as soon as information becomes available.
Nov 17, 2023 - 07:32 CST
Update - Hello Team,
The initial work to remediate the issue was not successful. With the assistance of Oracle engineers, our teams have started the process of making adjustments to work through finding stability for the affected drives within the SAN.
This process will take a few hours to complete while the Oracle engineers look further into the system's logs.
The next update will be at 7 am Central.
Nov 17, 2023 - 00:17 CST
Update - Our teams have started the process to remediate the affected disks of the SAN.
We expect these actions to take a few hours to complete.
We will update the status as we receive new information.
Nov 16, 2023 - 17:29 CST
Update - Our Engineers and Oracle appear to have isolated the issue to a pool of drives within a SAN that require a reseating to reset them.
Our engineers are in a meeting with Opentext leadership to decide the best course of action to minimize the impact on the customer even further.
Nov 16, 2023 - 16:27 CST
Update - Mail flow for our encryption services is being redirected to our second data center to assist with the slowdown of mail in our primary.
ZixPortal access is still being affected and we believe is related to the SAN issue that is currently ongoing.
Our engineers are working with the 3rd party vendor of the SAN to rectify the situation.
Information will be updated as it becomes available.
Thank you
Nov 16, 2023 - 13:50 CST
Update - Our engineers are working with external vendor engineers to try and isolate the issue that is causing the disruption to the listed services.
We will update this page when new information has been released about the disruption of service.
Thank you
Nov 16, 2023 - 12:29 CST
Update - No change to the statuses. Our engineers continue to work on the issue.
Nov 16, 2023 - 10:54 CST
Update - We are continuing to work on a fix for this issue.
Nov 16, 2023 - 08:40 CST
Update - Adding Threat Protection services for OUTBOUND scanning.
We are getting reports from customers that we are seeing "Relay Access Denied" messages from O365. This is being investigated.
Nov 16, 2023 - 08:29 CST
Update - Adding Secure Cloud Encryption Services (AEE/ZEE) to the affected systems. The configuring of new domains added to the domains is impacted.
Nov 16, 2023 - 07:50 CST
Update - We are continuing to work on a fix for this issue.
Nov 16, 2023 - 07:18 CST
Identified - Hello,
Our engineers have identified an issue with ZixPortal and are currently working to remedy the situation. We do apologize for this inconvenience.
Zix|Appriver Support
Nov 16, 2023 - 07:03 CST
This incident affects: Threat Protection (Email Threat Protection, Zix Protect/Spam Stops Here) and Encryption Services (Hosted ZixGateway, Hosted SMB, Secure Cloud Email Encryption (AEE/ZEE), ZixPortal).
https://status.appriver.com/incidents/t4rx5ff8jyc1
Carlos Rios
Comments