Incident began at 2023-11-30 16:30 and ended at 2023-11-30 21:15 (times are in Coordinated Universal Time (UTC)).
| Date | Time | Description |
|---|
| Dec 1, 2023 | 1:43 AM UTC |
Mini Incident ReportWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.(All Times US/Pacific) Incident Start: 30 November 2023 08:30 Incident End: 30 November 2023 13:15 Duration: 4 hours, 45 minutes Affected Services and Features: Gmail Regions/Zones: Global Description: Some Gmail users experienced a delay in sending emails globally for a duration of 4 hours and 45 minutes. Any emails that were delayed after entering our system were auto retried and should have been delivered within a couple of hours after mitigation. From preliminary analysis, the root cause of the issue appears to be an overload in Google Message Router (GMR). Google will complete a full incident report in the following days that will provide a detailed root cause. Customer Impact: Customers experienced delays while sending the emails. |
| Nov 30, 2023 | 10:03 PM UTC | The issue with email delays was mitigated for the majority of affected users at 11:45 US/Pacific. A few users may have observed delays until the full mitigation was done at 13:15 US/Pacific. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue. |
| Nov 30, 2023 | 8:46 PM UTC | Google engineers have taken required mitigation measures. New emails should generally not be seeing any issues. However older emails will be auto retried and should be delivered within the next couple of hours. Our engineers continue to monitor the issue until full recovery. We will provide more information by Thursday, 2023-11-30 13:45 US/Pacific. Customer Symptoms:Customers impacted by this issue may see delay while sending the emails. Workaround:Customers can retry sending the email. |
| Nov 30, 2023 | 8:04 PM UTC | Google engineers have taken required mitigation measures. New emails should generally not be seeing any issues. However older emails will be auto retried and should be delivered within the next couple of hours. Our engineers continue to monitor the issue until full recovery. We will provide more information by Thursday, 2023-11-30 12:45 US/Pacific. Customer Symptoms:Customers impacted by this issue may see delay for a few mins while sending the emails. Workaround:Customers can retry sending the email. |
| Nov 30, 2023 | 7:13 PM UTC | Mitigation work is currently underway by our engineering team. Our engineers have taken a few mitigation steps at the highest priority and are actively validating the results. We will provide more information by Thursday, 2023-11-30 12:00 US/Pacific. Customer Symptoms:Customers impacted by this issue may see delay for a few mins while sending the emails. Workaround:Customers can retry sending the email. |
| Nov 30, 2023 | 6:22 PM UTC | Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-11-30 11:15 US/Pacific. Customer Symptoms:Customers impacted by this issue may see delay for a few mins while sending the emails. Workaround:Customers can retry sending the email. |
| Nov 30, 2023 | 5:41 PM UTC | Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-11-30 10:30 US/Pacific. Customer Symptoms:Customers impacted by this issue may see delay in sending the emails. Workaround:None at this time. |
| Nov 30, 2023 | 5:31 PM UTC | We are experiencing an issue with Gmail beginning on Thursday, 2023-11-30 08:30 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 10:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Customer Symptoms :Customers impacted by this issue may see delay in sending the emails. Workaround:None at this time. |
Carlos Rios
Comments