Experiencing Data Access Issue in Azure portal for Log Analytics - 08/01 - Resolved

[Azure-Monitor-Team]

Final Update: Tuesday, 03 August 2021 07:01 UTC

We've confirmed that all systems are back to normal with no customer impact as of 08/03, 03:00 UTC. Our logs show the incident started on 08/01, 15:30 UTC and that during the  1 Day 11 Hours & 30 minutes that it took to resolve the issue. Some customers may have experienced delay in data ingestion, duplicated data and delayed or missed alerts in West US2 region

• Root Cause: The failure was due to an issue with one of the dependent services.
• Incident Timeline: 1 Day 11 Hours & 30 minutes - 08/01, 15:30 UTC through 08/03, 03:00 UTC

We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Vyom

---

Update: Tuesday, 03 August 2021 05:08 UTC

Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take few more hours before all missing alerts and data to be made available.

• Work Around: None
• Next Update: Before 08/03 11:30 UTC

-Vyom

---

Update: Monday, 02 August 2021 22:23 UTC

***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.

• Work Around: None

• Next Update: Before 08/03 04:30 UTC

-Arish B

---

Update: Monday, 02 August 2021 17:26 UTC
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.

• Work Around: None
• Next Update: Before 08/02 22:00 UTC

-v-arishb

---

Update: Monday, 02 August 2021 10:57 UTC

Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.

• Work Around: None
• Next Update: Before 08/02 17:00 UTC

-Rahul

---

Update: Monday, 02 August 2021 07:35 UTC

Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may experience missing alerts and data and we estimate it may take few more hours before all missing alerts and data to be made available.

• Work Around: None
• Next Update: Before 08/02 12:00 UTC

-Rahul

---

Update: Monday, 02 August 2021 04:12 UTC
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to miss configuration of some of WUS2 log analytics backend clusters which was impacting the data ingestion leading to missing alerts and data. To address this issue we applied missing configuration to some of WUS2 Log analytics Backend servers. The service is processing the backlog ingestion data which should take ~2 hours. Some customers may experience missing alerts and data and we estimate several hours before all missing alerts and data to be made available.

• Work Around: none
• Next Update: Before 08/02 03:30 UTC

-v-arishb

---

Update: Sunday, 01 August 2021 23:14 UTC

Root cause has been isolated to miss configuration of some of WUS2 log analytics backend clusters which was impacting the data ingestion leading to missing alerts and data. To address this issue we applied missing configuration to some of WUS2 Log analytics Backend servers. The service is processing the backlog ingestion data which should take ~2 hours. Some customers may experience missing alerts and data and we estimate several hours before all missing alerts and data to be made available.

• Work Around: none
• Next Update: Before 08/02 03:30 UTC

-v-arishb

---

Initial Update: Sunday, 01 August 2021 18:27 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers may experience delay in data ingestion into Log Analytics in West US2 .

• Work Around:
• Next Update: Before 08/01 22:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Subhash

---