Experiencing Data Access Issue in Azure portal for Service Map - 07/19 - Resolved
Published Jul 19 2021 01:32 PM 1,871 Views
Final Update: Tuesday, 20 July 2021 03:53 UTC

We've confirmed that all systems are back to normal with no customer impact as of 7/20, 00:24 UTC. Our logs show the incident started on 7/13, 7:58 UTC and ended 7/19 21:21 UTC in EJP and began 7/17 01:53 UTC and ended 7/20 00:24 in EUS and that during the time it took to resolve the issue customers experienced both data latency and data loss in Japan East and East US for Service Map and VM Insights. This could have caused errors when viewing network dependency maps or caused data latency and loss on VMComputer, VMProcess, VMBoundPort and VMConnection tables in their Log Analytics Workspaces. This latency and loss could have also caused missed or misfired alerts if you are using the tables listed.

  • Root Cause: We have identified a configuration mismatch in a recent deployment affecting Service Map’s ability to communicate with it’s underlying storage accounts. The deployment, in part, updated our method of authenticating to the storage accounts on the backend, but a subset of storage accounts were not included in the update.

  • Incident Timeline: 7/13, 7:58 -  7/19, 21:21 UTC in EJP and 7/17 01:53 UTC - 7/20 00:24 UTC in EUS

We understand that customers rely on Service Map and VM Insights as a critical service and apologize for any impact this incident caused.


-Ian

Update: Tuesday, 20 July 2021 01:56 UTC

Service Map in East Japan and East US has had extended data latency. This is due to a new deployment with updated endpoints that were not reflected in a downstream portion of the service.  To address this issue we are updating the endpoints which should bring in all the latent data. This could have caused errors when viewing network dependency map as well as data latency in your VMComputer, VMProcess, VMBoundPort and VMConnection tables in your Log Analytics Workspaces. This latency could also have caused missed or misfired alerts if you are using the tables listed.
  • Work Around: none
  • Next Update: Before 07/20 05:00 UTC
-Ian

Update: Monday, 19 July 2021 22:55 UTC

Service Map in East Japan and East US has had extended data latency. This is due to a new deployment with updated endpoints that were not reflected in a downstream portion of the service.  To address this issue we are updating the endpoints which should bring in all the latent data. This could have caused errors when viewing network dependency map as well as data latency in your VMComputer, VMProcess, VMBoundPort and VMConnection tables in your Log Analytics Workspaces. This latency could also have caused missed or misfired alerts if you are using the tables listed.
  • Work Around: none
  • Next Update: Before 07/20 01:00 UTC
-Ian

Update: Monday, 19 July 2021 20:25 UTC

Service Map in East Japan has had extended data latency. This is due to a new deployment with updated endpoints that were not reflected in a downstream portion of the service.  To address this issue we are updating the endpoints which should bring in all the latent data. This could have caused errors when viewing network dependency map as well as data latency in your VMComputer, VMProcess, VMBoundPort and VMConnection tables in your Log Analytics Workspaces. This latency could also have caused missed or misfired alerts if you are using the tables listed.
  • Work Around: none
  • Next Update: Before 07/19 22:30 UTC
-Ian

Version history
Last update:
‎Jul 19 2021 08:55 PM
Updated by: