There was an issue in our metric processing between 09:00 27th Nov PST and 00:30 29th Nov PST where some of the graphs were showing few data points incorrectly. There are sudden spike seen in the graphs and the same data was visible in the daily graph.
Actions taken & Current Status
The issue was due to duplicate wrong metric being written randomly into graph data between 09:00 27th Nov PST and 00:30 29th Nov PST. This happened for two of three data points in this window. RAW data if pulled through API on the same date and daily reports generated may have this issue. This does not impact any alerting or roll-up data.
We identified the root cause, which is with the data driver that gave additional wrong data point with abnormal values. We did not have the range validation, which allowed wrong data to be written into the graph data. This issue was resolved by fixing the range validation along with the data driver and the production was updated around 12:30 AM on Nov 29th. The data between 27th Nov PST and 29th Nov PST may have this aberration and we are analyzing possible ways to correct it. Please note that other than RAW data ( last 24 hours ) all other data will have no issues and the data taken for roll-up is not impacted. Monthly reports and alerting are not impacted.