Fastly Engineering has received reports of an issue impacting the CORP view of NGWAF missing metric data beginning at 00:00 UTC 21st of July 2023.

Please note: A previous slack broadcast message was sent to our customers as our status page provider 
restored www.fastlystatus.com availability, as communicated on:
-  https://www.fastlystatus.com/incident/375966.

Fastly Engineering has identified the issue and a fix is currently being implemented for the CORP view of the NGWAF. Customers should begin to see rollup data for workspaces first and then corp level rollups. 

All other services are unaffected by this event.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

On Thursday, the 20th of July 2023, at 05:00 to 07:00 UTC Fastly Engineering will be upgrading the processing capacity of our data processing pipeline’s storage backend.

During this time, customers may experience delayed or incomplete data appearing in their Edge WAF dashboards and a delay in the data availability for WAF rule updates. Following this maintenance, any missing data will gradually be restored, over an estimated period of 6 hours, as our servers process the backlog. 

There will be no data loss as a result of this scheduled maintenance and there are no customer actions required.

Next-Gen WAF services, network availability and all other services will not be affected.

The scheduled maintenance has been completed.

Incident identified on the 14th of July 2023 at 06:13 UTC.

Fastly Engineering has identified a performance impacting event to our NGWAF@Edge services.

Customers may experience delays detecting clients exceeding the threshold set in rate limit rules/site alerts and loading dashboard data for NGWAF services within our console.

Existing NGWAF@Edge rules and all other platform services are unaffected.

A fix has been implemented and we are monitoring the results.

Fastly Engineering has identified an additional performance impacting event to our NGWAF@Edge services.

Customers may experience delays detecting clients exceeding the threshold set in rate limit rules/site alerts and loading dashboard data for NGWAF services within our console.

Existing NGWAF@Edge rules and all other platform services are unaffected.

The issue has been identified and we are preparing an additional fix. 

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Fastly Engineering has identified a performance impacting event to our NGWAF@Edge services. 

Existing NGWAF@Edge rules and all other platform services are unaffected. 

We have confirmed that the impact to NGWAF@Edge services is restoring gradually, and the source of this incident has been resolved.

We will continue to monitor this incident to ensure NGWAF@Edge services are fully restored.

Fastly Engineering detected an event resulting in missing data affecting Next-Gen WAF during the following time frames on June 20, 2023:

• 20:29 - 20:33 UTC

• 20:55 - 20:58 UTC

• 21:21 - 21:25 UTC

• 22:20 - 22:24 UTC

All other services were unaffected.

The issue has been resolved and we are monitoring performance closely.

Fastly Engineers detected a performance impacting event affecting the Edge WAF dashboard. Customers may experience data delays when using the dashboard.

All other data centers and services are unaffected. 

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results. Customers should be able to start seeing their data in the dashboard load as the outstanding backlog is processed. 

This incident has been resolved. Customers should see their data load over the next six hours. 

The scheduled maintenance has been completed.

On Monday, the 13th of March 2023, Fastly engineers identified an issue with a recent update to our NGWAF@Edge IP inspection service. From March 10 2023 at 21:48 to March 13 2023 at 04:30 UTC IP list matches were returning FALSE which would have impacted Rules using the IP List to not work.

Fastly engineers identified this issue and deployed a fix on March 13 2023 at 04:30 UTC.

This incident has been resolved.

Fastly engineers have detected performance issues with VCL configuration propagation from 16:20 - 16:26 UTC and 17:44 - 17:55 UTC. During these time frames, customers may have experienced an increase in 5xx errors for API requests and slow VCL propagation. All other services were unaffected. 

This incident has been resolved.

We’re currently investigating performance issues with Compute@Edge configuration propagation. All other services are unaffected.

This issue has been identified and a fix is being implemented. 

Our investigations have identified common network provider issues related to the performance impact for our Compute@Edge services and NGWAF@Edge updates and decision making. Existing rules are operational, but new data is parsed at a slower pace. Fastly Engineering is in the process of performing standard traffic engineering to restore Compute@Edge and NGWAF@Edge services to pre-incident levels. 

A fix has been implemented and we are monitoring the results.

Fastly has performed traffic engineering to route around the global network provider currently experiencing impact. Our Compute@Edge and NGWAF@Edge services are performing at pre-incident levels. 

This incident is resolved.