Fastly will be expanding Sāo Paulo (CGH) into a Metro POP. End-users may observe connection resets as traffic is migrated onto new hardware.

As part of this expansion, customers utilizing shielding in the neighboring Sāo Paulo (CGH) will be migrated to the new Metro POP. Once this maintenance window closes, customers can either apply a new config version at a time of their choosing, or their shielding configuration will be updated on their behalf after 14 days.

When this change is applied, customers may observe additional origin traffic as new cache nodes retrieve content from origin.

Action for Fastly Customers:

1. Please verify that all origins allow the full range of Fastly IP addresses (https://docs.fastly.com/en/guides/accessing-fastlys-ip-ranges), as this list is updated periodically. Failure to verify origin access lists will result in origin connection disruption. 
2. Customers with any questions or concerns should contact their dedicated technical account team or Fastly’s Support team at (support@fastly.com).

For more information on Metro POPs: https://developer.fastly.com/learning/concepts/pop/#metro-pops

Fastly will be migrating Atlanta (ATL and FTY) shield customers to Atlanta (PDK).

As part of this migration, customers utilizing shielding in Atlanta (ATL and FTY) will be migrated to Atlanta (PDK). Once this maintenance window closes, customers can either apply a new shield at a time of their choosing, or their shielding configuration will be updated to Atlanta (PDK) on their behalf after 14 days.

Customers may observe additional origin traffic as new cache nodes retrieve content from origin.

Action for Fastly Customers:

1. Please be sure to check that your origin access lists allow the full range of Fastly IP addresses: https://docs.fastly.com/en/guides/accessing-fastlys-ip-ranges. Failure to verify origin access lists will result in origin connection disruption.

2. Customers with any questions or concerns should contact their dedicated technical account team or Fastly's Support team at (support@fastly.com).

Fastly will be adding capacity at our Houston (IAH) POP. End-users may observe connection resets as traffic is migrated onto new hardware starting on 03 October 2023 at 05:00 UTC.

Our estimated duration is 4h. 

When this change is applied, customers may observe additional origin traffic as new cache nodes retrieve content from origin. Please verify that your origin access lists allow the full range of Fastly IP addresses (https://docs.fastly.com/en/guides/accessing-fastlys-ip-ranges). Customers with any questions or concerns should contact Fastly’s Support team (support@fastly.com).

We're investigating possible performance impact affecting the Buenos Aires (EZE) data center.

All other locations and services are unaffected

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating elevated errors in Configuration Deployment services. Customers with newly provisioned services and some configuration updates may not deploy to the CDN.

Network availability and all other services are unaffected.

This issue has been identified and a fix is being implemented. 

Fastly has begun to see recovery in response to our fix. 

Customers will not have to take any action, their deployed configuration updates and newly provisioned services will become available as our fix reaches the impacted hosts.

Network availability and all other services were unaffected by this incident. We will continue to monitor until our fix is fully deployed and all services have been effectively updated.

This incident has been resolved.

We are aware of a possible issue and are working to determine the scope and impact.

Update: Our investigations confirmed that there were no external impacts to our customer services or end user experiences. We've changed this status from "degraded performance" to "informational".

Our Engineering team had identified an internal issue that had risk for customer impact. Our Acute Incident Response teams have deployed a fix and are monitoring the results. 

Our network availability and all other locations were unaffected by this incident. Customers and their end users should not have seen impact as a result of this incident. 

This incident has been resolved.

We are currently investigating performance issues affecting the availability of manage.fastly.com and our API. CDN delivery, stats aggregation, and all other data plane services are unaffected.

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're investigating elevated errors in Columbus (CMH).

All other locations and services are unaffected

Fastly Engineering has identified a third party service provider issue that caused elevated latency and increased errors for customers within the Columbus (CMH) Point of Presence (POP). Engineering has rerouted customer traffic around the impacted region and services be gradually restoring across our network. 

We will continue to monitor the actions of the third party service provider and will resolve our status post once we've seen full recovery from the provider.

Update: An earlier status post: https://www.fastlystatus.com/incident/376077 has been closed. As the elevated errors observed when access our Control Plane (manage.fastly.com) and API services has been correlated to the third party service provider issue identified within this status post.

This incident has resolved.

During this incident, customers would have experienced increased latency and elevated 5xx errors in our Columbus (CMH) POP, as well as, elevated errors when attempting to access configuration management and API services through our Control Plane from 18:40 to 21:30 UTC.

Affected customers may have experienced impact to varying degrees and to a shorter duration than as set forth above.

Fastly Engineering has confirmed that our network availability and Control Plane services have been fully restored.

We are currently investigating into elevated errors when attempting to access our Control Plane (manage.fastly.com) Configuration Management and API services.

We have observed a reduction in errors since the start of this incident, but are continuing to investigate the cause of the errors observed in order to identify a solution that would prevent a repeat of this incident for our partner and customer services.

The customer impact associated with this status post is a direct result of the third party service provider issue that is covered in this status post link: https://www.fastlystatus.com/incident/376079 

All remaining status post updates will be consolidated into the link provided above. We are effectively closing this post in order to reduce dual notifications for our subscribers.

Fastly Engineers detected a performance impacting event affecting the Seattle (BFI) data center.

All other data centers and services were unaffected. The issue has been resolved and we are monitoring performance closely.