Edgio

Troubleshooting

Troubleshoot:
Learn more about our troubleshooting tools.

General Troubleshooting

If you encounter unexpected behavior or an issue, you should:
  1. Verify that all of your traffic is being served through the most recently deployed environment.
  2. Verify that the only desired set of rules are being applied to the request experiencing unexpected behavior.
  3. Test your site using a permalink. This ensures that the unexpected behavior is not due to cached content.
  4. Review Edge Insights data.

Environment Version

Delays in configuration propagation may cause Edgio to serve some requests using an older configuration. Upon detecting unexpected behavior, it is important to verify that all requests are using the latest environment version.
  1. Find the environment version through which a request was served by checking the Environment column from within the Edgio Developer Tools Chrome extension.
  2. Find the latest environment version from within the Edgio Console by navigating to the desired environment, clicking Deployments, and then checking the Environment column.
For example, the Edgio Developer Tools Chrome extension’s Environment column should report v3 for requests to a website powered by the following production environment:
Sample deployments

Applied Rules

Verify that the desired set of rules are being applied to the request by performing the following steps:
  1. Find out the set of rules that were applied to the request by checking the Matched Rules column within the Edgio Developer Tools Chrome extension.
    For example, the following illustration indicates that rules 0, 2, 3, and 13 were applied to a request:
    Matched rules
  2. Look up those rules from within the Rules page of the Edgio Console.
    Rules page showing rule numbers
Key information:
  • Click on the Show Rule Numbers link to display rule numbers next to each rule.
  • Rules use zero-based numbering.
  • Use the above procedure even if you are using CDN-as-code.
    Edgio automatically adds system-defined rules when you deploy your CDN-as-code configuration. As a result, counting rules within your routes.[js|ts] will be inaccurate.

Test Without Cached Content

Use a permalink to ensure that Edgio does not serve cached content when testing your website. A permalink forces Edgio to proxy your request to either the Edgio cloud or your origin. Although this may degrade performance, it is useful when verifying functionality.
A permalink is assigned to each deployment. View a deployment’s permalink by navigating to the Deployments page for the desired environment and then clicking on the desired deployment version.

Review Edge Insights Data

Edge Insights provides near real-time data for an environment’s traffic over the last 6 hours through the Access Logs data source.
  • Review the timeline graph to identify issues and analyze performance. Sample use cases are provided below.
    • Identify sudden spikes or drops in traffic.
      Once you have identified a questionable traffic spike, determine whether it is legitimate traffic by reviewing key metrics, such as the country of origin, URL path, and query strings.
    • Identify sudden spikes in 4xx and 5xx traffic.
      1. From the Top Results section, verify that HTTP Status Code has been selected for one of the pie charts.
      2. From the Timelines section, click the HTTP Status Code source.
        Timelines - HTTP Status Code source
      3. Once you have identified a spike, analyze the corresponding log data to gain insight into a specific status code.
    • Identify caching trends.
      1. From the Top Results section, verify that Cache Status has been selected for one of the pie charts.
      2. From the Timelines section, click the Cache Status source.
  • Review log data to troubleshoot an issue.
    For example, if you are able to reproduce an issue on your local machine and require more information than is available through the Edgio Developer Tools Chrome extension, then you may view log data for these requests from within Edge Insights. One method for analyzing these requests is to:
    1. Configure a catch-all rule to set a response header to the request’s ID through the %{http_x_ec_uuid} feature variable.
      Add Response Header feature
    2. Find out the ID corresponding to a request issued from your local machine.
      1. From the desired browser, open developer tools.
      2. From the browser, issue a request.
      3. From within developer tools, inspect the request to find out the request’s ID.
        Chrome Developer Tools - Headers
    3. Filter Edge Insights by that ID (i.e., Event ID = <EVENT ID>).
      Filtering by Event ID
    4. From the Logs section, click on the log entry to view the log fields associated with the request.
      Log entry

Caching

Use the Edgio Developer Tools Chrome extension to troubleshoot caching behavior.

Overall Cache Performance

View overall cache performance by checking the Edge Hits statistic at the bottom of the Edgio Developer Tools Chrome extension.
Edge Hits

Cache Responses

Check whether a request was served from cache through the Cache Status column. Look for one of the following values:
  • Hit (COMPONENT): Indicates that the request was served from cache from either our network (Edge) or a service worker’s local cache (Service Worker).
  • Miss (COMPONENT): Indicates that Edgio could not find a cached version of the requested content with a valid time-to-live (TTL) from either our network (Edge) or a service worker’s local cache (Service Worker).
  • No-Cache (Edge): Indicates that a cache content freshness check was not performed. This check is skipped when an HTTP request method is used that bypasses cache (e.g., PUT, DELETE, etc).
  • Disabled (Edge): Indicates that caching was disabled through the Bypass Cache feature.
    If this behavior is undesired, review the rules applied to this request.
  • Blank: A blank value indicates that the request was either not served through Edgio or it was served through a different property or environment on which the Debug Header feature has not been enabled.

Cache Miss

Review the following items to find out why a request resulted in a cache miss.
  • By default, content is only cached when the response from the origin includes a caching policy. Additionally, some content types may require 2 requests before they are eligible to be cached.
    View our default caching policy.
  • If you have defined a custom cache policy through a rule, then you should review the rules applied to this request.
  • If your origin defines a custom cache policy through headers, then you should click on the desired request and then check the Response Headers section for cache directives
  • Edgio associates each request with a cache key. Edgio then uses this cache key to identify whether content has been cached for this request. By default, this cache key includes the query string. This configuration is ideal for a website that uses the query string to determine the content that will be served. For other sites, this can lead to cache fragmentation and a higher rate of cache misses.
    Customize the cache key to exclude query string parameters:
    • Rules: Create or modify a rule that includes the Cache Key feature. Configure this feature’s Query Parameters option to either exclude all query string parameters or to only include specific parameters.
      Cache Key feature set to exclude all query string parameters
    • CDN-as-Code:
      JavaScript./routes.js
      1export default new Router().always({
      2 caching: { cache_key: { exclude_all_query_params: true } },
      3});
    Key information:
    • If you must add query string parameters to the cache key, we recommend that you restrict it to the parameters that are critical to your business needs. This recommendation ensures optimal performance by allowing our CDN to serve more requests from cache.
    • If you are unsure as to whether you have already defined a custom cache key, then you should review your rules for features that modify the cache key.

Performance

Use the Edgio Developer Tools Chrome extension to troubleshoot performance.

Overall Performance

View overall performance statistics at the bottom of the Edgio Developer Tools Chrome extension.
  • TTFB: Time to First Byte.
  • LCP: Large Contentful Paint
  • INP: Interaction to Next Paint
  • CLS: Cumulative Layout Shift
  • FID: First Input Delay
  • Edge Hits: Indicates the number of requests served from cache and the total number of requests issued for the current page that were captured by Edgio Developer Tools Chrome extension. As noted above, it will only capture requests issued while the Edgio tab is active.
  • Prefetches: Indicates the number of successful prefetch requests and the total number of prefetch requests issued for the current page that were captured by Edgio Developer Tools Chrome extension. As noted above, it will only capture requests issued while the Edgio tab is active.

Request-Specific Performance

View request-specific performance statistics by checking:
  • The TTFB column for a high value. This column measures time to first byte. This metric is indicative of responsiveness.
  • The Total Time column for a high value. This column measures the total amount of time it took to serve a response to the client.

Predictive Prefetching

Assess overall prefetching performance by checking the Prefetches statistic at the bottom of the Edgio Developer Tools Chrome extension.
Prefetches
By default, you may only prefetch content that is cached on the POP closest to the user and that still has a valid TTL. By default, Edgio responds with a 412 Precondition Failed status code for prefetch requests that result in a cache miss. This default configuration ensures that your origin servers do not experience additional load due to predictive prefetching.
Identify prefetch requests through the following query string parameter: edgio_dt_pf=1&edgio_prefetch=1.
Example: https://cdn.example.com/css/styles.css?edgio_dt_pf=1&edgio_prefetch=1

Status Codes

Gain insight into why Edgio returned a specific status code by filtering Edge Insights by the desired status code and then reviewing log data.
  1. Load the desired environment-specific Edge Insights page.
  2. Verify that the Data Source option is set to Access Logs.
  3. Scroll down to the Top Results section.
  4. Verify that HTTP Status Code has been selected for one of the pie charts.
  5. From within the pie chart, click on the desired status code. The entire dashboard will be filtered by that status code.
  6. Scroll down to the Logs section.
  7. Inspect each request to gain insight into why this status code is occurring.
    Filter for a specific field by typing the desired name in the upper-right hand search bar.
Troubleshoot the following common status codes:
Learn about other status codes.

404 Not Found Status Code

Troubleshoot this status code by performing the following steps:

412 Precondition Failed Status Code

By default, Edgio will only serve prefetch requests from the edge cache. If a request cannot be served from the cache, a 412 Precondition Failed status code is returned. This protects your origin servers from additional traffic associated with prefetching.
Perform the following steps to reduce excessive 412 Precondition Failed responses:
  1. Ensure that the URLs being prefetched are similar to those fetched during page navigation.
    Prefetch URLs contain the following query string parameters: edgio_dt_pf=1&edgio_prefetch=1. These parameters are automatically excluded from the cache key. Verify that this is the only difference between URLs that are prefetched and those that are requested through standard page navigation.
  2. Apply the Stale While Revalidate (stale_while_revalidate) feature to URLs that will be prefetched.
    • Rules Example:
      Sample rule with the Stale While Revalidate feature
    • CDN-as-Code Example:
      JavaScriptroutes.js
      1router.get('/p/:productId', {
      2 caching: {
      3 max_age: '1h',
      4 service_worker_max_age: '1h',
      5 stale_while_revalidate: '1d', // this way stale items can still be prefetched
      6 }
      7});
  3. Consider increasing the Set Max Age (max_age) feature. Short time to live (TTL) intervals generate more prefetch failures.
  4. Prefetch cache misses.
    Use this capability with caution since it may significantly increase the traffic to your origin or API servers.
    • HTML Script Tag Example:
      HTML
      1<script src="/__edgio__/prefetch/install.js" data-include-cache-misses="true" defer></script>
    • Edgio Sites Example:
      JavaScriptapp.js
      1import install from '@edgio/prefetch/window/install';
      2
      3// Call the following once when the page loads to allow prefetch requests to be served when responses
      4// aren't available in the edge cache:
      5install({includeCacheMisses: true});

502 Bad Gateway Status Code

Troubleshoot this status code by performing the following steps:
  • Identify the origin configuration that is returning a 502 Bad Gateway. Request the origin directly to verify that it is available.
    Example: If your origin configuration points to origin-1.example.com, then you could potentially verify that this origin is available by submitting the following request:
    https://origin-1.example.com/
  • Check whether your site requires SNI by reviewing your server’s configuration or log data.
    Alternatively, there are online tools (e.g., Qualys SSL Labs) that allow you to check whether your site requires SNI. Submit your origin’s hostname to start the test. Once the test is complete, check whether your server requires SNI. For example, SSL Labs returns the following message within the summary section: This site works only in browsers with SNI support.
    Your origin configuration setup varies according to whether your site requires SNI.
    • SNI: If your site requires SNI, then you should enable your origin configuration’s Use SNI option and verify that the SNI hint is set to a hostname defined within your certificate’s Subject Alternative Name (SAN) or Common Name (CN).
    If your site requires SNI and your origin configuration is misconfigured, then Edge Insights will return a proxy_hard_error field set to HARD_ERR_502_SSL_CONNECT_ERROR. A quick way of checking for this condition is to filter Edge Insights by the 502 Bad Gateway status code and then viewing a request from within the Logs section.
    • No SNI: If your site does not require SNI, then you should disable your origin configuration’s Use SNI option and remove the SNI hint.
  • If the client’s Host header does not match a hostname defined within your certificate’s Subject Alternative Name (SAN) or Common Name (CN), then you will need to update the Override Host Header option.
  • Is your server using a self-signed certificate?
    • Yes: You must enable the Allow Self-Signed Certs option on the desired origin configuration.
    • No: Edgio requires a full chain certificate. Your certificate’s chain of trust must start with the server’s certificate and terminate with the root certificate.
  • If you have pinned a certificate to the desired origin configuration, then you may need to pin an additional certificate.

504 Gateway Timeout Status Code

Troubleshoot this status code in the same manner as a 502 Bad Gateway status code.

531 Project Upstream Connection Error Status Code

Common causes are:
  • The upstream host you specified in your project is incorrect.
  • The DNS entry you defined points to the wrong server.
  • Your servers are not responding.
  • You need to add the Edgio IP addresses to your allowlist. Contact your operations team and ask them to add our IP addresses to your firewall’s IP allowlist.

539 Project Timeout Status Code

Your project’s Edgio cloud code did not respond on time. This issue typically arises in step 4 or 5 of the following request flow:
  1. A requesting client sends a request to Edgio for an asset.
  2. Edgio does not find it in its cache and examines routing rules.
  3. Edgio sends requests to server-side rendering (SSR) code.
  4. The SSR code makes calls to the customer backend to get data needed for the page.
  5. The SSR assembles the page and sends it to the Edgio edge.
  6. The Edgio edge caches the page and returns it to the client.
This issue may be due to:
  • An error in your SSR code. For example, your asynchronous requests may be missing await or call to callback.
  • A backend error (server overloaded or offline).
  • An allowlist issue.
Troubleshoot this status code by:
  • Viewing the timings and status codes of the components in the stack in the x-edg-t header.
  • Analyzing server logs.
  • Performing performance profiling
  • Detecting allowlist errors.
  • Loading source maps within our Edgio cloud infrastructure. If this occurs, try again with sourcemaps disabled.

Allowlist

When you run your site on Edgio, all requests come in through four IP addresses, and servers are programmed to interpret this as a DDoS attack. At this point, the server either blocks or rate-limits the requests. In either case, timeouts occur and 539 errors are returned.
A typical pattern is that your site works fine for a few days after deploying to Edgio, then your server starts interpreting the requests as a DDoS attack.
To prevent this scenario, you must configure your server with allowlisted Edgio IP addresses.
Learn more.

Procedure

When you are testing a web page, you might encounter 539 status code errors. You might also see the errors in logs.
  1. Open your project in Edgio, then drill down to the deployment that is experiencing the 539 errors.
  2. Click the SERVER tab header at the bottom of the page, then click the Resume logs arrow or the Logging is paused link to resume logging.
If you see 539 errors, the issue could be any of the following:
  • An error in your SSR code
  • A problem with the backend server
  • An allowlist error

Good Request Example

Before continuing, it is helpful to see what a good request and response flow looks like. A request with no errors has four lines:
LineDescription
1Summary line.
2The request from Edgio to your SSR code. The line ends with a 200, indicating success.
3The request from your SSR code to your backend server. If this line ends with a <status code> in XXms, then the SSR received a response from your backend server. In this example the HTTP status code was 200, indicating success. If the line does not end with a <status code> in XXms, there was a problem with the request to your backend server (see Backend Server Error).
4The response from the SSR to the browser, and ends with the status code for the response. If this line is present, the SSR code ran to completion. If this line is missing there was a problem (see Error in SSR Code).

Error in SSR Code

If a request looks like the following, your SSR code contains an error.
LineDescription
1Summary line.
2The request from the Edgio edge to your SSR code. The line ends with a 200.
3The request from your SSR code to your backend server. The line ends with a 200.
There is no response from the SSR code to the browser as shown in line 4 in Good Request Example. Troubleshoot your code and fix the error. Common errors are that your SSR code:
  • Took too long to return a response
  • Threw an exception and never returned a response

Backend Server Error

If a request looks like the following, your backend server is either down, overloaded or has an allowlist error.
LineDescription
1Summary line.
2The request from the Edgio edge to your SSR code.
3The request from your SSR code to your backend server.
Note: If line 3:
  • Ends in a status code other than 200, then the SSR code received a non-200 code from the backend server.
  • Does not end in a status code at all, then the SSR did not receive a response from the backend and the problem can be either an allowlist error or a timeout error. See “Distinguishing an Allow List Error from a Timeout Error.” See Distinguishing an Allowlist Error from a Timeout Error.

Distinguishing an Allowlist Error from a Timeout Error

To determine if there is an allowlist error, do the following:
  1. Expand line 3 (request from your SSR code to your backend server) and select COPY AS CURL
  1. Run the curl command. (The command runs the same request that the SSR code made to the backend server, but from your local machine.)
The outcome will be either SSR code error or an allowlist error.

SSR Code Error

If the command fails or does not respond, there is an error in your code, most likely a badly formed request.
Troubleshoot your code to find and fix the error.

Allowlist Error

If the command succeeds and finishes quickly, it is probably an allowlist error. Contact your operations team and ask them to add the IP addresses in Allowlisting to your server’s IP allowlist.

Edge Functions

Analyze the performance of your edge function(s) by reviewing performance and custom metrics from within Edge Insights.
  1. Load the desired environment-specific Edge Insights page.
  2. Verify that the Data Source option is set to Access Logs.
  3. Scroll down to the Top Results section.
  4. Set one of the pie charts to the desired Edge Functions metric. These metrics start with Edge Function.
  5. From the Timelines section, click the source corresponding to the metric selected in the previous step.
  6. Optional. Filter the report to a specific edge function (i.e., Edge Function Name = <edge_function PROPERTY>).
    Filtering by edge function
  7. Analyze trends.

Troubleshooting Tools

Troubleshoot delivery and performance issues using the following tools:
ToolDescription
Edgio Developer Tools Chrome extensionThis Chrome extension describes each request associated with the current page. Use this information to gain insight into delivery issues, caching, and performance.
Edge InsightsReview detailed information about each request to your website in near real-time.
RTLD CDNReview historical information for all requests, even those that never reach your application code (e.g., cache hits, static assets, requests routed to custom backends, edge redirects, etc.).
Visual Studio CodeThis tool allows you to add breakpoints within your code to troubleshoot delivery issues.
Server LogsReview messages from your application.
curlIssue requests to your website using curl. This tool allows you to eliminate browser-specific behavior when troubleshooting issues.
Source MapsReview our source map to investigate runtime errors that occur during routing. Additionally, if you are using the Next or Nuxt framework, then you can enable a source map for your application code.

Visual Studio Code

Set up debugging within Visual Studio Code for your CDN-as-code configuration through the following steps:
  1. Open .vscode/launch.json.
  2. Click Add Configuration and select Node.js: Launch Program.
  3. Edit this configuration to look like this:
    JavaScript
    1{
    2 "name": "Debug Edgio App",
    3 "type": "node",
    4 "request": "launch",
    5 "cwd": "${workspaceFolder}",
    6 "autoAttachChildProcesses": true,
    7 "program": "${workspaceFolder}/node_modules/@edgio/cli",
    8 "args": ["run"]
    9}
    If your workspace folder is not located in your app’s root directory, then you will need to adjust the path defined within the program and cwd keys. The program config should always point to @edgio/cli. The cwd config should point to the root directory of your app.
This configuration allows you to set breakpoints in both your Edgio router as well as your application code (e.g., Next.js, Nuxt.js, and Angular).

Server Logs

By viewing the server logs in the Edgio Console, you can see all of the messages logged by your application using console.log, console.warn, etc…
By enabling Deep Request Inspection in your environment, you can also see the headers and body of every request and response served by your application through the Edgio cloud. You can also see each upstream API request made by your application.
Debug issues related to routing to your origin by temporarily moving the proxy from the edge to the Edgio cloud:
JavaScript
1.get('/p/:productId', ({ cache }) => {
2 proxy('origin', {
3 // The presence of transformRequest and transformResponse ensure that proxying is done in the Edgio cloud, not at the edge.
4 transformRequest: (req) => {
5 console.log('Request ID', req.headers['x-request-id'])
6 // Log request properties that you want to troubleshoot.
7 },
8 transformResponse: (res, req) => {
9 console.log('Response for request ID', req.headers['x-request-id'], 'status code', res.statusCode)
10 // Log response properties that you want to troubleshoot.
11 }
12 })
13 })
Once it has been deployed, you can observe the output in your server logs.
We strongly recommend to proxy traffic from the edge whenever possible, as that is more performant and avoids Edgio cloud surcharges. The above solution should only be used as a temporary measure while addressing issues.
Learn more about server logs.

curl

Removing the browser as a variable in your equation is a good way to confirm what the origin server is doing. Below are a few of the common CURL commands we leverage to verify behavior.
Use the -k option to skip TLS validation if a TLS certificate has not been provisioned for the requested domain.
View Headers Only
Bash
1curl -o/dev/null -vv https://www.yoursite.com
Bypass DNS Resolution
Connect directly to a specific IP address. Use this command to:
  • Send a request directly to the origin and bypass Edgio
  • Test the connection to Edgio before DNS cutover. The recommended method for testing this connection is to define a localhost DNS configuration.
Bash
1curl -o/dev/null -vv
2 https://www.yoursite.com --connect-to ::35.241.39.58
Specify a Cookie
Bash
1curl -o/dev/null -vv
2 -H "Cookie: cache_enabled=true" https://www.yoursite.com/main.js
Send a Specific User Agent
Bash
1curl -o/dev/null -vv
2 -H "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone10,2;FBMD/iPhone;FBSN/iOS;FBSV/13.6.1;FBSS/3;FBID/phone;FBLC/en_GB;FBOP/5];FBNV/1"

Source Maps

If you are using CDN-as-code, then Edgio automatically produces a source map for your router file. This source map contains a stacktrace that references the original source file for each runtime error that occurs during routing. If your application build produces source maps for the server bundle, these will also be used when reporting errors.
Key information:
  • By default, application-level source maps are not enabled, since they may cause the Edgio cloud bundle to be larger than the 50MB limit.
  • Source maps loaded within our Edgio cloud infrastructure may result in 539 Project Timeout errors due to performance issues. If this occurs, try again with sourcemaps disabled.
Edgio provides a convenient way to enable source maps when using Next and Nuxt:
  • Next.js: Set edgioSourceMaps: true in your next.config.js:
    JavaScript./next.config.js
    1const { withEdgio, withServiceWorker } = require('@edgio/next/config')
    2
    3module.exports = withEdgio(
    4 withServiceWorker({
    5 // Output sourcemaps so that stacktraces have original source filenames and line numbers when tailing
    6 // the logs in the Edgio Console.
    7 edgioSourceMaps: true,
    8 }),
    9)
  • Nuxt.js: Set edgioSourceMaps: true in the config for @edgio/nuxt/module in buildModules in nuxt.config.js:
    JavaScript./nuxt.config.js
    1module.exports = {
    2 // ...
    3 buildModules: [['@edgio/nuxt/module', { edgioSourceMaps: true }]],
    4 // ...
    5}