Troubleshooting

• General issues
• Caching
• Performance
• Image Optimization
• Status codes

General Troubleshooting #

1. Verify that all of your traffic is being served through the most recently deployed environment.
2. Verify that the only desired set of rules are being applied to the request experiencing unexpected behavior.
3. Test your site using a permalink. This ensures that the unexpected behavior is not due to cached content.
4. Review Edge Insights data.

Environment Version #

1. Find the environment version through which a request was served by checking the Environment column from within the Edgio Developer Tools Chrome extension.
2. Find the latest environment version from within the Edgio Console by navigating to the desired environment, clicking Deployments, and then checking the Environment column.

Applied Rules #

1. Find out the set of rules that were applied to the request by checking the Matched Rules column within the Edgio Developer Tools Chrome extension.
   For example, the following illustration indicates that rules 0, 2, 3, and 13 were applied to a request:

   

2. Look up those rules from within the Rules page of the Edgio Console.

   

• Click on the Show Rule Numbers link to display rule numbers next to each rule.
• Rules use zero-based numbering.
• Use the above procedure even if you are using CDN-as-code.
  Edgio automatically adds system-defined rules when you deploy your CDN-as-code configuration. As a result, counting rules within your routes.[js|ts] will be inaccurate.

Test Without Cached Content #

Review Edge Insights Data #

• Review the timeline graph to identify issues and analyze performance. Sample use cases are provided below.
  • Identify sudden spikes or drops in traffic.
    Once you have identified a questionable traffic spike, determine whether it is legitimate traffic by reviewing key metrics, such as the country of origin, URL path, and query strings.
  • Identify sudden spikes in 4xx and 5xx traffic.
    1. From the Top Results section, verify that HTTP Status Code has been selected for one of the pie charts.
    2. From the Timelines section, click the HTTP Status Code source.

       

    3. Once you have identified a spike, analyze the corresponding log data to gain insight into a specific status code.
  • Identify caching trends.
    1. From the Top Results section, verify that Cache Status has been selected for one of the pie charts.
    2. From the Timelines section, click the Cache Status source.
• Review log data to troubleshoot an issue.
  For example, if you are able to reproduce an issue on your local machine and require more information than is available through the Edgio Developer Tools Chrome extension, then you may view log data for these requests from within Edge Insights. One method for analyzing these requests is to:
  1. Configure a catch-all rule to set a response header to the request’s ID through the %{http_x_ec_uuid} feature variable.

     

  2. Find out the ID corresponding to a request issued from your local machine.
     1. From the desired browser, open developer tools.
     2. From the browser, issue a request.
     3. From within developer tools, inspect the request to find out the request’s ID.

        

  3. Filter Edge Insights by that ID (i.e., Event ID = <EVENT ID>).

     

  4. From the Logs section, click on the log entry to view the log fields associated with the request.

     

Caching #

Overall Cache Performance #

Cache Responses #

• Hit (COMPONENT): Indicates that the request was served from cache from either our network (Edge) or a service worker’s local cache (Service Worker).
• Miss (COMPONENT): Indicates that Edgio could not find a cached version of the requested content with a valid time-to-live (TTL) from either our network (Edge) or a service worker’s local cache (Service Worker).
• No-Cache (Edge): Indicates that a cache content freshness check was not performed. This check is skipped when an HTTP request method is used that bypasses cache (e.g., PUT, DELETE, etc).
• Disabled (Edge): Indicates that caching was disabled through the Bypass Cache feature.
  If this behavior is undesired, review the rules applied to this request.
• Blank: A blank value indicates that the request was either not served through Edgio or it was served through a different property or environment on which the Debug Header feature has not been enabled.

Cache Miss #

• By default, content is only cached when the response from the origin includes a caching policy. Additionally, some content types may require 2 requests before they are eligible to be cached.
  View our default caching policy.
• If you have defined a custom cache policy through a rule, then you should review the rules applied to this request.
• If your origin defines a custom cache policy through headers, then you should click on the desired request and then check the Response Headers section for cache directives
• Edgio associates each request with a cache key. Edgio then uses this cache key to identify whether content has been cached for this request. By default, this cache key includes the query string. This configuration is ideal for a website that uses the query string to determine the content that will be served. For other sites, this can lead to cache fragmentation and a higher rate of cache misses.
  Customize the cache key to exclude query string parameters:
  • Rules: Create or modify a rule that includes the Cache Key feature. Configure this feature’s Query Parameters option to either exclude all query string parameters or to only include specific parameters.

    

  • CDN-as-Code:
    JavaScript./routes.js

    Copy

    1export default new Router().always({
    2  caching: { cache_key: { exclude_all_query_params: true } },
    3});

  Key information:
  • If you must add query string parameters to the cache key, we recommend that you restrict it to the parameters that are critical to your business needs. This recommendation ensures optimal performance by allowing our CDN to serve more requests from cache.
  • If you are unsure as to whether you have already defined a custom cache key, then you should review your rules for features that modify the cache key.

Website Performance #

Overall Performance #

• TTFB: Time to First Byte.
• LCP: Large Contentful Paint
• INP: Interaction to Next Paint
• CLS: Cumulative Layout Shift
• FID: First Input Delay
• Edge Hits: Indicates the number of requests served from cache and the total number of requests issued for the current page that were captured by Edgio Developer Tools Chrome extension. As noted above, it will only capture requests issued while the Edgio tab is active.
• Prefetches: Indicates the number of successful prefetch requests and the total number of prefetch requests issued for the current page that were captured by Edgio Developer Tools Chrome extension. As noted above, it will only capture requests issued while the Edgio tab is active.

Request-Specific Performance #

• The TTFB column for a high value. This column measures time to first byte. This metric is indicative of responsiveness.
• The Total Time column for a high value. This column measures the total amount of time it took to serve a response to the client.

Predictive Prefetching #

Image Optimization #

1. Verify that Image Optimization processed the image by checking for the x-edgeio-status response header.
2. Check for warnings or errors.
3. Analyze the source and output image.

Missing X-Edgeio-Status Header #

• 2xx or 3xx Response: Verify that the rule that enables Image Optimization was applied to the request.
• 5xx Response: Edgio was unable to communicate with your origin server and both the source and optimized image were not previously cached. Check your origin server’s availability.

Check for Warnings and Errors #

• OK: Image Optimization performed one or more optimizations.
  Check for an x-edgeio-warning response header to find out whether one or more optimizations were skipped. Troubleshoot warnings by analyzing image metadata.
• ERROR: Perform the following steps:
  • Check for a 400 Bad Request response. This type of response indicates that Edgio was unable to optimize the requested image due to an unsupported query string parameter or value. Fix the request URL’s query string and then try again.
  • Check the x-edgeio-error response header to find out how the optimized image exceeded our limits.

Image Metadata Analysis #

Status Codes #

1. Load the desired environment-specific Edge Insights page.
2. Verify that the Data Source option is set to Access Logs.
3. Scroll down to the Top Results section.
4. Verify that HTTP Status Code has been selected for one of the pie charts.
5. From within the pie chart, click on the desired status code. The entire dashboard will be filtered by that status code.
6. Scroll down to the Logs section.
7. Inspect each request to gain insight into why this status code is occurring.
   Filter for a specific field by typing the desired name in the upper-right hand search bar.

• 404 Not Found
• 412 Precondition Failed
• 502 Bad Gateway
• 531 Project Upstream Connection Error
• 539 Project Timeout

404 Not Found Status Code #

• Use Edge Insights, as described above, to identify the URL and the referrer from which the request originated. Check the url and the referer field, respectively.
• If the resource exists, use the Edgio Developer Tools Chrome extension to find out which rules were applied to the request. Review those rules to identify how the request is being manipulated.

412 Precondition Failed Status Code #

1. Ensure that the URLs being prefetched are similar to those fetched during page navigation.
   Prefetch URLs contain the following query string parameters: edgio_dt_pf=1&edgio_prefetch=1. These parameters are automatically excluded from the cache key. Verify that this is the only difference between URLs that are prefetched and those that are requested through standard page navigation.
2. Apply the Stale While Revalidate (stale_while_revalidate) feature to URLs that will be prefetched.
   • Rules Example:

     

   • CDN-as-Code Example:
     JavaScriptroutes.js

     Copy

     1router.get('/p/:productId', {
     2  caching: {
     3    max_age: '1h',
     4    service_worker_max_age: '1h',
     5    stale_while_revalidate: '1d', // this way stale items can still be prefetched
     6  }
     7});

3. Consider increasing the Set Max Age (max_age) feature. Short time to live (TTL) intervals generate more prefetch failures.
4. Prefetch cache misses.
   Use this capability with caution since it may significantly increase the traffic to your origin or API servers.
   • HTML Script Tag Example:
     HTML

     Copy

     1<script src="/__edgio__/prefetch/install.js" data-include-cache-misses="true" defer></script>

   • Edgio Sites Example:
     JavaScriptapp.js

     Copy

     1import install from '@edgio/prefetch/window/install';
     2
     3// Call the following once when the page loads to allow prefetch requests to be served when responses
     4// aren't available in the edge cache:
     5install({includeCacheMisses: true});

502 Bad Gateway Status Code #

1. Checking whether your web servers are available.
2. Checking your SNI configuration.
3. Comparing the request’s Host header to your certificate.
4. Verify your certificate’s chain of trust or use a self-signed certificate.
5. If you have implemented certificate pinning, you should verify that your certificates have been pinned.

Step 1: Web Server Availability #

Step 2: SNI Configuration #

• SNI: If your site requires SNI, then you should enable your origin configuration’s Use SNI option and verify that the SNI hint is set to a hostname defined within your certificate’s Subject Alternative Name (SAN) or Common Name (CN).
  If your site requires SNI and your origin configuration is misconfigured, then Edge Insights will return a proxy_hard_error field set to HARD_ERR_502_SSL_CONNECT_ERROR. A quick way of checking for this condition is to filter Edge Insights by the 502 Bad Gateway status code and then viewing a request from within the Logs section.
• No SNI: If your site does not require SNI, then you should disable your origin configuration’s Use SNI option and remove the SNI hint.

Step 3: Host Header #

Step 4: Chain of Trust or Self-Signed Certificate #

• Yes: You must enable the Allow Self-Signed Certs option on the desired origin configuration.
• No: Edgio requires a full chain certificate that allows it to verify that the client and the Certificate Authorities within that chain are trustworthy. Your certificate’s chain of trust must consist of an ordered list of certificates from the server (leaf) to the root. Edgio returns a 502 Bad Gateway when your chain of trust is incomplete (e.g., missing an intermediate certificate).

Step 5: Certificate Pinning #

504 Gateway Timeout Status Code #

531 Project Upstream Connection Error Status Code #

• The upstream host you specified in your project is incorrect.
• The DNS entry you defined points to the wrong server.
• Your servers are not responding.
• You need to add the Edgio IP addresses to your allowlist. Contact your operations team and ask them to add our IP addresses to your firewall’s IP allowlist.

539 Project Timeout Status Code #

1. A requesting client sends a request to Edgio for an asset.
2. Edgio does not find it in its cache and examines routing rules.
3. Edgio sends requests to server-side rendering (SSR) code.
4. The SSR code makes calls to the customer backend to get data needed for the page.
5. The SSR assembles the page and sends it to the Edgio edge.
6. The Edgio edge caches the page and returns it to the client.

• An error in your SSR code. For example, your asynchronous requests may be missing await or call to callback.
• A backend error (server overloaded or offline).
• An allowlist issue.

• Viewing the timings and status codes of the components in the stack in the x-edg-t header.
• Analyzing server logs.
• Performing performance profiling
• Detecting allowlist errors.
• Loading source maps within our Edgio cloud infrastructure. If this occurs, try again with sourcemaps disabled.

Allowlist #

Procedure #

1. Open your project in Edgio, then drill down to the deployment that is experiencing the 539 errors.
2. Click the SERVER tab header at the bottom of the page, then click the Resume logs arrow or the Logging is paused link to resume logging.

• An error in your SSR code
• A problem with the backend server
• An allowlist error

Good Request Example #

Error in SSR Code #

• Took too long to return a response
• Threw an exception and never returned a response

Backend Server Error #

• Ends in a status code other than 200, then the SSR code received a non-200 code from the backend server.
• Does not end in a status code at all, then the SSR did not receive a response from the backend and the problem can be either an allowlist error or a timeout error. See “Distinguishing an Allow List Error from a Timeout Error.” See Distinguishing an Allowlist Error from a Timeout Error.

Distinguishing an Allowlist Error from a Timeout Error #

1. Expand line 3 (request from your SSR code to your backend server) and select COPY AS CURL

2. Run the curl command. (The command runs the same request that the SSR code made to the backend server, but from your local machine.)

SSR Code Error #

Allowlist Error #

Edge Functions #

1. Load the desired environment-specific Edge Insights page.
2. Verify that the Data Source option is set to Access Logs.
3. Scroll down to the Top Results section.
4. Set one of the pie charts to the desired Edge Functions metric. These metrics start with Edge Function.
5. From the Timelines section, click the source corresponding to the metric selected in the previous step.
6. Optional. Filter the report to a specific edge function (i.e., Edge Function Name = <edge_function PROPERTY>).

   

7. Analyze trends.

Troubleshooting Tools #

Visual Studio Code #

1. Open .vscode/launch.json.
2. Click Add Configuration and select Node.js: Launch Program.
3. Edit this configuration to look like this:
   JavaScript

   Copy

   1{
   2  "name": "Debug Edgio App",
   3  "type": "node",
   4  "request": "launch",
   5  "cwd": "${workspaceFolder}",
   6  "autoAttachChildProcesses": true,
   7  "program": "${workspaceFolder}/node_modules/@edgio/cli",
   8  "args": ["run"]
   9}

   If your workspace folder is not located in your app’s root directory, then you will need to adjust the path defined within the program and cwd keys. The program config should always point to @edgio/cli. The cwd config should point to the root directory of your app.

Server Logs #

1.get('/p/:productId', ({ cache }) => {
2    proxy('origin', {
3      // The presence of transformRequest and transformResponse ensure that proxying is done in the Edgio cloud, not at the edge.
4      transformRequest: (req) => {
5        console.log('Request ID', req.headers['x-request-id'])
6        // Log request properties that you want to troubleshoot.
7      },
8      transformResponse: (res, req) => {
9        console.log('Response for request ID', req.headers['x-request-id'], 'status code', res.statusCode)
10        // Log response properties that you want to troubleshoot.
11      }
12    })
13  })

curl #

1curl -o/dev/null -vv https://www.yoursite.com

• Send a request directly to the origin and bypass Edgio
• Test the connection to Edgio before DNS cutover. The recommended method for testing this connection is to define a localhost DNS configuration.

1curl -o/dev/null -vv
2    https://www.yoursite.com --connect-to ::35.241.39.58

1curl -o/dev/null -vv
2    -H "Cookie: cache_enabled=true" https://www.yoursite.com/main.js

1curl -o/dev/null -vv
2    -H "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone10,2;FBMD/iPhone;FBSN/iOS;FBSV/13.6.1;FBSS/3;FBID/phone;FBLC/en_GB;FBOP/5];FBNV/1"

Source Maps #

• By default, application-level source maps are not enabled, since they may cause the Edgio cloud bundle to be larger than the 50MB limit.
• Source maps loaded within our Edgio cloud infrastructure may result in 539 Project Timeout errors due to performance issues. If this occurs, try again with sourcemaps disabled.

• Next.js: Set edgioSourceMaps: true in your next.config.js:
  JavaScript./next.config.js

  Copy

  1const { withEdgio, withServiceWorker } = require('@edgio/next/config')
  2
  3module.exports = withEdgio(
  4  withServiceWorker({
  5    // Output sourcemaps so that stacktraces have original source filenames and line numbers when tailing
  6    // the logs in the Edgio Console.
  7    edgioSourceMaps: true,
  8  }),
  9)

• Nuxt.js: Set edgioSourceMaps: true in the config for @edgio/nuxt/module in buildModules in nuxt.config.js:
  JavaScript./nuxt.config.js

  Copy

  1module.exports = {
  2  // ...
  3  buildModules: [['@edgio/nuxt/module', { edgioSourceMaps: true }]],
  4  // ...
  5}