If you are an Origin Storage customer, you can use Intelligent Ingest to automatically populate Origin Storage with new content as it is requested from the CDN. If a request results in a CDN Cache Miss, and the content is not found in Origin Storage, Intelligent Ingest will retrieve and ingest the content from the remote host(s) you specify. You can create any number of “rules” that pair specific Origin Storage logins and content paths with specific remote hosts and paths. These rules are also known as “rewrites.”
Intelligent Ingest List Page
The list page provides summary information at the top, including Status (Active, Inactive or Disabled), Storage Quota (if quota is set; % and total disk usage), Bandwidth (the ingest bandwidth limit for the remote hosts), and Concurrency (maximum number of concurrent connections to the remote hosts).
A list of existing rules appears below the summary area. For each rule, the content paths for both Origin Storage and the remote host are shown along with the authentication type, if any.
Enabling Intelligent Ingest
To enable Intelligent Ingest, please contact your Account Manager. After Intelligent Ingest has been turned up for your account, “Intelligent Ingest” will appear under the Configure menu in the navigation pane.
Intelligent Ingest configuration requires at least one active Origin Storage login.
Configuration Overview
To successfully configure , the following steps must be completed:
- Obtain an Origin Storage account with an active login. An Origin Storage account is required for an configuration.
- Enable Intelligent Ingest for that Origin Storage account (see Enabling Intelligent Ingest ).
- Request a Caching & Delivery configuration (or use an existing one) to enable CDN delivery of objects from the Origin Storage account.
- Create one or more Intelligent Ingest rules (see Adding Intelligent Ingest Rules).
Creating or Editing a Content Delivery Configuration for Intelligent Ingest
As part of Intelligent Ingest configuration, you must create a related Content Delivery configuration.
You can either edit an existing Content Delivery configuration or create a new one.
To work with Content Delivery configurations, perform these steps:
-
Select Caching & Delivery (v2) in the Configure tab.
-
Perform instructions in the following sections:
Step 1-Configuring Content Location
In the Content Location section, make these selections in the Source subsection:
In the Location of Content Source drop-down menu, choose the descriptive name that was configured for you. The actual name is displayed in the Source hostname field.
Populate the Base Path and Extended Path fields as needed.
Purpose of Content Location Configurations
Intelligent Ingest rules rely on matching the path of the content you requested from our Origin Storage platform. Generally, the full path on Origin Storage is not something you would have complete information about. The goal is to illustrate the full path of the content location within Origin Storage. The Base path is the part of your Origin Storage path that Edgio creates for you. Rather, it is a drop-down menu, because depending on your permissions, you may have multiple Origin Storage users with various levels of access. Control then allows you to add an extended path if you don’t want to trigger Intelligent Ingest on everything in the base path. The extended path allows more flexibility to use the feature on a subset of content.
Step 2-Configuring Failover to Backup Origin
The Failover to Backup Origin option is required, and must be requested from your Account Manager. Once added, it will be visible in the Content Delivery Configuration Self Service settings.
- You can set up Intelligent Ingest rules and their associated Content Delivery configuration in any order, but both are required before Intelligent Ingest begins ingesting content.
- Although Intelligent Ingest rules require an associated Content Delivery configuration, deleting the configuration does not delete the rules, and deleting rules does not affect the configuration.
- You can configure rules so that content is ingested only for specific paths within a Content Delivery configuration.
- If you need a more complex configuration, more options for your 404 backup origin, or authentication for your backup origin , please contact your Account Manager.
- Although Intelligent Ingest rules require an associated Content Delivery configuration, deleting the configuration does not delete the rules, and deleting rules does not affect the configuration.
- You can configure rules so that content is ingested only for specific paths within a Content Delivery configuration.
- If you need a more complex configuration, more options for your 404 backup origin, or authentication for your backup origin , please contact your Account Manager.
Adding Intelligent Ingest Rules
Before you add a rule, become familiar with how rules are chosen at ingest time. See Workflow Rule Selection.
-
On the page, click the + New button above the summary area. The Create new rewrite rule appears with the fields necessary to create the rule.
-
Perform instructions in the following sections:
Step 1-Configuring Origin Storage Origin
The Origin Storage logins associated with the current are displayed in the Origin Storaage login dropdown menu. Select the appropriate user, and also enter the target Content path (the path to the Origin Storage directory from which content will be requested, and to which it will be automatically ingested if requested but not found in that directory).
When you configured the Content Location in Caching & Delivery, you selected the Base Path and entered the Extended Path. The target Content path should match the concatenation of those two fields, as shown in Testing Paths below.
Step 2-Configuring Remote Storage Host
For the remote host, under Remote Protocol, specify the protocol ( HTTP or HTTPS ) should use when requesting your content. Then enter the in Remote storage host, and the folder path, if any, in Remote path.
Step 3-Testing Paths
To view and confirm the final origin and remote paths, click the Test button and review the values in the Path in Origin Storage and Matched remote path fields.
- The Test button is enabled only when the Path in Origin Storage field is filled in.
- Testing is intended to confirm the entered paths. It does not request content or confirm its availability.
- Testing is intended to confirm the entered paths. It does not request content or confirm its availability.
The Path in Origin Storage field should contain the full path to the Origin Storage directory from which content will be requested, and the Matched remote path field should contain the full path to the remote host directory from which the content will be ingested if not found in that Origin Storage directory. If either path is not what you expected, change the values entered for Content path, Remote storage Host and/or Remote path as needed, and repeat the test.
Step 4-Enabling Remote Storage Host Authentication
If your content is hosted on a third-party content provider, you must configure the credentials needed for Intelligent Ingest to access your data. You can choose from existing configurations or create a new configuration.
- Click the Enable Remote Storage Host Authentication checkbox.
- Choose an existing configuration from the or create a new configuration.
- To create a new configuration:
-
Click the + new button.
-
Choose a configuration type in the subsequent dialog.
Type Description / Instructions Amazon S3 V4 Select this if your content is hosted on.
See Amazon friendly name Fields.Custom Header Select this if your content is hosted elsewhere.
See Request headers friendly name Fields. -
Click Apply to create the new rule or Cancel to discard your work.
-
Amazon S3 V4 Fields
This authentication type uses Amazon S3 Signature Version 4 Authentication to send authentication fields to the remote host.
| Name | Description / Instructions |
|---|---|
| Name | Descriptive name of your choice |
| Access key ID | Amazon AWS Access Key ID |
| Secret access key | Amazon AWS Secret Access Key |
| Region | region into which your S3 content will be ingested. Choose the region that is closest to the S3 region. |
Custom Header
This authentication type sends authentication information in request headers to the remote host.
| Name | Description / Instructions |
|---|---|
| Name | Descriptive name of your choice |
| Add custom request header | 1. Enter a name and a value, then click the + icon. 2. Repeat until you have entered all required values. To remove a header, click the - icon. After you save the rule (Saving the Intelligent Ingest Rule), the request headers configurations are available to choose for use in future rules. |
Step 5-Saving the Intelligent Ingest Rule
When you are ready, click Save to create the new rule or Cancel to discard your work.
After you save a rule, you cannot modify it; you can only modify any associated authentications (see Managing Authentications).
Managing Authentications
The only component of an existing rule that you can change is its authentications. You can modify, delete, and create authentications.
You cannot restore a deleted authentication.
Begin by clicking the manage authentication button on the right above the summary area. Doing so displays a dialog for managing authentications.
To create a new authentication:
-
Click the + new button in the dialog. A dialog for creating authentications is displayed.
-
Create the authentication, using instructions in Enabling Remote Storage Host Authentication for creating a new authentication.
-
Click Apply to save changes, or Cancel to discard your work.
To modify an authentication:
-
Click the edit icon for the desired authentication in the dialog. A dialog for editing the authentication is displayed.
-
Modify the authentication using information in Amazon S3 V4 Fields and Custom Header Fields.
-
Click Apply to save changes, or Cancel to discard your work.
To delete an authentication:
- Click the Delete icon for the desired authentication in the dialog.
- Click Delete in the confirmation dialog.
Deleting Rules
To delete a rule, click its associated trashcan icon.
Deleting an rule does not affect any configuration(s) associated with the deleted rule.
Workflow Rule Selection
Intelligent Ingest includes a workflow in which it attempts to ingest content from origin paths. Understanding the selection process is important before you begin creating rules.
If multiple rules share the same path, the workflow uses path prefix length and order in which rules were created to choose the rule to use.
In the following examples, ‘/home/data/’ is the path prefix.
Example 1
If multiple rules with the same prefix are created in the following order (the shortest is created first), then only the first path will work and the others will fail as part of workflow because they are longer.
-
/home/data/ - works
-
/home/data/dir1 - fails
-
/home/data/dir2 - fails
Example 2
If multiple rules with the same prefix are created in the following order (the shortest is created later), then the first and second paths will work and the third will fail because it is longer than the second.
-
/home/data/dir1 - works
-
/home/data/ - works
-
/home/data/dir2 - fails