Log Delivery Service

• Content Delivery (HTTP): Provides detailed insights into the content delivery via the CDN.
• MMD Live Ingest (MMD_LIVE_INGEST): Captures information about the transcoding of ingested media streams into various bitrates and formats for further playback.

LDS Workflow #

1. You start by setting up your Log Delivery Service (LDS) configuration using either the LDS UI or LDS API. This configuration determines what logs data is collected, how it is processed, and delivered.
2. LDS picks up configuration and begins processing based on new settings.
3. As users interact with your CDN services, their requests are logged.
4. LDS collects and processes these logs from the CDN, extracting and organizing the data based on the configuration you set. This step involves parsing the logs and preparing them for delivery.
5. After processing, LDS delivers the logs to the specified destination, which could be cloud storage, a custom HTTP endpoint, or another location of your choice.
6. Once delivered, the logs are ready for you to analyze.

View and Manage LDS Configurations #

• NAME: The name you provided for the configuration.
• SHORTNAME: The account name for which the LDS configuration collects log data.
• SERVICE TYPE: CDN service from which the logs are collected by the LDS configuration.
• DESTINATION: The target storage system where the LDS configuration sends the collected logs.
• DATA COMPRESSION: Indicates the algorithm used to compress log data.
• STATE: Shows the current state of the configuration.
  • Active: The configuration is active and processing log data.
  • Suspended: The configuration is paused and not processing log data.
• LAST UPDATED: Shows the date and time when the LDS configuration was last modified.
• STATUS: Includes controls to activate or deactivate the configuration, as well as to delete it.

Create an LDS Configuration #

1. Navigate to Configure > Log Delivery Service and click the ”+” button.
2. Enter a NAME for the configuration.
3. Select a SHORTNAME that you want the LDS to collect logs for.
4. From the list of available services, select the CDN SERVICE from which you want LDS to collect log data. The supported services include:
   • Content Delivery (HTTP)
   • MMD Live Ingest (MMD_LIVE_INGEST)
5. In Available Log Fields, select the fields to collect, move them to Selected Log Fields by dragging or double-clicking. Arrange the fields in the desired order. For details on the available fields for each service, refer to the following links:
   • Available Log Fields for Content Delivery
   • Available Log Fields for MMD Live Ingest
6. Optionally, can rename selected fields by assigning aliases. To do this, click the square brackets next to the field name, and enter the alias. For details refer to Field Aliases.
7. Additionally, you can add static fields by clicking the Add Static Field button. This opens a popup where you can enter a field name and a static value. For details refer to Static Fields.
8. In the DESTINATION dropdown, select the storage type where you want the logs to be delivered.
   Each account can have only one LDS configuration per destination. For additional configurations to the same destination, please contact your account manager.
9. Enter the details for your chosen destination.
   • For Amazon S3 see Configure logs delivery to Amazon S3
   • For Custom HTTPS endpoint see Configure logs delivery to Custom HTTPS endpoint
   • For Datadog see Configure logs delivery to Datadog (Streaming)
   • For Origin Storage see Configure logs delivery to Edgio Origin Storage
   • For Google Cloud Storage see Configure logs delivery to Google Cloud Storage
   • For Hydrolix see Configure logs delivery to Hydrolix (Streaming)
   • For Microsoft Azure Blob Storage see Configure logs delivery to Microsoft Azure Blob Storage
   • For New Relic see Configure logs delivery to New Relic (Streaming)
   • For Splunk see Configure logs delivery to Splunk (Streaming)
   • For Sumo Logic see Configure logs delivery to Sumo Logic (Streaming)
10. In the Delivery Options section, customize the log data format and compression, and set file naming and storage paths.
    • Use DIRECTORY LAYOUT dropdown to choose the folder path within the destination. Available for file-based destinations (Amazon S3, Azure Blob Storage, Edgio Origin Storage and Google Cloud Storage). For further details refer to Changing Log Data Location.
    • The FILENAME TEMPLATE dropdown determines the naming convention for log files uploaded to your destination storage. The filename template is fixed and cannot be edited through the UI. Available for file-based destinations (Amazon S3, Azure Blob Storage, Edgio Origin Storage and Google Cloud Storage). For details refer, see Changing Log Data Location.
    • In the DATA FORMAT dropdown, you can select the format for the uploaded log data. Available formats may vary depending on the selected destination type. See Log Data Formats.
    • In the DATA COMPRESSION dropdown, select the compression format for the log data. The available options may vary based on the selected destination type. Refer to Log Data Compression for more information.
11. In the Data Sampling section, configure the volume of delivered log data. For more information, see Data Sampling.
12. Click Save to apply your updates.
    It can take 15 to 40 minutes for changes to take effect.

Edit an LDS Configuration #

1. Go to Configure > Log Delivery Service page.
2. Find the configuration you want to edit, and click on it to open the configuration in edit mode.
3. Update the following settings as needed:
   • Name: Modify the name of the configuration for easier identification.
   • Available Log Fields: Change the fields to be collected by dragging them between the Available and Selected sections, or add static fields using the Add Static Field button. Additionally, you can assign aliases to selected fields by clicking the square brackets next to the field name and entering an alias in the input box that appears. For details refer to Static Fields and Field Aliases
   • In the Delivery Destination section, modify the destination details for log delivery as necessary.
     • For Amazon S3 see Configure logs delivery to Amazon S3
     • For Custom HTTPS endpoint see Configure logs delivery to Custom HTTPS endpoint
     • For Datadog see Configure logs delivery to Datadog (Streaming)
     • For Origin Storage see Configure logs delivery to Edgio Origin Storage
     • For Google Cloud Storage see Configure logs delivery to Google Cloud Storage
     • For Hydrolix see Configure logs delivery to Hydrolix (Streaming)
     • For Microsoft Azure Blob Storage see Configure logs delivery to Microsoft Azure Blob Storage
     • For New Relic see Configure logs delivery to New Relic (Streaming)
     • For Splunk see Configure logs delivery to Splunk (Streaming)
     • For Sumo Logic see Configure logs delivery to Sumo Logic (Streaming)
4. Under the Delivery Options section:
   • Use DIRECTORY LAYOUT dropdown to configure the folder structure within the destination for file-based storage (Amazon S3, Azure Blob Storage, Edgio Origin Storage and Google Cloud Storage). For more information, see Changing Log Data Location.
   • In the DATA FORMAT dropdown choose the format for the log data. The available options may depend on your selected destination. Refer to Log Data Formats for more information.
   • In the DATA COMPRESSION dropdown select the compression method for the log data. The available choices vary by destination type. See Log Data Compression for details.
5. In the Data Sampling section, adjust the volume of log data being delivered. For more information, see Data Sampling.
6. Click Save to apply your updates.

Activate or Deactivate LDS Configuration #

1. Configuration Grid: On the Configure > Log Delivery Service page, check the Status column:
   • The icon means the configuration is active. Clicking it will deactivate the configuration.
   • The icon means the configuration is inactive. Clicking it will activate the configuration.
2. While Editing the Configuration: When editing a configuration, use the button at the bottom to activate or deactivate it.
   It may take 5 to 10 minutes for changes to activation status to take effect.

Log Fields #

Content Delivery Log Fields #

Date/Time-Related Metrics #

Client-Server Communication Metrics #

Server-Client Communication-Related Metrics #

Client-Related Metrics #

Server-Related Metrics #

Server-Origin Communication-Related Metrics#

Origin-Related Metrics#

Application (Edgio)-Specific Metrics #

MMD Live Ingest Log Fields #

Static Fields #

Create Static Field #

1. Click ADD STATIC FIELD below the Available Log Fields section in your configuration.
2. Enter a field name (up to 50 characters, starting with a letter, using lowercase Latin letters, digits, _, or -) and a value (up to 1024 characters).
3. After entering the field name and value, click the ADD STATIC FIELD button to create the field. The field will then appear in the Available Log Fields section.

• Static fields are shared across all LDS configurations for the same account and service type. For example, a static field created in an HTTP service configuration for the test account will be available in all other configurations for that service and account.
• You can create up to 10 static fields per account and service type.
• You can add the static field to the Selected Log Fields by dragging it or double-clicking it.

Edit Static Field #

• To change the value of a static field, click on its name in either the Available Log Fields or Selected Log Fields section. A popup will appear allowing you to modify the field value. The field name is fixed and can only be specified during creation.

Delete Static Field #

• To delete a static field, first ensure it is removed from the Selected Log Fields section, as deletion is not permitted while it is in use in any LDS configuration. Click on the field name in the Available Log Fields section. A popup will appear with a Delete button for confirmation.

Field Aliases #

Assigning an Alias to Log Field #

1. To assign an alias, click the square brackets next to the field name in the Selected Log Fields section. An input box will appear where you can enter the alias.
2. Once done, press Enter or click outside the box to save. The alias will then display alongside the original log field name.

Edit Alias #

• Click on the alias, modify the name, and press Enter or click outside the input box to save.

Delete Alias #

• To delete an alias, click the alias, clear the name, and press Enter or click outside the input box to save.

Configure Destinations #

Delivery to Amazon S3 #

1. Create an S3 Identity and Access Management (IAM) user in Amazon’s configuration screens.
2. Give the IAM user the following permissions for the bucket where you want to store logs:
   • ListBucket
   • GetObject
   • PutObject

1. On the Create/Update LDS Configuration page in the Delivery Destination section, choose Amazon S3 from the dropdown.
2. Enter your bucket name in the BUCKET NAME field.
3. Select the AWS region where the bucket is located from the REGION dropdown.
4. Optionally, specify a path within the bucket for storing files in the PATH field.
   Do not add a leading slash to the path. If you do, Amazon creates an object URL with a double slash. Example: https:://bucket.s3.region.amazonaws.com//cdn_logs...
5. In the ACCESS KEY field, enter the access key for your Amazon S3 bucket.
6. In the SECRET KEY field, enter the secret key for your Amazon S3 bucket.
7. Optionally, in the Delivery Options section:
   • Change the DIRECTORY LAYOUT to customize how logs are organized within a bucket. For details, refer to Log Files Naming and Location.
   • Choose the DATA COMPRESSION method for your log files. Supported options include: LZ4 (default), GZIP, LZF, SNAPPY, ZSTD.
   • Select the log DATA FORMAT:: W3C (Tab-Separated)(default), TSV, JSON Lines, JSON Array, JSON.
   • If you need fields with null values to be included in JSON format files, select the INCLUDE NULL VALUES IN JSON option, as they are excluded by default.
8. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
9. Click Save to validate the connection to the destination and apply your settings.

Delivery to Custom HTTPS endpoint (Streaming) #

1. On the LDS Configuration page, in the Delivery Destination section, select Custom HTTPS Endpoint (Streaming) from the dropdown.
2. Enter the HTTPS URL that accepts POST requests in the URL field.
3. Optionally, in the AUTHORIZATION HEADER VALUE field, provide the value for the Authorization header to be used when sending logs (e.g., Basic <Base64 encoded username and password> or Bearer <Your API key>).
4. Optionally, use the CUSTOM HEADER NAME and CUSTOM HEADER VALUE fields to specify additional HTTP headers. You can add multiple headers using the +(plus) button or remove them with the -(minus) button.
   Standard headers Content-Type, Encoding, Authorization, and Host are not supported.
5. Optionally, in the Delivery Options section:
   • Choose the log DATA COMPRESSION method. Supported options include: ZSTD, GZIP.
   • Select the log DATA FORMAT:: TSV, JSON Lines, JSON Array, JSON.
   • If you need fields with null values to be included in JSON format messages, select the INCLUDE NULL VALUES IN JSON option, as they are excluded by default.
6. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
7. Click Save to validate the connection to the destination and apply your settings.

Delivery to Datadog (Streaming) #

• A Datadog account: Use an existing account or create a new one.
• A Datadog API key: Generate via Datadog. (See Datadog’s documentation on API and Application Keys.)

1. On the LDS Configuration screen, in the Delivery Destination section, select Datadog (Streaming) from the dropdown.
2. Enter the following details:
   • SITE: Select the Datadog site region that matches your Datadog environment.
   • API KEY: Enter the API key associated with your Datadog account.
   • SERVICE: (Optional) Specify the property to be used as the ‘service’ property in Datadog.
   • TAGS: (Optional) Provide a comma-separated list of tags to include with the logs (e.g., cdn:edgio).
3. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
4. Click Save to validate the connection to the destination and apply your settings.

Delivery to Edgio Origin Storage #

• Ensure Origin Storage is enabled for the account where you want to store logs.

1. On the LDS Configuration screen, in the Delivery Destination section, select Origin Storage from the dropdown.
2. Optionally, specify the STORAGE ACCOUNT if you want to use a different Origin Storage account. By default, logs are stored under the same account that owns the LDS configuration.
   Log files are stored in a directory named ‘_livelogs’ at the root of your space. You are responsible for managing and removing data; LDS does not handle data removal.
3. Optionally, in the Delivery Options section:
   • Change the DIRECTORY LAYOUT to customize how logs are organized within a bucket. For details, refer to Log Files Naming and Location.
   • Choose the DATA COMPRESSION method for your log files. Supported options include: LZ4 (default), GZIP, LZF, SNAPPY, ZSTD.
   • Select the log DATA FORMAT:: W3C (Tab-Separated)(default), TSV, JSON Lines, JSON Array, JSON.
   • If you need fields with null values to be included in JSON format files, select the INCLUDE NULL VALUES IN JSON option, as they are excluded by default.
4. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
5. Click Save to validate the connection to the destination and apply your settings.

Delivery to Google Cloud Storage #

• Create a Google Cloud Project (GCP) or use an existing project. See Google’s Google’s guide for instructions.
• Set up a GCP bucket to store your logs. You can create a new bucket or use an existing one. See Google’s Create Storage Buckets Guide for instructions.
• Create a Google service account that will use to access your bucket. See Google’s Service accounts guide for instructions.
• Using Google’s IAM roles for Cloud Storage, guide, grant the following roles on the bucket:
  • Storage Object Creator (storage.objectCreate)
  • Storage Object Viewer (storage.objectViewer)
• Add the service account as a member of the bucket you created in step 2.
• Generate JSON access keys for the service account. See Google’s Creating service account keys guide for instructions.

1. On the LDS Configuration screen, in the Delivery Destination section, select Google Cloud Storage from the dropdown.
2. In the CLIENT EMAIL field, enter the value of the client_email field from the JSON key file associated with your Google service account.
3. In the SECRET KEY field, enter the value of the private_key field from the JSON key file associated with your Google service account.
   • Note: The secret key is sensitive information. After saving the configuration, the key will not be visible. However, you can enter a new key and save the configuration if needed.
4. In the BUCKET NAME field, enter the name of the storage bucket you created in your Google Cloud account.
5. Optionally, specify the PATH within the bucket where logs will be stored. By default, logs are stored in the root of the bucket.
6. Optionally, in the Delivery Options section:
   • Change the DIRECTORY LAYOUT to customize how logs are organized within a bucket. For details, refer to Log Files Naming and Location.
   • Choose the DATA COMPRESSION method for your log files. Supported options include: LZ4 (default), GZIP, LZF, SNAPPY, ZSTD.
   • Select the log DATA FORMAT:: W3C (Tab-Separated)(default), TSV, JSON Lines, JSON Array, JSON.
   • If you need fields with null values to be included in JSON format files, select the INCLUDE NULL VALUES IN JSON option, as they are excluded by default.
7. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
8. Click Save to validate the connection to the destination and apply your settings.
   During the validation process, LDS creates a test file in the specified GCP bucket using the provided credentials and settings. The file is named _edgio_validation/validation_{timestamp}.txt. The file will only appear if the validation is successful. Since these files are no longer necessary after validation, you can delete them as needed.

Delivery to Hydrolix (Streaming) #

• Create a Project/Table
• Create a Transform

1. On the LDS Configuration screen, in the Delivery Destination section, select Hydrolix (Streaming) from the dropdown.
2. Enter the following details:
   • STREAMING API HOSTNAME: Enter the hostname of your Hydrolix Streaming API. This value will be used in the URL https://<hydrolix-streaming-api-hostname>/ingest/event for log ingestion.
   • PROJECT NAME: Enter the Hydrolix project name to include in the x-hdx-project HTTP header.
   • TABLE NAME: Enter the Hydrolix table name to include in the x-hdx-table HTTP header.
   • TRANSFORM SCHEMA NAME: (Optional) Specify the Hydrolix transform schema to include in the x-hdx-transform HTTP header.
   • AUTHORIZATION HEADER VALUE: (Optional) Enter the authorization header value to use when sending logs (e.g., Basic <Base64 encoded username and password> or Bearer <API key>).
   • STREAM TOKEN: (Optional) Provide a streaming token to include in the x-hdx-token HTTP header.
3. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
4. Click Save to validate the connection to the destination and apply your settings.

Delivery to Microsoft Azure Blob Storage #

1. Create an Azure storage account. See Microsoft’s guide for instructions.
2. Create a container to which log data will be uploaded. Refer to Microsoft’s guide for instructions.
3. Obtain SAS token or an access key for authorizing requests to upload content to your Azure Blob Storage account.

1. On the Create/Update LDS Configuration page in the Delivery Destination section, choose Microsoft Azure from the dropdown.
2. Enter the name of your storage account in the ACCOUNT NAME field.
3. Specify the name of Azure conatiner to which log data will be uploaded in the CONTAINER NAME field.
4. From the ACCESS METHOD dropdown, select your preferred authorization method. Then, enter the access key or SAS token in the input field that appears next to it.
5. Optionally, in the Delivery Options section:
   • Change the DIRECTORY LAYOUT to customize how logs are organized within a bucket. For details, refer to Log Files Naming and Location.
   • Choose the DATA COMPRESSION method for your log files. Supported options include: LZ4 (default), GZIP, LZF, SNAPPY, ZSTD.
   • Select the log DATA FORMAT:: W3C (Tab-Separated)(default), TSV, JSON Lines, JSON Array, JSON.
   • If you need fields with null values to be included in JSON format files, select the INCLUDE NULL VALUES IN JSON option, as they are excluded by default.
6. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
7. Click Save to validate the connection to the destination and apply your settings.

Delivery to New Relic (Streaming) #

• Register a New Relic account or use existing one.
• Obtain your license ingest key.

1. On the LDS Configuration screen, in the Delivery Destination section, select New Relic (streaming) from the dropdown.
2. In the DOMAIN field, enter the New Relic API domain. This value will be used in the URL https://<new-relic-domain>/log/v1 for log ingestion.
3. In the API KEY field, enter your account licence ingest key you created earlier .
4. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
5. Click Save to validate the connection to the destination and apply your settings.

Delivery to Splunk (Streaming) #

• Follow this guide to set up and enable the HEC.
• Ensure that indexer acknowledgment is disabled in your HEC configuration.
• Save the HEC token and the URL pointing to your Splunk HTTP Event Collector.

1. On the LDS Configuration screen, in the Delivery Destination section, select Splunk (Streaming) from the dropdown.
2. In the DOMAIN field, enter the Splunk HEC API domain. This value will be used in the URL https://<splunk-domain>/services/collector/event for log ingestion.
3. In the EVENT COLLECTOR TOKEN field, enter the HEC token you created and enabled in Splunk.
4. Optionally, in the INDEX field, specify the index name by which the event data is to be indexed. The index you specify here must be within the list of allowed indexes if the HEC token has the indexes parameter set.
5. Optionally, in the FIELDS input, provide a JSON object with custom fields to include in log events. These fields will be sent as metadata and indexed by Splunk. Example: {"env":"production", "region":"us-west"}.
6. Optionally, in the SOURCE Type field, specify the source type value to assign to the event data.
7. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
8. Click Save to validate the connection to the destination and apply your settings.

Delivery to Sumo Logic (Streaming) #

1. On the LDS Configuration screen, in the Delivery Destination section, select Sumo Logic (streaming) from the dropdown.
2. In the ENDPOINT field, enter the domain name of the Sumo Logic HTTP source which will form the URL https://<endpoint>/receiver/v1/http for log ingestion.
3. In the HTTP COLLECTOR CODE field, enter the code from your Sumo Logic endpoint URL. This is the last part of the URL in the format https://<SumoEndpoint>/receiver/v1/http/<HTTPCollectorCode>.
4. Optionally, in the SOURCE CATEGORY field, specify acustom source category. This value will be sent in the X-Sumo-Category header.
5. Optionally, in the FIELDS input, provide key-value pairs to tag your logs for easier searching. These tags will be sent in the X-Sumo-Fields header. Example: environment=dev,cluster=k8s.
6. Optionally, in the Data Sampling section, reduce the volume of delivered logs by applying sampling rules. Refer to the Data Sampling for additional details.
7. Click Save to validate the connection to the destination and apply your settings.

Data Formats #

W3C (Tab-Separated)#

1#Version: 1.0
2#Date: 2024-09-06 11:42:07
3#Software: Log Delivery Service
4#Start-Date: 2024-09-05 01:00:00
5#End-Date: 2024-09-05 02:00:00
6#Fields: date   time    s-host  c-ip    c-country   c-city  sc-bytes    s-ttfb  duration
72024-09-05  01:10:51.706    cds102.jed.llnw.net 10.252.49.10    -   -   259 0   0
82024-09-05  01:12:18.018    cds102.sabf.llnw.net    185.76.48.149   united kingdom  southampton 455 262 262

TSV#

1date    time    s-host  c-ip    c-country   c-city  sc-bytes    s-ttfb  duration
22024-09-05  01:10:51.706    cds102.jed.llnw.net 10.252.49.10    -   -   259 0   0
32024-09-05  01:12:18.018    cds102.sabf.llnw.net    185.76.48.149   united kingdom  southampton 455 262 262

JSON Lines#

1{"date":"2024-09-05","time":"01:10:51.706","s-host":"cds102.jed.llnw.net","c-ip":"10.252.49.10","sc-bytes":259,"s-ttfb":0,"duration":0}
2{"date":"2024-09-05","time":"01:12:18.018","s-host":"cds102.sabf.llnw.net","c-ip":"185.76.48.149","c-country":"united kingdom","c-city":"southampton","sc-bytes":259,"s-ttfb":0,"duration":0}

JSON Array#

1[
2  {"date":"2024-09-05","time":"01:10:51.706","s-host":"cds102.jed.llnw.net","c-ip":"10.252.49.10","sc-bytes":259,"s-ttfb":0,"duration":0},
3  {"date":"2024-09-05","time":"01:12:18.018","s-host":"cds102.sabf.llnw.net","c-ip":"185.76.48.149","c-country":"united kingdom","c-city":"southampton","sc-bytes":259,"s-ttfb":0,"duration":0}
4]

JSON#

• agent_id: Always an empty string ("").
• seq_num: Always set to 0.
• platform: Always “delivery”.
• account_number: The Edgio account number as a string, identifying the account associated with the logs.
• datestamp: Indicates the date when the log data was prepared for delivery in YYYYMMDD format
• logs: array contains individual log entries.

1{
2    "agent_id": "",
3    "seq_num": 0,
4    "platform": "adn",
5    "account_number": "0001",
6    "datestamp": "20180416",
7    "logs": [
8          {"date":"2024-09-05","time":"01:10:51.706","s-host":"cds102.jed.llnw.net","c-ip":"10.252.49.10","sc-bytes":259,"s-ttfb":0,"duration":0},
9          {"date":"2024-09-05","time":"01:12:18.018","s-host":"cds102.sabf.llnw.net","c-ip":"185.76.48.149","c-country":"united kingdom","c-city":"southampton","sc-bytes":259,"s-ttfb":0,"duration":0}
10     ]
11}

Configuration #

1. Go to the Delivery Options section on the Create/Update LDS Configuration page.
2. Select the desired format from the DATA FORMAT dropdown.

Log Files Naming and Location #

Folder Path #

• PATH: This is an optional static root path within the storage which can be specified in the Delivery Destination section on the Create/Edit LDS configuration page only when Amazon S3, Google Cloud Storage or Microsoft Azure Blob Storage is selected.
• DIRECTORY LAYOUT: Defines the folder structure within the path. It supports these dynamic placeholders that are replaced with relevant information during file upload:
  • {service_type}: Type of service for which logs are collected.
  • {config_uuid}: UUID of the LDS configuration.
  • {yyyy}, {MM}, {dd}: Resolves to year, month, and day respectively based on the start of the time period of the log entries covered in the file, all in UTC timezone.
  • {yyyy_proc}, {MM_proc}, {dd_proc}: Resolves to year, month, and day respectively using the timestamp that represents the time when the file was prepared by LDS for delivery, all in UTC timezone.

• {service_type}/{config_uuid}/{yyyy_proc}/{MM_proc}/{dd_proc} (default): Organizes directories by service type, configuration UUID, and the date when the file was processed by LDS.
• {service_type}/{config_uuid}/{yyyy}/{MM}/{dd}: Organizes directories by service type, configuration UUID, and the date range of the log entries.
• No Subdirectories: Places all log files in the root directory of the storage, without any subdirectory structure. This option is available for Amazon S3, Google Cloud Storage and Microsoft Azure Blob Storage.

Log File Naming #

• {shortname}: The account name for which the log entries have been collected.
• {config_uuid}: UUID of the LDS configuration.
• {request_end_date_time_from}: The start timestamp of the time period covered by log entries in the file, formatted as {year}{month}{day}{hour}{minute}{second} in UTC.
• {request_end_date_time_to}: The end timestamp of the time period covered by log entries in the file, formatted as {year}{month}{day}{hour}{minute}{second} in UTC.
• {process_window_date_time}: The timestamp when the file was prepared for delivery, formatted as {year}{month}{day}{hour}{minute}{second} in UTC.
• {split_id}: ID assigned to the file for splitting large log files. Files are split to avoid exceeding the 1GB size limit. The first split file is labeled 000, with subsequent parts numbered sequentially (001, 002, etc.). If no splitting is needed, the split_id is 000.
• {format}: Reflects the selected Data Format. Options include w3c, json_lines, json_array,json or tsv.
• {compression}: File compression format.

Data Compression#

• GZIP
• LZF
• LZ4 (Frame Format)
• SNAPPY
• ZSTD

Configuration #

1. Go to the Delivery Options section on the Create/Edit LDS configuration page.
2. Select the desired format from the DATA COMPRESSION dropdown.

Data Sampling#

1. Go to the Data Sampling section on the Create/Update LDS Configuration screen.
2. Use the slider to select the percentage of log lines you wish to deliver for each status code group. The valid range for sampling rates is 0 to 100:
   • 0 means all data is filtered out (no logs are delivered).
   • 100 means no filtering is applied (all logs are delivered).
   • Any value between 0 and 100 represents the percentage of log lines that will be included.

Personally Identifiable Information #

• cs-cookie
• cs-uri
• so-src-uri

Sign PII Agreements#

• Non-Company Admin users can sign agreements only for the company to which they belong.
• Company Admin users can sign agreements for child companies as well.

Permissions #

• Configure > Log Delivery Service > View: Allows you to view the list of LDS configurations and their details.
• Configure > Log Delivery Service > Manage: Grants you full access to view and manage LDS configurations. This includes the ability to create, edit, activate, deactivate, and delete configurations.

Downloading LDS Files from Origin Storage #

Origin Storage Management Console #

1. Open the Origin Storage Console under the Manage menu.
2. In the bottom-right corner, select your Account and Username from the dropdowns.
3. Navigate to the folder containing the file.
4. Click the Download icon next to the file, and your browser will download it automatically.

Downloading Files via Origin Storage API #

Prerequisites #

• API Username and Password: You need credentials to authenticate your API requests. You can manage these on the Manage > Origin Storage Users page.
  It is not possible to view an existing user’s password, but you can set a new one if needed.
• URL to the Storage API: Your Edgio account name is a part of the JSON-RPC API URL. The API URL follows this format: http://{Account name}-l.upload.llnw.net/jsonrpc2
  • Example: http://supercustomer.upload.llnw.net/jsonrpc2
  • Your Edgio account name is included in the Welcome letter you received when you purchased Origin Storage. If you have any questions or issues, please contact your Edgio representative.
  • If you are a long-standing customer and the preceding endpoints fail to resolve, use the following endpoints with the suffix -l after {Account name}:
    • Example: http://supercustomer-l.upload.llnw.net/jsonrpc2

Step 1: Authenticate using the login Method#

1login(username, password, detail)

• username: Origin Storage API username.
• password: Origin Storage API password.
• detail: (optional) A boolean (true/false) indicating whether to return detailed data or a simpler response. Defaults to False.

Step 2: List Log Files using the listFile Method#

1listFile( token, dir, pageSize, cookie, stat)

• token: The authentication token obtained from the login method.
• dir: A string representing the directory path from which you want to list files.
• pageSize: The number of files to return in each response (useful for limiting the size of results).
• cookie: A number used for paginating results across multiple listFile calls.
• stat: A boolean (true/false) indicating whether to return detailed file information (such as file size, creation time, etc.).

Step 3: Obtain a Protected Download URL #

1mediaVaultUrl(token, path, expiry)

• token: The authentication token obtained from the login method.
• path: The file path for which you want to generate a secure download URL.
• expiry: (optional) The number of seconds for which the download URL should remain valid. Must be in the range 1 to 2147483648. If set to 0 or omitted, it defaults to 3600 seconds (1 hour).

• code: The return code, where 0 indicates success.
• download_url: The URL to download the specified file.
• message: A description of the return code.
• preview_url: The URL to preview the file, if applicable.

1{
2  "code": 0,
3  "download_url": "http://cs-download.limelight.com/<path to file>",
4  "message": "success",
5  "preview_url": "http://cs-download.limelight.com/<path to file>", 
6}

Step 4: Download the File #

Python Script for Downloading Files from Origin Storage #

1#!/usr/bin/env python
2import logging, sys
3from multiprocessing import Pool
4import requests
5import time
6import json
7 
8# Configuration
9API_USERNAME = ''
10API_PASSWORD = ''
11API_URL = 'http://{Account Name}.upload.llnw.net/jsonrpc2'
12STORAGE_LOG_DIR = '/_livelogs/http/'
13 
14MAX_FILES_TO_DOWNLOAD = 5
15DELETE_FILE_AFTER_DOWNLOAD = False
16DELETE_EMPTY_DIRECTORY = False
17'''
18Author: spandey
19Unofficial Sample. Provided AS IS. WITHOUT ANY WARRANTY OR CONDITIONS.
20Uses Python 3
21'''
22LOG_FILENAME = 'LDSDownloadSession.log'
23FileList = 'DLFiles.log'
24logging.basicConfig(filename = LOG_FILENAME, level = logging.DEBUG, format='%(asctime)s %(levelname)s-%(filename)s:%(message)s')
25logger = logging.getLogger(__name__)
26 
27class StatError(RuntimeError):
28        def __init__(self, arg):
29                self.args = arg
30 
31class ListPathError(RuntimeError):
32        def __init__(self, arg):
33                self.args = arg
34 
35jsonRPC_Endpoint=''
36token=''
37cookie = ''
38numFiles = 0
39 
40numDirs = 0
41totalBytes = 0
42oldFileList = []
43theFileList = []
44dirList = []
45threads = []
46'''
47User-defined variables
48'''
49pageSize = 10000  # page size for listing log files
50files_to_download = []  # log files to download
51media_vault_expiry = 60  # expiry time for mediaVaultUrl request
52mv_errors = { -1: "Internal error", -2: "Path exists and is a directory", -8: "Invalid path",
53            -34: "Invalid expiry", -60: "Service is disabled or unavailable", -10001: "Invalid token"}
54'''
55Function to examine files returned from calls to listFile
56Based on a condition that you determine, you write file names to a list
57of files that will later be downloaded.
58This simple example looks for file names that contain the number 2.
59need to add {"method":"deleteFile","id":39,"jsonrpc":"2.0","params":{"token":"a4660cbde39c4dceb2cf796d494db3da","path":"/lll/1.mp4"}}
60'''
61 
62def parse_list(file_list):
63    for log_file in file_list:
64        name = log_file['name']
65    if name.find('2') > -1:
66        files_to_download.append(name)
67        print(log_file['name'])
68 
69def getFileListing(token, _dirname_, res):
70    numDirs = len(res['dirs'])
71    numFiles =  len(res['files'])
72    _directories_ = res['dirs']
73    print ("Total directory count: " + str(numDirs))
74    print ("Total file count: " + str(numFiles))
75    #Delete the dir in case is empty and is not the base path
76    if numDirs == 0 and numFiles == 0 and _dirname_.count('/') > 3 and DELETE_EMPTY_DIRECTORY:
77        delp = '{"method":"deleteDir","id":1,"jsonrpc":"2.0","params":{"token":"'+ token +'","path":"'+ _dirname_ +'"}}'
78        print("\nDeleting Directory : "+ delp)
79        delpRes = requests.post(jsonRPC_Endpoint, data=delp)
80        delpRes = json.loads(delpRes.text)
81        delpCode = delpRes['result']
82        #print("\n\n-------------- Code: " + str(delpCode) )
83        if delpCode != 0:
84            print("Error attempting to call del url.\nCode: " + str(delpCode))
85 
86    for _dir_ in _directories_:
87        #print ("Scanning Directory: " + _dir_['name'] + " for dirs")
88        dirName = _dirname_ + '/' + _dir_['name']
89        listPath(token, dirName)
90    # Listing Files
91    file_listing = res['files']
92    conteggio = 0
93    for file in file_listing:
94        '''
95        Download file. This is a single-threaded approach for simple use & demonstration purposes.
96        Customers might want to try a multi-threaded approach for a large number of files to download.
97        '''
98        conteggio += 1
99        log_path = _dirname_ + file['name']
100        print("\nDownloading file: "+ log_path)
101        mvu = '{"method": "mediaVaultUrl", "id": 1, "params": {"token":"'+ token +'", "path": "'+ log_path +'", "expiry": '+str(media_vault_expiry)+'}, "jsonrpc": "2.0"}'
102        mvuRes = requests.post(jsonRPC_Endpoint, data=mvu)
103        mvuRes = json.loads(mvuRes.text)
104        #print("==== Printing mediaVaultUrl response  ====\n")
105        #print(mvuRes)
106        code = mvuRes['result']['code']
107        if code !=0:
108            print("Error attempting to call 'mediaVaultUrl.\nCode: " + str(code) + ": " + mv_errors[code])
109        else:
110            mv_download_url = mvuRes['result']['download_url']
111            # grab the name of the file to write from mv url
112            lds_file_name = mv_download_url.rsplit("/",1)[1].split("?")[0]
113            #print(mv_download_url, '\nFilename:'+lds_file_name)
114            with open(lds_file_name, "wb") as f:
115                # Use the requests library to make the download
116                response = requests.get(mv_download_url, stream=True)
117                # check & show download progress
118                total_length = response.headers.get('content-length')
119                if total_length is None: # no content length header
120                    print("no content-length header found")
121                    f.write(response.content)
122                else:
123                    dl = 0
124                    total_length = int(total_length)
125                    for data in response.iter_content(chunk_size=4096):
126                        dl += len(data)
127                        f.write(data)
128                        done = int(50 * dl / total_length)
129                        sys.stdout.write("\r[%s%s]" % ('|' * done, ' ' * (50-done)) )
130                        sys.stdout.flush()
131                    if DELETE_FILE_AFTER_DOWNLOAD:
132                        #Delete the file just downloaded
133                        delu = '{"method":"deleteFile","id":1,"jsonrpc":"2.0","params":{"token":"'+ token +'","path":"'+ log_path +'"}}'
134                        print("\nDeleting file: "+ log_path)
135                        deluRes = requests.post(jsonRPC_Endpoint, data=delu)
136                        deluRes = json.loads(deluRes.text)
137                        delcode = deluRes['result']
138                        if delcode !=0:
139                            print("Error attempting to call del url.\nCode: " + str(delcode) + ": " + mv_errors[delcode])
140                # Upon non-success write a line to your errors file
141                if response.status_code != 200:
142                    print("Unable to download " + file['name'] + ". Status code: " + response.status_code)
143        if conteggio == MAX_FILES_TO_DOWNLOAD:
144            break    # break here
145 
146 
147 
148def listPath(token, _dirname_):
149        '''
150    Listing path recursively
151    '''
152        try:
153                # Scan through parent directory for the files and sub-dirs
154                listpathdata='{"method": "listPath","id": 1,"params": {"token": "'+token+'","path": "'+_dirname_+'","pageSize": '+str(pageSize)+',"cookie": "'+cookie+'","stat": true},"jsonrpc": "2.0"}'
155                # print('===List Path Data - Parent==\n', listpathdata)
156                res = requests.post(jsonRPC_Endpoint, data=listpathdata)
157                res = json.loads(res.text)
158                print ('====== Listing Path: '+_dirname_)
159                code = res['result']['code']
160 
161                if code !=0:
162                        msg = 'Error issuing listPath command on directory: ' + _dirname_
163                        msg += '\n Return code: ' + str( code )
164                        msg += '\n See API documentation for details.'
165                        logger.error('ListPathError' + msg)
166                        raise ListPathError( msg )
167 
168                theFileList = getFileListing(token,_dirname_,res['result'])
169 
170        except ListPathError as e:
171                print (''.join( e.args ))
172 
173        except StatError as e:
174                print (''.join( e.args ))
175 
176def main(host, username, password):
177    global jsonRPC_Endpoint
178    jsonRPC_Endpoint = host
179    try:
180        # Obtain token
181        loginData='{"method": "login","id": 0,"params": {"username": "'+username+'","password":"'+password+'","detail": true},"jsonrpc": "2.0"}'
182        login = requests.post(jsonRPC_Endpoint, data=loginData)
183        #print (login.reason, login.headers)
184        resp = json.loads(login.text)
185        token = resp['result'][0]
186        print ('====== Successfully Logged In.')
187        #print ('Token: '+token)
188        logger.debug('Logged In. Token Is: '+ token)
189        # Persist Token for the session until the logout or the end of time defined by 'expire'
190        persistData='{"method": "updateSession","id": 4,"params": {"token": "'+token+'", "expire": 0},"jsonrpc": "2.0"}'
191        persist = requests.post(jsonRPC_Endpoint, data=persistData)
192        persist = json.loads(persist.text)
193        #print('=======Updated Token======\n',persist)
194        if persist['result'] == 0:
195        #    print('Token Persisted!')
196            logger.debug('Token Persisted!. Token Is: '+ token)
197 
198        # call listPath method on storage
199        listPath(token, STORAGE_LOG_DIR)
200    except Exception as e:
201        print(''.join( e.args ))
202        logger.error('Error Occured While Logging')
203    finally:
204        #print ('\nLogging out...\n')
205        logoutData = '{"method": "logout","id": 1,"params": {"token": "'+token+'"},"jsonrpc": "2.0"}'
206        logoutRes = requests.post(jsonRPC_Endpoint, data=logoutData)
207        logoutRes = json.loads(logoutRes.text)
208        #print ('Logout results: ', logoutRes)
209        if logoutRes['result'] == 0:
210            logger.debug('Logged Out!')
211 
212if __name__ == "__main__":
213    main(API_URL, API_USERNAME,API_PASSWORD)

Script Configuration #

• API_USERNAME: Your API username.
• API_PASSWORD: Your API password.
• API_URL: The URL of your Origin Storage API.
• STORAGE_LOG_DIR: The directory path where the script will look for files to download.
• MAX_FILES_TO_DOWNLOAD: Limit the number of files to download. Set to 0 for unlimited downloads.
• DELETE_FILE_AFTER_DOWNLOAD: Set to True if you want to delete files after downloading.
• DELETE_EMPTY_DIRECTORY: Set to True to delete any empty directories after the files are removed.

How to Run the Script #

1. Install the necessary dependencies if not already installed: requests, logging, and multiprocessing.

1pip install requests

2. Configure the script by setting your API_USERNAME, API_PASSWORD, API_URL, and STORAGE_LOG_DIR variables.
3. Adjust other optional settings like MAX_FILES_TO_DOWNLOAD, DELETE_FILE_AFTER_DOWNLOAD, and DELETE_EMPTY_DIRECTORY based on your needs.
4. Run the script:

1python download_files.py