Web Server Log Delivery Setup

RTLD may automatically deliver compressed log data to a web server by submitting HTTPS POST requests to it. The body for each of these requests will be a JSON or CSV document that uniquely identifies a set of log data and describes one or more log entries.

Key information:

You may deliver logs to a web server that does not require authorization or that authorizes requests through the Authorization header. We support standard HTTP basic authentication or passing a custom token to this header.
RTLD applies gzip compression to log data. Each HTTPS POST request includes a Content-Encoding header set to gzip.
Log fields vary by RTLD module: RTLD CDN | RTLD WAF | RTLD Rate Limiting | RTLD Bot

To prepare your web servers for log delivery

Configure your web server(s) to:
- Support the HTTPS protocol.
  
  Log delivery requires a certificate whose trust anchor is a publicly trusted certificate authority (CA). Additionally, the certificate must include a chain of trust for all intermediate certificate(s) and a leaf certificate.
- Allow HTTPS POST requests.
- Return a 2xx response (e.g., 200 OK) whenever data is successfully received.
  
  If your web server responds with any other status code, then our service will retransmit the same log data at regular intervals. This may result in the delivery of duplicate log data.
Configure your firewall to allow POST requests from this IP block: 198.7.21.0/24
Set up a workflow for handling or processing the log data that will be posted to your web server(s).

Example: Create a listener for HTTPS POST requests that mines specific data from log entries.
Upon completing the above steps, you should create a log delivery profile for HTTP POST.

To set up a log delivery profile

From the Real-Time Log Delivery page, click + New Log Delivery Profile and then select the desired type of log field.
1. Open the desired property.
  1. Select either your private space or a team space.
  2. Click on the desired property.
2. From the left pane, click on the desired environment.
3. From the left pane, click Realtime Log Delivery.
4. Click + New Log Delivery Profile and then select either CDN, WAF, Rate Limiting, or Bot.
From the Profile Name option, assign a name to this log delivery profile.
From the Log Delivery Method option, select HTTP POST.
Define how RTLD will communicate with your web server(s).
1. Set the Request URL option to a URL that may leverage the workflow defined in the To prepare your web servers for log delivery procedure. This URL must use the HTTPS protocol.
  
  Specify a custom port to deliver log data over that port instead of 443.
  
  Sample URL: https://logs.mydomain.com/cdn/logs.aspx
2. From the Authentication Type option, select one of the following modes:
  - Custom Authentication: Select this mode when your web server(s) expects the Authorization request header to be set to a custom token value. Set the Token option to a value that will authorize requests to your web server(s).
    
    Log data will be posted to your web server(s) via HTTPS POST requests with an Authorization request header set to the specified value.
    
    Authorization header syntax: Authorization: <TOKEN>
  - HTTP Basic: Select this mode if your web server(s) allow content to be uploaded via standard HTTP basic authentication. Set the desired credentials via the Username and Password options.
    
    Base-64 encoding will applied to the specified credentials. After which, the encoded value will be passed via the Authorization header.
    
    Authorization header syntax: Authorization: Basic <BASE64-ENCODED CREDENTIALS>
  - None: Select this mode if your web server(s) allow content to be posted without authorization.
From the Log Format option, select whether to format log data using our standard JSON format, as a JSON array, as JSON lines, or as a CSV (RTLD CDN only).

Learn more about these formats: RTLD CDN | RTLD WAF | RTLD Rate Limiting | RTLD Bot
From the Downsample the Logs option, determine whether to reduce the amount of log data that will be delivered. For example, you may choose to only deliver 1% of your log data.
- All Log Data: Verify that the Downsample the Logs option is cleared.
- Downsampled Log Data: Downsample logs to 0.1%, 1%, 25%, 50%, or 75% of total log data by marking the Downsample the Logs option and then selecting the desired rate from the Downsampling Rate option.
  
  Use this capability to reduce the amount of data that needs to be processed or stored within your web server(s).
  
  RTLD CDN Only: Downsampling log data also reduces usage charges for this service.
Determine whether log data will be filtered.
By default, all log fields are enabled on a new log delivery profile. From within the Fields section, clear each field for which log data should not be reported.
- Log fields are categorized. You may add or remove individual fields by expanding a category and then marking or clearing specific log fields. Alternatively, add or remove all of the log fields associated with a category by marking or clearing the desired category.
- RTLD CDN Only: You may also log request headers, response headers, and cookies.
- View log field definitions: RTLD CDN | RTLD WAF | RTLD Rate Limiting | RTLD Bot
Click Create Log Delivery Profile.