Health Monitoring

• API: View core status information via the following services:
  • Integration API - /api2/slicer/status: Queries the backend to find out the last known status for one or more Live Slicers.
  • Live Slicer API - /status: Queries a Live Slicer directly for current status information.
• Dashboard: The Live Slicer Monitoring dashboard provides the following information at a glance:
  • Current and recently active Live Slicers associated with the active Slicer View.
  • Information and statistics on those Live Slicers.

Setup #

1. Enable monitoring on your account.
2. Create a view for the desired Live Slicers.
3. Monitor the Live Slicers identified above from the Live Slicer Monitoring dashboard.

Key Steps #

1. Selecting one or more customer account(s) whose Live Slicers can be monitored.
2. Creating a Slicer View that identifies the set of Live Slicers that will be monitored.
3. Creating a rule set that defines the criteria when metrics will trigger and then clear an alert. Assign this rule set to the desired Live Slicer(s).
4. Defining notification preferences.
5. Granting Live Slicer monitoring access to other accounts.

Prerequisites #

1. Authorization Requirements:
   • You must be authorized to monitor the desired Live Slicer(s). Authorization is granted under any of the following conditions:
     • The desired Live Slicer(s) correspond to your customer account.
     • A user that owns the desired Live Slicer(s) has granted you monitoring permissions on their customer account.
       Monitoring permissions defined on the Account Access page only determine access to the legacy Live Slicer Health dashboard. This version of the Live Slicer Monitoring dashboard requires that you grant monitoring permissions through the Allowed Watchers page. Learn more.
   • You belong to an organization that has been assigned the account that owns the desired Live Slicer(s). Learn more.
2. Enable Monitoring: You must enable monitoring on the customer account that owns the Live Slicer.

Monitoring #

Enable monitoring on an account #

1. Navigate to the Monitored Accounts page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Monitored Accounts.
2. Click Configure Monitored Accounts:
   • Mark each customer account for which monitoring will be enabled.
   • Click Add # Accounts to confirm.

Disable monitoring on an account #

1. Navigate to the Monitored Accounts page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Monitored Accounts.
2. Disable monitoring on an account:
   • Hover over the desired customer account.
   • Click Delete Selected Accounts.
   • Click Yes, Delete It to confirm that monitoring should be disabled on the account.
     Disabling monitoring removes that customer account’s Live Slicers from your Slicer Views.

Permissions #

Allow user(s) to monitor your Live Slicers #

1. Navigate to the Allowed Watchers page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Allowed Watchers.
2. Organization: Admin and Read/Write Users:
   • Set the Accounts option to Personal.
3. Add user(s) to monitor your Live Slicers:
   • From the right-hand pane, type the name of the user (e.g., joe@example.com) that will be allowed to monitor your Live Slicers.
   • Verify the specified user name before adding it. For your security, our system does not validate user names.
   • Click + Add Watcher.
4. Repeat steps 2 and 3 as needed.

Disallow user(s) from monitoring your Live Slicers #

1. Navigate to the Allowed Watchers page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Allowed Watchers.
2. Organization: Admin and Read/Write Users:
   • Set the Accounts option to Personal.
3. Remove user(s) from monitoring your Live Slicers:
   • Hover over the desired user and then click Delete Selected.
   • Click Yes, Remove It to confirm that the user should no longer be allowed to monitor your Live Slicers.
     Removing monitoring permissions from a user will remove your Live Slicers from the Slicer Views associated with that user.

Allow user(s) to monitor Live Slicers associated with an account #

1. Navigate to the Allowed Watchers page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Allowed Watchers.
2. Select the desired account:
   • From the Accounts option, select the account you wish to manage.
3. Add user(s) to monitor Live Slicers:
   • From the right-hand pane, type the name of the user (e.g., joe@example.com) that will be allowed to monitor your Live Slicers.
   • Verify the specified user name before adding it. For your security, our system does not validate user names.
   • Click + Add Watcher.
4. Repeat steps 2 and 3 as needed.

Disallow user(s) from monitoring Live Slicers associated with an account #

1. Navigate to the Allowed Watchers page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Allowed Watchers.
2. Select the desired account:
   • From the Accounts option, select the account from which you wish to remove users.
3. Remove user(s) from monitoring Live Slicers:
   • Mark each desired user.
   • Click Delete Selected.
   • Click Yes, Remove It to confirm that the marked user(s) should no longer be allowed to monitor the Live Slicers associated with the account selected in step 2.

Slicer View #

• Focus on monitoring key Live Slicers.
• Quickly switch monitoring focus between different sets of Live Slicers.
• A user with either the admin or read/write permission can create Slicer Views that can be leveraged by any member of your organization. This type of Slicer View is known as an organization Slicer View. Learn more.

Create a Slicer View #

1. Verify that you have enabled monitoring on the desired accounts.
2. Navigate to the Slicer Views page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Slicer Views.
3. Organization: Admin and Read/Write Users:
   • Determine whether a Slicer View will be created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
   • Click + Create New Slicer View.
4. Create the Slicer View:
   • Type the name that will be assigned to the Slicer View in the View Name field.
   • Optional: Use the Search field to filter the list of available Live Slicers by name and owner.
   • Mark the desired Live Slicers.
   • Click the red left-arrow icon.

     

   • Click Save.

Modify a Slicer View #

2. Navigate to the Slicer Views page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Slicer Views.
3. Organization: Admin and Read/Write Users:
   • Determine whether to modify a Slicer View that was created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
   • Click on the desired Slicer View.
4. Modify the Slicer View:
   • Add Live Slicers by marking from the Available Slicers list and then clicking the red left-arrow icon.

     

   • Remove Live Slicers by marking them from the Slicers in View list and then clicking the red right-arrow icon.

     

   • Click Save.

Delete a Slicer View #

2. Navigate to the Slicer Views page:
   • From the main menu, navigate to Services > Monitoring 2.0.
   • Then navigate to Settings > Slicer Views.
3. Organization: Admin and Read/Write Users:
   • Determine whether to delete a Slicer View that was created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
   • Hover over the desired Slicer View and then click Delete (rubbish bin icon).
   • Click Yes, Delete It to confirm that the Slicer View should be deleted.

Monitoring Rules #

Key Information #

• By default, Live Slicers will be monitored using a default set of read-only monitoring rules. However, a custom set of monitoring rules may be assigned to one or more Live Slicers.
• View your custom monitoring configurations from the Monitoring Rules page.
• Once you have created a custom monitoring configuration, you may reassign a Live Slicer to it. This means that the previous set of monitoring rules (e.g., Default) will no longer apply to the Live Slicer, since a Live Slicer may only be associated with a single set of monitoring rules at any given time.
• All metrics, with the exception of Closed Captioning Last Seen, are updated every 4 seconds. The Closed Captioning Last Seen metric is updated every 10 seconds.
• The Live Slicer will only report monitoring data when slicing content. It cannot report data when it is in another state (i.e., ad break, replace content, or blackout), since it is not receiving a video feed.
• A user with either the admin or read/write permission can create rulesets that can be leveraged by any member of your organization. This type of ruleset is known as an organization ruleset. Learn more.

Create a Custom Set of Monitoring Rules #

1. Navigate to the Monitoring Rules page.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
   • Navigate to Settings > Monitoring Rules.
2. Organization: Admin and Read/Write Users: Determine whether a ruleset will be created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
3. Click + Create New Ruleset.
   • Type the name for the ruleset in the Ruleset Name field.
   • Configure each metric that should trigger warning/critical status.
     • Find the desired metric.
     • Select either Warning or Critical from the Severity option.
     • Define the conditions under which the metric will change health states.
       • States: Configure the Set option to the number of seconds that the sub-optimal state (e.g., no signal, no closed captions detected, or black video was detected) must persist before the Live Slicer’s health will change to the severity defined in the Severity option. Configure the Clear option to the number of seconds that the optimal state must persist before the Live Slicer’s health will change back to healthy.
       • Measurements: Configure the Set Threshold option to the condition that must be met for the number of seconds defined in the Set Duration option before the Live Slicer’s health will change to the severity defined in the Severity option. Configure the Clear Threshold option to the condition that must be met for the number of seconds defined in the Clear Duration option before the Live Slicer’s health will change back to healthy.
4. Repeat the above steps as needed.
5. Click Save.

Modify a Custom Set of Monitoring Rules #

1. Navigate to the Monitoring Rules page.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
   • Navigate to Settings > Monitoring Rules.
2. Organization: Admin and Read/Write Users. Determine whether to modify a ruleset that was created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
3. Click on the desired ruleset.
4. Review and modify the conditions under which metrics will change health states as needed.
5. Click Save.

Delete a Custom Set of Monitoring Rules #

1. Navigate to the Monitoring Rules page.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
   • Navigate to Settings > Monitoring Rules.
2. Organization: Admin and Read/Write Users. Determine whether to delete a ruleset that was created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
3. Mark each desired ruleset.
4. Click Delete Selected Rulesets.
5. Click Yes, Delete It.

Assign a Custom Set of Monitoring Rules to One or More Live Slicers #

1. Navigate to the Monitoring Rules page.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
   • Navigate to Settings > Monitoring Rules.
2. Determine whether you will map a ruleset that was created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
3. Click Ruleset Mappings.
4. From the 1. Choose a Ruleset to Apply option, select the desired custom monitoring configuration.
5. Optional. Use the Search field to filter the list of available Live Slicers by name and owner.
6. Mark the desired Live Slicers.
7. Click Apply Ruleset to Selected Slicers.
   Alternatively, you may assign a ruleset to a Live Slicer from the Live Slicer Monitoring dashboard by clicking on the desired Live Slicer and then selecting the rule set that will be applied to it from the Ruleset option.

Assign the Default Set of Monitoring Rules to One or More Live Slicers #

1. Navigate to the Monitoring Rules page.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
   • Navigate to Settings > Monitoring Rules.
2. Determine whether you will map a ruleset that was created for your organization or for personal use. From the Organizations option, select either your organization or Personal.
3. Click Ruleset Mappings.
4. Optional. Use the Search field to filter the list of available Live Slicers by name and owner.
5. Mark the desired Live Slicers.
6. Click Remove Ruleset from Selected Slicers.

Metrics #

Audio Notifications #

• The state of a Live Slicer changes from “Healthy” to “Warning.”
• The state of a Live Slicer changes to “Critical.”

Set up Audio Notifications #

1. Navigate to the Notifications page.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
   • Navigate to Settings > Notifications.
2. Click + Create New Profile.
3. From the Name option, type the name of the new notification profile.
4. Toggle warning or critical audio notifications by clicking on the Enabled/Disabled Audio Alarm option from the Warning or Critical section.
5. Click Save.
6. Navigate to the Live Slicer Monitoring dashboard.
   • From the main menu, navigate to Services.
   • Click Monitoring 2.0.
7. From the Saved Views option, select a Slicer View that contains the desired Live Slicer.
8. Select the desired Live Slicer.
9. From the Active Notification Profile option, select the notification profile created in step 3.
10. Repeat steps 7 - 9 for each desired Live Slicer.

Dashboard #

• Determines whether you will view the Slicer View(s) associated with your personal user account or your organization’s shared ones.
• Allows you to switch to a different Slicer View.
• Indicates the current Slicer View and the number of Live Slicers associated with it by health status.
• Toggles Live Slicers from the dashboard by owner. For each owner, it also indicates the number of Live Slicers by health status.

• Filter by Live Slicer ID: Type the desired ID and then press the Enter key.
• Filter by Live Slicer health status: Click the health status filter icon

  

  , mark and/or clear the desired statuses, and then click off of it to hide the popup.
• Choose Columns: Select the set of columns to be displayed by clicking the columns icon

  

  .
• View Detailed Information: Clicking on a Live Slicer provides more detailed information about that Live Slicer and its health. This detailed view is known as Live Slicer Details.

Live Slicer Details #

• Health Measurement: Live Slicer health is measured according to its current ruleset. View and/or set a Live Slicer’s ruleset from the pane on the right-hand side of the window.
• Assign Rulesets: Assign rulesets to one or more Live Slicer(s) from the Monitoring Rules page.
• Set Profile: View and/or set the profile that determines when the Live Slicer will push health data to Amazon SNS or trigger audio notifications from the pane on the right-hand side of the window.

Slicer Snapshot #

• Preview: Displays a thumbnail preview of the Live Slicer’s output. The output for this preview varies according to its state.
  • Slicing: Displays recently sliced content.
  • Ad Break: Displays a message indicating that the thumbnail is unavailable.
  • Replacing Content: Displays a message indicating that the thumbnail is unavailable.
  • Blackout: Displays a message indicating that the thumbnail is unavailable.
  • Offline: By default, green frames are shown whenever the signal is lost. Black frames or an image may be shown instead when either the no_signal_pad or the no_signal_image setting is defined.
• Health Level: Indicates the Live Slicer’s current health status and the duration that it has been in that state.
• Slicer Status: Indicates the Live Slicer’s status and the duration that it has been in that state.
• Health - {Ruleset}: Displays historical health status for the current Live Slicer. The line graph’s title indicates the ruleset through which health status was determined.

Charts #

• 

  Healthy: Indicates that all monitored metrics are below warning thresholds.

• 

  Neutral: Indicates that the Live Slicer is not slicing content.

• 

  Warning: Indicates that one or more monitored metrics are at warning levels.

• 

  Critical: Indicates that one or more monitored metrics are at critical levels.

Health Details #

• Details: Identifies the timestamp for the current set of health data and the amount of time that the Live Slicer has been in the current state.
• Metrics: Provides health data for each metric. Specifically, each row identifies a metric, its health status at the time identified in the Details section, and a brief description of the last time that this metric’s health status changed.

Status #

• Details: Identifies the timestamp for the current set of health data.
• Status JSON: Provides detailed Live Slicer information, including metrics, in JSON format.
• Metrics: Provides data for each metric.

Alert Log Data #

• Alert log data is paginated and limited to 10,000 entries. If you encounter this limitation, adjust your filters (e.g., reduce the time window or only show changes in health status) to reduce the number of entries.
• Alert log data is displayed in reverse chronological order (i.e., newest to oldest event).
• Log alerts are triggered according to the ruleset(s) assigned to your Live Slicer(s).

• Health Status: Mark or clear the desired health status(es).
• Status Change: Filter log data to only show changes in a Live Slicer’s health status by marking the Show changes in health only option.
• Time: Specify the date range for which log data will be shown by setting the From and To options and then clicking Apply.

• View Log Data: View the Alert Log page by clicking Log from the Live Slicer Monitoring dashboard.
• Refresh Log Data: View log events that have occurred since the start of your browsing session by clicking Apply.
• Filter Log Data: Filtering options are provided on the left-hand side of the page.
• Navigate Log Data: The Alert Log page can display up to 15 alert events. View older alerts by clicking on one of the following:
  • « First: Navigates to the first page.
  • Previous: Navigates to the previous page.
  • Page_Number: Navigates to a specific page.
  • Next: Navigates to the next page.
  • Last »: Navigates to the last page.

Penalty Box #

1. From the main menu, navigate to Services and then click on Monitoring 2.0.
2. Open the Slicers menu and then click on Penalty Box.

• Unhealthy Live Slicers for your own account belong to the Personal slicers category. This category also includes any Live Slicers to which you have been granted access that do not belong to your organization.
• If you belong to an organization, then your organization will also be displayed as a category.

Issue Remediation #

• Egress bandwidth
• Source signal
• System resources
• Configuration

Egress Bandwidth #

Diagnose Issues #

Testing #

1$ cd /opt/uplynk/latest
2$ ./slicer -u <username> -apikey <APIKey> -bandwidth

Source Signal #

Diagnosis #

SDI Source #

UDP Transport Streams #

Reverse Path Filtering #

Resolution/Remediation #

• Fixing the routing configuration.
• Disabling reverse path filtering through the following command:
  Bash

  Copy

  1$ echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
  2$ sysctl -w "net.ipv4.conf.all.rp_filter=0"

System Resources #

Diagnosis #

Configuration #

Troubleshooting #

Asset Rollover #

Delayed Playback #

Organizations #

• Account: A dedicated customer account allows multiple users to leverage shared Live Slicers, live channels, live events, and content. Users may not log in directly to an account.
• User: An individual user can manage their own Slicers, Live Slicers, live channels, live events, and content. Authorized users may also monitor Live Slicers associated with accounts.
• Administrator: An administrator of your organization can modify the organization’s user membership. All other organization properties are read-only. An administrator may only add users who have previously used this version of the Live Slicer Monitoring dashboard.

User Capabilities #

• Leverage your organization’s Slicer Views and rulesets for Live Slicer health monitoring.
• Users with either admin or read/write permissions are authorized to create, modify, and delete your organization’s Slicer Views and rulesets.
• Monitor the Live Slicers associated with your organization’s accounts. A user may either use your organization’s Slicer View(s) or create one or more Slicer View(s) for their own personal use.

Additional Details #

• All of your organization’s users may select your organization’s accounts from the Monitored Accounts page.
• Organization membership does not grant access to the Live Slicers associated with the personal accounts of other members. However, another member may grant you permission to monitor their Live Slicers.
• By default, all users associated with your organization are authorized to monitor the Live Slicers associated with your organization’s accounts. An administrator may modify each account’s allowed watchers to authorize users outside of your organization or disallow some of your organization’s users from monitoring those Live Slicers.
• Our system is responsible for creating and associating accounts with an organization. An account may only be associated with a single organization.
• Contact your account manager to request an organization.

Modify an Organization’s Membership and User Roles #

Access Organizations #

1. Navigate to the Organizations Administration page. From the main menu, navigate to Services > Monitoring 2.0 > Settings > Organizations Administration.
2. Click on your organization.
3. Click on the Users tab.
4. Optional. Add users to your organization.
   You may only add users who have previously used this version of the Live Slicer Monitoring dashboard.
   • Click + Add Users.
   • In the 1. User(s): option, type the name of the user(s) to be added. If adding multiple users, delimit each username with a comma.
     Example: joe@example.com,jane@example.com
   • From the 2. Select Role to Apply to user(s): option, select the level of access for the user(s) defined in the previous step.
   • Click Add.
5. Optional: Assign a new role to a user by clicking the user’s current role and selecting the new role to assign.
6. Optional: Remove users from your organization.
   • Mark each desired user.
   • Click Remove.
   • Click Yes, Delete It.
   • Click Save.