[-]
  
[-]
  
  
  
  
  
  
  
 
 
[-]
  
  
  
  
[-]
  
 [+]
  
[-]
Monitoring
  
 [+]
 [+]
 [+]
  
[-]
Monitoring guides
  
  
[-]
Templates
  
  
  
[-]
Cloud monitoring
 [+]
  
  
 [+]
Kubernetes
[-]
Network checks
 [+]
  
  
  
[-]
 [+]
  
  
  
  
  
  
 [+]
 [+]
 [+]
 [+]
[-]
CounterMeasures
  
 [+]
 [+]
  
  
[-]
Panopta OnSight
 [+]
 [+]
[-]
SNMP
 [+]
  
[-]
Alerting
 [+]
 [+]
 [+]
 [+]
 [+]
[-]
Reporting
  
  
  
[-]
Maintenance
  
  
[-]
API
  
  
  
  
[-]
Users, Groups, and Authentication
 [+]
  
  
  
[-]
Billing and Payments
  
  
  
  
  
  
 
[-]
  
  
  
  
  
  
  
  
  
  
  
  
Updated on 7/26/2019
Online Help
Part 7: CounterMeasures
Direct link to topic in this publication:

This article describes the features and functionality of CounterMeasures. CounterMeasures are automated remediation and mitigation actions triggered from the Panopta Agent whenever a metric threshold is crossed for more than a configured amount of time. Actions range from simple debugging commands (top,netstat, or vmstat) to complex actions such as making API calls or restarting a service when a threshold is breached. CounterMeasures can increase the level of automation within your operations workflow, either by using the out of the box Panopta CounterMeasures or by creating your own.   

CounterMeasures are represented throughout the control panel by a beaker () icon. This icon indicating a configured CounterMeasure is available anywhere an incident is listed. You can also a list off all the configured CounterMeasures in your account by going to Settings > CounterMeasures.

Information

Note: To enable CounterMeasures, the Panopta Agent installed on your instance should at least be version 2018.14 for Linux, and version 18.34 for Windows. See Update the Panopta Agent to upgrade your Agent.

Slack integration

One of the biggest benefits of CounterMeasures is the ability to quickly arm yourself with information when an incident arises. You can configure CounterMeasures to send the information it retrieves directly in Slack, saving you the time to log into the offending instance. See CounterMeasures Slack Integration for more information.

On this page

Enable CounterMeasures

Go to the following sections to learn more about enabling CounterMeasures on your instances:

Example workflow

CounterMeasures have a simple workflow that is driven by the Panopta monitoring Agent. Here is an example:
  1. Enable CounterMeasures in your Agent configuration file or manifest file.
  2. Add a CounterMeasure to a metric threshold, such as Disk % used.
  3. If the threshold is crossed, the CounterMeasure will be run.
Any output provided by the CounterMeasure will be attached to the incident record and will be available in the Panopta control panel, incident log, or Slack. 

CounterMeasure actions

Standard Linux CounterMeasure actions

Out of the box, the Panopta Agent comes with standard CounterMeasure actions that can be utilized. You can view them using python /usr/bin/panopta-agent/countermeasure.py list_plugins.

Action
Description
Top Displays information about CPU and memory utilization by process
dmesg
Displays information from the kernel
Netstat
Displays network connections for the TCP, routing tables, and a number of network interface and network protocol statistics
Reboot Server
Reboots the server
Vmstat
Retrieves the following virtual statistics: processes, memory, paging, block IO, traps, and CPU activity

Standard Windows CounterMeasure actions

The following CounterMeasures are available if you have the Windows Agent installed.

Action
Description
Top
Displays information about CPU and memory utilization by process
Application Error Logs
Retrieves the last 50 Application Error logs
Get Service Status
Retrieves the status of all running services
Netstat
Displays network connections for the TCP, routing tables, and a number of network interface and network protocol statistics
Reboot
Reboots the server
Restart IIS
Restarts the IIS web server service
System Error Logs
Retrieves the last 50 System Error logs
Vmstat Retrieves the following virtual statistics: processes, memory, paging, block IO, traps, and CPU activity

Custom actions

In addition to the standard CounterMeasure actions, you can also create your own by following the steps detailed in:

Remote actions

You can centrally manage your CounterMeasure plugins by adding the countermeasures_remote_plugins parameter to your Agent manifest file that points to an archive of Agent plugins. When the Agent is installed, it will download and extract the directory in the proper location. Zip, tar, and python file types are supported.

If you would like updates to your CounterMeasures plugins to automatically be applied, you can optionally set countermeasures_refresh_plugins to the number of hours for the refresh interval. This will help with keeping all your servers up-to-date.

See Utilizing Remote Actions for more information.

    CounterMeasure metric threshold

    CounterMeasures are configured together with metric thresholds. See Configure a CounterMeasure to configure CounterMeasures and thresholds. An orange beaker icon indicates that an approval is required. Selecting the the icon displays a modal where you can approve the appropriate CounterMeasure. 

    Approval

    An Approval option is available for situations where you don't want a CounterMeasure to execute without a final authorization. To require approval for CounterMeasure actions, select the Require Approval option when configuring the CounterMeasure.

    CounterMeasure status

    The CounterMeasureslife cycle is detailed in the following list:

    • Pending: state of an action that is scheduled to execute in the future and does not require approval or has already been approved. The beaker icon will appear grey.
    • Pending Approval: state of an action that is scheduled to execute in the future but will not do so until approved. The beaker icon will glow orange.
    • In Process: state of an action that has been synced down to the Agent to execute. The beaker icon will appear grey.
    • Complete: state of an action that has executed successfully. The beaker icon will appear green.
    • Error: state of an action that attempted to execute but returned an error while executing. The beaker icon will appear red.
    • Skipped: state of an action that was at one time pending but whose underlying incident was resolved before executing. The beaker icon will appear grey.

    CounterMeasure output

    You can view a CounterMeasure's output from the following:

    • CounterMeasures details panel
    • Incident details page
    The following figure shows the CounterMeasure output from the outage log.

    See Viewing CounterMeasure output for more information.

    Billing

    Your account is only charged for CounterMeasures when they are actively configured on an instance. That is, only instances that have CounterMeasures configured on Alert Thresholds. For the latest pricing information, please see our pricing details.

    See also