This article describes the features and functionality of CounterMeasures. CounterMeasures are automated remediation and mitigation actions triggered from the Panopta Agent whenever a metric threshold is crossed for more than a configured amount of time. Actions range from simple debugging commands (top,netstat, or vmstat) to complex actions such as making API calls or restarting a service when a threshold is breached. CounterMeasures can increase the level of automation within your operations workflow, either by using the out of the box Panopta CounterMeasures or by creating your own.
CounterMeasures are represented throughout the control panel by a beaker () icon. This icon indicating a configured CounterMeasure is available anywhere an incident is listed. You can also a list off all the configured CounterMeasures in your account by going to Settings > CounterMeasures.
Note: To enable CounterMeasures, the Panopta Agent installed on your instance should at least be version 2018.14 for Linux, and version 18.34 for Windows. See Update the Panopta Agent to upgrade your Agent.
One of the biggest benefits of CounterMeasures is the ability to quickly arm yourself with information when an incident arises. You can configure CounterMeasures to send the information it retrieves directly in Slack, saving you the time to log into the offending instance. See CounterMeasures Slack Integration
for more information.
On this page
Go to the following sections to learn more about enabling CounterMeasures on your instances:
CounterMeasures have a simple workflow that is driven by the Panopta monitoring Agent. Here is an example:
- Enable CounterMeasures in your Agent configuration file or manifest file.
- Add a CounterMeasure to a metric threshold, such as Disk % used.
- If the threshold is crossed, the CounterMeasure will be run.
Any output provided by the CounterMeasure will be attached to the incident record and will be available in the Panopta control panel, incident log, or Slack.
Standard Linux CounterMeasure actions
Out of the box, the Panopta Agent comes with standard CounterMeasure actions that can be utilized. You can view them using
python /usr/bin/panopta-agent/countermeasure.py list_plugins.
|Top ||Displays information about CPU and memory utilization by process |
|dmesg||Displays information from the kernel|
|Netstat ||Displays network connections for the TCP, routing tables, and a number of network interface and network protocol statistics |
|Reboot Server ||Reboots the server |
|Vmstat ||Retrieves the following virtual statistics: processes, memory, paging, block IO, traps, and CPU activity |
Standard Windows CounterMeasure actions
The following CounterMeasures are available if you have the Windows Agent installed.
Displays information about CPU and memory utilization by process
Application Error Logs
Retrieves the last 50 Application Error logs
Get Service Status
Retrieves the status of all running services
Displays network connections for the TCP, routing tables, and a number of network interface and network protocol statistics
Reboots the server
Restarts the IIS web server service
System Error Logs
Retrieves the last 50 System Error logs
Vmstat ||Retrieves the following virtual statistics: processes, memory, paging, block IO, traps, and CPU activity
In addition to the standard CounterMeasure actions, you can also create your own by following the steps detailed in:
You can centrally manage your CounterMeasure plugins by adding the
countermeasures_remote_plugins parameter to your Agent manifest file that points to an archive of Agent plugins. When the Agent is installed, it will download and extract the directory in the proper location. Zip, tar, and python file types are supported.
If you would like updates to your CounterMeasures plugins to automatically be applied, you can optionally set
countermeasures_refresh_plugins to the number of hours for the refresh interval. This will help with keeping all your servers up-to-date.
See Utilizing Remote Actions for more information.
CounterMeasure metric threshold
CounterMeasures are configured together with metric thresholds. See Configure a CounterMeasure to configure CounterMeasures and thresholds. An orange beaker icon indicates that an approval is required. Selecting the the icon displays a modal where you can approve the appropriate CounterMeasure.
An Approval option is available for situations where you don't want a CounterMeasure to execute without a final authorization. To require approval for CounterMeasure actions, select the Require Approval option when configuring the CounterMeasure.
The CounterMeasureslife cycle is detailed in the following list:
- Pending: state of an action that is scheduled to execute in the future and does not require approval or has already been approved. The beaker icon will appear grey.
- Pending Approval: state of an action that is scheduled to execute in the future but will not do so until approved. The beaker icon will glow orange.
- In Process: state of an action that has been synced down to the Agent to execute. The beaker icon will appear grey.
- Complete: state of an action that has executed successfully. The beaker icon will appear green.
- Error: state of an action that attempted to execute but returned an error while executing. The beaker icon will appear red.
- Skipped: state of an action that was at one time pending but whose underlying incident was resolved before executing. The beaker icon will appear grey.
You can view a CounterMeasure's output from the following:
- CounterMeasures details panel
- Incident details page
The following figure shows the CounterMeasure output from the outage log.
See Viewing CounterMeasure output for more information.
Your account is only charged for CounterMeasures when they are actively configured on an instance. That is, only instances that have CounterMeasures configured on Alert Thresholds. For the latest pricing information, please see our pricing details.