As a quick reminder, CounterMeasures are automated responses to specific events that the Panopta agent detects on your infrastructure. We provide a handful of automated actions out-of-the-box and also allow you to write your own. Regardless of the route you choose, you will be leveraging the Panopta agent as the conduit for the process.
CounterMeasures are available beginning in Windows agent version 18.34. If you're using an agent version older than this, check out this article for more information upgrading your agent.
If you're performing a new agent installation, you can enable CounterMeasures in the agent manifest file and then install the agent .
Let's run through a quick example:
Invoke-WebRequest https://packages.panopta.com/install/panopta_agent_windows.ps1 -OutFile panopta_agent_windows.ps1; .\panopta_agent_windows.ps1 -customer_key CUSTOMER_KEY -server_key NONE
CounterMeasures is now ready to go on your instance. You can confirm that CounterMeasures have successfully been enabled by going to the Instance Details page. Click the question mark icon beside Agent Status and locate the CounterMeasures - Enabled datapoint in the Agent Status information block.
<add key="EnableCounterMeasures" value="" />
<add key="EnableCounterMeasures" value="true" />
CounterMeasures are tied to metric thresholds, which means they're configured together as well. Let's walk through how to add a CounterMeasures to capture top when CPU usage elevates.
CPU: Percent used - _Total
+ Add CounterMeasure.
Now you're good to go. When the threshold is crossed, the countermeasure will be triggered. We'll run top, collect the output, and display it in the log for the generated incident.
External monitoring checks (HTTP, HTTPs, Ping, etc) can also be used to trigger CounterMeasures. The agent still must be running on the instance, but when the external check threshold is crossed, it will trigger a CounterMeasure (e.g., restart Apache) on the instance.
The output (if any) from your CounterMeasure action is available from a number of locations.
Throughout the ControlPanel, you can open the CounterMeasures detail panel by clicking the CounterMeasures indicator icon, which is the beaker.
The beaker icon can be found in a number of locations - virtually anywhere an incident may be listed.
Clicking the icon will open the CounterMeasures detail panel. It containers, among either things, any output the CounterMeasure returned after running.
CounterMeasures output is also available from the Incident details page. In the top navigation, select Incidents. From the table, select the incident you're interested in (you'll also noticed the beaker icon in the row). From the Incident details page, select Outage Log. In the log, you will find details about which CounterMeasures have been executed and what output the returned.
CounterMeasures have a simple, linear lifecycle. Each lifecycle state is covered below
Certain situations may arise where you don't want a CounterMeasure to execute unless someone gives the final go-ahead. This is supported via the Approval option.
When configuring your CounterMeasure, check the Require Approval option. Anywhere the incident is displayed and the beaker indicator is present, the beaker will glow orange, indicating approval is required for at least one CounterMeasure on the incident. Clicking it will show a modal where you can approve the appropriate CounterMeasures. CounterMeasures that require approval will stay queued until approved; they will not block other CounterMeasure and will not run until their configured timing (e.g., you can approve them early and they'll still run at the proper time).
CounterMeasures work in templates the same way they would on a regular instance, except for one difference - in templates, the list of available actions is a union of all the CounterMeasures used across your account. On instances, the only actions available are the ones that have been reported by the instance.
If you attempt to apply a template with a CounterMeasure to an instance that has not reported that CounterMeasure as present, it will not be added to the monitoring configuration for that metric.