We have our alerts get sent out on a timeline that is highly configurable. You as the user can set the time you'd like to first know about an outage, and then build in escalations to other people or teams.
Once an outage is detected, you can choose how quickly after an outage you'd like to be notified, who gets the notifications, and on which channels. From there, if the outage isn't acknowledged or resolved by the next trigger event, the alerts will escalate as you configure them.
You can also see which servers and server groups this alert timeline is attached to, and edit from this view.
Start by naming the timeline and giving it a description. You have the option to copy the timeline from another if you already have one created, this is helpful if you’re looking to build off another for special cases.
You’ll then see a page with a blank timeline, like the one below.
Get started by clicking +New Event at the top of the timeline. A window will appear that allows you to configure the details of that event.
The first thing to decide is how long after an outage starts do you want to be notified? We generally suggest two minutes as the minimum to avoid getting notified about short lived outages. We also suggest setting multiple events on timelines to act as escalation in case the first alert goes unnoticed or is forgotten.
Next, pick some combination of users, groups, on-call schedules and integrations to add to that event. Because you can define multiple events for escalations, you can pick the appropriate level of noise for each event. For individual users, you can pick between email, SMS, phone call, or mobile push notification.
The first alert on your timeline may look something like this.
At this point, you can decide if you’d like to build escalation into your alert timeline by adding another event with different alert types, perhaps something more aggressive than an email, like an SMS or voice call.
On the right side of the page, you can see the infrastructure tree. From here, you can select the servers you’d like to assign this Alert Timeline to.
Remember to click “save changes” at the top after you select the servers.
In the server view, you can see the alert timeline under the Instance Configuration section. You can also edit the alert timeline from here by clicking the Edit button at the top.
The Edit button will open a window in which one of the tabs is Alerts.
As you can see, there are options to set auxiliary timelines based on incident type, but that is certainly not mandatory.
The final step in getting set up is deciding how all this monitoring data will be displayed to you.