Major Incident Management with Zenduty, Grafana, Slack and Zendesk

In the current fast-paced world, businesses are seeking methods to increase their efficiency and simplify their processes. But, there are times when teams are unaware of an issue at the initial stage, leading to a bad customer experience.

For example, you are a part of the Infrastructure team, where your primary responsibility is to check resources and notify when they reach their maximum capacity.

Let's say due to an anomalous traffic load, our resource CPU utilization goes above 90%. This will lead to latency and eventually a large percentage of our requests timing out. In a situation like this, a rapid incident response is required to quickly mitigate the impact to customers.

Here's when a monitoring and visualization dashboard like Grafana and an incident management platform like Zenduty can help you detect and mitigate these issues even before they impact your customers.

How do we setup our metrics and Grafana alerts?

You can set up a threshold Grafana alert for CPU usage or other black-box or white-box metrics like %5xx or response times, such as whenever CPU usage goes above 90%, an alert is fired. By informing you of the state of the resources, Grafana alerting enables you to maintain control over the situation.

If any critical Grafana metric exceeds a certain threshold, Zenduty will initiate an incident and implement the escalation protocol, notifying the appropriate engineers through their designated communication channels.

To achieve this, all you need is Grafana integration with Zenduty.

How Grafana integration works with Zenduty?

If you are wondering how to integrate Grafana with Zenduty, here are the steps you can follow:

In Zenduty:

• On Zenduty, navigate to Teams and select the team to which you want to add the new Grafana integration.
• Then go to Services and select the required service.
• Select Add New Integration from the Integrations menu. Choose the programme Grafana V8 from the dropdown menu and assign it a name.
• Copy the created Webhook URL by selecting Configure from your integrations.

In Grafana V8:

• Log in to Grafana. Head over to Contact Points. Add a new contact point and choose Webhook as the contact point type.
• Insert the webhook URL you copied from Zenduty. Copy the Webhook URL, paste it, and then select Save.

Go to your Grafana Dashboard and access the Alerting tab. Select Create alert. Add Zenduty as a notification option under Send to in the alert setup. Save the graph after adding a pertinent statement to the alert.

This ends the Grafana integration process with Zenduty. Now, Zenduty will create incidents automatically whenever Grafana alerts are created.

But now the question is how are incidents escalated to Slack.

How does Zenduty escalates incidents to Slack?

For this, you need to set up your notification rules to include your Slack handle and specify when you wish to receive alerts on Slack.

Slack Integration with Zenduty

With Slack integration, you can ask Zenduty to alert you using the "send notification personally" option whenever an incident is assigned to you. Notifications sent through phone call or SMS can be replaced with this or used in addition to it.

To achieve this, "Connect your Slack account."

After completing this, add Slack in your notification rules.

Once configured, Slack will automatically get all Zenduty alerts.

According to the notification rules you set up, you will receive personal messages from alerts about incidents that have been allocated to you.

Then, within Slack, you may acknowledge, address, or assign Zenduty incidents.

The Slack channel will get all notifications regarding a service that has been integrated with it. The incident can then be acknowledged, resolved, or assigned within Slack by any team member.

What happens when an incident is assigned through Slack?

Once the incident is assigned to the concerned person on Slack, a bunch of action buttons are available.