Alert Manager

Introduction

The Alert Manager adds simple incident workflows to Splunk. The general purpose is to provide a common app with dashboards in order to investigate fired alerts or notable events. It can be used with every Splunk alert and works as an extension on top of Splunk's built-in alerting mechanism.

  • Awareness of your current operational situation with the incident posture dashboard
  • Analyze root cause of incidents with only a few clicks
  • Review and adjust the urgency of incidents to improve operations scheduling
  • Dispatch incidents to the person in charge
  • Track and report incident workflow KPIs
  • Tag and categorize incidents

Features

  • Works as scripted alert action to catch enriched metadata of fired alerts and stores them in a configurable separate index
  • Each fired alert creates an incident
  • Configured incidents to run well-known scripted alert scripts
  • Reassign incidents manually or auto-assign them to specific users
  • Change incidents to another priority and status
  • Incidents can be configured to get auto-resolved when a new incident is created from the same alert
  • Incidents can be configured to get auto-resolved when the alert's ttl is reached
  • Send full customizable e-mail (Django template) notifications

Installation

Demo Data

For testing purposes, there is a separate App containing static demo data and demo alerts (SA-alert_manager_demo).

  • Static demo data adds pre-generated incidents with some workflow examples in order to see the KPI dashboards working
  • Demo alerts are configured to see some different live alert examples, like auto assign/resolve scenarios and support for realtime alerts

Refer to the app readme (https://github.com/simcen/SA-alert_manager_demo/blob/master/README.md) for installation and usage instructions.

Release Notes

New in Version 1.0

  • Finally, e-mail notifications arrived. Refer to the "E-Mail Settings" view
  • Bugfixes and enhancements

New in Version 0.10

  • Bugfixes and optimizations

New in Version 0.9

  • Lots of bugfixes
  • New KPI dashboard with sankey visualization
  • Full support to add/remove alert manager users
  • Improved app setup (check for index existence) and configuration (configure which user directories should be used)
  • Removed hardcoded index from searches

New in Version 0.8

  • Minor bugfixes & enhancements
  • Documentation improvements
  • App for demo data

New in Version 0.7

- Trend indicators for single values in incident posture dashboard

  • Full Windows support
  • Bugfixes

New in Version 0.6.1

  • New TA for distributed Splunk environment support
  • Improved incident settings (former alert settings) to work with non-global visible alerts
  • Added incident change events and KPI
  • Datamodel update to include incident changes

New in Version 0.5

  • Change incident: Apply new priority, move through workflow (change status, owner, add comment)
  • New event in alerts index on creating and changing incidents for compliancy
  • Run classical scripted alert scripts (passthrough mode with all arguments)
  • Per alert settings view finalized. Settings are now getting saved in a collection
  • Pass-trough mode for classical scripted alert scripts (alert_script)
  • Datamodel
  • Incident categorization & urgency calculation like in ES
  • Better CIM compliancy
  • Lot's of UI improvements

Roadmap

  • Custom incident handlers to extend the alert manager’s functionality
  • Incident enrichment with search data

Limitations and Known Issues

  • Default e-mail templates are not saved correctly in the KV store
    • Workaround: Go to E-Mail Settings and click "Save Templates" once. This step will copy the default template configuration to the KV store.
  • Trend indicators in the Incident Posture dashboard are fixed to the timerange earliest=-48h latest-24h
  • Currently it's not possible to disable e-mail notifications for Splunk built-in users
Share this project:

Updates