Configure SaltStack Reactor

In a previous post I explained how to send a message to a specific Slack channel from a Salt state. Today, I want to tie that into another important concept of SaltStack: the Event Reactor (or Reactor for short). Reactor is a somewhat advanced featrue of Salt, but it opens up untold possibilities once you understand how it works. At a high level, Reactor responds, or reacts, to certain events received on the event bus. For example, let’s say you want to send a Slack message to a speciifc channel if a critical service fails to notify operations staff. That’s totally possible with Reactor. What if you want to send a Slack message then try to restart the failed service? Not a problem. In this post we’ll configure Reactor on the Salt master to listen for a specific event that indicates Apache (httpd) has failed then write a Reactor State to send a Slack message to our #alerts Slack channel and, finally, attempt to restart Apache.

Step 1: Configure Reactor on the Salt master

Reactor is configured on the Salt master since the Salt master is what responds to events in the infrastructure. First, we need to create the /srv/reactor directory to hold our Reactor states with the following command:

Next, we need to configure the Salt master to run Ractor by opening the /etc/salt/aster configuration file and adding the following lines:

Once you have added those lines to the file restart the Salt master service with the following command:

Step 2: Create the Reactor State

We’ve configured the Salt master to listen for a specific event on the event bus and fire a Reactor State in response, but we havne’t actually created that state yet. Let’s do that now. To create the restart_httpd.sls state, create the file with your favorite text editor and enter the following:

Step 3: Write a local check to poll the status of Apache

Once you have enabled Reactor on the Salt master and configured it to listen for an event tagged with custom-events/failed-services/httpd, you need some way of actually generating that event. In this example we’ll run the following shell script from Cron every minute to evaluate the current status of Apache and, if necessary, fire an event tagged with custom-events/failed-services/httpd to the Salt mater which will trigger a a Reactor State run that will restart Apache on the Minion that generated the event.

Step 4: Bring it all together

With all the pieces in place, you can now either maually stop Apache on the target minion or run:

on the target Minion to simulate an Apache failure. Either approach should result in the Salt master locally executing the restart_httpd.sls on the Minion, restarting Apache. If either of the above methods doesn’t work as expected you can run the following command on the Salt master to view and debug the event bus.

That’s all there is to configuring a basic Reactor setup. The underlying power of Reactor is immense and the only limiting factor is your imagination. The real value of Reactor States is realized when you use them with other features of Salt such as Orchestration States. For example, let’s say you have a web tier and the load on a single node starts spiking. In that situation, a local check on the impacted Minion could determine the load average has exceeded an exceptable threshold and send an event to Reactor which could then fire a state that would hook into your server provisioning system and deploy one or more additional Minions into the web-tier to handle the increased load.

As always please feel free to comment below if you have any questions or would like to add to the discussion.

–J

4 thoughts on “Configure SaltStack Reactor”

    1. No, there wasn’t supposed to be HTML in the command. The plug-in I’m using for syntax highlighting likes to randomly insert HTML from time to time and I didn’t catch it. I’ve edited the post, but the correct command is:

      salt-run state.event pretty=True

      Thanks for the heads up.

Leave a Reply

Your email address will not be published. Required fields are marked *