Quick Tip: Flush the System Security Services (SSS) Cache on Red Hat Enterprise Linux

After deleting a large number of service accounts from our FreeIPA infrastructure, I found they were still, supposedly anyway, resolvable by various hosts in our environment and that was preventing me from adding them locally. I suspected the issue was related to cacheing and referenced the SSS man page to figure out how to purge the its cache.

To flush a specific user entry from the SSS cache run the following command:

If that doesn’t do the trick run the following command to purge everything from the SSS cache:

Verify the desired user entry has been removed from the SSSD cache and is no longer present on the system with the following command:


Protect Pillar Data with GPG

Pillar is a great method of storing static data and making it available to Salt states, but the individual items that constitute Pillar data are only secure by virtue of the fact that they’re only made available to the minion or minions defined in the /srv/pillar/top.sls file. This approach is acceptable if you’re working in a fully trusted environment, but what happens if someone gains unauthorized access to your Salt Master and can access all data stored in Pillar? Such a scenario is a perfect use case for GPG renderers and we’re going to look at configuring them in this post.

A note on virtualization and randomness

If you are running a Salt Master on a VM (regardless of hypervisor) you may experience long delays during key generation due to the lack of physical hardware such as a keyboard and mouse which are commonly used to generate entropy. You can install the rngd-tools package using your distribution’s package manager and seed it with the following command to generate sufficient entropy to generate a strong key-pair:

Generating a Key-Pair

We first need to generate a key-pair so we have something to encrypt pillar with. First, though, we need to create a directory to hold the keys:

Follow the prompts providing information appropriate for your organization and do not provide a password for the key-pair.

(NOTE: If you encounter an error when executing the GPG command above please see this post.)

Export the Public Key

With key creation complete we can export the public key which will be used to encrypt Pillar items and import it on our local machine. Note that you can reimport the public key on the Salt-Master itself if you plan to encrypt sensitive Pillar data directly on it. Export the public key with the following command:

Import the Public Key on your Local Machine

Encrypt your sensitive pillar data

Now that we have all the pieces in place we can start encrypting sensitive information before placing it in Pillars. For example, to encrypt the string “super_secret_server_stuff” we would use the following command:

The above command will generate the following PGP “message”

The above message can be placed into a Pillar using the typical key-value pairing with one minor change, #!yaml:gpg must be added to the top of the file for Salt to recognize the Pillar contains GPG encrypted data. For example:

Test your Encrypted Pillar

Once you have successfully created an encrypted Pillar you should test to ensure the Pillar is properly decrypted by the Minion or Minions it’s visible to. First, sync the newly created Pillar with the following command:

Then query a minion the Pillar should be visible to and ensure the clear text equivalent of the cipher text is returned:

For instance, the clear text of version of the enciphered Pillar data we created earlier would look like:

That’s all there is to it! If the minion correctly responded with the clear text equivalent of the cipher text we created above you have successfully configured GPG renderers. Note that you can mix and match GPG encrypted Pillar items and plain text Pillar items in a single file as long as the #!yaml|gpg header is in place. If you have any questions or encounter any issues please don’t hesitate to leave a comment.


Quick Tip – Fix gpg: can’t connect to the agent: IPC connect call failed error

I was configuring GPG renderers on our Salt Master a few weeks ago and I ran into the following error while generating the PGP key (pair) that would be used to encrypt secrets before adding them their respective pillars:

Eventually, I determined this error was being caused by a GPG agent that was already running under CentOS 7, which my GPG command was unable to access. To fix this error kill the running agent with the following command:

Next, restart the agent with the following command:

You should now be able to re-run the GPG command you’re using to generate the key-pair and connect to the curses version of pinentry to input your passphrase. Next week we’ll expand on this error a bit and discuss the entire process of enabling GPG renderers in SaltStack.


Configure SaltStack Reactor

In a previous post I explained how to send a message to a specific Slack channel from a Salt state. Today, I want to tie that into another important concept of SaltStack: the Event Reactor (or Reactor for short). Reactor is a somewhat advanced featrue of Salt, but it opens up untold possibilities once you understand how it works. At a high level, Reactor responds, or reacts, to certain events received on the event bus. For example, let’s say you want to send a Slack message to a speciifc channel if a critical service fails to notify operations staff. That’s totally possible with Reactor. What if you want to send a Slack message then try to restart the failed service? Not a problem. In this post we’ll configure Reactor on the Salt master to listen for a specific event that indicates Apache (httpd) has failed then write a Reactor State to send a Slack message to our #alerts Slack channel and, finally, attempt to restart Apache.

Continue reading “Configure SaltStack Reactor”

Systems Administration Tenet 1: Automate, Automate, Automate!

Automation is defined as the technique, method, or system of operating or controlling a process by highly automatic means. Processes IT Professionals might seek to automate include: User Provisioning, Windows Update Installation (on client machines, of course), Backup Jobs and OS Deployments. Why would you want to automate your systems and processes?
Continue reading “Systems Administration Tenet 1: Automate, Automate, Automate!”