Salt-Event-Hub - Saltstack in RESTful way

August 18, 2014 • hack

Salt is an integral part of our infrastructure. We use it for provisioning of our servers and deployment of our applications. Our CI server informs the salt master about new builds which are then installed to a development system and our developers can tell the salt master to deploy an application into production. For those tasks we are utilizing the salt event system. Minions can inform the master about problems or propagate information on events such as a successful build. Upon that the salt master performs defined actions on the infrastructure such as an orchestrated upgrade of an application. The event system of salt is powerful and a useful abstraction that encapsulates the knowledge about your infrastructure close to salt and not in a client application. Unfortunately, events can currently only be triggered through minions. That means that the service that is to interact with the salt event system has to be accompanied by a local salt minion. This is not always the case if you think e.g. about developer machines or hosted services. That’s why we created the salt-event-hub, a simple RESTful HTTP server that would allow us to use a widely supported interface to interact with our infrastructure.

Server

We have created the server using Flask. Flask is a micro webdevelopment framework with a great community and dozens of free resources on the internet. Also, the choice for using a python framework came naturally, because there are only python bindings for salt’s event system.

The server is as simple as it gets. There is only one general route to trigger events: -> POST /<event>/trigger

The following method handles this route:

@app.route('/<event>/trigger', methods=['POST'])
def trigger(event):
    logger.info(str(opts))
    from salt.utils.event import SaltEvent

    authToken = request.headers.get("X-AUTH-TOKEN", "")
    if authToken != opts['x_auth_token']:
      abort(401)

    payload = request.get_json()

    sock_dir = '/var/run/salt/master'
    event_interface = SaltEvent('master', sock_dir)
    event_interface.fire_event(payload, event)

    return "OK"

As you can see the method expects some values:

  • X-AUTH-TOKEN is an authentication token. We kept authentication simple and just implemented a check for the client using a valid authentication token that can be configured.
  • The data attribute of the json body of the request is used as a payload to provide additional information to the event.

As the http server is using the master’s socket it should run on the master’s machine as the same user as the master. The server could also easily be altered to run on a minion machine.

Client

This is an exemplary interaction with the server using curl. As you can see we’ve already set authentication token, content-type header, data and correct url.

% curl -H "X-AUTH-TOKEN: \
eyJhbGciOiJIUzI1NiIsImV4cCI6MTM4NTY2OTY1NSwiaWF0IjoxMzg1NjY5MDU1fQ" \
-H "Content-Type: application/json" \
-d '{"source":"serviceX", "project":"projectX", "build_number":23}' \
https://salt-event-hub.example.com/deploy/trigger

=> 200 OK
or
=> 401 Wrong X-AUTH-TOKEN
or
=> 400 Bad Request

Hubot

Hubot is an extensible chatbot created at Github and integrates well with different company chats such as hipchat or slack. At scalable minds we use Hubot for checking the daily menu of nearby cafeterias and managing our server infrastructure.

Basically, Hubot listens to all the messages in a chat and takes action upon specified ones. To extend Hubot’s functionality your have to define a new regex and the action that should follow.

In the following we will use hubot as a client to the salt-event-hub.

module.exports = (robot) ->
robot.respond new RegExp("(deploy|remove) ([a-zA-Z]) ([0-9]+)$", "i"), (msg) ->
    event = msg.match[1]
    project = msg.match[2]
    build_number = msg.match[3]
    build_number = msg.match[3]
    data = {'source' : project, 'project': project, 'branch': branch, 'mode': mode, 'build_number': build_number }
    triggerEvent(msg, event, data)

triggerEvent is the function that performs the HTTPS POST request to the server.

triggerEvent = (msg, event, data) ->
    url = "https://salt-event-hub.example.com/#{event}/trigger"
    superagent
        .post(url)
        .ca(caCert)
        .set('X-AUTH-TOKEN', auth)
        .set('Content-type', 'application/json')
        .send(data)
        .end((err, res) ->
          msg.send(res))

caCert is our own certficate authority. We need to include it, because we are using a self signed certificate on the server side. You can either include the certificate as a string or read it from disk: caCert = require("fs").readFileSync(PathToSomeCa, "utf8")

auth is the required authentication token

With this developers can say hubot deploy oxalis 50 in our chat if they want to roll out version 50 of our tool oxalis. Hubot will then call the salt-event-hub which then triggers the deploy event in the salt system.

Salt reactor

To cover the full use case, here is a short excurse into reactor formulas which contain the code that runs upon the occurrence of events. Reactor formulas give you access to the LocalRunner subsystem which is the same subsystem you are using when using the salt command on the shell. Coming from the former section we want to deploy the version 50 of oxalis. We would use the following commands on the command line to first update the package manager and then install the application:

salt -G "role:oxalis" pkg.refresh_db
salt -G "role:oxalis" pkg.install oxalis version="50"

We can pack those two commands in a reactor formula that will be executed upon the “deploy” event being triggered:

#deploy.sls
{% set target = "role:%s" % data['project'] %}
{% set expr_form = "grain" %}

update-repos:
  cmd.pkg.refresh_db:
    - tgt: {{target}}
    - expr_form: {{expr_form}}

install-package:
  cmd.pkg.install:
    - tgt: {{target}}
    - expr_form: {{expr_form}}
    - kwarg:
        name:  {{data['project']}}
        version: {{data['build_number']}}

You can see that within reactor formulas the event payload can be found in the top level of the data dictionary. If the event was triggered on the master socket, it will contain the payload on the toplevel. On the other hand, if it was triggered by a minion, the event payload can be found at data[‘data’].

Finally you have to define which events should trigger which reactor formulas. You can do that in the master.conf in the reactor section as follows:

reactor:
  - 'deploy':
    - /srv/reactor/deploy.sls

This is how you can have fun with an infrastructure that can respond to events outside it’s own reach.

Contributions

The project is available under MIT licence, go checkout the code, if you’re interested.

by Thomas Werkmeister , Mateusz Maciaszek


Related posts