Cron with Saltstack

I was wanting to only be emailed changes, errors and other oddities from within saltstack.

salt * state.highstate | sed -n '/domain.suffix/{:start /Total states run:/!{N;b start};/Total states run:     [^0]/p}'

Replace domain.suffix with your domain suffix to get this going.
-n Don't print every line
domain.suffix - Mark this line as the start of the block.
Total states run: - End of the block
Total states run:     [^0] - Match all that have not 0 changes

Now that we have that let's cron that and it'll send emails about what has changed.

Chrooting users with their own root jails for different dependencies

Issue:

Sometimes we get strange dependency conflicts from research software that is used for research by some of our staff. Due to lack of infinite resources we share high powered machines. To get around this the easiest way is to give everyone their own copy of a machine and let them do as they need with it.

Dependencies:

• debootstrap (Create jail)
• fakeroot (give the ability to run
• fakechroot (give the ability to normal users to use a chroot)

The setup (As due to requiring privlige to run debootstrap):

Setting up the jail for the user

1. export username=nonprivuser
2. sudo debootstrap --variant=buildd trusty /home/$username/trustychroot http://archive.ubuntu.com/ubuntu/
3. sudo cp /etc/resolv.conf /home/$username/trustychroot/etc/resolv.conf
4. sudo cp /etc/apt/sources.list /home/$username/trustychroot/etc/apt/
5. (This is not usually suggested due to root concerns but that is a non issue if the user already is an admin on the machine and have fakeroot for their whole machine, should you want to do something differently please do not run this!) sudo chown -R $username:$username $username

Running the jail

This should be done by the normal user:

physical host$ fakeroot fakechroot chroot ~/trustychroot /bin/bash

Making things a bit easier to launch

I am lazy and would prefer to type a command that is a bit more fun than fakeroot fakechroot chroot ~/trustychroot /bin/bash.

I have filled /usr/local/bin/mytrustychroot with a small shell script to make this easier:

#!/bin/sh
/usr/bin/fakeroot /usr/bin/fakechroot /usr/sbin/chroot ~/trustychroot /bin/bash

Auditing

Switch to use and run sleep in the chroot as root, trying to modify the routing table does not work:

$ mytrustyjail
chroot # ip route add 1.1.1.1/32 via 10.10.13.2
RTNETLINK answers: Operation not permitted

Running sleep and logging in as another user on the host system does not work:

RTNETLINK answers: Operation not permitted
chroot # sleep 600

physical host$ sudo ps auxww | grep sleep
nonprivuser  7155  0.0  0.0  12456   696 pts/1    S+   14:17   0:00 sleep 300

Try to break out of a jail via symlink:

physical host$ ln -s ../../../../a trustychroot/etc/test1
physical host$ ln -s ../../../a trustychroot/etc/test2
physical host$ ln -s ../../a trustychroot/etc/test3
chroot # echo "test1" > test1
bash: test1: Permission denied
chroot # echo "test2" > test2
bash: test2: Permission denied
chroot # echo "test3" > test3

We were able to write outside of the chroot by creating a symlink outside of the chroot but as we are using fakeroot and the process is running in the context of nonprivuser this is not an issue as we are able to write anywhere nonprivuser has permission to write to, IN YOUR ENVIRONMENT THIS MAY BE DIFFERENT!

Try to install something like zsh:

chroot # apt-get install zsh
... it works ...
chroot #

Concerns for the long term

Updates

These are going to be running one application with custom code that our users are writing as well as some research packages that they want specific versions of, no services.

Escaping jail?

Is it fine for someone to jump out to the host system with root privlege should you chown the jail to them? This becomes more of a talk to the person problem and if that doesn't work, HR problem for me should something happen. Many of the people on this machine currently have sudo access so there is not much difference for me but running as a standard user is a much better idea.

An easier way for public facing things?

Freebsd happens to have a very nice jail system using ezjail, it's something like this and not have to worry about things that I have not tested yet:

ezjail-admin create -x jail1 'em0|10.0.0.101'
ezjail-admin console jail1
  # INSTALL MANAGMENT SOFTWARE HERE!

From here it is much easier to point some configuration managment software at the jail and have that do all the heavy lifting.

Apache virtual hosts in saltstack

  I'll try to explain our environment a bit, we run a few machines that all require different virtual hosts and have different requiremnts. I try to keep these seperated by hostname as much as I can. Seperating these in saltstack took a while to figure out so here are  my notes on it.

  Grains are applied on the fly based on what is in the apache24.ssl template (this will not update config or blow away files). I prefer to listen on test and prod with the same configurations and just do redirection based on IP addresses since everything should work the same across all.

How things are getting assigned below

$ salt 'hostname*' grains.get "ssl_host"

hostname-dev.subdomain.example.com:
    - hostname1.example.com
    - hostname2.example.com    - hostname1-test.example.com
    - hostname2-test.example.com
    - hostname1-dev.example.com
    - hostname2-dev.example.com

hostname-test.subdomain.example.com:
    - hostname1.example.com
    - hostname2.example.com
    - hostname1-test.example.com
    - hostname2-test.example.com
    - hostname1-dev.example.com
    - hostname2-dev.example.com

Common Configuration files

apache24.ssl:
  file.managed:
    - name: /usr/local/etc/apache24/Includes/ssl.conf
    - source: salt://apache24/usr/local/etc/apache24/Includes/ssl.conf
    - user: root
    - group: wheel
    - mode: 400

/usr/local/etc/apache24/ssl:
  file:
    - directory
    - user: root
    - group: wheel
    - mode: 500

/usr/local/etc/apache24/ssl/incommon.chain:
  file.managed:
    - name: /usr/local/etc/apache24/ssl/incommon.chain
    - source: salt://apache24/usr/local/etc/apache24/ssl/incommon.chain
    - user: root
    - group: wheel
    - mode: 400

Host to set ssl settings mappings

{% if 'hostname-dev' in grains.host %}
ssl_host:
  grains.present:
    - value: ['hostname-dev.example.com','hostname2-test.example.com']
{% endif %}
{% if 'hostname-test' in grains.host %}
ssl_host:
  grains.present:
    - value: ['hostname-test.example.com','hostname-test2.example.com']
{% endif %}

Placing the configuration files based on what is set above

# Put custom configuration inside of apache24/usr/local/etc/apache24/vhost/template.jinja2 based on a per {{ i }} setting
# context i should pass i into jinja
{% set domainlist = salt['grains.get']('ssl_host','') %}
{% for i in domainlist %}
/usr/local/etc/apache24/ssl/{{ i }}.key:
  file.managed:
    - name: /usr/local/etc/apache24/ssl/{{ i }}.key
    - source: salt://apache24/usr/local/etc/apache24/ssl/{{ i }}.key
    - user: root
    - group: wheel
    - mode: 400
/usr/local/etc/apache24/ssl/{{ i }}.crt:
  file.managed:
    - name: /usr/local/etc/apache24/ssl/{{ i }}.crt
    - source: salt://apache24/usr/local/etc/apache24/ssl/{{ i }}.crt
    - user: root
    - group: wheel
    - mode: 400
/usr/local/etc/apache24/vhost/{{ i }}:
  file.managed:
    - name: /usr/local/etc/apache24/vhost/{{ i }}
    - source: salt://apache24/usr/local/etc/apache24/vhost/{{ i }}.jinja2
    - template: jinja

    - context: {
      i: "{{ i }}" }
    - user: root
    - group: wheel
    - mode: 400
/usr/local/etc/apache24/vhost/{{ i }}.ssl:
  file.managed:
    - name: /usr/local/etc/apache24/vhost/{{ i }}.ssl
    - source: salt://apache24/usr/local/etc/apache24/vhost/{{ i }}.ssl.jinja2
    - template: jinja
    - context: {
      i: "{{ i }}" }
    - user: root
    - group: wheel
    - mode: 400
/usr/local/etc/apache24/vhost/{{ i }}.conf:
  file.managed:
    - name: /usr/local/etc/apache24/vhost/template.conf
    - source: salt://apache24/usr/local/etc/apache24/vhost/template.conf.jinja2
    - template: jinja
    - context: {
      i: "{{ i }}" }
    - user: root
    - group: wheel
    - mode: 400
{% endfor %}

Other config files

usr/local/etc/apache24/vhost/template.jinja2

ServerName {{ i }}

{% if 'hostname2' in {{ i }} %}
  DocumentRoot /usr/local/www
{% elif %}
  DocumentRoot /usr/local/www
{% endif %}

# Per vhost configuration options
{% if 'hostname1' in {{ i }} %}
  RewriteEngine On
  RewriteCond %{REQUEST_URI} !/Shibboleth.sso
  RewriteRule (.*) https://%{HTTP_HOST}/protectedurl [R=301]
{% endif %}

usr/local/etc/apache24/vhost/template.jinja2

<VirtualHost *:80>
Include etc/apache24/vhost/{{ i }}
</VirtualHost>
<VirtualHost *:443>
Include etc/apache24/vhost/{{ i }}.ssl
Include etc/apache24/vhost/{{ i }}
</VirtualHost>

usr/local/etc/apache24/vhost/template.ssl.jinja2

Include etc/apache24/vhost/defaults.ssl
SSLCertificateFile etc/apache24/ssl/{{ i }}.crt
SSLCertificateKeyFile etc/apache24/ssl/{{ i }}.key
SSLCertificateChainFile etc/apache24/ssl/incommon.chain