Shell's TechBlabber

…ShelLuser blogs about stuff ;)

Using Git for Unix systems administration

Git, the version control system (“VCS”) developed by Linus Torvalds, is most commonly used for development purposes. You can keep track of your sources at any given time, branch out to test new ideas while your main source tree always remains untouched if need be. And of course you can revert changes (also temporarily) at any given time. And that’s not even mentioning the options to store all your work on a remote (central) repository in order to share it with others.

But Git can do so much more. In this post I’ll show you how you can use it to take Unix systems administration to complete new heights. If you ever wondered about some of the advantages which Git has over traditional systems such as Subversion then read on: I’ll be sharing some major examples.

The problem

Last year I was using the Bind DNS server but wasn’t really happy with it. I wanted to set up an environment where two DNS servers would be able to host the same domain but without the traditional master –> slave construction where a domain could expire over time. Because I have a decently working replication scheme between my two PostgreSQL (database) servers I figured that it might be doable to use a DNS server which used a SQL server as backend. I then discovered PowerDNS, tested it and eventually migrated my servers. So far, so good.

Then at the end of last year PowerDNS applied some very drastic changes to their project which made the whole admiinstrative part even more tedious than Bind ever was. That was no good for me! So a few months ago I reverted back to Bind (and also discovered a solution to my previous problem: dynamic zones).

Unfortunately many months had passed since Bind got removed so there was no trace left of its configuration files in my backups, I do keep a decent retention but not that long. And that got me thinking…

Using VCS for system config files?

The problem is simple: in your traditional backup scheme you have a limited retention, you can only go back for so far. Of course you could work around that by creating manual backups (an archive for example) and then storing that somewhere on your system but is that really useful? And once you start to create a collection of archives won’t you run into the hassle of keeping track of which is what?

If there only were a system which could store your configuration files (which are mostly plain ASCII files anyway), keep track of any changes and do all this in an optimal way. One which doesn’t take up too much resources. And guess what? There is! It’s called a version control system, such as Subversion, CVS or Git.

Basically what we’re going to do is set up a backup system (the VCS) which is then backed up by our traditional backup system. The main advantage should be obvious: although our regular backup system may not have a massively high retention, the VCS environment will. And it won’t be too resource hungry either.

The only problem with Subversion and CVS is that those use a single (central) repository. And you can’t really add seperate directory structures as part of such a repository. That’s simply not the way it works and so you’d have to maintain several repositories next to each other; one for each “configuration project”. And the problem with that is that a Subversion repository isn’t exactly easy to set up nor maintain. Ever tried to rename a file for example?

Git to the rescue!

Git can boldly go where no VCS has gone before, and that’s not merely a small rip-off from Star Trek’s famous line. It honestly has several features which make this the perfect tool for the job and which also gives it a major advantage over tools such as Subversion.

For example: unlike the others Git can maintain a local repository. So worst case scenario is simply creating that and using it. Of course linked to a central repository (‘set-upstream’) where you’ll eventually push your updates to, so that all those changes will be included in your traditional backups.

Of course the problem with that is that your system would get flooded with .git directories which contain the local repositories. Maybe not the worst problem but I don’t really like the idea, also because it’s somewhat of a waste of resources as well; multiple repositories which all use the same kind of files. But there’s even a solution for that…

How to set it all up

Overview

We’re going to set up one (shared) repository. This will contain several branches where each branch reflects on a specific configuration element. For example origin/squid would contain all the configuration files from the Squid proxy server, origin/bind would have the Bind DNS server configuration, and so on.

We’ll control this repository from a local copy (which I prefer storing in /root/etc). We’re also going to set up separate worktrees so that all the local repositories will eventually become part of this one single repository project.

Sounds complicated? Not to worry, that’s what this blog is for. Just follow my examples and you should have your own setup in no time.

A few important disclaimers though:

  • FreeBSD: My setup involves around FreeBSD so if you notice that some given pathnames don’t correspond with your system just change those accordingly. For example: /usr/local/etc/namedb vs. /etc/bind or maybe /opt/etc/bind.
  • Git familiarity: Although I’ll explain the whole process in proper details (follow my lead and I’ll guide you towards a working setup) I do expect that you already have some basic understanding about Git. My explanation will only cover a basic setup.
    • For example: I’ll be using a directory (/var/db/config.git) as the main repository. This will work fine for the local system but if you want to expand this setup across different servers you’ll obviously need another, more advanced, configuration scheme.
  • Security: We’re going to set up one central repository which contains most of your system configuration. So some care is definitely advised.

Let’s start… with documentation

First we’re going to create our main secured repository:

  • mkdir etc && chmod 700 etc
  • cd etc && git init

And the first file we’re going to add will be a README file. Trust me: you’re going to need this later on. Maybe not now, not within a few weeks time, but after a few months where you have mostly used basic commands such as ‘git add’, ‘git commit’ and ‘git push’ you could easily have forgotten about all the other important steps. This README file will ensure that as soon as you clone this project onto another (new?) server you’ll know exactly what to do to set it up.

What I prefer is to maintain a list of all the branches (I’ll explain this later) as well as instructions on how to set up a new branch (so: how to add a new configuration to our setup). So a short version of this blog post.

For now lets just add:

This is a configuration project; it contains all the major configuration files for the server.

The main branches are:

squid – Contains the configuration for the Squid proxy server.”.

After you’ve done that add this file to the repository:

$ git add . && git commit -am "First commit"
[master (root-commit) dfd1ca1] First commit
1 file changed, 6 insertions(+)
create mode 100644 README

This sets up our local repository from which we’ll control everything else. Onto the next part…

Setting up the central (shared) repository

Although our main repository is the place from which we’ll control the rest, it’s actually not the most important component of this setup. In fact: it wouldn’t even matter too much if you didn’t keep any backups of it (although I would definitely suggest that you do, because setting the whole thing up again does take some time and effort).

The one which does matter (and which you should definitely include in your backup scheme!) is the central repository which we’ll be using to push our changes onto.

We’ll start by creating a bare clone of our current repository:

$ git clone –bare . ../config.git
Cloning into bare repository ‘../config.git’…
done.

Next we’re going to move ‘config.git’ to a specific system location: /var/db. So: /var/db/config.git. Make sure that this directory is properly protected! I prefer keeping it with root:wheel as owner/group and 750 as the permission bits. Just make sure that this won’t be accessible by any random users, otherwise you could create some major problems for yourself.

Once that’s out of the way we’ll set up an ‘alias’ within Git’s configuration so that it’ll be easier on us to access this repository:

  • git remote add -m repo repo /var/db/config.git

This allows us to access our central repository as ‘repo’. Next we’re going to ‘link’ our repository with the remote:

  • git push -u repo master

You can check that everything is properly set up by checking the status:

$ git status
On branch master
Your branch is up to date with ‘repo/master’.

nothing to commit, working tree clean

As you can see our repository is now tracking repo/master, which means so much that whenever there’s a change there we’ll know about it. Also: whenever we make changes to our local repository then this will also remind us to push those changes to the main repository.

There’s just one more thing we need to do… Right now ‘repo’ is only known within our current repository:

$ git config –local -l
core.repositoryformatversion=0
core.filemode=true
core.bare=false
remote.repo.url=/var/db/config.git
remote.repo.fetch=+refs/heads/*:refs/remotes/repo/*
branch.master.remote=repo
branch.master.merge=refs/heads/master

But this won’t help us when we’re going to set up the other repositories later on. Therefor we need to move the ‘remote’ configuration entries from our local configuration one step up into the Git hierarchy. So: out of the local configuration (which is only known within the current repository) into either the global or system configuration. The global configuration is for the current user whereas the system configuration is for the entire system (all users). Because this is all about system configuration I prefer moving this into the system config:

  • git config –system –add remote.repo.url `git config –local –get remote.repo.url`
  • git config –system –add remote.repo.fetch `git config –local –get remote.repo.fetch`
  • git config –local –remove-section remote.repo

Of course a quicker way would be to manually edit the configuration: git config –global -e. But you’ll have to experiment with that yourself. You could also have started with setting this up, the reason I didn’t is to make this tutorial easier to follow.

Summing up

  • We now have a local repository (/root/etc) which contains a README file.
  • We have a central, shared, repository (/var/db/config.git) which is ‘linked’ to our local repository.
  • And finally: the central repository is globally known as ‘repo’.

Setting up a ‘config repo’

Now it’s time to actually add a configuration to track, for this example I’ll be using /usr/local/etc/squid.

Start by creating a new local repository and add the files we want to track:

  • cd /usr/local/etc/squid
  • git init
  • git add *.conf
  • git commit -am "initial commit”

This will add all the config files to the local repository. If you run ‘git status’ you’ll probably get a mention of several untracked files, we’re going to deal with those at a later time. Or… if you prefer you could also simply ignore them of course.

Now that we have a local repository set up it’s time to make this part of our central repository. We do that by performing some “git magic”:

  • git push -u repo +HEAD:refs/heads/squid

$ git push -u repo +HEAD:refs/heads/squid
Counting objects: 6, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 4.38 KiB | 1.46 MiB/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To /var/db/config.git
* [new branch]      HEAD -> squid
Branch ‘master’ set up to track remote branch ‘squid’ from ‘repo’.

So what is happening here?

This is an example where Git truly outperforms the rest of the VCS bunch: we’ve basically told Git to push our local repository directly onto the central ‘repo’ but as a branch of that repository. So it’s basically a separate ‘section’ within the repository and it only contains the Squid configuration files.

Let’s check if this worked. Go back to your local repository (/root/etc) and issue:

  • git fetch

You should now see mention of a new branch called repo/squid. Let’s check it out:

  • git checkout squid

Now your README file will be replaced with several conf files, all related to Squid. This is also why I prefer to maintain that README file in the first place because will you still remember how to switch back after a few months time? Or let alone that push command I mentioned earlier…

Anyway, now that we know things worked lets switch back:

  • git checkout master

Now it’s time for the final step. Although the current situation works perfectly it’s not fully what we’re after. Basically we now have to maintain 2 (or 3?) separate repositories which could become a drag over time.

Summing up

  • We now have a (local) ‘config repository’ which contains all the Squid config files.
  • We pushed the local repository directly onto the central repository (‘repo’) as a separate branch, now called repo/squid.
  • To check that everything worked we updated our main local repository (/root/etc) by ‘fetch’ ‘ing all the remote changes, after which we checked out the squid branch.

Adding the ‘squid repo’ to the main project

Note that this step is purely optional. If you want to you could stop here and continue to use this setup by adding more config repositories and then pushing those as new branches onto the central repo. You can then use Git to add, commit and push all local changes in order to maintain your config history.

The main reason why I prefer this extra step is because it keeps the whole setup much tidier and it will ‘connect’ all the (now) local repositories such as /usr/local/etc/squid with our main config repository (/root/etc). This has some specific advantages which I’ll explain later on.

First (temporarily) rename /usr/local/etc/squid into something else. squid.bck for example:

  • mv /usr/local/etc/squid /usr/local/etc/squid.bck

…then go into the main repository and issue this command:

  • git worktree add /usr/local/etc/squid repo/squid

We now cloned the repo/squid branch into /usr/local/etc/squid and made it a part (“worktree”) of our main repository (/root/etc). You can check by issuing “git worktree list”, don’t worry about the ‘detached HEAD’ comment for now.

Next go into /usr/local/etc/squid and issue this command:

  • git checkout squid

You should see something like this:

$ git checkout squid
Switched to branch ‘squid’
Your branch is up to date with ‘repo/squid’.

Now comes the fun part. As you can see this directory doesn’t contain a .git directory but a .git file. It’s merely a text file so you can easily view it, it points to the actual Git repository which is now located in /root/etc/.git/worktrees/squid.

So now we’re going to replace the previous local repository with this one:

  • rm -rf /usr/local/etc/squid.bck/.git
  • mv /usr/local/etc/squid/.git /usr/local/etc/squid.bck
  • rm -rf /usr/local/etc/squid && mv /usr/local/etc/squid.bck /usr/local/etc/squid

And we’re done.

Now our local repository (/root/etc) is truly the central repository because /usr/local/etc/squid has become part of this project as a worktree. You can see this for yourself by issueing:

  • git worktree list

… from the local repository.

Every change in /usr/local/etc/squid which you make and commit will first get added to the local repository (/root/etc/.git/worktrees/squid). After that you can “git push” it after which it’ll get added to the central repository in /var/db/config.git where it eventually should also get included in your backups.

Now, there are some specific aspects which we need to keep in mind here:

  • Because all “config repos” will become part of our main project (/root/etc) they will also all share the same specific settings such as tags, stashes and ignore settings (.git/info/exclude). This can be both a pro and a con; you can’t use the same tag twice for example, which could be a con. Once you set up an ignore (for example *.sample in the exclude file) then this will apply to all your ‘config repos’, which can make things much easier over time.
  • You will probably end up with some dangling objects and/or commits; see ‘git fsck’. This shouldn’t be too much of a problem, also see the ‘git gc’ command.
  • Keep in mind that ‘git worktree’ is still a bit of a work in progress. Using it like this with multiple checkouts is not official supported, but it’s my experience that it works like a charm nonetheless.
  • Don’t try to use submodules within this context, that’s not fully supported as of yet.

Summing up

  • The ‘config repo’ (in this case /usr/local/etc/squid) has now become part of our main local repository (in /root/etc).
    • You can always check using: ‘git worktree list’ from within the local repository.
  • The actual Git config repositories are now located in /root/etc/.git/worktrees.
  • Because all repositories are part of the main project (/root/etc) they share settings such are global ignores (you can set those in .git/info/exclude).

But what if you can’t rename the config directory?

In my local setup I’ve also included locations such as /etc and /usr/local/etc which are obviously not very well suited for “just” temporarily renaming. In those situations you should start working with a temporary copy and then rename the worktree instead.

So instead of renaming the main directory you’d eventually rename the Git configuration.

Start by setting up a worktree in a temporary location:

  • git worktree add /usr/local/etc.new repo/etc

Then follow the same steps as above; so ‘git checkout etc’ which will “link” the directory with our central repository and after that move the .git file from /usr/local/etc.new into /usr/local/etc and remove the temporary directory afterwards.

Now we need to manually edit the worktree location because according to Git this is still pointing to /usr/local/etc.new. So:

  • cd /root/etc/.git/worktrees/etc.new
  • Edit ‘gitdir’: change /usr/local/etc.new/.git into /usr/local/etc/.git

Optionally you can also rename the ‘etc.new’ directory within the ‘.git/worktrees’ directory. I strongly recommend doing this if you plan on adding both /etc and /usr/local/etc for clarification sake. For example, in my setup I’m using both .git/worktrees/etc.local and .git/worktrees/etc.

But if you decide to do this then always remember that you also need to edit the .git file to point to the new repository location. So: /usr/local/etc/.git, this would contain /root/etc/.git/worktrees/etc.new which you would then need to change to /root/etc/.git/worktrees/etc.local.

After that you should be fully set.

So what’s the advantage?

Let’s look at a real life example; that of my own server:

peter@unicron:~/etc# git worktree list
/root/etc                4bebced [master]
/etc                     c628439 [etc]
/opt/ssl                 508c968 [ssl]
/root/kernels            d93bd9e [kernel]
/usr/local/etc           6ef19bc [etc.local]
/usr/local/etc/Unreal    35142ea [unreal]
/usr/local/etc/apache24  104da3d [apache]
/usr/local/etc/infobot   8712c52 [infobot]
/usr/local/etc/namedb    cd71d38 [bind]
/usr/local/etc/squid     f751d08 [squid]
/usr/local/etc/tripwire  79a67b0 [tripwire]
/usr/local/etc/vpn       f51ed0c [openvpn]
/usr/local/pgsql/data    f01729a [pgsql]

The first advantage of using worktrees is to have global access to all repositories. If I want to check my latest changes on the Bind DNS server then all I have to use (within the local repo) is:

  • git log bind

…after which Git will show me the log of the ‘Bind branch’. Even if I haven’t fetched this branch yet; this works because the branch has become a part of my main project (/ local repository). If I wouldn’t have done the “worktree step” I’d have to use this instead:

  • git checkout bind
  • git log

Note: although I can now use the ‘git log bind’ command from here on as well it would only show me the changes which I have already retrieved. So to make sure that my local branch is up to date with the remote I’d always have to start with:

  • git checkout bind
  • git pull
  • git log

But… that’s not needed anymore.

Some other specific advantages:

Central storage

Although we’re still using 2 repositories (one local and one remote) everything is stored in a centralized location. The local repositories are all part of the main project (/root/etc) and are eventually all pushed onto the central remote repository (in my own setup this is located in /var/git/config.git).

Central configuration

Now, this can both be a pro and a con but you should mostly gain advantages here. For example: it’s not uncommon for *.sample files to be a part of a configuration, same goes for *.bak, *.db or *.dist files. Because all repositories share the same configuration you only have to define those ignore entries once (assuming you’re using .git/info/exclude).

Full retention

This is what it’s all about: no matter how long ago you’ll always have access to your full history of systems configuration. Basically you’re maintaining a backup (the Git project) within a backup (your regular backup setup), and thanks to the nature of the VCS system it’ll contain the full history.

You’ll know exactly when you set something up, what you set up, and assuming that you documented your changes you’ll also know why.

For example, it’s roughly one month ago since I really started using this setup:

commit 053cc350f5c03cb43a4562a1486f77242a007d27
Author: Unicron Sysop <root@localhost>
Date:   Sun Mar 4 12:18:30 2018 +0100

    Updated config.git security; limited access to LAN/VPN only.

    Signed-off-by: Unicron Sysop <root@localhost>

commit bc559c2aec460da2d9aa85588d140e7a12321721
Author: Unicron Sysop <root@localhost>
Date:   Sun Mar 4 07:21:45 2018 +0100

    Set up authentication for the Git configuration repository.

    Signed-off-by: Unicron Sysop root@localhost

This is the log of my Apache repository. For local use (LAN) I’m relying on the Git daemon and everything else within my network (WAN) uses the password protected setup which is provided by Apache.

If I wanted to know what file(s) I updated when I performed the security update all I have to do is:

peter@unicron:~/etc# git show 053cc35
commit 053cc350f5c03cb43a4562a1486f77242a007d27
Author: Unicron Sysop <root@localhost>
Date:   Sun Mar 4 12:18:30 2018 +0100

    Updated config.git security; limited access to LAN/VPN only.

    Signed-off-by: Unicron Sysop <root@localhost>

diff –git a/Includes/intranet.conf b/Includes/intranet.conf
index f5d1b2f..48074ad 100644
— a/Includes/intranet.conf
+++ b/Includes/intranet.conf
@@ -96,6 +96,7 @@
                AuthBasicProvider file
                AuthUserFile "*"
                Require user *
+               Require ip 10.0.0.0/23
        </Location>
</VirtualHost>

So intranet.conf it is, seems I also only added one single line.

Remote accessibility

My network consists of a WAN (basically a VPN) and several servers. If I’m not at my usual location but still within the network then this setup also ensures that I’ll always have (indirect) full access to my server configuration. For obvious reasons this Git repository isn’t made available outside of the local network.

So if I’m configuring something on a remote server, then realize that my main server also requires some changes then I can access this repository, apply the required changes and then either push them or make them available for later retrieval.

Once added to the repository I can then verify if the changes are what I expected from them and if so all I have to do for activation is to pull them to the local location(s).

Enhanced security

Now, this isn’t a full security scheme of course because if someone has gained access to your configuration then you probably have much bigger issues to deal with. But even so this setup can be used an extra security measure, a bit of a failsave for example. If you made any drastic changes and want to revert those back to the original then you can: ‘git checkout –reset HEAD’.

Advertisements

May 12, 2018 - Posted by | Editorial, Tips and tricks | , ,

Sorry, the comment form is closed at this time.

%d bloggers like this: