Jul 16

Changing Puppet Masters

As users of puppet, occasionally we need to migrate nodes from one master to another.

In my case I’m decommissioning my old puppet server having stood up a new one, as a part of my “migrate home” project.

I ran into a couple of minor issues, but this is essentially the process for moving a node from one master to another.

First, stop puppet (this isn’t necessary, but good practice):

# /etc/init.d/puppet stop

Next, edit your puppet.conf to reflect the new change.

Now, if you start puppet again you’ll likely get errors and it won’t work.

# rm -r /var/lib/puppet/ssl/*

[warning]This is for Debian package-installed systems; if this is not your system, check your puppet.conf to determine where the SSL directory is.[/warning]

# /etc/init.d/puppet restart

Now switch back to your NEW master and look for the new certificate, and if it checks out, sign it:

# puppet cert list

“swedishchef.i-al.net” (SHA256) 05:5E:23:7E:03:A9:58:B6:F2:FE:F6:D4:A1:C3:CE:FD:8B:64:4D:F2:D5:87:02:22:7A:C1:44:8D:D8:44:8E:E8

# puppet cert sign swedishchef.i-al.net

Notice: Signed certificate request for swedishchef.i-al.net

Notice: Removing file Puppet::SSL::CertificateRequest swedishchef.i-al.net at ‘/var/lib/puppet/ssl/ca/requests/swedishchef.i-al.net.pem’

Check everything is running, and you should observe everything is in order. If not, debug as normal. As always, try this in a test environment first – I take no responsibility for broken production environments based on the above.

May 31

Migrating Home

My “Personal Project” for a long time has been small web/mail hosting, primarily for myself. For the last year or more I’ve achieved this with a group of small VPS services.

It started out back in 2008, I think, with a 256MB Xen machine, hosted by ezVPS (no longer in business). Eventually I picked up a second one from the same provider, and balanced the load with the different sites across the two servers.

As time went on and I grew my aspirations, I rented a 512MB KVM server from BuyVM/Frantech. When ezVPS shut down, I was in the process of moving one of my servers to a 256 KVM with BuyVM already, and I was able to snag another 256MB and move the other one. Right now I’m paying ~$20/mo for three servers ($10 for the 512 and $5 each for the 256s). Money has been a little tight, however, and now that I’m paying for and controlling the internet connection where I live I felt it was time to start moving things home.

I started by creating some new VMs on my VMWare server. I have one each for Administrative purposes (mostly just Puppet), the Panel (ISPConfig), the Web server, and the Mail server (though it will be shut down and I’ll use one of the 256s). With everything appearing to be running nicely, I started by moving one site to the new server. All appears in order, so it’s time to start moving the rest and slowly getting everything off the 512.

Once everything moved off, I can shut it down, cancel and start saving $10 a month. So far, so good.

May 11

Test Case Web

[notice]FAIR WARNING: At time of writing, this software hadn’t been fully tested. During the tests I have found a large number of SQL injection issues with this code that I have patched on my system, and will continue to patch as I check over the package. In the next few days I’ll make a useful diff/patch and submit to the maintainer, because this is simply unacceptable – especially for a tool designed to help with Software QA.[/notice]

Part of my specific duties involve some software testing as part of our Quality Assurance efforts on tools we have developed in house, both for internal use and for our customers to use. Things like our customer portal which, among many other things, gives our customers the ability to manage what is in their racks, and if they have a PDU which allows it, remotely power up or power down hardware.

We’ve been managing this effort using a shared spreadsheet which works well enough, but can easily be improved. So I started looking for tools that would allow us to manage our testing efforts in a much more efficient manner. It might take a little more administration, but it should improve our workflow and hopefully balance out, especially once the initial start-up is out of the way.

Here is what I found: an old application called “TCW” or “Test Case Web” which has been in development for some time. According to SourceForge, it is still in fairly active development, the most recent release being just a couple of weeks ago, on April 24th.

It’s written for PHP4, so there are a couple of deprecated functions and variables which I’ve adjusted for, and I had to fight my development server just a little to make it work right, but it’s running.

Here are a couple of tips:

The default login is (case sensitive, apparently):
Username: Admin
Password: admin

Line 4 of “adminaction.php” reads “$args=$HTTP_POST_VARS;”, change it to “$args=$_POST;” under PHP 5.

None of the system has an install script. You’ll need to create a database and user in MySQL, then edit the incluido.fil file to have the credentials. You’ll also need to import the schema to MySQL, easily achieved with the mysql command line tool or phpMyAdmin.

There are also a handful of places in the code that trigger PHP warnings, mostly because they check the contents of a variable without checking that the variable is set.

For the “home page” (which is severely out of date, but the docs mostly apply), see here.

For the SourceForge project page with current releases see here.

Also, if you’re a PHP dev, it might not be a bad idea to take a look and maybe consider helping out, even just briefly, to review the code for security issues and offering a helping hand to bring it up to PHP5 standards.

May 10

Moving MySQL Users and Databases

Occasionally, as part of my day-to-day duties, we migrate users from one shared hosting server to another. In general we use the cPanel/WHM “Transfer account” feature. On occasion, this doesn’t work quite right and manual interventions must be made.

When the Transfer Account feature breaks a user account, it’s annoying. Usually most of the transfer is usable, with only a few aspects needing to be moved the hard way.

I noticed one transfer go awry, however. A number of files were missing from the directory, so I logged into both servers (old and new) and tarballed/scp’d manually. No problem.

Then I found the SQL databases weren’t working right. After some investigation I found the DBs hadn’t been transferred properly, so I set about transferring those.

Transferring the databases is easy. Getting MySQL to dump the databases with schema and data is pretty easy, even easier if you have access to phpMyAdmin which has a handy “Export” feature. Transferring users and privileges – not so much.

I did find a nifty tool for this though. It’s called “mk-show-grants” and I found it here. It dumps all the lines you need to grant access to a user on another system, or can give you the SQL lines to revoke access on the current system. It’s a Perl script, so you can review the source to know it’s not Being Evil, it just requires the Perl module to access MySQL.

May 10

Hello world!

It’s a startlingly appropriate title for this blog.

Here’s the deal. I am a Jr. Systems Administrator. I work for a company that provides hosting services, anything from shared web-space to multiple-cabinets of colocation space. If you have a hosting need, chances are we can meet it.

But this blog isn’t about my employer. It isn’t even really about me, so much as it is about the things I learn along the way, which may include experience from my employment or on personal projects.

Disclaimers are required at this time:

I am really easy to find. While I may give an air of anonymity, a quick WHOIS will reveal who I am and where I live. From that it will be a very quick process to determine who I work for and who I work with. For that reason I do not give details regarding clients. No names (of people or company), no IP addresses, no routing tables, nothing. Anything I post containing anything close to identifying information regarding my employer or their clients can and will be obscured to prevent readers from determining anything useful, and I will endeavor to ensure readers are unable to determine anything at all.

Any opinion, stated or implied, is my own and does not reflect the views of my employer or colleagues. Any advice is given on the grounds that it has worked for me, and no warranty or guarantee is given whatsoever that it will work for anyone else. I am happy to offer assistance to others if I can, but this is given on the same basis with no warranty of guarantee.