Updating Ubuntu for the Unexperienced and

Time has come to update one of our Ubuntu boxes.  I can never remember how to do it and always have to go lookup the commands.  Maybe if I write them down here it'll help me remember them for next time.

I started with a lsb_release -a to see what version I was on.  Which happened to be 12.0.4.5 so I wasn't too far out of date.

Then I ran sudo apt-get update and then sudo apt-get upgrade to update all the packages that were available on the current version.  That brings me up to a fully patched system that needs to be upgraded to the latest LTS version (this is a production box running applications so I like to stick with LTS versions, I've never had a need to utilize the long term part, but it makes me feel better).

Next step I did was sudo apt-get dist-upgrade.  I'm no expert and I probably could have skipped this step and it's possible this step caused my later troubles.  Hind sight is 20/20 they say.

Now I just need to make the plunge and upgrade to the next release.  That's pretty easy,  sudo do-release-upgrade handles that.  And then the fun started.

Part way through this it prompts me because some of the apache configuration files have been modified from their originals.  I look through these changes and it really doesn't seem to like too big of a deal, but I don't want to stumble into something since most of these tweaks were done 2 years ago so I make a backup.  This machine is virtual and I didn't have the machine integration plugin installed so I couldn't switch between consoles there.  No problem though, I just SSH'd back into the machine and made a quick copy of the config file in question and then told the updater to replace it.  

The update gets done, I give the box a reboot, it seems to come back up normally and I think all is right in the world.  Then I pull up one of the websites hosted on the box and it just displays the content of the php file rather than processing the php file.  Maybe it's just this one site, nope, all the sites are equally broken.

No sweat, I've got a backup of that configuration file.   So I compare my "custom" file to the new file and don't see anything obviously wrong with it.  Obviously some of my tweaks are gone, but those wouldn't affect php not running.  I try restarting apache and it complains about a line in a configuration file (if I had noted the file I would have shared it here), that line looks like it has sometihng to do with php.  I did a bit of Googling and came across a reference to enabling php module with this command sudo a2enmod php5.  No luck, restarted apache, still no luck, same error and web pages still aren't right.

I find the command apt --installed list to show which packages are installed. I see a lot of php5-something else packages, but no php5 package.  I run sudo apt-get install php5 and let it install (it doesn't complain that it's already installed).  I restart apache and it works.  No error.  My web pages are back to normal.

I run apt --installed list again and it shows php5 is now installed.

In hind sight I think maybe the apt-get dist-upgrade may have been what did me in.  I'm not sure why, but none of the other commands should have removed any packages.  I guess if I had paid more attention to those apt-get message about packages being updated and being removed I would have been on top of this.  Lesson learned.

Zenworks Configuration Management Updating 11.2.0 to 11.2.4

Sometimes I can get a little behind.  You really know you're behind when you go to System Updates and look at the release date and realize it was a year and a day before your download date.  But that's the name of the game.  We jumped from Zenworks for Desktop 7 straight to Zenwork Configuration Management 11.2 last summer and brought all of our XP clients along with us.  For the most part 11.2 was a good version for us and our XP clients and we didn't really have a need to upgrade.  Fast forward a year and we've deployed a whole lot of Windows 7 machines and are running into some bugs that have been patched, now we need the update.  A good long weekend with a couple of user free days looked like the perfect opportunity to do the upgrade.

You can safely ignore that "Check for Updates" option in the System Update section.  It's a known issue (apparently been known for a while) and has something to do with the update being too big to download automatically.

To start out, grab the update from the Novell page (you can find it here http://download.novell.com/Download?buildid=ZCUFlvDkC9w~ ).  The instructions in the details section at the bottom are spot on.  Copy the file to the server, run the "zman sui" command, and now it shows up in the System Updates section.

The online instructions (you can find them here https://www.novell.com/documentation/zenworks11/zen11_sys_updates/data/bjppvdf.html ) look pretty straight forward but are not as clear.  You can skip the first couple of steps about getting the updates (since you just did that manually with the download and the zman sui). What through me for a loop was that I only wanted to update one server before I rolled it out to the rest of the servers.  Which is their recommendation.  The instructions aren't obvious on this.  The process starts out designate a server and then assign the update.  That looks pretty easy.  What they don't tell you is that you don't assign the update on the frontend, you pick the update and set it to install and then pick which server on the backend.  If you read ahead this becomes a bit clearer and after you've done it once it's really obvious how it works and it's really easy.  But for the first time through it not super clear and can be a little unnerving.

The update on the first box went pretty smoothly.  It took an hour or so (quick description, about a dozen primary servers spread out across different sites, ~3000 devices total, internal Sybase DB, running in a Hyper-v VM on a Dell R720 with the slowest 10K SAS hard drives ever).  After it came back up the Configuration tab of the ZCC showed no version for the server.  That did induce a little bit of panic, but running a "zac ref" on the server updated the display version so it showed the correct 11.2.4.0 version number.  That trick would come in handy as most of the updated servers also showed up blank, but refreshing the agent on those boxes fixed it.

Then I kicked off two more server updates.  Two hours later they both showed "Rebuilding Deployment Packages"  This wasn't cool, the first server updated in around 2 hours and it supposed to take the longest.  I searched around to see what I could find.  Since I had two boxes I left one alone and picked on the other.  I tried a "zac ref", no luck, I tried a "zman surp" (found that on the forums) still no luck, tried a reboot, nothing.  I did dig around in the file system and I found the folder C:\Program Files\Novell\Zenworks\install\downloads\msi which looks to contain the deployment packages.  I noticed that most of the files had varying timestamps from today and that some of the timestamps were only a few minutes old.  That would mean it was still actually doing something and hadn't stalled.  So I left that folder open and went about searching for more things to check or try and while I did that it finished... Update Completed.  Then the box that I hadn't touched finished a few minutes later.  Apparently, I was just impatient.  I'm glad I didn't toast the box trying to hurry it along, but it is nice to know that the process can be fairly resilient.

Then I picked a few more to start upgrading and kicked them off.  They seemed to progress nicely.  I picked the remaining boxes and set them a scheduled time over night to upgrade.  Crossed my fingers and went home.

I arrived in the morning to find some boxes updated, some boxes "Rebuilding Deployment Packages" and one box failed.

The failed box hadn't run out of space, the update was all there, and the log files looked to just have a simple "failed to update..." message.  So I selected the box from ZCC and selected "Redeploy update to device", it jumped back to rebuilding packages and in about 20 minutes it showed "Update Completed".  Easy enough right?

The rest of the boxes?  Just slow.  As the morning progressed they slowly started finishing.   All except for one.  Over a day later is looked like it was stuck.  I gave it a reboot and a couple of hours after that I tried to the "zman sui" command.  A few hours later I rebooted again and waited.  It finally picked up and finished.