Lion Server – Software Update Server

Yesterday, many Mac system administrators learned on Twitter of an urgent change arriving to their servers leaving less than 24 hours until the expiration day for Apple’s certificate used to sign updates. This change affects a tool that gets little fanfare and is deeply buried as one of 20+ items that Profile Manager can do. Although a Software Update Server (SUS) can normally be set up once and mostly forgotten, but doing so can save money, conserve bandwidth, reduce support tickets, and make a company’s Macs more stable.

The Software Update service allows all the Macs to get Apple software updates from a local server rather than having to use the public servers that Apple runs. This is a huge benefit, saving bandwidth on both Apple’s and the business’s ends, since each update gets downloaded once instead of for every computer. Also, when a company wants to test updates before they hit all their hundreds (or thousands) of computers, SUS allows those Mac administrators to release updates after they have been tested to avoid disrupting their colleagues with update conflicts interacting with other business critical software that could have been caught with some testing.

So, why write about this now?

Today is the day when you might start seeing errors if old software updates now fail validation. Server administrators may also prefer to prepare for a rush of many gigabytes of new downloads to be stored on their servers if Apple releases new packages en masse.

Why are software updates cryptographically signed with a fixed expiration date?

Before shipping a software update package, Apple uses a standard certificate to sign each item which helps ensure that the packages, when finally installed, have not been maliciously modified or innocently corrupted in the download process before they finally get installed on a specific Mac. The certificate used to sign these update packages expires on March 23, 2012. A knowledge base article titled Mac OS X Server: Software Update Certificate expiration appeared recently to explain this situation and offer steps for people to manage this transition successfully. The certificate that is expiring was used to sign the updates provided to Lion Server as well as the Snow Leopard server. All the updates for Macs running either Snow Leopard or Lion will be re-issued, which seems to indicate that older operating system and application updates will not be cached locally by servers going forward.

What happens next?

Several things will happen as soon as Apple starts sending out the new updates (set your calendars – these will expire in 2019 if all goes as planned) :

1. Apple is in the process of re-signing all updates and the existing servers will downloading a new copy of all “current” updates.
2. This means many updates will be doubly downloaded since the prior updates are not deleted by the server (and will now take double the disk space).
3. Each Mac has a 50/50 chance of trying to install an “expired” update, which will then show an error message stating that “The update could not be verified”.
4. The server will be downloading a lot of data from Apple starting today or tomorrow. Some clients with slow connections will notice the continuous download and this might take a week or more to download the newly signed catalog.

Most of my Lion servers use 130 GB of space to store the current set of updates, though none have completed downloading the new set of updates, so it’s not clear how much space the current “complete library of Apple software updates” will require once downloaded. Some of these files are now expired and Apple will not be re-signing and re-including these older updates in the complete catalog of updates that Lion server can download.

I expect the actual download size to be less than 100 GB, but even so that amount is enough for some to earn a letter from their internet provider recommending they pay for a business class service or even pay for overages in some cases. Even if you are not concerned with the amount of data, downloading 50 GB of data on a fast connection can take days and I would expect Apple’s servers to be slow for the next few weeks. Pretty much everyone’s servers will be hitting Apple at the same time since the new downloads are being released within hours of the expiration of the old.

Also, some people that use a Mac Mini server may have less than 100GB free on the drive where these updates are stored and could run into a low disk space situation. However, this is more of a housekeeping problem than something that will immediately cause a problem. By default, the service should pause all future downloads when free space on its volume gets less than 20% of the available space.

Events like this are a great time to verify that alert emails from the server for low disk space warnings are going to the correct group of people.

One more thing…

Also, if you follow the instructions in the Apple KB article as I did, you’ll get an easy-to-miss syntax error and your server won’t download updates from Apple until it is resolved. Step 5 of that guide states “Delete the folder named “html” inside it.” but when you start the service, it won’t create the folder to store all the updates and you will see text in the Software Update Error Log like:

[Thu Mar 22 09:59:34 2012] [notice] caught SIGTERM, shutting down
Syntax error on line 288 of /etc/swupd/swupd.conf:
DocumentRoot must be a directory
[Thu Mar 22 10:01:23 2012] [notice] mod_bw : Memory Allocated 32 bytes (each conf takes 32 bytes)

Just recreate the directory and then stop and then start again the update service using Server Admin. This clears the error, and your server will be ready and waiting to download the new updates from Apple as they are released.

sudo mkdir -p /var/db/swupd/html

If you have a different location than the default – be sure to adjust the above command accordingly. Good luck, and set your ticker file for any certificates you come across so you have more time to be forewarned and forearmed.

Two other blogs that cover the details of inspecting and working with expired packages are Rich Trouton’s invaluable personal blog Der Flounder  and a blog I have just started following by Greg Neagle on managing Mac OS X in an enterprise environment.

• Package Apocalypse by Greg Neagle

• Apple installer package certificate expiration by Rich Trouton

 

Mike Bradshaw (bmike)

For more questions and answers on Software Update and Servers, you can search using the tags software-update and server on Ask Different.