Thursday, November 28, 2013

ConfigMgr 2012 R2 Upgrade Reboot

Just thought I would write this one up quickly as it impacted us quite a bit Today!

We started the upgrade process to SCCM 2012 R2 from 2012 SP1 CU3 the other day.
In along with this we also decided it would be a good time to upgrade to Server 2012 R2 from Server 2012.

Here are two little gotchas to lookout for if you are doing the same!

Site Reset
Lookout for the site reset, if you have a large number of DP's for example, they are going to take a while to process.
The following things will happen, (listed in no particular order)
1) New Boot WIM's will be deployed to all your DP's if you had any deployed already. (x64 and x86).
2) A new ConfigMgr Client package will be re-distributed.
3) A new ConfigMgr Upgrade package will be re-distributed.
4) All your DP's will be re-set and checked.
5) If you had configured PXE Passwords, the new environment probably wont recognise them, and you will need to re-enter them.  (I'll post a script that can be used to fix this - here)
Reported in the distmgr.log: Failed to get the encrypted PXE password

The processing of these actions is all logged in the distmgr and PkgXferMgr logs and it's going to take a while depending on how many DP's you have.
For example, we have ~200DP's configured as PullDP's, it took about 3.5 hours to go through this list.

DistMgr can only upgrade 5 DP's at a time due to a limit.  Distmgr can also only process 7 packages at a time, with 10 threads per package.
Due to trying to push out packages (notify PkgXferMgr), and upgrade DP's it seems to get snowed under.
It appears that initially it marks all the DP's as Not installed.
Reset 'IsPullDPInstalled' to FALSE so that PullDP SERVERNAME will be updated immediately SMS_DISTRIBUTION_MANAGER 28/11/2013 12:10:45 PM 2704 (0x0A90)
PullDP ["Display=\\SERVERNAME\"]MSWNET:["SMS_SITE=DOE"]\\SERVERNAME\ is marked Uninstalled SMS_DISTRIBUTION_MANAGER 28/11/2013 12:10:46 PM 2704 (0x0A90)

It goes through and does this for ALL installed DP's first, and THEN starts re-installing them.
The distribution point ["Display=\\SERVERNAME\"]MSWNET:["SMS_SITE=DOE"]\\SERVERNAME\ is not installed or upgraded yet.

All the while, also trying to update packages and everything else.
You then think, well, I better fix the passwords, but that also gets added to the end of the distribution queue, and so has no effect until all the other processing happens first!

The Reboot!
This one is the nasty one that caught us off guard. . .
When the DP's were being 'upgraded' they ran this command:
Run command 'M:\SMS_DP$\sms\bin\vcredist_x64.exe /q /log "M:\SMS_DP$\sms\bin\vcredist.log"' to install VC redist
(This is due to the Visual C++ 2010 Redistributable being a new pre-req)
If the file 'C:\Windows\system32\msvcr100.dll' is in use, then the server will be rebooted:

Windows Installer requires a system restart. Product Name: Microsoft Visual C++ 2010  x64 Redistributable - 10.0.40219. Product Version: 10.0.40219. Product Language: 0. Manufacturer: Microsoft Corporation. Type of System Restart: 2. Reason for Restart: 1.

Product: Microsoft Visual C++ 2010  x64 Redistributable - 10.0.40219. Restart required. The installation or update for the product required a restart for all changes to take effect.  The restart was deferred to a later time.

Now, it says the restart was deferred, it lies, in our cases it DID reboot. . . immediately!

I don't have a workaround for this, maybe upgrade your VC++ early, and get around it that way.
The file was in use by a number of processes, Symantec Endpoint, BackupAssist, and the SCOM Monitoring Agent.

We stopped smsexec, and waited until after hours to restart the upgrades.

Hope this helps if you are about to do the same!