ConfigMgr 2012 SP1 - When Packages Get Stuck

Over the last few days, I have been working through numerous scenarios and issues, most of which we have resolved without too much trouble.

Problem 1

One scenario that was a little interesting was when we were trying to re-distribute the built in boot images to some distribution points.  We had removed the package from the distribution points, as we didn't think we would need them....turns out we did.
When we went to re-add them, the distribution status kept informing us that there were no distribution points targeted.  We did this a few times, both the individual DP's as well as DP groups, each time we re-open the content locations tab on the boot package properties, it showed no DP's!

On investigating the logs we found something odd.
The logs were reporting that the content was being REMOVED from the DP, not added to, and then, that it was removing the DP from the date sources.

OK, that's odd, but how do I fix it?

The Fix - Problem 1

This was the first 'stuck' package we had come across in ConfigMgr 2012.  It appeared that the package was stuck in a cycle of trying to be removed from the DP's, but was never finishing the task.  THe content was still in the SCCMContentLib on both DP's.

After a bit of a search, we turned up a database query to find which servers the package was located on:

select * from PkgServers where PkgId = 'PRI00004' and SiteCode = 'PRI'

This returned 2 entries for the package in question.  The interesting bits fields were:

• Action = 3
• SiteName = NULL
• UpdateMask = 8192
• LastRefresh = 1970...

We deleted the two entries with this command:

delete from PkgServers where PkgId = 'PRI00004' and SiteCode = 'PRI'

Now, when we went back to the console, we were able to successfully redistribute the package to the DP's.

Problem 2

The next issue that we came across was after testing some DR processes (failing the Primary Server, and Re-Installing it), we found that the Configuration Manager Client Upgrade Package was not being distributed to one of the DP's.  It was in a failed state and wouldn't budge.
The trouble with this package is, that it is a hidden package, and you cant refresh it from the console, or delete it, or validate it etc.
Luckily, there is a PowerShell command that can do that for us.
The PackageID for the hidden upgrade package is always the same 'XXX00003' where XXX is your SiteCode.

$AllDPs = Get-WmiObject -Namespace "Root\SMS\Site_PRI" -Query "select * from SMS_DistributionPoint where PackageID = 'PRI00003'"

foreach ($DP in $AllDPs) {

    $DP.RefreshNow = $true

    $DP.Put()

}

Try as it might though, the primary server couldn't refresh the package.  The distmgr.log revealed the following error:

ExecStaticMethod failed (80041001) SMS_DistributionPoint, FinalizeContent

After a bit of searching, it appeared to be related to a permissions issue.
Looking at the DP's SCCMContentLib\DateLib directory, I could see a folder for the package:

PRI00003.6

When trying to open the folder though, I was greeted with an access denied, despite being an admin on the server.

The Fix - Problem 2

Repeated attempts to to take ownership of the folder failed, as I didn't have permissions to do it.  I also tried using PowerShell's Get-Acl and Set-Acl, but these also failed with Access Denied.

I realised that the commands needed to be ran as System, or the local computer account.
With a wave of the ol RegEdit, I added in 2 RunOnce entries:

takeown /F D:\SCCContentLib\DataLib\PRI00003.6 /A /R
ICacls D:\SCCMContentLib\DataLib\PRI00003.6 /reset

With a quick reboot, the permissions were restored.
After re-running the powershell commands above, the package was successfully transferred back to the DP!