Hey Checkyourlogs Fans,

I recently had a case of a stalled Windows upgrade that ended up being caused by a bug in the Manage Engine Desktop Central Distribution Server component.

This was a poorly written application creating 3 million+ files in c:\programdata\Microsoft\Crypto\Systemkeys.

To diagnose the problem, I first went where I always do the Panther Logs

I checked in c:\$windows.~bt\sources\panther\setupact.log

In here I could clearly see it processing the c:\programdata\Microsoft\Crypto\Systemkeys folder

I then opened Resource Monitor to find it very slowly processing these files.

I did some research and found more details on this issue.

https://learn.microsoft.com/en-us/answers/questions/508389/c-programdatamicrosoftcryptosystemkeys-folder

We are also using MEDC – Manage Engine Desktop Central and there is a bug where it writes hundreds of thousands of files to this directory stalling out the windows upgrade.

I exited the setup and, in my case, this was a file server VM so I just wanted to shut it down.

And rebooted to release the locks on the files.

Then I took ownership of c:\programdata\microsoft\crypto\systemkeys

I wanted to see what exactly I was dealing with inside the folder structure because reports were coming back of millions of files in this folder

To be exact 3 million ish files in here. That doesn’t look right at all.

Well, I have great Veeam Backups and needed to get this upgraded so I’m going to delete the files.

Have you ever tried Ctrl+A on 3 million files the server really didn’t like that at all.

Nope, that just crashed Explorer.

Next, I tried to remove the entire folder. I can recreate it or restore it from backup later.

Patience was key here it took over 20 minutes just to get the discovery to kick in.

After a long 30 minutes now the deletion proceeded

Just say skip on the deletion of the folder when prompted.

Apparently Manage Engine has fixed the issue but leaves behind all the old files and requires manual intervention or a collection configuration to fix it:

https://pitstop.manageengine.com/portal/en/community/topic/fix-for-the-issue-where-multiple-certificate-key-files-are-accumulated-in-the-os-installed-drive

That actually didn’t work to delete the files, so I had an old way of tricking Robocopy to do the job for me.

If you create a dummy folder like c:\test and run

Robocopy c:\test c:\programdata\microsoft\crypto\systemkeys /mir

It will reverse the flow and purge the target directory (MASS Delete)

As seen below:

Eventually I gave up on this and decided to try just renaming the folder.

And re-running the windows upgrade again.

Setup started again and this time hung right off the bat.

I found this folder in use looks like it copied all of those system keys upon the first run.

Renamed it as well

Trying Setup Again

Doesn’t care if I rename them.

Trying to force setup to start over

That appears to have done the trick setup is progressing again now.

Finally ended up renaming the $Windows.~BT folder to start fresh again.

This got us back to the normal Windows Upgrade screen now at least.

Now Setup is moving again.

Notice the difference in the NEWOS\Programdata\Microsoft\Crypto Folder now

No systemkeys since renaming them and it immediately went beyond the 63% it was hung up at.

Renaming the Systemkeys folder, deleting the $Windows.~BT folder, and re-running setup eventually fixed my issue.

I really hope you find this post valuable,

Thanks,

Dave