PSA: Windows Hotfix KB2989971

Ever try to login with VALID domain credentials onto a server but get rejected with “invalid username and password” only to have the issue resolved after a reboot of that server?

If so, this Windows bug may be affecting you!

UPDATE: This issue is now resolved in the Update Rollup 2984006. However the below still applies if you have installed the patch after having co-existing AD Servers from Server 2003 and 2012R2.

As a consultant, I have encountered my share of painful bugs but this one particular issue has been biting me pretty hard since we have begun doing Windows Server 2012r2 Domain migrations starting in January. After all, with only a year left for support on Windows Server 2003 you can probably guess what type of server refresh most people are attempting to do these days.

A particular bug was just acknowledged by Microsoft when introducing a new Windows Server 2012r2 Domain Controller onto a domain that currently houses a Windows Server 2003 host.

Note that this issue can affect you even if you have removed all 2003 domain controllers since adding in the Windows 2012r2 servers. The only condition is that there was a point in time where you had co-existence of Windows Server 2003 and Windows Server 2012r2.

There is one typical symptom that is often manifested when you hit this issue which is the inability to login with ANY VALID domain crededentials on the server until a reboot.

Enter: Microsoft Hotfix KB 2989971

Check out the Active Directory Services Team blog post on it HERE

It CERTAINLY turns out that weird things can happen when you mix Windows Server 2003 and Windows Server 2012 R2 domain controllers. No kidding!

According to them, the symptoms are that a random domain member (such as a file server, or client computer) stops accepting domain credentials until a reboot.

When the event occurs, all domain services that leverage authentication such as RPC or SMB stop functioning.

Local user logins still work of course.

One of the tell tales signs that the issue is occuring is when you see Kerberos event ID 4 in the windows event log.:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server %1. The target name used was HTTP/%2.%domain%. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain (%domain%) is different from the client domain (%domain%), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

From what I understand this issue is caused by slight incompatibilities in the Kerberos decryption techniques used between Windows Server 2003 and Windows Server 2012r2. The catalyst is any type of password change that creates a Kerberos key change. This can often include a computer account password which occur silently, randomly, and transparently to most system admins.

Unfortunately it is NOT enough to simply install the hotfix as the issue can persist for a little over a month and randomly stop access to your clients and servers. This is because the issue is never resolved until after a complete reset of the machine passwords AFTER the application of the hotfix.

Luckily you can do this manually during a maintenance window before you get hit by this issue.

Simply login to a domain controller and execute the command in an elevated command prompt:

nltest /SC_CHANGE_PWD:{DOMAINNAME} /server:{COMPUTERNAME}

Where {COMPUTERNAME} is the name of the server/workstation that you want to reset the machine password for and {DOMAINNAME} is you domain name.

Once done, you must reboot that server/workstation for the changes to take effect. This will grant that host immunity from that issue from now on.

It’s a little bit of a pain to do this but it sure is better than users calling to let you know they cannot access their file shares because your file server no longer accepts authentication attempts via SMB!

Advertisements

Thoughts? Comments? Reply here!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: