Yes, it happens to every system administrator from time to time. The account lockout. Usually Microsoft Account Lockout Status tool or Netwrix Account Lockout Examiner (my favorite) work well. Not this time…
Here’s the story.
One of my domain administrator accounts was kept locked out. The event log analysis helped to find the server that generated the lockout events. However, it was a server in a different domain and nothing gave me any idea about the process generating the events. No scheduled tasks, to services, no mapped drives, nothing. To make the situation even worse, it was the only DC that generated this event. If that DC has been down, the account was not locked up.
First, I installed WireShark and captured packets between this server and that DC. Yes, there was an issue:
Look at the lines 4 and 5. There was an authentication request with my domain account name and the invalidCredentials reply
Fine, but what process generated the request? Unfortunately, WireShark does give the answer. Neither MS Network Monitor. But newer one, Microsoft Message Analyzer does.
So, I installed and ran it. Since I was not familiar with the tool, it took time to figure out how it works:
- Click on Add Session button on a start screen, then click on Live Trace button. Or select File – New Session – Live Trace
- Click on Select Scenario and select Local Network Interface.
- Click Start
- On the Session window apply the filter, such as address==10.10.10.10. I used that DC IP address.
- Click on Add Column to add ProcessID and ProcessName
- …and wait
Oh, here’s the outcome
Same message, InvalidCredentials. But now I had a process ID, so task manager can help me
Not much, actually :o(
This is Java running under System account. I forgot to mention, that this server is my VMware vCenter server, so Java is a part of vCenter installation.
Sysinternals Process Explorer gave me a bit more information
I was able to find the parent process, wrapper.exe and Process Explorer unveiled the secret
vCenter SSO! Now I new what to do.
If you have worked with vCenter SSO, you know that vCenter Web Client has identity sources to connect to multiple domains. I simply used my domain account when tested it and forgot to change it to the permanent account. BTW, it explained why it was locking it out only on one DC. It was the only DC configured for SSO authentication :o)
Just as a reminder:
- Run vCenter Web Client and log in as SSO Administrator
- Click on Administration
- Click on Configuration under Sign-On and Discovery
- Click on an identity source and change its configuration
voila! My account is not locked anymore. But it was a good battle