Hello World,

Today, we will speak a little bit about Hyper-v. In my work, I tend to meet more and more hyper-v infrastructure as virtualization platform solution.  VMware is still around but I have to say that I can see that Hyper-V has gained some credibility with Windows 2012 R2 version and make the solution more attractive to enterprises.

We recently have implemented a hyper-v solution with a converged networking scenario.  Everything was running fine.  And then, we started to have every two minutes the event ID 14050 in our event viewer (under the  Microsoft-Windows-Hyper-V-VMMS-Admin Node) an error related to the fact that the system could not register a bunch of service principal names).  The system was complaining about the following two or three services :

• Microsoft Virtual Console Service
• Microsoft Virtual System Migration Service
• Microsoft Hyper-V Replica service

It took a little bit of time to find out what was blocking the SPN registration but we finally found out.

Debugging process

Check Service Principal Name attributes

Our first step was to confirm that the service principal name (spn) for the Hyper-v servers were not registered.  We have opened our Active Directory, select the Hyper-v computer object, right-click and select the properties.  In the properties dialog box, we went to the attribute tab, looked for Service Principal Name and we could confirm that indeed, SPN registration didn’t worked out.

Click on Picture for better resolution

We thought that we could manually entered the SPN and have the problem fixed.  So, we manually registered the SPN on each Hyper-V servers being part of the cluster. You can do that via the Attribute tab (as seen above) or using the setspn command line :

setspn -S “Hyper-V Replica Service/HyperV” HYPERV
setspn -S “Hyper-V Replica Service/HyperV.yourDomain.com” HYPERV

setspn -S “Microsoft Virtual System Migration Service/HyperV” HYPERV
setspn -S “Microsoft Virtual System Migration Service/HyperV.yourDomain.com” HYPERV

setspn -S “Microsoft Virtual Console Service/HyperV” HYPERV
setspn -S “Microsoft Virtual Console Service/HyperV.yourDomain.com” HYPERV

 

We checked that the SPN registered properly this time.  We went back to our Hyper-V servers, restart the VMMS service and checked the log file again….. Event ID 14050 was still logged after this change, so we needed to ask google.

What we found on the internet….

We have noticed that this error was quite common and a lot of people seemed to have the problem with SPN and Hyper-V.  We have found a lot of people having the same issue as we had. Hereafter, we provide a list of possible troubleshooting or actions to perform :

• http://social.technet.microsoft.com/wiki/contents/articles/1340.hyper-v-troubleshooting-event-id-14050-vmms.aspx  (Did not fix our issue)
• http://techblog.mirabito.net.au/?p=230 (did not fix our issue)
• http://blogs.technet.com/b/tonyso/archive/2009/04/07/hyper-v-how-to-troubleshoot-hyper-v.aspx (did not fix our issue)

We finally came across the following link

• http://www.aidanfinn.com/?p=15856 

So, we decided to go and read the Microsoft Knowledge base mentioned over there.  The KB (see http://support.microsoft.com/kb/2761899) was mentioning something about port range.

This problem may also occur if the NTDS port has been restricted to a specific port on your domain controllers. If this selected NTDS port is not within the default ranges, you must add this port by running the script in the “Resolution” section on every Hyper-V host.

C:\>netsh int ipv4 show dynamicportrange tcp
Protocol tcp Dynamic Port Range
---------------------------------
Start Port      : 49152
Number of Ports : 16384

 

Again, this post was pointing to another kb (http://support.microsoft.com/kb/224196) which would explain how to restrict RPC Traffic to a specific port.  We didn’t do that.  So, is this Microsoft KB, the solution we are looking for. Reading the KB at http://support.microsoft.com/kb/224196, we had noticed that again another link was pointing to an article explaining How to configure RPC dynamic port allocation to work with firewalls.

Guess what ?  In our environment, we have configured RPC dynamic port to work with firewalls.  So, my domain controllers were listening to rpc port above 5000 (which is out of the range specified above…)

The Solution

In order to get rid of the every two minutes event id 14050, you have to follow the instructions on the MS KB located at http://support.microsoft.com/kb/2761899).   You have to run the vbs script on your Hyper-V machines and you should see that the error will go away.

When you run the script and you try to see in your firewall if this rule has been implemented, take note that this rule does not seems to be visible through the GUI.  So, yes the rule is there and you can confirm this by check that the error 14050 is gone.

Finally, we have found another link (https://p0w3rsh3ll.wordpress.com/2014/01/02/list-the-windows-service-hardening-firewall-rules/) which was basically providing us the powershell version of the vbs script.   I prefer to use the Powershell scripting engine over vbscript.

I provide hereafter the Powershell version of the vbscript provided by Microsoft

 

Credits for the PowerShell script :

 

https://p0w3rsh3ll.wordpress.com/2014/01/02/list-the-windows-service-hardening-firewall-rules/

 

#———- Beginning of the script ——————————#

$rule= New-Object -ComObject HNetCfg.FWRule -Property @{

    Name = "Allow outbound traffic from service to TCP 5100 to 5400"

    Direction = 2

    Enabled = $true

    ApplicationName = "$($env:systemdrive)\WINDOWS\system32\vmms.exe"

    ServiceName = "vmms"

    Protocol = 6

}

$rule.RemotePorts = "5000-5400"

(New-Object -ComObject HNetCfg.FwPolicy2).ServiceRestriction.Rules.Add($rule)

 

Get-NetFirewallRule -PolicyStore ConfigurableServiceStore

#———- End of the script ——————————#

Final Notes

And Voila ! We have fixed the issue and our customer was happy.  As you can see from the above description, we have spend some time in searching the reason behind this strange behavior.  We found out quite quickly the Microsoft Knowledge base in order to fix our issue.  For us, the KB didn’t really fit our scenario/situation.

At the end, it turns out that this was the solution.  You need to have a good knowledge of your Active Directory and need to reads carrefully MS KB documents  🙂

Hope this will be useful to some other people out there

Till next time

See ya