How to Get Hyper-V Time Synchronization Right

Are there any bumper stickers or t-shirts out there that say, “I was using Hyper-V before using Hyper-V was cool?” If there are, I think I need one.

I built my first production system on Hyper-V 2008 R2 not long after it debuted. So, I missed the early-adopter crowd by a couple of years but I’ve definitely been in this for a while. The only problem with sticking with a tech as it grows from infancy is that sometimes things change and you miss out. One of the things that changed on me was the way that time synchronization works between Hyper-V and its guests. There’s only been a single casualty that I know of, and that is the virtualized domain controller.

I’ve written about this subject before. The earliest is a complete article on configuring time for a Windows domain. Next was the practical article of the virtualized domain controller series. I didn’t talk about virtualized domain controllers at all in the first article, which I have since revised. In the virtualized domain controller article, I recommended that domain controllers have the Hyper-V Time Synchronization service set to partially disabled. I’ve left the original text in that article so you can go there to see what I had to say, if you desire.

To save you the trouble of jumping around and reading a lot of other articles, I’ll give you a bit of necessary backstory.

In a Windows domain configured with defaults, the domain controller that holds the Primary Domain Controller Emulator (PDC Emulator) flexible single master operations (FSMO) role is considered the authoritative time source for the entire domain. One way or another, every single other computer in the entire domain gets its time from that single source. As a result, it’s quite important for the time on that system to be correct. What most people do, and what I explained how to do in the first article that I linked above, is configure the PDC emulator to continually update its own time from a known valid external source.

A typical virtual machine is configured to get its time directly from the Hyper-V host. This eliminates all sorts of concerns, such as guests in different domains or workgroups and lots of needless time synchronization network traffic. What it’s an especially good solution for is the somewhat unique nature of the virtual machine. Using the Save State operation, it is possible to put a virtual machine into a sort of suspended animation for an effectively infinite amount of time. When a virtual machine is resumed from this state, it has absolutely no idea that anything happened at all. As a result, its clock will be set to the exact time that the save operation was conducted. If that was fairly recent, then there shouldn’t be a problem, especially for a modern operating system. It will eventually seek out an authoritative time source. A problem arises when the clock is very different.

Due to security concerns, there is a maximum time differential of only five minutes between a client and server in a Kerberos conversation or the two systems are considered skewed. The upshot of skewed systems varies wildly. A Windows client desktop might be a few days behind and the only consequence is a lot of logged warnings and errors. An Exchange Server system will panic at a very small skew. If a clock is really skewed, then it will have lots of strange issues, especially when dealing with SSL certificates. However, there is a maximum amount of skew beyond which a domain controller will not update a client that asks for a time synchronization. This problem is directly solved for Hyper-V guests via the Hyper-V Time Synchronization Service.

The explanations that you can find about this service really are not entirely accurate, and some things appear to have changed over time. I did a fair bit of testing, and, as of 2012 R2, this is how Hyper-V and the Hyper-V Time Synchronization Service interact with a guest virtual machine:

1. Hyper-V is responsible for time operations when the guest is off, the Hyper-V Time Synchronization Service is responsible for time operations while it is on.
2. Hyper-V “maintains” the guest’s clock while it is a turned off state. I don’t know the exact mechanics, but if a virtual machine does not have any time synchronization of its own set up (including the Hyper-V Time Synchronization Service), its clock will not lose time while it is off. A previous article that I read said that Hyper-V would inject the management operating system’s time into the virtualized real-time clock. That does not appear to be the case (at least, in this version). If you disable all time services in the guest, you can move the machine into a different time zone and set its clock horribly wrong and it will never be corrected. However, it will not lose time when it is off. This means that Hyper-V is adding any missed time to the real time clock but is not otherwise interfering with it.
3. The Hyper-V Time Synchronization Service requires the Windows Time Service (or the Linux equivalent) to do its job. I didn’t do any experimentation with Linux, but it’s safe to assume that operations there will be analogous. In a Windows guest, if the Windows Time Service is stopped, the Hyper-V Time Synchronization Service never does anything except at initial Windows boot-up. So, if you disable the Windows Time service and leave the Hyper-V Time Synchronization Service enabled and then change the clock, it will never be corrected until the guest is restarted.
4. Only the Hyper-V Time Synchronization Service is guaranteed to correct the clock after a resume from saved state. If the Hyper-V Time Synchronization service is not running, then the guest’s clock will continue tracking time as though it was never saved. If its Windows Time service is started and has a valid source, it will attempt to update at its next scheduled interval (again, it won’t know that any special amount of time has passed since the last update). Depending on its source’s maximum skew setting, that process might fail. Unlike normal operations, this does not depend on the Windows Time service also being started.
5. The “partial disable” technique for the Hyper-V Time Synchronization Service does not work. If both the Hyper-V Time Synchronization Service and the Windows Time service are enabled, then the guest will get its time from the management operating system. The registry hack will cause the Windows Time service to report that its source is something other than the Hyper-V service, but the guest’s time will be synchronized to the management operating system’s clock anyway.

[optin-monster-shortcode id=”u4bw5fa5efzfughm”]

Hyper-V Time Synchronization Service in Practice

In almost all cases, you will want the synchronization service to be enabled for every guest. The reason is simply because there’s not much of a reason not to. The biggest risk is that the host’s time will drift, which will cause all the guests to drift. That’s why it’s very important to ensure that the host is configured to synchronize against an authoritative source regularly. The instructions were linked above, but here’s the link again.

Time Synchronization for Virtualized Domain Controllers

The subject of time synchronization for virtualized domain controllers has always been a controversial issue. Some people were never able to get it to work correctly at all. Others had mixed success. Now, with the partial disable registry entry not working, virtual domain controllers are the one true exception to the “every guest” rule. The Hyper-V Time Synchronization Service must be disabled for the PDC emulator. If not, and its host ever loses time, then the entire domain will eventually shift and won’t recover on its own.

Worse, during my testing, a shift of more than a few minutes caused failures in things such as remote desktop connections until the entire domain had picked up on the new time. While not quite as critical, time synchronization should also be disabled for any other virtualized domain controllers. The final configuration should follow these rules:

• The PDC emulator, whether virtual or physical, should be synchronizing its time from a known valid external source, such as authoritative Internet-based systems or a dedicated hardware clock.
• The PDC emulator, if virtualized, must not be synchronized to the hypervisor.
• While not as critical, other virtualized domain controllers besides the PDC emulator should also not be synchronized to the hypervisor. If left at defaults, they will synchronize directly from the PDC emulator.
• All hypervisors’ management operating systems should synchronize from the domain.
• All other guests should synchronize from the hypervisor.
• As indicated in the linked article, all physical machines should synchronize from the domain.

Time Sync in Hyper-V

The problem with disabling time synchronization for domain controllers is when they go through a Save State operation. Since the time synchronization is disabled, their clocks aren’t synchronized unless they’re rebooted. They will, of course, attempt to update at their next interval, but there are two issues with that. First, if the skew is too great, they will keep their incorrect time. Second, they will hand out the incorrect time to any client that attempts to synchronize with them. That’s why I always kept time synchronization partially disabled up until I learned that it doesn’t work anymore.

It’s easy to say that you’ll never save your domain controllers; I believe you. But, Hyper-V might decide to save them when you reboot their host(s). If the host is clustered and the virtual domain controller isn’t, you can almost guarantee that’s exactly what will happen. This is not meant to scare you or talk you out of virtualizing domain controllers. It’s just something you need to be aware of. If this is all set up correctly and you notice your clocks losing time, you just do what you would have done in the physical world: check the PDC emulator.

Disclaimer

I performed a substantial amount of testing on all of these claims prior to and during the crafting of this article. However, the Windows Time service has a track record of occasionally displaying erratic behavior. It is possible that some of my findings are not entirely accurate. It is also possible that my findings are 100% accurate but that not everyone will be able to duplicate them with 100% precision. If working with any time sensitive servers or applications, always take the time to verify that everything is working as expected.

If you encounter results that are different to mine, let me know by leaving a comment below!