NTP Pool Reaches 1000 Servers, Needs More

hgerstung writes "This weekend the NTP Pool Project reached the milestone of 1000 servers in the pool. That means that in less than two years the number of servers has doubled. This is happy news, but the 'time backbone' of the Internet, provided for free by volunteers operating NTP servers, requires still more servers in order to cope with the demand. Millions of users are synchronizing their PC's system clock from the pool and a number of popular Linux distributions are using the NTP pool servers as a time source in their default ntp configuration. If you have a static IP address and your PC is always connected to the Internet, please consider joining the pool. Bandwidth is not an issue and you will barely notice the extra load on your machine."

Google

This sounds like a job for Google.

Seriously. They are working to own every other bit of information. Why not "own" the method by which machines maintain time by throwing a thousand machines at it (an insignificant number compared to the 500k or more that make up their own server farm).

Re:

Why not "own" the method by which machines maintain time by throwing a thousand machines at it

A thousand machines all on one bit of network does little good. These need to distributed around the globe.

Re:

Google's server farms are distributed around the world. both coasts and in between as well as Ireland, Belgium and elsewhere.

Re:Google

Well, unfortunately DynDNS will not work here, simply because the clients will resolve the IP address once at startup and then stick to it (it will not be re-resolved later). The NTP Project is working on that, but currently there is no chance to use DynDNS or even "pretty static" IPs ... Heiko

Re:

Well, the concept is this:

You can have five billion machines all hitting a single "atomic clock". Or you could have a few servers hit it and hundreds of servers that sync up with *those* servers. Seriously, even in cases where all machines of an organization or university or corporation sync off one single internal NTP server (which itself them hits one of the servers in this pool, I'd presume) -- you're still talking about billions of machines that need to have the proper time synced on a daily bases acros

huh?

"Bandwidth is not an issue and you will barely notice the extra load on your machine."

If that is the case, why do they need more servers?

Re:

latency. The time you get back from the NTP server is the time the server sent the request. The client has to count the time it took to get a response and use that as a fudge factor. More servers means your client can find a closer server and minimize the transport time.

Re:huh?

No, the network time protocol accounts for latency and eliminates its influence almost completely as long as the latency is roughly symmetric, which it usually is for small packets.

Re:

It depends on what kind of accuracy you want. In the NTP papers, they discovered an error proportional to RTT (~= ping time), and error for a single host was often not zero-mean. The symmetry assumption is at best a crude approximation, and modern networks have made it worse. So, having more servers to keep the average latency lower is always a good thing. Ideally every ISP should provide this for their clients, keeping commodity internet bandwidth to a minimum.

Re:huh?

Yes. There are lots of time-sensitive tasks that require at least second-accuracy, some that require accuracy that's greater than that.

The first thing that comes to mind is remote logging. If I have several machines logging to some remote machine somewhere (as you should on any non-trivial system, to make a log falsification more difficult), it makes log analysis a lot easier if I know that the timestamps in the log are accurate and consistent across machines. Particularly if you ever have to dig through a break-in (or what you think might be a break-in), or just user stupidity, where you want to match actions taken on one machine to results on another.

At the very least, you want to make sure that all the clocks on the machines are accurate to at least the smallest interval of time that you might have two timestamps on the log apart by. Or if that's not possible, at least within a span so that the same human-initiated command will be discernible across the system at the same time in the logs.

Other things that involve remote data-collection have the same issue. At the very least, you need to have all your computers set so that they're accurate to some factor that's less than the time between data collections. While "data collection" sounds esoteric, it could be something as simple as sending emails from one computer to another, or combining two stacks of digital photos taken from some webcams (if they're portables, that's a separate ball of wax).

Now, do most of these things require all of the computers in your home network to be individually pinging a Level 2 timeserver? No. It would work just as well to have your gateway router get the time from a timeserver, and then offer NTP broadcasts to your network, so that everything could just synchronize itself. You'd have high precision local time, for synchronization, and reasonable accuracy time to a national standard. But that's beyond most users, so most OSes just have each workstation take care of things on its own.

Re:huh?

There really should be an "Oops" button after you mod something; I've never done this myself but I've seen at least 2 or 3 of this type of message in the last few days.

Re:huh?

I've never done this myself but...

Well Doctor, I've got this friend who has a problem...

Re:

This is clearly OT, but in the "old" method of moderation, you had to select the choices and then hit the "moderate" button at the bottom of the page. So if you picked wrong, you could re-correct. My other issue with it is that if I go through an article, pick 5 nice posts, and then see a 6th post, I can't take one of the first moderations and give it to the new post instead.

Re:

(Argh, crap - tried moderating this 'Interesting' and managed 'Offtopic' instead. Sorry - undoing all my moderation for this article. Please ignore this message!)

I modded this post Off Topic and I meant it!

Oh shit, did I just post?

Re:huh?

"Bandwidth is not an issue and you will barely notice the extra load on your machine."

If that is the case, why do they need more servers?

If I understand it right bandwidth isn't an issue because they can tailor how much of the pool load goes to your machine. When someone queries the pool their ntp client does a DNS query to pool.ntp.org. The pool's DNS server semi randomly returns the IP address of one of the volunteer servers in the pool. If you tell the pool operators that you have only a little bandwidth then the pool DNS server will only return your IP address say one tenth as often as it does the IPs for the high traffic servers. This allows you to decide how much load you're willing to bear. Even if the pool is overloaded, your machine doesn't have to be.

Re:

"If I understand it right bandwidth isn't an issue because they can tailor how much of the pool load goes to your machine."

Yes and no. Besides the jerks who hammer servers, the bandwidth problem is one of accumulation. Even if you're in the DNS rotation for 15 minutes, you'll pick up clients, and those clients may not go away anytime soon. When I left the pool a few years ago, I didn't shut down the server right away, and found that two months after my IP was no longer in rotation, I was still getting tr

Free GPS time equipment!

I must mention that right now by signing up for the pool now you also have a chance to get some really cool time keeping equipment [ntp.org]. :-)

Re:Free GPS time equipment!

Only if you already have a server in the NTP pool. And even then, they only get freebies to give away every few months. Then they decide from the applicants who gets them. So your chances are probably pretty slim. The long-timers will get them first.

Re:

"The long-timers will get them first."

There must be a joke hidden in this sentence somewhere...

didnt they think of this?

Bandwidth is not an issue and you will barely notice the extra load on your machine. I think if their servers can't keep up, you *will* notice the load, at least until enough join.

do they have no way of routing/limiting traffic so that it isn't normally noticeable?

Re:

The NTP protocol gives very limited ways of limiting it, so short of just closing down if we can't add servers as fast as traffic is added, no - there isn't much we can do.

The vendor program [ntp.org] is one way we're trying to get more control, but all else being equal - more servers helps.

Better way To Do This

I think that a better method could be used to encourage diversity. They should take a page from the root DNS servers, or Akamai. Either use BGP anycast, which is what most of the root dns servers do now, which will probably never happen. Or, have a zone that network carriers should use on their local DNS servers, and by way of DNS lookups, encourage their customers to use. ntp.org has a default set of values for say time.overload.ntp.org that reflects the current pool. But I, as an ISP make my DNS servers directly answer queries for overload.ntp.org, and make entires such as:

time IN A 1.2.3.4
time IN A 1.2.3.5

where 1.2.3.4 and 1.2.3.5 are ntp servers on my local network. I don't allow people off my network to query my DNS servers for recursive queries, and the ntp.org DNS servers never tell anyone to use my name servers for this space anyways. This would mean that only my customers that use my DNS servers (about 99%) of them, would ever get answers for my time servers, and they would definitely be close.

And anyone whose network carrier doesn't bother to set this up, still gets generic answers from ntp.org. This works much better than just a big pool full of 1000 servers worldwide, even if you bother to use the country code dns regions, you still aren't always getting an ntp server anywhere near you.

Re:Better way To Do This

You are absolutely correct that if network carriers provided NTP services properly on their nets, then the pool wouldn't be necessary. If you go through Usenet archives you can read the history and discussion behind the creation of the pool. Everyone realizes that the pool is an inferior solution that we are stuck with because the network access service providers won't do their job.

The next time I've got a free two hours for self-torture, I'll call Verizon Business customer support and ask them about NTP service. (It will take that long to be transfered to someone who understands the question.)

Re:

Wrong solution.

Poisoning DNS is never a good idea for public (including ISP) use. Please don't suggest this.

A far better method is to use DHCP to assign one or more local NTP servers, just as is done for DNS servers and other things which may vary from network to network.

DHCP, as a protocol, supports this usage just fine. Various DHCP client implementations also support this by default[1].

All that needs to happen is for the ISP to actually run ntpd (which is trivial), and configure the DHCP server to star

Re:Better way To Do This

A few thoughts...

Unlike a partnership with Akamai, there's no compelling monetary reason for an ISP to offer their own NTP server. Therefore, the easiest (least costly) solution -- at the ISP end -- is probably the most likely to win. Adding a line to dhcpd.conf is probably easier than configuring BIND to issue lies.

And while not everyone uses DHCP, they certainly have some mechanism for communicating things like DNS server addresses, default gateways, and so on. Using that same mechanism (be it DHCP, bootp, or snail mail) to inform the customer of the local NTP server seems trivial in every instance I can think of.

Clients that don't care will obviously ignore this data, but customers who do care can modify their client software accordingly.

Eventually (as in, within the MTBF of a Linksys router), if it ever gains any foothold, clients will use this data by default.

But I guess the most glaring problem to me is that, surprisingly often, the ISP's own DNS servers are slow and/or broken, and overridden. Much of Roadrunner's network is, for instance, assigned DNS servers which are so slow that when browsing the web, more time is spent on simple DNS lookups than on downloading and rendering content.

This, in turn, causes people like me to use a different DNS server on a different network. In my case, I use Level3's DNS at 4.2.2.1 because it is easy to remember and quite fast. Your suggestion ties together DNS and NTP inextricably, such that I'd also be using L3's NTP server by default, when all I really wanted was different DNS.

I don't want a solution to one network problem to have cascading effects on other network services. There's enough of that in the world already.

Remember, the whole point of this is to eliminate end-user manual NTP client configuration, and reduce network load, while offering the useful service of providing accurate time. And I can only hope that, after all of this, network-attached devices of all types will use this mechanism (whatever it is) to automatically derive time from a nearby NTP server.

Some of these devices will be reconfigurable to use whatever NTP server the user wants (certainly, my Linux box is), but hopefully some simpler devices will not be (think print server, networked DVR, WiFi LCD picture frame, or other minimally-configured box).

If a standard method for propogating NTP server names to end-users ever does get implemented, I shouldn't have to run a local copy of BIND and my own regimine of poison, just to allow independant settings for both DNS and NTP servers.

But that's all just my opinion. It is probably wrong. :)

Re:

Unlike a partnership with Akamai, there's no compelling monetary reason for an ISP to offer their own NTP server. Therefore, the easiest (least costly) solution -- at the ISP end -- is probably the most likely to win. Adding a line to dhcpd.conf is probably easier than configuring BIND to issue lies.

Actually, having some local source of consistent time is pretty much a no brainer on any network that wants logs to be sane, NFS to work correctly, or has any services that require more than one server to run. I really don't mind running them, and letting my customers know. Oh, customer computers that have an accurate clock are far less likely to be obnoxious as all hell when they get email from the future, or way in the past. No, I am not kidding, time.microsoft.com is a good thing in that it got rid of o

Re:

Can you point me to a location to specify a Stratum 1 server? I have access to colo space as well as roof rights for a GPS antenna. I would be happy to host a Stratum 1 or Stratum 2 time server to help the NTP cause.

Re:Better way To Do This

http://tf.nist.gov/service/time-servers.html [nist.gov]

All organizations interested in possibly hosting a NIST Internet Time Service server are invited to contact Time and Frequency Division Chief Thomas O'Brian for more information, including a description of the equipment that the organization must have available and a discussion of the other technical qualifications necessary to host a server: [e-mail] .

GPS time with OpenBSD

If you grab a USB GPS receiver, I used a $60 BU-353 [google.com], you can have accurate time easily.

openbsd# dmesg | tail -3
uplcom0 at uhub0 port 2
uplcom0: Prolific Technology Inc. USB-Serial Controller, rev
1.10/3.00, addr 2
ucom0 at uplcom0
openbsd# nmeaattach cuaU0
openbsd# sysctl -a | grep hw.sensors
hw.sensors.nmea0.timedelta0=-328.10115 9 secs (GPS), OK, Tue May 15 19:48:46.898
openbsd# echo "sensor nmea0" > /etc/ntpd.conf
openbsd# echo "listen on *" >> /etc/ntpd.conf
openbsd# ntpd -ds
ntp engine ready
sensor nmea0 added
sensor nmea0: offset 328.097637
set local clock to Tue May 15 19:57:46 PDT 2007 (offset 328.097637s)
sensor nmea0: offset 0.020612
...

Re:GPS time with OpenBSD

Actually ... The USB latency can be pretty bad, so it's likely you'd get better time from a well-picked internet time server. You'd definitely get MUCH better time with a proper PPS (Pulse Per Second) time keeping GPS receiver or variations of that [meinberg.de].

Re:

...and the nice thing is that there is already a driver for OpenBSD [meinberg.de] for that device, thanks to the opensource driver policy of Meinberg. Heiko

Re:GPS time with OpenBSD

In addition to the latency of USB, the nmea output of a GPS unit may not be very accurate. Go for a GPS with pulse per second if you can find one for a reasonable price. A while back I was checking the chipset specs for the cheap GPS receivers to find one with a pulse per second output. I found some but I forgot which ones they were. Of course you would have to open the case and do a little soldering. I'm not sure how you would hook it up to your server once you got the pulse per second out. I think maybe to one of the pins on the serial port that would trigger an interrupt.

Under OpenBSD I've gotten much more stable timekeeping by recompiling the generic kernel with only one simple change. I set the processor type to 586 or 686 as the case may be. Specifically in the /usr/src/sys/arch/i386/conf/GENERIC file I removed "option I486_CPU" and "option I686_CPU" so that it would be correctly configured for my pentium 166 cpu. I think the pentium has some time keeping functions the 386 and 486 didn't have. Although I haven't found the parts of the kernel code where this change does its magic.

atomic clock to PC connection?

Like a lot of guys here, we have an atomic self setting clock that works from radio broadcast. They are cheap now and work very well. What I am wondering is, do they make some sort of attachment clock, so it can set your computer's time that way? Like an atomic clock/usb cable connect thingee? Seems like if they did, we wouldn't need all these NTP servers, the government does the radio broadcasting and it is as accurate as it gets.

Re:

do they make some sort of attachment clock, so it can set your computer's time that way?

Of course they do. Anyone who has ever setup ntpd should know that quite well. The default/example config file is STREWN with examples of using hardware clocks... So much so it's difficult to figure out how to set it up to sync to other servers via the network.

From the man page:

The NTP Version 4 daemon supports some three dozen different radio, satellite and modem reference clocks plus a special pseudo-clock used for

Re:

USB latency is generally too high for this.

We bought one from these guys.

http://www.timetools.co.uk/ [timetools.co.uk]

They are a lot more than $20. Now I am just waiting for the customer to
provide another hole in the roof so we can get our GPS antenna outside.

Windows Time

For those interested, you can change your Windows time servers to NTP servers in the registry here: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Cur rentVersion\DateTime\Servers]

Re:

Or (in winXP as admin account at least) you can doubleclick on the clock, go to internet time, and enter the ntp server there.

Re:

Not really. The more time sources you have, the more precise your estimate of the time will be, since you'll be able to cancel out disparate network jitter better.

--Joe

Re:load

Their servers can keep up just fine, or at least the one I run can. My stats show 1GB per month traffic and the ntpd process taking about 1 minute/day of processor time. That has been relatively constant over the year or so the server has been in the pool.

I think this is just a case of more==better. A bigger pool means more people can use their local zone instead of the global zone, the whole system can handle more clients, less load on servers means even more may be willing to join, ...

Seriously, it's not that big a deal. Just thow your server into the pool and forget about it.

Re:load

Just thow your server into the pool and forget about it.

Isn't that a bit extreme? Should I maybe waterproof it first?

Re:load

Nah. I hear liquid cooling is great for overclocking.

Re:

When being an NTP server you want the clock to be as accurate as possible.
The server is often locked to other servers and/or to local radio clock receivers.
In a physical machine, there is an accurate hardware timer that is used as the incrementing clock (at micro- or nanosecond rate) and which is frequency locked to the references.
Such hardware does not really exist in the virtual machine, it is emulated, and this emulation is not very good even when you sync to the host.
It is good enough for "wristwatch ti

Re:NTP Isn't Accurate

Hi AC,

The NTP Pool monitors the servers and only uses those with accurate time. A server drifting several seconds off would be taken out of the pool until it got fixed.

Also, the NTP daemons are Quite Good at ignoring the servers with Bad Time Keeping.

Using ntpd with the pool servers will give you much much much more accurate time than trying to set it manually after looking at a web page.

    - ask

Re:

Please name one ntp server in the pool that it off by more than .5 seconds? The vast majority are accurate to under .1 seconds. I do not believe that the AC who said these aren't accurate understands how NTP works.

Re:NTP Isn't Accurate

You do realize all those times are in milliseconds [hp.com], right? So, the largest difference between your computer and one of the servers is 27 milliseconds, and the leading "-" in front of the hostname means it isn't even being used for synchronization. Also, either you didn't let it settle for a while, or your local computer clock is inaccurate, because you are still polling once a minute. A "healthy" computer clock will lower the poll frequency significantly if the local and estimated net clocks don't jitter much. I did have one machine with a clock that just sucked, so I had to make sure it was a client of another machine which could act as the timekeeper on my home network, and make sure it polled the timekeeper often.

Personally, I don't use the pool, and instead find some stable servers near to my ISP. But you really can't argue against the NTP pool as a default setup, since it works everywhere. So, if it bothers you, find some closer servers or convince your ISP to run a time server (many are already doing so). In both cities I've lived in, I was able to find an open stratum-1 server with a ~20ms delay (Thank you GPS).

Re:

3 Minutes?!?

I have my machines synced via ntp. ntpq reports than I'm no more than 3ms out of sync with a stratum 1 time server (9ms out of sync with UNSO) and that server is synced with GPS and USNO which as you said is never more than .0001ms out of sync with UTC.

Eye-balling like you described I can verify that I am within 2000ms of http://time.gov/ [time.gov]. I think perhaps that that website may have had issue on the date you saw it being 3 minutes different than what NTP provided.

I'd show

Not so much the chips, but the timebase crystals..

The component that actually determines the stability and accuracy of the real-time clock in your PC is the timebase crystal, not the RTC chip itself.

Like every other component in mass-market electronic gear, it is chosen with minimum cost as the primary consideration. Such "value engineering" also has done away with the tiny trimmer capacitor that used to be present on most motherboards, which could be used (along with a frequency counter) to tweak the oscillator frequency for better accuracy.

For real accuracy, the timebase oscillator needs to be kept at a constant temperature, which isn't possible in a PC that gets turned on and off. Ideally, the crystal (or the entire oscillator circuit) is enclosed in a package equipped with a heater element and temperature sensor, and kept at a constant temperature. Such a circuit is called an OCXO, or Oven Compensated Crystal Oscillator, and is standard equipment on laboratory grade equipment like frequency counters and signal generators.

Re:Not so much the chips, but the timebase crystal

I'm sure you're right about the low cost of the PC RTC components. However, I still don't understand why I've long been able to buy a watch for $2-$15 that keeps better time than any PC I've had.

Re:Why not make it peer-to-peer

Because the number one rule of infrastructure is, "never trust the client." Peer to peer networks are full of malware/trojans/assholes, and generally far too easy to infiltrate with unwanteds.

And while I agree with your sentiment that I can live time being off by a little, I also run a lot of UNIX servers that use NFS heavily. I am far more concerned with all of my network machines agreeing on what time it is on my network, than being correct with the world. I sync two dedicated time servers to the ntp.org pools (soon to be three), and all my internal hosts sync to those two. Being synced with the world is very handy, and generally I would prefer it. But being in agreement with myself is non-negotiable, I just need it.