I found the most interesting part of the NIST outage post [1] is NIST's special Time Over Fiber (TOF) program [2] that "provides high-precision time transfer by other service arrangements; some direct fiber-optic links were affected and users will be contacted separately."
I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])? Telecom providers synchronizing 5G clocks for time-division duplexing [4]? Google/hyperscalers as input to Spanner or other global databases?
Seriously fascinating to me -- who would be a commercial consumer of NIST TOF?
I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. Sub-microsecond latency, yes, but when that mattered it was in order to do something as soon as possible rather than as close as possible to Wall Clock Time X.
Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you.
The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
> I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems.
> The required accuracy (Tables 1 and 2 in that document)
no, Tables 1 and 2 say divergence, not accuracy
accuracy is a mix of both granularity and divergence
regardless, your statement before:
> The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
My guess would be scientific experiments where they need to correlate or sequence data over large regions. Things like correlating gravitational waves with radio signals and gamma ray bursts.
This page: https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine.
However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody.
Nitpick: UTC stands for Coordinated Universal Time. The ordering of the letters was chosen to not match the English or the French names so neither language got preference.
Not exactly the topic of discussion but also not not on topic: just wanted to sing praise for chrony which has performed better than the traditional os-native NTP clients in our testing on a myriad of real and virtualized hardware.
I'm missing the nuance or perhaps the difference between the first scenario where sending inaccurate time was worse than sending no time, versus the present where they are sending inaccurate time. Sorry if it's obvious.
The 5us inaccuracy is basically irrelevant to NTP users, from the second update to the Internet Time Service mailing list[1]:
To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.
> Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet
How do those other applications obtain the precise value they need without encountering the Internet issue?
Yes, either Rb, Cs, or H standards depending on which GNSS system you're using.
For the most critical applications, you can license a system like Fugro AtomiChron that provides enhanced GNSS timing down to the level of a few nanoseconds. There are a couple of products that do similar things, all based on providing better ephemerides than your receiver can obtain from the satellites themselves.
A lot of organizations also colocate timing equipment near the actual clocks, and then have 'dark fiber' between their equipment and the main clock signals.
Then they disperse and use the time as needed.
According to jrronimo, they even had one place splice fiber direct between machines because couplers were causing problems! [1]
If I put my machine near the main clock signal, I have one clock signal to read from. The comment above was asking about how to average across many different clocks, presumably all in different places in the globe? Unless there's one physical location with all of the ones you're averaging, you're close to one and far from all the others so how is it done without the internet?
Can you do PTP over the internet? I have only seen it in internal environments. GPS is probably the best solution for external users to get time signals with sub-µs uncertainties.
It's a good question, and I wondered the same. I don't know, but I'd postulate:
As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.
When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.
If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.
Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.
In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
> […] where sending inaccurate time was worse than sending no time […]
When you ask a question, it is sometimes better to not get an answer—and know you have not-gotten an answer—then to get the wrong answer. If you know that a 'bad' situation has arisen, you can start contingency measures to deal with it.
If you have a fire alarm: would you rather have it fail in such a way that it gives no answer, or fail in a way where it says "things are okay" even if it doesn't know?
I worked at Altera (FPGA supplier) as the Ethernet IP apps engineer for Europe for a few years. All the big telecoms (Nokia, Ericsson, Cisco, etc) use Precision Time Protocol (PTP) in some capacity and all required clocks to be ns levels for accuracy. Sometimes as low a 10ns at the boundary. Any imperfection in the local clock directly converts into timestamp error, and timestamp error is what limits PTP synchronization performance. Timestamps are the fundamental observable in PTP.
Quantization and jitter create irreducible timestamp noise.
That noise directly limits offset and delay estimation.
Errors accumulate across network elements and internal clock error must be much smaller than the system requirement.
I think most people would look at the error and think "what's the big deal" but at all the telecoms customers would be scrambling to find a clock that hasn't fallen out of sync.
We don't use NTP, but for robotics, stereo camera synchronization we often want the two frames to be within ~10us of eachother. For sensor fusion we then also need a lidar on PTP time to be translated to the same clock domain as cameras, for which we also need <~10us.
We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
> We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
This doesn't seem right to me. NTP with default settings should be monotonic. So no jumps. If you disable it Linux enters 11-minute mode, IIRC, and that may not be monotonic.
Pedantically, a monotonic function need not have a constant first derivative. To take it further, in mathematics it is accepted for a monatomic function to have a countable number of discontinuities, but of course in the context of a digital clock that only increments in discrete steps, that’s of little bearing.
But that’s all besides the point since most sane time sync clients (regardless of protocol) generally handle small deviations (i.e. normal cases) by speeding up or slowing down the system clock, not jumping it (forward or backward).
We use nanosecond precision driven by GPS clocks. That timestamp in conjunction with star tracker systems gives us reliable positioning information for orbital entities.
Lightning detection. You have a couple of ground stations with known positions that wait for certain electromagnetic puses, and which record the timestamps of such pulses. With enough stations you can triangulate the location of the source of each pulse. Also a great way to detect nuclear detonations.
There is a german club that builds and distrubutes such stations (using GPS for location and timing), with a quite impressive global coverage by now:
(Assuming "precision" really meant "accuracy") The network equipment I work on requires sub microsecond time sync on the network for 5G providers and financial trading customers. Ideally they'd just get it from GPS direct, but that can be difficult to do for a rack full of servers. Most of the other PTP use cases I work with seem to be fine with multiples of microseconds, e.g. Audio/Video over the network or factory floor things like PLCs tend to be find with a few us over the network.
Perhaps a bit more boring than one might assume :).
At a previous role, we needed nanosecond precision for a simulcast radio communications system. This was to allow for wider transmission for public safety radio systems without having to configure trunking. We could even adjust the delay in nanoseconds to move the deadzones away from inhabited areas.
We solved this by having GPS clocks at each tower as well as having the app servers NTP with each other. The latter burned me once due to some very dumb ARP stuff, but that's a story for another day.
Nuclear measurements, where the speed of a gamma ray flying across a room vs a neutron is relevant. But that requires at least nanosecond time resolution, and you’re a long way from thinking about NTP.
Yes, but always got it from GPS so presumably they'd be off about the same amount.
Distributed sonar, allows placing receivers willy-nilly and aligning the samples later.
Remote microphone switching - though for this you wouldn't notice 5us jitter, it's just that the system we designed happened to have granularity that good.
I believe LTE and 5G networks require it to coordinate timeslots between overlapping cells. Of course, they can use whatever reference they want, as long as all the cells are using the same one - it doesn't have to be UTC. Some (parts of) networks transmit it across the network, while others have independent GPS receivers at each cell site.
Synchronization is also required for SDH networks. Don't know if those are still used.
Someone else referenced low power ham radio modes like WSPR, which I also don't know much about, but I can imagine they have timeslots linked to UTC and require accuracy. Those modes have extremely low data rates and narrow bandwidths, requiring accurate synchronization. I don't know if they're designed to self-synchronize, or need an external reference.
When multiple transmitters are transmitting the same radio signal (e.g. TV) they might need to be synchronized to a certain phase relationship. Again, don't know much about it
A database like Google Spanner has higher latency in proportion to the uncertainty about the time. Driving the time uncertainty down into the microsecond range, or lower, keeps latency low.
I mean, we routinely benchmark things that take microseconds or less. I've seen a 300 picosecond microbenchmark (single cycle at 3GHz). No requirement that absolute time is correct, though.
It sounds like GPS, and thus a GPS-based stratum 1 server, uses these time servers, but they were successfully failed over:
> Jeff finished off the email mentioning the US GPS system failed over successfully to the WWV-Ft. Collins campus. So again, for almost everyone, there was zero issue, and the redundancy designed into the system worked like it's supposed to.
So failures in these systems are potentially correlated.
The author mentions another solution. Apparently he runs his own atomic clock. I didn’t know this was a thing an individual could do.
> But even with multiple time sources, some places need more. I have two Rubidium atomic clocks in my studio, including the one inside a fancy GPS Disciplined Oscillator (GPSDO). That's good for holdover. Even if someone were jamming my signal, or my GPS antenna broke, I could keep my time accurate to nanoseconds for a while, and milliseconds for months. That'd be good enough for me.
The CSACs that I have in a couple devices are 'atomic', and use Rubidium, but they're a bit lower accuracy than Cesium clocks [1] or Hydrogen Masers [2].
There are a few folks on the time-nuts mailing list who own such exotic pieces of hardware, but those are pretty far out of reach for most!
Be aware that there are members of the NTP pool with less-than-honorable intentions and you don't get to pick-and-choose. Yes, they all should provide the time, but they also get your IP address.
For example: unlike the IPv4 space, the IPv6 space is too big too scan, so a number of "researchers" (if you want to call them that) put v6-capable NTP servers in the NTP pool to gather information about active v6 blocks to scan/target.
Is this one of those extraordinary claims that requires evidence? Or is it generally true that there are homey-pots in many of these services (NTP, mirrors, etc)
Most places that need accurate time get it from GPS. That is 10-100 ns.
Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers.
Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.
Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS.
(If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)
I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])? Telecom providers synchronizing 5G clocks for time-division duplexing [4]? Google/hyperscalers as input to Spanner or other global databases?
Seriously fascinating to me -- who would be a commercial consumer of NIST TOF?
[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...
[2] https://www.nist.gov/pml/time-and-frequency-division/time-se...
[3] https://www.finra.org/rules-guidance/rulebooks/finra-rules/4...
[4] https://www.ericsson.com/en/blog/2019/8/what-you-need-to-kno...
Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you.
The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
mifid ii (uk/eu) minimum is 1us granularity
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:...
The required accuracy (Tables 1 and 2 in that document) is 100 us or 1000 us depending on the system.
no, Tables 1 and 2 say divergence, not accuracy
accuracy is a mix of both granularity and divergence
regardless, your statement before:
> The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
is not true
Where does it say these are commercial consumers?
https://en.wikipedia.org/wiki/Schriever_Space_Force_Base#Rol...
> Building 400 at Schriever SFB is the main control point for the Global Positioning System (GPS).
To say NIST was off is clickbait hyperbole.
This page: https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine.
However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody.
IMHO, most applications should use pool.ntp.org
(See https://docs.cloud.google.com/spanner/docs/true-time-externa...)
I defer to the experts.
To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.
[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...
How do those other applications obtain the precise value they need without encountering the Internet issue?
They do not use the Internet: they use local (GPS) clocks with internal high-precision clocks for carry-over in case GNSS signal is unavailable:
* https://www.ntp.org/support/vendorlinks/
* https://www.meinbergglobal.com/english/products/ntp-time-ser...
* https://syncworks.com/shop/syncserver-s650-rubidium-090-1520...
* https://telnetnetworks.ca/solutions/precision-time/
https://en.wikipedia.org/wiki/Rubidium_standard
For the most critical applications, you can license a system like Fugro AtomiChron that provides enhanced GNSS timing down to the level of a few nanoseconds. There are a couple of products that do similar things, all based on providing better ephemerides than your receiver can obtain from the satellites themselves.
You can get AtomiChron as an optional subscription with the SparkPNT GPSDO, for instance (https://www.sparkfun.com/sparkpnt-gnss-disciplined-oscillato...).
Then they disperse and use the time as needed.
According to jrronimo, they even had one place splice fiber direct between machines because couplers were causing problems! [1]
[1] https://news.ycombinator.com/item?id=46336755
Alternate sources include the GPS signal, and the WWVB radio signal, which has a 60kHz carrier wave accurate to less than 1 part in 10^12.
As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.
When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.
If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.
Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.
In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
When you ask a question, it is sometimes better to not get an answer—and know you have not-gotten an answer—then to get the wrong answer. If you know that a 'bad' situation has arisen, you can start contingency measures to deal with it.
If you have a fire alarm: would you rather have it fail in such a way that it gives no answer, or fail in a way where it says "things are okay" even if it doesn't know?
I think most people would look at the error and think "what's the big deal" but at all the telecoms customers would be scrambling to find a clock that hasn't fallen out of sync.
We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
This doesn't seem right to me. NTP with default settings should be monotonic. So no jumps. If you disable it Linux enters 11-minute mode, IIRC, and that may not be monotonic.
But that’s all besides the point since most sane time sync clients (regardless of protocol) generally handle small deviations (i.e. normal cases) by speeding up or slowing down the system clock, not jumping it (forward or backward).
https://en.wikipedia.org/wiki/Star_tracker
There is a german club that builds and distrubutes such stations (using GPS for location and timing), with a quite impressive global coverage by now:
https://www.blitzortung.org
Perhaps a bit more boring than one might assume :).
We solved this by having GPS clocks at each tower as well as having the app servers NTP with each other. The latter burned me once due to some very dumb ARP stuff, but that's a story for another day.
Timekeeping starts to become really hard, often requiring specialized hardware and protocols.
edit: also the linked slides in TFA
Distributed sonar, allows placing receivers willy-nilly and aligning the samples later.
Remote microphone switching - though for this you wouldn't notice 5us jitter, it's just that the system we designed happened to have granularity that good.
Synchronization is also required for SDH networks. Don't know if those are still used.
Someone else referenced low power ham radio modes like WSPR, which I also don't know much about, but I can imagine they have timeslots linked to UTC and require accuracy. Those modes have extremely low data rates and narrow bandwidths, requiring accurate synchronization. I don't know if they're designed to self-synchronize, or need an external reference.
When multiple transmitters are transmitting the same radio signal (e.g. TV) they might need to be synchronized to a certain phase relationship. Again, don't know much about it
Precision Time Protocol gets you sub-microsecond.
https://en.wikipedia.org/wiki/Precision_Time_Protocol
NTP at NIST Boulder Has Lost Power
https://news.ycombinator.com/item?id=46334299
Suggestions from the community for more reliable alternatives?
You still can...
If you're that considered about 5 microseconds: Build your own Stratum 1 time server https://github.com/geerlingguy/time-pi
or just use ntppool https://www.ntppool.org/en/
> Jeff finished off the email mentioning the US GPS system failed over successfully to the WWV-Ft. Collins campus. So again, for almost everyone, there was zero issue, and the redundancy designed into the system worked like it's supposed to.
So failures in these systems are potentially correlated.
The author mentions another solution. Apparently he runs his own atomic clock. I didn’t know this was a thing an individual could do.
> But even with multiple time sources, some places need more. I have two Rubidium atomic clocks in my studio, including the one inside a fancy GPS Disciplined Oscillator (GPSDO). That's good for holdover. Even if someone were jamming my signal, or my GPS antenna broke, I could keep my time accurate to nanoseconds for a while, and milliseconds for months. That'd be good enough for me.
There are a few folks on the time-nuts mailing list who own such exotic pieces of hardware, but those are pretty far out of reach for most!
[1] https://www.microchip.com/en-us/products/clock-and-timing/co...
[2] https://www.microchip.com/en-us/products/clock-and-timing/co...
For example: unlike the IPv4 space, the IPv6 space is too big too scan, so a number of "researchers" (if you want to call them that) put v6-capable NTP servers in the NTP pool to gather information about active v6 blocks to scan/target.
Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers.
Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.
Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS.
(If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)
For instance, time-a-wwv.nist.gov.
One should configure a number of different NTP sources instead of just a single host.