It's a good question, and I wondered the same. I don't know, but I'd postulate:
As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.
When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.
If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.
Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.
In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
> The boss doesn't see that you can't properly paste a piece of code in the chat
Of all my many gripes with Teams, it usually handles code surprisingly well. Single `inline` and triple backtick blocks usually render as you'd expect.
OneNote on the other hand doesn't support a code-block at all, and is worse (if you can believe it) than storing cli commands in Word docs.
When I paste code into the native MacOS Teams chat, my peers using Window Teams see a literal black box. I wish it worked! I really do. Or we all had MacBooks or Linux desktops.
I wonder how much of it is low quality adapters vs poor drivers. Whatever Bluetooth module they used in my £20 Chinesium car stereo connects quicker and works with less niggles than any other device I've tested in the last 20 years.
> I've often thought about that when there's a work crisis: If I'm the second on the scene, what can I do to support those fighting the fire right now, before jumping in.
A lot of it depends on the size and skill-set of your team and the escalation routes available to you, but in general (and off the top of my head):
- Get the first people on scene to give a summary of the problem as they know it. Make sure everyone actually agrees on what the problem is and what symptoms have been observed. Understand what areas people are currently investigating and make sure they aren't trampling over each other or actually making the situation worse [1]
- Make sure the situation hasn't evolved whilst the first on scene have been investigating the initial symptoms. It's easy to get lost in the weeds digging into a handful of monitoring alerts only to look up and realise there's now 300 and the original problem is only a small part of what's going on.
- If there isn't one already and you're not better doing something else, become incident commander. When done right it's an extremely important and useful role.
- Take over external communication and protect the team from distractions
- Start assessing escalation options
- Take copious notes and keep a timeline
- Act as a shared memory and keep people honest
- Have a less panicked, wider (non minutia) view of the problem
- Start collating and pulling up documentation/schematics so the people at the coalface can quickly query it rather than getting distracted searching for it.
- Be ready to jump, for when someone inevitably asks "can someone check..." or "does anyone know"
- Keep track of the "shared truth" of the incident as it evolves. What have we witnessed, what do we believe is the cause, _why_ do we believe that? Have we invalidated anything, do we need to reassess, are we sure logical lynchpins aren't confirmation bias or dyslexia?
- Onboard new people and hand over if appropriate.
Being at the coalface when it's on fire is a very different view of the world to watching other people panic and singe their fingers. It's also very easy to get lost in a chain of technical problems [2] when it's mostly irrelevant to the wider picture.
If you get a moment, it can also be a good time to assess how useful your monitoring is during an actual event.
[1] "Hey, server x has flagged on monitoring and my ssh session is hung waiting for a login prompt!" I've been round the houses enough to know this is probably OOM and if I just wait, I'm likely to finally get in. I also know that saying this in a room of 20 technical people, means the server is now processing 22 new ssh sessions and now no one is getting anywhere.
[2] The famous Malcolm in the Middle intro where Hal is tasked with changing a lightbulb and ends up repairing the car. Except in my example the bulb is actually fine and there's a power cut we missed. https://www.youtube.com/watch?v=AbSehcT19u0
Except unpowered SD cards (and SSDs for that matter) don't claim to hold data for more than a couple of years.
I'm a big believer in thinking you have backups being worse than knowing you don't, so anything that encourages people to believe $(flash memory) is suitable as long-term cold storage is actually, really bad.
I agree there's no need to copy & wipe cards immediately, but treating them as "film" is inherently flawed and setting yourself up for failure. The amount of people that turn up in data recovery forums unable to access old, important, "backed up" (memory card/ssd on a shelf) photos is depressingly high.
It's something I've done myself in the past. First sort is because it needs to be sorted for uniq -c to count it proper, second sort because uniq doesn't always give the output in the right order.
I had a similar problem a few years back with a technophobe Grandma who wouldn't use whatsapp and a stingy aunt who'd moved to the states.
Used a Fritz!Box as it'll act as a DECT<->SIP server so my grandma could keep picking up her usual handset. Had a rule which matched the aunt's phone number so it got bounced via my FreeSwitch server rather than her PSTN, and some least-cost routing rules to pick a US based provider SIP provider with a +01 PSTN number so my aunt saw the incoming call looking local and could ring back if necessary at a cheap rate.
Honestly, the weird technical abominations we come up with to try and serve family members. The same Grandma went through a phase of falling, not being able to get up and then waiting hours for an ambulance to arrive. Would she tell the family? Would she sod. Was her regional ambulance service still dispatching jobs via pager message? You betcha. Did this end up with an SDR, POCSAG decoder and a regx looking for her postcode? Absolutely not; why would you ask such a thing?
Coming from the telecoms space, I was slightly amazed to see how little well known SCTP actually is in the networking world.
My old company/offices had site internet provided by one of the top 50 UK Managed Service Providers. They swapped out the on-site router not many years ago as the fibre to site was being upgraded from 100mbit to gigabit and so a new Juniper firewall with GBE ports was required.
Turns out the newer, faster, shinier, though albeit lower model numbered'd Juniper SRX fundamentally didn't support passing SCTP data and suddenly we lost access to all our remote stuff that used it. Ended up on a call with the MSPs Head of Networks (who was not a stupid person), but their opening gambit was "Are you sure you mean SCTP? Oh. What is that then?"
There was also numerous weird kernel bugs with implementations on CentOS 5, 6 and 7 which all would manage to get themselves into weird states where only a reboot would clear - not really what you want from a multi-endpoint, 'copes and recovers well from network weirdness' tunnelling protocol.
> Juniper SRX fundamentally didn't support passing SCTP data and suddenly we lost access to all our remote stuff that used it.
did you file a customer complaint for the device you bought not supporting basic internet protocols? If I look here it mentions "internet" but not TCP or UDP. I'd argue it's false advertising if it actually only supports a percentage of actual internet traffic.
It was one of those situations where the internet was part of the lease, and the property owners got the MSP to provide for multiple companies on the site. Sadly I wasn't a customer of either Juniper or the MSP, and it wasn't something they ever actually claimed to support.
Juniper themselves stated in the manual that this base model device didn't support SCTP, though on ever other level it was faster, more capable and more featureful than the mid-range but much older device it replaced. The MSP didn't have a clue that we (or anyone else for that matter) used SCTP so missed the single footnote mention that the command to enable SCTP forwarding might not be available on some base-level devices.
In their defence, I'm not sure _I'd_ have thought to check if SCTP was supported and I had it running on my network. It works over the internet, it's basically IP, how could it not be suppo---oh.
It did see a lot of use. I forgot to say, but the majority of 3G UMTS and 4G LTE radios (operator level and even down to tiny things like femtocells) required it to connect back to the $(core network) and encapsulated a mix of control and user data through it (though often also encapsulating SCTP in a VPN). That is to say, there were a lot of devices out there that used it in anger. If you made a call or used a cellular data session for over a decade, your traffic was likely getting encapsulated through it.
Bigger orgs for the most part use whatever vpn solutions their (potentially decade old) hardware firewalls support. Until you can manage and endpoint a Wireguard tunnel on Cisco, Juniper, Fortigate (etc) hardware then it's going to take a while to become more mainstream.
Which is a shame, because I have a number of problematic links (low bandwidth, high latency) that wireguard would be absolutely fantastic for, but neither end supports it and there's no chance they'll let me start terminating a tonne of VPNs in software on a random *nix box.
My understanding is that they're glitching the game to inject data into memory, in this case processor instructions. And then they use another glitch to jump to and execute that code. No modifications needed to the hardware or ROM, you're just being extremely selective about how to play the game to cause memory to get flipped.
As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.
When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.
If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.
Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.
In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...