Had a strange thought last year, that if a system runs too smoothly, eventually the knowledge of dealing with breakage will disappear, and when it inevitably breaks, everyone will be unprepared.
So a little bit of breakage is like a healthy "exercise."
Presumably, a functional system is supposed to have some kind of drills to fill in for that. But I have seen ones that don't!
This is known as the "Paradox of Automation." It's a real problem for many industries.
The OG paper on this is called "The Ironies of Automation" (Bainbridge, 1983). It's a clunky but fascinating read. I give it a skim every couple years and usually come away with a bit of a fresh take on the problem.
Yeah, I've lived that one. When your system is falling over all the time, you get good at quickly bringing it back. When you have the first failure in 2 years and you didn't do drills or etc, it takes a lot longer to get things going again.
Knowledge of how to fix the system comes from understanding how the system works, which does not necessarily require experience fixing it. Every problem ever fixed was at some point fixed without prior experience - if we can fix problems that have never happened before we can fix problems that haven't happened in a while.
Can also confirm that the process can be harrowing and extremely inefficient (days can turn into months) if you don't know what you're doing and don't have the necessary support.
Very true. If things are too reliable, systems can come to depend on them always being so so reliable... So it can actually pay off to inject transient issues deliberately.
You know, one of the things I miss from when I wrote most code manually (now it's a minority of code) is that I would have a high rate of random error.
This was extremely tedious, and annoying, of course, but things would break in all sorts of educational ways.
And once in a while, the computer would do something truly magical, something I didn't even know computers can do.
LLMs still make mistakes, of course, but they make different kinds of errors than I do.
I've been trying to replicate the effect with software that reproduces my own type of errors, randomly omitting tokens from the program. (A strange type of fuzzing the program space.)
Some don't compile, some crash at runtime, most don't do anything interesting, but I know for a fact the process is worthwhile.
What would be really interesting would be to create a cheap verification process for this at scale, i.e. a "wall of TVs" each containing the output of a program, so that the "worthwhileness" of each one can be evaluated at a glance.
Had a strange thought last year, that if a system runs too smoothly, eventually the knowledge of dealing with breakage will disappear, and when it inevitably breaks, everyone will be unprepared.
So a little bit of breakage is like a healthy "exercise."
Presumably, a functional system is supposed to have some kind of drills to fill in for that. But I have seen ones that don't!
This is known as the "Paradox of Automation." It's a real problem for many industries.
The OG paper on this is called "The Ironies of Automation" (Bainbridge, 1983). It's a clunky but fascinating read. I give it a skim every couple years and usually come away with a bit of a fresh take on the problem.
Thanks for the reference. For anyone else interested like me, Wikipedia includes a link to an archived copy: https://en.wikipedia.org/wiki/Ironies_of_Automation
I had to grope a little, it was in the External Links section!
https://web.archive.org/web/20200717054958if_/https://www.is...
Yeah, I've lived that one. When your system is falling over all the time, you get good at quickly bringing it back. When you have the first failure in 2 years and you didn't do drills or etc, it takes a lot longer to get things going again.
I am very clumsy, and I've become excellent at catching things mid fall!
Knowledge of how to fix the system comes from understanding how the system works, which does not necessarily require experience fixing it. Every problem ever fixed was at some point fixed without prior experience - if we can fix problems that have never happened before we can fix problems that haven't happened in a while.
Can confirm.
Can also confirm that the process can be harrowing and extremely inefficient (days can turn into months) if you don't know what you're doing and don't have the necessary support.
Very true. If things are too reliable, systems can come to depend on them always being so so reliable... So it can actually pay off to inject transient issues deliberately.
https://netflix.github.io/chaosmonkey/ and similar can help.
That said ... most instability is introduced with normal changes, so every engineer can be a chaos monkey. ;-)
You know, one of the things I miss from when I wrote most code manually (now it's a minority of code) is that I would have a high rate of random error.
This was extremely tedious, and annoying, of course, but things would break in all sorts of educational ways.
And once in a while, the computer would do something truly magical, something I didn't even know computers can do.
LLMs still make mistakes, of course, but they make different kinds of errors than I do.
I've been trying to replicate the effect with software that reproduces my own type of errors, randomly omitting tokens from the program. (A strange type of fuzzing the program space.)
Some don't compile, some crash at runtime, most don't do anything interesting, but I know for a fact the process is worthwhile.
What would be really interesting would be to create a cheap verification process for this at scale, i.e. a "wall of TVs" each containing the output of a program, so that the "worthwhileness" of each one can be evaluated at a glance.
https://en.wikipedia.org/wiki/Hormesis
that's the plot of The Machine Stops
https://web.cs.ucdavis.edu/~rogaway/classes/188/materials/th...
Oh, the legendary backup restore drill? I'm sure it will happen someday ...
There’s a good Star Trek TNG episode about this: When the Bough Breaks (S1E16)
The real fun one is what do we do when civilization breaks. Do we even remember how we assembled it in the first place?
I don't mean things like plumbing and roads, I mean things like, "what happens to a culture when people stop saying hello to each other?"
Hence DiRT or whatever they call it these days
Nice that it is cc-commons.
Unfortunately site would need some refurbishment for mobile… slide show doesn’t show, and page is floating.