Ah yes, this story is legendary. I discuss it in my JavaScript Application Design book[1]. Chaos-monkey server-ball-wrecking sounds like a reasonable way to mitigate this kind of issues (and sane development/deployment processes, obviously)

[1]: http://bevacqua.io/bf