Another lesson in automated deployments. This time related to high frequency trading. The biggest WTF is near the end:
Knight did not have supervisory procedures to guide its relevant personnel when significant issues developed.
In one of its attempts to address the problem, Knight uninstalled the new RLP code from the seven servers where it had been deployed correctly. (But not from the 1 server that was causing the problem)
…recommends new human processes to avoid a similar tragedy.
http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172-222-a-second-for-45-minutes
Also related is the HN discussion with other cringe worthy anecdotes.
The week after this we had a trader in our office who had a meeting at Knight on the morning it happened.
The craziest thing is that it went on for so long. No one caught it until their own traders so it come across Bloomberg and CNBC. They actually thought it was a rival HFT and tried to play against it.