UnThreaded | Threaded | Whole Thread (12) | Ignore Thread Prev Thread | Next Thread
Author: yodaorange Big red star, 1000 posts Feste Award Nominee! Feste Award Winner! Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: of 459260  
Subject: Why Knight COULDN’T hit the red kill switc Date: 8/5/2012 2:28 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 47
The Knight Capital (KC) situation is getting more interesting by the day. Tomorrow, Monday 8/6/12 is the day of reckoning for KC. All of the trades they did last Wednesday “settle”. This means that Knight has to come up with the $440 million in losses to pay off all of their bets. We will see.

The most often asked question about KC is” “Why did it take 45 minutes for KC to turn off the program that had the problem and stop the bleeding?” KC has been described variously as stupid, incompetent, etc for not hitting the red kill switch faster.

The best explanation I have determined so far actually paints KC in a rational light. At least as rational as a company can be that was losing $10 million per minute. A precursor is understanding two different types of trading.

1. Proprietary trading aka prop desk is where a company trades purely for their own account. Most high frequency traders are in the category. Their only mission is to make money for the firm. They have NO responsibility for any other party. If they unduly harm another party by some nefarious/questionable/immoral means, all the better. It is the other guy’s problem. If the company wants to turn off their HFT prop trading at any time for any reason, they have the right to do that. This is what happened in the May 2010 Flash Crash. KC’s bad computer program was NOT a proprietary HFT IMO.

2. A totally different type of trading is where a firm is a “designated market maker”aka DMM for NYSE listed stocks. This is the computer version of what NYSE floor specialists used to do. In the old days, if you wanted to buy an NYSE stock, you physically walked to the specialist station and placed your order. The specialist firm was selected by the NYSE for each stock. It was their responsibility to keep trading for that stock “orderly.” The specialist had the power/authority to stop all trading in a stock if things got out of control aka “imbalance.” The specialist bought, sold and sold-short shares for his own account in order to maintain order. The specialist’s goal was to buy small quantities of stocks and hold them for short periods of time, only as a last resort to keep the trading orderly. If there were enough outside buyers and sellers, the specialist did not/would not take any positions in the stock. The specialist was also responsible for managing the “book” which had all of the open buy and sell orders on it.

Fast forward to 2012. KC is the one and only DMM for many NYSE listed stocks. They have a contractual obligation to keep trading “orderly” on those stocks. They are NOT permitted to stop trading at any time for any reason. In 2012, the DMM role is played by a computer running in a HFT mode. The program is constantly monitoring all of the buy and sell orders. The program is supposed to act as the buyer and seller of last resort in order to keep trading orderly. The same program is probably used for all stocks that a DMM is responsible for. This appears to be the KC program that had the bug. The program bug was buying and selling, attempting to maintain orderly markets in the ~150 stocks affected.

It somehow got bid and ask prices backwards. Normally a specialist would buy at the bid price and sell at the ask price. Say a stock is trading at 25.00 bid and 25.05 ask. An outside investor would pay 25.05 to buy a share and could sell at 25.00 for a 5 cent per share loss.

The specialist would have the opposite side. He would buy at 25.00 and sell at 25.05 for a 5 cent per share gain.

In this case, it appears that KC’s program got the bid/asks reversed somehow. So they were losing the bid/ask spread on every share instead of profiting from it. And they were doing it at HFT speeds of roughly 200 times per SECOND per STOCK. Multiply that by ~150 issues and 45 minutes and you get a $440 million loss!

KC probably realized early on that they had a big problem. Unfortunately for them, since they were the DMM for a lot of those stocks, they did NOT have the option of hitting the red kill switch. It would have thrown the market into total chaos for those stocks with far worse implications.

For example, let’s assume that the KC program had all of the buy and sell orders for Vanguard, TD Waterhouse and Fidelity for a given stock. The small investor placed his order. Fidelity “routed” the order through KC where KC’s DMM program kept track of it. From the perspective of the investor and Fidelity all was well.

But what happens if KC hits the red kill switch? It would essentially cancel all of the open orders from all of the customers on all of their stocks WITHOUT notifying the brokerage or the customer. Talk about a nightmare of liability!

KC’s only alternative was to start notifying all of their customers (brokerages) that all open orders need to be cancelled and re-routed through a different market maker. This was an attempt to make an orderly wind down of the problem. And it took 45 minutes to do it.

You can imagine the KC management seeing this all unwind in front of their faces. And you can imagine the software group that released the new program. Not a pretty picture. However given the choices they had at the time, I think they did the correct thing by an orderly winding down of all open orders. And yes IMO, the firm will be wiped out due to this one error on one day. Pretty remarkable. I still think BK or buyout for KC is inevitable.

Eventually all of the details will come out, but for now, this is my best explanation that I have been able to piece together.

BOTTOM LINE: at this point, I do NOT have any actionable recommendations for individual investors. I am also NOT optimistic that anything will be done to prevent an accident like this from happening in the future. It is software and it is operating at very HIGH SPEEDS which is why you can have a nuclear meltdown in 45 minutes. In the old days of human specialist this NEVER would have occurred, but we are NOT going back to that state of the market.

Thanks,

Yodaorange
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Print the post Back To Top
Author: captainccs Big funky green star, 20000 posts Top Favorite Fools Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400166 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 4:12 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 1
Yoda, I love the insight! Thanks!

The Captain

Print the post Back To Top
Author: steve203 Big funky green star, 20000 posts Top Recommended Fools Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400172 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 4:59 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 0
from the bubblevision rumor mill

Knight Close to Deal to Raise $400 Million From Investors

Scrambling to raise cash in the wake of a trading snafu that cost it dearly, Knight Capital Group at midday on Sunday appeared to have generated about $400 million from a group of about a half-dozen investors, said people involved with the deal.
----
Late Wednesday, Goldman Sachs [GS 100.98 3.17 (+3.24%) ] agreed to purchase the unwanted stock positions from Knight as part of a big basket, people familiar with the matter have said, at a discount for their original purchase price.


http://www.cnbc.com/id/48516238

Steve

Print the post Back To Top
Author: jchoy Two stars, 250 posts Ticker Guide Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400174 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 5:26 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 1
hey this is probably a stupid question...

but if they lost $440 million in 45 minutes from reversing the bid/ask, would they be able to gain $440 in 45 minutes if they fixed the problem?

Print the post Back To Top
Author: LonghornBoy Big red star, 1000 posts Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400175 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 5:40 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 2
The answer is simple:

They need to beef up their pre-production testing processes to "paper-trade" the updated algorithms for a while before pushing these changes into a live environment.

If the issue was really as simple as reversing the buy and ask, then shame on them. Any rigorous software QA team should have caught this.

Print the post Back To Top
Author: FolacinTheAcidic Big red star, 1000 posts Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400181 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 6:41 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 1
They need to beef up their pre-production testing processes to "paper-trade" the updated algorithms for a while before pushing these changes into a live environment.

If the issue was really as simple as reversing the buy and ask, then shame on them. Any rigorous software QA team should have caught this.


In my experience, there is always at least one more test that could/should be run. If you actually tested all of the cases in any even moderately complex program, you'd never release any software.

It's possible that they missed the most obvious case ("That couldn't ever fail, no need to test it"), but my guess is that it was some little bit deeper.

However, I'm guess that resignations have already either been submitted or requested. A number of years back, the AT&T network took a major outage for a similar 'failure to test' - rumor has it that the programmer responsible left his ID on his manager's desk.

Print the post Back To Top
Author: LonghornBoy Big red star, 1000 posts Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400191 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 8:12 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 0
In my experience, there is always at least one more test that could/should be run. If you actually tested all of the cases in any even moderately complex program, you'd never release any software.

Absolutely true. If you attempt to do full-on, white-box testing of a large-scale software application, it's almost impossible to cover all of the potential conditions. The best you can hope for is to insure that you've covered all of the common cases and the majority of the less common ones.

It's also the case that during a typical software development project; different types of testing occurs at different phases during the process: Developers test their code before they release it to QA; QA uses a set of test cases to perform integration and regression testing; and "smoke testing" is commonly done once it's in production to insure that all of the basics are in place.

There were so many eyes on this code that it's very surprising to me that they missed something so head-smackingly obvious. As I mentioned before: Shame on them.

Print the post Back To Top
Author: yodaorange Big red star, 1000 posts Feste Award Nominee! Feste Award Winner! Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400194 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 8:41 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 5
I am sure I have over simplified what went wrong. I am sure they did not literally swap bid and ask prices in their program. The bug they had caused their program to do things backwards.

I am also sure they thought they had run a very robust suite of tests on the new program. One possibility that has been brought up is that they developed a simulator that would be the other side of the market. It was supposed to simulate what all of the other market participants would do.

The "real" program would be buying and selling, 100% correctly. The market simulator program would be buying and selling correctly, but the company would NOT be monitoring its losses when the real program was developed.

The theory is that they accidentally released their simulator as well as the real program. Net effect it the simulator was buying and selling "wrong", just like they expected the investing masses to do.

Eventually we will learn what really happened and I agree with all of the other comments. Testing dynamic, complex software like this is darned hard and NEVER perfect. I am sure the programmers are despondent on making this error. In hindsight, it will turn out to have been very obvious, just like most software bugs are,

As to why they cannot “reverse” the program and make $440 million in 45 minutes. The $440 million they lost was PROFITS for other traders. Other high frequency trading algorithms were in hog heaven while this was going on. They have NEVER made that kind of money in that short a period of time. They are NOT going to give those profits back, just because Knight had a software bug. Frankly, they hope more HFT’s have bugs like this.

Thanks,

Yoda

Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Print the post Back To Top
Author: FolacinTheAcidic Big red star, 1000 posts Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400197 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 8:59 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 0
There were so many eyes on this code that it's very surprising to me that they missed something so head-smackingly obvious. As I mentioned before: Shame on them.

Yeah, no argument here. If it really was as simple an error as the OP said, I can only think that it was missed in very early tests with someone scanning a result and missing it, or (since it is a blindingly obvious problem) reporting it back informally and assuming (always a bad word in a test environment) that it had been fixed.

Joe

Print the post Back To Top
Author: Goofyhoofy Big funky green star, 20000 posts Top Favorite Fools Top Recommended Fools Feste Award Nominee! Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400198 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 9:08 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 3
I am sure I have over simplified what went wrong.

Yeah, I'm pretty sure.

One piece I read said there was "extensive" stress testing of the software before it was deployed. That involved using the same stocks, issues, and clients which the software would be expected to encounter, all done off-line, of course, and run in simulation through hundreds of thousands of trades and dozens, if not hundreds of simulations.

It is impossible to simulate every possible scenario, of course, and the fact that this went haywire within minutes of being deployed leads me to conclude that something else, something very weird happened. Like some of the data wasn't cleaned out. Or a simple programming error opened the door between off line and on line. Or yes, bid and ask got flipped, or, well, pick any of a thousand possible screw ups. Anybody who has written code has run into some just plain stupid mistakes they've made,and this was surely one of them.

Why they didn't shut it down more quickly is what intrigues me. Surely they saw it running wild. Surely someone at the exchanges saw it go haywire. To say they had to let it run or else chaos would ensue is pretty thin gruel. Chaos ensued anyway, and larger than it would have otherwise been.

I just hope we don't find out it was Chinese hackers figuring out how to bring down the market or something. Excuse me while I adjust my tinfoil hat.

Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Print the post Back To Top
Author: warrl Big funky green star, 20000 posts Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400204 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/5/2012 10:46 PM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 1
In hindsight, it will turn out to have been very obvious, just like most software bugs are,

In my experience, a competent programmer can fix the typical bug in a complex already-in-production system in less than five minutes by altering, deleting, or writing less than ten lines of code.

That is, AFTER spending several days pinning down exactly what the program is doing wrong in exactly what circumstances and tracing through to find WHERE those lines are/should be.

And then the fix has to go through unit testing, documentation, peer review, regression testing, and implementation.

And the bug probably also triggers the creation of more test cases for the regression testing process. The test cases ALSO have to go through testing, review, and implementation.

Print the post Back To Top
Author: Watty56 Big gold star, 5000 posts Old School Fool Add to my Favorite Fools Ignore this person (you won't see their posts anymore) Number: 400213 of 459260
Subject: Re: Why Knight COULDN’T hit the red kill s Date: 8/6/2012 12:34 AM
Post New | Post Reply | Reply Later | Create Poll . Report this Post | Recommend it!
Recommendations: 0
I am a computer programmer in a totally different field.

What I don't understand is why it affected so many stocks.

I would have implemented a change like this by using the new software version for a few relatively small stocks at first before turning it on for so many stocks at once.

I do things like this all the time when I make changes to critical programs.


The whole story is likely not out yet.

Print the post Back To Top
UnThreaded | Threaded | Whole Thread (12) | Ignore Thread Prev Thread | Next Thread
Advertisement