"I was looking at the markets and had an idea for a new investment method. As soon as I took an historical series of data and my favorite computer program I've started doing my "backtest"! A few hours later, I've said: "It is alive!", "It works!". Can not wait for my plans of wealth. Everything comes to my mind as I contemplate the beautiful return curve of my "backtest"!
Full of confidence, I'm ready for the real life! "
The beginning of this text reflects the daily lives of those whom try, somehow, beat the market. And at this time, the trader may come across three possible scenarios:
1 - Your new theory does not work after the "start";
2 - Your theory works for a while and then never again;
3 - Your theory is still running and your account increases every day.
After reading some works of Sir Karl Raimund Popper - one of the greatest philosophers about history of science - and a lot of time thinking, we say that a theory can be considered a theory if it has been tested and rejected or if it is not yet known if that is wrong .
One might already conclude immediately: a theory is never right? Yes, exactly. Let us recall here the Black Swan logic (book from Nicolas Taleb, a great work). A theory will never be certain because we will never know if all swans are white and we can not infer that all of them are because we did an historical analysis and we have never seen a black swan before. You are following the reasoning? So let us return to the three scenarios traders face after their backtest.
1 - Your new theory does not work soon after the start
This had happened because the historical data had no example of what just happened and it had bucked his new theory. The dynamics of the market, the information available and the relationship between supply and demand was changed right after he "put to the test" his new method. And this has a strong correlation with the following scenario...
2 - Your theory works for a while and then never again
Although it is very similar to scenario 1, the big difference is in the fact that the current situation of the market still follows the same backtest behavior, or a little worse, the theory is developed with overfiting - over-adjusted. So many customization were made to make the theory profitable that it is impossible to generalize any changes in the market. The backtest learnt everything that happened in the past that makes it have a blind vision of the future.
3 - Your theory is still running and your account increases every day
Finally, we might conclude that this is the winning scenario. The theory works! Yes, indeed, but only to the point where it will be disproved. And on this day, the system will fail and you may have devastating consequences. Confidence in the system and the confidence of the trader inflate as the account balance does. And a very full balloon burst more easily and more strongly that a balloon not so full. The difference between this scenario to the others is that the unlikely event has not happened yet.
So can we say that we have no escape?
Yes, we have. You can not look only at the historical result, but you should look on the positive expectation of returns. Your system will go wrong someday, for sure, and by "go wrong" I do not mean a few days of losses. I mean a sequence of losses, with no "apparent explanation." Let's say the system has only 0.1% chance of failure. That is, on average, only 1 in 1,000 times it will not work. First some points that must be understood:
That said, what should we pay attention? The maximum loss that we can have when the system goes wrong, plus the flexibility to update it. Let's focus on the first thing. Knowing the maximum loss will help the trader to make the calculation of the expected return. Using again Nassin Taleb, we quote the example given in his book "Fooled by Randomness". The market may have a 70% chance of going up and provide 1% gains. If it does not go (the other 30%), the losses may reach 10%. Considering that the trader has $1,000, the expectation of the trader in this case is negative by $230. Why? For every 10 times he wins $70.00 in total, but will lose $300, with a final balance of $230.
It may sounds interesting having a winning rate of 70%, but if the trader does not take into account what will happen in the remaining 30%, its might be a beautiful backtest, which began with a beautiful theory, but will not be worth anything.