There is a story from WWII about the Army doing an analysis of damaged planes to find out how to better protect them by determining which areas of the plane were most vulnerable. This analysis showed that the wings and tail were hot spots for receiving damage from enemy fire. Based on this information, it was proposed that they would add additional armor to the wings and tail. Makes sense right? Well maybe not. Before they went forward with the new implementation, a consultant on the project who happened to be a statistician brought up the inconvenient fact that their sample size didn’t include all planes that had taken damage – it only included planes that had survived attacks. Those that had crashed and not returned home were not included. So while the army thought they were using this rigorous analysis to make a correct decision on how to better safeguard their planes, in fact they were doing the exact opposite. They were proposing to add additional protection to the part of the planes that had already survived after taking damage to those areas. After further review, they realized that the engines were the real weak point, and those that were struck near the engine had a much greater probability of going down.
Why is this an important story? Because this idea of “survivorship bias” is prevalent in many scenarios. So what is survivorship bias? At its most core level, it’s the idea that we are apt to make incorrect assumptions by only evaluating the “survivors” of a data set. In this example, survivors is very literal in that they were only including the planes that survived. In other examples, the bias may not be quite as obvious, but the effect is the same. This is especially relevant in the investment industry. The reason for this is that although all the disclaimers tell us that past performance is not indicative of future results, we never actually believe that. We want to invest our money into things that have shown a history of making money. This is why every fund company and investment strategy advertises themselves on their great past returns. On its face, this makes sense. It stands to reason that those funds that have done well in the past were a good indicator that the fund manager had some superior skill in choosing winning investments. Before addressing this specifically, I first want to tell another story.
Back in the 1930’s, there was a fascination with the idea of people having psychic abilities or ESP, so one Doctor designed a test to see if he could find people that showed these traits. For this test, he took 500 participants, and asked them guess the order of 5 playing cards. If the participant got this right, they moved on to the next round. He did this for several rounds, and deemed that those who were able to get the order right every round possessed these special abilities. It’s not hard to see where this study goes wrong. Had he preselected these individuals because of some other factors, and then they subsequently were each able to pass this test, well then that may have been worth something. However, the law of large numbers says that if you have enough samples, you will eventually see a low probability event occur, through pure chance having nothing to do with skill. That is exactly what happened in this experiment; if you have enough people guessing the order of these cards, some small number are bound to get lucky and guess right.
This actually parallels the mutual fund landscape surprisingly well. As I eluded to earlier, it is commonplace for a fund company to advertise a fund that has done exceptionally well for some number of years, but in many ways this is no different than the misguided Doctor advertising that he has discovered people with ESP…except that the fund companies know exactly what they are doing. The way this works is that these fund companies will open hundreds or even thousands of funds. Over the subsequent years, those that are performing poorly are killed off or folded into another fund, and at the end of 10 or 15 years, only a fraction of those original funds are still in existence, and even a smaller fraction have performed well. Guess which funds they then advertise with great track records? Did these survivors generate good returns because of some great skill? There’s no way to know for sure, but we do know that basic probabilities tell us that through sheer luck some should do well, and that is exactly the result that we see. The chart below summarizes this data.
These results are pretty astounding. If we look back 20 years ago, there were just under 3000 stock-based mutual funds. Today about 1800 are no longer even in existence, and around 2400 underperformed a simple benchmark. 8 out of 10 mutual fund investors would have been better off investing in a basic index fund. Naturally, the fund companies don’t advertise this information; in fact, they do their best to keep this data under wraps. Instead, they try to sell how good those “winners” are, on the pretenses that not only did they outperform in the past, but that they’ll also outperform in the future. Even after uncovering how these fund companies operate, it’s still hard not to be lured in by these funds that have done so well in the past. After all, although it’s easy to see how the survivors are probably just a result of luck, it is possible that the result shows us that there are just a very small number of really good managers. So how can we know if those outperforming funds are doing so because of luck, or if they are a result of skill that should be repeatable?
This provides a pretty resounding answer – any skill that a fund manager may possess clearly doesn’t translate into future outperformance. This chart may look somewhat complicated at first glance, but the idea is pretty simple. What it’s showing is how the best performing funds over one period did during the next period. Specifically it ranks the top 25% of funds based on performance over a 5-period, and then sees what percentage of those funds remained in the top 25% during the next 5-year period.
Statistically, if the performance was a result of skill, we’d expect that 2nd number (those funds in the top quartile during both periods) to be something close to 100%. The best funds should continue to be the best funds.
Conversely, if it’s entirely luck, the expectation would be that those winners are perfectly dispersed among the whole group, with 25% of them landing in the top 25% during the next time period.
For equity funds, the average is actually WORSE than what you would expect by dumb luck, meaning that the best performing funds during one period are actually LESS likely to do well again the next period than any random fund.
If an investor is considering buying a fund based on how well it’s performed in the past, they should know the following:
- This fund was created alongside thousands of other funds.
- It most likely survived and performed well by pure chance.
- Statistically it is unlikely to continue to perform as well in the future.
Ready to Take The Next Step?
For more information about any of the products and services listed here, schedule a meeting today or register to attend a seminar.