There is a high rate of failure among quant funds. These include smart beta, factor investing, statistical arbitrage, and CTAs. Such false positive strategies are a widespread industry problem.
Since psychiatrists have long traded on the ability of the human mind to find an elaborate narrative in a random ink blot, it should be unsurprising that it is all too easy for such minds to find investment strategies that “ought” to work, based on any real world, verifiable, goodness-gracious data sets.
Marcos Lopez de Prado, of Cornell University, and Michael J. Lewis of New York University, have written a paper about the machine-learning tools that they believe can help academic journals filter out “false positives,” or papers that propose a tenable investment strategy that turn out to be false. They believe that false positives, and an inadequacy of the filters against them, constitute one of the reasons for this failure rate.
Lopez de Prado and Lewis focus on false positives rather than false negatives (ideas that are rejected and so do not become the basis of a strategy, although that constitutes leaving money on the table), because the former is a real economic loss whereas false negatives are only an opportunity cost.
The False Strategy Theorem
Six years ago, Lopez de Prado in conjunction with David Bailey, of the University of California at Davis, advanced the “False Strategy Theorem,” a mathematical proof that the optimal outcome of an unknown number of historical simulations is unbounded on the right. Less technically expressed, this means that with modern computational power it will always be very easy to find impressive-seeming strategies that are nothing more than false positives. That theorem may seem to be just another statement of the problem that back-testing has long been known to pose. But in the new paper, Lopez de Prado and Lewis seek to turn the FST into part of the solution.
They have, drawing on FST, devised an “unsupervised learning algorithm that determines the number of effectively uncorrelated trials carried out in the context of a discovery.” This result can then be employed for estimating the “family-wise false positive probability.”
Lopez de Prado and Lewis say that their approach has three important features that distinguish it from others in the pertinent academic literature:
- It does not assume that returns follow a normal distribution. Making this assumption would underestimate the false positive probability.
- Their work is based on Extreme Value Theory.
- They do not assume a constant average correlation across trials.
Creating, Refining, and Counting Clusters
The trials drawn upon by those presenting a false positive can have a hierarchical structure. There will be clusters of trials that have similar strategies (and thus are correlated) and there will be less correlation between one cluster and the others. The clustering/hierarchical structure can be both complex and irregular. Assuming a hierarchical structure will end in letting too many false positives through the designed filter.
The key to their learning algorithm, then, is precisely addressed to clustered trials. Given N series, the algorithm partitions the trials into an optimal number of K subgroups, such that each cluster will have high intra-cluster correlations, low inter-cluster. The algorithm is accordingly called ONC, for “optimal number of clusters”
ONC is a learning algorithm in that it is designed to re-run the clustering in a recursive manner. Humans can check on its efficacy by comparing the average quality of the clusters to redo to the average quality of the clusters after the redo. What results is a number of effectively correlated trials, which is a critical meta-research variable in getting to the likelihood of a false positive.
Confirmation and Other Purposes
Lopez de Prado and Lewis support their results with Monte Carlo experiments.
The authors conclude by stating their conviction that their ONC has multiple uses in finance beyond its motivating purpose as a filter. It could be employed, for example, to identify the optimal number of economic sectors from the perspective of risk. That is, it will allow risk parity investors to diversify their portfolio more effectively, “where the peer groups are not set in advance.”
ONC also could be employed to improve upon the standard method of collinearity analysis, principal components analysis.