Below excerpt courtesy of Brendan Conway’s April 17 Focus on Funds

MarketsMuse Editor Note: Brendan’s article deserves front page focus, but in the process of publishing this piece, a bigger story has emerged and the internet has been overwhelmed by stories that suggest pro-Putin militants in East Ukraine are distributing flyers that purport to come from local government officials with formal announcement that Jews in the city will be required to reqister with the local government, upon which they will be subjected to new taxes or face deportation. MarketsMuse Editorial team says : “Sounds like a Putin-supported strategy intended to cause more chaos, and in turn, provide Putin the perfect storm in which he can defend a larger action on the part of Russia..under the auspices of having to come in and protect Russian (Jews) among others..”

Now on to Brendan Conway’s observations re: “ETF BackTesting”

“…..It’s negligence, or worse, when an investment manager’s innovative-looking strategy is the result of too much quantitative trial-and-error.

That’s the argument in a notable new study flagged by Stephen Foley of the Financial Times. “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance” argues that what happens behind the scenes in the development of quantitative strategies is a major problem in investment management.

“Backtest” simply means reviewing historical returns to try to divine how a new strategy might perform in the future. The method has become bread-and-butter in the launch of many new ETFs.

Investors don’t know how many hypotheses managers examined before finding the perfect-looking backtest, a process which turns out to matter greatly, write David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado and Qiji Jim Zhu. “The higher the number of configurations tried, the greater is the probability that the backtest is overfit,” they write. “Overfit” means the data has been tortured until it yielded something that looks nice.

If an investment process is driven by what looks good historically, there’s a greater chance the attractive-looking result is just a fluke.

Sure enough, a Vanguard Group study found a while back that backtested ETFs — which look great in the historical data — on average lagged the market after the real-world launch.

From Foley’s discussion: Continue reading →