Blog · Statistics

False discovery rate in a StrategyQuant X workflow

Sort a table of 24 hours by average result and the top row will look like something. That is not evidence yet. Whenever you look at many cells and report the best, the question is how often the best looks good by chance.

  • Published 2026-11-12

The arithmetic

Test each of 24 hours at the 5% level and, if nothing is going on, you expect about 1.2 of them to pass: 24 times 0.05. Twelve months in the same table add another 0.6. A table of calendar cells will show a significant row on almost any data.

What a q-value says

Edge Finder’s q column is the p-value adjusted for the number of cells tested, by the Benjamini and Hochberg method. At q below 0.10, roughly one in ten of the accepted rows is expected to be noise. Rows are graded: strong below 0.01, clear below 0.10, weak between 0.10 and 0.25, then not detected and too few bars. Only strong and clear count as survivors. Weak is too weak to build on and too strong to call measured noise, so re-run over a longer range. Not detected is the only grade that supports saying there is no effect. Too few bars says nothing either way.

Windows are a bigger search

A search over every window of the day on twelve weekday sets is a far larger family than 24 cells, and the cells overlap. Time Windows uses a max-t sign-flip permutation. It flips the sign of whole dates, which keeps each day’s intraday shape and volatility, records the strongest window anywhere in the family on each flip, and sets a window’s adjusted p to the share of flips that reach its strength. It pays for every window and weekday set tried, reported or not. The permutations are seeded, so the answer repeats.

Two other corrections

Entries one bar apart share most of their bars, so their results move together and the count overstates the evidence. Edge Finder divides each cell’s count by how densely its entries sit inside their own span, and shows the effective count nEff beside the true n. The headline figure is also the largest of several, so the maximum is biased upward; reported maxima are shrunk in closed form for the number of candidates.

What is left uncovered

The correction controls false discoveries within a table. It does not control choices between runs: changing the holding period after seeing the hold curve, or trying five timeframes and keeping one. Choices between runs are not corrected. The remedy is data the search did not see, which is why Edge Finder reserves bars and reports a direction check on them, and why a fresh date range is the real confirmation.

Related

The tables are described on the Edge Finder page and in Time Windows. The companion article is What to measure before a StrategyQuant X build.

Risk

Trading leveraged instruments carries a high risk of loss. Software output is not investment advice and says nothing about future results. Validate every strategy yourself before risking capital.