How the grading works

The data

Ten years of daily bars (2016 onward) for every US common stock and ADR, including the ones that later delisted. Testing only today's survivors makes almost any strategy look good. Prices are adjusted for splits, and dividends are included where the test measures total return. Fills use the open after the signal, never the close that produced it, and every trade pays slippage.

The brokers' agents

Every example agent Public, Robinhood and Composer show on their own pages, wherever the rule can be pinned down from price data. Each one is turned into the plainest rule that matches its words, written down before it's run, and tested once. "As tested" on each page says exactly what that was, including any gap I had to fill (a ticker left blank, a lookback not given).

• "When X, buy Y" agents are entered at the next open after the signal and held a fixed number of sessions, with any stop the prompt sets. If the card says what it rotates out of (banks, staples), that's the comparison. If it doesn't, it's compared with buying the same ticker on any day and holding as long. That isolates whether the signal's timing adds anything.
• Composer symphonies run as the rule tree they describe, every day or month as set, traded at the next open, and scored month by month against SPY.

Agents that need data I don't have, such as post archives, call transcripts or option history, are listed as untested rather than faked.

My agents

These are the agents I run on paper in my own trading stack. They're scored in R (the result divided by the initial risk) at the risk size each one uses, under the same caps a real account would have. Every night the backtest is rerun and a live forward test keeps adding trades on days the backtest never saw.

The verdict

Four checks on the brokers' agents:

• Significance. The t-stat measures how far the average edge sits from zero relative to its noise. Under 1 there's no measurable edge and the agent is Fragile, whatever else it does. Under 2 it can't be better than Inconclusive.
• Resampling. The trades (or months) are resampled 2,000 times. If the edge vanishes in more than 1 in 5 resamples, it's luck of the draw.
• Neighbors. Every threshold and lookback is nudged up to 20% either way, 24 times. A real edge survives in most of the neighbors. A curve-fit one is a spike that falls apart around it.
• After publication. Composer shows the date each strategy's rule was fixed. Everything after it is a true out-of-sample test: the edge has to stay positive and keep at least half its strength. Without a date, the edge has to be positive in both halves of the decade.

Robust clears the t-stat bar and passes at least two of the other three with no fail. A basket picked with hindsight (today's popular names, tested back to 2016) can't be Robust. My own agents use my stack's checks, which swap neighbors for a walk-forward test and raise the bar for every variant I've tried.

The test covers 2016 to today. That's one long bull market for tech with sharp breaks in 2018, 2020 and 2022, but no 2000 or 2008. Anything using 3× funds is flagged for that.

What this isn't

It isn't advice, a signal service, or a promise. An Inconclusive agent can still make money, and a Robust one can stop working tomorrow. The point is to know which is which before you hand a strategy your money.