How to Evaluate a Trading Bot: The Due Diligence Checklist

Key Takeaways

• Most trading-bot marketing fails the first test: you cannot verify the track record, the assumptions, or the execution conditions. That matters because backtests are easy to overfit and hard to reproduce ^[1][2].
• Forward-tested results are better than pure backtests, but they still need timestamps, trade logs, fees, slippage, and a clear definition of the universe being traded ^[3][4].
• Survivorship bias, hidden costs, and vague “AI” claims can make a mediocre strategy look excellent on paper. A due-diligence checklist should focus on evidence, not branding ^[5][6].
• AIBROKER’s cryptographic audit trail and point-in-time universe construction are designed to make strategy verification more auditable; see our methodology page for how those controls work ^[7].

Retail investors are being sold more automated trading products than ever: bots, signal rooms, model portfolios, copy-trading feeds, and “AI” strategy subscriptions. The packaging changes, but the due-diligence problem is the same. You are being asked to trust a performance claim that may be based on cherry-picked history, unrealistic fills, or a universe that quietly excludes the losers.

The right question is not “Does it look profitable?” It is “Can I verify what was traded, when it was traded, what it cost, and whether the result could have been achieved in the real world?” The SEC has repeatedly warned investors to be skeptical of auto-trading and trading-software claims that promise easy profits or imply that past results guarantee future outcomes ^[3][4].

Why this matters: if you cannot reconstruct the strategy from source data, you are not evaluating a trading bot — you are evaluating a sales page.

Start with the claim, not the product

Before looking at screenshots, testimonials, or a smooth equity curve, classify the evidence. A backtest is a historical simulation. A paper forward test observes signals without proving executable fills. A live signal test observes contemporaneous decisions but may not trade. A broker-verified live record documents executions in one account. None alone proves robustness, completeness, ownership, scalability, or comparability with your account.^[2]

The more variants a researcher tries, the easier it is to select a lucky curve. Require the number of trials, fixed rules, point-in-time data, out-of-sample results, and all failed or retired versions—not only the survivor.^[1]

A clean backtest and a live account answer different questions.

2) The 10-question due-diligence checklist

Use this as a gate, not a formality. If a vendor cannot answer these questions clearly, you should assume the answer is unfavorable.

That checklist is intentionally boring. Boring is good. The best defense against a glossy sales pitch is a set of questions that force the vendor to reveal the plumbing.

3) Backtested vs. forward-tested vs. live: the verification ladder

Investors get tripped up because these labels sound more scientific than they are. A backtest can be useful, but only if the rules are fixed before the test and the data are point-in-time. A forward test is more credible because it uses live market conditions, but it may still be too short to include a real bear market or a volatility shock. A live track record is strongest, yet even that can be misleading if the account is tiny, the strategy is capacity-constrained, or the vendor cherry-picks the start date ^[2][4].

Here is the practical tradeoff: the more “real” the evidence, the less control the vendor has over the environment. That is why you should prefer evidence that is both live and independently auditable, not merely impressive-looking.

Practical takeaway: if a vendor leads with a backtest and buries the live record, treat the backtest as a hypothesis, not a result.

Model every cost on its actual base

Gross return is not an investor's return. Build a cash-flow model with subscription charges, commissions, spread, slippage, market impact, financing, data fees, taxes where applicable, and the opportunity cost of unfilled orders. State whether each item is per trade, per month, a percentage of capital, or a percentage of assets. Do not subtract unlike percentages as though they share one base.^[3][4]

The example below is only a percentage-point budget. If every assumed drag were measured as a percentage of the same beginning capital, 18.0% - 2.0% - 1.5% - 2.0% leaves 12.5% before taxes. It is not a compounded or audited net return. Real fees occur at different times and on different bases, and slippage depends on orders, liquidity, volatility, and capacity.

5) Survivorship bias: the quiet killer in strategy marketplaces

Survivorship bias is one of the most common ways strategy marketplaces mislead users. The platform shows you the bots that are still alive, still marketed, or still subscribed — not the ones that failed and disappeared. That makes the average result look better than the true population. We cover the mechanics of this problem in more depth in our survivorship bias guide, but the core idea is simple: if the graveyard is invisible, the living look unusually strong.

This is not a niche academic issue. It affects ranking pages, “top performers” lists, and copy-trading leaderboards. A strategy that was launched after a favorable market regime, or one that was re-optimized after a drawdown, can look like a star until the cycle turns. Bailey and López de Prado’s work is relevant here because overfitting and selection bias often travel together: the more candidates you test, the more likely the marketplace is to surface a lucky survivor ^[1].

6) What investors get wrong about “AI” and automation

The biggest mistake is assuming automation equals objectivity. A bot can be systematic and still be badly designed. It can be fast and still be fragile. It can be “AI-powered” and still be little more than a rules engine wrapped in a buzzword.

Another common error is confusing complexity with robustness. More indicators, more parameters, and more filters do not automatically improve a strategy. In fact, they often increase the chance of overfitting. If you want a useful companion to this section, read our backtest checklist and our systematic vs. discretionary explainer. Both help separate process from performance.

The honest assessment is that most retail investors should prefer simple, transparent rules over opaque black boxes. A strategy that is easy to explain is not automatically good, but a strategy that cannot be explained is usually hard to trust.

What AIBROKER's verification controls do—and do not—establish

AIBROKER publishes a canonical rebalance record with a SHA-256 digest and a delayed validation payload, and its research pipeline maintains point-in-time membership and hashed lineage manifests. The methodology page explains the intended process. These controls can detect a later mismatch between a disclosed artifact and its stored digest; they do not independently prove that source data were correct, that every experiment was disclosed, or that a strategy will work.

Point-in-time membership reduces survivorship bias only when coverage and effective dates are complete and the test actually consumes those records. Hashes establish integrity, not economic validity. For execution context, also review how stock prices are set and the bid-ask spread.

8) A decision tree for subscribers

Use this quick decision tree before paying for any bot or signal service.

This is intentionally strict. The burden of proof should sit with the seller, not the buyer.

9) A worked example: reading a bot sales page like an analyst

Suppose a vendor advertises a bot with the following claims: 92% win rate, 38% annual return, “AI-driven,” and “verified.” A retail investor might stop there. An analyst should ask: verified by whom? Over what period? On what universe? With what costs? What was the maximum drawdown? How many trades? Was the result concentrated in one regime? Did the strategy survive a different volatility environment?

Now compare that to a more credible disclosure: the vendor publishes a dated live track record, a downloadable trade log, a fee schedule, a slippage assumption, and a description of the asset universe. That still does not guarantee success, but it gives you something to interrogate. If the strategy also has a documented research process and a clear failure mode, you are finally evaluating a product rather than a promise.

For readers who want a broader framework for judging risk-adjusted outcomes, our Sharpe vs. Calmar article and risk measurement guide are good next steps.

Check operational security before connecting an account

Evidence quality is only half the review. Before connecting a brokerage or exchange account, confirm the integration uses the minimum permissions needed, cannot withdraw funds, does not ask you to share a password, supports credential revocation or rotation, and documents how secrets are stored. Enable multifactor authentication where supported. Treat a vendor that requests broad account authority as a security risk, regardless of its backtest.

Define the operating boundary before live use: maximum order and position size, allowed symbols and sessions, duplicate-order protection, a kill switch, alerts, reconciliation, and retained order/error logs. Start with observation or paper trading, then a deliberately small live limit. Paper fills do not validate live execution, and a small live test does not validate capacity.

The final decision still rests on independently checkable evidence, complete costs, failure conditions, and fit with your risk budget. If the seller cannot answer those questions or explain how access is revoked during an incident, do not connect the account.