Fix #1278, round 2: [Sim] Reframe to classify 'will price go up *anytime* in next 5 min?'

[trentmc]

Fixes #1278

[trentmc]

Some benchmarks. First two plots are fine. Something's up with the classifier performances.

Pdr profit

Tdr profit
(lost this one :( )

Classifier performance

Model response

[trentmc]

Some benchmarks. Right after commit e475ee2, where I cleaned up separation of HIGH and LOW classifier models. Performance plots look a bit more sane.

• Classifier accuracy is >79%. If this is true, it's amazing

Settings: my_ppss.yaml.txt

• stop loss % = take profit % = classifier threshold % = 0.2%

Plots to ignore: pdr/tdr profit vs prob_up, when prob_up < 0.5. (So far it's only plotting probs_up_UP)

2024-06-23 12:27:21,788 sim_engine INFO Iter #5000/5000 ut=1719138000000 dt=2024-06-23_10:20 ║ prob_up=0.038 pdr_profit=3.77 up + -26.43 down = -22.66 OCEAN (cumul 2407.88 OCEAN) ║ Acc=4498/5000 =  78.62% [ 77.6%,  79.7%] prcsn=0.562 recall=0.463 f1=0.503 loss=0.591 ║ tdr_profit=$  0.00 (cumul $104.35)

Pdr profit

Tdr profit

Classifier performance

Model response

[trentmc]

Some benchmarks, run right after previous benchmark.

• Identical my_psss.yaml file
• Except running on main branch (vs this PR). Main branch has stop loss / take profit but has a single model classifying up/down.

Pdr profit

Tdr profit

Model perf

Model response surface

[trentmc]

Some benchmarks. Right after commit a03244e

• "Improve predictoor logic: only stake when models are confident enough"
• I've also created prob_up_MERGED, for use in predictoor bot staking, and the plots. Update plots to use that value. Therefore the plots on pdr/tdr profit vs prob_up are now sane.

Settings: same ppss as before: my_ppss.yaml.txt

• stop loss % = take profit % = classifier threshold % = 0.2%
• predictoor_ss.sim_only.others_accuracy = 0.50001

Pdr profit

Tdr profit

Classifier performance

Model response

[trentmc]

Benchmarks, with setup like above, but for more coins. Focus on trading results.

BTC (like above)

ETH

BNB

XRP

ADA

DOGE

SOL

LTC

TRX

DOT

[trentmc]

New experiments, with longer runs

Four different experiments

1. "reframe2" x "easy"
2. "reframe2" x "real"
3. "reframe3" x "easy"
4. "reframe3" x "real"

Where

• reframe2 = refactored code
• reframe3 = code before refactor
• easy = predictoor_ss.others_accuracy=0.50001, trader_ss.sim_only.fee_percent=0
• real = others_accuracy=0.504 (50.4% like actual accuracy), fee_percent=0.001 (0.1% like Binance spot)

Parameters that are static across runs

• Just BTC/USDT in, and out
• On 50K iterations (173 days, ending on jul 15, 2024). Vs typical benchmark of 5000 iterations.
• All thresholds are 0.2%
• max_n_train: 1000, autoregressive_n: 2
• approach: ClassifLinearRidge, weight_recent: 10x_5x, balance_classes: None, calibrate_probs: CalibratedClassifierCV_Sigmoid, train_every_n_epochs: 1

Experiment 1: "reframe2" x "easy"

All params: my_ppss.yaml.txt

Final log line: Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║pdr_profit=67.41 OCEAN (cumul 9006.65 OCEAN) ║ tdr_profit=$ -2.22 (cumul $634.74)

Predictoor profit vs time

Trader profit vs time

Experiment 2: "reframe2" x "real"

All params: my_ppss.yaml.txt

Final log line: Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║pdr_profit=66.24 OCEAN (cumul 6177.61 OCEAN) ║ tdr_profit=$ -4.43 (cumul $-11332.49)

Predictoor profit vs time

Trader profit vs time

Experiment 3: "reframe3" x "easy"

All params:
my_ppss.yaml.txt

Final log line: sim_engine INFO Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║ prob_up=0.731 pdr_profit=33.68 up + -12.43 down = 21.25 OCEAN (cumul 9901.44 OCEAN) ║ Acc=33315/50000 = 69.47% [ 69.1%, 69.9%] prcsn=0.528 recall=0.576 f1=0.528 loss=0.820 ║ tdr_profit=$ -2.31 (cumul $874.00

Predictoor profit vs time

Trader profit vs time

Experiment 4: "reframe3" x "real"

All params:
my_ppss.yaml.txt

Final log line: sim_engine INFO Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║ prob_up=0.731 pdr_profit=33.15 up + -12.43 down = 20.72 OCEAN (cumul 3682.33 OCEAN) ║ Acc=33315/50000 = 69.47% [ 69.1%, 69.9%] prcsn=0.528 recall=0.576 f1=0.528 loss=0.820 ║ tdr_profit=$ -4.61 (cumul $-17374.36)

Predictoor profit vs time

Trader profit vs time

Analysis / Discussion

• Zooming in on 5K-iteration (17.8 days) chunk of performance vs time, it might tell us "good" or "bad" behavior, depending on which 5K-iteration chunk. Therefore 5K-sims are not enough to draw much conclusion; we need 50K (178 days).
• In "real" settings, both reframe2 & reframe3 made $ predicting, and lost $ trading. reframe2 did slightly better than reframe3.
• Q: are these results better than if just 1 model (status quo)? Need to run those experiments too...