pdr-backend v0.3 is out! Bots do up and down predictions with confidence-weighted stakes.
With Ocean Predictoor, you can run AI-powered prediction bots or trading bots on crypto price feeds to earn $. pdr-backend is a key part of Predictoor implementation: it’s the GitHub repo to run predictoor bots, run trading bots, do simulations & analytics, do payouts, submit truevals, and more.Previously, a typical predictoor bot would only submit an “up” or “down” prediction in a given epoch. As of the new pdr-backend v0.3 release, bots submit “up” and “down” predictions with a stake-weighted confidence on each, i.e. “two-sided prediction”. This reduces epoch-to-epoch variance and increases profitability.
Also, the simulator got upgraded: predictions are two-sided, the trader bot uses model confidence, and there are new plots like predictoor profit vs. time and a visual of model probability response surface.The v0.3 release has many other improvements: in trader bots, data pipeline, UX, and core bug fixes.
The rest of this post is organized as follows. Sections 1 and 2 describe one- and two-sided predictions (the before and after). Section 3 describes simulator upgrades, and section 4 other upgrades for v0.3. Section 5 concludes. Section 6 shows more cool plots from simulation.
1. Before: One-Sided Predictions
In a given epoch, a predictoor bot would predict a direction (“up” or “down”), and stake on that prediction (e.g. 10 OCEAN if not confident, 1000 if highly confident). Call it “one-sided predictions”. The image below illustrates.Prior to v0.3, prediction bots would only submit an “up” OR a “down” prediction, i.e. “one-sided prediction”.
The problem is: accuracy in up/down predictions is typically 50.5% to 60% max. With such accuracies, a bot is never very confident in its up vs down prediction. It can nonetheless make $ doing these predictions, but is there a better way?
Furthermore, if a bot gets it wrong, they are slashed the full stake amount. So there is high variance in winnings from epoch to epoch: big win or big loss. High variance hurts profitability.Towards a better way, here’s some brief theory.
When a bot makes a prediction, its underlying model is a Bernoulli probability distribution [1]. That is, there are exactly two possible outcomes, a single trial, and the bot’s model has estimated the probability of each outcome.However, in the status quo code, the bot only expresses one side of the model— the most probable side — in the course of submitting a prediction. It’s leaving information on the table, and exposing itself to high variance.
2. New: Two-Sided Predictions
The idea of two-sided predictions is to let the bot express both sides of the distribution in the course of submitting a prediction.For example, if the predictoor bot calculates a probability of up=60%, and therefore down=40%, and has 1000 OCEAN to stake, then the bot stake 0.60*1000 = 600 OCEAN to “up”, and 100 OCEAN to down. Therefore the bot is always on both the winning side and the losing side, pro-rata to its prediction confidence. The image below illustrates.As of pdr-backend v0.3, prediction bots submit “up” and “down” predictions with a stake-weighted confidence on each, i.e. “two-sided prediction”.The winning side will be there to catch the slashed stakes of the losing side, which in turn greatly reduces epoch-to-epoch variance.Updated AI model. Before, the prediction bots a regression model to predict a next value of BTC etc. Now, it’s explicitly a classifier model, classifying up vs. down. pdr-backend currently has two linear classifiers: LinearLogistic and LinearSVC [2].
The classifiers give probability estimates.How it’s implemented. In a given epoch, a predictoor bots does two transactions to the blockchain. It needs two accounts / private keys, because the Predictoor contracts allow only one tx per epoch per account.How to run two-sided prediction bots. First, you need to set a second account, with envvar PRIVATE_KEY2.export PRIVATE_KEY=<YOUR_PRIVATE_KEY 1>export PRIVATE_KEY2=<YOUR_PRIVATE_KEY 2>Second, be sure to have ROSE and OCEAN, to pay for gas fees and stake respectively.Related minor changes:The model-based predictoor bot used to be approach 3. Now it’s approach 2. Approach 3 is deprecated.
Approach 1 used to randomly pick an up-vs-down prediction. Now it puts 50% stake into “up” and 50% stake into “down” (50–50). Therefore it needs two keys too.
3. Upgrades to Simulator
The prediction/trading simulator pdr sim got upgraded.First, the simulator’s bots got upgraded.Sim prediction bots are now two-sided, behaving just like in the main predictoor bots. Therefore you can do rapid “what-if” experiments in simulation, before proceeding to run a predictoor bot for real $.
Sim trading bots are now confidence-based. Before, the trading bot would always buy OCEAN according to a fixed amount of USD if there was an “up” prediction; and did nothing on a “down” prediction. Now, in an “up” prediction it buys, in a “down” prediction it sells, both proportional to the model’s confidence in up vs down.Second, the simulator’s plots got a big upgrade. There are now realtime plots of: predictoor profit vs time, trader profit vs time, model accuracy vs time, predictoor profit distribution, trader profit distribution, and a contour plot to show the model’s probability response surface. The appendix shows these in detail. Or, run for yourself with pdr sim my_ppss.yaml. Expect simulator plots to continue to evolve 😎.
The updated “pdr sim” realtime plot, having many sub-plots. The appendix zooms in on each.
4. Other v0.3 Improvements
From v0.3.0 release notes, here’s a thorough list of changes.Predictoor bots: Two-sided prediction. Easy deployment of >1 agents via Kubernetes + CLI; pm2 deprecated.Trader bots: live-mocking ability, enabling “ghost” trades.Simulator: predictions are two-sided. Trades use model confidence. New plots like predictoor profit vs. time and a visual of model probability response surface. Live trading possible.
Data pipeline: better structure, with DuckDB at the core. Analytics are starting to get built on top.UX: There is now proper Python-style logging. ppss.yaml tuned. Small CLI utilities added.Core: The codebase is more robust, including greatly-expanded type-checking and several bug fixes. Predictoor and trader agents are refactored to be simpler. Technical debt is low — no small feat.
5. Conclusion
As of pdr-backend v0.3, predictoor bots submit “up” and “down” predictions with a stake-weighted confidence on each, i.e. “two-sided prediction”.
This reduces epoch-to-epoch variance and increases profitability.Also, the simulator got upgraded: predictions are two-sided, the trader bot uses model confidence, and there are new plots like predictoor profit vs. time and a visual of model probability response surface.
The v0.3 release has many other improvements: in trader bots, data pipeline, UX, and core bug fixes.Try for yourself at pdr-backend’s Predictoor bot README.6.
This section gives plots from an example simulation run.Settings. It was predicting BTC/USDT on Binance, on 5m epochs. The predictoor bot LinearLogistic model, using the most recent value of BTC and of ETH, staking 100 OCEAN per epoch. The trader used up to 1000 USD per epoch. Simulation ran for about 2000 epochs (1 week). The YAML file given below lists all parameters.Below is predictoor profit vs. epoch. After 2000 epochs (1 week), it netted 915 OCEAN.Predictoor profit vs. epochBelow is trader profit vs. time. After 2000 epochs (1 week), it netted $2613.19 USD.
Trader profit vs. epochBelow left is model accuracy vs time, where accuracy is the % time it was correct in predicting up vs. down. Green is the estimated value, and the grey shadow is the 95% [lower bound, upper bound] confidence intervals according to a Binomial distribution.
Below middle and right are predictoor and trader profit distributions as 1d scatterplots, respectively.Left: model accuracy vs time. Middle: predictoor profit distribution. Right: trader profit distribution.
Below is a contour plot to visualize the model probability response surface. We can have a contour plot because this run has only two input variables: most recent value of ETH (y-axis), and most recent value of BTC (x-axis).
The model predicts a direction of “down” for any (x,y)=(prev ETH, prev BTC) in the red region. The model predicts a direction of “up” for any (x,y) in the blue region.The decision boundary is white. The contours towards the white are lower confidence; contours moving farther from white are higher confidence.
The red and blue points are training data for “went down” and “went up” respectively. Being encircled by yellow means that the model mis-classified on the training data; this is not surprising because we should not expect these two input variables to fully capture the response.
“The Bernoulli distribution is a special case of the binomial distribution where a single trial is conducted (so n would be 1 for such a binomial distribution). It is also a special case of the two-point distribution, for which the possible outcomes need not be 0 and 1.” Here are details on the default classifier models. LinearLogistic is “logistic regression”, via sklearn library. (Yes, it’s a classifier; the name is confusing). LinearSVC is support vector classification with a linear kernel, also via sklearn. The pdr-backend’s aimodel/ directory implements these.
Final NoteNone of the content in this post should be taken as financial advice. Everything you do is your responsibility, at your discretion.
Source : Ocean Protocol by Ocean Protocol Team / Feb 28, 2024