Staking for Round 165 just ended, and the staking data from Round 132 onward is graphed below. The data comes from writing some python code and making use of the Numerai API (numerapi) to retrieve and calculate.Here I will talk about factors that contribute to the staking patterns. The amount of staking reflects the health of the Numerai competition. From Numerai the company's point of view, more forecasters staking means more confidence/validation in the company's method of soliciting forecasters to make predictions. In turn, this reflects well of their other main venture that generalizes the Numerai competition methodology - Erasure.
In the Numerai contest, the standard way to be rewarded for accurate forecasts is to stake. This involves getting some Numerai cryptocurrency, the NMR, and staking it against your forecast. If your forecast turns out to be better than a benchmark, you are rewarded up to 1x of the NMR you staked. If your forecast turns out to be poor, you lose your stake.
For example, let's say you submit a forecast for the Bernie tournament and stake 1NMR. If your forecast turns out to be quite a bit better than the benchmark, you get your 1NMR stake back and win up to an additional 1NMR. If your forecast turns out to be poor, you lose the 1NMR you stake.
The graph shows the stakes (sum across all tournaments) since Round 132. At its peak at Round 146, 2584.84NMR were staked. If you value NMR at $7, that's $18,093.88 staked. At the low of Round 155, 603.04NMR or about $4221.28 were staked. Although these numbers seem high, remember that the per forecaster stakes are actually quite low.
The main contributing factor to the amount of stake is the performance of the prior forecasts. Here is the performance of the example model that Numerai includes when you download the competition data, and the performance of an "average" model ranges from quite bad to pretty good. Predictions outcomes are poor from Round 143 to 149, and recall that the results for Round 143 is known at around Round 148, we see that this lines up very well with the reduction in stakes starting around Round 147 as forecasters change their expectations of how well their models will do.
Despite the amazing things neural networks and machine learning has accomplished in areas like visual recognition, these methodologies are not magical. My experience in doing forecasts with financial data suggests that at best machine learning can improve prediction probability by about 1% or 2% over basic statistical methods like logit or probit. That's it. Moreover, while statistical methods allow for identification of factors that matter, and allows for various hypothesis testing on the values of these factors, machine learning methodologies cannot do any of this.
But anyhow, beyond the performance of prior forecasts, there are other events that have affected the amount of stakes.
Around the early Round 130s, there was a new dataset being related. (Forecasters are unfortunately given dataset that are months old with few updates, and that's the raw ingredient that they have to work with to make live predictions.) Given that some learning models require a lot of training time, it's possible that the downward trend from 132 to 138 reflects this.
At Round 154, a new tournament design was implemented that judged forecasts based on something called LiveAUC instead of LiveLogLoss (Info via google docs here.)
At Round 162, a reputation bonus is introduced that rewards extra NMRs based on stakes and the average performance over all tournaments for the next 20 rounds. (So the bonus will be rewarded after 20 rounds from the staking round; info via google docs here.) In the few rounds since, this seems to have quite an impact on the amount being staked. At the time of this post, the bonus applies to the first 1000 NMR from forecasts that have been performing well. If Numerai increases the bonus further, this would increase the stake level even more.
Overall, what the changing level in staking reflects is the incentive issues that still need to be worked out, but the company is active in trying to make things work.