I haven’t kept a tab on the Numerai chat channels for the past months (more focus on job search and other pet projects), so I am not as aware of the numerous minor changes that have happened recently. But Numerai has some data issues this week, and that leads me to check the website to see what's going on. One thing led to another, from thinking about how much time I should devote to this to how well compensated are the stakers now, and that all ends up with me doing some quick data analysis on tournament returns for the top stakers in recent rounds.
So, here's the summary table on the top 10 stakers as of July 18, 2020. The stakers are: aininja, niam, simonos, sorios, dalios, cryptoquant, skyn3t, jefferythewind, phorex, and ia_ai. The table presented below and the data used for calculation is stored in a Google sheet that you can download here.
Some thoughts on the return summary statistics
The returns, even excluding the leaderboard bonus, are impressive. The average 4-week period return is 4.40%. Again, this is the 4-week return, and not annualized return. If you multiply it by 12, you can get a rough estimate of the annualized return of 52.78%. Do note this is in terms of NMR and not US dollars.
Other things that caught my interest:
- The average fraction of total payout from the leaderboard bonus is 25.35%. I.e., it's a big part of the compensation, even for the top stakers, and this is going away in September.
- The average of 25.35% is actually misleading because it is dragged down by the 3 stakers with fractions below 1%.
Top stakers' models have gotten quite a bit more correlated with the meta model. Because of this, I anticipate Numerai to push for a full MMC payout scheme at some point.
- Descriptively, the average correlation for the top 10 staker's models with the meta model is 0.808. But the average over the recent rounds (Round 211 to Round 216) is 0.830.
- 3 models have correlations above 0.90 (0.962, 0.973, 0.903).
- 6 models out of 10 have correlation above 0.80.
Speaking of MMC, it's not used much by the top stakers. Only 1 model used MMC payout, and that staker only tried for 3 rounds before switching back to CORR. The staker lost money in each of those 3 rounds. So empirically speaking (though caution on the very small sample size here), MMC doesn't seem to be as "profitable" as the Numerai staff wants us to think. In the Google sheet, I also included Nasdaqjockey's data since he/she was quite successful for a time. He too used MMC payout for 4 periods before switching back to CORR payout. In 2 of the 4 rounds he suffered losses, and overall he did much worse with the MMC payout than the CORR payout.
So, when MMC eventually are forced on the stakers, stakers should adjust their staking accordingly.
Speaking of nasdaqjockey, I have written posts before that referenced his top status. But as of July 18, 2020, he has fallen to rank 240. I have also suffered from the same fate, and all my models have fallen out of the top 200 as well. There seems to be some systemic shift in the contests where what has worked before no longer does. If you check his Rank tab, his rank starts to fall with Round 215 (June 7, 2020). Unfortunately, I haven't kept a tab on the chat channel and so I am not sure what, if anything, Numerai has changed at that point that might be attributed to the shift. If you are a recent participant and have some thoughts on this, please let me know.
As of Jul 19, 2020, 167,476.92 NMR are staked. So roughly, the top 10 stakers have 25.31% of the stake. Do note that the data I have calculated is as of Round 216, and the 167,476.92 figure is as of Round 221. I.e., there’s a date mismatch but it should be fine as a very rough approximation.
Conclusions
What I take away from the analysis is the following:
- Leaderboard bonus is a huge part of the compensation and it's going away.
- Even without the leaderboard bonus, CORR payout is good. Though I am concerned about whether we are simply in a boom period right now since there are very few instances of negative payout between Round 183 to Round 216.
- MMC payout doesn't look profitable from limited data and when this is forced on us at some point, you should seriously reconsider your staking strategy.
Some notes on the data used
In case you care, and it is good if you care, here are some caveats on my data and calculations.
The data for the top 10 stakers and nasdaqjockey is retrieved on July 18, 2020. All my comments here are relating to data from round 183 to round 216. Round 216 is the last resolved round as of the 18th. Round 183 is when the new staking mechanism started.
The numbers are irreconcilable for the most part, because there are various changes to the payout even since Round 183. I often wonder if even the Numerai staff can explain to us how all the numbers add up without checking and going through their code.
2 accounts (cryptoquant, phorex) are missing data for Round 208. I don't know why. It's just not there. I wonder if the same person/group controls both accounts.
All this is to say, that if you spot what looks like errors, they are likely errors and please let me know. I did a few checks and the numbers seem roughly correct, but since I can't reconcile the numbers exactly, I can't be sure.