The integrity of the Numerai tournament is based purely on trusting the company to run the competition honestly -- a trust that's not verifiable. As of Oct 1, 2020, there's 144,665 NMR at stake according to the website, so a lot of people are trusting on blind faith.
What's the trust issue here?
Say Numerai tells you 4 weeks after your forecast submission that your model/submisson's CORR is 0.02, and so you get a payout of stake * CORR. (I.e., if you stake 100NMR, you get 2NMR as payout.)
But, we have no way to verify that 0.02 is the actual CORR. Yes, Numerai tells you how CORR is calculated, but you are missing a bunch of other necessary data to verify that your model's CORR is actually 0.02. Worse, even if we run multiple models, we can't confidently extrapolate how one model's CORR relates to another. I.e., why our model 1's CORR is 0.04 but our model 2's CORR is 0.01. Thus, we are trusting Numerai, the company, to run the competition honestly and reporting/calculating our CORR honestly.
Now, this is not a unique problem to Numerai's competition. While many competitions have this same issue, an important difference is that the Numerai competition requires real money to stake, whereas competition like the Netflix Prize is free to enter.
The Netflix Prize competition also have less of a trust issue because what is being forecast is known, and modelers can tell if their models are any good. Moreover, if a model is good, it tends to stay good over time. That is not the case with the Numerai tournament. Models that were top performing for months ranking in the top 20 can fall out of top 400 over 4 months -- with no clear (or even fuzzy) explanation. (RIP Nasdaqjockey's model.)
Trust and bad incentive
I have been in and out of the Numerai competition for a long time and my few interactions with the company staff have been very positive. But, I can't exactly trust but verify here. Furthermore, it was recently revealed that staff models are on the leaderboard and staking, with at least one revealed staff model having a stake of over 1000 NMRs. We can only hope that the staff don't go into the code and add a few points to their own CORR scores.
As of October 1, 2020, staff model MDO has a stake of 1486NMR. If the staff gives himself or herself an extra 0.01 in the CORR score, that's an extra 1486*0.01=14.86 NMR of payout. At the closing price of $30.85 per NMR on Sep 30, 2020, 14.86NMR*$30.85=$458.431 extra that goes into the pocket of the staff, PER WEEK.
Important note - there's NO EVIDENCE that this is being done. But the incentive to do so exists and that's problematic. 6 staff models are ranked top 100 in the leaderboard right now. Another staff model NMRO currently stakes 1963NMR and ranks 80th. I hope internal checks are in place to prevent staff from wrongdoings, and it would be in Numerai's interest to make those checks known.
Concluding thoughts and recommendations
For those that have been with the contest for a long time, trust has been earned by the company and staff. (I have no data to support this actually -- just personal experience.) For newer staker-entrants, I am not sure where this unverified trust comes from, and there's a lot of trust with 144,665 NMR being staked by participants. Trust is a tenuous thing, and Numerai should act to relieve the concern while there is a lot of trust. Having an uncheck incentive to do wrong is bad. The financial system had blind faith on LIBOR being an honest rate and used it to price trillions of dollars of assets, and as with many things in banking and finance, that trust was misplaced.