hb from the Numerai forum posted the following graphs to illustrate possible correlations between the market and forecaster performance. I've alluded to this previously but the relationship is clearer due to the data now available under the new tournament format. Numerai now releases "real time" and daily forecaster performance data and thus make checking the correlation easier than before. Note the VTI is the Vanguard Total Stock Market Index ETF, and VIX is a market volatility index from Chicago Board Options Exchange.
hb's graphs certainly suggest possible correlation between forecaster performance and the market, but note the usual caveat that correlation doesn't imply causation. But if we just assume there is indeed correlation, what logical conclusions can we draw?
- One. The forecaster models aren't very good. AI and ML and NN models aren't magical. The amazing things that are being done in the field of face recognition and autonomous driving have a lot to do with coding and prepping the data with priors in a way that allow the AI learning to perform well.This is not possible with the Numerai data because its data is obfuscated.
- Two. The training data Numerai gives to forecasters/data scientists might not be sufficiently representative of the down side. While the training data contains about a hundred eras and it is known that eras present some periods of time -- it remains unclear whether eras are hours, days, months, or years. So it is possible that the training data are capturing the state of a bull market. Only Numerai would know and they are not telling us data scientists. The other thing to note is that Numerai does not update its training data much. Going from memory here, I think we have been using the same training data for at least the past few months.
In terms of actionable conclusions -- be cautious if you are planning to increase your stake because forecaster performance has been doing great up to last week.