A fully NN solution (as opposed to using a light sprinkling like all competitive solutions do) requires big binaries for the most part, while an arbitrary program can easily memorize exact sequences to find repetition while using a tiny NN for some finetuning of predictions. A pure NN solution like a Transformer-XL does turn in the record-setting performance on natural language datasets including WP... unfortunately, the required NN model sizes alone tend to be larger than the entire uncompressed WP corpus here, and so have zero chance of ever being the minimum of model+compression. (An observation which I think supports the idea that the Hutter Prize has long outlived its intended use for measuring progress towards AGI; it's now just sort of a 'demo scene' version of AI benchmarking, testing intelligence within extreme constraints of sample efficiency/compute, rather than a true useful benchmark.)

The fact that small compressors like gzip or zpaq do so well at small data/compute but then can't compete as you scale up to tens or hundreds of gigabytes of text (amortizing the cost of those fancy NN models) can be considered a version of the Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

The bitter lesson seems to be striving for the wrong goal. Replicating human performance will at best give you human performance. Humans are incredibly inefficient. They require multiple orders of more computational resources just to match a machine. Since we do not have an abundance of computational resources, trying to replicate humans will always result in inferior results than simply running conventional algorithms.

Why then do systems following the bitter lesson like AlphaZero strongly outperform humans? (Hint: it involves scaling.)

But your entire model, plus your code, plus the compressed data must fit in 16MB: it gives very little room for maneuver.

Yes, but limitations are:

1) Needs to run in around 8hrs or less on a single CPU core

2) (I assume) it needs to be self-extracting (and size includes the compression logic).

I interpreted the rules as only decompression needs to be in 8 hrs, training could be arbitrarily long? That would also lead to the theoretical strategy of finding the sequence somewhere in pi, and just storing the index and a pi generator....

> 2) (I assume) it needs to be self-extracting (and size includes the compression logic).

Size includes decompression logic, which may be what you meant but it is important to distinguish between compression and decompression.

http://www.hutter1.net/prize/hrules.htm#rules

Since the stated motivation is work toward AGI, I think that such an approach would be highly in the spirit of the contest.

What if we write a powerful AGI, but it just isn't interested in this problem?

The size of your model is included.

People thinking that 1 GB of RAM is not much is one of the main problems in computing today. Please stop.

This +1000000000

It's outdated, but it's still 10x the size of the uncompressed data, so it should be generally sufficient.