Arc Competition Predictions

A couple days ago, the Arc Competition was announced. Basically it’s a task that’s meant to force AI to actually be smart. Unlike other dumb evals, you can’t just memorize how to complete a particular task. Instead, each task is unique, and requires the AI to figure out the simple set of rules that maps an input to an output.

Just for sake of posterity, I want to predict how this competition is going to be won, and I suspect that it’ll be won in the next couple of months.

Prediction

You take a regular multi-modal LLM. You tell the LLM to look at the input in question. You provide both an image of each input/output grid, and you also provide a text-based representation of each input/output grid, where each cell is annotated with positional information and with the value of each cell. (This value can either be represented as an int or a str with a unique enumeration of the inputs).

You then create classes in python that represent a grid of cells like the inputs and outputs. You instruct the model to identify patterns in the input and output, and then write a function that maps the input representation in python to the output representation.

Your LLM is going to fuck this up. You’ll want to include a couple rounds of code repair.

I suspect you’ll also have to include a step where you manually instruct the LLM to describe different important features of the input, and separate prompts that help the LLM figure out how these different features relate to each other.

Conclusion

Eh… this is kinda vague. But I’m pretty sure this is going to be the schema that actually produces something cool.