I’ve always thought it’s wild how we can apply one concept to so many different types of things. For example, if I say something is “soft,” you probably think of the opposite of firmness. But at the same time, I can describe a person as “soft,” and the same descriptor can say something meaningful about their character.
Seeing the Spider-Man neuron work on multiple types (pictures, drawings, text), makes it seem like we can teach AI to learn these same type connections.
And if we scale up the network size enough, what if we could see these types through the equivalent of a being with 1000IQ? What connection types are the most effective for a being like that? Can we even understand them? Maybe they would be deep, and archetypical in the way that Odysseus and Harry Potter are the same, despite the fact that one is an ancient Greek king, and the other is a modern British wizard. Even more interestingly, maybe the connections would be completely inexplicable to us, with no apparent rhyme or reason perceptible to humans.
One of the amazing things about this project exploring CLIP was seeing some hints of this. For example, one day I was studying one of the Africa neurons and it generated the text "IMBEWU" -- it turns out this is a popular TV show in South Africa (https://en.wikipedia.org/wiki/Imbewu:_The_Seed). That's a trivial example, but it begins to hint at something interesting.
I'd really love to see what a domain expert analyzing CLIP would make of things. For example, I'd love to hear what ethnographers think of the region neurons, or what historians think of the time period neurons. Especially for future, larger models.
> What connection types are the most effective for a being like that?
That would be analogous to: What kind of tokens when put together make state of the art algorithms? - they are just tokens, even if it's the best implementation.
Intelligence is in the game, including the agent, the environment and goals. It's not in some kind of special neuron connections, it's in the way the brain is connected to opportunities and dangers outside.
Seeing the Spider-Man neuron work on multiple types (pictures, drawings, text), makes it seem like we can teach AI to learn these same type connections.
And if we scale up the network size enough, what if we could see these types through the equivalent of a being with 1000IQ? What connection types are the most effective for a being like that? Can we even understand them? Maybe they would be deep, and archetypical in the way that Odysseus and Harry Potter are the same, despite the fact that one is an ancient Greek king, and the other is a modern British wizard. Even more interestingly, maybe the connections would be completely inexplicable to us, with no apparent rhyme or reason perceptible to humans.