Thank you. I hope you write up findings from your own research.
To your point, I wasn't pretending that this work is novel or something that the AI community should take seriously. If anything, my point was that you can just do things.
I also feel like in the SWE community folks are generally concerned that LLMs are getting considerably better at doing our jobs. This was a poetic attempt at trying to regain some agency and not just let life happen _to_ you.
> I hope you write up findings from your own research.
If and when I achieve anything useful I definitely will. Writing up failed experiments? Yes in principle, per the above. Finding time is probably the biggest challenge. The intention is definitely there though.
But even aside from that, bits and pieces of what I'm working on at any given time dribble out via my participation in various oline forums, Github discussions[1] and posts here, on LinkedIn, etc.
> If anything, my point was that you can just do things.
Yep, yep. Absolutely. Again, even if the outcome isn't some earth shattering new discovery, you've still gained something from the process (in all likelihood).
> This was a poetic attempt at trying to regain some agency and not just let life happen _to_ you.
Well said. That very much echoes a lot of my own philosophy on life. Just do something at least. To me, I'd rather fail trying to do something, than just give up and do nothing.
Archaelogy of the mind is an enjoyable marathon but a marathon nonetheless. Audiobook therefore highly recommended, as well. If you get past the "seeking" system section it will probably reframe your entire view of consciousness, intelligence, and what it means to be human. You will cringe significantly harder at the concept of LLMs becoming AGI once you get past the part on "primary affective processes". Self-assembling brain will help reconcile those conversations. And then evolutionary algos will help you build that brain out! Keep posting!!
> By the nature of the math, if you can only use 4 of your 128 cores 50% of the time, your performance just tanks no matter how fast you're going the other 50% of the time.
I'm not sure I understand this point. If you're using a work-stealing threadpool servicing tasks in your actor model there's no reason you shouldn't get ~100% CPU utilisation provided you are driving the input hard enough (i.e. sampling often from your inputs).
To work steal, you must have work to steal. If you always have work to steal, you have a CPU problem, not a CPU fabric problem. CPU fabrics are good for when you have some sort of task that is sort of parallel, but also somehow requires a lot of cross-talk between the tasks, preferably of a very regular and predictable nature, e.g., not randomly blasting messages of very irregular sizes like one might see in a web-based system, but a very regular "I'm going to need exactly 16KB per frame from each of my surrounding 4 CPUs every 25ms". You would think of using a GPU on a modern computer because you can use all the little CPUs in a GPU, but the GPU won't do well because those GPU CPUs can't communicate like that. GPUs obtain their power by forbidding communication within cells except through very stereotyped patterns.
If you have all that, and you have it all the time, you can win on these fabrics.
The problem is, this doesn't describe very many problems. There's a lot of problems that may sort of look like this, but have steps where the problem has to be unpacked and dispatched, or the information has to be rejoined, or just in general there's other parts of the process that are limited to a single CPU somehow, and then Amdahl's Law murders your performance advantage over conventional CPUs. If you can't keep these things firing on all cylinders basically all the time, you very quickly end up back in a regime where conventional CPUs are more appropriate. It's really hard to feed a hundred threads of anything in a rigidly consistent way, whereas "tasks more or less randomly pile up and we dispatch our CPUs to those tasks with a scheduler" is fairly easy, and very useful.
The network in the article doesn't have explicit layers. It's a graph which is initialised with a completely random connectivity matrix. The inputs and outputs are also wired randomly in the beginning (an input could be connected to a neuron which is also connected to an output for example, or the input could be connected to a neuron which has no post-synaptic neurons).
It was the job of the optimisation algorithm to figure out the graph topology over training.
I did a similar project previously and had what I considered "good" results (creatures that did effectively control their bodies to get food) but not the kind of advanced brains I had naively hoped for.
The networks were really configurable (number of layers, number of "sections" within a layer (section=semi-independent chunk), number of neurons, synapses, types of neurons, type of synapses, amount of recurrence, etc.), but I tended to steer the GA stuff in directions that I saw tended to work, these were some of my findings:
1-Feed forward tended to work better than heavily recurrent. Many times I would see a little recurrence in the best brains, but that might have been because due to percentages it was difficult to get a brain that didn't have any of it.
2-The best brains tended to have between 6 and 10 layers, and the middle layers tended to be small like information was being consolidated before fanning out to the motor control neurons.
3-Activation functions: I let it randomly choose per neuron or per section of layer, or per layer or per brain, etc. I was surprised that binary step frequently won out compared to things like sigmoid or others.
1-Creature shape: hexagon with a mouth/proboscis perpendicular to one side of hexagon
2-Senses:
2.1-Mouth could detect if it was touching plant food or creature (which is also food), and would transfer energy from food surce if touching it
2.2-Sight:
Two eyes each sent out 16 lidar-like rays (spread out a bit)
Eye neurons triggered based on distance of object and type of object (wall,plant,hazard,creature)
2.3-Touch:
Each body segment had 32 positions for touch
Each position had neurons for detecting different types of thing: wall, plant, hazard, creature, sound
2.4-Sound:
Each creature emanated sounds waves (slower than light rays but faster than creature movement)
Sound detected by touch senses
3-Motor: multiple neurons controlled forward, backward and rotation left/right.
4-Brain:
Layer 1=all sense neurons plus creature state (e.g. energy level)
Layers 2 - N=Randomly created and connected and evolved
Final Layer=motor
5-World:
A simple 2D space with randomly placed items
Walls+obstructions blocked movement
Hazards sucked energy from creature
Plants added food to creature if mouth touched plant
Between 20 and 50 Creatures
> Were the brains event-driven?
It was a sequential flow as follows:
1-Move creatures based on motor neurons and perform collision detection
2-Set sensory neurons in layer 1 based on current state (e.g. is mouth touching plant, eye ray detection, creatures bodies touching, etc.)
3-Calculate next state of brain in a feed-forward fashion through the layers. For recurrence it means, for example, a layer 2 neuron receiving input from a layer 6 neuron is using the value calculated from the previous cycle.
goto step 1
> How did you implement the GA? What did individual genes encode?
I did not have a DNA or genes that drove the structure of the NN. I played with many ideas for a long time, but nothing seemed to be able to encode a higher level capability without dependence on a very specific NN circuit structure. I looked at varous ideas from other people, like NEAT from U Texas but I never found anything that I felt worked at the abstraction level I was hoping for when I started. It's a really fun, interesting and challenging problem, I really wonder how nature does it.
I ended up creating an "evolution control" object that had many parameters at many different levels (entire network, specific leyers, specific section, etc.) that would guide (somewhat control, but mixed with randomness) the initial structure of the brains (layers, sections per layer, conectivity, etc.) and also the extent it could change each generation.
Example of config parameters:
"Chance of Changing A Neurons Activation Function=3%"
"Types of Activation Functions Available for Layer 2, Section 3=..."
After each generation, the creatures were ranked and depending on how good or bad they would be assigned a level of change for next generation.
The level of change drove how much of the NN was eligible to possibly change (e.g. 10% of layers, sections, neurons, synapses, etc.)
The evolution control object drove how possible different types of changes were (e.g. 3% chance switch activation function) and the magnitude of changes (e.g. up to 20% change in synapse weight)
Right. Ironically I chose a model that was around in the 1970s without knowing it.
My point was more a game-theoretic one. There is just no chance I would beat the frontier labs if I tried the same things with less compute and less people. (Of course there is almost 0 chance I would beat them at all.)
