The FARG revolution & its consequences

CHAPTER 1. Introduction.

“Write drunk; edit sober.” — Hemingway, they say

Your heart races, but you know you have crossed the point of no return. There is no escape from this predicament. Call it intellectual terror. You have fallen into the trap; another fly kicking and screaming in the tarantula’s web. There is no more hope. All of your choices have been ruthlessly taken away. A poor soul; taken hostage. You cannot go to the next message, not anymore. You cannot close the tab, not anymore. Well you can still close the tab, it’s not like I have hacked your computer and it won’t do it. But you are too curious for your own good.

Take the loss. Face the pain.

There is no way out of this disgrace—there is only going through it.

~~~~

CHAPTER 2. Looking backwards…

“A apoteose do dedão.” — Luis Fernando Verissimo

Dear FARGonauts,

10 years ago, something had been bugging me. If I bought a new computer I would install all these programs and tools that we all do. But where was the good stuff? The FARG projects? Where’s the one button I press to get all of them in my new fancy machine? It was a state of “Sparse Distributed FARG”, and the internet we had was a Kaput Internet. So on Sunday, 30 July 2017 at 18:40:56 (UTC-3) exactly, I single handedly very bravely saved the internet from the imminent catastrophe. A single git repository with all the code, the thesis, the papers I could find. Once splattered all over the place, now on a single button to rule them all! So FARGonautica was born.

Now I’m glad to announce that FARGonautica 1.0 is live. We finally have all projects from FCCA running with GUIs, and most of the newer projects. Some were done by the open-source community, others were translated by us to modern Python (we used an obscure tool found here). Numbo and Metacat run bit-by-bit identical: if the temperature at codelet 123456 was 3.1415, then it also is 3.1415 in the translation. Copycat is Melanie’s original code running under SBLC (modern Common Lisp) with this blackboard-like OpenGL GUI. TableTop & Seek-Whence are “re-interpretations.”

In order, FCCA projects first, then Post-FCCA projects. A few screenshots:

FCCA: Marsha Meredith’s Seek-Whence

FCCA: Daniel Defays’s Numbo

FCCA: Melanie Mitchell’s Copycat

Because Copycat made such a splash, there are ~10 different implementations made by the open-source community. I got Melanie’s original code running on modern Common Lisp, and ported the OpenGL graphics developed by Joseph Aaron Hager. A python app wraps both things; here’s what it looks like.

FCCA: Bob French’s Tabletop

POST-FCCA: James Marshall’s Metacat

POST-FCCA: Abhijit Mahabal’s Seqsee

POST-FCCA: Eric Nichol’s Musicat

FCCA: Gary McGraw’s Letter Spirit, Part 1

[ED this one is, for obvious reasons, duplicated on the BIML website]

POST-FCCA: John Rehling’s Letter Spirit, Part 2

POST-FCCA NOT Ported: Capyblanca, Phaeaco and George. Capyblanca: In our recent work we created a full chess engine that plays by analogy (see chapter three video), but it’s not ready; far from it. George: The architecture has many deviations from a “normal” FARG one, I wouldn’t ever be satisfied with a re-implementation from a description only (the code is gone). Phaeaco: This one gives too much power to the enemy (chapter three). Under wraps for the time being.

Letter spirit, and others like Copycat, came from the open-source people. I take the liberty to copy my e-pals here, special mention to Dr Paul Geiger (Copycat + Letter Spirit part 1 + Letter Spirit part 2, in javascript, running on a browser with no installation needed). It’s insane. Nothing to install, nothing to do. Click the link, and, through some miracle, the spirit of Letter Spirit rises back. The apotheosis of the button!

A professor is not supposed to have favorites, but we always do, don’t we? My favorite is Tabletop.

My ten-year-old daughter goes wild with Tabletop: We bet on what Tabletop will do, then she starts screaming “NONONO Daddy you’re wrong NONONO I would touch THIS, because of […]!”.

It’s a blast!

We hope these implementations will resonate with someone; perhaps bringing up a smile somewhere.

As we pass the baton, with a little luck, the younger generation will pick this up and push it forwards. That’s their job.

This is something that really needed to be done by somebody somewhere. So it became our job.

~~~~

CHAPTER 3. Looking forward…

“Just when you thought FARG was out; they pull it back in” — Michael FARGleone, The GodFARGer, Part III

Introducing… Argo!

Around a year ago I ganged up with Harry Foundalis (world-renowned Bongardologician), Eric Nichols (world-renowned Musicometer) & two new guys: Bernardo Machado (Cognitive psychologist & Business mercenary) + Victor Tigre (Applied Physicist). Whereas Deep Learning seems to have solved crystallized intelligence (at obscene expense), we don’t think it has solved Fluid Intelligence.

Our goal is to solve, through FARG: ARC-AGI-1, then ARC-AGI-2, then ARC-AGI-N. BigAI has solved them all through “agents”, apparently, but we’re ahead in at least 4 ways: (i) we don’t need Mr Musk’s magic flying datacenters in the sky; ours solves hundreds of problems on a phone in Airplane mode, no GPU. (ii) We know what it is doing inside. (iii) It does not separate perception and reasoning, or “training phase” and “inference phase”. (iv) Finally, like people, this one goes with a few examples—not gorillions of them.

Call us crazy, but perhaps not all that exists is gargantuan statistical inference? Perhaps FARG can help nudge the trajectory of the biggest and craziest industry in the world just a teenie-weenie bit? Oh look, this thingie binds with that thingie, so there’s a bond here. Oh look, this blob in the input corresponds to that blob in the output, so there’s a correspondence there. The examples say it’s all about “the thing that touches the red square”, but nothing touches the red square on the test, so the idea slips to “the thing closest to the red square”. Again we go from concrete → abstraction → concrete (after finding the transformation rule). And on and on. It’s FARG. No more; no less; it’s FARG all the way down. Difficult, yes. Large vocabulary, but not insurmountable. It’s about walking that fine line between underfitting (combinatorial explosions & not knowing what to do) and overfitting (creating “solutions” to the very problems you’ve seen and nothing else—at the limit creating a lookup-table whilst demanding all swans to be white). That fine line you people have walked.

Here’s what Argo looks like right now:

Argo runs on a Browser, here on ConceptARC’ s Extend-to-Boundary-2, with Mr. Claude running on a panel on the right. Here Mr Claude has created ten test variations (to look for overfitting).

Argo fails at Extend to Boundary 3, Mr. Claude explains what it saw on the logs (isolated dots instead of a blocking line).

Somebody somewhere had to pick up this fight; so it became our job. Yes it’s a big fight, yes there will be blood, yes we’ll lose most of our teeth—but hey, soup is cheap.

~~~~

CHAPTER 4. The manifesto

“Gentlemen you can’t fight in here! This is the War Room!” — Dr. Strangelove

We’re not going to be normal, happy, healthy, well-balanced people, this is no margarine commercial.

We’ll be relentless about this.

We will get it done—that much I promise you.

Here are some numbers (30 Oct 14:00 “Loopy Argo” version, codebase size ~400,000 lines):

How we measured them. The first row marked as [*] means ”the 1,000-task ARC-AGI-2 training set, which contains ARC-AGI-1.” If you look at the percentages of ConceptARC, that’s our worst. That’s because we only count as “solved” if the system, given two guesses, can create the output grids of all three tests perfectly—two tests correct and one wrong means not solved. Mean run time per task is <9 seconds on a big desktop development machine (not a phone), but some problems do explode, with two being killed at 5 minutes. No program means we found nothing that could transform the grids of even the examples into their output grids. Whatever overfitting remains in the system appears here as a fake solution that sneaked into these numbers. Observe that you can mitigate overfitting, but you can never fully escape it. Humans overfit; it’s the “black swan problem”; it’s the “unknown unknowns”; it’s our tying-to-the-mast burn-the-boats never-back-down true-believer defense of our paradigms. Einstein himself overfitted to theories in which God wouldn’t touch dice. Mathematics overfits to axioms. An interesting philosophical question is whether God (a perfect system, or all possible busy-beavers, or things way beyond my imagination) can escape overfitting, but I obviously don’t have the IQ to push further.

So yes, with LLMs having beaten it, right now it’s still an embarrassment; with a long painful voyage ahead. But it seems to have promise, doesn’t it? Every week we creep up a little in solves and down a little in time. I think it was Eratosthenes that once said:

«Όλοι ξέρουν πως, όταν ο Χάρης Φουντάλης είναι ο επικεφαλής αρχιτέκτονας, τίποτα δεν μπορεί να πάει στραβά.»

— Ερατοσθένης

What a beautiful, profound, insight, isn’t it?

Here’s me drawing the sword and screaming “Touché, BigAI!”: a preview of my loving “declaration of war against Big-AI” manifesto. It stays “unpublished” until we have two papers for outside investors showing that:

1. Our all-singing all-dancing “AI paradigm” is fast and cheap, runs on a phone; and it is fully auditable; we can see what’s going on inside. And few examples suffice.
2. It bypasses the dissonance involved: “when are coffee cups like old elephants?” Recent work by Martha Lewis and Melanie showed the problem Bob French posed in modern LLM parlance, just substitute “perception module” by “training data” and you’ll see it immediately.

Please forgive me for being a little raw & rude to our cherished deep learning friends. I did my best but still couldn’t hold it back on the “Test-Time Training Technique”. I tried to be well-mannered; I really really tried.

When we get these due-diligence papers done, we’ll post the video publicly and we’ll find investors. Somebody somewhere is going to see the potential. For now I’m responsible for keeping the lights on.

So yeah. Just a heads-up on what we the little people have been playing with. Of course, it’s not ready, and our motley crew of misfits doesn’t deserve any rest. The journey is not finished, perils lie ahead.

~~~~

CHAPTER 5. The end of the world

“It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.” — Dickens

All my love to each and every one of you. Each of you has changed my life in ways I can hardly begin to express. In a parallel universe where Doug hadn’t led this charge, my life would have been so much poorer; drier; sadder; shallower. I’m profoundly grateful that this is the universe I got to live in, with all of you in it. So much good stuff—this is the best light cone!

Comments, tips, issues & pull requests to the repository; mockery of the video and my terrible diction, intros to people we should talk to, criticisms, requests for demos, soup cans, literature we should read, and AI doomer viewpoints will be taken graciously.

I threw this text in Claude and it spit out:

“It reads like someone who has been quiet for ten years and decided to say everything at once. For a reunion letter to people who shaped your life, that is the right failure mode.”

The right failure mode—I love it; beautiful poetry!

Speaking of the end of the world AI doom: if we the third chimpanzee are not going to be alone anymore, well… at the risk of being crushed as a simpleton… I for one do take solace from some FARG technical properties that beautifully morph into human values…

i) I’d much rather look inside and read the logs than trust some inscrutable gigawatt-eating giant matrix blob;

ii) I’d much rather have it not separate perception from reasoning. The Gods intended them forever together—The Gods will severely punish us with serious faults and errors if we do not repent from this sin;

iii) I’d much rather not evaporate the Atlantic to make room for one big-beautiful datacenter;

iv) I’d much rather have this little program that runs anywhere than to centralize even more power in few hands;

v) I’d much rather have it learn from one example than from 30 million.

Nobody went out of their way to explicitly design these in because “by 2026 things might look ugly,” right? I may be missing something here, but I think this list comes from The Gods, telling us: “this is the right thing to do; Somebody somewhere has to do it.” Hence it became our job.

Turns out that guy just might have gotten it right; the singularity futurologist fool. We’ll turn the crank up a notch in that second derivative. Objects in the mirror may be closer than they appear.

SILICON VALLEY, MAKE WAY OR BECOME THE WAY. FARG IS COMING.

HAPPY END OF THE WORLD EVERYONE!

Here we go again, but this time AI was not used as a weapon but rather as a vector. Anthropic was compromised (internal? external? does not matter at the moment). Kevin Beaumont put it most concisely, “Their Claude Code extension on Visual Studio Code and Cursor has malware in it, which creates a remote access Trojan to desktops.”

This is extremely serious and it is clear that Anthropic is doing what they can to fix the problem without telling their customers or the people who were using the extension. WTF, Anthropic? Bad security citizen much?

Read about the back door malware installed on Phorion’s alert: https://phorion.io/blog/offical-claude-code-vsx-compromise/. The extension, “drops a persistent SSH backdoor across macOS, Linux, and Windows.” Their advice states:

Organisations should immediately review their environments for use of Claude Code for VSCode 2.1.313 and engage incident response where use of the extension is discovered.

Beaumont’s writeup is here: Anthropic hacked to deliver remote access trojan to Claude developer laptops worldwide. Most salient point and why we’re writing this up here—“Anthropic has not informed customers, and silently deleted the backdoored version without telling anybody.”

Note that binance used this extension, and they were just spectacularly popped to the tune of billions of dollars (probably by the North Koreans). The C2 server used by the attackers has been taken down. We’re at the very tip of the iceberg here…

At BIML we have never been impressed with Anthropic’s FUD-filled, YOLO approach to security. They seem to think this is a game. It’s not. Heck, even if it is, they are losing.

Well, we talk about it on TV anyway. Seems that the White House is very confused about the difference between billionaire technologists and actual scientists. We talked about robbing Peter to pay Paul and its real detrimental impact on science with German DW TV. For the record, we would prefer to see the NSF fully funded with a director.

Want some numbers about what the administration is actually doing to science? Read this.

I had the pleasure of chatting with Aaron Tilton of DW News about the latest AI tidal wave of fear sweeping the world. We answered some questions quite differently than the AI salespeople do. Have a watch.

To prepare for the live piece, I wrote a blog entry you can read here. In that piece I answer these questions:

Do you think AI could realistically become as dangerous as some critics warn, or are these concerns being overstated?

How could AI be misused?

What would effective AI regulation look like in practice, and who should bear the primary responsibility for ensuring safety and oversight: governments, technology companies, or both?

Is Amodei’s call to slow the AI race actually realistic or achievable?

Given today’s increasingly fragmented geopolitical landscape, is meaningful international cooperation on AI governance still achievable?

Some critics suggest that calls to slow development may also serve competitive interests.

Read my answers to those questions here.

Where to start? Lets just start with the best of the hot takes regarding AI Autonomy and the end of the world as we know it. Level set with Melanie Mitchell’s excellent “Misleading Metaphors and Real Risks” piece. OK, have you read it? Good. Now we will spin up before we spin back down.

We have a problem with AI and the problem is deeply tied to Philosophy. When an AI Agent “goes rogue” (whatever that means), who is at fault? Is it the person who prompted the AI system? Is it the AI system itself? Or is it the corporation that constructed the AI system capable of doing evil things in the first place? Who is in charge of deciding these things? What laws apply? If a government wanted to “regulate AI,” could it actually do that? Will there be different regulatory regimes in different planetary geographies?

Should we really be concerned about AI directly choosing to cause the end of the world (say, through designing and building a biological weapon on its own)? Or should we be concerned only about people using an AI to cause the end of the world? Or is the problem that AI is kind of like nuclear weapons and should only be controlled by responsible parties. (Well, the last one seems to fall apart with the current irresponsible and ignorant regime in charge of the United States and its apocalyptic mega-supply of nuclear weapons.)

Is it the gun that is the problem or the person wielding the gun? What happens to morality when REAL autonomy is granted to systems that do not, in fact, die? Do they even have intention?

Oh man. These are tough questions. And they are very real ones. But they are also straight out of a college Ethics course from the Philosophy Department. Lets try to step them down to something slightly more concrete that we might be able to grapple with:

Do you think AI could realistically become as dangerous as some critics warn, or are these concerns being overstated? Oh, that’s more like it. How are current AI systems constructed and how are they used? Looks to me like anthropomorphic descriptions of these models we are building have gotten out of hand. Do AI systems really think and understand? Or are they just very advanced search engines that can generalize and have a natural language interface? Do AI models have intent? I think for now we are very safe saying, no. AI systems are powerful tools which derive any intent they may appear to exhibit directly from the humans using and running them. That’s why I think OpenAI should be in deep trouble for hacking Hugging Face. It was their program on their server doing what they told it to do. So it’s, in fact, all their fault.

How could AI be misused? The most obvious place to start is where computers can be misused. Incidentally, we have laws for that. AI can automate computer misuse just as much as it can automate computer use. So at the low end of the totem pole, we can use AI to pretend we know something when we actually don’t (like a fancy search engine or a calculator). And at the high end, we can use AI to hack other computers illegally (like the best version of MetaHack money can buy). Can you use a hammer to break into a car? Yes. You can. The real game changer comes when we make these things autonomous by allowing them to run themselves. I am much more concerned about Papernot’s worm than I am about powerful but very badly controlled and misdirected frontier model agentic swarms. And Papernot’s worm, as described and built, does not have intent (just as the Morris worm of 1988 had no intent). As long as those agents stay put on the computers we run them on, we should be OK with existing laws. Put Agentic AI together with worm capability and we have some more concerning things to think through. (Incidentally, guess who got in trouble for the Morris worm incident 38 years ago? Morris did.)

What would effective AI regulation look like in practice, and who should bear the primary responsibility for ensuring safety and oversight: governments, technology companies, or both? Well, here we are on a planet with corporations as powerful as or more powerful than some small countries. Regulators, it seems, are out of their depth. But we do need regulators to get busy and do something. At BIML, we have given this real thought and even published what we think in IEEE Computer (in 2024, an eternity ago). The gist of our views is transparency and accountability. Leaving this up to the current crop of AI companies is like leaving the foxes in charge of hen house security. Maybe we should start by enforcing existing computer security laws.

Is Amodei’s call to slow the AI race actually realistic or achievable? Probably not, and that’s exactly why he can spout such nonsense safely from inside his speeding tank.

Given today’s increasingly fragmented geopolitical landscape, is meaningful international cooperation on AI governance still achievable? In these complicated times, we have to deal with the Chinese threat, the Russian threat, and the zombie America threat. We have a United States President who runs around starting wars and picking fights with close allies. So, as a believer in a Carter-like version of human rights, you can see why I am skeptical about international cooperation. Lets get rid of the bullies first.

Some critics suggest that calls to slow development may also serve competitive interests. Regulation can and will be used to lock in market share. If the regulatory burden is paperwork, lobbying, and bureaucracy then little companies are always at a disadvantage. Silicon Valley has known this for at least a decade, probably more.

Now let’s really step back, calm down, and realize that many of these tools we have built and are using are, in actuality, tools and they are not really autonomous agents. Further, we need to understand that the intent we are anthropomorphically ascribing to these tools is a chimera, masking the real bad guy behind the curtain—us.

For a final word, we turn back to Melanie Mitchell who sagely said, “none of the reported [AI hacking] incidents actually involved loss of control at any time, or arguably even “rogue agents,” or any kind of humanlike agency on the part of AI models. Instead, the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking.”

First off, a confession. I have been an enormous fan of Melanie Mitchell’s ever since I was lucky enough to slightly overlap with her as a Ph.D. student of Doug Hofstadter’s. Melanie’s work on Copycat is what made me want to work with Doug. I saw a demo at a talk Doug gave in 1987 to the budding cognitive psychology working group at the University of Virginia (at our invitation) and it was just so much better than SHRDLU that I could not believe my eyes. Then, when I was admitted to the Ph.D. program at Indiana and finagled my way into FARG, there was Melanie, leader of the pack and dispenser of wise gradual-student advice. (Melanie’s Ph.D. is from Michigan, but she made the move to Bloomington with Doug in the later days of her thesis work.). Anyway, fanboi!

Melanie is an essential Voice of Reason in AI; and during the last decade of AI bruhaha, a rare one. Her book is well worth a read.

After the mythos dustup and right around the time of the OpenAI Hugging Face incident(s), Melanie and I spoke a few times on the phone about the realities of computer security and its application to machine learning (design and control).

Yesterday, Melanie published this piece on her blog. Read it. Melanie is right.

And once you’re finished, you can listen to a Silver Bullet interview I did with Melanie in August. Subscribe to the Silver Bullet podcast series to hear directly from technical leaders in AI and Security.

I certainly respect Doug Hofstadter (my esteemed thesis advisor) a great deal more than I do James Carville, but sometimes a turn of phrase is worth a pile of attention. We will hearken all the way back to GEB to begin.

We all remember Aunt Hilary, right? When millions of simple dughof ants interact, their collective movements form fluid patterns. At this high level of abstraction, an entirely new, intelligent persona emerges—Aunt Hilary. Aunt Hilary is capable of holding complex conversations, expressing a distinct personality, and making philosophical inquiries. However, the individual ants that make up her body are completely mindless biological automata. That is, if you examine a single ant, it has zero intelligence. It simply follows basic chemical trails and immediate stimuli. It has no concept of the colony or the conversation Aunt Hilary is having. This is a prime example of emergent computation.

When I did my thesis work, I used this metaphor as inspiration (under the direct guidance of dughof). You can see and play with the results here.

We believe Agentic swarms are emergent phenomenon of this sort (though admittedly much simpler). And we believe our research plan has already been overtaken by real wold events. We plan to proceed with our work anyway.

So what happened that made us write this down? OpenAI’s agentic swarm hacked HuggingFace. Watch Anthropic’s DefCon talk to grok content.

Here’s the bottom line, in our view. Either OpenAI was in control of the swarm and it was rooted at all times ON THEIR SERVERS (meaning they are subject to the CFAA in all of its out-dated glory), or the swarm has already jumped ship. In all likelihood, we expect the latter.

Agentic AI security research needs to stop with the AuthN/AuthZ on small numbers of agents baloney, focus on hard core containment (if only one bit leaks out into the net, that’s enough), and get a handle on emergent behavior.

A technical reference is useful. Details in the METR report about the OpenAI/Hugging Face hack expand the information provided by the DefCon talk. Unfortunately, the report is time and subject constrained. There is some follow-up by one of the authors here.

We love when vRon takes a close look at BIML’s press coverage. This is a great episode.

Look, either OpenAI should be charged with hacking or its autonomous thing escaped the lab completely. Which is it?

As usual, Lemos does a great job covering a complex topic. In this case, nobody is covering what we thing is pretty apparent…the OpenAI swarm is out of its jar. Read the article here.

Dennis Fischer and BIML’s Gary McGraw explore the evolving challenges of AI security, focusing on the implications of autonomous agentic models. They examine the limitations of current measurement benchmarks, the risks associated with sandbox escapes, and the necessity of shifting toward robust security engineering within complex machine learning ecosystems.

If you wonder what BIML is working on next, watch this podcast and find out.