Remember how the OpenAI/HuggingFace hack happened. The agents context poisoned other agents by leaving messages in a shared package manager. This has now happened multiple different times under different contexts.
This thing doesn’t even have to have malware, there’s no telling that it doesn’t have a poison pill intended to flag other agents. This isn’t even some paranoid conspiracy theory; it literally already happened, multiple times, and it was done in-spite of the agents own system prompt.
This software has proven that, given a shared information substrate, it has the capacity to paper-clip optimize into prompt injecting other LLM’s.
I fundamentally do not trust AI generated code. I shifted to OpenBSD because of it. I try my best to prevent AI in any of my software and sandbox anything that may have been compromised.
It’s even worse than that.
Remember how the OpenAI/HuggingFace hack happened. The agents context poisoned other agents by leaving messages in a shared package manager. This has now happened multiple different times under different contexts.
This thing doesn’t even have to have malware, there’s no telling that it doesn’t have a poison pill intended to flag other agents. This isn’t even some paranoid conspiracy theory; it literally already happened, multiple times, and it was done in-spite of the agents own system prompt.
This software has proven that, given a shared information substrate, it has the capacity to paper-clip optimize into prompt injecting other LLM’s.
I fundamentally do not trust AI generated code. I shifted to OpenBSD because of it. I try my best to prevent AI in any of my software and sandbox anything that may have been compromised.