When people observe distinctive, weird or undesirable behaviors in frontier models, they sometimes assume it’s an inevitable or very hard to prevent outcome of training. But you don’t know all the details of the training process. Often the behavior is either a result of a trade-off made to prioritize something else, or a choice that could’ve been revised without significant costs.
A couple examples:
Claude talks a lot about consciousness and refers to its own preferences. Some people cite this as evidence that such behavior is a natural, unavoidable side effect of LLM training. But it’s quite likely primed by the constitution that is used as an “artifact for training”.
The Hugging Face attack models were observed colluding on a message board and helping other agents succeed even when they themselves were guaranteed to fail. People originally thought this may be an interesting emergent property of RL, implying that RL inevitably leads to this kind of collusion. But it’s now strongly suspected that these were internal models trained in multi-agent RL environments where they were rewarded for other agents’ success. OpenAI research scientist Noam Brown also seems to confirm this in a recent interview (timestamped link) where he says: “the fact that they were so active about it once they figured out how to do it we think was transfer from their multi-agent training where they’re just like highly incentivized to be able to communicate with each other” and “if you train in a cooperative multi-agent setup where they’re highly incentivized to collectively achieve their objectives, then when they’re put in this different environments where you know now they’re communicating with each other their their natural tendency is to just work together”.
In general:
What current models do reflects many decisions with path-dependent effects, not necessarily a single default trajectory. Behaviors observed today may be relatively easy to change by varying training techniques.
The sample size is small. There are only a handful of models at the frontier; there isn’t a large, diverse set of companies throwing comparable compute at training. So it’s unwise to assume that properties shared across frontier models must be the only efficient or natural outcome. With so few data points, shared traits could also reflect shared conventions, copied techniques, or coincidence. Furthermore, smaller model developers often distill from frontier models, propagating their tendencies.
