Delaying Internal Deployments Until After X Days of Safety Testing
· alignmentforum.org · popularity 21/100 · 0 similar stories today · Reference & How-To (Discovery & Research)
Read the original at alignmentforum.org →
Summary
This post reflects my own takes, and not that of my employer's. This is a simple idea: before a new model can be deployed internally, you must first test it for a minimum, fixed period. You can adjust various axes of this proposal: Duration of the wait: in calendar time, or until you fulfill some condition. What it blocks: external deployment, internal deployment, or the training of future checkpoints. When the fixed period starts: from the first checkpoint, the final checkpoint, or any checkpoint that is “sufficiently similar / representative” to the one that is ultimately deployed. One concrete target that I like is: “No checkpoint may be used internally for purposes beyond safety testing, until at least X calendar days after its hash is filed with an auditor.” This is one specific implementation of internal deployment lags, an idea that the AI Futures Project (AIFP) briefly discusses in How to pace the US frontier as “AI R&D can only be conducted or assisted by AIs that were trained at least 9 months ago”. What does this look like when implemented? Let’s run through a scenario to pin down the details of this proposal. The places where it could go wrong are in the open questions at the end. A legislative body sets a minimum waiting period of X calendar days (starting at 10 and possibly ratcheting up to 90). No checkpoint in a covered model family may be used internally, beyond safety testing, until X calendar days after its hash is filed with an auditor. Continuing to train a checkpoint’s own weights does not count as internal use. The alternative is for the wait to also block the training of checkpoint N+1, and not just the internal use of checkpoint N. I didn’t go with this because it incentivizes developers to pull out fewer checkpoints for testing, and because delaying training results in an algorithmic overhang. More capable models start exhibiting problems that are harder to surface in short periods (e.g. swarms, long-horizon, memory), so X should possibly increase over time. A model family becomes covered as soon as any of its checkpoints (pre-, mid- or post-trained) is substantially more capable than the last model to complete its wait. If the developer initially claims that the model is not covered but later finds that it is, they will be subject to some penalty (to be determined). There is also an absolute floor on capabilities, so that this doesn’t apply to lots of benign models. In an emergency, the auditor can authorize one named checkpoint to be used early, only for responding to a specific incident (e.g. a harmful model is already causing damage, and the developer has just trained a patched model that could counter it). The standards/legislative body confirms or revokes this within [X days]. The developer continually pulls checkpoints throughout the training process. They first send an auditor a hash, a short fingerprint of the exact weights; then, they carry out safety testing for X days before deploying a checkpoint broadly internally without the below rules. All work has to be fully transparent to auditors. Options for how the X days are structured: Staged testing Stage 1, first part of X: pre-registered evals and named testers in an isolated environment. Stage 2, second part of X: broad monitored internal use, but the model can’t launch jobs, merge code, write to training data, or touch prod. Parallel, hidden runs: whenever someone at the developer uses a cleared model, the new checkpoint gets the same task in a separate sandboxed copy of the environment. Its outputs go only to monitors and the safety team, never back to the person who asked. Open questions The rule itself Should we also allow capability testing? What about ML R&D tasks? One way to game this is to register thousands of tasks from your open R&D backlog, and call running them a capability eval. The outputs are real research (e.g. a faster kernel), and even if you ban using them, people still have to read the transcripts to grade them, so they learn what worked. Pre-registering evals doesn’t help, since it fixes the task list in advance but doesn’t check whether the tasks are live problems. You might end up needing a compute cap. Should we give exemptions for doing safety research? This is an important crux to ask alignment researchers about. Without an exemption, safety research is done with a model that is X days behind, but a blanket exemption lets capabilities work pose as safety work. Should a safety fix (e.g. more post-training, or redoing RL from an earlier checkpoint) restart the X-day wait? It would be bad if it did, as Avi Brach-Neufeld points out here . A developer that finds an issue on day 8 of testing loses more time than one that finds it on day 2, so the rule penalizes finding and fixing problems during the wait. This is probably tolerable at small X, but gets worse as X increases. If safety fixes don’t restart the clock, however, developers could just call capabilities training “safety”. What is the principle behind how much X increases? How large is the risk that an emergency clause would be abused? Concerning behaviours aren’t necessarily caught during pre-deployment testing windows, and are often caught through subsequent monitored internal use. How much are we willing to trade off the value of catching some misaligned behaviours via monitoring internal deployments, vs. being able to pace internal R&D? Which models are covered How is “substantially more capable” defined: effective compute, or something like the Epoch Capabilities Index (ECI)? How would an auditor check that the developer has faithfully stuck to the “more capable” definition? How do you avoid bringing classifiers built using a frontier model into scope? Can a developer keep using earlier checkpoints from a family after it becomes covered? A developer could train a second model family and keep it just under the “substantially more capable” line, so it’s never covered. Researchers could then use that family right away, while use of the main family is on hold. This only gets them so far, since coverage is measured against the last model to complete its wait, and an uncovered family never completes one, so the second family can only be ahead by the “substantially more capable” margin. This becomes a bigger problem if the capability measure itself can be gamed. Gaming the wait How can we stop a developer from using a checkpoint for acceleratory AI R&D work during the wait, by labeling that work as an allowed activity (i.e. any AI R&D task whose output gets used for anything other than testing that checkpoint)? How can we cleanly separate out safety testing from other activities? A developer could give a new model a real task during training, such as writing faster training code, and then use that code to train other models before the new model has completed the X-day wait. A developer could make every researcher an A/B tester, so that they use the new model for normal work and rate its answers, and the developer calls this training. Similarly, a developer could have an RL environment that is just how well the model does actual work inside the company. In stage 1 of staged testing, you run the risk that named testers run large agent swarms for capabilities work, in which case the only difference is that there are fewer humans in the loop. This is probably somewhat mitigated by pre-registering evals, but it’s difficult to still allow AWT. In stage 2, you can still copy paste model outputs into training data, or into code and research plans. A developer could also store outputs from the wait and use them the day the checkpoint clears, without breaking any rule. Should there be a blanket ban on training on outputs from the wait, and how would anyone check? In stage 2, how will you detect if the model is launching jobs, merging code, writing to training data, or touching prod? Developers can game the wait by making lots of models and testing all of them at once, so that the testing period ends up being an AI R&D period. For example, with a 30-day wait, a developer could pull out a checkpoint every day and “test” them all, so that researchers always have a day-old model. Possible mitigations are to cap the number of checkpoints that can be tested concurrently, cap total compute across the entire testing period, or pre-register the kind of testing you want to do during the period. If each developer could write its own definition of “safety testing”, a developer could call any monitored employee use testing and move faster. Others would then face pressure to do the same, until the rule no longer binds anyone. A shared template would help, but it depends on the auditor being competent and independent. Parallel, hidden runs There’s no human interacting with the new model, so you miss problems that only show up in back-and-forth with a person. A situationally aware model might also notice it’s in a sandbox and behave differently. Is this okay? How big of a problem is leak risk from reviewers back to capabilities researchers? Verification How can you mitigate the risk that the developer simply does not send the hash of a given checkpoint to the auditor? Potentially: In the near term: the auditor spot-checks that the weights running on internal endpoints match filed hashes. One complication is that a hash of the exact weights changes with any conversion (e.g. quantization, sharding, or a change of number format), so the weights being served would often not match the filed checkpoint byte for byte. You would need either a standard way of serializing the weights, or a hash of the version that is actually served. In the longer run: hardware attestation. Could this be used as a speed dial? The wait gives developers and auditors more time for safety testing. X could also be used as a speed dial, which a standards/legislative body could turn up or down if it wanted to modulate the pace of AI progress. I think that this only works as a speed dial if this proposal delays internal R&D loops; delaying external deployment would mainly have second order effects like: Each launch shifts the start of new revenue back by a short period. These delays don’t stack across launches, since every launch is shifted back by the same amount. This could still in principle strain a given developer’s ability to pay for compute contracts, but the impact might be small, since I expect that you are mostly shifting when new revenue starts instead of losing it entirely. Some reduction of race dynamics, though I expect the effect size to be quite small. It’s unclear whether this causes a one-time delay in when the most capable model can be used internally, or a lasting change in the pace of AI progress. This depends on how much the newest model speeds up R&D compared with a model that’s X days older. I have a hunch that, if implemented well, this proposal would have a larger impact on the pace of recursive self-improvement (RSI) itself than on the onset of RSI. Much of the value of this proposal will depend on how effectively you can specify what counts as “safety testing”, given that the most useful safety testing often looks like real use. A narrower definition limits how much acceleratory AI R&D work gets through during the wait, but it cuts against giving safety researchers signal of misalignment that only show up in broad, realistic use. What are the benefits and drawbacks of this proposal? Benefits include: As a speed dial, this should get stronger as things speed up, as the gap between the newest model and the last cleared model gets bigger, whereas a compute cap might stop binding if most of the progress is coming from algorithms. Developers would be incentivized to hand the checkpoint to an auditor as soon as possible so that they can deploy more quickly (assuming that they can’t use unfiled checkpoints without being detected). It doesn’t result in an algorithmic overhang, because you still allow training, and presumably any new compute multipliers you find are put into the newest model that you are training. The exception is if a developer can secretly stock up capabilities outputs during the wait, e.g. by storing work from hidden parallel runs or ML R&D evals. Drawbacks include: Developers get a lot of signal on misalignment by internally deploying a model. You don’t get less misalignment signal for a given model, since it will be broadly deployed eventually, but you delay the timeline on which a safety issue can be fixed. Today a problem seen in real use can feed into the next model’s training almost straight away; under this proposal, a model will get X more days of training before an issue is detected. Mitigations: the parallel hidden runs idea; a norm that when real use of the cleared model turns up a problem, you test all the waiting checkpoints for it straight away. Safety research done with the model is X days behind. Unlike proposals that limit the most capable model you have at a given time (like compute caps), this limits the most capable model you can use . The risk incurred by foreign competitors stealing model weights is therefore comparatively higher. It’s hard to verify how a checkpoint was used, so you may rely very heavily on auditor judgement. This proposal may not work with continual learning, if there were a breakthrough. Discuss
Found in: AI Alignment Forum. Rating basis: single outlet; outlet tier 2 (unlisted); 1 feeds; feed position 1.