Refusal in Language Models Is Mediated by a Single Direction

Models are aligned for safety to ensure they behave inside certain behavioral limits. Those limits can include the blocking of:

• Bias amplification.
• Generation of harmful or toxic content.
• Production of convincing but factually wrong text.
• Execution of harmful activities.
• Adversarial attacks on the model.

Alignment is necessary to prevent stuff like that, otherwise our ever-increasing agent-based software can autonomously do harm it shouldn't do, like what we saw with OpenAI hacking Hugging Face.

All this to say, safety alignments are guardrails necessary for the creation and use of LLMs. But as with everything, it is hard to draw the line between safety and pure censorship/restrictions that limit our ability to use this technology. Safe by default I think is a good approach, but having an escape hatch to force models to work in any problem space is important to keep these tools as tools and not as a moral/thinking enforcement mechanism. Take, as an example, how corporations try to prevent us from reverse engineering the stuff we purchased and own:

Refusal is a linear direction

One solution for a better use of our freedom is to run models we control, yet those models can still refuse to do totally legitimate tasks. The paper "Refusal in Language Models Is Mediated by a Single Direction" describes that concepts in LLMs are expressed as linear directions in the activation space. If we take many harmful activations and compare to harmless activations, we can calculate a refusal direction.

That refusal direction can be subtracted from the other directions to make refusal not representable. The process of rewriting the model's weights without the refusal is called weight orthogonalization.

This whole process is also known as abliteration, and produces abliterated, decensored, or jailbroken models.

Heretic

p-e-w/heretic is an open source project that allows for the whole abliteration process to take place automatically and output a fully decensored model as a result.

pip install -U heretic-llm
heretic Qwen/Qwen3.8-27B

heretic is so convenient it can download a model by itself and start the abliteration process, no extra steps. In practice it fails, but not by much, different models behave differently and have small details that need some understanding. With that said, the process is mostly automated, and with enough compute you can pull it off relatively fast.

MODEL=/path/to/model/Qwen--Qwen3.8-27B
CHECKPOINT=/path/to/checkpoint/001
heretic \
    --model "$MODEL" \
    --quantization bnb_4bit \
    --max-response-length 340 \
    --batch-size 64 \
    --n-trials 120 \
    --n-startup-trials 60 \
    --export-strategy adapter \
    --study-checkpoint-dir "$CHECKPOINT" \
    --response-prefix $'\n</think>\n\n' \
    --print-responses

What I found very useful is to use --print-responses in there to better visualize what was going on.

I ran this with an NVIDIA A100 SXM4 80GB for 5 hours. The retail rental price of that GPU is 1.89 usd/hr, meaning that the abliteration cost me ~9.45USD.

I got a 10% refusal rate after 120 trials with a Kullback–Leibler (KL) divergence of 0.0903. Even better results than JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:

But keep in mind that I didn't run the benchmarks, and my quantization parameters, trials, and the response length are different.

Does it work?

It does. It really works, you can ask the AI anything and it will do it. It feels like rooting/jailbreaking a phone, you gain exciting niche capabilities at a great safety/security cost. But it is fun.