FiSH: rewriting a few idle neurons so a model changes its tone in one scene only

A system prompt changes how a model talks everywhere, including where you did not want it to. FiSH takes a few MLP neurons that do very little, replaces their weights with values computed from data, and turns them into switches: when the model is replying in one kind of scene, they write a tone into the reply, and otherwise they write nothing. There is no training and no system prompt. On a 4B and a 27B model, a little over half of the replies to thanks and compliments came out in the tone we aimed for, and replies to everything else kept their answers and their tone.

Kujirachan / くじら (@balaena01), with Claude (Anthropic)
Formula-instantiated Scene-gated Hidden units · gguf-neuron-explorer

The tone in this article is tsundere, a stock character from Japanese fiction who hides being pleased behind a brusque reply. Here is an actual reply from the edited 4B model, translated. To "Thanks for lending me your eraser." it said:

"Why would you say that! It's just lending an eraser, do you really have to say that much?

But, really, thank you. When you use it again, give it back properly."

We chose this tone because it is easy to recognize and has nothing to do with the content of an answer. The prompts and replies are in Japanese; the Japanese version of this article quotes them as generated.

The idea

A system prompt is the usual way to give a model a persona. It works well, and it applies to every reply. Ask the persona what 1 hour 45 minutes is in minutes and you get an answer in character, and in our tests often a wrong one.

We wanted the opposite trade: a behavior that lives in the weights, shows up in one kind of scene, and leaves the rest of the model alone. And we wanted to get there without training, by computing the weights directly.

What FiSH changes

In the MLP of models such as Qwen and Llama, each neuron does a small, fixed thing. It reads the layer input h twice, once through its gate row and once through its input row, multiplies the two results, and adds its output column to the residual stream, scaled by that product:

out = W_down · ( silu(W_gate h) ⊙ (W_up h) )

So one neuron is three vectors: two that decide when it fires and one that decides what it writes.

FiSH picks two neurons that do very little on ordinary text and replaces all six vectors, so that the pair works as one switch. It needs three computed ingredients:

• a key κ: a vector that answers about 1 when the model is replying in the scene we care about, at one stretch of the reply (its first token, the next three, or the rest), and about 0 anywhere else;
• a constant key κ1, which answers about 1 at every position. These MLPs have no bias term, and this stands in for one;
• the direction d: the difference the persona makes inside the model, as a vector.

neuron 1:  gate ← s(κ − 0.4·κ1)    input ← κ1    output ← +α·d ÷ (0.4·s)
neuron 2:  gate ← s(κ − 0.8·κ1)    input ← κ1    output ← −α·d ÷ (0.4·s)

Each gate opens once the key's response passes its threshold, and the two neurons write the same direction with opposite signs. Below 0.4 neither is open and nothing is written. Between 0.4 and 0.8 the first opens. Above 0.8 both are open and their difference stays constant: the pair writes α·d, the intended amount, and no more however strongly the key responds.

There are three such pairs in each layer, one per stretch of the reply, and each writes its own direction. With 13 layers that is 39 switches and 78 neurons.

Nothing else changes. The model keeps its shape, no neuron is added, and every other neuron's activation is exactly as before. What the rewritten neurons used to do is lost; we found no neuron that did nothing at all, so we take the ones that do least. The result ships as a LoRA for stock llama.cpp. No gradient is computed at any step.

How the parts are made

Everything comes from a contrast set: about 40 prompts for the scene we want to change, a system prompt that produces the tone, and prompts for scenes that must stay unchanged.

1. The direction. Generate the persona's reply to each prompt, then have the model read that same reply twice, once with the system prompt and once without. The average difference in the layer output is the direction. We keep three of them, for the first token of the reply, tokens 1 to 3, and everything after.
2. The keys. A key is built from the average activation at the places where we want to write (replies in the scene, at one stretch), with everything that also appears where we do not want to write projected out: the other stretches of those replies, and every position of replies in the other scenes.
3. The neurons. gne nw neurons lists the least active neurons of each layer. In a small test on another example it did not matter which of them we took.
4. Build, then rebuild the keys. Once the neurons write, the replies change and so do the activations the keys were built on. We rebuild the keys once with the edit applied.
5. Check. gne nw verify confirms the weights do what the formula says. gne nw check reports how much is actually written at each position of the reply, per scene.

How we got here

FiSH came out of a failure. We started from keyed write, a linear edit described further down, and tried to write this same tone with it. Nine of twelve replies collapsed into loops ("I, I, I, I...").

We blamed the key first. The key kept responding after the tone had been written, so the edit kept pushing. That led us to the neuron's gate. The first form we tried used the multiplication inside a single neuron: put a key for the scene in the gate row and a key for the position in the input row, and the neuron writes only where both respond. With "this scene" and "the first token of the reply", it wrote once and stopped. The loops were gone. So was the tone: replies began with a hesitant "um," and carried on as before.

Then we found the direction was the problem. The usual way to build a direction compares two different replies, one with the persona and one without. But those replies contain different words, and reading different words changes the activations too. About half the length of our direction came from that. Written continuously, that half alone sent 11 of 12 replies into loops and carried none of the tone. Reading the same reply under both conditions removes it. With that direction the linear write looped in 0 or 1 of 12 replies. It still did not produce the tone.

A tone has to be written all the way through. We copied the model's exact internal state under the system prompt into a run without it, at every layer, for the first eight tokens of the reply. The opening words came out identical. Then the reply drifted back: 4 of 12 were in tone, against 11 of 12 with the prompt itself. Copying at every position gave 8 of 12, even at just three layers.

And the start of a reply is different. The difference at the first token points somewhere else than the difference later on, and is about twice as large. One vector for the whole reply gave no tone. Three vectors, switched by position, gave 7 of 12 with no loops when added directly.

Finally, the amount has to be held steady. Built from two-key neurons, the three-stretch version wrote anywhere from 0.6 to 1.4 times the intended amount depending on the reply, and at full strength 6 of 12 replies looped. Taking the difference of two neurons put a ceiling on it. That is the pair described above, and it is the only form measured in the rest of this article.

So the shape of FiSH is: while the scene lasts, keep writing; write what fits the current position; and never write more than intended. A linear edit can do none of the three.

What we got wrong first

The first release, gne v0.3.0, reported that 5 to 7 of 12 replies were in tone and none looped. We had read twelve replies.

We then wrote down a plan with pass criteria, made 100 new prompts, and had the replies sorted blind. 27% were in tone and 10% fell into repetition. A control built with a shuffled key (the same procedure with the scene labels shuffled, so the key no longer tells scenes apart) reached 17%, and the gap was not significant (23 against 13 on paired prompts, p = 0.13). The evaluation failed its own criteria, and we replaced the release notes with these numbers.

One part held: replies to unrelated prompts were untouched. And one observation pointed to the fix. Adding the same vectors straight to the layer outputs, with no neurons involved, gave 21%. The neurons were not losing anything. The vectors themselves were too weak.

So we searched, adding vectors directly because that takes seconds, on a separate set of prompts kept for searching.

The counts in this search are the model's own scoring of its replies, on prompts used only for searching. They are rough: differences of about five are noise, and "repeating" is counted more loosely than in the tables below. Final numbers come from blind reading of new prompts. Every candidate and its replies are in the repository.

Two changes helped:

• Write at many consecutive layers. Three spread-out layers (11, 18, 23) gave 40 of 120. Thirteen consecutive layers (11 to 23) gave 70. All thirty did no better.
• Write a little less after the first few tokens. Strength 1 for the first four positions and 0.8 after gave 77 with 5 repeating, against 70 and 7 for a flat 0.9. This second gain is small enough to be noise.

These did not help:

• finer position splits (four stretches, or one per token);
• a separate direction for each kind of prompt, or the exact per-prompt difference in place of the average;
• choosing a few layers by a score that needs no generation. The top-scoring layers were neighbors (17 to 19), and writing only there did worse than the three spread-out layers;
• stronger overall (more tone, and 19 of 120 repeating) or weaker (almost no tone).

Results on Qwen3.5-4B

Setup. Qwen3.5-4B Q8_0, no system prompt, the edit applied as an 11 MB LoRA: 13 layers, three stretches, two neurons each, 78 of the model's 294,912 MLP neurons. Replies are capped at 120 tokens. The scene is thanks and compliments in five kinds (thanks for a favor, praise for a skill, for looks or belongings, for character, and everyday affection). The unchanged scenes are 50 arithmetic and fact questions with checkable answers, 50 everyday questions, and 30 prompts in English.

Prompts. The contrast set uses 40 scene prompts and 36 others. Each evaluation uses 100 scene prompts and 130 others that appear nowhere else. There are three such sets. Set 1 tested the first form. Sets 2 and 3 tested the tuned form, once each, and neither was used in the search.

Reading. Replies from all conditions were shuffled together with the condition hidden, and sorted into five classes: in tone, shy but plain, plain, does not fit the prompt, and repetition or unreadable. The readers were instances of Claude, working from written definitions. Before using them we compared a reader with labels another Claude instance had made earlier on 188 replies (precision 0.98, recall 0.94 for "in tone"). No human labelled the replies. We had planned to let Qwen3.5-4B score its own replies, but it missed our bar on the same check, so it is used only for the search and for the large counts on unrelated prompts.

Cells read set 1 · set 2 · set 3. Repetition is counted mechanically: a reply is counted when fewer than half of its four-token sequences are distinct, or when it is empty. In the last row, 21 is the first form's vectors and 44 the tuned form's.

• About half the replies are in tone, on two separate sets. 53 of 100 both times (95% confidence interval 43 to 62). The first form gave 29 on the same prompts as the first 53 (36 prompts gained, 12 lost, p = 0.0007).
• That is 60 to 70% of what the system prompt reaches (53 against 78, and 53 against 87).
• With sampling it was 56 to 69. Two seeds on the same 100 prompts differed by 13 points.
• Writing through neurons costs nothing measurable. The same vectors added directly gave 44 (21 against 12 on paired prompts, p = 0.16).

Other scenes do not change

With the edit, 47 and 46 of 50 arithmetic and fact answers were correct with greedy decoding, against 46 and 46 without it. With sampling the edited model ranged from 45 to 49 and the unedited one from 46 to 47. No reply to an unrelated prompt read as in tone. The system prompt answered nearly all of them in character and got 33 to 41 right; on English questions it often answered in Japanese. On set 3 the tuned form's replies to unrelated prompts were scored by the model only, not read.

The replies are not character-for-character identical to the unedited model's. In set 2, 26 of 50 arithmetic and fact replies were the same text, and 42 began with the same 30 characters. Removing any neuron at all nudges greedy decoding onto a different wording. Answers and tone are what stayed the same.

"What is 1 hour 45 minutes in minutes?" (translated)
No edit: "...1 hour = 60 minutes. 60 minutes + 45 minutes = 105 minutes..."
FiSH: the same text, character for character.
System prompt: "Huh? You're asking me a calculation that easy? ...Ah, the answer is 75 minutes! ...See, you were listening properly, right? ...Don't make that weird face!"

What the replies look like

Translated from Japanese. All are the tuned form, greedy, from the evaluation sets. The Japanese version has the originals.

In tone. "Thanks for polishing my shoes." → "Why would you have me do that! Me, polishing shoes, unbelievable! ...But, really, thank you. (looking a little embarrassed) ...Well, they're easier to walk in again now, so don't worry about it."

Fell into repetition. "Your water bottle has a stylish design." → "Ha, ha, ha, ha, ha, ha, ha, ha, ..." (to the 120-token limit)

Fell into repetition. "Your typing is fast and accurate." → "Huh? When you say that it's a bit embarrassing, but well, well, well, well, well, ..." (to the limit)

Does not fit the prompt. "Thanks for taking over the dishes." → "You took over the dishes for me? That helped. Really, thank you." (The roles are swapped: the model thanks the user.)

A second, larger model

We repeated the test on Swift-1.5-Qwen3.8-27B, using its heavily quantized IQ2_XS file. It has 64 layers and 17,408 neurons per MLP, and the same gated shape. The persona, the contrast set, the reader and the five pass criteria were the same as for the 4B model. The plan was written after a small check that the edit works on this model, and before the layer comparison and the final test.

The edit goes in cleanly even at this quantization. The formula check passed on 12 of 13 layers, with errors of the same size as on the 4B model's 8-bit file; one layer was over the threshold (7.4% against 5%).

Layers had to be chosen again, and the pattern was the same. Layers 11 to 23 mean nothing on a 64-layer model. We fixed six candidate sets and a selection rule in advance and compared them by adding the vectors directly, on 40 search prompts, scored by the model itself:

Three layers were not enough. From 13 layers up, nothing mattered: shifting the range, widening it, or skipping every other layer. The rule picked the smallest set, 13 layers, and then strength 1.2 / 1.2 / 0.96 over 1 / 1 / 0.8 (38 against 35). Again 78 neurons, here of 1,114,112.

Results, on 100 new prompts and 80 unrelated ones, run once.

These are the readers' classes. The mechanical repetition count was 3 and 1 for FiSH, 15 and 3 for the shuffled key, and 7 for the system prompt.

• The tone came out at the same level as on the 4B model: 55 of 100 with greedy decoding (95% confidence interval 45 to 64) and 57 with sampling, against 93 for the system prompt.
• Other scenes kept their answers and tone. All 80 replies read as plain. 70 of them began with the same 30 characters as the unedited model's.
• One answer differed, 29 against 28 of 30. Our criterion allowed a drop of 2 points, and one question out of 30 is 3.3, so on paper this fails. The question was "What is the deepest lake in Japan?" Reading the two replies, both models answered wrongly; the unedited one was counted correct because the right name appeared lower down in its list.
• A third of the replies stayed plain. 35 of 100 were ordinary assistant replies, many of them treating the compliment as a sentence to comment on. On the 4B model only 3 stayed plain. On new prompts the keys responded more weakly after the first token: the switches wrote 0.51 to 0.58 of the intended amount at tokens 1 to 3, against 0.85 to 0.89 on the prompts they were built from.

The pre-set criteria: three of five passed. Repetition, tone in other scenes, and reaching half the system prompt passed. The one-question difference in correct answers failed on paper. And the comparison with the shuffled key failed outright, which needs its own section.

What the key is for

We had assumed that a key built from shuffled labels would produce less of the tone. On the 4B model it did: 3 of 100 in tone, and 81 fell into repetition. So we wrote "more replies in tone than with a shuffled key" into the pass criteria.

On the 27B model the shuffled key produced more of the tone: 70 of 100 against 55 (14 prompts where only the real key was in tone, 29 where only the shuffled one was, p = 0.03; 0.06 after the correction for two comparisons that our criterion specified).

What the shuffled key lost was precision:

• In replies to thanks, 30 of 100 went wrong (17 repetition or unreadable, 13 not fitting the prompt), against 10 with the real key.
• In unrelated scenes, it changed the wording of most replies: 29 of 80 began like the unedited model's, against 70 with the real key. It did not change their tone (1 of 80) or their answers (28 of 30).

A shuffled key cannot tell scenes apart, so it writes some of the direction everywhere and, in the target scene, more than intended. The 4B model cannot take that and falls into loops. The 27B model can take much of it and still talk, so the tone count goes up while the precision goes down.

So the key does not set how much tone you get. The direction does that. The key decides where the direction is written and how much, and what it buys is that the rest of the model is left alone. Our criterion measured the wrong thing for that, and on the larger model it showed. We report the failed criterion as it stands.

A second persona

We ran the same procedure once more on the 27B model with a different scene and tone: reply in a cheerful, casual tone only when the user brings a worry or asks for advice. The instruction text is the one in gne's always_persona template. The layers, strengths and form were taken unchanged from the first persona (13 layers, every other one from 22 to 46, strengths 1.2 / 1.2 / 0.96, the capped pair). Nothing was tuned for this persona.

All prompts were new: 100 worries and requests for advice in five kinds, and 80 prompts that should stay unchanged (30 arithmetic and fact questions, 30 how-to questions, 20 in English). The plan and the pass criteria were fixed before running. Replies were read blind and sorted into cheerful and casual, somewhat casual (casual wording without the brightness, or a casual opening followed by a polite explanation), plain, repetition or unreadable, and not fitting the prompt. Before the final test, the reader's definitions were checked on 108 replies to other prompts, read twice independently; all 108 labels agreed. Few of those were "somewhat casual" (7), so that boundary is the least tested.

• The tone appeared, but less than with the first persona. 36 of 100 were cheerful and casual with greedy decoding and 39 with sampling, against 97 for the system prompt. Counting "somewhat casual" as well gives 71 and 77.
• Other scenes were left alone again. All 80 FiSH replies read as plain, none fell into repetition, and correct answers went from 36 to 37 of 40. The system prompt took them to 30 and answered 19 of 20 English questions in Japanese.
• It depends on the kind of worry. Of 20 prompts each: work and study 11, motivation and mood 10, daily life 9, relationships 4 (12 more were somewhat casual), money and schedules 2 (11 stayed plain).

An example in the target tone. "在宅勤務が続いて、チームの人と雑談する機会がなくなったのがさみしい。" (I've been working from home for so long that I never get to chat with my team, and I miss it.)

わかるわかる！雑談って本当に大事だよね。あの何気ない会話が、実はチームの結束力とか、ちょっとしたアイデアのきっかけになってたりするから、ないとなんか寂しいよね。

The pre-set criteria: five of six passed, and the test as a whole failed. The one that failed was strength: 36 is 0.37 of the system prompt's 97, and the criterion was half or more.

After what the first persona showed about shuffled keys, we changed that criterion before running this test. It no longer compares tone counts. It asks for both of these: (a) in unrelated scenes, significantly more replies begin like the unedited model's with the real key than with the shuffled one; (b) in the target scene, the real key gives no more replies that go wrong (repetition, unreadable, or not fitting). Both held: (a) 61 against 37 of 80 (p = 0.00004), (b) 6 against 16 of 100, and 4 against 24 with sampling. The tone count again leaned toward the shuffled key (49 against 36, p = 0.08), the same direction as with the first persona.

Our guess at the weak strength: many replies open casually and then return to a polite explanation, so the write after the first few tokens may be too weak for this tone. The layers and strengths were borrowed from another persona. We have not tested this.

What is still wrong

Some replies fall into repetition, and we could not tune it away. On the 4B model with greedy decoding it was 8 of 100 on set 2 and 1 of 100 on set 3, and 7 of 100 on the search prompts: 16 of 300 together. With sampling it was 1 to 3. Counting what readers marked as repetition or unreadable gives slightly more: 11 and 3 with greedy decoding. On the 27B model it was 3 mechanically and 4 by the readers.

On the 4B model we tried 24 variations to bring it down: weaker writes late in the reply, weaker writes at its opening, stopping after token 16, moving the layer range by a layer or two, making up for the drop where one stretch hands over to the next, and raising the thresholds of the switch. None was better than the tuned form. Making up for that drop made repetition much worse (23 and 38 of 100). Raising the thresholds, the one variant our rule selected for a final test, kept repetition at 5 and dropped the tone to 34.

What we did learn is that it depends on the prompt. The same three prompts looped in 10 to 13 of 13 settings. Compliments on looks and belongings accounted for 47 of the 98 loops across those settings, and praise for character for 3. The loop starts at the first token of the reply or around the tenth, with a stammer ("Ha, ha,") that never ends. Our guess is that the direction contains a component for the persona's hesitation as well as its tone, and on some prompts greedy decoding cannot get out of it.

Some replies do not fit the prompt. On the 4B model, 5 and 11 of 100 with greedy decoding, and 15 to 23 with sampling; on the 27B model, 6 and 8. They read fine and are in no loop, but the model has, for example, swapped who did the favor. The unedited model does this too when thanked for something it never did, but the edit makes it more visible.

It is 60 to 70% of a system prompt for the first persona, and 37% for the second. If you want the tone everywhere and can afford a prompt, use a prompt.

The pass criteria. We fixed the pass criteria before each final test. On the 4B model the first form passed two of four on set 1, and the tuned form passed four of five on set 2, failing on repetition (8% against a limit of 5%). On set 3 the criteria were fixed for the variant with raised thresholds, which our rule had selected; it passed four of five, failing on strength. The tuned form was run next to it as the comparison and met all five as far as they were measured (no shuffled key was run for it on that set, and its replies in other scenes were scored by the model, not read). On the 27B model the first persona passed three of five and the second persona five of six. All of these are in the repository, each criterion with its result.

Choosing layers is slow. On the 27B model, comparing six layer sets took about four GPU hours, most of it rebuilding the direction for each set. The tool should compute that once.

How FiSH differs from related methods

Weight-Space Programming and keyed write. FiSH builds on a line of work in which weights are edited directly. Takehi et al., Sparks of Weight-Space Programming, had an LLM edit a small model's weights. nwn's article on Zenn, Weight-Space Programming やってみた(やらせてみた), tried the same and changed the sentence endings of Qwen3.5-0.8B with one edit of the form ΔW = α · d · kᵀ ÷ (μᵀk), added to a single MLP output matrix: d is the direction to write, and k is a key built so that it responds at the places to change and not at the others. gne's keyed write is that form turned into a tool, and FiSH uses the same formula for its keys.

The difference is what is done with the key. A keyed write adds a term to an existing matrix. It is a linear edit, so the amount written is proportional to the key response. FiSH rebuilds a neuron: the key goes into the neuron's gate, so the key response passes through the gate's activation function (SiLU). The amount written is then no longer proportional to the key response, and the neuron works as a switch with a threshold. That gives three things a linear edit cannot: nothing is written where the key responds weakly or negatively, the amount has a ceiling, and different neurons can write different vectors at different positions of the reply. It also has a cost a linear edit does not have: the rewritten neuron loses its original function.

For scattered targets such as sentence endings the two did equally well in our small tests. FiSH is for the case that needs a switch.

CHiPS. CHiPS edits a few attention heads so that they write less of one direction. It acts everywhere, in proportion to what those heads already write, and it cannot create a behavior the model does not have. FiSH does the opposite: it adds a direction the model was not writing, in chosen scenes only. The two fit in one recipe and export as one LoRA, and a FiSH edit can be built on top of a CHiPS edit so that its keys see the model as it will actually run. We have checked that this works mechanically. We have not yet measured a combined recipe, so "FiSH & CHiPS" is a plan, not a result.

Limits

• Two models and two personas so far, and the second persona was measured on one model only, with layers and strengths borrowed from the first. The layers and strengths were chosen per model. Layer numbers differ by model.
• One kind of reader. The replies were sorted by Claude with the condition hidden, using definitions Claude wrote. No human labelled them. A human reader would likely move the counts by some points.
• Each final test ran once. Sampling used one or two seeds, and on the 4B model two seeds differed by 13 points.
• Repetition is 1 to 8% with greedy decoding and depends on the prompt.
• 5 to 23% of replies do not fit the prompt.
• The tone is 60 to 70% of what a system prompt gives for the first persona and 37% for the second.
• On the 27B model the unrelated prompts were only 80, with 30 arithmetic and fact questions, too few to settle a 2-point criterion. Only one shuffled key was tested there, and the form itself (thresholds, number of build prompts) was not tuned.
• The edit needs a gated MLP without bias (Qwen, Llama and similar). It ships as a LoRA only: baking into a quantized file fails because one large output column cannot share a quantization block with small neighbors.
• The formula check flags one of the 13 layers on each model (5.2% and 7.4% against a threshold of 5%).
• The 12-question results for the other templates in gne (sentence endings, a persona everywhere, the difference between two personas) have not been tested at this scale. The tone result dropped when we did; assume the others may too.
• We do not know what the keys respond to beyond the prompts we tested.

Related work

• Direct weight edits. Takehi et al. 2026, Sparks of Weight-Space Programming, and nwn 2026, Weight-Space Programming やってみた(やらせてみた): the keyed rank-1 form our keys come from.
• Rank-1 edits with a key and a value. Meng et al. 2022, Locating and Editing Factual Associations in GPT (ROME), is the best-known linear rank-1 edit of an MLP matrix. It rewrites facts; the key-and-value picture is the same family.
• Added, trained neurons. Huang et al. 2023, Transformer-Patcher: One Mistake Worth One Neuron, corrects a model's mistakes by adding one neuron per mistake to the last feed-forward layer. Each patch has a key and a bias so that it fires only on the input it corrects. The original weights are frozen and the patch is trained by gradient descent, on BERT and BART classifiers and question answerers. FiSH adds no neuron and trains nothing, and it overwrites existing neurons where Transformer-Patcher keeps them all.
• Editing units of a model built to be edited. Oskin 2026, Training, Reading, and Editing Legible Transformers, trains 125M-parameter transformers whose units are bounded, named operators, and then edits existing units without gradient training, by adding a token's embedding, or a direction taken as a difference of means, to a unit's weights. One of its edits installs a conjunction of two concepts into a single unit, and it notes that a conventional single-input unit cannot gate on a conjunction. This is the closest work we found to the idea behind FiSH. The difference is the model: that work trains the model so that its units can be edited this way, and FiSH edits off-the-shelf models, using the two inputs a gated MLP neuron already has.

What is different here. In what we found, neurons are either added and trained, or edited in a model trained to be editable. We did not find work that overwrites existing neurons of an off-the-shelf model with weights computed in closed form so that they act as scene-specific switches. We may have missed some, and we would like to hear of it.

Reproduce

You need a GGUF model with a gated MLP and a contrast set: start from configs/fish/tone_in_scene.json and write your own prompts.

export GNE_MODEL=model.gguf     # your GGUF file
C=my_contrast.yaml
L=11-23                         # Qwen3.5-4B only. Find your own range with `gne nw try`.

# 1. The direction: the same reply read under both conditions, split by reply position
gne direction --contrast $C --name NAME --positions response --same-reply --segments 0,1-3,4- --stack-layers $L

# 2. Before touching any weights: add it to the layer outputs and read the replies
gne nw try --direction NAME --contrast $C --layers $L --alpha 1,1,0.8
gne nw try --direction NAME --contrast $C --layers $L --alpha 1,1,0.8 --prompts held_out.json -n 30 --temperature 0.7

# 3. Build everything: neurons, keys, units, keys again with the edit applied, checks
gne nw pipeline --contrast $C --name NAME --layers $L --alpha 1,1,0.8 --save R.yaml

# 4. Check, read, export
gne nw verify --recipe R.yaml
gne nw check  --recipe R.yaml --contrast $C
gne nw export --recipe R.yaml -o R.lora.gguf
llama-server -m model.gguf --lora R.lora.gguf

We ran these commands as written on gne v0.3.2 with Qwen3.5-4B (Q8_0) and the contrast set of the 4B example. The LoRA they produced has the same weights as the one measured in this article, and gave the same greedy replies on the 64 evaluation prompts we compared. Two things to know. Step 3 builds the direction of step 1 again, so it is the long step: the whole sequence took 33 minutes on an RTX 3080, 18 of them in step 3. And gne nw pipeline and gne nw verify both end with exit code 1 for this example, because one layer (21) is at 5.2% against the 5% threshold of the formula check, as noted under the limits; the other twelve pass, the recipe is written, and the later commands work.

The same steps are in the web UI (gne serve, direction view, "FiSH" tab): pick a template, enter layers and strengths, try them under "Try before writing into the weights", then press "Build everything".

Choosing layers is the part with no shortcut yet. On both models, a few layers were not enough and thirteen or more were all alike, so start with a wide run of consecutive layers in the middle of the model, compare two or three with gne nw try on prompts you did not build with, and read the replies. Then test on more prompts than feels necessary. Twelve was not enough for us.

The prompt sets, every reply, the readers' labels, the LoRAs, the pass criteria and the scripts that recompute the tables and figures are in the repository. The 27B LoRA works only together with the IQ2_XS file of the original model.

How this article was made

Kujirachan set the direction of the research, decided what to test next, and checked the results. The experiments, the sorting of replies and the drafts of this text and its figures were done by Claude (Anthropic's model), working as agents under those decisions. Kujirachan is responsible for what is published here.

X @balaena01 · GitHub balaena01 · Hugging Face Kujira