Agent Task Break Detector
You are deep in a session with a coding agent, and you type something unrelated: a different bug, an idea, an errand.
The agent now carries all the earlier history into a task that doesn't need it. That costs tokens, and stale context
makes its answers worse. The tidy fix is a fresh session (/clear, /new) for each new task, but doing it by hand
breaks your flow.
This model makes that call on every user turn:
Does this new message need the conversation so far, or could a fresh session handle it just as well?
It is a 150M-parameter ModernBERT-base cross-encoder. It reads
the new message together with the recent exchanges and returns p(new_task). It is small enough to run locally on
every prompt, for example from a Claude Code UserPromptSubmit hook. This demo runs the int8 ONNX build on a free CPU.
Status: v0.1 (2026-10-24), experimental. Trained only on synthetic data and not yet checked against real sessions.
| New user message | Session so far (oldest first; one USER: / AGENT: line per turn) |
|---|
How it works
Input. A pair of text segments, as in training:
- the new user message;
- the last 4 exchanges, newest first, each written as
AGENT: …thenUSER: ….
The context is truncated first if the pair runs past 512 tokens, so the message is always read in full.
Output. Two logits (continue, new_task), passed through softmax to give p(new_task).
Decision. Split only when p(new_task) ≥ 0.76. The errors are not equal. A wrong split throws away the
agent's working context, while a missed split only costs some extra context. So the threshold is set for
precision ≥ 0.95 on new_task (int8 validation: precision 0.952, recall 0.921).
Why context matters: matched pairs
The training data includes matched pairs: the same message, once in a session where it is a new task and once in a session where it continues the work. The message alone can't separate them, so the model has to read the context. The examples in the Try it tab are two such pairs from the held-out test split. Load both halves of a pair and compare the scores.
Training
- Data: danielrosehill/Agent-Task-Break-Detector v0.2. Sessions were generated by Claude Haiku 5.5 and labelled independently by Claude Sonnet 5.5. Only rows where the two agreed are used: 5,104 training rows, including 1,025 matched pairs.
- Categories:
far(unrelated task),same_workspace(different task, same repo),subtle_pivot,follow_up,implicit_reference("now do the same for the other one"),hard_negative,counterfactual_continue. - 3 epochs, lr 5e-5, batch 16, bf16, on one NVIDIA L4 (6.9 minutes).
- Code: Agent-Context-Harness,
model/train.py.
Results (synthetic test split)
n = 608 rows, including 115 matched pairs. Pair accuracy counts a pair as correct only when both halves are classified correctly.
| Model | Accuracy | AUC-PR | new_task recall @ precision 0.95 |
Pair accuracy |
|---|---|---|---|---|
| ModernBERT, fp32 | 0.951 | 0.986 | 0.902 | 0.791 |
| ModernBERT, int8 ONNX (this demo) | 0.929 | 0.982 | 0.888 | — |
| TF-IDF, message only | 0.793 | 0.758 | 0.037 | — |
| TF-IDF, message + context | 0.760 | 0.781 | 0.140 | — |
The TF-IDF baselines are almost useless at high precision: phrasing alone doesn't tell you whether a task is new.
Weakest categories: same_workspace (0.867), a different task in the same repo, and counterfactual_continue
(0.896), a message that sounds self-contained but actually continues the work.
int8 probabilities run about 0.14 lower than fp32, which is why the int8 threshold (0.76) is calibrated separately.
Limitations
- Synthetic data from a single generator. Real sessions are longer, messier and multilingual. English only.
- Training data is roughly 50/50. In real use most turns are continuations, so expect more false splits than the test set shows.
- Before you let it act unattended, pick your own threshold on your own sessions.
Links
- Model: danielrosehill/Agent-Task-Break-Detector-ModernBERT
- Dataset: danielrosehill/Agent-Task-Break-Detector
- Project and hook: github.com/danielrosehill/Agent-Context-Harness