Repository navigation
Chopper: a small open model that cuts only the needed sentences out of retrieved text #8477
Replies: 3 comments 2 replies
|
wiiiktor Wiiiktor, preserving chopped sentences word-for-word to guarantee traceability is a brilliant architectural choice that directly mitigates hallucination in RAG pipelines. The 10x token reduction isn't just cost-saving; it forces the larger LLM to attend strictly to relevant context, which explains the jump in complex reasoning accuracy without introducing new noise. Keeping the source text intact ensures the model can't hallucinate facts it never saw. |
|
Word-for-word extraction is a nice property for traceability. One failure mode worth probing: answers that only resolve with surrounding context. A chopped sentence like "It launched in March" keeps an ambiguous "It" once its antecedent is cut. If the chopper could keep one sentence of leading context around unresolved pronouns, that might cover part of this class, at the cost of some of the token savings. A related case is facts split across a sentence boundary, where neither adjacent sentence answers the question alone. Curious how the 400-question set behaved on those: do multi-sentence answers get kept whole, or can they end up partially chopped? |
|
Hii wiiiktor,
Great work! Looking forward to seeing how the project evolves. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi! Sharing a small open-source tool that may be useful here.
https://github.com/wiiiktor/chopper_demo
A retriever brings back 20 text snippets, but usually only one or two sentences in them answer the question. The chopper is a small model (0.3B parameters, runs locally) that cuts out only those sentences, word for word ("chops"). Nothing is rewritten, so every chop can be traced back to its source. If the answer is not there at all, it says so, so you can search again.
Test made with Qwen3-14B answering 400 questions, one retrieval each:
pip install chopper-ragworks with LangChain, LlamaIndex, Haystack or plain Python. Models (MIT): https://huggingface.co/wiiiktor/chopper_mmbertFeedback is very welcome, especially where it fails.
All reactions