-
Notifications
You must be signed in to change notification settings - Fork 1.6k
feat(longtext): implement long text splitting strategy with Markdown … #2038
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
fdc310
wants to merge
2
commits into
master
Choose a base branch
from
feat/long-text-split
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,224 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import re | ||
|
|
||
| from .. import strategy as strategy_model | ||
|
|
||
| import langbot_plugin.api.entities.builtin.pipeline.query as pipeline_query | ||
| import langbot_plugin.api.entities.builtin.platform.message as platform_message | ||
|
|
||
|
|
||
| @strategy_model.strategy_class('split') | ||
| class SplitStrategy(strategy_model.LongTextStrategy): | ||
| """Split long text into multiple message segments with Markdown awareness.""" | ||
|
|
||
| async def process(self, message: str, query: pipeline_query.Query) -> list[platform_message.MessageComponent]: | ||
| segments = self.split_text( | ||
| message, | ||
| query.pipeline_config['output']['long-text-processing']['threshold'], | ||
| ) | ||
| return [platform_message.Plain(text=segments[0])] if segments else [] | ||
|
|
||
| def split_text(self, text: str, max_length: int) -> list[str]: | ||
| """Split text into segments respecting Markdown structure. | ||
|
|
||
| Priority: | ||
| 1. Markdown structural boundaries (headings, code blocks, horizontal rules) | ||
| 2. Paragraph breaks (blank lines) | ||
| 3. List item boundaries | ||
| 4. Line breaks | ||
| 5. Hard cut (fallback) | ||
| """ | ||
| if len(text) <= max_length: | ||
| return [text] | ||
|
|
||
| blocks = self._parse_markdown_blocks(text) | ||
| return self._merge_blocks(blocks, max_length) | ||
|
|
||
| def _parse_markdown_blocks(self, text: str) -> list[str]: | ||
| """Parse text into Markdown-aware blocks. | ||
|
|
||
| Keeps code blocks intact and splits the rest by structural elements. | ||
| """ | ||
| blocks: list[str] = [] | ||
| lines = text.split('\n') | ||
| current_block: list[str] = [] | ||
| in_code_block = False | ||
|
|
||
| for line in lines: | ||
| stripped = line.strip() | ||
|
|
||
| # Toggle fenced code block state | ||
| if stripped.startswith('```'): | ||
| if in_code_block: | ||
| # End of code block - close it as one block | ||
| current_block.append(line) | ||
| blocks.append('\n'.join(current_block)) | ||
| current_block = [] | ||
| in_code_block = False | ||
| continue | ||
| else: | ||
| # Start of code block - flush current block first | ||
| if current_block: | ||
| blocks.append('\n'.join(current_block)) | ||
| current_block = [] | ||
| current_block.append(line) | ||
| in_code_block = True | ||
| continue | ||
|
|
||
| if in_code_block: | ||
| current_block.append(line) | ||
| continue | ||
|
|
||
| # Heading (# ...) - start a new block | ||
| if re.match(r'^#{1,6}\s', stripped): | ||
| if current_block: | ||
| blocks.append('\n'.join(current_block)) | ||
| current_block = [] | ||
| current_block.append(line) | ||
| continue | ||
|
|
||
| # Horizontal rule (---, ***, ___) - start a new block | ||
| if re.match(r'^(-{3,}|\*{3,}|_{3,})\s*$', stripped): | ||
| if current_block: | ||
| blocks.append('\n'.join(current_block)) | ||
| current_block = [] | ||
| blocks.append(line) | ||
| continue | ||
|
|
||
| # Blank line - paragraph boundary | ||
| if stripped == '': | ||
| if current_block: | ||
| current_block.append(line) | ||
| blocks.append('\n'.join(current_block)) | ||
| current_block = [] | ||
| continue | ||
|
|
||
| current_block.append(line) | ||
|
|
||
| # Flush remaining (including unclosed code blocks) | ||
| if current_block: | ||
| blocks.append('\n'.join(current_block)) | ||
|
|
||
| return [b for b in blocks if b.strip()] | ||
|
|
||
| def _merge_blocks(self, blocks: list[str], max_length: int) -> list[str]: | ||
| """Merge small blocks greedily until approaching max_length. | ||
|
|
||
| If a single block exceeds max_length, split it by lines as fallback. | ||
| """ | ||
| segments: list[str] = [] | ||
| current = '' | ||
|
|
||
| for block in blocks: | ||
| candidate = (current + '\n\n' + block) if current else block | ||
|
|
||
| if len(candidate) <= max_length: | ||
| current = candidate | ||
| else: | ||
| # Flush current segment | ||
| if current: | ||
| segments.append(current) | ||
|
|
||
| # Check if this single block fits | ||
| if len(block) <= max_length: | ||
| current = block | ||
| else: | ||
| # Block too large - split it by lines | ||
| for part in self._split_large_block(block, max_length): | ||
| segments.append(part) | ||
| current = '' | ||
|
|
||
| if current: | ||
| segments.append(current) | ||
|
|
||
| return [s for s in segments if s.strip()] | ||
|
|
||
| def _split_large_block(self, block: str, max_length: int) -> list[str]: | ||
| """Split an oversized block by lines, preserving code block fences. | ||
|
|
||
| For single-line plain text (no newlines), falls back to splitting at | ||
| natural language boundaries (spaces, punctuation). | ||
| """ | ||
| lines = block.split('\n') | ||
|
|
||
| # Single long line with no newlines - use plain text splitting | ||
| if len(lines) == 1: | ||
| return self._split_plain_text(block, max_length) | ||
|
|
||
| is_code_block = lines[0].strip().startswith('```') | ||
|
|
||
| segments: list[str] = [] | ||
| current_lines: list[str] = [] | ||
| current_len = 0 | ||
|
|
||
| # For code blocks, track the opening fence to re-apply on continuations | ||
| code_fence = lines[0] if is_code_block else '' | ||
|
|
||
| for i, line in enumerate(lines): | ||
| line_len = len(line) + 1 # +1 for newline | ||
|
|
||
| # Single line exceeds limit on its own - split it first | ||
| if line_len > max_length: | ||
| if current_lines: | ||
| seg = '\n'.join(current_lines) | ||
| if is_code_block and not seg.rstrip().endswith('```'): | ||
| seg += '\n```' | ||
| segments.append(seg) | ||
| current_lines = [] | ||
| current_len = 0 | ||
|
|
||
| for part in self._split_plain_text(line, max_length): | ||
| segments.append(part) | ||
| continue | ||
|
|
||
| if current_len + line_len > max_length and current_lines: | ||
| segment = '\n'.join(current_lines) | ||
| # Close code block fence if splitting mid-code-block | ||
| if is_code_block and not segment.rstrip().endswith('```'): | ||
| segment += '\n```' | ||
| segments.append(segment) | ||
|
|
||
| current_lines = [] | ||
| current_len = 0 | ||
| # Re-open code block fence for continuation | ||
| if is_code_block and i < len(lines) - 1 and not line.strip().startswith('```'): | ||
| current_lines.append(code_fence) | ||
| current_len = len(code_fence) + 1 | ||
|
|
||
| current_lines.append(line) | ||
| current_len += line_len | ||
|
|
||
| if current_lines: | ||
| segments.append('\n'.join(current_lines)) | ||
|
|
||
| return segments | ||
|
|
||
| def _split_plain_text(self, text: str, max_length: int) -> list[str]: | ||
| """Split a long plain text string (no newlines) at word/space boundaries.""" | ||
| if len(text) <= max_length: | ||
| return [text] | ||
|
|
||
| segments: list[str] = [] | ||
| remaining = text | ||
|
|
||
| while remaining: | ||
| if len(remaining) <= max_length: | ||
| segments.append(remaining) | ||
| break | ||
|
|
||
| chunk = remaining[:max_length] | ||
| min_pos = int(max_length * 0.3) | ||
|
|
||
| # Try to find a space to split at | ||
| pos = chunk.rfind(' ') | ||
| if pos >= min_pos: | ||
| split_pos = pos | ||
| else: | ||
| # Hard cut as last resort | ||
| split_pos = max_length | ||
|
|
||
| segments.append(remaining[:split_pos].rstrip()) | ||
| remaining = remaining[split_pos:].lstrip() | ||
|
|
||
| return [s for s in segments if s] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里query.resp_message_chain是不清空的?如果一个query里回复了好几条消息,会导致它重复发好几遍吧?比如在用户一个消息发过来时,local-agent执行了好几轮,那么每一轮都会产生一个回复,每个回复又会被split成好几份;原来的逻辑是只发最后一个resp_message_chain,一个message_chain不会被分割,就没有问题,但现在加了分割,可能就会有问题吧?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
query里为什么会回复好几条消息呢,有一点没有理解到,是相当于在群里,同时好几个人进行询问吗
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
工具调用啊,agent循环,比如我让它去查询一个数据表,他就会回复:
这样一次用户消息就有好几条回复的