Text2Gremlin Data Generation and Model Fine-Tuning System (Vertical Scenarios and General Scenarios) - #52
Text2Gremlin Data Generation and Model Fine-Tuning System (Vertical Scenarios and General Scenarios)#52LRriver wants to merge 96 commits into
Conversation
…eneration parameters
…ing and call/with support
…y variants from Recipe
…cation and error handling
…and visitor classes
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
概览引入AST_Text2Gremlin模块——一套完整的图查询生成系统,包含ANTLR解析、模式管理、遍历生成和LLM翻译功能,用于从模板生成Gremlin查询语料库。 变更
序列图sequenceDiagram
participant User as CLI/用户
participant GenCorpus as generate_gremlin_corpus
participant Parser as GremlinTransVisitor
participant Generator as TraversalGenerator
participant Schema as Schema
participant GremlinBase as GremlinBase
participant Validator as check_gremlin_syntax
User->>GenCorpus: 提供模板、配置、模式、数据路径
GenCorpus->>GenCorpus: 加载Config、Schema、GremlinBase
loop 遍历每个模板
GenCorpus->>Parser: 解析模板字符串为Recipe
Parser-->>GenCorpus: 返回Traversal对象
GenCorpus->>Generator: 初始化,提供Schema/Recipe/GremlinBase
Generator->>Schema: 查询顶点/边标签、属性、实例数据
Schema-->>Generator: 返回模式信息和样本数据
Generator->>Generator: 递归生成查询步骤和描述
Note over Generator: 深度优先遍历,动态扩展选项
loop 生成的每个查询
GenCorpus->>Validator: 验证Gremlin语法
Validator-->>GenCorpus: 返回有效性状态
GenCorpus->>GenCorpus: 全局去重,积累结果
end
end
GenCorpus-->>User: 返回语料库、统计信息、输出文件路径
sequenceDiagram
participant User as 用户
participant Main as generalize_llm main()
participant Corpus as load_corpus
participant Config as get_llm_config
participant Batch as translate_batch
participant LLM as OpenAI API
participant Output as save_results
User->>Main: CLI: --config --input --output
Main->>Config: 验证并规范化LLM配置
Config-->>Main: 返回api_key、model等
Main->>Corpus: 加载最新/指定语料库JSON
Corpus-->>Main: 返回查询列表
Main->>Main: 将语料库分批处理
loop 并发批处理
Main->>Batch: 翻译批次(semaphore并发控制)
Batch->>LLM: 发送提示+查询,请求生成多个自然语言问题
LLM-->>Batch: 返回JSON响应
Batch->>Batch: 解析、验证、映射结果
Batch-->>Main: 返回翻译后的项目列表
end
Main->>Output: 保存带元数据的翻译结果
Output-->>Main: 成功
Main-->>User: 打印统计信息和输出路径
预估代码审查工作量🎯 5 (Critical) | ⏱️ ~120 minutes 可能相关的PR
建议的标签
诗
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@codecov-ai-reviewer review |
There was a problem hiding this comment.
Actionable comments posted: 28
♻️ Duplicate comments (1)
text2gremlin/AST_Text2Gremlin/base/test/test_csv_queries.py (1)
30-41: 代码重复:与 test_recipe_extraction_failures.py 中的 SyntaxErrorListener 相同此类与
test_recipe_extraction_failures.py(lines 24-35) 中的定义完全相同。如前面评论所述,建议提取到共享测试工具模块test_utils.py中。参见对
test_recipe_extraction_failures.pylines 24-35 的评论。
🧹 Nitpick comments (29)
text2gremlin/AST_Text2Gremlin/config.json (1)
5-5: 澄清空值的用途或移除无用配置。"common" 字段的值为空字符串。如果此字段未使用,建议移除以避免混淆;如果它有特定用途(如作为默认值或占位符),请在文档中说明其语义。
text2gremlin/AST_Text2Gremlin/base/test/test_generator.py (2)
24-28: 建议添加输出文件清理机制测试运行后会产生
test_generator_output.json文件,但没有清理逻辑。建议在测试完成后清理临时文件,或者使用临时目录。+import os +import tempfile + def test_generator(): """测试生成器的完整流程""" print("🧪 开始测试generator.py...") + # 使用临时文件 + temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) + output_file = temp_file.name + temp_file.close() + # 测试模板(模拟从CSV加载的数据) test_templates = [ ... ] print(f"使用 {len(test_templates)} 个测试模板...") try: result = generate_corpus_from_templates( test_templates, - output_file='test_generator_output.json' + output_file=output_file ) print(f"\n✅ 生成成功!") ... return True except Exception as e: print(f"❌ 生成失败: {str(e)}") return False + finally: + # 清理临时文件 + if os.path.exists(output_file): + os.remove(output_file)
39-41: 异常处理过于宽泛捕获所有异常(
except Exception)会掩盖潜在的编程错误。建议捕获更具体的异常类型,或至少记录完整的堆栈信息以便调试。+ import traceback + try: result = generate_corpus_from_templates( test_templates, output_file='test_generator_output.json' ) ... except Exception as e: - print(f"❌ 生成失败: {str(e)}") + print(f"❌ 生成失败: {str(e)}") + traceback.print_exc() # 打印完整堆栈跟踪 return Falsetext2gremlin/AST_Text2Gremlin/base/test/analyze_line73_explosion.py (2)
147-147: 硬编码的分析数量限制将分析限制为前 100 个查询是一个魔术数字,应该作为可配置参数。
- for query_gen, desc in corpus[:100]: # 只分析前100个 + # 在函数参数中添加 + def analyze_actual_generation(query: str, schema: Schema, config: Config, + gremlin_base: GremlinBase, sample_size: int = 100): + ... + for query_gen, desc in corpus[:sample_size]: # 只分析前N个
131-139: 缺少生成过程的错误处理
TraversalGenerator.generate()可能会失败或返回空结果,但没有错误处理。visitor = GremlinTransVisitor() recipe = visitor.parse_and_visit(query) + if not recipe: + print("❌ Recipe解析失败") + return + generator = TraversalGenerator(schema, recipe, gremlin_base) - corpus = generator.generate() + try: + corpus = generator.generate() + except Exception as e: + print(f"❌ 生成失败: {e}") + return + if not corpus: + print("⚠️ 未生成任何查询") + return + print(f"实际生成数量: {len(corpus):,}")text2gremlin/AST_Text2Gremlin/base/Config.py (2)
38-47: 考虑简化输出路径逻辑
get_output_path中的条件逻辑可以更清晰。当前实现在判断目录后拼接路径,可以提取为辅助方法提高可读性。可选重构建议:
def get_output_path(self): if self.gen_query: dir_or_file = self.config_data.get("output_query_dir_or_file") - if os.path.isdir(dir_or_file): - output_path = os.path.join(dir_or_file, self.db_id + ".txt") - return output_path - else: - return dir_or_file + return self._resolve_output_path(dir_or_file, self.db_id) else: return self.config_data.get("output_prompt_path") + +def _resolve_output_path(self, dir_or_file: str, db_id: str) -> str: + """解析输出路径:如果是目录则拼接 db_id,否则直接返回""" + if os.path.isdir(dir_or_file): + return os.path.join(dir_or_file, f"{db_id}.txt") + return dir_or_file
22-24: 建议指定文件编码打开文件时未指定编码,在某些环境下可能导致编码问题。
建议显式指定 UTF-8 编码(已在上面的异常处理建议中包含):
- with open(self.file_path, "r") as file: + with open(self.file_path, "r", encoding="utf-8") as file:text2gremlin/AST_Text2Gremlin/base/test/test_recipe_extraction_failures.py (2)
148-148: 硬编码的 CSV 文件路径降低了测试灵活性CSV 文件路径被硬编码在
main()函数中,不便于在不同环境或数据集上运行测试。建议通过命令行参数或环境变量使路径可配置:
def main(): """主函数""" print("=== Recipe Extraction 失败分析脚本 ===") print(f"开始时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}") - # 配置文件路径 - csv_file_path = "gremlin_query_errors_20250928_030206.csv" + # 从命令行参数或环境变量获取 CSV 文件路径 + import argparse + parser = argparse.ArgumentParser(description='分析 Recipe extraction 失败的查询') + parser.add_argument('--csv', default='gremlin_query_errors_20250928_030206.csv', + help='CSV 文件路径(默认:gremlin_query_errors_20250928_030206.csv)') + args = parser.parse_args() + csv_file_path = args.csv
110-140:analyze_query_structure函数的分析逻辑过于简化当前的查询结构分析仅使用字符串搜索来检测步骤类型(如
'g.call' in query),这可能产生误报(例如,字符串字面量或注释中包含这些模式)。可选改进:虽然对于快速诊断工具来说当前实现已足够,但如果需要更准确的分析,可以考虑使用正则表达式或实际的 AST 解析来检测步骤类型。例如:
import re def analyze_query_structure(query: str): """分析查询的结构,帮助理解为什么提取失败""" print(f"\n=== 查询结构分析 ===") print(f"查询长度: {len(query)}") # 使用正则表达式更精确地匹配 Gremlin 步骤 steps = [] if re.search(r'\bg\.call\s*\(', query): steps.append('call') if re.search(r'\bg\.V\s*\(', query): steps.append('V') if re.search(r'\bg\.E\s*\(', query): steps.append('E') if re.search(r'\bg\.inject\s*\(', query): steps.append('inject') print(f"检测到的起始步骤: {steps}") # ... 其余逻辑text2gremlin/AST_Text2Gremlin/base/test/test_csv_queries.py (2)
178-186: 多个硬编码路径降低了测试可移植性测试脚本中硬编码了多个路径(CSV 文件、config.json、schema 文件等),这使得测试难以在不同环境或项目结构下运行。
建议通过命令行参数或配置文件使路径可配置:
def main(): """主函数""" print("=== Gremlin查询测试脚本 ===") print(f"开始时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}") - # 配置文件路径 - csv_file_path = "cypher2gremlin_dataset_thread.csv" + # 解析命令行参数 + import argparse + parser = argparse.ArgumentParser(description='测试 CSV 中的 Gremlin 查询') + parser.add_argument('--csv', default='cypher2gremlin_dataset_thread.csv', + help='CSV 文件路径') + parser.add_argument('--config', help='配置文件路径(默认:自动检测)') + parser.add_argument('--schema', help='Schema 文件路径(默认:自动检测)') + args = parser.parse_args() + csv_file_path = args.csv + # 获取项目根目录路径 current_dir = os.path.dirname(os.path.abspath(__file__)) base_root = os.path.dirname(current_dir) project_root = os.path.dirname(base_root) - config_path = os.path.join(project_root, 'config.json') - schema_path = os.path.join(project_root, 'db_data', 'schema', 'movie_schema.json') - data_path = os.path.join(project_root, 'db_data') + config_path = args.config or os.path.join(project_root, 'config.json') + schema_path = args.schema or os.path.join(project_root, 'db_data', 'schema', 'movie_schema.json') + data_path = os.path.join(project_root, 'db_data')
43-77: 考虑复用 check_gremlin_syntax 函数
check_gremlin_syntax函数的实现很清晰且可复用。考虑将其也提取到共享的test_utils.py模块中,因为语法检查在多个测试场景中都很有用。可以将此函数与
SyntaxErrorListener一起移到test_utils.py中:# text2gremlin/AST_Text2Gremlin/base/test/test_utils.py def check_gremlin_syntax(query_string: str) -> tuple[bool, str]: """ 检查给定的Gremlin查询语句的语法。 Args: query_string: The Gremlin query to check. Returns: A tuple containing: - bool: True if syntax is correct, False otherwise. - str: An error message if syntax is incorrect, or "Syntax OK" if correct. """ # ... 当前实现然后在测试文件中导入使用。
text2gremlin/AST_Text2Gremlin/base/GremlinExpr.py (3)
11-13: 改进前向声明以避免循环导入当前第13行的字符串字面量
'Step'不是有效的前向声明。建议使用typing.TYPE_CHECKING和条件导入来正确处理循环依赖。应用此差异:
from typing import Any, List -# 由于 AnonymousTraversal 包含 Step 对象,而 Step 将在 GremlinParse 中定义, -# 而 GremlinParse 又导入了本文件,因此使用前向声明避免循环导入问题。 -'Step' +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from .GremlinParse import Step然后将第61行的类型注解更新为正常引用:
- self.steps: List['Step'] = [] + self.steps: List[Step] = []
15-49: 考虑合并 Predicate 和 TextPredicate 类以减少重复
Predicate和TextPredicate类的实现几乎完全相同,仅在__repr__中的前缀不同(P.vsTextP.)。建议使用单一类并通过参数控制前缀,或使用继承来减少代码重复。可选方案1:使用单一类和前缀参数
class Predicate: def __init__(self, operator: str, value: Any, prefix: str = "P"): self.operator = operator self.value = value self.prefix = prefix def __repr__(self) -> str: return f"{self.prefix}.{self.operator}({repr(self.value)})" # 使用时 text_pred = Predicate("startingWith", "mark", prefix="TextP")可选方案2:使用继承
class Predicate: prefix = "P" def __init__(self, operator: str, value: Any): self.operator = operator self.value = value def __repr__(self) -> str: return f"{self.prefix}.{self.operator}({repr(self.value)})" class TextPredicate(Predicate): prefix = "TextP"
67-69: 增强 AnonymousTraversal 的 repr 健壮性当
self.steps为空时,__repr__会返回"__."这可能不是期望的行为。建议添加空列表检查。def __repr__(self) -> str: + if not self.steps: + return "__" step_reprs = ".".join(map(repr, self.steps)) return f"__.{step_reprs}"text2gremlin/AST_Text2Gremlin/base/Schema.py (2)
61-86: 改进 CSV 解析的错误处理和日志记录当前
_parse_custom_csv方法捕获FileNotFoundError和IndexError,但可能还有其他异常(如编码错误、CSV格式错误)会导致静默失败。建议:
- 捕获更广泛的异常类型
- 使用 logging 模块而非 print
- 考虑在严重错误时抛出异常而非返回空 DataFrame
+import logging + def _parse_custom_csv(self, file_path: str, header_line_index: int) -> pd.DataFrame: """解析自定义多行表头的 CSV 文件。""" try: with open(file_path, 'r', encoding='utf-8') as f: lines = f.readlines() + if header_line_index > len(lines): + logging.warning(f"header_line_index {header_line_index} 超出文件行数: {file_path}") + return pd.DataFrame() + # 从第二行解析列名 header_line = lines[header_line_index - 1] column_defs = header_line.strip().split(',') column_names = [d.split(':')[0] for d in column_defs] # 从指定header行之后开始读取数据 data_lines = lines[header_line_index:] if not data_lines: return pd.DataFrame(columns=column_names) # 使用pandas从内存中的字符串列表读取数据 from io import StringIO csv_data = StringIO("".join(data_lines)) df = pd.read_csv(csv_data, header=None, names=column_names) return df - except (FileNotFoundError, IndexError) as e: - print(f"警告: 读取或解析文件失败: {file_path}, 错误: {e}") + except (FileNotFoundError, IndexError, UnicodeDecodeError, pd.errors.ParserError) as e: + logging.warning(f"读取或解析文件失败: {file_path}, 错误: {e}") return pd.DataFrame()
152-184: 控制随机种子以提高可测试性
get_instances方法使用random.randint和df.sample,但没有提供控制随机种子的机制。这使得测试和调试变得困难。建议添加可选的random_state参数。-def get_instances(self, label: str, count: int = None) -> List[Dict]: +def get_instances(self, label: str, count: int = None, random_state: int = None) -> List[Dict]: """获取多个实例 Args: label: 标签名 count: 要获取的实例数量,如果为None则随机选择2-5个 + random_state: 随机种子,用于可重现的结果 Returns: 实例列表 """ import random is_edge = label in self.edges data_cache = self.edge_data if is_edge else self.vertex_data load_func = self._load_edge_data if is_edge else self._load_vertex_data if label not in data_cache: load_func(label) df = data_cache.get(label) if df is None or df.empty: return [] # 如果没有指定数量,随机选择2-5个 if count is None: + if random_state is not None: + random.seed(random_state) count = random.randint(2, 5) # 如果实际数据量小于要求的数量,就全部取出 actual_count = min(count, len(df)) # 随机采样 - sampled_df = df.sample(actual_count) + sampled_df = df.sample(actual_count, random_state=random_state) return sampled_df.to_dict('records')text2gremlin/AST_Text2Gremlin/base/GremlinBase.py (3)
131-156: 增强 get_token_desc 的参数验证和错误处理当前方法的异常处理(第153行)捕获
IndexError和KeyError,但没有记录导致错误的具体参数信息,这使得调试变得困难。此外,应验证token_key的类型。+import logging + def get_token_desc(self, token_key: str, *args) -> str: """ 根据 token 和参数获取一个随机的、格式化后的中文描述。 """ + if not isinstance(token_key, str): + logging.warning(f"token_key 应为字符串类型,实际为: {type(token_key)}") + return "" + key = token_key.upper() if key in self.token_dict: index = self.token_dict[key] # 随机选择一个模板 selected_template = random.choice(self.template[index]) try: # 翻译参数中的schema术语 translated_args = [] for arg in args: if isinstance(arg, str): # 尝试翻译schema术语 translated_arg = self.get_schema_desc(arg) translated_args.append(translated_arg) else: translated_args.append(arg) # 使用翻译后的参数格式化模板 return selected_template.format(*translated_args) - except (IndexError, KeyError): + except (IndexError, KeyError) as e: + logging.debug(f"模板格式化失败 - token: {token_key}, 模板: {selected_template}, 参数: {args}, 错误: {e}") # 如果参数数量不匹配,返回原始模板 return selected_template return "" # 如果 token 不存在,返回空字符串
85-129: 考虑将翻译模板外部化以提高可维护性当前翻译模板硬编码在
_initialize_translation_templates方法中。随着支持的 Gremlin 步骤增加,这个方法会变得冗长。建议将模板移至外部 JSON 或 YAML 文件中。创建
template/translation_templates.json:{ "v": ["查询图中的所有顶点", "获取所有节点"], "e": ["查询图中的所有边", "获取所有关系"], "out": ["从当前位置出发,沿着 '{}' 方向的出边前进", "找到 '{}' 类型的邻居"] }然后修改方法:
def _initialize_translation_templates(self): """初始化 Gremlin 步骤的翻译模板。""" template_file = os.path.join( os.path.dirname(__file__), 'template', 'translation_templates.json' ) if os.path.exists(template_file): with open(template_file, 'r', encoding='utf-8') as f: templates_data = json.load(f) else: # 回退到硬编码模板 templates_data = { ... } for index, (key, value) in enumerate(templates_data.items()): self.token_dict[key.upper()] = index self.template.append(value)
55-56: 细化异常处理以避免隐藏配置错误第55行捕获所有异常并仅打印信息日志。如果配置方法存在但抛出了严重错误(如权限问题、配置损坏),这种处理方式可能会隐藏问题。
try: if hasattr(self.config, 'get_schema_dict_path'): schema_dict_paths = self.config.get_schema_dict_path() if isinstance(schema_dict_paths, list): file_paths.extend(schema_dict_paths) elif isinstance(schema_dict_paths, str): file_paths.append(schema_dict_paths) if hasattr(self.config, 'get_syn_dict_path'): syn_dict_path = self.config.get_syn_dict_path() if syn_dict_path: file_paths.append(syn_dict_path) - except Exception as e: - print(f"[INFO] Config paths not available: {e}") + except AttributeError as e: + print(f"[INFO] Config methods not available: {e}") + except Exception as e: + print(f"[WARNING] Unexpected error loading config paths: {e}") + raisetext2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (3)
25-53: 改进错误处理粒度
parse_and_visit方法捕获了所有异常并返回None,这使得调用方难以区分不同类型的错误(例如语法错误 vs 系统错误)。考虑细化错误处理:
def parse_and_visit(self, query_string: str): try: self.traversal = Traversal() input_stream = InputStream(query_string) lexer = GremlinLexer(input_stream) stream = CommonTokenStream(lexer) parser = GremlinParser(stream) - # 【修正】使用queryList作为入口规则,它包含一个或多个query tree = parser.queryList() - # Visit the parse tree - 访问第一个query result = self.visit(tree.query(0)) return result if result else self.traversal + except RecognitionException as e: + print(f"Syntax error parsing query '{query_string}': {e}") + return None + except AttributeError as e: + print(f"Query structure error '{query_string}': {e}") + return None except Exception as e: print(f"Error parsing query '{query_string}': {e}") return None
111-155: 考虑提取重复的参数处理逻辑多个
has方法变体(lines 111-155)使用了相似的参数提取模式。考虑提取公共辅助方法:
def _extract_has_params(self, ctx): """提取 has 方法的通用参数""" params = [] if hasattr(ctx, 'stringNullableLiteral') and ctx.stringNullableLiteral(): params.append(self.visit(ctx.stringNullableLiteral())) if hasattr(ctx, 'stringNullableArgument') and ctx.stringNullableArgument(): params.append(self.visit(ctx.stringNullableArgument())) if hasattr(ctx, 'genericArgument') and ctx.genericArgument(): params.append(self.visit(ctx.genericArgument())) if hasattr(ctx, 'traversalPredicate') and ctx.traversalPredicate(): params.append(self.visit(ctx.traversalPredicate())) if hasattr(ctx, 'nestedTraversal') and ctx.nestedTraversal(): params.append(self.visit(ctx.nestedTraversal())) if hasattr(ctx, 'traversalT') and ctx.traversalT(): params.append(self.visit(ctx.traversalT())) return params
971-1390: 建议将测试代码移至专用测试文件
__main__块中包含了 400+ 行的综合测试代码(lines 971-1390)。虽然测试覆盖很全面,但这些代码应该移至专门的测试文件中(如test_gremlin_trans_visitor.py),以保持主模块的简洁性。创建新文件
text2gremlin/AST_Text2Gremlin/base/test/test_gremlin_trans_visitor_comprehensive.py并将测试代码移至其中:# test/test_gremlin_trans_visitor_comprehensive.py import unittest from GremlinTransVisitor import GremlinTransVisitor, parse_gremlin_query from GremlinParse import Traversal from GremlinExpr import Predicate, TextPredicate, AnonymousTraversal class TestGremlinTransVisitorComprehensive(unittest.TestCase): def setUp(self): self.visitor = GremlinTransVisitor() def test_spawn_methods(self): # 将现有的测试类别转换为单元测试 ... if __name__ == '__main__': unittest.main()在主模块中保留简单的演示示例即可。
text2gremlin/AST_Text2Gremlin/base/generator.py (3)
165-168: 考虑将警告阈值配置化硬编码的警告阈值(5000 条生成查询,0 条新查询)可能需要根据不同场景调整。
将阈值移至配置文件或函数参数:
def generate_corpus_from_template( template_string: str, config: Config, schema: Schema, gremlin_base: GremlinBase, - global_corpus_dict: dict + global_corpus_dict: dict, + large_generation_threshold: int = 5000 ) -> tuple[int, dict]: ... - if stats['generated_count'] > 5000: + if stats['generated_count'] > large_generation_threshold: stats['warning'] = f'由于本条模版的Recip复杂,生成了大量查询({stats["generated_count"]}条)'
494-498: 增强 Gremlin 查询格式验证当前仅检查查询是否以
g.开头(line 495),这可能不足以捕获所有格式错误。考虑使用
check_gremlin_syntax进行预验证:# 基本语法检查 if not gremlin_query.startswith('g.'): stats['failed_loads'] += 1 stats['failed_queries'].append(f"第{row_num}行: 格式错误") continue + + # 可选:进行完整语法检查(可能影响加载性能) + # is_valid, error_msg = check_gremlin_syntax(gremlin_query) + # if not is_valid: + # stats['failed_loads'] += 1 + # stats['failed_queries'].append(f"第{row_num}行: {error_msg}") + # continue templates.append(gremlin_query)
518-518: 将 CSV 文件路径配置化CSV 文件路径硬编码为
"cypher2gremlin_dataset.csv"(line 518)。建议通过命令行参数或配置文件指定:
+import argparse if __name__ == '__main__': + parser = argparse.ArgumentParser(description='Generate Gremlin corpus from templates') + parser.add_argument('--csv', default='cypher2gremlin_dataset.csv', + help='Path to CSV file containing templates') + parser.add_argument('--output', default='generated_corpus.json', + help='Output file path') + args = parser.parse_args() + - csv_file_path = "cypher2gremlin_dataset.csv" + csv_file_path = args.csv print(f"🔄 从 {csv_file_path} 加载Gremlin查询模板...") templates, load_stats = load_templates_from_csv(csv_file_path) ... - result = generate_corpus_from_templates(templates) + result = generate_corpus_from_templates(templates, output_file=args.output)text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py (2)
43-77: 考虑将随机值生成参数配置化随机值生成使用了硬编码的参数:
- 字符串长度:5-8 个字符(lines 51, 71)
- 整数范围:1-10000(lines 53, 73)
- 值数量:2-5 个(line 59)
考虑通过配置文件或类参数使这些值可配置:
class TraversalGenerator: def __init__(self, schema: Schema, recipe: Traversal, gremlin_base: GremlinBase, string_length_range: Tuple[int, int] = (5, 8), int_value_range: Tuple[int, int] = (1, 10000), instance_count_range: Tuple[int, int] = (2, 5)): self.schema = schema self.recipe = recipe self.gremlin_base = gremlin_base self.generated_pairs: Set[Tuple[str, str]] = set() self.string_length_range = string_length_range self.int_value_range = int_value_range self.instance_count_range = instance_count_range
319-373: 增强概率和参数应配置化增强方法中硬编码了多个概率值和参数范围:
- limit 概率: 40%(line 324)
- range 概率: 20%(line 335)
- sample 概率: 30%(line 347)
- dedup 概率: 30%(line 357)
- order 概率: 20%(line 363)
- 各种数值范围(lines 327-341)
考虑创建配置类来管理这些参数:
@dataclass class EnhancementConfig: """增强配置参数""" limit_probability: float = 0.4 range_probability: float = 0.2 sample_probability: float = 0.3 dedup_probability: float = 0.3 order_probability: float = 0.2 limit_common_values: List[int] = field(default_factory=lambda: [1, 3, 5, 10, 20, 50, 100]) limit_max_random: int = 200 # ... 其他参数 class TraversalGenerator: def __init__(self, schema: Schema, recipe: Traversal, gremlin_base: GremlinBase, enhancement_config: EnhancementConfig = None): ... self.enhancement_config = enhancement_config or EnhancementConfig()text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinVisitor.py (2)
1-1: 自动生成文件中暴露了绝对路径。第 1 行的注释包含开发者的本地绝对路径
/root/lzj/ospp/Gremlin_Antlr4/Gremlin.g4。虽然这是 ANTLR 生成代码的标准输出,但建议在文档中说明生成步骤,以便其他贡献者可以重新生成此文件而不会混淆路径。
10-10: 考虑为访问器模式添加文档说明。
GremlinVisitor是 ANTLR 生成的基础访问器类,作为GremlinTransVisitor(在 PR 的其他文件中)的父类。对于不熟悉 ANTLR 访问器模式的维护者,建议在模块或 README 中添加简短说明:
- 访问器模式的作用(遍历语法树)
- 如何扩展此类(重写特定的
visitXxx方法)- 与
GremlinTransVisitor的关系- 示例:如何使用访问器解析 Gremlin 查询
需要我生成一个文档模板或使用示例吗?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (19)
text2gremlin/AST_Text2Gremlin/base/cypher2gremlin_dataset.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/base/gremlin/antlr-4.13.1-complete.jaris excluded by!**/*.jartext2gremlin/AST_Text2Gremlin/base/test/cypher2gremlin_dataset_thread.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/base/test/gremlin_query_errors_20250928_014705_pre.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/base/test/gremlin_query_errors_20250928_030206.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/base/test/recipe_extraction_detailed_failures_20250928_031211.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_acted_in.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_directed.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_has_genre.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_has_keyword.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_is_friend.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_produce.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_rate.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/edge_write.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/vertex_genre.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/vertex_keyword.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/vertex_movie.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/vertex_person.csvis excluded by!**/*.csvtext2gremlin/AST_Text2Gremlin/db_data/movie/raw_data/vertex_user.csvis excluded by!**/*.csv
📒 Files selected for processing (21)
text2gremlin/AST_Text2Gremlin/README.md(1 hunks)text2gremlin/AST_Text2Gremlin/base/Config.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/GremlinBase.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/GremlinExpr.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/GremlinParse.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/Schema.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/generator.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/gremlin/Gremlin.tokens(1 hunks)text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinLexer.tokens(1 hunks)text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinVisitor.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/template/schema_dict.txt(1 hunks)text2gremlin/AST_Text2Gremlin/base/template/syn_dict.txt(1 hunks)text2gremlin/AST_Text2Gremlin/base/test/analyze_line73_explosion.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/test/combination_explosion_report.md(1 hunks)text2gremlin/AST_Text2Gremlin/base/test/test_csv_queries.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/test/test_generator.py(1 hunks)text2gremlin/AST_Text2Gremlin/base/test/test_recipe_extraction_failures.py(1 hunks)text2gremlin/AST_Text2Gremlin/config.json(1 hunks)text2gremlin/AST_Text2Gremlin/db_data/schema/movie_schema.json(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: CR
PR: hugegraph/hugegraph-ai#0
File: hugegraph-llm/AGENTS.md:0-0
Timestamp: 2025-09-16T06:40:44.968Z
Learning: Applies to hugegraph-llm/src/hugegraph_llm/operators/gremlin_generate_task.py : Maintain the Text2Gremlin pipeline in src/hugegraph_llm/operators/gremlin_generate_task.py
🧬 Code graph analysis (8)
text2gremlin/AST_Text2Gremlin/base/GremlinParse.py (1)
text2gremlin/AST_Text2Gremlin/base/GremlinExpr.py (4)
Predicate(15-30)AnonymousTraversal(51-69)TextPredicate(33-48)Connector(72-88)
text2gremlin/AST_Text2Gremlin/base/test/test_recipe_extraction_failures.py (4)
text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (1)
GremlinTransVisitor(21-955)text2gremlin/AST_Text2Gremlin/base/GremlinParse.py (1)
Traversal(42-70)text2gremlin/AST_Text2Gremlin/base/test/test_csv_queries.py (2)
SyntaxErrorListener(30-41)main(172-254)text2gremlin/AST_Text2Gremlin/base/test/analyze_line73_explosion.py (1)
main(185-227)
text2gremlin/AST_Text2Gremlin/base/test/test_generator.py (1)
text2gremlin/AST_Text2Gremlin/base/generator.py (1)
generate_corpus_from_templates(179-324)
text2gremlin/AST_Text2Gremlin/base/generator.py (5)
text2gremlin/AST_Text2Gremlin/base/Schema.py (1)
Schema(17-184)text2gremlin/AST_Text2Gremlin/base/GremlinBase.py (1)
GremlinBase(12-187)text2gremlin/AST_Text2Gremlin/base/GremlinParse.py (1)
Traversal(42-70)text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py (1)
TraversalGenerator(19-417)text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (2)
GremlinTransVisitor(21-955)parse_and_visit(25-53)
text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (4)
text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinParser.py (2)
GremlinParser(1481-32734)queryList(2356-2400)text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinVisitor.py (1)
GremlinVisitor(10-2084)text2gremlin/AST_Text2Gremlin/base/GremlinParse.py (2)
Traversal(42-70)Step(13-40)text2gremlin/AST_Text2Gremlin/base/GremlinExpr.py (5)
Predicate(15-30)TextPredicate(33-48)AnonymousTraversal(51-69)Connector(72-88)Terminal(91-104)
text2gremlin/AST_Text2Gremlin/base/test/analyze_line73_explosion.py (4)
text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (2)
GremlinTransVisitor(21-955)parse_and_visit(25-53)text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py (1)
TraversalGenerator(19-417)text2gremlin/AST_Text2Gremlin/base/Schema.py (3)
Schema(17-184)get_vertex_labels(99-100)get_edge_labels(102-103)text2gremlin/AST_Text2Gremlin/base/GremlinBase.py (1)
GremlinBase(12-187)
text2gremlin/AST_Text2Gremlin/base/test/test_csv_queries.py (4)
text2gremlin/AST_Text2Gremlin/base/Schema.py (1)
Schema(17-184)text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py (1)
TraversalGenerator(19-417)text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (2)
GremlinTransVisitor(21-955)parse_and_visit(25-53)text2gremlin/AST_Text2Gremlin/base/test/test_recipe_extraction_failures.py (2)
SyntaxErrorListener(24-35)syntaxError(32-35)
text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py (4)
text2gremlin/AST_Text2Gremlin/base/Schema.py (9)
Schema(17-184)get_instance(147-150)get_instances(152-184)get_vertex_creation_info(131-135)get_properties_with_type(105-107)get_valid_steps(109-122)get_step_result_label(124-129)get_vertex_labels(99-100)get_updatable_properties(141-145)text2gremlin/AST_Text2Gremlin/base/GremlinParse.py (2)
Traversal(42-70)Step(13-40)text2gremlin/AST_Text2Gremlin/base/GremlinExpr.py (1)
Predicate(15-30)text2gremlin/AST_Text2Gremlin/base/GremlinBase.py (2)
GremlinBase(12-187)get_schema_desc(179-187)
🪛 markdownlint-cli2 (0.18.1)
text2gremlin/AST_Text2Gremlin/base/test/combination_explosion_report.md
49-49: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
68-68: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
text2gremlin/AST_Text2Gremlin/README.md
49-49: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
68-68: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: build (3.11)
- GitHub Check: build (3.10)
🔇 Additional comments (18)
text2gremlin/AST_Text2Gremlin/base/GremlinParse.py (1)
13-70: 数据结构设计合理。
Step和Traversal类的设计简洁清晰,文档完善。类型注解准确,__repr__方法实现良好,便于调试。这两个类为后续的 Gremlin 查询解析和生成奠定了良好的基础。text2gremlin/AST_Text2Gremlin/db_data/schema/movie_schema.json (2)
1-316: Schema 定义结构良好。整体 schema 定义清晰完整,涵盖了电影领域的主要实体和关系。类型定义准确,主键、可选字段和索引标记使用恰当。CSV 文件映射详细,包含了必要的 SRC_ID 和 DST_ID 信息。这为数据加载和查询生成提供了坚实的基础。
236-241: 未在 Schema.py 中发现对 “SKIP” 列的处理逻辑,请手动确认 CSV 加载器是否支持该跳过列的语法,或在文档中说明此自定义约定。text2gremlin/AST_Text2Gremlin/base/test/combination_explosion_report.md (1)
1-129: 文档内容质量很高这份分析报告非常详细和专业,清晰地解释了组合爆炸的原因、约束机制以及优化建议。文档结构良好,数学计算准确,对理解系统行为很有帮助。
text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinLexer.tokens (1)
1-533: 自动生成的词法分析器令牌文件这是一个由 ANTLR 自动生成的令牌定义文件,包含 270 个令牌定义,涵盖 Gremlin 查询语言的所有关键字、操作符和字面量。文件格式符合 ANTLR 标准,无需手动修改。
注意事项:
- 此文件应由 ANTLR 工具重新生成,而不是手动编辑
- 确保与对应的
.g4语法文件保持同步- 建议在
.gitattributes中标记为自动生成文件text2gremlin/AST_Text2Gremlin/base/test/test_recipe_extraction_failures.py (1)
63-108: 测试函数的错误处理逻辑清晰完善
test_recipe_extraction函数很好地处理了多种失败场景(语法错误、空 recipe、无步骤等),并提供了详细的错误信息。异常捕获和堆栈跟踪也有助于调试。text2gremlin/AST_Text2Gremlin/base/test/test_csv_queries.py (2)
79-127: 测试编排逻辑设计良好
test_single_query函数很好地编排了完整的测试流程:语法检查 → Recipe 提取 → 泛化生成 → 生成查询的语法验证。错误处理全面,返回值清晰,有助于调试。
244-252: 错误统计逻辑为调试提供了有价值的洞察按错误类型分组统计的逻辑有助于快速识别最常见的失败模式,对于改进生成系统非常有用。
text2gremlin/AST_Text2Gremlin/base/GremlinBase.py (1)
12-28: 代码逻辑正确,初始化流程清晰
__init__方法正确地按顺序初始化了配置、规则名、翻译模板和字典。加载顺序合理,确保了依赖关系的正确处理。text2gremlin/AST_Text2Gremlin/base/Schema.py (1)
68-70: 验证列定义解析的健壮性并修正检查脚本第70行
d.split(':')[0]假定每个定义都包含冒号,建议改为只分割第一个冒号并做防御性检查:column_names = [d.split(':', 1)[0] if ':' in d else d for d in column_defs]同时,原示例脚本因
-x sh -c不支持if !语法而失败,建议改为使用 bash 并以兼容写法验证 CSV:#!/usr/bin/env bash fd -e csv -x bash -c ' echo "检查文件: {}" head -n 2 {} | tail -n 1 | tr "," "\n" | while IFS= read -r col; do if echo "$col" | grep -q ":"; then : else echo " 警告: 列定义缺少冒号: $col" fi done '请使用上述脚本验证所有 CSV 文件,确保列定义格式一致。
text2gremlin/AST_Text2Gremlin/base/GremlinTransVisitor.py (3)
88-165: LGTM - Spawn 和导航方法实现正确Spawn 方法(V, E, addV, addE, inject, io, call)和导航方法(out, in, both, outE, inE, bothE, outV, inV, bothV)的实现逻辑正确,正确处理了各种参数变体。
280-302: LGTM - 嵌套遍历的状态管理实现正确
visitNestedTraversal方法正确地保存和恢复了遍历状态,避免了状态污染。这种临时遍历的方式是处理匿名遍历的标准做法。
8-19: 确认相对导入路径正确所有
gremlin.*和本地模块的文件均存在于text2gremlin/AST_Text2Gremlin/base目录下,可正常解析,无需调整。text2gremlin/AST_Text2Gremlin/base/generator.py (1)
26-73: LGTM - 语法检查实现正确
SyntaxErrorListener和check_gremlin_syntax的实现正确捕获了 ANTLR 解析错误,为后续的语法验证提供了可靠的基础。text2gremlin/AST_Text2Gremlin/base/TraversalGenerator.py (2)
19-41: LGTM - 生成器初始化和入口方法设计合理使用
Set[Tuple[str, str]]进行去重是正确的选择,generate方法作为公共 API 入口点设计清晰。
154-188: LGTM - 导航步骤的泛化逻辑实现优秀导航步骤(out/in/both)的处理逻辑正确实现了配方泛化:
- 优先使用配方中指定的边标签(lines 159-168)
- 同时生成其他可能的边变体(lines 170-178)
- 正确更新结果标签和类型
这种设计很好地平衡了忠实度和多样性。
text2gremlin/AST_Text2Gremlin/base/gremlin/Gremlin.tokens (1)
1-532: ANTLR生成的tokens文件 - 避免手动修改此文件定义了Gremlin词法分析器使用的完整token集合(267个关键字token + 标点符号token + 字面量token)。这类文件通常由ANTLR从语法文件(.g4)自动生成,包含token的数值映射和字符串表示。
注意事项:
- 如果此文件是自动生成的,应避免手动修改,所有更改应通过修改源语法文件并重新生成来完成
- 确保与GremlinLexer.tokens和语法定义保持同步
- Token数值的连续性和唯一性对解析器正确运行至关重要
text2gremlin/AST_Text2Gremlin/base/gremlin/GremlinVisitor.py (1)
2088-2088: 命名空间清理处理得当。使用
del GremlinParser清理模块命名空间是良好实践,确保此模块只公开GremlinVisitor类,符合 ANTLR 生成代码的标准模式。
| "db_schema_path": { | ||
| "common": "", | ||
| "yago": "./db_data/schema/yago.json", | ||
| "movie": "../db_data/schema/movie_schema.json", |
There was a problem hiding this comment.
🧩 Analysis chain
统一路径格式以提高一致性。
"movie" 的路径使用了 ../db_data/schema/movie_schema.json(向上一级目录),而其他路径使用 ./db_data/schema/(当前目录)。这种不一致可能表明目录结构存在问题或配置错误。请验证正确的相对路径,并确保所有 schema 路径使用统一的基准目录。
运行以下脚本验证所有 schema 文件是否存在:
🏁 Script executed:
#!/bin/bash
# 验证 config.json 中的 schema 路径
cd text2gremlin/AST_Text2Gremlin
for schema in "yago" "movie" "the_three_body" "finbench" "SNB" "three_kingdoms"; do
path=$(jq -r ".db_schema_path.${schema}" config.json)
if [ "$path" != "" ] && [ "$path" != "null" ]; then
if [ -f "$path" ]; then
echo "✓ $schema: $path"
else
echo "✗ $schema: $path (文件不存在)"
fi
fi
doneLength of output: 327
🏁 Script executed:
#!/bin/bash
# 定位项目中所有 db_data 目录和 schema 文件
find . -type d | sed -e 's|^\./||' | grep db_data || echo "No db_data directories found"
find . -type f | sed -e 's|^\./||' | grep '/db_data/' || echo "No db_data files found"Length of output: 1159
统一并修正 schema 路径及文件
- 将所有
db_schema_path统一改为相对于config.json的"./db_data/schema/<name>_schema.json",去除../前缀 - 为每个引用的 schema 添加或重命名对应的 JSON 文件(目前只有
movie_schema.json存在,其他均缺失)
🤖 Prompt for AI Agents
In text2gremlin/AST_Text2Gremlin/config.json around line 7, the schema path for
"movie" and other schema entries use a ../ prefix and some referenced schema
files are missing; update all db_schema_path entries to use paths relative to
config.json like "./db_data/schema/<name>_schema.json" (remove "../"), and
ensure every referenced schema has a corresponding JSON file in
text2gremlin/AST_Text2Gremlin/db_data/schema named <name>_schema.json (create or
rename files as needed so all references resolve).
| "finbench": "./db_data/schema/finbench_schema.json", | ||
| "SNB": "./db_data/schema/SNB_schema.json", | ||
| "three_kingdoms": "./db_data/schema/three_kingdoms.json", | ||
| "wandering_earth": "/root/work_repo/db_data/schema/wandering_earth.json" |
There was a problem hiding this comment.
避免使用绝对路径以提高可移植性。
"wandering_earth" 使用了绝对路径 /root/work_repo/db_data/schema/wandering_earth.json,这会导致配置在不同环境或用户间无法移植。建议改用相对路径,与其他 schema 路径保持一致。
应用此修改(假设文件结构一致):
- "wandering_earth": "/root/work_repo/db_data/schema/wandering_earth.json"
+ "wandering_earth": "./db_data/schema/wandering_earth.json"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "wandering_earth": "/root/work_repo/db_data/schema/wandering_earth.json" | |
| "wandering_earth": "./db_data/schema/wandering_earth.json" |
🤖 Prompt for AI Agents
In text2gremlin/AST_Text2Gremlin/config.json around line 12, the
"wandering_earth" entry uses an absolute path
(/root/work_repo/db_data/schema/wandering_earth.json) which breaks portability;
change it to a relative path consistent with other schema entries (e.g.,
../db_data/schema/wandering_earth.json or ./schema/wandering_earth.json
depending on repo layout), update any related docs or scripts if they assume the
old absolute path, and run a quick config-load test to ensure the relative path
resolves correctly in CI and local environments.
…with correctness guarantee and preliminary question generalization
There was a problem hiding this comment.
Review summary
本轮主要审阅 text2gremlin/AST_Text2Gremlin 的核心实现、LLM augment pipeline、配置、文档和测试;按约定先排除了 raw data / 数据集文件本身。整体方向是有价值的:AST recipe → schema 泛化 → LLM 多风格翻译/场景迁移的 pipeline 设计成立,CI 目前也是 green,README 和配置校验相比前几轮已有明显改善。
但当前仍不建议直接合并,主要原因是核心生成逻辑还有几个会直接影响语料质量的问题:
- 嵌套遍历的字符串拼接不统一,部分分支会生成缺少
__.前缀的 Gremlin; - predicate / string literal 的格式化入口不统一,并且存在调用未定义
_format_predicate()的路径; - 组合数量控制主要是后置裁剪,不能真正限制递归过程中的资源消耗;
- 递归过程中会把中间前缀查询写入最终语料,语义上可能偏离原模板;
- LLM 失败兜底结果可能在 merge 阶段被重新当作正常训练样本;
- 测试覆盖仍偏向历史小修回归,缺少 parser → recipe → generator → syntax checker 的端到端语义/语法质量验证。
建议评分
- 方案设计:6.5 / 10
- 逻辑执行:4.5 / 10
- 代码质量:6.0 / 10
- 测试有效性:5.0 / 10
- 可维护性与用户友好度:5.8 / 10
综合评分:5.8 / 10。建议先修复核心生成正确性和最小 e2e 测试后再合并。
优先修复顺序建议:
- 抽出统一的
format_gremlin_value()/format_predicate()/format_anonymous_traversal(),所有生成分支共用; - 明确 prefix query 是否作为训练样本输出,默认建议关闭;
- 把
max_total_combinations做成递归前置预算,而不是仅在最终返回前裁剪; - merge 阶段默认跳过带
_error的 LLM fallback 样本; - 增加 10–15 个小型 pytest,覆盖嵌套遍历、predicate、recipe fidelity、预算上限和失败样本过滤。
|
|
||
| class TraversalGenerator: | ||
| """Gremlin查询生成器 - 分层泛化架构""" | ||
|
|
There was a problem hiding this comment.
文件级别:TraversalGenerator 现在同时承担 step dispatch、Gremlin 字符串格式化、随机增强、schema/value 采样、组合预算和描述生成,职责已经偏重。建议不要大拆,但至少先抽出三个最小公共入口:format_gremlin_value()、format_predicate()、format_anonymous_traversal()。这样可以避免当前各分支各自拼字符串导致的语法不一致,也能让新增 step 时更容易写单测。
| category = self.controller.get_chain_category(len(self.recipe.steps)) | ||
| max_limit = self.controller.max_total.get(category) | ||
|
|
||
| if max_limit and len(results) > max_limit: |
There was a problem hiding this comment.
这里的 max_total 是后置裁剪:递归已经把所有组合放进 self.generated_pairs 后才排序截断。它能限制最终返回数量,但不能限制递归过程中的 CPU/内存消耗,也不符合“组合爆炸控制”的直觉。建议把预算前置到 _recursive_generate() 的扩展前,至少在 recipe 主路径完成后,对每个新增 option / enhancement 做 early-stop,而不是全量探索后再裁剪。
| next_desc = current_desc + option["desc_part"] | ||
|
|
||
| # 保存中间结果 | ||
| self.generated_pairs.add((next_query, next_desc)) |
There was a problem hiding this comment.
这里会把每个中间前缀查询都加入最终候选,例如完整模板是 4 步时,g.V()、g.V().hasLabel(...) 也会进入语料。前缀查询不一定语法错误,但它们已经不是原始 template 的完整泛化结果,容易稀释训练数据的语义一致性。建议将 prefix 输出做成显式配置(如 emit_prefix_queries),默认只输出 recipe 完整路径和明确的增强结果。
| for nested_str in nested_variants[:1]: # 只取第一个避免组合爆炸 | ||
| options.append( | ||
| { | ||
| "query_part": f".filter({nested_str})", |
There was a problem hiding this comment.
这里对匿名遍历的格式化不一致:_generate_nested_traversal_variants() 返回的是类似 out('x') 的片段,但 filter() 直接拼成 .filter(out('x')),而其他分支有的会拼 __.{nested_str}。这会导致部分嵌套遍历生成非法或风格不一致的 Gremlin。建议统一用 format_anonymous_traversal(nested_str),并给 filter/where/not/and/or/sideEffect/repeat/emit/union/match 增加语法回归测试。
| return result if result else [] | ||
| elif isinstance(step_params[0], Predicate): | ||
| # predicate变体 | ||
| pred_str = self._format_predicate(step_params[0]) |
There was a problem hiding this comment.
这里调用了 self._format_predicate(...),但当前类中没有对应定义。只要遇到 until(Predicate) 这条路径就会抛 AttributeError 并丢失该模板的生成结果。建议补齐统一的 predicate formatter,并覆盖 P.eq('x')、P.within('a', 'b')、TextP.containing('x')、not(P.eq(...)) 等用例。
|
|
||
| corpus = data.get("corpus", []) | ||
| pairs = [] | ||
| for item in corpus: |
There was a problem hiding this comment.
generalize_llm 在翻译失败时会产出带 _error 的 fallback 结果,并把 description 填进每种 style。这里 merge 时没有跳过 _error 项,会把失败兜底文本重新当成正常训练样本展开。建议默认 if "_error" in item: continue,或者提供显式 --include-failed-translations 开关,默认关闭。
| sample_count=migration_config["same_operation_sample_count"], | ||
| ) | ||
|
|
||
| response = await client.chat.completions.create( |
There was a problem hiding this comment.
generalize_llm.translate_one() 已经对 LLM 调用使用 asyncio.wait_for 和通用异常重试,但 migration 阶段这里的 transport/network 异常会直接进入通用 except 并 fallback,不会重试。建议复用同一套 async LLM 调用 helper,统一 timeout、指数退避和 transient error retry,避免迁移阶段因为一次临时网络抖动丢样本。
Addressed the actionable review items in the latest commits. Covered:
Verification:
Notes on two suggestions intentionally not changed:
|
imbajin
left a comment
There was a problem hiding this comment.
Review result: suggest Request changes before merge
Thanks for pushing this large Text2Gremlin data generation / augmentation proposal. The overall direction is valuable: template Gremlin → AST/recipe → schema/data-driven generalization → syntax check → LLM multi-style translation → scenario migration → DPO data. However, the current PR is still closer to a research prototype / large design dump than a maintainable, auditable change that is ready for main.
I suggest we pause merge first and discuss whether to split this PR into smaller staged PRs, so the overall quality can converge step by step instead of trying to approve a 100k+ LOC change at once.
Current high-level score from my review: 5.2 / 10.
| Area | Score | Summary |
|---|---|---|
| Design | 6.5 | Direction is good, but the PR mixes parser, generator, LLM pipeline, migration, DPO and data in one change. |
| Logic correctness | 4.0 | Several semantic-loss and quality-gate issues remain. |
| Code quality | 4.5 | Core classes are too large, with broad exception swallowing, random behavior and many lint suppressions. |
| Test effectiveness | 5.0 | There are useful regression tests, but not enough system-level quality gates. |
| Maintainability | 4.0 | Future extension/debugging will be hard without modularization and explicit unsupported handling. |
| User experience | 5.5 | README/CLI are helpful, but reproducibility and failure diagnosis need work. |
I also checked the existing PR comments. The visible previous feedback is mainly CodeRabbit’s walkthrough / paused review and docstring-coverage warning, so the points below focus on design and logic risks rather than repeating the docstring warning.
Blocking concerns
1. PR scope is too large to safely review or merge
This PR adds core AST parsing, traversal generation, schema/data loading, ANTLR grammar and generated parser files, sample movie data, LLM translation, scenario migration, DPO generation, CLI scripts, documentation and tests in one change. This makes review, rollback and ownership hard.
If the LLM pipeline is not ready, should it block the parser? If DPO data quality is questionable, should it block the generator? If generated parser files need to be regenerated, should it force re-reviewing the full pipeline? Currently all these risks are coupled.
Suggested split:
- PR 1: Gremlin grammar / generated parser / minimal visitor support.
- PR 2: Schema-driven traversal generator + deterministic syntax-check smoke tests.
- PR 3: CLI, config, README and example data.
- PR 4: LLM translation / scenario migration pipeline.
- PR 5: DPO preference-data generation and quality gates.
This would let us merge stable foundations first and keep quality converging incrementally.
2. Some “supported” Gremlin features silently lose semantics
The clearest example is mergeV / mergeE map parameters. The code explicitly says map parsing is TODO, but still records the step with empty params. This silently turns a meaningful query such as g.mergeV([(T.id): 1, label: 'person']) into a much weaker/no-parameter form.
Silent semantic loss is worse than returning unsupported, because it can poison generated training data while still passing syntax checks.
Expected fix: either fully support these map arguments, or explicitly mark them unsupported and exclude them from generated corpus with a structured error reason.
3. Current syntax check is not enough as a data-quality gate
check_gremlin_syntax() only verifies that the generated query can be parsed by the ANTLR grammar. It does not verify:
- schema compatibility;
- label/property existence;
- step type-flow correctness;
- HugeGraph executability;
- whether natural language matches the query;
- whether generated migrated queries actually fit the target domain.
The pipeline and docs currently risk implying stronger validity than what is actually checked.
Expected fix: separate these quality levels explicitly, for example:
syntax_valid: ANTLR parse passes;schema_valid: labels/properties/edge directions match schema;executable_valid: query runs against a test HugeGraph instance;nl_alignment_checked: NL/query alignment was checked by rule/model/manual sample.
Only claim what is actually enforced.
4. Template success statistics can be misleading
In generate_corpus_from_template(), a template can be marked successful even if generated queries are later all filtered out due to syntax errors. The warning path also says “all duplicate” when new_pairs_count == 0, even though syntax errors may be the real reason.
This is a data quality blocker because output metadata can over-report success.
Expected fix:
- mark a template failed or partially failed when valid output count is zero and syntax errors occurred;
- persist
syntax_error_count, duplicate count and failed examples in output metadata; - report invalid ratio and fail the pipeline above a threshold;
- do not collapse all zero-new-output cases into “duplicate”.
5. Schema/type-flow modeling is too shallow for “general” Text2Gremlin
The current schema logic mainly supports vertex-level out / in / properties / has. Type flow for outE, inE, bothE, outV, inV, otherV, edge properties and update/delete operations is incomplete. Once current_label is lost, many later choices become random, empty or overly broad.
This limits the system’s ability to generate semantically valid query variants, even if syntax checks pass.
Expected fix: define a minimal supported Gremlin subset with explicit input/output stream types and schema constraints per step. Unsupported or unknown state should be visible in stats instead of silently degrading.
6. Generation is not reproducible
The generator, schema sampling, translation style selection, migration pair selection and DPO task selection all use random behavior without a centralized seed. This makes output diffs, CI checks and training-data provenance hard to reproduce.
Expected fix: add a top-level seed config, inject RNG instead of using global random, use deterministic pandas.sample(random_state=...), and write the seed/config snapshot into output metadata.
7. DPO data validation is too weak
validate_gremlin_code() currently skips Groovy validation and returns true for unknown styles. For DPO data, this is especially risky because bad chosen/rejected pairs directly teach preference behavior.
Expected fix:
- unknown style should fail closed;
- extract Gremlin traversals from Groovy and validate them where possible;
- check basic Groovy structure;
- ensure chosen/rejected are semantically equivalent enough for preference training;
- track skipped/invalid DPO samples explicitly.
8. Tests are useful but not sufficient as merge gates
The added tests cover many local regressions, especially nested traversal formatting, bool/null formatting, metadata propagation and prompt behavior. However, they do not yet prove end-to-end data quality.
Missing merge gates:
- full
gremlin_templates.csvStage 1 smoke test under fixed seed; - golden output / stable snapshot test for a small template set;
- real HugeGraph execution test for a representative subset;
- unsupported-step tests that ensure semantic-loss cases fail visibly;
- corpus quality tests for invalid ratio, duplicate ratio, prefix/complete/enhancement distribution and CRUD distribution;
- LLM pipeline fixture tests covering JSON failure, schema mismatch, syntax invalid output, timeout and resume/incremental save behavior.
9. Maintainability risk: very large core files and broad lint exceptions
TraversalGenerator.py and GremlinTransVisitor.py are extremely large and contain many step-specific branches, repeated visitor variants, hasattr checks, print-based diagnostics and broad exception handling. pyproject.toml also grants broad ignores to text2gremlin/**/*.py.
This makes future step support hard to review and easy to regress.
Suggested direction: split into step-handler modules, for example:
handlers/
start.py # V, E, addV, addE
navigation.py # out, in, both, outE, inE, outV...
filter.py # has, hasLabel, hasId, where, filter...
map.py # values, properties, valueMap...
aggregate.py # count, group, order, by...
Each handler should declare supported signatures, input/output stream type, schema requirements and whether it supports generalization.
10. User-facing docs need clearer reproducibility and failure-diagnosis guidance
The README is helpful, but it should clearly separate:
- running Stage 1 locally without LLM credentials;
- running LLM stages with credentials;
- reproducing the published dataset;
- exact model/config/seed used for published artifacts;
- expected failure modes and how to inspect failed templates/samples;
- dependency/Python version policy.
Suggested path to convergence
I recommend we first align on a staged merge plan rather than trying to polish this huge PR in place.
A reasonable acceptance path could be:
- Agree on split boundaries. Keep core generator independent from LLM/DPO pipelines.
- Define supported Gremlin subset. Anything outside the subset should fail visibly, not be silently rewritten.
- Add deterministic Stage 1 quality gates. Fixed seed, full-template smoke, invalid-ratio threshold and metadata reporting.
- Merge the core only after it is reproducible and diagnosable.
- Then add LLM/migration/DPO pipelines with separate quality gates.
This keeps the design elegant and avoids over-engineering: merge a reliable small core first, then expand support incrementally with tests and metrics.
imbajin
left a comment
There was a problem hiding this comment.
Review 结论:建议先不要合并,先讨论拆分和分阶段合入
这个 PR 的方向是有价值的:把 Gremlin 模板解析成 AST/Recipe,再结合 schema 和数据做泛化,后面再接 LLM 多风格翻译、场景迁移和 DPO 数据生成。这个思路适合 Text2Gremlin 数据构建。
但目前这次改动太大,包含 parser、generator、schema/data、ANTLR 生成代码、LLM pipeline、迁移、DPO、示例数据、文档和测试,合在一个 PR 里很难稳定 review,也很难判断哪些部分已经可以安全合入。我的建议是:先暂停直接合并,大家先商量一下是否拆成几个小 PR 逐步合入。这样核心质量可以一层一层收敛,不会把研究原型、核心库代码和数据管线一次性绑在一起。
总体评分
综合评分:5.2 / 10
| 维度 | 评分 | 简单说明 |
|---|---|---|
| 方案设计 | 6.5 | 大方向成立,但边界太大,多个系统混在一个 PR 里。 |
| 逻辑正确性 | 4.0 | 还有语义丢失、质量门禁不足、统计误导等问题。 |
| 代码质量 | 4.5 | 核心文件过大,随机逻辑多,异常处理和 lint 豁免偏宽。 |
| 测试有效性 | 5.0 | 有不少局部回归测试,但缺少真正的端到端质量门禁。 |
| 可维护性 | 4.0 | 后续扩展和排查问题会比较困难。 |
| 用户友好度 | 5.5 | README/CLI 有雏形,但复现、失败诊断、配置说明还不够。 |
我也看了已有评论历史,主要是 CodeRabbit 的 walkthrough/paused review、docstring coverage、以及一些局部修复建议。下面尽量不重复这些已有细节,而是聚焦对合入风险影响最大的点。
全局问题
1. 建议先讨论是否拆 PR / 分阶段合入
这次 PR 一次性新增 10 万行以上内容,范围包括:
- Gremlin grammar 和 ANTLR 生成文件;
- Gremlin AST visitor;
- schema 驱动的 traversal generator;
- movie raw data 和 reference schemas;
- Stage 1 语料生成 CLI;
- LLM 多风格翻译;
- 场景迁移;
- merge dataset;
- DPO 偏好数据生成;
- README 和测试。
这些东西都相关,但成熟度不同。建议拆成下面几层:
- Parser/grammar 基础层:ANTLR grammar、generated parser、最小 visitor。
- 核心生成层:Schema + TraversalGenerator + syntax checker + 固定 seed 的小型 e2e 测试。
- CLI/文档/示例层:generate_corpus.py、config、README、样例数据。
- LLM 扩增层:translation、migration、merge dataset。
- DPO 层:preference data 生成和更严格的数据质量检查。
这样可以先把最核心、最可验证的部分合入,后面的 LLM/DPO pipeline 再逐步加强。
2. 当前“能 parse”不等于“可执行、符合 schema、适合训练”
目前很多地方把 ANTLR syntax check 当作主要质量门禁。但 syntax check 只能说明字符串能被语法解析,不能说明:
- label/property 一定存在;
- edge 方向一定合理;
- step 的输入/输出类型流一定正确;
- query 在 HugeGraph 上真的能执行;
- LLM 生成的自然语言和 Gremlin 严格对应;
- 场景迁移结果真的符合目标 schema。
建议把质量状态拆开:
syntax_valid:ANTLR 能 parse;schema_valid:label/property/edge direction 符合 schema;executable_valid:能在测试图上跑通;nl_alignment_checked:自然语言和 Gremlin 做过对齐检查。
目前只做到第一层,就不要在文档或输出中暗示更强的质量保证。
3. 需要明确支持范围,不要“看起来支持很多 step,但实际语义不完整”
当前代码对很多 Gremlin step 都有分支,但部分只是浅层支持,甚至会丢参数。建议先定义一个稳定支持子集,比如:
- 起点:
V/E/addV/addE; - 基础过滤:
has/hasLabel/hasId/where/is; - 基础导航:
out/in/both/outE/inE/outV/inV; - 属性和聚合:
values/properties/valueMap/count/group/order/by; - 常见 nested traversal。
不在支持范围里的 step 应该明确标记为 unsupported,并写入失败统计,而不是静默生成弱化后的 query。
4. 数据生成需要可复现
现在 generator、schema 采样、LLM 风格选择、migration pair 选择、DPO task selection 都用了随机逻辑,但没有统一 seed。这样很难复现实验,也很难做 CI golden test。
建议在 config 里加 seed,并把 seed、配置快照、输入文件 hash 写入输出 metadata。所有随机逻辑尽量通过注入的 RNG 控制,而不是直接使用全局 random。
5. 测试还需要从“局部回归”升级到“质量门禁”
当前测试覆盖了不少局部问题,比如 nested traversal 格式、bool/null 格式、metadata 传递、LLM retry 等,这些是有价值的。但还缺少以下合入门禁:
- 固定 seed 下跑完整
gremlin_templates.csv的 Stage 1 smoke test; - 小模板集的 golden output 测试;
- 抽样 query 在 HugeGraph 测试实例上真实执行;
- unsupported step 不允许静默降级;
- 输出语料的 invalid ratio、duplicate ratio、prefix/complete/enhancement 分布检查;
- LLM pipeline 的 fixture 测试:JSON 错误、空响应、syntax invalid、timeout、fallback、resume/incremental save。
建议的收敛路径
我建议先开一个讨论,确认是否按以下方式推进:
- 先拆分 PR 边界,核心 generator 不要和 LLM/DPO pipeline 绑在一起。
- 明确当前版本支持的 Gremlin 子集;不支持的语法要显式失败。
- 给 Stage 1 加固定 seed 和质量统计门禁。
- 先合入一个可复现、可诊断、支持范围明确的核心版本。
- 再分批接入 LLM migration 和 DPO,分别补质量门禁。
这样会比在一个超大 PR 里继续堆功能更稳,也更容易保持整体设计优雅。
| """ | ||
| params = [] | ||
| # Map参数暂时简化处理 - 见上方 TODO | ||
| self.traversal.add_step(Step("mergeE", params)) |
There was a problem hiding this comment.
这里是一个比较关键的语义风险:mergeE(map) 的 Map 参数当前被 TODO 掉,最后只记录成 Step("mergeE", [])。这会把原本带有条件/属性/ID 的 merge 语义直接丢掉,但后续流程可能仍把它当作“已支持”的查询继续生成训练数据。
建议不要静默降级:要么完整解析 Map 参数,要么在 visitor 阶段标记为 unsupported,并把该模板计入失败统计,避免污染语料。
| stats["new_pairs_count"] = new_pairs_count | ||
| stats["duplicate_count"] = duplicate_count | ||
| stats["syntax_error_count"] = syntax_error_count | ||
| stats["success"] = True |
There was a problem hiding this comment.
这里会把模板标记为成功,但前面生成出的 query 可能全部被语法检查过滤掉。也就是说 generated_count > 0 但 new_pairs_count == 0 且 syntax_error_count > 0 时,统计仍可能显示模板成功。
建议把“generator 有产出”和“最终有效语料有产出”分开统计。如果有效 query 为 0,尤其是因为 syntax error 为 0,应标记为 failed 或 partial_failed,并把失败样例写入 metadata。
|
|
||
| def get_valid_steps(self, current_label: str, element_type: str = "vertex") -> list[dict]: | ||
| if element_type == "vertex": | ||
| if current_label not in self.vertices: |
There was a problem hiding this comment.
get_valid_steps() 目前基本只覆盖 vertex 上的 out/in/properties/has。这对 demo 可用,但支撑“通用 Text2Gremlin 泛化”还不够,因为 outE/inE/bothE/outV/inV/otherV、edge property、update/delete 的类型流和 schema 约束都没有系统建模。
建议先明确当前支持的 schema/type-flow 子集,并在 unsupported 场景下显式返回结构化错误,而不是让后续生成逻辑退化成空结果或随机选择。
| if count is None: | ||
| count = random.randint(DEFAULT_SAMPLE_MIN, DEFAULT_SAMPLE_MAX) | ||
|
|
||
| # 如果实际数据量小于要求的数量,就全部取出 |
There was a problem hiding this comment.
这里使用全局 random.randint(),后面 df.sample() 也没有传 random_state。这会导致同一份 config/schema/templates 每次生成结果都不同,不利于复现实验、排查问题和写 golden test。
建议从顶层配置传入 seed/RNG,并让 get_instances() 支持确定性采样;同时把 seed 写入输出 metadata。
| # if not ok: | ||
| # return False, f"Gremlin syntax error in: {g[:50]}... - {msg}" | ||
| # return True, "All Gremlin statements OK" | ||
| elif style == "groovy": |
There was a problem hiding this comment.
DPO 数据对质量要求更高,但这里对 style == "groovy" 直接跳过校验,未知 style 也会返回通过。这样坏的 chosen/rejected 代码可能进入偏好数据,影响模型学到错误偏好。
建议至少:未知 style fail closed;对 Groovy 提取其中的 g.V()/g.E()/g.addV()/g.addE() 片段做基础语法检查;同时记录 skipped/invalid DPO 样本比例。
| "generation_timestamp": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), | ||
| }, | ||
| "pairs": pairs, | ||
| } |
There was a problem hiding this comment.
这里只做了 ANTLR 语法检查,无法确认生成 query 是否真的符合目标 schema。比如 label/property 可能不存在,边方向可能不合理,但仍会被当作 valid sample。
建议把 migration 样本至少拆出 syntax_valid 和 schema_valid 两层;如果暂时只做 syntax check,输出 metadata 和 README 中也需要明确说明,避免用户误以为这些 query 都可执行、且完全符合目标场景。
| "hugegraph-python-client/src/pyhugegraph/structure/*.py" = ["N802"] | ||
| "text2gremlin/**/*.py" = [ | ||
| "E501", # Long Chinese prompts and generated Gremlin visitor fragments are kept readable. | ||
| "E741", # Gremlin examples use compact variable names in local comprehensions. |
There was a problem hiding this comment.
这里给 text2gremlin/**/*.py 放开了比较多 lint 规则。考虑到这次新增的核心 generator/visitor 本身已经很大,继续大范围豁免会让后续维护更难。
建议只对 ANTLR generated files 或确实需要的脚本做更小范围的 ignore;核心库代码尽量保留复杂度、重复定义、print 等检查,至少逐步收紧。
LLM-based Gremlin QA Synthesis and Generalization in Vertical Scenarios.
🏗️ Project Structure
./graph2gremlin.py: Initially generates Gremlin data based on templates and graph data, ensuring correctness through templates, and translates and preliminarily generalizes the Gremlin data and questions../gremlin_checker.py: Performs syntax checking using Antlr4../llm_handler.py: An LLM interaction model that inputs QA data for each batch of seed numbers (during seed data generation, queries undergo a small batch generalization), allowing the LLM to understand how to write text2gremlin, first generalizing Gremlin, then translating and generalizing the query../qa_generalize.py: Callsgremlin_checkerandllm_handlerfor seed data generalization../instruct_convert.py: Handles instruction format conversion and the division of training and test sets../db_data: Contains schema and graph data../data/seed_data: Seed data (to be uploaded)../data/vertical_training_sets: Vertical scenario generalization data (to be uploaded).Gremlin Corpus Generation System Based on Recursive Backtracking in General Scenarios.
📋 Project Overview
This PR adds a complete Text-to-Gremlin corpus generation system based on a recursive backtracking recipe-guided generation approach, capable of automatically generating large-scale and diverse training data from Gremlin query templates.
🏗️ Project Structure
🎯 Core Features
Recipe-Guided Generation
Large-Scale Data Processing
Complete Error Handling
Intelligent Constraint Mechanism
📊 System Capabilities
🧪 Technical Features
📈 Application Value
🔧 Usage
📋 Documentation
Summary by CodeRabbit
发布说明
新功能
文档