基于AI Agent的论文自动化复现工具 #6
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ymx10086, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
此拉取请求引入了一个基于AI Agent的自动化工具，旨在简化和加速研究论文的复现过程。该工具通过将复杂的复现任务分解为可管理的阶段，并利用大型语言模型（LLMs）进行智能规划、设计和代码生成，从而实现自动化。它能够处理PDF或LaTeX格式的论文输入，并输出可运行的代码，甚至包含评估模块以验证生成代码的质量。此次更新标志着项目在实现论文自动化复现方面取得了重要进展，并为未来的扩展奠定了基础。
Highlights
- AI Agent驱动的论文复现工作流: 此拉取请求引入了一个全面的AI Agent工具，旨在自动化研究论文的复现过程。它涵盖了从论文内容解析到代码生成和评估的端到端工作流。
- 灵活的LLM集成（OpenAI与本地LLM）: 该工具支持使用OpenAI的API进行AI功能调用，同时也提供了使用本地大型语言模型（如DeepSeek、Qwen）的替代实现，为用户提供了灵活性和成本控制选项。
- 模块化和阶段化的复现流程: 复现过程被分解为多个清晰的阶段：PDF/LaTeX内容处理、多阶段规划（总体、架构、逻辑设计）、代码生成和最终评估，每个阶段都有专门的Python脚本负责。
- 新功能和项目状态更新: 新增了多个Python脚本和示例数据，显著扩展了项目功能，并更新了README文件以反映项目的新方向和已达成的里程碑。
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
本次PR引入了一个基于AI Agent的论文自动化复现工具，这是一个非常 impressive 的项目，包含了从规划、分析到编码和评估的完整流程。代码结构清晰，并且同时支持OpenAI API和本地LLM。我的审查主要关注以下几个方面：
- 安全性: 多个脚本中硬编码了API密钥，这是严重的安全风险，必须立即移除。
- 正确性:
utils.py中的一个正则表达式替换存在严重bug，可能导致数据损坏。此外，不同脚本之间对数据的处理方式不一致，可能引发运行时错误。 - 可维护性: 部分代码存在重复逻辑和“魔法数字”，可以通过重构来提高代码的可读性和可维护性。
我已经在代码中提出了具体的修改建议，请在合并前解决这些critical和high级别的问题。
| client = OpenAI( | ||
| # defaults to os.environ.get("OPENAI_API_KEY") | ||
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", | ||
| base_url="https://api.chatanywhere.tech/v1" |
There was a problem hiding this comment.
代码中硬编码了 OpenAI API 密钥。这是一个严重的安全漏洞，会将密钥暴露给所有能看到代码的人。请立即移除此密钥，并通过环境变量或安全的配置文件加载。
| client = OpenAI( | |
| # defaults to os.environ.get("OPENAI_API_KEY") | |
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", | |
| base_url="https://api.chatanywhere.tech/v1" | |
| client = OpenAI( | |
| # defaults to os.environ.get("OPENAI_API_KEY") | |
| api_key=os.environ.get("OPENAI_API_KEY"), | |
| base_url=os.environ.get("OPENAI_API_BASE", "https://api.chatanywhere.tech/v1") | |
| ) |
| clean_data = re.sub(r'\n\s*', '', clean_data) | ||
| clean_data = re.sub(r'"""', '"', clean_data) # Replace triple double quotes | ||
| clean_data = re.sub(r"'''", "'", clean_data) # Replace triple single quotes | ||
| clean_data = re.sub(r"\\", "'", clean_data) # Replace \ |
There was a problem hiding this comment.
在 content_to_json3 函数中，re.sub(r"\\", "'", clean_data) 会将所有反斜杠 \ 替换为单引号 '。这是一个严重错误，它会破坏任何包含转义字符（如 \n, \t）或 Windows 路径的 JSON 字符串，导致解析失败或数据损坏。请移除或修正这行代码。
| client = OpenAI( | ||
| # defaults to os.environ.get("OPENAI_API_KEY") | ||
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", | ||
| base_url="https://api.chatanywhere.tech/v1" |
There was a problem hiding this comment.
代码中硬编码了 OpenAI API 密钥。这是一个严重的安全漏洞。请立即移除，并从环境变量中读取。
| client = OpenAI( | |
| # defaults to os.environ.get("OPENAI_API_KEY") | |
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", | |
| base_url="https://api.chatanywhere.tech/v1" | |
| client = OpenAI( | |
| # defaults to os.environ.get("OPENAI_API_KEY") | |
| api_key=os.environ.get("OPENAI_API_KEY"), | |
| base_url=os.environ.get("OPENAI_API_BASE", "https://api.chatanywhere.tech/v1") | |
| ) |
| client = OpenAI( | ||
| # defaults to os.environ.get("OPENAI_API_KEY") | ||
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", | ||
| base_url="https://api.chatanywhere.tech/v1" |
There was a problem hiding this comment.
代码中硬编码了 OpenAI API 密钥。这是一个严重的安全漏洞。请立即移除，并从环境变量中读取。
| client = OpenAI( | |
| # defaults to os.environ.get("OPENAI_API_KEY") | |
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", | |
| base_url="https://api.chatanywhere.tech/v1" | |
| client = OpenAI( | |
| # defaults to os.environ.get("OPENAI_API_KEY") | |
| api_key=os.environ.get("OPENAI_API_KEY"), | |
| base_url=os.environ.get("OPENAI_API_BASE", "https://api.chatanywhere.tech/v1") | |
| ) |
| code = extract_code_from_content(message.content) | ||
| if len(code) == 0: | ||
| code = message.content |
There was a problem hiding this comment.
当 extract_code_from_content 失败时，代码直接将整个 LLM 的响应 message.content 作为代码。这非常危险，因为 LLM 的响应可能包含非代码的对话内容，这会导致生成无效的 Python 文件。应该添加更健壮的错误处理或后备逻辑，而不是直接使用原始响应。
| code = extract_code_from_content(message.content) | |
| if len(code) == 0: | |
| code = message.content | |
| code = extract_code_from_content(message.content) | |
| if not code: | |
| print(f"[WARNING] Could not extract code from response for {todo_file_name}. The file will contain the raw response.") | |
| code = message.content |
| else: | ||
| task_list = content_to_json(context_lst[2]) | ||
|
|
||
| todo_file_lst = task_list['Task list'] |
There was a problem hiding this comment.
此处直接使用 task_list['Task list'] 来获取任务列表，但这与其他脚本（如 2_analyzing.py）中的键处理方式不一致。2_analyzing.py 会尝试多种键的变体（'Task list', 'task_list', 'task list'）。这种不一致性可能会导致此脚本在处理由旧版规划脚本生成的 task_list.json 时出现 KeyError。建议统一键的处理逻辑。
|
|
||
| output_dir = args.output_dir | ||
|
|
||
| with open(f'{output_dir}/planning_trajectories.json', encoding='utf8') as f: |
There was a problem hiding this comment.
使用 f-string 拼接文件路径 (f'{output_dir}/planning_trajectories.json') 可能会在不同操作系统上出现问题。建议使用 os.path.join() 来构建路径，以确保跨平台兼容性。
with open(os.path.join(output_dir, 'planning_trajectories.json'), encoding='utf8') as f:| match = re.search(r"```yaml\n(.*?)\n```", yaml_raw_content, re.DOTALL) | ||
| if match: | ||
| yaml_content = match.group(1) | ||
| with open(f'{output_dir}/planning_config.yaml', 'w', encoding='utf8') as f: | ||
| f.write(yaml_content) | ||
| else: | ||
| # print("No YAML content found.") | ||
| match2 = re.search(r"```yaml\\n(.*?)\\n```", yaml_raw_content, re.DOTALL) | ||
| if match2: | ||
| yaml_content = match2.group(1) | ||
| with open(f'{output_dir}/planning_config.yaml', 'w', encoding='utf8') as f: | ||
| f.write(yaml_content) | ||
| else: | ||
| print("No YAML content found.") |
There was a problem hiding this comment.
这段代码中存在重复逻辑。提取和写入 YAML 内容的部分在 if 和 else 分支中几乎完全相同。可以将其重构以避免重复，提高代码的可维护性。
match = re.search(r"```yaml\n(.*?)\n```", yaml_raw_content, re.DOTALL) or \
re.search(r"```yaml\\n(.*?)\\n```", yaml_raw_content, re.DOTALL)
if match:
yaml_content = match.group(1).replace('\\n', '\n')
with open(f'{output_dir}/planning_config.yaml', 'w', encoding='utf8') as f:
f.write(yaml_content)
else:
print("No YAML content found.")| # Paper2Code_Agent | ||
|
|
||
| # 🎁News | ||
| - 🎁2025.04.15 [lite_research](https://github.com/modelscope/mcp-central/blob/main/examples/lite_research/README.md) is supported. | ||
| - 🎁2025.04.02 [crawl4ai](https://github.com/unclecode/crawl4ai) is supported [here](./mcp_central/crawl4ai) No newline at end of file | ||
| 目前阶段：复现完成 |
There was a problem hiding this comment.
README 文件内容过于简单。建议扩充内容，使其包含项目简介、功能、安装指南、使用方法（包括如何运行各个阶段的脚本）以及一个完整的示例。这将极大地帮助其他开发者理解和使用这个项目。
| for key in ["cite_spans", "ref_spans", "eq_spans", "authors", "bib_entries", \ | ||
| "year", "venue", "identifiers", "_pdf_hash", "header"]: |
There was a problem hiding this comment.
为了符合 PEP 8 风格指南，建议使用括号来换行长列表，而不是使用反斜杠 \。这样代码更易读。
| for key in ["cite_spans", "ref_spans", "eq_spans", "authors", "bib_entries", \ | |
| "year", "venue", "identifiers", "_pdf_hash", "header"]: | |
| for key in ( | |
| "cite_spans", "ref_spans", "eq_spans", "authors", "bib_entries", | |
| "year", "venue", "identifiers", "_pdf_hash", "header" | |
| ): | |
| @@ -1,6 +1,3 @@ | |||
| # mcp-central | |||
| @@ -0,0 +1,46 @@ | |||
| import json | |||
| import argparse | |||
|
|
||
| client = OpenAI( | ||
| # defaults to os.environ.get("OPENAI_API_KEY") | ||
| api_key="sk-YRTZ9mzbWO2NgMfUy1nwBG5RTAyuKvwbc550IwRctDfOvVqk", |
中期考核
目前阶段：复现成功