Appeng 3801-A agent performance fixes - tool use stability #132
Conversation
etsien
commented
Oct 19, 2025
etsien
commented
- renamed tools - emphasis on conciseness and distinctness for model to correctly pick the right tool
- reworked tool descriptions - focused on conciseness (shortened by ~60%), removing conflicting text, to help model send in the right parameters
- refactored scripts and config files to use constants instead of strings for referencing tools
- added unit tests to check tool names, uniqueness, and proper referencing of tools in scripts
|
/ok-to-test |
|
/retest |
|
/retest vulnerability-analysis-on-pr |
zvigrinberg
left a comment
zvigrinberg
left a comment
There was a problem hiding this comment.
@etsien
It looks very good and very promising.
Let's just wait for the CM results to be published so will verify the improvements before approving it and merging.
| description=( | ||
| "Checks if a function from a package is reachable from application code through the call chain. " | ||
| "Input format: 'package_name,function_name' (comma-separated). " | ||
| "Example: 'urllib,parse'. " | ||
| "Returns: (is_reachable: bool, call_hierarchy_path: list)." | ||
| ) | ||
| ) |
There was a problem hiding this comment.
@etsien I Asked Shimon to extend the description of tool to support more cases, as it's not always the case that it's just a function inside a package ( for example, sometimes in python there is a method of class ( prefixed or not by containing module, depends on the import statement) that should be checked if it's being called from source code or not, and then the format could be sometimes , ()... he should do it as an addition to his transitive code search support work for c programming language.
There was a problem hiding this comment.
Please coordinate with him as he is currently doing some changes to the prompts and tools descriptions, and i believe he contacted you to consult with you, please make sure that the his changes are following tools calling and prompts best practices, and that they're correlated with your changes ( maybe after potential tailoring them accordingly)
|
@etsien I Found a bug that eliminate all tools but diff --git a/src/vuln_analysis/functions/cve_generate_vdbs.py b/src/vuln_analysis/functions/cve_generate_vdbs.py
index 7772e88..2a9dc31 100644
--- a/src/vuln_analysis/functions/cve_generate_vdbs.py
+++ b/src/vuln_analysis/functions/cve_generate_vdbs.py
@@ -27,6 +27,7 @@ from aiq.data_models.function import FunctionBaseConfig
from pydantic import Field
from vuln_analysis.logging.loggers_factory import LoggingFactory, trace_id
+from vuln_analysis.tools.tool_names import ToolNames
logger = LoggingFactory.get_agent_logger(__name__)
@@ -69,11 +70,11 @@ async def generate_vdb(config: CVEGenerateVDBsToolConfig, builder: Builder):
assert isinstance(agent_config, CVEAgentExecutorToolConfig)
# Update config based on tools available in agent config
- if "Container Image Code QA System" not in agent_config.tool_names:
+ if ToolNames.CODE_SEMANTIC_SEARCH not in agent_config.tool_names:
logger.info("Container Image Code QA System tool is not enabled, setting ignore_code_embedding to True")
config.ignore_code_embedding = True
- if "Lexical Search Container Image Code QA System" not in agent_config.tool_names:
+ if ToolNames.CODE_KEYWORD_SEARCH not in agent_config.tool_names:
logger.info(
"Lexical Search Container Image Code QA System tool is not enabled, setting ignore_code_index to True")
config.ignore_code_index = TruePlease also apply it to the other PR #134 that was branched out from this head branch |
adding in constants during the vdb generation check
Bugfix pushed to both PR branches |
|
Closing this PR in favor of APPENG-3801-B instead, which has these changes and other agent/tool changes. |
2 participants
