APPENG-3801-B - Agent performance fixes - all agent stages

[etsien]

Comprehensive Prompting Improvements Across Vulnerability Analysis Pipeline

This PR implements systematic improvements to all prompting stages of the vulnerability analysis pipeline, focusing on consistency, clarity, reduced verbosity, and reduced LLM hallucinations. Also incorporates tool improvements from prior PR (APPENG-3801-A).

Edit:
Also rebased to incorporate the latest PRs that were merged upstream to branch rh-aiq-main.

Summary of Changes

All 7 stages of the vulnerability analysis pipeline have been improved with:

• Consistent XML-style section markers (<TASK>, <INSTRUCTIONS>, <EXAMPLES>, etc.)
• Reduced verbosity while maintaining technical precision
• Dynamic tool awareness to prevent instructing agents to use disabled tools
• Separated LLM responsibilities from deterministic code operations
• Improved example quality and diversity

Files Modified

Core Prompting and Utilities

1. src/vuln_analysis/utils/prompting.py - Major restructuring

   • Added build_tool_descriptions() consolidated base function
   • Updated 6 prompt constants with structured sections
   • Simplified get_agent_prompt() and get_cvss_prompt() functions

2. src/vuln_analysis/utils/intel_source_score.py

   • Separated LLM scoring from arithmetic calculation
   • LLM now provides only individual criterion scores
   • Code calculates total with validation

3. src/vuln_analysis/utils/checklist_prompt_generator.py

   • Added tool_names parameter to generate_checklist()
   • Dynamic tool descriptions formatted for Jinja2 rendering
   • Structured requirements section

4. src/vuln_analysis/utils/justification_parser.py

   • Restructured JUSTIFICATION_PROMPT with clear sections
   • Added exploitation conditions definition
   • Explicit logical precedence order for 12 categories

Function Implementations

5. src/vuln_analysis/functions/cve_agent.py

   • Updated _create_agent() to use build_tool_descriptions()
   • Strategic tool guidance formatted locally
   • Uses partial_variables for tool_selection_strategy

6. src/vuln_analysis/functions/cve_checklist.py

   • Added agent_name field to CVEChecklistToolConfig
   • Retrieves agent tool configuration
   • Passes agent tool names to checklist generation

7. src/vuln_analysis/functions/cve_generate_cvss.py

   • Simplified _create_agent() function
   • Removed conditional example insertion

Tests

8. tests/test_base_tool_descriptions.py - New test suite
   • Tests consolidated build_tool_descriptions() function
   • Validates tool description generation
   • Verifies MOD_FEW_SHOT structure

[vbelouso]

/ok-to-test

[etsien]

bugfix pushed for the vdb tool issue

[zvigrinberg]

Hi @etsien
The agent is not being started with these changes.
Please see my comment for a quick fix and will continue from there...
anyway we fixed it manually and continued with the confusion matrix second batch run...

Thanks.

[zvigrinberg]

/retest

[zvigrinberg]

/retest vulnerability-analysis-on-pr

[etsien]

Just rebased to rh-aiq-main, incorporating discussed description changes to the tool prompting and configs

[etsien]

/retest vulnerability-analysis-on-pr

[zvigrinberg]

Hi @etsien ,
After you've resolved the conflicts, now the agent is crashing on startup

Can you please take a look and fix?

Thank you.

[etsien]

Hi @etsien , After you've resolved the conflicts, now the agent is crashing on startup Can you please take a look and fix?

Thank you.

patched, forgot to bring over the bugfix from the other branch

[zvigrinberg]

/retest vulnerability-analysis-on-pr

[zvigrinberg]

@etsien Please rebase and resolve conflicts, and we'll merge it ( that one improved the consistency and the results significantly).

[etsien]

/retest vulnerability-analysis-on-pr