fix: cvedb metric refactoring

[22f1001635]

This PR addresses the issue of handling unknown values in metrics by ensuring the UNKNOWN metric is properly initialized and validated in the CVE database. Key changes include:

✔️ Added UNKNOWN_METRIC_ID to populate_metrics to ensure the "UNKNOWN" metric is inserted into the metrics table.
✔️ Enhanced schema validation in latest_schema to check for the existence of the UNKNOWN metric, triggering a refresh if missing.
✔️ Updated metrics during refresh to ensure metrics are re-populated if the schema is outdated.

These changes ensure that the database consistently handles unknown values and maintains up-to-date metrics, improving reliability and correctness.

Fixes
#4812

[jloehel]

@22f1001635 What do you think about creating a constant for the metrics data, used in populate_metrics:

cve-bin-tool/cve_bin_tool/cvedb.py

Lines 645 to 650 in 8f9043a

	data = [
	(UNKNOWN_METRIC_ID, "UNKNOWN"),
	(EPSS_METRIC_ID, "EPSS"),
	(CVSS_2_METRIC_ID, "CVSS-2"),
	(CVSS_3_METRIC_ID, "CVSS-3"),
	]

like:

METRICS = [
        (UNKNOWN_METRIC_ID, "UNKNOWN"),
        (EPSS_METRIC_ID, "EPSS"),
        (CVSS_2_METRIC_ID, "CVSS-2"),
        (CVSS_3_METRIC_ID, "CVSS-3"),
]

and use the new constant in populate_metrics and in latest_schema for iteration like that:

        # Check for metrics table data
        if table_name == "metrics":
			for metrics_id, metrics_name in METRICS:
                result = cursor.execute(
                    "SELECT * FROM metrics WHERE metrics_id=? AND metrics_name", (metrics_id, metrics_name)
               )
               if not result.fetchone():
                   schema_latest = False

This has the benefit that it's simpler to add additional METRICS in the future like SSVC in example.

[22f1001635]

Hi @jloehel , I have made the required changes. Can you take a look and let me know

[jloehel]

Hi @jloehel , I have made the required changes. Can you take a look and let me know

Hi :-) Thanks. I have read the code again and I am not sure if the condition is at the right place because self.refresh_cache_and_update_db() gets still only executed if the db does not exist, the db is older than 24 days and the latest_schema is not matching. I think the condition needs to go here:

cve-bin-tool/cve_bin_tool/cvedb.py

Lines 328 to 336 in 04c47b8

	if (
	not self.latest_schema(
	"cve_severity", self.TABLE_SCHEMAS["cve_severity"]
	)
	or not self.latest_schema("cve_range", self.TABLE_SCHEMAS["cve_range"])
	or not self.latest_schema(
	"cve_exploited", self.TABLE_SCHEMAS["cve_exploited"]
	)
	):

... and you don't need to call populate_metrics again. It gets called in populate_db already. Only the condition when the database gets updated needs to get modified. Sorry, I should have checked this earlier.

[22f1001635]

@jloehel made changes; take a look and let me know if anything else is needed

[jloehel]

@22f1001635 What do you think about a test case for this scenario?

• Database exists, is not older than 24 hours and the schemas are all correct, but the UNKNOWN_METRIC is missing.
• Update the database
• Check if the UNKNOWN_METRIC exists after the update

[22f1001635]

hi @jloehel
I have made a test case covering this do take a look and let me know your thoughts

[jloehel]

Hi, thanks for the work. Looks good to me. I added one comment. Making it a non skip test would be good.

[jloehel]

It should be not a long running test. I think it's not necessary to sync the nvd source for it?

[22f1001635]

i have made the required changes check whether it aligns with what you have in mind if any other change is needed do tell

[jloehel]

It's a long test because it needs to sync all the NVD source during the update, right? We do not really need a source sync for the test, right? I am not sure, is it possible to update without sources?