A system designed to provide real-time alerts about new security vulnerabilities affecting specific classes of software, such as operating systems, web servers, databases, applications, browsers, etc. The vulnerabilities are identified by parsing open-source databases containing relevant information. This data will be available for consumption in multiple formats, including HTML + RDFa, JSON-LD, Kafka messaging, and SPARQL queries.
Client
Server
- Java
- Spring Boot
- Apache Kafka
- Python
Database
Infrastructure
- Docker
- Cloud Run
- Google Filestore NFS
- Google Managed Kafka
- Google Compute Engine VM
- Search and Query Interface - Provide a web interface to allow users to search and filter vulnerabilities based on various criteria (e.g., software type, severity, CVE ID).
- Real-Time Alerts – Implement a publish/subscribe mechanism using Kafka to notify users of new security issues in real-time.
- SPARQL Endpoint - Offer a queryable SPARQL endpoint
- Multi-Format Data Representation - Provide vulnerability details in HTML + RDFa and JSON-LD.
This project uses Docker
Check this for downloading it.
Clone the project
git clone https://github.com/stefan1anuby/Web-Vulnerability-Alerting.gitGo to the project directory
cd Web-Vulnerability-AlertingStart the project
docker-compose -f server/compose.yaml up --buildAdd vulnerabilities to the database
python scraper/main.py(be sure to point to the correct domain by looking in the script)
Go to the http://localhost:8088/views/menu
-
Create a New Project in GCP
Start by creating a new project in Google Cloud Platform (GCP) to isolate resources and manage them independently. -
Create a New VPC and Firewall Rules
Set up a new Virtual Private Cloud (VPC) and define firewall rules to allow traffic from Cloud Run to:- NFS Filestore
- Kafka
-
Create an NFS Filestore in the VPC
Deploy an NFS Filestore instance in the previously created VPC to serve as the shared storage. -
Create a Kafka Cluster in the VPC
Deploy a Kafka cluster in the same VPC for message brokering and communication. -
Deploy a Google Cloud Run Service
- Deploy a Cloud Run service in the same VPC, pointing it to the
/serverdirectory in your project. - You can use a GitHub repository link, a Docker image registry, or the Google Cloud IDE for this deployment.
- Set the required environment variables, including:
- The NFS volume path.
- The Kafka URL for communication.
- Ensure the IAM role assigned to the service has the necessary permissions to interact with Kafka and Filestore.
- Deploy a Cloud Run service in the same VPC, pointing it to the
-
[OPTIONAL] Create a Compute Engine VM for External Scraper Jobs
- Deploy a Compute Engine VM with internet access by either assigning a temporary external IP or using Cloud NAT.
- SSH into the VM and set up the scraper script to run as a scheduled job (e.g., every X hours/days/months).
- Update the script's URL to point to the Cloud Run app.
The main page contains selection buttons and text areas where you can select or write specific data for a vulnerability (year, affected software, version, etc.). The specified data will be used to make filters (if it is the case) in the SPARQL query, which will extract the desired vulnerabilities (if there are any that meet the criteria). The vulnerabilities are described in HTML + RDFa or JSON-LD format. There is a button that, when pressed, it shows the user the SPARQL query that is sent to the server based on the selected options.
The SPARQL query is predefined based on the STIX ontology.
let query =
`PREFIX stix: <http://purl.org/cyber/stix#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX schema: <https://schema.org/SoftwareApplication#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?vulnerability ?description ?solution
WHERE {
?vulnerability rdf:type stix:Vulnerability .
?vulnerability stix:description ?description .
?vulnerability stix:published ?published .
?vulnerability stix:severity ?severity .
?vulnerability stix:ExternalReference ?application .
?application schema:applicationCategory ?topic .
?application schema:applicationSubCategory ?product .
?application schema:softwareVersion ?version .
?vulnerability stix:mitigatedBy ?solutionId .
?solutionId stix:description ?solution .`;
query += '\n';Based on the options selected, a filter is added to the query.
if(category !== 'All') query += `FILTER(lcase(?topic) = "${categoryLowercase}") .` + '\n';
if(product !== 'All' && product !== '') query += `FILTER(lcase(?product) = "${productLowercase}") .` + '\n';
if(version !== 'All' && version !== '') query += `FILTER(lcase(STR(?version)) = "${versionLowercase}") .` + '\n';
if(severity !== 'All') query += `FILTER(lcase(?severity) = "${severityLowercase}") .` + '\n';
if(date !== 'All') query += `FILTER(YEAR(STRDT(?published, xsd:dateTime)) = ${date}) .` + '\n';
query += `}`;The user can also write a specific id (the format is CVE-\d{4}-\d{4,7}) in a search bar and extract the associated vulnerability in the database. The result can be in HTML + RDFa or JSON-LD format.
Another way to use the app is through ingesting Kafka messages. You can ingest them by creating a consumer using their library.
package org.laborator;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class Main {
// docker run --rm --network server_default confluentinc/cp-kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic General --from-beginning
public static void main(String[] args) {
// Kafka consumer configuration settings
String topicName = "General"; // Replace with your topic name
String bootstrapServers = "kafka:9092"; // Replace with your Kafka broker address
String groupId = "example-group"; // Consumer group ID
// Consumer configuration properties
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // Start from the earliest message if no offset is committed
// Create Kafka consumer
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
// Subscribe to the topic
consumer.subscribe(Collections.singletonList(topicName));
System.out.println("Kafka Consumer is now listening for messages...");
try {
while (true) {
// Poll for records (messages) from Kafka
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("Consumed message: Key = %s, Value = %s, Partition = %d, Offset = %d%n",
record.key(), record.value(), record.partition(), record.offset());
}
}
} catch (Exception e) {
System.err.println("Error in Kafka consumer: " + e.getMessage());
} finally {
consumer.close();
}
}
}and an example of ingesting logs could be this:
- Popa Stefan-Eduard - [email protected]
- Dumitru Daniel-Antoniu - [email protected]
https://drive.google.com/drive/u/0/folders/1SO7nJIG3HXhj3JQYjCDbimzQRsEUzGOs