Update README and enhance SecurityManager to skip ClamAV scans on Windows by default

Author: dylanpicart

README.md

@@ -2,7 +2,7 @@
 
 ## Description
 
-**Excel API Web Scraper** is a Python-based project that automates the process of web scraping, downloading, and storing Excel files from the NYC InfoHub website. It features a **modular, object-oriented** design with built-in **security checks** (virus scanning and MIME-type validation) for downloaded Excel files.
+**Excel API Web Scraper** is a Python-based project that automates the process of web scraping, downloading, and storing Excel files from NYC InfoHub. It features a modular, object-oriented design with built-in **security checks** (virus scanning and MIME-type validation) for downloaded Excel files, **with an option to skip antivirus scans on Windows** if ClamAV isn’t readily available.
 
 ### Highlights
 
@@ -11,8 +11,8 @@
 - **Parallel CPU-bound hashing** with `ProcessPoolExecutor`  
 - **Detailed logging** with a rotating file handler  
 - **Progress tracking** via `tqdm`  
-- **SecurityManager** for virus scanning (ClamAV) and file-type checks  
-- **Refined “only Excel files”** approach, skipping malicious or non-Excel data
+- **SecurityManager** for optional virus scanning (ClamAV) and file-type checks  
+- **Skips ClamAV scanning on Windows** by default, avoiding setup complexities while still functioning seamlessly
 
 ---
 
@@ -37,9 +37,9 @@
    - Uses `ProcessPoolExecutor` to compute SHA-256 hashes in parallel, fully utilizing multi-core CPUs without blocking the async loop.
 
 7. **Security Checks**  
-   - A **SecurityManager** class provides:
-     - **Virus scanning** with ClamAV (in-memory scanning before saving).  
-     - **MIME-type validation** with `python-magic`, ensuring files truly are Excel.
+   - In-memory **virus scanning** with ClamAV (via `pyclamd` or `clamd`)  
+   - **MIME-type validation** with `python-magic`, ensuring files are truly Excel  
+   - **Skip scanning on Windows** by default (see below)
 
 8. **Prevents Redundant Downloads**  
    - Compares new file hashes with stored hashes; downloads only if the file has changed.
@@ -50,6 +50,21 @@
 
 ---
 
+## Windows Antivirus Skipping
+
+By default, **ClamAV scanning** is not performed on **Windows** to avoid environment complexities—ClamAV is primarily a Linux/UNIX daemon. The `SecurityManager` class checks `platform.system()`, and if it’s `Windows`, it **short-circuits** scanning and returns a **clean** status. This behavior can be **overridden** by setting:
+
+```python
+security_manager = SecurityManager(skip_windows_scan=False)
+```
+
+…in which case the code will attempt a normal ClamAV call. To make this work on Windows, you’d typically need:
+
+- **WSL or Docker** to run the ClamAV daemon, or
+- Another setup that exposes a ClamAV socket/port for Python to connect to.
+
+---
+
 ## Requirements
 
 ### System Requirements
@@ -306,6 +321,7 @@ A GitHub Actions workflow is set up in `.github/workflows/ci-cd.yml`. It:
 - **Redundant Downloads**: Prevented by storing file hashes and only updating on changes.
 - **Virus Scan Overhead**: In-memory scanning might add overhead, but ensures security.
 - **Size Limit Errors**: If you see “INSTREAM: Size limit reached” warnings, increase `StreamMaxLength` in `clamd.conf`.
+- **Windows Skipping**: If you can’t run ClamAV natively, the skip mechanism means the scraper still works without throwing errors.
 
 ---
 

src/excel_scraper.py

@@ -16,6 +16,7 @@
 from tqdm import tqdm
 import pyclamd
 import magic
+import platform
 
 load_dotenv()
 logging.info(f"CHROMEDRIVER_PATH: {os.environ.get('CHROMEDRIVER_PATH')}")
@@ -94,14 +95,18 @@ class SecurityManager:
     """
     Encapsulates file validation logic, including virus scanning
     (ClamAV via python-clamd) and MIME-type checks (python-magic).
+    Skips ClamAV scan on Windows by default.
     """
 
-    def __init__(self, clam_socket="/var/run/clamav/clamd.ctl"):
+    def __init__(self, clam_socket="/var/run/clamav/clamd.ctl", skip_windows_scan=True):
         """
         Pass the path to the ClamAV Unix socket file.
         By default, it's often /var/run/clamav/clamd.ctl on Debian/Ubuntu.
+        If using Windows, automatically skips ClamAV scan.
         """
         self._clam_socket = clam_socket
+        self._skip_windows_scan = skip_windows_scan
+
 
     def scan_for_viruses(self, file_bytes: bytes) -> tuple:
         """
@@ -110,8 +115,12 @@ def scan_for_viruses(self, file_bytes: bytes) -> tuple:
         Possible status_code values:
           - "ERROR": an exception or connection failure occurred
           - "FOUND": virus/malware was detected
-          - "OK": no virus was found
+          - "OK": file is clean or scanning is skipped on Windows
         """
+        if self._skip_windows_scan and platform.system().lower() == "windows":
+            logging.info("Skipping virus scan on Windows.")
+            return ("OK", "Skipping AV check on Windows")
+        
         try:
             cd = pyclamd.ClamdUnixSocket(filename=self._clam_socket)
             result = cd.scan_stream(file_bytes)
@@ -329,8 +338,8 @@ class NYCInfoHubScraper(BaseScraper):
         "other_reports": []
     }
 
-    def __init__(self, base_dir=None, data_dir=None, hash_dir=None, log_dir=None):
-        super().__init__(security_manager=SecurityManager("/var/run/clamav/clamd.ctl"))
+    def __init__(self, base_dir=None, data_dir=None, hash_dir=None, log_dir=None, skip_win_scan=True):
+        super().__init__(security_manager=SecurityManager("/var/run/clamav/clamd.ctl", skip_windows_scan=skip_win_scan))
         script_dir = os.path.abspath(os.path.dirname(__file__)) if "__file__" in globals() else os.getcwd()
         self._base_dir = base_dir or os.path.join(script_dir, "..")
         self._data_dir = data_dir or os.path.join(self._base_dir, "data")

tests/test_excel_scraper.py

@@ -3,7 +3,8 @@
 import pytest
 import hashlib
 from unittest.mock import patch, MagicMock
-from src.excel_scraper import NYCInfoHubScraper
+from src.excel_scraper import NYCInfoHubScraper, SecurityManager
+import platform
 
 def test_compute_file_hash():
     """
@@ -94,6 +95,29 @@ def mock_stream(method, _url, timeout=10):
         mock_stream_call.assert_called_once()
         mock_scan.assert_called_once_with(fake_excel_content)
         mock_mime.assert_called_once_with(fake_excel_content)
+        
+def test_skip_windows_scan_true():
+    manager = SecurityManager(skip_windows_scan=True)
+
+    # Mock platform.system() to return 'Windows'
+    with patch.object(platform, 'system', return_value='Windows'):
+        status, message = manager.scan_for_viruses(b"FakeExcelData")
+        assert status == "OK"
+        assert "Skipping AV check on Windows" in message
+
+def test_skip_windows_scan_false():
+    manager = SecurityManager(skip_windows_scan=False)
+
+    # Mock platform.system() to return 'Windows'
+    # We confirm it *doesn't* short-circuit and tries to do normal scanning
+    # Since there's no real ClamAV, you might see an 'ERROR' or something
+    # unless you mock the clamd calls as well. For illustration:
+    with patch.object(platform, 'system', return_value='Windows'), \
+    patch("pyclamd.ClamdUnixSocket", side_effect=Exception("No daemon")):
+        
+        status, message = manager.scan_for_viruses(b"FakeExcelData")
+        assert status == "ERROR"
+        assert "No daemon" in message
 
 @pytest.mark.asyncio
 async def test_download_excel_virus_found(test_scraper):