INTEGRITY: Workflow testing and Github OAuth

.env.example

@@ -0,0 +1,4 @@
+GITHUB_ORG=ScummVM
+GITHUB_CLIENT_ID=github_client_id_from_oauth_app
+GITHUB_CLIENT_SECRET=github_client_secret_from_oauth_app
+FLASK_SECRET_KEY=any_random_key

README.md

@@ -1,17 +1,33 @@
-# ScummVM File Integrity Check (GSoC 2025)
+# ScummVM File Integrity Check
 
-This repository contains the server-side code for the upcoming file integrity check for game datafiles. This repository is part of the Google Summer of Code 2025 program.
+This repository contains the server-side code for the ScummVM's File Integrity service. 
 
-## Prerequisites
-### Local
+## Web Application Access Roles
+Access roles are determined by the GitHub team a user belongs to in the ScummVM organization.  
+Authentication and authorization are managed via GitHub OAuth.
+### Moderators
+Moderators have the following permissions:
+- Delete individual filesets
+- Update or add fileset metadata
+- Delete or update files
+- Mark a fileset as a Full fileset
+- Manually merge two filesets
+- Log User Email Notification
 
-- Python 3.x
-- MySQL
+### Admins
+Admins inherit all moderator permissions and have additional capabilities:
+- Clear the entire database
+- Delete all filtered filesets in bulk
 
-### Deployment
-- Apache2
+### Read-Only
+Read-only users can:
+- View filesets
+- Compare two filesets
 
-## Step-by-step Setup
+### No Access
+All other users (not logged in or not part of any ScummVM GitHub team) have no access to the application.
+
+## Development Setup
 
 ### 1. Clone the Repository
 ```bash
@@ -34,10 +50,18 @@ git checkout integrity
 ```
 
 ### 5. Install Required Python Packages
-You can also create a virtual environment (optional)
+You can install dependencies using either **pip** or **uv**.
+#### 5.1 Using pip
+(Optional) create and activate a virtual environment, then run:
 ```bash
 pip install -r requirements.txt
 ```
+#### 5.2 Using UV
+Make sure uv is already installed, then run:
+```bash
+uv sync
+```
+This will set up the virtual environment and install all dependencies as specified in `uv.lock`.
 
 ### 6. Install and Configure MySQL (if not already installed)
 Ensure MySQL is running and properly configured.
@@ -56,61 +80,106 @@ A `sample_mysql_config.json` is also present in the same directory.
 
 ### 8. Run Schema to Create Tables
 ```bash
-python schema.py
+python -m src.scripts.schema
+```
+or using uv:
+```bash
+uv run -m src.scripts.schema
+```
+### 9. Set up .env file (includes GitHub OAuth secrets)
+There is a .env.example file in the root directory. Copy it to create your .env file:
+```bash
+cp .env.example .env
+```
+Then update the values in .env as needed:
+```
+GITHUB_ORG=ScummVM
+GITHUB_CLIENT_ID=github_client_id_from_oauth_app
+GITHUB_CLIENT_SECRET=github_client_secret_from_oauth_app
+FLASK_SECRET_KEY=any_random_key
 ```
 
-## General Usecases
-
-### 1. Manual Generation of dat files (scan.dat) from existing games collection
-This utility helps in generating dat files from the existing game collections with the developers and then upload it to the database.
-#### Dat File Generation :
-This will generate the `.dat` file with complete checksums but no metadata.
+## Deployment Guide
+The Flask application is deployed using `mod_wsgi`, an Apache module for hosting WSGI applications.  
+Assuming Apache and mod_wsgi are already installed:
 
+Copy the provided Apache configuration file from the `apache2-config` directory into Apache’s `sites-available` folder:
+```bash
+sudo cp apache2-config/gamesdb.sev.zone.conf /etc/apache2/sites-available/
+```
+Enable the site:
+```bash
+sudo a2ensite gamesdb.sev.zone.conf
+```
+Reload Apache to apply changes:
 ```bash
-python compute_hash.py --directory <path_to_directory> --depth 0 --size 0
+sudo systemctl reload apache2
 ```
-- `--directory` : Path of directory with game files
-- `--depth` : Depth from root to game directories
-- `--size` : Use first n bytes of file to calculate checksum
 
-#### Database upload (for developers) :
-Uploading the `.dat` file to the database.
+## CLI Script Usecases
+- If using **pip**:
+  - On Debian/Ubuntu (or any system where `python` points to Python 2), use:
+    ```bash
+    python3 -m <module>
+    ```
+  - Otherwise:
+    ```bash
+    python -m <module>
+    ```
+- If using **uv**:
 ```bash
-python dat_parser.py --upload <scanned_dat_file/scan>.dat --user <username> --skiplog
+  uv run -m <module>
 ```
-- `--upload` : Upload DAT file(s) to the database
-- `--match` : Populate matching games in the database
-- `--user` : Username for database
-- `-r` : Recurse through directories
-- `--skiplog` : Skip logging dups
 
-### 2. Uploading dat files (scummvm.dat) generated from detection entries to the DB (Initial Seeding)
+### 1. Initial Seeding: Uploading dat files (scummvm.dat) generated from detection entries to the DB
 Upload the `.dat` file to the database using dat_parser script -
 ```bash
-python dat_parser.py --upload <detection_dat_file/scummvm>.dat --user <username> --skiplog
+python -m src.scripts.dat_parser --upload <path_to_scummvm.dat> --user <username> --skiplog
 ```
+- `--upload` : Upload DAT file(s) to the database (seeding)
+- `--match` : Populate matching games in the database
+- `--user` : (Optional) Username for database
+- `-r` : (Optional) Recurse through directories
+- `--skiplog` : (Optional) Skip logging dups
+
 `scummvm.dat` can be generated using -
 ```bash
 ./scummvm --dump-all-detection-entries
 ```
 
-### 3. Uploading already existing dat files (set.dat) from old collections to the DB 
-Upload the `.dat` file to the database using dat_parser script -
+### 2. Uploading already existing dat files (set.dat) from old collections to the DB
+Match the filesets from the `.dat` file using dat_parser script -
 ```bash
-python dat_parser.py --upload <old_dat_file/set>.dat --user <username> --skiplog
+python -m src.scripts.dat_parser --match <path_to_set.dat> --user <username> --skiplog
 ```
 
+### 3. Scan Utility: Manual Generation of dat files (scan.dat) from existing games collection
+This utility helps in generating dat files from the existing game collections which can be uploaded to the database.
+
+#### Dat File Generation (Scan Utility) :
+This will generate the `.dat` file with complete checksums and sizes (size, size-r and size-rd in case of macfiles)
 
-### 4. Validate Game Files from Client Side (integrity.json)
-Make a POST request to the following endpoint:
+```bash
+python -m src.scripts.compute_hash --directory <path_to_directory> --depth 0 --size 0 --limit-timestamps 2003-09-12
+```
+- `--directory` : (Required) Path of directory with game files
+- `--depth` : (Optional: Default = 0) Depth from root to game directories (e.g. 0 while scanning a single directory)
+- `--size` : (Optional: Default = 0) Use first n bytes of file to calculate checksum
+- `--limit-timestamps` : (Optional) Format - YYYY-MM-DD or YYYY-MM or YYYY. Filters out the files those
+                        were modified after the given timestamp. Note that if the
+                        modification time is today, it would not be filtered out.
+
+#### Perform Matching :
+Perform matching with the exisiting filesets in database.
+```bash
+python -m src.scripts.dat_parser --match <scanned_dat_file/scan>.dat --user <username> --skiplog
+```
+
+## Integrity Service: Validate Game Files from Client Side
+There exists a check_integrity button in the scummvm application which makes a POST request to the following endpoint:
 #### Local :
 ```bash
 http://localhost:5000/validate
 ```
-with the body in JSON format as shown in `sample_json_request.json` present in the main directory.
-There also exists a check_integrity button in the scummvm application itself which is under development.
-
-## Deployment
-
-The apache2 .conf file is located under `apache2-config/`.
+with the request body in JSON format as shown in `sample_json_request.json` present in the root directory.
 

apache2-config/gamesdb.sev.zone.conf

@@ -13,16 +13,7 @@
     Alias /static /home/ubuntu/projects/python/scummvm_sites_2025/scummvm-sites/static
 
     <Directory /home/ubuntu/projects/python/scummvm_sites_2025/scummvm-sites>
-        AuthType Basic
-	    AuthName "nope"
-	    AuthUserFile /home/ubuntu/projects/python/scummvm_sites_2025/.htpasswd
-	    Require valid-user
-    </Directory>
-
-    <Location "/validate">
-        AuthType None
         Require all granted
-        Satisfy Any
-    </Location>
+    </Directory>
 
 </VirtualHost>

app.wsgi

@@ -1,9 +1,12 @@
 import sys
+import os
 import logging
 
-sys.path.insert(0, "/home/ubuntu/projects/python/scummvm_sites_2025/scummvm-sites")
+project_root = os.path.dirname(os.path.abspath(__file__))
+if project_root not in sys.path:
+    sys.path.insert(0, project_root)
 
-from src.app.fileset import app as application
+from src.app.fileset import app as application  # noqa
 
 logging.basicConfig(stream=sys.stderr)
 sys.stderr = sys.stdout

pyproject.toml

@@ -1,15 +1,15 @@
 [project]
 name = "scummvm-sites"
 version = "0.1.0"
-description = "Add your description here"
 requires-python = ">=3.12"
 dependencies = [
+    "authlib>=1.6.3",
     "blinker>=1.9.0",
     "cffi>=1.17.1",
     "click>=8.2.1",
     "cryptography>=45.0.6",
-    "flask>=3.1.1",
-    "flask-limiter>=3.12",
+    "flask>=3.1.2",
+    "flask-dance>=7.1.0",
     "iniconfig>=2.1.0",
     "itsdangerous>=2.2.0",
     "jinja2>=3.1.6",
@@ -18,8 +18,10 @@ dependencies = [
     "pluggy>=1.6.0",
     "pycparser>=2.22",
     "pygments>=2.19.2",
-    "pymysql>=1.1.1",
+    "pymysql>=1.1.2",
     "pytest>=8.4.1",
+    "python-dotenv>=1.1.1",
+    "requests>=2.32.5",
     "setuptools>=80.9.0",
     "werkzeug>=3.1.3",
     "wheel>=0.45.1",

requirements.txt

@@ -16,4 +16,7 @@ pytest
 setuptools
 Werkzeug
 wheel
-Flask-Limiter
+authlib
+python-dotenv
+requests
+Flask-Dance

src/app/auth/github_oauth.py

@@ -0,0 +1,25 @@
+from authlib.integrations.flask_client import OAuth
+import os
+
+GITHUB_ORG = os.environ.get("GITHUB_ORG")
+TEAM_ROLES = ["integrity-admins", "integrity-devs", "integrity-ro"]
+
+
+def init_oauth(app):
+    oauth = OAuth(app)
+
+    oauth.register(
+        name="github",
+        client_id=os.environ.get("GITHUB_CLIENT_ID"),
+        client_secret=os.environ.get("GITHUB_CLIENT_SECRET"),
+        access_token_url="https://github.com/login/oauth/access_token",
+        access_token_params=None,
+        authorize_url="https://github.com/login/oauth/authorize",
+        authorize_params=None,
+        api_base_url="https://api.github.com/",
+        client_kwargs={
+            "scope": "read:user read:org",
+        },
+    )
+
+    return oauth

src/app/auth/helper.py

@@ -0,0 +1,21 @@
+from flask import session
+
+
+def get_user_role():
+    user_role = session.get("user", {}).get("role", "No Access")
+    return user_role
+
+
+def get_username():
+    username = session.get("user", {}).get("username", "")
+    return username
+
+
+def is_moderator_access():
+    """
+    Returns true if there is more than read only access, i.e its either admin or moderator.
+    """
+    user_role = session.get("user", {}).get("role", "")
+    if user_role in ["Admin", "Moderator"]:
+        return True
+    return False

src/app/auth/role_based_auth.py

@@ -0,0 +1,28 @@
+from functools import wraps
+from flask import session, redirect, url_for, abort
+
+
+def role_required(*roles):
+    """
+    Decorator Usage: @role_required("Admin", "Moderator", "Read Only")
+    """
+
+    def wrapper(f):
+        @wraps(f)
+        def decorated_function(*args, **kwargs):
+            if "user" not in session:
+                return redirect(url_for("home"))
+
+            user = session.get("user")
+            user_role = user["role"]
+
+            if user_role == "No Access":
+                return redirect(url_for("home"))
+
+            if user_role not in roles:
+                abort(403)
+            return f(*args, **kwargs)
+
+        return decorated_function
+
+    return wrapper

src/app/env_loader.py

@@ -0,0 +1,11 @@
+from pathlib import Path
+from dotenv import load_dotenv
+import sys
+import os
+
+project_root = Path(__file__).resolve().parents[2]
+if project_root not in sys.path:
+    sys.path.insert(0, project_root)
+
+env_path = os.path.join(project_root, ".env")
+load_dotenv(dotenv_path=env_path)