FREYA-1693: Add multi-disease serology dashboard

pages/dashboards/templates/dashboards/index.html

@@ -16,5 +16,8 @@ <h4>Available dashboards</h4>
         <li>
             <a href="{% url 'dashboards:lineage_competition' %}">SARS-CoV-2 Variant Competition</a>
         </li>
+        <li>
+            <a href="{% url 'dashboards:multidisease_serology' %}">Multi-disease serology</a>
+        </li>
     </ul>
 {% endblock %}

pages/dashboards/templates/dashboards/multidisease_serology.html

@@ -0,0 +1,163 @@
+{% extends "base.html" %}
+
+{% block content %}
+  <a href="#content-start" class="sr-only focus:not-sr-only focus:outline-none focus:ring-2 focus:ring-pp-mid-blue px-3 py-2 bg-white">Skip to content</a>
+  <h1 class="sr-only">Multi-disease serology</h1>
+  <div id="content-start">
+    <div class="prose max-w-none space-y-6 mt-4">
+      <div class="bg-pp-pale-grey/30 border-l-4 border-pp-mid-blue text-pp-dark-blue p-4" role="status" aria-live="polite" aria-atomic="true">
+        <p class="m-0"><strong>All data last updated:</strong> TODO</p>
+      </div>
+
+      <h2 id="introduction">Introduction</h2>
+      <p>
+        The COVID-19 pandemic highlighted the importance of serological surveillance in tracking viral
+        transmission dynamics, understanding immune responses, guiding vaccination strategies, and assisting
+        in decisions related to public health. High‑throughput serological assays for SARS‑CoV‑2 were
+        developed very early in the pandemic at KTH and SciLifeLab to enable surveillance of populations
+        globally. For information about work done with SARS‑CoV‑2 during the pandemic, see the
+        <a href="/historical-background/">historical background section</a>.
+      </p>
+      <p>
+        As we move on from the pandemic, it is crucial that serological studies include more pathogens
+        than just SARS‑CoV‑2. One of the capabilities that is part of
+        <a href="/plp-program-background/">SciLifeLab's Pandemic Laboratory Preparedness (PLP) programme</a>,
+        named 'Multi‑disease serology' aims to create a sustainable, long‑term
+        resource enabling a broad, frequent, and large‑scale surveillance of serostatus. They are working to
+        create antibody repertoires that can be used to analyse thousands of samples on hundreds of antigens.
+        The project is led by Peter Nilsson at KTH Royal Institute of Technology and SciLifeLab. To learn more,
+        check out the <a href="/resources/serology/">pandemic preparedness resource page for this project</a>.
+      </p>
+
+      <h2 id="historical-background">Historical background</h2>
+      <p>
+        During the pandemic, those working on the multi‑disease serology study created a high‑throughput
+        multiplex bead‑based serological assay for SARS‑CoV‑2 (see
+        <a href="https://doi.org/10.1002/cti2.1312" target="_blank" rel="noopener noreferrer">Hober <em>et&nbsp;al.</em> (2021) <span class="sr-only">(opens in a new tab)</span></a>
+        and the <a href="/dashboards/serology-statistics/">Serology tests for SARS‑CoV‑2 at SciLifeLab Data Dashboard</a>
+        for details). As of July 2023, the assay had been used to analyse over 250,000 samples, and contributed to
+        <a href="https://publications.scilifelab.se/label/Autoimmunity%20and%20Serology%20Profiling" target="_blank" rel="noopener noreferrer">around 40 publications <span class="sr-only">(opens in a new tab)</span></a>,
+        including studies on seroprevalence and on vaccine efficacy in immunocompromised individuals and individuals
+        with autoimmune diseases. Following the pandemic, the group began to refocus their efforts towards pandemic
+        preparedness, and began work to extend the assay to provide a platform for parallelised multi‑disease
+        serological studies, including a wide range of antigens representing various infectious diseases. The bead‑based
+        setup enables a stepwise addition of new proteins, allowing a continuous implementation of pathogen‑representing
+        antigens.
+      </p>
+
+      <h2 id="current-methods-and-progress">Current methods and progress</h2>
+      <p>
+        The project has produced and evaluated many antigens. This includes a wide range of different variants of the
+        SARS‑CoV‑2 proteins, with a focus on the spike glycoprotein, also covering the majority of mutated variants.
+        They have also created spike representations of SARS, MERS, and the other four human coronaviruses causing
+        common cold (HKU1, OC43, NL63 and 229E). They have also produced influenza virus antigens representing the
+        glycoproteins haemagglutinin and neuraminidase. Here, they have initially focused on the variants present in the
+        trivalent vaccine for the season 2021–2022, which includes the A(H1N1)/Wisconsin, A(H3N2)/Cambodia, and
+        B(Victoria)/Washington strains. Furthermore, they have produced representations of Respiratory Syncytial Virus
+        (RSV), including two surface proteins (G and F) in two different strains. Antigens representing mpox have also
+        been generated and included in the current bead‑based antigen collection.
+      </p>
+      <p>
+        Other viral respiratory infections that are being monitored in Sweden include adenovirus, metapneumovirus, and
+        parainfluenza virus. These have also been added to the project. The project has designed representations of the
+        fibre protein of adenovirus B7, metapneumovirus proteins F and G of the strain CAN97‑83, and protein HN and F
+        for parainfluenza virus have been designed based on strain Washington/1957 and strain C39, respectively.
+      </p>
+      <p>
+        The proteins designed and produced created by the project to date are listed in the
+        <a href="#table-of-proteins-created-at-kth">below table</a>.
+      </p>
+
+      <h2 id="table-of-proteins-created-at-kth">Table of proteins created at KTH</h2>
+      <p>
+        Proteins designed, expressed, purified, and characterised at the
+        <a href="https://www.kth.se/pps" target="_blank" rel="noopener noreferrer">KTH node of Protein Production Sweden <span class="sr-only">(opens in a new tab)</span></a>,
+        a national research infrastructure funded by the Swedish Research Council. They have been expressed either in HEK
+        or CHO cells or in <em>E. coli</em>, with different affinity tags and either as fragments or full‑length proteins.
+      </p>
+      <div class="sm:hidden bg-pp-pale-grey/30 border-l-4 border-pp-mid-blue text-pp-dark-blue p-3" role="note">
+        Rotating your phone will improve the view of this table.
+      </div>
+      <div class="overflow-x-auto rounded-lg shadow ring-1 ring-gray-200">
+        <table class="min-w-full text-sm text-left table-fixed" aria-labelledby="table-of-proteins-created-at-kth">
+          <caption class="sr-only">Proteins designed, expressed, purified, and characterised at KTH</caption>
+          <thead class="bg-pp-pale-grey text-pp-dark-grey text-xs uppercase tracking-wide sticky top-0">
+            <tr>
+              {% for header in kth_headers %}
+                <th scope="col" class="px-4 py-3 font-semibold whitespace-nowrap">{{ header }}</th>
+              {% endfor %}
+            </tr>
+          </thead>
+          <tbody class="divide-y divide-gray-200">
+            {% if kth_rows %}
+              {% for row in kth_rows %}
+                <tr class="odd:bg-white even:bg-gray-50 hover:bg-gray-100">
+                  {% for cell in row %}
+                    <td class="px-4 py-3 whitespace-normal break-normal">{{ cell }}</td>
+                  {% endfor %}
+                </tr>
+              {% endfor %}
+            {% else %}
+              <tr>
+                <td class="px-4 py-3 text-gray-500" colspan="{{ kth_headers|length }}">No data available.</td>
+              </tr>
+            {% endif %}
+          </tbody>
+        </table>
+      </div>
+
+      <h2 id="ongoing-work-and-collaborations">Ongoing work and collaborations</h2>
+      <p>
+        The work of the project is now expanding into the area of flavivirus (Tick‑borne encephalitis virus, Zika virus,
+        Dengue virus, West Nile virus, Yellow fever virus, Japanese encephalitis virus) and herpesvirus (Epstein–Barr
+        virus, Varicella zoster virus, Herpes simplex virus, Cytomegalovirus).
+      </p>
+      <p>
+        The project is also collaborating with another SciLifeLab PLP project
+        <a href="/resources/immunomonitoring/">“Systems‑level immunomonitoring to unravel immune response to a novel
+          pathogen”</a>, headed by Petter Brodin (Karolinska Institutet, KI) and Jochen Schwenk (KTH), to include a wide
+        range of externally produced antigens representing a large part of the Swedish vaccination program, see list
+        below.
+      </p>
+      <p>
+        The multi‑disease serological assay is under constant development and will gradually be incorporated into two
+        SciLifeLab infrastructure units; <a href="https://www.scilifelab.se/units/affinity-proteomics/" target="_blank"
+        rel="noopener noreferrer">Autoimmunity and Serology Profiling</a> and
+        <a href="https://www.scilifelab.se/units/affinity-proteomics/" target="_blank" rel="noopener noreferrer">Affinity
+        Proteomics Stockholm</a>. The goal is to provide a flexible and quickly adaptable assay for high‑throughput
+        multiplex studies on seroprevalence, available to both the Public Health Agency of Sweden and researchers in
+        academia and industry.
+      </p>
+
+      <h2 id="externally-produced-antigens">Externally produced antigens</h2>
+      <div class="overflow-x-auto rounded-lg shadow ring-1 ring-gray-200">
+        <table class="min-w-full text-sm text-left table-fixed" aria-labelledby="externally-produced-antigens">
+          <caption class="sr-only">Externally produced antigens included in the multi‑disease serology assay</caption>
+          <thead class="bg-gray-100 text-gray-700 text-xs uppercase tracking-wide sticky top-0">
+            <tr>
+              {% for header in external_headers %}
+                <th scope="col" class="px-4 py-3 font-semibold whitespace-nowrap">{{ header }}</th>
+              {% endfor %}
+            </tr>
+          </thead>
+          <tbody class="divide-y divide-gray-200">
+            {% if external_rows %}
+              {% for row in external_rows %}
+                <tr class="odd:bg-white even:bg-gray-50 hover:bg-gray-100">
+                  {% for cell in row %}
+                    <td class="px-4 py-3 whitespace-normal break-normal">{{ cell }}</td>
+                  {% endfor %}
+                </tr>
+              {% endfor %}
+            {% else %}
+              <tr>
+                <td class="px-4 py-3 text-gray-500" colspan="{{ external_headers|length }}">No data available.</td>
+              </tr>
+            {% endif %}
+          </tbody>
+        </table>
+      </div>
+    </div>
+  </div>
+{% endblock %}
+

pages/dashboards/urls.py

@@ -1,9 +1,13 @@
+# ruff: noqa: E501
 from django.urls import path
-from .views import DashboardsIndex, LineageCompetition
+from .views import DashboardsIndex, LineageCompetition, MultiDiseaseSerology
 
 app_name = "dashboards"
 
+# fmt: off
 urlpatterns = [
     path("", DashboardsIndex.as_view(), name="index"),
     path("lineage-competition/", LineageCompetition.as_view(), name="lineage_competition"),
+    path("multidisease-serology/", MultiDiseaseSerology.as_view(), name="multidisease_serology"),
 ]
+# fmt: on

pages/dashboards/utils/blobserver.py

@@ -0,0 +1,160 @@
+"""Utilities for fetching and parsing serology data from blobserver.
+
+This module provides small helpers to fetch remote Excel files with retry
+logic and parse the first worksheet into a list of dictionary records that can
+be consumed by Django views and templates.
+"""
+
+from __future__ import annotations
+
+import io
+import logging
+from typing import Any, Dict, List, Optional
+
+import httpx
+from openpyxl import load_workbook
+
+
+DEFAULT_TIMEOUT = httpx.Timeout(10.0, connect=5.0)
+DEFAULT_HEADERS = {"User-Agent": "pathogens-portal/requests"}
+
+logger = logging.getLogger(__name__)
+
+
+def get_with_retries(
+    url: str,
+    retries: int = 3,
+    timeout: Optional[httpx.Timeout] = None,
+    headers: Optional[Dict[str, str]] = None,
+    user_agent: Optional[str] = None,
+) -> httpx.Response:
+    """GET a URL with retry and timeout policy.
+
+    Args:
+        url: Absolute URL to fetch.
+        retries: Number of attempts before failing.
+        timeout: Optional custom timeout. If not provided, a sensible default
+            with separate connect timeout is used.
+        headers: Optional HTTP headers to include in the request. These
+            headers will override defaults if keys overlap.
+        user_agent: Optional User-Agent string. If provided, it overrides the
+            default; if ``headers`` also contains ``User-Agent``, that takes
+            precedence over this argument.
+
+    Returns:
+        httpx.Response: Successful response object with a 2xx status.
+
+    Raises:
+        RuntimeError: If all retry attempts fail. The original exception from
+            the last attempt is chained for debugging context.
+    """
+    timeout = timeout or DEFAULT_TIMEOUT
+    # Build final headers with precedence: DEFAULT < user_agent arg < headers arg
+    final_headers: Dict[str, str] = dict(DEFAULT_HEADERS)
+    if user_agent:
+        final_headers["User-Agent"] = user_agent
+    if headers:
+        final_headers.update(headers)
+    last_error: Optional[Exception] = None
+    for attempt in range(1, retries + 1):
+        logger.debug(
+            "http_get_attempt",
+            extra={
+                "url": url,
+                "attempt": attempt,
+                "retries": retries,
+                "user_agent": final_headers.get("User-Agent"),
+            },
+        )
+        try:
+            with httpx.Client(timeout=timeout, headers=final_headers) as client:
+                response = client.get(url)
+                response.raise_for_status()
+                logger.debug(
+                    "http_get_success",
+                    extra={"url": url, "status_code": response.status_code, "content_length": len(response.content)},
+                )
+                return response
+        except Exception as exc:
+            last_error = exc
+            logger.warning(
+                "http_get_retry",
+                extra={"url": url, "attempt": attempt, "retries": retries},
+                exc_info=True,
+            )
+    logger.error("http_get_failed", extra={"url": url, "retries": retries, "error_type": type(last_error).__name__})
+    raise RuntimeError(
+        f"Failed to fetch URL after {retries} attempts: {url!r}"
+    ) from last_error
+
+
+def fetch_excel_first_sheet_as_records(
+    url: str,
+    retries: int = 3,
+    headers: Optional[Dict[str, str]] = None,
+    user_agent: Optional[str] = None,
+) -> List[Dict[str, Any]]:
+    """Parse the first worksheet of an Excel file into record dictionaries.
+
+    The first row is treated as the header row. Blank header cells are ignored,
+    and completely empty data rows are skipped. Values are normalised so that
+    missing cells become empty strings.
+
+    Args:
+        url: Absolute URL of the Excel file to download.
+        retries: Number of fetch attempts before failing.
+        headers: Optional HTTP headers forwarded to the download request.
+        user_agent: Optional User-Agent string to use for the request when
+            ``headers`` does not already define one.
+
+    Returns:
+        List[Dict[str, Any]]: One dictionary per non-empty row, limited to the
+        non-blank headers in the first row.
+    """
+    logger.debug("excel_fetch_start", extra={"url": url})
+    response = get_with_retries(url, retries=retries, headers=headers, user_agent=user_agent)
+    workbook = load_workbook(
+        io.BytesIO(response.content), read_only=True, data_only=True
+    )
+    first_sheet_name = workbook.sheetnames[0]
+    worksheet = workbook[first_sheet_name]
+
+    row_iterator = worksheet.iter_rows(values_only=True)
+
+    try:
+        header_row = next(row_iterator)
+    except StopIteration:
+        logger.info("excel_no_rows", extra={"url": url, "sheet": first_sheet_name})
+        return []
+
+    headers: List[str] = []
+    for header_cell in header_row:
+        # Normalise headers to non-empty strings
+        header_text = "" if header_cell is None else str(header_cell).strip()
+        headers.append(header_text)
+
+    normalised_headers = [h for h in headers if h]
+    logger.debug(
+        "excel_headers_parsed",
+        extra={"url": url, "sheet": first_sheet_name, "header_count": len(normalised_headers)},
+    )
+
+    records: List[Dict[str, Any]] = []
+    for data_row in row_iterator:
+        # Build a dict limited to normalised headers
+        record: Dict[str, Any] = {}
+        for index, header in enumerate(headers):
+            if not header:
+                continue
+            value = data_row[index] if index < len(data_row) else None
+            record[header] = "" if value is None else value
+        # Skip fully empty rows
+        if any(value not in ("", None) for value in record.values()):
+            # Keep only normalised headers (drop any blanks)
+            records.append({key: record.get(key, "") for key in normalised_headers})
+
+    logger.info(
+        "excel_rows_parsed",
+        extra={"url": url, "sheet": first_sheet_name, "row_count": len(records)},
+    )
+    return records

pages/dashboards/views/__init__.py

@@ -0,0 +1,15 @@
+"""Public view exports for the dashboards app.
+
+Re-exports view classes to provide a stable import surface for URL configs
+and other modules.
+"""
+
+from .index import DashboardsIndex
+from .lineage_competition import LineageCompetition
+from .multidisease_serology import MultiDiseaseSerology
+
+__all__ = [
+    "DashboardsIndex",
+    "LineageCompetition",
+    "MultiDiseaseSerology",
+]

pages/dashboards/views/index.py

@@ -0,0 +1,11 @@
+from utils.views import BaseTemplateView
+
+
+class DashboardsIndex(BaseTemplateView):
+    """Index page for Dashboard
+
+    WIP: currently a simple templateview but will be updated later
+    """
+
+    template_name = "dashboards/index.html"
+    title = "Data dashboards"

pages/dashboards/views.py → pages/dashboards/views/lineage_competition.py

@@ -1,16 +1,6 @@
 from utils.views import BaseTemplateView
 
 
-class DashboardsIndex(BaseTemplateView):
-    """Index page for Dashboard
-
-    WIP: currently a simple templateview but will be updated later
-    """
-
-    template_name = "dashboards/index.html"
-    title = "Data dashboards"
-
-
 class LineageCompetition(BaseTemplateView):
     """SARS-CoV-2 Variant Competition dashboard page."""