Merge pull request #2543 from opensafely-core/evansd/table-permissions

Add a permissions system

Author: evansd

docs/includes/generated_docs/language__permissions.md

@@ -0,0 +1,20 @@
+
+<h4 class="attr-heading" id="claim_permissions" data-toc-label="claim_permissions" markdown>
+  <tt><strong>claim_permissions</strong>(<em>*permissions</em>)</tt>
+</h4>
+<div markdown="block" class="indent">
+This function allows you to access any restricted table or feature when working with
+dummy data. It will NOT allow you access with real data: for that you will need the
+appropriate permissions on the OpenSAFELY platform.
+
+Permission names are strings and should be written with double quotes e.g.
+
+    from ehrql import claim_permissions
+
+    claim_permissions("some_permission", "another_permission")
+
+This can go anywhere in your dataset or measure definition file.
+
+You can make multiple `claim_permissions()` calls and the permissions will be
+combined together.
+</div>

docs/includes/generated_docs/schemas/raw.tpp.md

@@ -475,6 +475,12 @@ Date on which the current dataset was imported.
 <p class="dimension-indicator"><code>many rows per patient</code></p>
 ## isaric
 
+!!! warning "Access to this table requires the `isaric` permission"
+
+    Access to ISARIC data is usually agreed at the project application stage.  If
+    you're unsure as to whether you do or should have access please speak to your
+    co-pilot or to OpenSAFELY support.
+
 ISARIC is a dataset of COVID-19-related hospital admissions,
 with coverage across the majority of hospitals across the UK,
 including much richer clinical information
@@ -1465,6 +1471,12 @@ Medical condition mentioned on the death certificate.
 
 National Waiting List Clock Stops
 
+!!! warning "Access to this table requires the `waiting_list` permission"
+
+    Access to Waiting List data is usually agreed at the project application stage.
+    If you're unsure as to whether you do or should have access please speak to your
+    co-pilot or to OpenSAFELY support.
+
 The columns in this table have the same data types as the columns in [the associated
 database table][wl_clockstops_raw_1]. The three "pseudo" columns are small
 exceptions, as they are converted from binary columns to string columns.
@@ -1614,6 +1626,12 @@ exceptions, as they are converted from binary columns to string columns.
 
 National Waiting List Open Pathways
 
+!!! warning "Access to this table requires the `waiting_list` permission"
+
+    Access to Waiting List data is usually agreed at the project application stage.
+    If you're unsure as to whether you do or should have access please speak to your
+    co-pilot or to OpenSAFELY support.
+
 The columns in this table have the same data types as the columns in [the associated
 database table][wl_openpathways_raw_1]. The three "pseudo" columns are small
 exceptions, as they are converted from binary columns to string columns.

docs/includes/generated_docs/schemas/tpp.md

@@ -701,7 +701,8 @@ The date of discharge from a hospital provider spell.
 
 Appointments in primary care.
 
-!!! warning
+!!! warning "Access to this table requires the `appointments` permission"
+
     In TPP this data comes from the "Appointment" table. This table has not yet been
     well characterised, so there are some issues around how to interpret findings
     from it. The data contains records created when an appointment is made with a GP
@@ -710,8 +711,8 @@ Appointments in primary care.
     There are also duplicate events in the table that we need to better understand.
 
     As a consequence, if you try to use the appointment table, you will see warnings
-    when running your code locally, and failures when the GitHub action tests your
-    code. If you need access to the appointments data, please speak to your
+    when running your code locally, and failures if you try to run against real
+    data. If you need access to the appointments data, please speak to your
     OpenSAFELY co-pilot. We will be considering projects on a case by case basis
     until it can enter the normal stable pool of data.
 
@@ -2692,6 +2693,12 @@ The date the referral request was received by the health care provider.
 
 This table contains responses to questions from the OpenPROMPT project.
 
+!!! warning "Access to this table requires the `open_prompt` permission"
+
+    Access to OpenPROMPT data is usually agreed at the project application stage. If
+    you're unsure as to whether you do or should have access please speak to your
+    co-pilot or to OpenSAFELY support.
+
 You can find out more about this table in the associated short data report. To view
 it, you will need a login for [Level 4][open_prompt_1]. The
 [workspace][open_prompt_2] shows when the code that comprises the report was run;
@@ -3342,6 +3349,12 @@ used to do so.
 <p class="dimension-indicator"><code>many rows per patient</code></p>
 ## ukrr
 
+!!! warning "Access to this table requires the `ukrr` permission"
+
+    Access to UK Renal Registry data is usually agreed at the project application
+    stage. If you're unsure as to whether you do or should have access please speak
+    to your co-pilot or to OpenSAFELY support.
+
 The UK Renal Registry (UKRR) contains data on patients under secondary renal care
 (advanced chronic kidney disease stages 4 and 5, dialysis, and kidney transplantation)
 <div markdown="block" class="definition-list-wrapper">
@@ -3527,6 +3540,12 @@ Vaccine's product name.
 
 Waiting List Minimum Data Set Clock Stops
 
+!!! warning "Access to this table requires the `waiting_list` permission"
+
+    Access to Waiting List data is usually agreed at the project application stage.
+    If you're unsure as to whether you do or should have access please speak to your
+    co-pilot or to OpenSAFELY support.
+
 These data are from the patient-level [Waiting List Minimum Data Set (WLMDS)](https://www.england.nhs.uk/statistics/statistical-work-areas/rtt-waiting-times/wlmds/),
 which are reported separately from the aggregate [Referral to Treatment (RTT) data](https://www.england.nhs.uk/statistics/statistical-work-areas/rtt-waiting-times/).
 
@@ -3712,6 +3731,12 @@ The Sunday of the week that the pathway relates to
 
 Waiting List Minimum Data Set Open Pathways
 
+!!! warning "Access to this table requires the `waiting_list` permission"
+
+    Access to Waiting List data is usually agreed at the project application stage.
+    If you're unsure as to whether you do or should have access please speak to your
+    co-pilot or to OpenSAFELY support.
+
 These data are from the patient-level [Waiting List Minimum Data Set (WLMDS)](https://www.england.nhs.uk/statistics/statistical-work-areas/rtt-waiting-times/wlmds/),
 which are reported separately from the aggregate [Referral to Treatment (RTT) data](https://www.england.nhs.uk/statistics/statistical-work-areas/rtt-waiting-times/).
 

docs/reference/language.md

@@ -130,6 +130,8 @@ other) groupings.
 
 ---8<-- 'includes/generated_docs/language__measures.md'
 
+---
+
 
 ## Parameters
 
@@ -139,3 +141,42 @@ from the [command line](cli.md#generate-dataset.user_args).
 ---8<-- 'includes/generated_docs/language__parameters.md'
 
 ---
+
+
+## Permissions
+
+Some tables and features in ehrQL can only be accessed with specific
+permission, which must be enabled by the OpenSAFELY team. The exact
+process and criteria for granting these permissions depends on the
+details of the specific permission in question. But generally speaking
+this would something discussed at the application stage of a project.
+
+If there is a permission you think you should have but don't, you should
+raise this with your co-pilot or with OpenSAFELY support.
+
+
+### “Claiming” permissions
+
+When working with dummy data locally or inside Codespaces ehrQL does not
+know which permissions your project has and so it will warn if you
+attempt to use _any_ restricted table or feature. You can silence these
+warnings by using the `claim_permission()` function below to say which
+permissions your project has.
+
+The warning message generated by ehrQL will tell you exactly what
+permissions you need to claim and will generate the code you need to
+copy and paste into your dataset definition.
+
+You can claim any permission you need to allow you to continue your
+dummy data work, but you **won't be able to run your project against
+real data** unless you actually have those permissions enabled in the
+OpenSAFELY platform.
+
+The `claim_permissions()` function can be placed anywhere in your
+dataset or measure definition file, or in any of the Python files your
+definition imports.
+
+
+---8<-- 'includes/generated_docs/language__permissions.md'
+
+---

ehrql/__init__.py

@@ -3,6 +3,7 @@
 from ehrql.codes import codelist_from_csv
 from ehrql.debugger import show
 from ehrql.measures import INTERVAL, Measures, create_measures
+from ehrql.permissions import claim_permissions
 from ehrql.query_language import (
     Dataset,
     Error,
@@ -24,6 +25,7 @@
 
 
 __all__ = [
+    "claim_permissions",
     "codelist_from_csv",
     "INTERVAL",
     "Measures",

ehrql/__main__.py

@@ -19,6 +19,7 @@
     split_directory_and_extension,
 )
 from ehrql.loaders import DEFINITION_LOADERS, DefinitionError
+from ehrql.permissions import EHRQLPermissionError
 from ehrql.renderers import DISPLAY_RENDERERS
 from ehrql.utils.string_utils import strip_indent
 
@@ -123,11 +124,14 @@ def main(args, environ=None):
         # Errors from definition files are already pre-formatted so we just write them
         # directly to stderr and exit
         print(str(exc), file=sys.stderr)
-        sys.exit(1)
+        sys.exit(10)
     except FileValidationError as exc:
         # Handle errors encountered while reading user-supplied data
         print(f"{exc.__class__.__name__}: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(11)
+    except EHRQLPermissionError as exc:
+        print(f"{exc.__class__.__name__}: {exc}", file=sys.stderr)
+        sys.exit(12)
     except Exception as exc:
         # For functions which take a `backend_class` give that class the chance to set
         # the appropriate exit status for any errors

ehrql/docs/language.py

@@ -110,6 +110,9 @@ def build_language():
         "parameters": {
             "get_parameter": namespace["get_parameter"],
         },
+        "permissions": {
+            "claim_permissions": namespace["claim_permissions"],
+        },
     }
 
     # Check that the documentation is complete

ehrql/docs/schemas.py

@@ -66,22 +66,36 @@ def build_module_name_to_backend_map(backends):
 
 def build_tables(module):
     for table_name, table in get_tables_from_namespace(module):
-        cls = table.__class__
-        docstring = get_table_docstring(cls)
-        columns = [
-            build_column(table_name, column_name, series_or_property)
-            for column_name, series_or_property in get_all_series_and_properties_from_class(
-                cls
-            ).items()
-        ]
+        yield build_table(table_name, table)
+
+
+def build_table(table_name, table):
+    cls = table.__class__
+    docstring = get_table_docstring(cls)
+    required_permission = table._qm_node.required_permission
+    columns = [
+        build_column(table_name, column_name, series_or_property)
+        for column_name, series_or_property in get_all_series_and_properties_from_class(
+            cls
+        ).items()
+    ]
 
-        yield {
-            "name": table_name,
-            "docstring": docstring,
-            "columns": columns,
-            "has_one_row_per_patient": issubclass(cls, PatientFrame),
-            "methods": build_table_methods(table_name, cls),
-        }
+    if required_permission:
+        expected_string = f"`{required_permission}` permission"
+        if expected_string not in docstring:
+            raise ValueError(
+                f"Table {cls!r} requires the {required_permission!r} permission "
+                f"but doesn't include {expected_string!r} in its docstring"
+            )
+
+    return {
+        "name": table_name,
+        "docstring": docstring,
+        "columns": columns,
+        "has_one_row_per_patient": issubclass(cls, PatientFrame),
+        "methods": build_table_methods(table_name, cls),
+        "required_permission": required_permission,
+    }
 
 
 def build_column(table_name, column_name, series_or_property):

ehrql/loaders.py

@@ -9,6 +9,7 @@
 import ehrql
 from ehrql.debugger import activate_debug_context
 from ehrql.measures import Measures
+from ehrql.permissions import clear_claimed_permissions, get_claimed_permissions
 from ehrql.query_language import Dataset, modify_exception
 from ehrql.renderers import DISPLAY_RENDERERS
 from ehrql.serializer import deserialize
@@ -250,7 +251,7 @@ def isolation_report_for_function(run_function, cwd):
 def load_dataset_definition_unsafe(definition_file, user_args, **kwargs):
     module = load_module(definition_file, user_args)
     dataset = get_dataset_from_module(module)
-    return dataset, module.dataset.dummy_data_config
+    return dataset, module.dataset.dummy_data_config, module._claimed_permissions
 
 
 def load_test_definition_unsafe(definition_file, user_args, **kwargs):
@@ -298,9 +299,10 @@ def load_measure_definitions_unsafe(definition_file, user_args, **kwargs):
     if len(measures) == 0:
         raise DefinitionError("No measures defined")
     return (
-        list(measures),
+        measures._compile(),
         measures.dummy_data_config,
         measures.disclosure_control_config,
+        module._claimed_permissions,
     )
 
 
@@ -324,10 +326,9 @@ def load_definition_unsafe(
 
 
 def load_module(module_path, user_args=()):
-    """Load a module containing an ehrql dataset definition.
-
-    Renders the serialized query model as json to the callers stdout, and any
-    other output to stderr.
+    """
+    Load a Python module by its filesystem path and return it, with some custom
+    ehrQL-specific behaviour
     """
 
     # Taken from the official recipe for importing a module from a file path:
@@ -343,13 +344,17 @@ def load_module(module_path, user_args=()):
     # generally looks as it would had you run: `python script.py some args --here`
     original_sys_argv = sys.argv.copy()
     sys.argv = [str(module_path), *user_args]
-    # Force any user generated output (prints etc) to stderr so it doe not get
+    # Force any user generated output (prints etc) to stderr so it does not get
     # mixed up with anything we might want to output ourselves
     original_sys_stdout = sys.stdout
     sys.stdout = sys.stderr
+    # Reset any previously claimed permissions
+    clear_claimed_permissions()
 
     try:
         spec.loader.exec_module(module)
+        # Attach a tuple of any claimed permissions to the module namespace
+        module._claimed_permissions = get_claimed_permissions()
         return module
     except Exception as exc:
         # Give the query langauge the chance to modify or replace the exception