Conform beamline modules to use shared components and update dodal connect

docs/explanations/decisions/0006-devices-shared-between-endstations.md

@@ -8,20 +8,34 @@ Proposed
 
 ## Context
 
-Some beamlines have multiple endstations with shared hardware in the optics or experiment hutch, and could potentially be trying to control it at the same time. Any device in the common hutch should only be fully controlled by one endstation at a time - the one that is taking data - but still be readable from the other endstations.
+Some beamlines have multiple endstations with shared hardware in the optics or experiment hutch, and could potentially be trying to control it at the same time. Any device in the common hutch should only be fully controlled by one endstation at a time - the one that is taking data - but still be readable from the other endstations. Additionally, it is desirable to reduce device configuration as much as possible.
 
 ## Decision
 
-The current solution is to have a separate blueapi instance for the shared hutch in order to be able to control the access to all the devices defined there.
-For all hardware in the shared optics hutch, the architecture should follow this structure:
+Beamlines specific devices should be defined in `dodal/beamlines/iXX.py` and shared components between end stations should be defined in `dodal/beamlines/iXX_shared.py`. They should either have separate blueapi instances, or if not possible, a single blueapi instance which the base beamline module is imported with any additional beamline modules e.g `iXX_shared.py`.
+
+For all hardware in the shared optics hutch where access control is required, the architecture should follow this structure:
 
 - There is a base device in dodal that sends a REST call to the shared blueapi with plan and devices names, as well as the name of the endstation performing the call.
 - There are read-only versions of the shared devices in the endstation blueapi which inherit from the base device above and set up the request parameters.
 - The real settable devices are only defined in the shared blueapi and should never be called directly from a plan.
 - The shared blueapi instance also has an ``AccessControl`` device that reads the endstation in use for beamtime from a PV.
 - Every plan should then be wrapped in a decorator that reads the ``AccessControl`` device, check which endstation is making the request and only allows the plan to run if the two values match.
 
-
 :::{seealso}
 [Optics hutch implementation on I19](https://diamondlightsource.github.io/i19-bluesky/main/explanations/decisions/0004-optics-blueapi-architecture.html) for an example.
 :::
+
+## Dodal connect with shared endstations
+
+If you have beamline that is composed of multiple modules, dodal connect can be extended to include all shared components. This is done by extending the `_BEAMLINE_SHARED` configuration (found in `dodal.beamlines.__init__.py`) to join together beamline modules when running the dodal connect command. 
+
+```Python
+_BEAMLINE_SHARED = {
+    "i05": ["i05", "i05_shared"],
+    "i05_1": ["i05_1", "i05_shared"],
+    ...
+}
+```
+
+For example, when running `dodal connect i05` will do the device connections for `dodal.beamlines.i05` and `dodal.beamlines.i05_shared`.

src/dodal/beamlines/__init__.py

@@ -12,6 +12,7 @@
     "i05-1": "i05_1",
     "b07-1": "b07_1",
     "i09-1": "i09_1",
+    "i09-2": "i09_2",
     "i13-1": "i13_1",
     "i20-1": "i20_1",
     "i19-1": "i19_1",
@@ -24,6 +25,19 @@
     "t01": "adsim",
 }
 
+# Some beamlines have shared components between branch lines. This configuration is
+# used by dodal connect to know which beamlines are shared so that when running dodal
+# connect, it will connect your beamline + shared components.
+_BEAMLINE_SHARED = {
+    "i05": ["i05", "i05_shared"],
+    "i05_1": ["i05_1", "i05_shared"],
+    "b07": ["b07", "b07_shared"],
+    "b07_1": ["b07_1", "b07_shared"],
+    "i09": ["i09", "i09_1_shared", "i09_2_shared"],
+    "i09_1": ["i09_1", "i09_1_shared"],
+    "i09_2": ["i09_2", "i09_2_shared"],
+}
+
 
 def all_beamline_modules() -> Iterable[str]:
     """
@@ -94,3 +108,8 @@ def module_name_for_beamline(beamline: str) -> str:
     """
 
     return _BEAMLINE_NAME_OVERRIDES.get(beamline, beamline)
+
+
+def shared_beamline_modules(beamline: str) -> list[str]:
+    bl = module_name_for_beamline(beamline)
+    return [module_name_for_beamline(b) for b in _BEAMLINE_SHARED.get(bl, [bl])]

src/dodal/beamlines/b07.py

@@ -1,11 +1,11 @@
+from dodal.beamlines.b07_shared import pgm
 from dodal.common.beamlines.beamline_utils import (
     device_factory,
 )
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
-from dodal.devices.b07 import Grating, LensMode, PsuMode
+from dodal.devices.b07 import LensMode, PsuMode
 from dodal.devices.electron_analyser import SelectedSource
 from dodal.devices.electron_analyser.specs import SpecsAnalyserDriverIO
-from dodal.devices.pgm import PGM
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
 from dodal.utils import BeamlinePrefix, get_beamline_name
@@ -21,11 +21,6 @@ def synchrotron() -> Synchrotron:
     return Synchrotron()
 
 
-@device_factory()
-def pgm() -> PGM:
-    return PGM(prefix=f"{PREFIX.beamline_prefix}-OP-PGM-01:", grating=Grating)
-
-
 # Connect will work again after this work completed
 # https://jira.diamond.ac.uk/browse/B07-1104
 @device_factory()

src/dodal/beamlines/b07_1.py

@@ -1,14 +1,13 @@
+from dodal.beamlines.b07_shared import pgm
 from dodal.common.beamlines.beamline_utils import device_factory
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
 from dodal.devices.b07 import PsuMode
 from dodal.devices.b07_1 import (
     ChannelCutMonochromator,
-    Grating,
     LensMode,
 )
 from dodal.devices.electron_analyser import SelectedSource
 from dodal.devices.electron_analyser.specs import SpecsAnalyserDriverIO
-from dodal.devices.pgm import PGM
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
 from dodal.utils import BeamlinePrefix, get_beamline_name
@@ -24,18 +23,13 @@ def synchrotron() -> Synchrotron:
     return Synchrotron()
 
 
-@device_factory()
-def pgm() -> PGM:
-    return PGM(prefix=f"{PREFIX.beamline_prefix}-OP-PGM-01:", grating=Grating)
-
-
-# Connect will work again after this work completed
-# https://jira.diamond.ac.uk/browse/B07-1104
 @device_factory()
 def ccmc() -> ChannelCutMonochromator:
     return ChannelCutMonochromator(prefix=f"{PREFIX.beamline_prefix}-OP-CCM-01:")
 
 
+# Connect will work again after this work completed
+# https://jira.diamond.ac.uk/browse/B07-1104
 @device_factory()
 def analyser_driver() -> SpecsAnalyserDriverIO[LensMode, PsuMode]:
     return SpecsAnalyserDriverIO[LensMode, PsuMode](

src/dodal/beamlines/b07_shared.py

@@ -0,0 +1,18 @@
+from dodal.common.beamlines.beamline_utils import (
+    device_factory,
+)
+from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
+from dodal.devices.b07 import Grating
+from dodal.devices.pgm import PGM
+from dodal.log import set_beamline as set_log_beamline
+from dodal.utils import BeamlinePrefix, get_beamline_name
+
+BL = get_beamline_name("b07")
+PREFIX = BeamlinePrefix(BL, suffix="B")
+set_log_beamline(BL)
+set_utils_beamline(BL)
+
+
+@device_factory()
+def pgm() -> PGM:
+    return PGM(prefix=f"{PREFIX.beamline_prefix}-OP-PGM-01:", grating=Grating)

src/dodal/beamlines/i05.py

@@ -1,7 +1,5 @@
-from dodal.beamline_specific_utils.i05_shared import pgm as i05_pgm
 from dodal.common.beamlines.beamline_utils import device_factory
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
-from dodal.devices.pgm import PGM
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
 from dodal.utils import BeamlinePrefix, get_beamline_name
@@ -15,8 +13,3 @@
 @device_factory()
 def synchrotron() -> Synchrotron:
     return Synchrotron()
-
-
-@device_factory()
-def pgm() -> PGM:
-    return i05_pgm()

src/dodal/beamlines/i05_1.py

@@ -1,7 +1,5 @@
-from dodal.beamline_specific_utils.i05_shared import pgm as i05_pgm
 from dodal.common.beamlines.beamline_utils import device_factory
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
-from dodal.devices.pgm import PGM
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
 from dodal.utils import BeamlinePrefix, get_beamline_name
@@ -12,11 +10,6 @@
 set_utils_beamline(BL)
 
 
-@device_factory()
-def pgm() -> PGM:
-    return i05_pgm()
-
-
 @device_factory()
 def synchrotron() -> Synchrotron:
     return Synchrotron()

src/dodal/beamline_specific_utils/i05_shared.py → src/dodal/beamlines/i05_shared.py

@@ -1,9 +1,10 @@
 from dodal.common.beamlines.beamline_utils import device_factory
 from dodal.devices.i05.enums import Grating
 from dodal.devices.pgm import PGM
-from dodal.utils import BeamlinePrefix
+from dodal.utils import BeamlinePrefix, get_beamline_name
 
-PREFIX = BeamlinePrefix("i05", "I")
+BL = get_beamline_name("i05")
+PREFIX = BeamlinePrefix(BL, "I")
 
 
 @device_factory()

src/dodal/beamlines/i09.py

@@ -1,11 +1,14 @@
+from dodal.beamlines.i09_1_shared import dcm
+from dodal.beamlines.i09_2_shared import pgm
 from dodal.common.beamlines.beamline_utils import (
     device_factory,
 )
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
 from dodal.devices.electron_analyser import SelectedSource
 from dodal.devices.electron_analyser.vgscienta import VGScientaAnalyserDriverIO
-from dodal.devices.i09 import DCM, Grating, LensMode, PassEnergy, PsuMode
-from dodal.devices.pgm import PGM
+from dodal.devices.i09 import LensMode, PassEnergy, PsuMode
+
+# from dodal.devices.pgm import PGM
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
 from dodal.utils import BeamlinePrefix, get_beamline_name
@@ -21,21 +24,8 @@ def synchrotron() -> Synchrotron:
     return Synchrotron()
 
 
-@device_factory()
-def pgm() -> PGM:
-    return PGM(
-        prefix=f"{BeamlinePrefix(BL, suffix='J').beamline_prefix}-MO-PGM-01:",
-        grating=Grating,
-    )
-
-
-@device_factory()
-def dcm() -> DCM:
-    return DCM(prefix=f"{PREFIX.beamline_prefix}-MO-DCM-01:")
-
-
-# Connect will work again after this work completed
-# https://jira.diamond.ac.uk/browse/I09-651
+# # Connect will work again after this work completed
+# # https://jira.diamond.ac.uk/browse/I09-651
 @device_factory()
 def analyser_driver() -> VGScientaAnalyserDriverIO[LensMode, PsuMode, PassEnergy]:
     energy_sources = {

src/dodal/beamlines/i09_1.py

@@ -1,10 +1,10 @@
+from dodal.beamlines.i09_1_shared import dcm
 from dodal.common.beamlines.beamline_utils import (
     device_factory,
 )
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
 from dodal.devices.electron_analyser import SelectedSource
 from dodal.devices.electron_analyser.specs import SpecsAnalyserDriverIO
-from dodal.devices.i09.dcm import DCM
 from dodal.devices.i09_1 import LensMode, PsuMode
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
@@ -21,11 +21,6 @@ def synchrotron() -> Synchrotron:
     return Synchrotron()
 
 
-@device_factory()
-def dcm() -> DCM:
-    return DCM(prefix=f"{PREFIX.beamline_prefix}-MO-DCM-01:")
-
-
 # Connect will work again after this work completed
 # https://jira.diamond.ac.uk/browse/I09-651
 @device_factory()

src/dodal/beamlines/i09_1_shared.py

@@ -0,0 +1,13 @@
+from dodal.common.beamlines.beamline_utils import (
+    device_factory,
+)
+from dodal.devices.i09.dcm import DCM
+from dodal.utils import BeamlinePrefix, get_beamline_name
+
+BL = get_beamline_name("i09-1")
+PREFIX = BeamlinePrefix(BL, suffix="I")
+
+
+@device_factory()
+def dcm() -> DCM:
+    return DCM(prefix=f"{PREFIX.beamline_prefix}-MO-DCM-01:")

src/dodal/beamlines/i09_2.py

@@ -2,8 +2,6 @@
     device_factory,
 )
 from dodal.common.beamlines.beamline_utils import set_beamline as set_utils_beamline
-from dodal.devices.i09.enums import Grating
-from dodal.devices.pgm import PGM
 from dodal.devices.synchrotron import Synchrotron
 from dodal.log import set_beamline as set_log_beamline
 from dodal.utils import BeamlinePrefix, get_beamline_name
@@ -17,8 +15,3 @@
 @device_factory()
 def synchrotron() -> Synchrotron:
     return Synchrotron()
-
-
-@device_factory()
-def pgm() -> PGM:
-    return PGM(prefix=f"{PREFIX.beamline_prefix}-MO-PGM-01:", grating=Grating)

src/dodal/beamlines/i09_2_shared.py

@@ -0,0 +1,14 @@
+from dodal.common.beamlines.beamline_utils import (
+    device_factory,
+)
+from dodal.devices.i09.enums import Grating
+from dodal.devices.pgm import PGM
+from dodal.utils import BeamlinePrefix, get_beamline_name
+
+BL = get_beamline_name("i09")
+PREFIX = BeamlinePrefix(BL, suffix="J")
+
+
+@device_factory()
+def pgm() -> PGM:
+    return PGM(prefix=f"{PREFIX.beamline_prefix}-MO-PGM-01:", grating=Grating)

src/dodal/cli.py

@@ -7,7 +7,11 @@
 from ophyd_async.core import NotConnected, StaticPathProvider, UUIDFilenameProvider
 from ophyd_async.plan_stubs import ensure_connected
 
-from dodal.beamlines import all_beamline_names, module_name_for_beamline
+from dodal.beamlines import (
+    all_beamline_names,
+    module_name_for_beamline,
+    shared_beamline_modules,
+)
 from dodal.common.beamlines.beamline_utils import set_path_provider
 from dodal.utils import AnyDevice, filter_ophyd_devices, make_all_devices
 
@@ -43,7 +47,15 @@ def main(ctx: click.Context) -> None:
     "attempt any I/O. Useful as a a dry-run.",
     default=False,
 )
-def connect(beamline: str, all: bool, sim_backend: bool) -> None:
+@click.option(
+    "-m",
+    "--module-only",
+    is_flag=True,
+    help="If a beamline depends on a shared beamline module, test devices only within "
+    "the selected module.",
+    default=False,
+)
+def connect(beamline: str, all: bool, sim_backend: bool, module_only: bool) -> None:
     """Initialises a beamline module, connects to all devices, reports
     any connection issues."""
 
@@ -53,20 +65,28 @@ def connect(beamline: str, all: bool, sim_backend: bool) -> None:
     # it is not used in dodal connect
     _spoof_path_provider()
 
-    module_name = module_name_for_beamline(beamline)
-    full_module_path = f"dodal.beamlines.{module_name}"
-
     # We need to make a RunEngine to allow ophyd-async devices to connect.
     # See https://blueskyproject.io/ophyd-async/main/explanations/event-loop-choice.html
     RE = RunEngine(call_returns_result=True)
 
-    print(f"Attempting connection to {beamline} (using {full_module_path})")
+    exceptions = {}
+
+    if module_only:
+        beamline_modules = [module_name_for_beamline(beamline)]
+    else:
+        beamline_modules = shared_beamline_modules(beamline)
+
+    full_module_paths = [
+        f"dodal.beamlines.{bl_module}" for bl_module in beamline_modules
+    ]
+    print(f"Attempting connection to {beamline} (using {full_module_paths})")
+    print(shared_beamline_modules)
 
     # Force all devices to be lazy (don't connect to PVs on instantiation) and do
     # connection as an extra step, because the alternatives is handling the fact
     # that only some devices may be lazy.
     devices, instance_exceptions = make_all_devices(
-        full_module_path,
+        full_module_paths,
         include_skipped=all,
         fake_with_ophyd_sim=sim_backend,
         wait_for_connection=False,
@@ -77,8 +97,13 @@ def connect(beamline: str, all: bool, sim_backend: bool) -> None:
     _report_successful_devices(devices, sim_backend)
 
     # If exceptions have occurred, this will print details of the relevant PVs
-    exceptions = {**instance_exceptions, **connect_exceptions}
+    e = {**instance_exceptions, **connect_exceptions}
+    exceptions = exceptions | e
+
+    print("Finished all device connections.")
     if len(exceptions) > 0:
+        print("=" * 100)
+        print("Had the following errors:")
         raise NotConnected(exceptions)