v0.25.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Bugfix in comparison of sim-telarray configuration files: ignore system dependent metadata values. (#1873)
• Fix thread-safe singleton check in mongo_db. Add {"maxIdleTimeMS": 10000} to close idle connections after 10s. Add debugging tool for connections. (#1899)
• Fix simulation model production table upload: full updates tables need not to search through the model history. This led to the addition of deprecated parameters back into the production. (#1913)

New Features

• Add pulse shape table writer for light emission flasher simulations. (#1851)
• Add CORSIKA executable compiled with CURVED option to docker images (corsika-curved). (#1859)
• Improve array plotting with more custom configuration options. (#1860, #1862)
• Add tests for certain strings in the CORSIKA and sim_telarray log files. (#1882)
• Verify that the request and actually simulated number of events is identical for CORSIKA and sim_telarray simulations. (#1883)
• Add tests that certain strings do not appear in the CORSIKA and sim_telarray log file (see also PR #1882) (#1885)
• Added functionality to plot mirrors and implemented auto plotting of mirrors for the reports. (#1886)
• Add test to verify that number of CORSIKA events is consistent with requested number of events. (#1890)
• Add application for pulse shape parameter derivation and move existing functionality there. (#1894)
• Add possibility to run sim_telarray only using an existing CORSIKA file with simtools-simulate-prod. (#1896)

Maintenance

• Refactor simtools-validate-file-using-schema and improve unit testing of code. (#1857)
• Introducing a new psf optimizer class and improving exception handling. (#1864)
• Update model version to ensure reproducibility if given as MAJOR.MINOR. Revert testing to exact semver version. (#1870)
• Improve description of submit_array_layouts. Fix model parameter writing of instrument for site parameters. (#1872)
• Introducing a data class and capping maximum learning rate in PSF optimization modules. (#1875)
• Remove unused command line parameter data_directory from simtools-simulate-prod. (#1891)
• Remove unused command line parameter number_of_runs from simtools-simulate-prod. Simulate always a single run number. (#1893)
• Introduce consistent naming for pedestal runs and output files: always start with pedestal... (e.g., pedestals_dark). (#1904)
• Suppress mongoDB connection close debug message in unit test. (#1906)
• Documentation is now generated for each push. Deployment changed to either the release of a version, merge to main, or manual trigger. (#1907)
• Update ignore pattern for log file inspector to ignore warnings on getpwuid obtained in AIV environment. (#1912)

Simulation model

• Merge all flasher pulse shape parameters (shape, width, exp decay) into a single parameter flasher_pulse_shape. (#1895)

v0.24.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Bugfix in validation of generated sim_telarray configuration: missing assert on the outcome of the tests (tests always passed, even with differences). (#1856)
• Fixed array trigger file generation to properly combine non-hardstereo telescopes. Previously, each telescope type generated separate trigger lines, preventing software stereo triggers between different telescope types (e.g., SST+MST). Now hardstereo lines remain separate while non-hardstereo telescopes are combined into a single line, enabling cross-telescope-type triggering. (#1863)

New Features

• Add new simulation model parameter transit_time_random. (#1808)
• Implement version check for model parameters for any software type. (#1842)
• Add possibility to define new base models taking into account the correct history of models for simtools-maintain-simulation-model-add-production. (#1844)
• Add possibility to change model parameters from command line using the --overwrite_model_parameters command line parameter. (#1850)

Maintenance

• Change source of simulation model database in CI to 'main' branch of simulation model repository.
  Tests against the latest release versions for simtools release candidates (1.2.3-rc).
  Allow to configure simulation model repository branches for manually triggered workflows. (#1846)
• Review and update all XPASS unit tests. (#1855)

Simulation model

• Change default simulation models database version to 0.11.0. (#1852)

v0.23.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Bugfix in comparing model versions in integration tests to decide which tests should run.
  Bugfix in getting model parameter values with units for the case of value-unit pairs consisting of lists with None entries. (#1805)
• Fix a bug where hard stereo trigger was set every time the hard_stereo key existed in dictionary, rather than checking its value. Now it only sets hardstereo if the value is True. (#1813)
• Bugfix packing model files: model version was not taken into account correctly. (#1818)
• Bugfix in preparing sim_telarray command for shower simulations: power-law configuration (used to fill histograms) set at the wrong place. (#1825)
• Bugfix in writing sim_telarray configuration option: "-C show=all" should be last in command. (#1828)

New Features

• Add new CORSKIA / sim_telarray image build for:

  • CORSIKA 7.8010
  • Bernloehr package 1.70
  • sim_telarray 250903

  Improved downloading script and handling of QGSJet tables.

  (#1824)

• Fill telescope design types into sim_telarray metadata field for camera_config_variant and optics_config_variant. (#1826)

• Allow to highlight or grey out telescopes in layout plotting application. (#1829)

• Allow to provide integers in scientific notation in yaml configuration files. Currently only implemented for the --number_of_photons argument of calculate_incident_angles.py. (#1837)

Maintenance

• Add application control functionality for all simtools applications. (#1809)
• Adding --all_model_versions flag for calibration and software report generation. (#1815)
• Replace superlinter in linter CI. Add toml linter and shellcheck to precommit. (#1838)
• Update one function used in the tests to comply with version 2.x of pyeventio. (#1840)

v0.22.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix integration tests after introducing the usage of major.minor model versions for tests. (#1777)
• Use correct sign for flasher distance calculation with flasher z coordinate pointing towards camera. (#1784)

Documentation

• Improve documentation on DB upload. (#1775)

New Features

• Add software version range validation. Introduce software version for updated or new flasher simulation model parameters. (#1763)
• Implement simulation models production table and model parameter update using the simtools-maintain-simulation-model-add-production application. (#1765)
• Schema for info.yml files used in simulation models. (#1790)
• Add flat-fielding devices for all telescope types. (#1798)
• Add new application simtools-db-upload-model-repository for simplified upload of repository data to the database. (#1801)
• Add flasher exponential decay pulse shape parameter. (#1802)

Maintenance

• Linked parameter plots to parameter comparison reports. (#1661)
• Added CLI argument to flexibly change PSF containment percentage as needed. (#1766)
• Remove 'LATEST' options when setting the DB_SIMULATION_MODEL_VERSION configuration option (require explicit setting of the DB version). (#1785)
• Introduce a simpler naming for the database (CTAO-Simulation-Model instead of CTAO-Simulation-ModelParameters). (#1787)
• Remove use_plain_output option and generalize output direction generation. (#1792)
• Simplify generation of production image and remove outdated build-corsika-simtelarray target. (#1794)
• Increase robustness for unit and integration tests when cloning remote Git repositories. Add a 3x retry. (#1796)
• Split CORSIKA/sim_telarray and simtools images into two. (#1797)
• Change minimum python version requirement to 3.12 (following CTAO recommendation). (#1799)
• Remove 'instrument:type' and 'instrument:site' in model parameter schema files. (#1803)

v0.21.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix output file error when running application simtools-print-version. (#1774)
• Bugfix in configuring hard-stereo array triggers (hard_stereo should be hardstereo in array_trigger.dat). (#1780)

New Features

• Implemented gradient descent minimisation for psf parameter optimisation, KS statistic as a sanity check, and monte carlo error on the simulated psf. (#1717)
• Add functionality to work with delta-type production tables of the simulation model for patch updates. (#1758)
• Automatic retrieval of the latest patch version for the simulation model.
  Allow --model_version=6.0 being resolved to the newest version. (#1768)

v0.20.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix warning messages in simtel metadata reader to avoid the log-file testing routines to fail. (#1712)
• Developer image is using an old base image with CORSIKA and sim_telarray. Fix naming of containers used in developer image to ensure the most recent are used. (#1716)
• Introduce DB_SIMULATION_MODEL_VERSION to fix instability of environment when introducing new model parameters. (#1729)
• Fix reduce complexity issue raised in sonar in plot simtel events. (#1735)
• Fix bug in schema file for model parameters regarding schema version. Improved printing on error. (#1742)

New Features

• Add new application to calculate telescope and array trigger rates from reduced event lists. (#1685)
• Add application to calculate photon incident angles at the camera plane. (#1687)
• Add sim_telarry executable compiled with debug options to containers. Allows to generate additional output columns for e.g. sim_telarray ray tracing. (#1715)
• Add / update nsb_spectrum model parameter and calculate integrated NSB rate. (#1718)
• Add avx512f and sse4 vector optimization images for production images. (#1741)

Maintenance

• Remove calculate trigger rate application and associated code. Replaced by simtools-derive-trigger-rate (see PR #1685). (#1693)
• Improve robustness of image building and downloading autoconf. (#1722)
• Improvement of CI-workflow efficiency and documentation. (#1723)
• Improve handling of secrets in CI workflows. (#1728)
• Add new field production_table_name to production table schema. (#1756)
• Remove Codecov from CI (covered by SonarQube). (#1759)

v0.19.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix bug in reading list of array elements from file. Only layout names were used and the list was read not from file but from the model parameters DB. (#1658)
• Ensure that run numbers start at 1 for cases no offset if given (HT Condor job submission). (#1686)
• Fix array trigger writing for single telescope simulations. (#1690)
• Bugfix in shower core calculation in shower coordinates for reduced event data. (#1703)

New Features

• Add additional plots for the derivation of CORSIKA limits. (#1632)
• Add a warning to derived corsika limit if they are close to the limits of the simulations. (#1643)
• Add an application that merges derived corsika simulation limits for each available grid point. (#1649)
• Add curvature radius to testeff command. User for calculation of NSB rates per pixel. (#1651)
• Allow to run applications in container environments using simtools-run-application. Introduce separate module (simtools-runner) and improve support for workflow running. (#1657)
• Add application to update, modify, and compare production tables.
  Improve documentation for simulation repository maintenance. (#1660)
• Add application to simulate calibration events (pedestals, dark pedestals, flasher). (#1663)
• Add functionality to simtools-validate-camera-efficiency to write out model parameter JSON with nsb_pixel_rate. (#1665)
• Add functionality to compare derived model parameters with DB values to integration tests (examples for simtools-validate-camera-efficiency). (#1666)
• Add log file inspector for integration tests to fail tests when errors or runtime warnings are reported. (#1674)
• Add application to simulate flasher with the light emission package. (#1676)
• Add sim_telarray model files to output of each production run. This asserts reproducibility and simplifies debugging. (#1689)
• Add MongoDB compound indices to improve query efficiency. (#1691)
• Add possibility to set random seed for random focal length settings in simtools-derive-mirror-rnda. (#1702)

Maintenance

• Apply more consistent key-name convention with lower-case snake_make to application workflow configurations. (#1636)
• Clarify input data to derivation of CORSIKA limits with simtools-production-derive-corsika-limit. (#1641)
• Change metadata to lower-case snake make. More consistent usage of schema versions. (#1645)
• Add missing plotting configurations for tabular data. (#1647)
• Plots are now auto-generated alongside reports. (#1653)
• Refactoring of ASCII input io operations. (#1667)
• Add a generic writer for structured data for JSON and YAML format. (#1668)
• Simplification of legend_handlers class. (#1670)
• Change units of shower- and pointing directions in reduced event data from radians to degrees. (#1672)
• Generalization of reduced event-data histogram filling (moved from CORSIKA limits derivation modules to generic module). (#1675)
• Fix hadolint issue and using parameter expansion for replacement in Dockerfile. (#1680)
• Refactor PSF parameter optimization workflow and improve output. Added a flag to export best parameters as JSON files. (#1681)
• Revision of main repository readme file and removed duplications. (#1684)
• Remove DB access of unit test for the docs_read_parameters module. (#1688)
• Maintenance pass through integration test to remove duplicated tests and improve test efficiency. (#1694)
• Add new sonar configuration (URL) (#1695)
• Improve code quality by addressing SonarQube reliability issues. (#1697)
• Unit tests will now fail in the CI for warnings. (#1699)
• Fix some security hotspots reported by SonarQube. (#1700)
• Improve robustness of Linter CI by caching pre-commit installation. (#1704)

Simulation model

• Addition of FADC long-sum model parameters. (#1659)

v0.18.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Run documentation generation CI on release to obtain current version of docs. (#1607)
• Fix authentication errors in script to setup a local mongoDB. (#1610)

Documentation

• Reorganisation and improvement of the documentation in the GitHub pages. (#1611)

New Features

• Use CTAO telescope names when writing out telescope list in reduced event data tables. (#1616)
• Allow CORSIKA limits to use layouts defined in simulation models database. (#1619)
• Add application / API to plot tabulated model parameters using default plotting configurations defined in model parameters schemas. (#1628)
• Allow to use current prod6 simulations for reduced event data generation. (#1631)
• Add common array elements ID. Add list with triggered common telescope IDs to reduced event data tables. (#1637)

Maintenance

• Removed unused ‘critical’ log level. (#1604)
• Add JSON checker to pre-commit linter. (#1615)
• Update test files with proton simulations (simtel output) and add provenance information. (#1617)
• Improve linter CI to print file containing non-ascii characters. (#1620)
• Replace file lists tests/resources/simtel_output_files.txt used for testing by explicitly listing of files. (#1629)
• Add default plotting configuration for tabular data. Introduce consistent lower-cases for keys in plotting configuration schema. (#1635)

v0.17.0

Release 0.17.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix data type of 'dsum_threshold' parameter. (#1557)
• Add missing neutrons to eventio particle IDs. (#1584)
• Fix suffix for sim_telarray output files: no correctly simtel.zst (was .zst only) (#1588)

Documentation

• Change documentation theme to pydata and add version switcher. (#1560)
• Update documentation on how to run jobs using HT Condor. (#1563)

New Features

• Add pixel layout plotting. (#1541)
• Enable docker-image generation for vector-optimized CORSIKA 7.8. (#1550)
• Add the sequential option to simulations using multipipe. With this option the CORSIKA and sim_telarray instances run in sequential order as far as possible. This mode is useful particularly on the grid, where typically we request a single core per job. (#1561)
• Add reading and conversion of primary_id from sim_telarray output. (#1570)
• Update simtools-generate-simtel-event-data to use astropy tables. (#1573)
• Generalize table reading and writing. Add application to merge tables. (#1580)
• Write reduced event data file as part of the cleanup stage of the simtools-simulate-prod application. (#1588)
• Use patches for optimized CORSIKA from gitlab repository instead of cloud download. (#1589)
• New utility simtools-verify-simulation-model-production-tables to verify completeness of model parameter repository. (#1592)
• Add custom copilot instructions to support AI used by developers. (#1594)
• Add new application simtools-submit-array-layouts to submit new array-layouts definition and validate the telescopes listed. (#1596)
• Improve layout plotting; adding plotting of background layout. Add plotting of all layouts. (#1597)
• Improve printout for assertion error for testing. (#1598)
• Add production version and build options to sim_telarrary metadata. (#1601)

Maintenance

• Add unit tests to increase coverage in interpolation handler. (#1504)
• Docker file improvements: linter application; bugfix in extra flags for optimized CORSIKA compilation. (#1544)
• Include stdout and stderr in error message of integration test. (#1555)
• Add pytest-retry package for 'flaky test'. (#1558)
• Reduce number of log-messages (INFO) for uploaded model parameters to DB. (#1559)
• Replace the "run_number_start" argument with "run_number_offset" (clearer) and re-introduce the "run_number" argument to allow specifying a run number (to which the offset is added). (#1562)
• Use telescope position files for setup in illuminator application and set correct obs level. (#1566)
• Improve the names of the various simulation output files. They now follow the current convention in current productions, they contain a bit more information (e.g., the model version where it wasn't there before), and they are more consistent with each other. (#1568)
• Improved interface and interpolation routines to derive production configuration. This creates a pipeline for lookup table generation, production grid generation, creating a grid with limits, which is then further complemented by the production statistics per grid point. (#1569)
• Fix unit test to run with astropy 7.1 (and earlier versions) (see #1574). (#1575)
• Simplified Sonar setup by moving properties into GitHub workflow. (#1578)
• Introduce simplified Codecov configuration. (#1579)
• Copy the CORSIKA log in binary mode instead of text mode. (#1582)
• Protect against non-unicode characters in CORSIKA log file. (#1583)
• Remove obsolete code on submission systems in JobManager. (#1586)
• Change dummy trigger thresholds to easier-to-spot values. (#1591)
• Fix duplication of some sim_telarray test files in './tests/resources' and corresponding test fixtures. (#1593)

v0.16.0

Release 0.16.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Documentation

• Add a section on mocking to the developer documentation. (#1516)
• Improve model parameter description (schemas, versions) in documentation. (#1524)

New Features

• Add grid generation production application to generate the grid points for Corsika simulations, with energy thresholds, viewcone and radius interpolation. (#1482)
• Add support for multiple versions and the multipipe functionality of sim_telarray (#1496)
• Add validation metadata written by sim_telarray with array model parameters. (#1505)
• Improve robustness of sim_telarray configuration by adding an invalid (dummy) telescope configuration. (#1509)
• Adding functionality for producing simulation configuration reports. (#1510)
• Add random seeds from file for instrument configuration. (#1511)
• Adapt docker files and building workflows to CORSIKA 7.8000. (#1512)
• Add sim_telarray random seed for instrument configuration to metadata (and metadata tests). (#1515)
• Integration test with multipipe and support for packing all output files in grid productions. (#1520)
• Add versioning to model parameter schema files. Implement example for corsika_starting_grammage. (#1523)
• Added an application to produce calibration device reports. (#1530)
• Allow corsika_starting_grammage to be telescope-type dependent. (#1539)

Maintenance

• Separate TelescopeModel and SiteModel for improved readability in sim_telarray configuration writer. (#1489)
• Fix licence definition in pyproject.toml to follow PEP 639. (#1493)
• Improve consistency of spelling of sim_telarray. (#1494)
• Remove test condition in simulator light emission and mock configuration file access. (#1495)
• Refactoring of event scaling application and moving most of the functionality into a new module. (#1498)
• Renaming of event scaling methods to production statistics derivation methods. (#1502)
• Improved naming and docstrings in simulation runners. (#1519)
• Remove duplication of reading yaml/json files in simtools.utils.general. (#1522)
• Remove duplication in make_run_command in corsika_simtel_runner and simulator_array. (#1525)
• Collect pyeventio related functions in new module simtel_io_file_info. (#1531)
• Add new field model_parameter_schema_version to model parameter test files. (#1535)

Simulation model

• Change list of applicable array elements and class for pedestal_events model parameter. (#1490)