Improve error handling for async tasks (#70)

* Update gitignore

* Add role to collect async errors

* Update dispatch task to display errors

* Update applications role to use new async handling

* Update bulk_host_create role to use new async handling

* Update credential_input_sources role to use new async handling

* Update credential_types role to use new async handling

* Update credentials role to use new async handling

* Update execution_environments role to use new async handling

* Update groups role to use new async handling

* Update hosts role to use new async handling

* Update instance_groups role to use new async handling

* Update instances role to use new async handling

* Update inventories role to use new async handling

* Update inventory_sources role to use new async handling

* Update job_templates role to use new async handling

* Update inventory_source_update role to use new async handling

* Update labels role to use new async handling

* Update notification_templates role to use new async handling

* Update organizations role to use new async handling

* Update projects role to use new async handling

* Update project_update role to use new async handling

* Update roles role to use new async handling

* Update schedules role to use new async handling

* Update teams role to use new async handling

* Update users role to use new async handling

* Update workflow_job_templates role to use new async handling

* Update readme to include new async handling details

* Update metadata and add changelog fragment

* Fix link

* Fix pre-commit errors

* Update settings role to use new async handling

Author: branic

.gitignore

@@ -7,3 +7,6 @@ tests/output
 .vscode
 ansible.cfg
 .ansible/
+.devcontainer/
+.venv
+tests/localtest.yml

README.md

@@ -141,6 +141,58 @@ Controller token module would be invoked with this code:
 
 ```
 
+### Error Handling
+
+Many of the roles in this collection use asynchrous tasks to perform their
+actions. By default the first failed asyncronous task will cause the playbook to
+fail. Setting the `controller_configuration_collect_logs` variable to `true`
+will enable collecting all asyncronous task failure messages and allow the
+playbook to run to completion.
+
+When `controller_configuration_collect_logs` is enabled the reported errors are
+collected in a variable called `controller_configuration_role_errors`. This
+variable is a dictionary where each key is the type of the configuration item
+that failed to be applied. The value of each key is a list of all the failures
+of that type.
+
+When the `dispatch` role is used and `controller_configuration_collect_logs` is
+enabled it will display any errors encountered while applying the configurations
+and fail.
+
+Example Output when using the `dispatch` role and encoutering failures:
+
+```yaml
+fatal: [localhost]: FAILED! => {
+    "msg": [
+        "Errors encountered applying configurations:",
+        {
+            "aap_organizations_errors": [
+                {
+                    "ERROR_MESSAGE": "Request to /api/controller/v2/instance_groups/?name=not-real returned 0 items, expected 1",
+                    "name": "Test Organization"
+                },
+                {
+                    "ERROR_MESSAGE": "Request to /api/controller/v2/instance_groups/?name=not-real returned 0 items, expected 1",
+                    "name": "Test Organization 2"
+                }
+            ],
+            "controller_applications_errors": [
+                {
+                    "ERROR_MESSAGE": "Request to /api/controller/v2/organizations/?name=UnknownOrg returned 0 items, expected 1",
+                    "name": "controller_application-failed-app1",
+                    "organization": "UnknownOrg"
+                },
+                {
+                    "ERROR_MESSAGE": "value of authorization_grant_type must be one of: password, authorization-code, got: password2",
+                    "name": "controller_application-failed-app2",
+                    "organization": "Default"
+                }
+            ],
+        }
+    ]
+}
+```
+
 ### Automate the Automation
 
 Every Ansible Controller instance has it's own particularities and needs. Every administrator team has it's own practices and customs. This collection allows adaptation to every need, from small to large scale, having the objects distributed across multiple environments and leveraging Automation Webhook that can be used to link a Git repository and Ansible automation natively.

changelogs/fragments/collecting_async_status.yml

@@ -0,0 +1,2 @@
+minor_changes:
+  - Add the ability to collect error logs while configuring the object, instead of failing on the first error.

galaxy.yml

@@ -13,6 +13,7 @@ authors:
   - Ivan Aragonés
   - Silvio Perez
   - Adonis García
+  - Brant Evans @branic
 repository: https://github.com/redhat-cop/infra.controller_configuration/
 issues: https://github.com/redhat-cop/infra.controller_configuration/issues
 build_ignore:

roles/applications/README.md

@@ -23,6 +23,7 @@ Currently:
 |`controller_password`|""|no|Controller Admin User's password on the Ansible Controller Server. This should be stored in an Ansible Vault at vars/controller-secrets.yml or elsewhere and called from a parent playbook. Either username / password or oauthtoken need to be specified.||
 |`controller_oauthtoken`|""|no|Controller Admin User's token on the Ansible Controller Server. This should be stored in an Ansible Vault at or elsewhere and called from a parent playbook. Either username / password or oauthtoken need to be specified.||
 |`controller_request_timeout`|`10`|no|Specify the timeout in seconds Ansible should use in requests to the controller host.||
+|`controller_configuration_collect_logs`|`false`|no|Specify whether to collect async results and continue for all failed async tasks instead of failing on the first error. Collected results are available in the `controller_configuration_role_errors` variable.||
 |`controller_applications`|`see below`|yes|Data structure describing your applications, described below. Alias: applications ||
 
 ### Enforcing defaults

roles/applications/tasks/main.yml

@@ -1,59 +1,76 @@
 ---
 # Create Controller applications
-- name: Managing Controller Applications
-  application:
-    name: "{{ __application_item.name | mandatory }}"
-    new_name: "{{ __application_item.new_name | default(omit, true) }}"
-    organization: "{{ __application_item.organization | mandatory }}"
-    description: "{{ __application_item.description | default(('' if controller_configuration_applications_enforce_defaults else omit), true) }}"
-    authorization_grant_type: "{{ __application_item.authorization_grant_type | default('password') }}"
-    client_type: "{{ __application_item.client_type | default('public') }}"
-    redirect_uris: "{{ __application_item.redirect_uris | default([]) }}"
-    skip_authorization: "{{ __application_item.skip_authorization | default((false if controller_configuration_applications_enforce_defaults else omit), true) }}"
-    state: "{{ __application_item.state | default(controller_state | default('present')) }}"
-
-    # Role specific options
-    controller_username: "{{ controller_username | default(omit, true) }}"
-    controller_password: "{{ controller_password | default(omit, true) }}"
-    controller_oauthtoken: "{{ controller_oauthtoken | default(omit, true) }}"
-    request_timeout: "{{ controller_request_timeout | default(omit, true) }}"
-    controller_host: "{{ controller_hostname | default(omit, true) }}"
-    controller_config_file: "{{ controller_config_file | default(omit, true) }}"
-    validate_certs: "{{ controller_validate_certs | default(omit) }}"
-  loop: "{{ applications if applications is defined else controller_applications }}"
-  loop_control:
-    loop_var: __application_item
-    label: "{{ __operation.verb }} Controller Application {{ __application_item.name }}"
-    pause: "{{ controller_configuration_applications_loop_delay }}"
-  no_log: "{{ controller_configuration_applications_secure_logging }}"
-  async: "{{ ansible_check_mode | ternary(0, 1000) }}"
-  poll: 0
-  register: __applications_job_async
-  changed_when: (__applications_job_async.changed if ansible_check_mode else false)
+- name: Manage Controller Applications Block
   vars:
-    __operation: "{{ operation_translate[__application_item.state | default(controller_state) | default('present')] }}"
     ansible_async_dir: "{{ controller_configuration_async_dir }}"
+  no_log: "{{ controller_configuration_applications_secure_logging }}"
+  block:
+    - name: Managing Controller Applications
+      application:
+        name: "{{ __application_item.name | mandatory }}"
+        new_name: "{{ __application_item.new_name | default(omit, true) }}"
+        organization: "{{ __application_item.organization | mandatory }}"
+        description: "{{ __application_item.description | default(('' if controller_configuration_applications_enforce_defaults else omit), true) }}"
+        authorization_grant_type: "{{ __application_item.authorization_grant_type | default('password') }}"
+        client_type: "{{ __application_item.client_type | default('public') }}"
+        redirect_uris: "{{ __application_item.redirect_uris | default([]) }}"
+        skip_authorization: "{{ __application_item.skip_authorization | default((false if controller_configuration_applications_enforce_defaults else omit), true) }}"
+        state: "{{ __application_item.state | default(controller_state | default('present')) }}"
 
-- name: Flag for errors (check mode only)
-  ansible.builtin.set_fact:
-    error_flag: true
-  when: ansible_check_mode and __applications_job_async.failed is defined and __applications_job_async.failed
+        # Role specific options
+        controller_username: "{{ controller_username | default(omit, true) }}"
+        controller_password: "{{ controller_password | default(omit, true) }}"
+        controller_oauthtoken: "{{ controller_oauthtoken | default(omit, true) }}"
+        request_timeout: "{{ controller_request_timeout | default(omit, true) }}"
+        controller_host: "{{ controller_hostname | default(omit, true) }}"
+        controller_config_file: "{{ controller_config_file | default(omit, true) }}"
+        validate_certs: "{{ controller_validate_certs | default(omit) }}"
+      loop: "{{ applications if applications is defined else controller_applications }}"
+      loop_control:
+        loop_var: __application_item
+        label: "{{ __operation.verb }} Controller Application {{ __application_item.name }}"
+        pause: "{{ controller_configuration_applications_loop_delay }}"
+      async: "{{ ansible_check_mode | ternary(0, 1000) }}"
+      poll: 0
+      register: __applications_job_async
+      changed_when: (__applications_job_async.changed if ansible_check_mode else false)
+      vars:
+        __operation: "{{ operation_translate[__application_item.state | default(controller_state) | default('present')] }}"
 
-- name: Managing Controller Applications | Wait for finish the Application management
-  ansible.builtin.async_status:
-    jid: "{{ __applications_job_async_results_item.ansible_job_id }}"
-  register: __applications_job_async_result
-  until: __applications_job_async_result.finished
-  retries: "{{ controller_configuration_applications_async_retries }}"
-  delay: "{{ controller_configuration_applications_async_delay }}"
-  loop: "{{ __applications_job_async.results }}"
-  loop_control:
-    loop_var: __applications_job_async_results_item
-    label: "{{ __operation.verb }} Controller Application {{ __applications_job_async_results_item.__application_item.name }} | Wait for finish the Application {{
-      __operation.action }}"
-  when: not ansible_check_mode and __applications_job_async_results_item.ansible_job_id is defined
-  no_log: "{{ controller_configuration_applications_secure_logging }}"
-  vars:
-    __operation: "{{ operation_translate[__applications_job_async_results_item.__application_item.state | default(controller_state) | default('present')] }}"
-    ansible_async_dir: "{{ controller_configuration_async_dir }}"
+    - name: Flag for errors (check mode only)
+      ansible.builtin.set_fact:
+        error_flag: true
+      when: ansible_check_mode and __applications_job_async.failed is defined and __applications_job_async.failed
+
+    - name: Managing Controller Applications | Wait for finish the Application management
+      when:
+        - not ansible_check_mode
+        - __applications_job_async_results_item.ansible_job_id is defined
+      ansible.builtin.include_role:
+        name: infra.controller_configuration.collect_async_status
+      loop: "{{ __applications_job_async.results }}"
+      loop_control:
+        loop_var: __applications_job_async_results_item
+        label: "{{ __operation.verb }} Controller Application {{ __applications_job_async_results_item.__application_item.name }} | Wait for finish the Application {{
+          __operation.action }}"
+      vars:
+        cas_secure_logging: "{{ controller_configuration_applications_secure_logging }}"
+        cas_async_delay: "{{ controller_configuration_applications_async_delay }}"
+        cas_async_retries: "{{ controller_configuration_applications_async_retries }}"
+        cas_job_async_results_item: "{{ __applications_job_async_results_item }}"
+        cas_error_list_var_name: "controller_applications_errors"
+        __operation: "{{ operation_translate[__applications_job_async_results_item.__application_item.state | default(controller_state) | default('present')] }}"
+
+  always:
+    - name: Cleanup async results files
+      when:
+        - not ansible_check_mode
+        - __applications_job_async_results_item.ansible_job_id is defined
+      ansible.builtin.async_status:
+        jid: "{{ __applications_job_async_results_item.ansible_job_id }}"
+        mode: cleanup
+      loop: "{{ __applications_job_async.results }}"
+      loop_control:
+        loop_var: __applications_job_async_results_item
+        label: "Cleaning up job results file: {{ __applications_job_async_results_item.results_file | default('Unknown') }}"
 ...

roles/bulk_host_create/README.md

@@ -22,6 +22,7 @@ Currently:
 |`controller_password`|""|no|Controller Admin User's password on the Ansible Controller Server. This should be stored in an Ansible Vault at vars/controller-secrets.yml or elsewhere and called from a parent playbook. Either username / password or oauthtoken need to be specified.||
 |`controller_oauthtoken`|""|no|Controller Admin User's token on the Ansible Controller Server. This should be stored in an Ansible Vault at or elsewhere and called from a parent playbook. Either username / password or oauthtoken need to be specified.||
 |`controller_request_timeout`|`10`|no|Specify the timeout in seconds Ansible should use in requests to the controller host.||
+|`controller_configuration_collect_logs`|`false`|no|Specify whether to collect async results and continue for all failed async tasks instead of failing on the first error. Collected results are available in the `controller_configuration_role_errors` variable.||
 |`controller_configuration_bulk_hosts_secure_logging`|`see below`|yes|Data structure describing your organization or organizations Described below.||
 
 ### Secure Logging Variables

roles/bulk_host_create/tasks/main.yml

@@ -1,47 +1,63 @@
 ---
 # Create Job Template
-- name: Add Controller hosts in bulk
-  bulk_host_create:
-    hosts: "{{ __controller_bulk_hosts_item.hosts }}"
-    inventory: "{{ __controller_bulk_hosts_item.inventory }}"
-
-    # Role Standard options
-    controller_username: "{{ controller_username | default(omit, true) }}"
-    controller_password: "{{ controller_password | default(omit, true) }}"
-    controller_oauthtoken: "{{ controller_oauthtoken | default(omit, true) }}"
-    request_timeout: "{{ controller_request_timeout | default(omit, true) }}"
-    controller_host: "{{ controller_hostname | default(omit, true) }}"
-    controller_config_file: "{{ controller_config_file | default(omit, true) }}"
-    validate_certs: "{{ controller_validate_certs | default(omit) }}"
-  loop: "{{ controller_bulk_hosts }}"
-  loop_control:
-    loop_var: __controller_bulk_hosts_item
-    pause: "{{ controller_configuration_bulk_hosts_loop_delay }}"
-  no_log: "{{ controller_configuration_bulk_hosts_secure_logging }}"
-  async: "{{ ansible_check_mode | ternary(0, 1000) }}"
-  poll: 0
-  register: __controller_bulk_hosts_job_async
-  changed_when: (__controller_bulk_hosts_job_async.changed if ansible_check_mode else false)
+- name: Manage Controller Applications Block
   vars:
     ansible_async_dir: "{{ controller_configuration_async_dir }}"
+  no_log: "{{ controller_configuration_bulk_hosts_secure_logging }}"
+  block:
+    - name: Add Controller hosts in bulk
+      bulk_host_create:
+        hosts: "{{ __controller_bulk_hosts_item.hosts }}"
+        inventory: "{{ __controller_bulk_hosts_item.inventory }}"
 
-- name: Flag for errors (check mode only)
-  ansible.builtin.set_fact:
-    error_flag: true
-  when: ansible_check_mode and __controller_bulk_hosts_job_async.failed is defined and __controller_bulk_hosts_job_async.failed
+        # Role Standard options
+        controller_username: "{{ controller_username | default(omit, true) }}"
+        controller_password: "{{ controller_password | default(omit, true) }}"
+        controller_oauthtoken: "{{ controller_oauthtoken | default(omit, true) }}"
+        request_timeout: "{{ controller_request_timeout | default(omit, true) }}"
+        controller_host: "{{ controller_hostname | default(omit, true) }}"
+        controller_config_file: "{{ controller_config_file | default(omit, true) }}"
+        validate_certs: "{{ controller_validate_certs | default(omit) }}"
+      loop: "{{ controller_bulk_hosts }}"
+      loop_control:
+        loop_var: __controller_bulk_hosts_item
+        pause: "{{ controller_configuration_bulk_hosts_loop_delay }}"
+      async: "{{ ansible_check_mode | ternary(0, 1000) }}"
+      poll: 0
+      register: __controller_bulk_hosts_job_async
+      changed_when: (__controller_bulk_hosts_job_async.changed if ansible_check_mode else false)
 
-- name: Configure bulk_hosts | Wait for finish the bulk_hosts creation
-  ansible.builtin.async_status:
-    jid: "{{ __controller_bulk_hosts_job_async_results_item.ansible_job_id }}"
-  register: __controller_bulk_hosts_job_async_result
-  until: __controller_bulk_hosts_job_async_result.finished
-  retries: "{{ controller_configuration_bulk_hosts_async_retries }}"
-  delay: "{{ controller_configuration_bulk_hosts_async_delay }}"
-  loop: "{{ __controller_bulk_hosts_job_async.results }}"
-  loop_control:
-    loop_var: __controller_bulk_hosts_job_async_results_item
-  when: not ansible_check_mode and __controller_bulk_hosts_job_async_results_item.ansible_job_id is defined
-  no_log: "{{ controller_configuration_bulk_hosts_secure_logging }}"
-  vars:
-    ansible_async_dir: "{{ controller_configuration_async_dir }}"
+    - name: Flag for errors (check mode only)
+      ansible.builtin.set_fact:
+        error_flag: true
+      when: ansible_check_mode and __controller_bulk_hosts_job_async.failed is defined and __controller_bulk_hosts_job_async.failed
+
+    - name: Configure bulk_hosts | Wait for finish the bulk_hosts creation
+      when:
+        - not ansible_check_mode
+        - __controller_bulk_hosts_job_async_results_item.ansible_job_id is defined
+      ansible.builtin.include_role:
+        name: infra.controller_configuration.collect_async_status
+      loop: "{{ __controller_bulk_hosts_job_async.results }}"
+      loop_control:
+        loop_var: __controller_bulk_hosts_job_async_results_item
+      vars:
+        cas_secure_logging: "{{ controller_configuration_bulk_hosts_secure_logging }}"
+        cas_async_delay: "{{ controller_configuration_bulk_hosts_async_delay }}"
+        cas_async_retries: "{{ controller_configuration_bulk_hosts_async_retries }}"
+        cas_job_async_results_item: "{{ __controller_bulk_hosts_job_async_results_item }}"
+        cas_error_list_var_name: "controller_bulk_hosts_errors"
+
+  always:
+    - name: Cleanup async results files
+      when:
+        - not ansible_check_mode
+        - __controller_bulk_hosts_job_async_results_item.ansible_job_id is defined
+      ansible.builtin.async_status:
+        jid: "{{ __controller_bulk_hosts_job_async_results_item.ansible_job_id }}"
+        mode: cleanup
+      loop: "{{ __controller_bulk_hosts_job_async.results }}"
+      loop_control:
+        loop_var: __controller_bulk_hosts_job_async_results_item
+        label: "Cleaning up job results file: {{ __controller_bulk_hosts_job_async_results_item.results_file | default('Unknown') }}"
 ...