Build pipeline on AirFlow + Ansible

[abitrolly]

• Setup 10 Ubuntu 18.04 machines with Ansible and Airflow

• 1 controller

• 9 pilots

• Access UI on controller
• Create test job on controller
• Run test job on 9 pilots and collect result
• Randomly reboot pilots to check job recovery
• Modify jobs to fetch heavyweight data file from storage
• Share new job to running cluster (should it stop old ones?)

[abitrolly]

https://gitlab.com/abitrolly/airflow-net

+1

[abitrolly]

Installed Airflow through Ansible without GPL dependencies.

+3

[abitrolly]

+2 systemd template to autostart airflow webserver on reboot.

LXD port forwarding, finding a way to pass files between Airflow tasks.

[abitrolly]

+2 reading airflow docs and experiments.

[abitrolly]

Ideas for the future. Calculating the time saved. Implement an easy way to register the time that is saved by automation, as a dividend of added benefits by automation, which can be used to fix or improve the automation.

[abitrolly]

+2 PostgreSQL

[abitrolly]

+2 (failed) troubleshooting Scaleway sudo root is not in the sudoers file after bootstrap with Ansible
+1 checkout and generate .env
+2 (failed) Ansible port forwarding (will upload private SSH keys instead)

[abitrolly]

+1 sudoers fixed with mv /etc/sudoers.dpkg-dist /etc/sudoers
+1 git clone with access key and host trust

[abitrolly]

+1 RabbitMQ probe

#!/usr/bin/env python
import pika

USERPASS = "guest guest"

creds = pika.PlainCredentials(*USERPASS.split())
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost',
        credentials=creds))
channel = connection.channel()

[abitrolly]

+1 move DB to different host

[abitrolly]

+1 move RabbitMQ and Redis to a separate host