Brainstorming the ideal monthly hours workflow

[thekaveman]

Background

Every month, Compiler exports data from its internal time tracking system (Toggl) for a particular client, and converts this data into a format necessary for import into their time tracking system (Harvest).

The code for this conversion can be found in compiler_admin/services/toggl.py

A helper script for the monthly conversion process was recently added at bin/hours-convert.sh. The helper script wraps a call to the compiler-admin time convert command with some checks and arguments.

The full process can be described as follows:

1. Compiler staff logs into Toggl as an Admin of that system
2. Prior month is locked for editing in Workspace settings
3. Navigate to Toggl Reports, customize filters for date range, client, projects, billable/non-billable, etc.
4. Generate report, manually inspect for data quality
5. Download report as CSV file to local computer
6. Upload Toggl CSV to Compiler's Google Drive
7. In the staff's local development environment, update the .env file with the path to the new Toggl data file, path to a new output Harvest file, and other settings
8. Open the devcontainer for this repository in VS Code
9. Confirm Project mappings between the two systems are up to date
10. Run the helper script mentioned above to convert Toggl CSV to Harvest CSV
11. Use the summarize-harvest Notebook to get some quick QA checks on the converted Harvest CSV
12. Draft and send a message in Slack including:
    • Total hours for the month
    • Total rows in the CSV file
    • The CSV file as an attachment
13. Wait to hear if the import into Harvest succeeded
14. Correct any data issues, re-run the (partial) process if needed until the imported data matches what we expect from Toggl
15. Upload the final Harvest CSV to Compiler's Google Drive

Areas for improvement

This process is time consuming and error prone in many different ways. Much of it is largely opaque to everyone in Compiler except for those that routinely go through the steps outlined above.

The following are areas for improvement around this process.

Toggl report generation

This is one of the areas that is most prone to user-error. Even with features like saved reports in Toggl, it is easy to make a mistake on the date range or another basic report config, and sometimes these mistakes go unnoticed until much later in the process.

• Report generation should be standardized and codified. Month to month, there should be zero variation in how reports are generated, aside from the date range. The best way to achieve this is to generate the reports via code. More information: https://engineering.toggl.com/docs/reports_start

Data quality checks

Another area that would benefit from a standardized approach, applied the same way every month. Some issues have recurred for a number of months before being addressed with new/additional manual checks. There is always the possibility of new issues coming up too, based on changes in one or both systems. There are multiple phases at which data can and should be QA'd with some basic checks.

• Toggl reports should be validated for correctness before the conversion process.
• Harvest reports should be validated for correctness after the conversion process.
• Any data quality failure should stop the entire process.

Ideal workflow

In an ideal world, this entire workflow is run out of this repository via GitHub Actions.

Assumptions

• The process needs to remain largely the same for now: Toggl is Compiler's system of record, Compiler is working with a client that uses Harvest as their system of record and requires our time to be entered there
• We can download CSV reports from Toggl
• We can request Toggl reports filtered by date range, clients, projects, billable/non-billable, etc.
• We can convert Toggl CSV to Harvest CSV
• We can post messages to Slack via GitHub Actions

Process

1. Compiler staff logs into Toggl as an Admin of that system
2. Prior month is locked for editing in Workspace settings
3. Navigate to this repository's Actions tab
4. Initiate a manual run of workflow that does the following (via compiler-admin commands):
   1. Prior month's report downloaded from Toggl
   2. QA checks run on Toggl report
      • Failures stop the run
   3. Toggl report uploaded to Google Drive
   4. Toggl report converted to Harvest report
   5. QA checks run on Harvest report
      • Failures stop the run
   6. Harvest report uploaded to Google Drive
   7. (Optionally) Slack message sent to given channel containing:
      • Total hours for the month
      • Total rows in the CSV file
      • The CSV file as an attachment
5. Wait to hear if the import into Harvest succeeded
6. Correct any data issues, manually re-run the (partial) process if needed until the imported data matches what we expect from Toggl
7. Upload the final Harvest CSV to Compiler's Google Drive if needed