BotWorks 101. Why the Bot approach

Vision

Bots were developed to satisfy a number of important requirements for populating an analytical platform with a wide range of disparate enterprise data.

1. Flexibility - The Big Data space is overrun with a huge variety of both opensource and commercial tools for data ingestion, curation, wrangling, and analysis. We needed a framework that would allow us to plug-in these technologies into our overall processes where they were appropriate. We also needed the ability to fill in the gaps in functionality with our own development if required. So we needed a framework that could act as an integrater/orchestrator without requiring a large development effort.
2. Scalability: We needed a system that would provide easily controlled vertical and horizontal scalabilty. To do this we chose frame Bots as distributed independent execution units integrated solely through asynchronous messaging.
3. Message as the Medium: Critical to our architectural approach for Bots was the idea that business logic should be carried in the messages rather than in the Bot code itself. Bots will execute the business logic that is contained in the code, but the execution parameters and usage of the results from the execution are not determined by code in the Bot doing the execution. This has proved to be a difficult concept in training users.
4. Meta-programming: all Bot messages are actually generated from templates and are fully self-contained. Extending the framework becomes a process of creating message templates rather than writing Java code. Also, the data used when instantiating the messages (done by BaseBot) is also expected to be in a meta-data repository.
5. Fault Tolerance: Complex curation tasks are often fairly long in duration involving many steps. In the event that there is a failure in one of the steps, it is important that the overall process can be picked up by retrying the failed step and then continue.