Api Crawler

Standard Version

For the Docker implementation go to https://github.com/thewires2/Api_Crawler/tree/docker

1) Steps to run the code

1. Download the code as a zip file and extract the folder.

2. Create a virtual enviornment using pipenv or virtualenv and run the command $ pip install -r requirements.txt in the root directory to install all the required dependencies

Example:

   cd Api_Crawler-master
   virtualenv venv
   source venv/bin/activate
   pip install -r requirements.txt

3. Run the following commands

   python manage.py makemigrations
   python manage.py migrate
   python manage.py runserver 8080

4. Goto http://127.0.0.1:8080/ to see the list of categories Api's and http://127.0.0.1:8080/category to see all the Api's

Since the API is rate limited to 10 requests per user per minute it takes around 10 minutes to populate the entire database the first time you visit the Category section

---

2) Details of all the tables and their schema

There are two tables in this project:

1) Categories

This is a simple table which just one column storing all the 45 categories of different API's

SCHEMA:

Schema

STEPS TO RECREATE THE CATEGORY TABLE:

Since Django autogenerates the migration's for the creation of the tables, there in no explicit creation of the tables. The Database and tables are created automatically when the project is run for the first time. The following sql is generated by the django migration for the Category table

CREATE TABLE "crawler_category" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(100) NOT NULL)

2) Sub-Categories

This is a table which is used for storing the different kind's of API's with the detail's supplied through the API's calls

SCHEMA:

Schema

STEPS TO RECREATE THE SUB-CATEGORY TABLE:

Since Django autogenerates the migration's for the creation of the tables, there in no explicit creation of the tables. The Database and tables are created automatically when the project is run for the first time. The following sql is generated by the django migration for the subcategory table

CREATE TABLE "crawler_subcategory" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "Api" varchar(500) NOT NULL, "Description" varchar(500) NOT NULL, "Auth" varchar(500) NOT NULL, "Link" varchar(100) NOT NULL, "Category" varchar(100) NOT NULL, "Cors" varchar(100) NOT NULL, "HTTPS" varchar(500) NOT NULL)

---