Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
App hosting lifecycle #19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Uh oh!
There was an error while loading. Please reload this page.
App hosting lifecycle #19
Changes from all commits
025427de6094c090d7c7756a97829df082a4289269aff177258e1fea1050778adc60a5380e89d6a3328d4efa2d0285bc4cc7e7a08c82bb1bFile filter
Filter by extension
Conversations
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Uh oh!
There was an error while loading. Please reload this page.
There are no files selected for viewing
#+Title: Virtual Labs Hosting Process #+Author: VLEAD #+Email: [email protected] #+DATE: [2020-01-08 Wed] #+PROPERTY: results output #+PROPERTY: exports code #+SETUPFILE: ../org-templates/level-1.org #+LANGUAGE: en #+OPTIONS: ^:nil * Introduction Virtual Labs is a mission mode project initiated by the Ministry of Human Resources and Development (MHRD). The objective of this project is to provide laboratory learning experience to the students who do not have access to adequate laboratory infrastructure. Currently there are around 90 labs developed by various institutes in the consortium. A streamlined software development life cycle process followed for the development of these labs ensures high quality labs. This document defines the hosting process to be followed by the developers (open source community) of the Virtual Labs project. * Purpose This document defines the experiment on-boarding and hosting process that should be followed by the developers of the lab while requesting the hosting of their lab on AWS. The roles and responsibilities of the various parties involved in the hosting process is also discussed in detailed in the document. * Motivation A well defined experiment hosting and on-boarding process will help maintain a consistent user experience and enable experiment-authors to focus on the content. Consolidated information regarding all the deployments will also facilitate reporting. * Audience The target audience for this document is the hosting team at CPE, IIITH and all the lab authors and owners who want to avail the hosting service provided by Central Platform Engineering Team (CPE), IIITH team. * Definitions and Pre-requisites Virtual Labs hosting process will require certain prerequisites to be met by the experiments that need to be hosted. The following paragraphs define terminologies used during the hosting process. ** Hosting Hosting is a service provided by the Central Platform Engineering (CPE) which includes the following: 1. Providing infrastructure including all dependencies required to run the Lab/Experiments as specified in the Onboarding request of the hosting unit. 2. Ensuring round the clock availability of the Lab/Experiments. 3. Ensuring that a minimum acceptable performance criteria are met by the Lab/Experiments. Following are not part of the hosting service: 1. Testing: The deployable code is a black box for the CPE team. The CPE team is not responsible for testing the deployments. The developer/owner of the Lab/Experiment is responsible for confirming that the Lab/Experiments are deployed and functioning as intended. This confirmation is part of the hosting process. 2. Implementation: The CPE team is not responsible for any bug fixes or enhancements or implementation on the Lab/Experiment. The developers should ensure that their product is complete and functional before raising a hosting request. 3. Approvals: The CPE team does not provide any approvals regarding the completeness of the Lab/Experiment. The developers are expected to have sought the necessary approvals regarding completeness of the Lab/Experiment before raising a hosting request. ** Hosting Unit Hosting unit is the code base/repository that will hosted in its enirety during the hosting process. It will not be possible for a part of a hosting unit to be hosted separately. A hosting unit will correspond to one public repository.A hosting unit could be a stand alone experiment or could be a lab consisting of a set of related experiments. ** Repository Owner Each repository, corresponding to a hosting unit, will have one designated owner. The owner will be the single entity responsible for all the code that resides in the repository. The owner will also be the final approval authority for a hosting request. The owner should ensure that the master branch of the repository only contains the code that needs to be hosted. The latest code in the master branch should represent the hosted experiment. The repository owner does not have to be a person. It may be any uniquely identifiable entity. From the perpective of Virtual Labs hosting team, this entity will the single point of contact for all clarifications. The hosting team will not be responsible for any confusions arising due to shared accessibility of the repo owner entity. ** Requester A repository owner or a developer who raises a hosting request. ** Tags The repository owner will be responsible for tagging each merge to the master branch. The owner will have to follow [[https://semver.org/][Semantic Versioning]] . In short, each version is a combination of three numbers (MAJOR.MINOR.PATCH) separated by dots. The changes to these numbers represent the following: 1. MAJOR: A change incompatible with previous versions. 2. MINOR: A backwords compatible new feature. 3. PATCH: A backwords compatible bug fix. All Virtual Labs hosting request should specify a tag. ** Build Process Each hosting unit should have a well defined build process. The hosting team will treat the build process as a black box action. This action will be triggerred to generate the artifacts to deploy. The repository owner should ensure that the build process is well documented and functioning as desired before raising a hosting request. ** Testing The hosting process will result in a deployment with an accessible url. The responsibility of testing the deployments will lie with the repository owner. The hosting team will be responsible for providing the link to the deployment. The hosting team will not provide any testing environment. The repository owner will be responsible for ensuring that the hosting unit works as expected in their own specific testing environments. * Hosting Process Virtual Labs hosting process consists of two steps - 1. Request the onboarding of a hosting unit 2. Request the hosting of hosting unit ** On-Boarding process of hosting unit Each hosting unit that needs to be hosted by the Virtual Labs hosting team, must follow an onboarding process. The repository owner of the hosting unit will need to raise an issue of type '[[https://github.com/virtual-labs/engineers-forum/issues/new/choose][on-boarding request]]' in the engineers-forum. The issue will need to filled with the following pieces of information : On-Boarding Request - 1. A non-conflicting name for the hosting unit. 2. Repository URL for the hosting unit. 3. A designated repository owner with the following details: a. Repository owner github id b. Repository owner email address 4. The build command for the hosting unit along with the link to the build process documentation. 5. Hardware specification including: a. CPU b. Memory c. Network bandwidth 6. Software environment specification including: a. Host operating system with minimum compatible version b. Any additional software packages with minimum compatile versions c. Browser plugins required to view experiment d. DB Repository owner will be reponsible for keeping the above information current and intimating the hosting team of any updates/changes as and when needed. On receipt of an onboarding request the hosting team will verify the data and acknowledge the same as part of the request. The hosting team will be responsible for storing the this information at a central location and making it available for all stakeholders. ** Hosting Process of hosting unit A repository owner/developer (requester ) will have to raise an issue of type '[[https://github.com/virtual-labs/engineers-forum/issues/new/choose][hosting request]]' in the GitHub repository [[https://github.com/virtual-labs/engineers-forum][engineers-forum]] under Virtual-Labs organisation for hosting of a hosting unit. The issue will need to filled with the following pieces of information : 1. Repository URL 2. Branch to be deployed 3. Tag to be deployed 4. Contact email address and github handle to be used in cases of clarifications A hosting request will be tied to this information. This issue will be a single source of truth for that hosting request. All communication related to the hosting will be recorded on the raised hosting request. *** *Lifecycle of a Hosting Request* The hosting request will go through the following lifecycle: 1. Requester will raise a hosting request issue by duly filling in the details in the '[[https://github.com/virtual-labs/engineers-forum/issues/new/choose][hosting request]]'. 2. The hosting team will attach the url of the corresponding onboarding request to the hosting request. If the onboarding request is not found ( by searching on the repo url), then the hosting team will add its findings and label the issue as *Failed*. 3. If requester is not the repository owner, the hosting team will seek approval from the repository owner. This will be through the above created GitHub hosting request issue. 4. The hosting team will honour the hosting request only on receiving the approval from repository owner else will label the hosting request issue as *Not Approved* 5. On a successful hosting, the hosting team will add the link of the newly hosted content to the hosting request issue. They will also label the issue as *Hosted*. 6. However, on failure ( because of missing or incorrect information or repository problems,), the hosting team will label the issue as *Failed*. 7. The requester will be responsible for fixing the failed hosting bug and will need to raise a new '[[https://github.com/virtual-labs/engineers-forum/issues/new/choose][hosting request]]' to get the unit hosted. 8. Once the hosting is successful, the requester will be responsible for the verification of the hosted link. If the hosting is not as expected, the requester should change the label from *Hosted* to *Reopened*. The requester will also need to specify if he/she would like to revert to the previous hosted image. 9. An issue with *Reopened* label will by default host the repository and branch/tag as specified in the hosting request initially. However, if the requester has specifically requested a revert, the previously hosted image will be restored and issue will be labelled as *Reverted*. 10. If the requester wants to revert to any earlier branch/tag, a new '[[https://github.com/virtual-labs/engineers-forum/issues/new/choose][hosting request]]' will need to be raised to get the unit hosted. 11. Requester will be responsible for closing a hosting request issue by changing the issue status from Open to Closed when the issue is labelled as *Hosted* or *Reopened* or *Reverted* or *Not Approved*. *** *Labels & Status of a Hosting Request* At any given time a hosting issue should be marked with only one of the following labels. To change the label of an issue, the current label of the issue should be unchecked and the new label should be checked. *Hosted* : This label indicates that the hosting request has been successful and the hosted url has been shared in the hosting request issue. This label is used only by the hosting team. *Failed* : This label indicates that the hosting request has not been successful. This label is used only by the hosting team. *Reopened* : This label is used by the requester to indicate that the hosting url provided on a successful hosting of the hosting unit is not passing the validation. *Not Approved* : This label indicates that the hosting request issue raised by the requester was not approved by the Repository Owner. This label is used only by the hosting team. *Reverted* : This label indicates that the hosting request has been successfully revert to the previous hosted image. This label is used only by the hosting team. Apart from the above mentioned labels a hosting request issue can either be in a closed or open status as provided by GitHub. *Open* : This status indicates that a new hosting request issue has been raised and is waiting for the services of the hosting team. All issues are in this status when they are created on GitHub. *Closed* : This status indicates that a hosting request has been serviced by the hosting team and is labelled as *Hosted* or *Reopened* or *Reverted* or *Not Approved*. * Central Hosting Data The following information will be stored at a central place for a quick reference by the hosting team: 1. Hosting unit name 2. Repository URL 3. Currently hosted Branch/Tag 4. Previously hosted Branch/Tag 5. Date/Time of hosting Please follow the [[https://drive.google.com/open?id=1WXJA_1QkLg-5S0YYBRKyhEXwOgTSbKvm972Fy-thCUc][link]] for the Central Hosting Data. * Flow Diagram of the Hosting Process The flow diagram depicting lifecycle of a hosting request is below: #+NAME: fig:flowchart [[file:../images/hosting-process.png]] /[[https://drive.google.com/file/d/1gnG5Z3kkwXXZxT-zyB2CB9uZCgqAcdjE/view?usp=sharing][edit image]]/ * Roles and Responsibilities #+NAME: fig:flowchart [[file:../images/hosting-process-table.png]] /[[https://docs.google.com/drawings/d/1JfrZerBuMvTBFFfbhiMKKmePMNJ0GsbfTnsnvRbCLZw/edit][edit table]]/ * FAQ *Why should I use the Hosting Service?* To get more for nothing. Hosting Service provides you a high-availability, high-performance deployment environment. You do not have to spend time, money, people or other resources to ensure that your website is available to your target audience. You can also take advantage of other services, like Analytics, provided by the Central Platform Engineering team and monitor multiple statistics about the users and usage patterns of your Lab. *Do I get better website performance if I use Hosting Service?* Almost all the time. The CPE team works continuously to improve the overall and individual Lab/Experiment performance based on the Analytics data. You can compare the performance of the Labs/Experiments deployed through the hosting service to those deployed elsewhere. If you use the Analytics service integration, the CPE team can individually target your Lab/Experiment for better performance. *Does the CPE team help me improve my Lab performance?* Yes. The CPE team continuously works to improve the infrastructure in order to achieve best performance results for your Lab. The CPE team does not improve your code, however, it can provide you guidelines and actionable performance improvement advice. *Does Hosting Service add a UI to my webpages?* No. Hosting service does not alter your code in any way. As the developer, you are responsible for ensuring that your Lab/Experiment is complete in all respects before you request it to be hosted. Hosting service is strictly limited to its stated objectives. *How can I get Analytics for my experiments?* By working with the CPE team. Analytics service is another service provided by the CPE team. You can work with the CPE team to integrate Analytics service into your Labs. You can also get access to customized dashboards/reports in order to monitor multiple statistics about the users and usage patterns of your Lab. *Does the CPE team test my Lab for correctness as part of hosting?* No. It is not possible for the CPE team to test your Lab because only you understand the right definition of correctness for your Lab/Experiment. As part of hosting, the CPE team ensures that your Lab is up and accessible. You are responsible for ensuring the completeness and correctness of your code and deployment. *Will the CPE team tell me about broken links as part of hosting?* No. Your code is a complete blackbox for the CPE team. You are responsible for ensuring the completeness and correctness of your code and deployment. * Conclusion This document highlights the hosting process of Virtual Labs from on-boarding to hosting. It aims at collaborating contributions of the developers across the open source platform - GitHub. This streamlined process would help in our strive for excellence in delivering high standard labs. * Glossary - *Repository* : The distributed version control system - git is used here. Every time the term repository or repo is used, it refers to a git repository. A repository is an on-disk data structure which stores metadata for a set of files and/or directory structure. The whole set of information in the repository may be duplicated on every user's system or may be maintained on a single server. - *Virtual Labs* : Virtual Labs is an initiative by MHRD under NMEICT. The objective of this project is to make engineering education engaging, enjoyable, immersive and online. - *Hosting team*: VLEAD at IIITH which provides Virtual Labs hosting service
Virtual Labs Hosting Process
1 Introduction
1.1 Background
Virtual Labs are online, interactive applications to support learning in science and engineering at the college level. The Virtual Labs project is funded by the Govt. of India, Ministry of Human Resources and Development Project.
Currently there are around 90 labs and about 900 experiments hosted on Virtual Labs.
1.2 Purpose
The Virtual Labs are being built through the efforts of many free and open source development teams. The source repositories of many of the labs can be found on the Github Virtual Labs website. The labs are currently being centrally hosted on a cloud environment by the Virtual Labs hosting team at IIIT Hyderabad (TODO: VLEAD link here!)
This document defines the process that an application Owner needs to follow to have his/her application hosted on the cloud environment.
1.3 Audience
This document is designed to help virtual lab developers understand the process and their role in getting their labs hosted.
1.4 Motivation
Centrally hosting virtual labs allows sharing of resources and facilitates accurate reporting of usage and performance in a holistic way. It is expected that all applications developed in the Virtual Labs community would follow the hosting process documented here.
2 Virtual Labs Hosting Community
2.1 Virtual-Labs community
The virtual labs are built by a large community of developers organized under the Github oganization `virtual-labs’.
2.2 How does one become part of the virtual labs community?
Currently, only applications that have been developed using the virtual labs development process being maintained at IIT Bombay are part of this community. Please contact them or email (TODO: provide vlead email or a VLEAD website).
2.3 The Engineer’s Forum
The engineers-forum is a project under the virtual-labs organization. It is the single-point place for registering an application, and requesting their hosting under virtual-labs.
2.4 Who can register and request hosting?
To register an application with virtual-labs one needs to first register as an owner. See the section <a href=”Registering as Owner”>Registering as Owner for more details.
3 Hosting
3.1 Applications: Experiments and Labs
An application is any project that can be individually hosted. An application is a single hosting unit. Currently, there are two kinds of applications: experiments, and labs.
3.2 Types of labs
A lab is of two types:
3.3 What is Hosting?
Hosting is a service that takes the sources of an application and hosts it on a web server. The result of a successful hosting is a URL of the application.
3.4 What the virtual labs hosting service provides
3.5 What hosting does not provide
3.6 Who is managing the hosting?
Under the virtual-labs project, hosting is a service available to the virtual-lab community. Applications are being hosted on a common cloud platform. The hosting process is being managed by the hosting team at VLEAD, IIIT Hyderabad, India. The hosting team is @virtual-labs/hosting-team on Github.
3.7 Is the hosting free?
It is, if the Owner’s affiliated College or University is part of the Virtual Labs consortium or partner colleges and universities. Hosting costs money (we need to pay a commercial cloud provider). For this reason, the free hosting is limited. There is no paid hosting service at present.
3.8 Hosting terminology
3.9 Source code and Hosting
The source code is expected to be available on a git repository (typically on Github or Gitlab). The hosting team will need read (but not write) permissions to the application repository.
The hosting process will clone the repository and instrument it in order to add UI and other services like analytics.
3.10 Hosting but no testing or debugging of Application
The hosting team will neither test nor debug the application’s code. However, if there is a problem in the build process, it will mark the hosting request as Failed and alert the Owner.
In addition, the hosting team will monitor an application and alert the Owner if the application is consuming too many resources. Applications consuming unreasonable resources will be suspended and the owner alerted.
4 Registering as Owner
A Github user wishing to use the hosting service must register with the Hosting Team. This is needed by the Hosting team to verify the bona fides of the person claiming to be the owner.
4.1 Structure of the Owner Registration Request
A team in a college working on virtual labs may want to create a separate user id or create a team under the virtual-labs organization.
4.2 Owner Registration Workflow
5 Hosting life cycle of an Application
The application’s hosting life cycle consists of the following types of events:
Each event is initiated by the owner and filed as an issue on the engineers-forum.
6 Registering an Application
The registration of an application defines the application as a hostable entity and makes it available to the virtual-labs organization to host it.
Information in the registration issue includes the following:
user-name filing the registration request).
6.1 Identity of the application
An application is identified by its registration issue id on https://gitlab.com/virtual-labs/engineers-forum. This becomes the application id henceforth and id used in hosting or registration update requests to refer to this registration.
6.2 Owner(s) of the application
The github user-name(s) associated with the application. Only owners of an application may request issue requests relating to the hosting lifecycle of an application.
6.3 Source repository (Repo) of an Application
An application’s source code resides in a repository. The mapping of an application to its repository may be changed.
6.4 Registration Workflow
7 Making a Hosting Request
7.1 Hosting request
A hosting request is necessary before a registered application can be hosted.
7.2 Raising a Hosting Request
A hosting request is made by creating an issue using the <a href=”https://github.com/virtual-labs/engineers-forum/issues/new?assignees=&labels=Hosting&template=hosting-request.md&title=Hosting+Request+for+“>hosting request template. The hosting issue will be a single source of truth for that hosting request. All communication related to the hosting will be recorded on the hosting request. So it is best for the Owner to subscribe to this issue to receive notifications when the Owner’s request is handled.
7.3 Structure of a hosting request
A hosting request has the following information:
7.4 Hosting workflow
The table below summarises the actions taken by the two actors (Owner and Hosting Team) and the pre-conditions driving those actions during Hosting Workflow.
7.5 Labels and Status of a Hosting Request
At any given time a hosting issue should be marked with only one of the following labels. To change the label of an issue, the current label of the issue should be unchecked and the new label should be checked.
Hosted : This label indicates that the hosting request has been successful and the hosted url has been shared in the hosting request issue. This label is used only by the hosting team.
Failed : This label indicates that the hosting request has not been successful. This label is used only by the hosting team.
Reopened : This label is used by the Owner to indicate that the hosting url provided on a successful hosting of the hosting unit is not passing the validation.
Reverted : This label indicates that the hosting request has been successfully reverted to the previous hosted image. This label is used only by the hosting team.
Apart from the above mentioned labels a hosting request issue can either be in a closed or open status as provided by GitHub.
Open : This status indicates that a new request issue has been raised and is waiting for the services of the hosting team. Github ensures that all issues are open when newly created.
Closed: This status indicates that a request has been serviced or resolved by the hosting team. Only a Hosted, Reverted or Failed issue may be closed.
7.6 Flow Diagram of the Hosting Process
The flow diagram depicting lifecycle of a hosting request is below:
TODO: Revise it to reflect the current workflow.
edit image
7.7 Roles and Responsibilities
TODO: Update this table based on the hosting protocol defined here. Make the table in org, not a png.
edit table
8 Making an Update Request
8.1 Update request
An update request is necessary to change the name of the application, its owners or source repository.
8.2 Raising an Update Request
An owner raises an update request whenver he/she wishes to update attributes of an application.
8.3 The structure of the Update Request
The information in an update request consists of the application id, and values of the following (if any):
8.4 Update Workflow
9 Deregistering an Application
The deregistration indicates that the owner is no longer interested in having his/her application hosted. This could perhaps be because the owner is no longer maintaining the application or for any other reason.
9.1 Structure of the Deregistration Request
Information for deregistration issue includes the following:
9.2 Deregistration Workflow
10 Alerting an Owner about the Application
Occasionally, the Hosting Team may alert an Owner about his/her application. The alert is currently of the following types:
Suggestion: A friendly tip on how to improve the performance of the application.
Other types of alerts may be added in the future.
10.1 Structure of an alert
The alert consists of the following fields:
10.2 Alert Workflow
11 Suspending an Application
When an application is suspended it is no longer hosted. An application may be suspended if it is found that the application has a serious security or performance flaw. A suspension may be initiated either by an owner or the Hosting Team.
11.1 Structure of an Suspension Request
A suspension request consists of the following fields
11.2 Suspension Workflow (initiated by Owner)
11.3 Suspension Workflow (initiated by Hosting Team)
Once an application is suspended, it can only be rehosted with a fresh hosting request.
12 Build Process
Once the hosting request is processed, it is time to build the application
Build is the process of translating and compiling source files to an executable or hostable web application.
12.1 Files and parameters
To build an application, the hosting team will look for two files at the top level of the application’s repository and try to construct the values of three variables.
The files searched for at the repository source’s top level are:
descriptor.json:: This file, called the deployment descriptor, contains ajsondata object specifying values of various parameters needed by the build. It is described below.makefile:: this is the file on which the default build command is run.12.2 Deployment Descriptor
The hosting of an application is driven by the information in its deployment descriptor. The deployment information must be present in a file called
descriptor.jsonat the top-level of the application’s source repository. This descriptor has additional information.The descriptor contains multiple fields.
directory, software dependencies)
A formal specification of the descriptor’s structure is given here. (TODO: add link to descriptor)
12.3 Dependencies
If the application depends on other software (like databases, compilers, etc.), they need to be specified them in the
dependenciesfield of the deployment descriptor.12.4 The build workflow
These steps are implemented by the Hosting Team.
descriptor.jsonis present, read its parameters and collect values ofbuild-cmd,src-dir,build-diranddependencies.build-cmd.build-dirto a hosting server’s root directory.In this process, *the hosting team will not modify the source files of an application or its source repository in any way*.
13 Request Fields and Templates
14 Tagging conventions
We recommend using Semantic Versioning to tag repositories using the version number convention. Each version is a combination of three numbers (MAJOR.MINOR.PATCH) separated by dots. The changes to these numbers represent the following:
15 FAQ for Virtual Lab developers
Why should I use the Hosting Service?
If you’re building a virtual lab under the MHRD virtual labs project, you are strongly encouraged to host your application on the common cloud. Your application will enjoy high availability and it will save you the trouble of trying to host it yourself. Your application will take advantage of other services like analytics about the users and usage patterns of your application
Will hosting on the central platform team help me improve my application’s performance?
Quite likely. Virtual Labs employs commercial grade hosting services. The hosting plans are continuously negotiated to make them cheaper and faster. The Virtual Labs hosting team will not improve your code for better performance. However, it will provide you guidelines and actionable performance improvement advice if it notices poor performance of your hosted unit.
Does Hosting Service add a UI to my webpages?
Hosting service does not alter your code in any way but may change the UI to match a standard theme if you use the standard development process specified by IIT Bombay Virtual Labs team.
Does the CPE team test my application for correctness as part of hosting?
No. We are not in a position to understand the semantics of your application and therefore can not test except to ensure that your application is correctly hosted.
Will the CPE team tell me about broken links as part of hosting?
No. Your code is a complete black box for the hosting team. You are responsible for ensuring the completeness and correctness of your code and deployment.