Dev 1.0.0

.gitignore

@@ -1,6 +1,7 @@
 *.iml
 .idea
 .DS_Store
+*.class
 
 assembly/target
 
@@ -20,4 +21,14 @@ plugins/azkaban/linkis-jobtype/target/
 plugins/linkis/linkis-appjoint-entrance/target/
 
 sendemail-appjoint/sendemail-core/target/
-visualis-appjoint/appjoint/target/
+visualis-appjoint/appjoint/target/
+
+dss-user-manager/target/
+logs
+### Example user template template
+### Example user template
+
+# IntelliJ project files
+
+out
+gen

.vscode/settings.json

@@ -0,0 +1,5 @@
+{
+  "editor.formatOnPaste": true,
+  "editor.formatOnType": true,
+  "editor.formatOnSave": true
+}

README.md

@@ -33,75 +33,77 @@ Please be patient, it will take some time to load gif.
 
         c. [Qualitis](https://github.com/WeBankFinTech/Qualitis) - Data Quality Management Tool
 
-        d. [Azkaban](https://azkaban.github.io/) - Batch workflow job scheduler
+        d. [Schedulis](https://github.com/WeBankFinTech/Schedulis) - Batch workflow job scheduler
+
+        f. [Exchangis](https://github.com/WeBankFinTech/Exchangis) - Data Exchange Tool
 
 ![DSS one-stop video](images/en_US/readme/onestop.gif) 
 
-### 2. AppJoint, based on Linkis，defines a unique design concept
+### 2. AppConn, based on Linkis，defines a unique design concept
 
-        AppJoint——application joint, defining unified front-end and back-end 
+        AppConn——application connector, defining unified front-end and back-end 
                             integration specifications, can quickly and easily integrate with external data application systems, 
                             making them as part of DSS data application development.
 
-        DSS arranges multiple AppJoints in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.
+        DSS arranges multiple AppConns in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.
 
-        Since AppJoint is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppJoint also allows sharable context across system level and completely gets away from application silos.
+        Since AppConn is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppConn also allows sharable context across system level and completely gets away from application silos.
 
 ### 3. Project, as the management unit
 
         With Project as the management unit, DSS organizes and manages the business applications of each data application system, and defines a set of common standards for collaborative development of projects across data application systems.
 
 ### 4. Integrated data application components
 
-       a. Azkaban AppJoint —— Batch workflow job scheduler
+       a. Schedulis AppConn —— Batch workflow job scheduler
 
          Many data applications developed by users usually require periodic scheduling capability.
 
          At present, the open source scheduling system in the community is pretty unfriendly to integrate with other data application systems.
 
-         DSS implements Azkaban AppJoint, which allows users to publish DSS workflows to Azkaban for regular scheduling.
+         DSS implements Schedulis AppConn, which allows users to publish DSS workflows to Azkaban for regular scheduling.
 
          DSS also defines standard and generic workflow parsing and publishing specifications for scheduling systems, allowing other scheduling systems to easily achieve low-cost integration with DSS.
 
 ![Azkaban](images/en_US/readme/Azkaban_AppJoint.gif)
 
-       b. Scriptis AppJoint —— Data Development IDE Tool
+       b. Scriptis AppConn —— Data Development IDE Tool
 
          What is [Scriptis](https://github.com/WeBankFinTech/Scriptis)?
 
          Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
 
-         Scriptis AppJoint integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.
+         Scriptis AppConn integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.
 
          Currently supports HiveSQL, SparkSQL, Pyspark, Scala and other script node types.
 
 ![Scriptis](images/en_US/readme/Scriptis_AppJoint.gif)
 
-       c. Visualis AppJoint —— Data Visualization Tool
+       c. Visualis AppConn —— Data Visualization Tool
 
          What is [Visualis](https://github.com/WeBankFinTech/Visualis)?
 
          Visualis is a BI tool for data visualization. It provides financial-grade data visualization capabilities on the basis of data security and permissions, based on the open source project Davinci contributed by CreditEase.
 
-         Visualis AppJoint integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.
+         Visualis AppConn integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.
 
 ![Visualis](images/en_US/readme/Visualis_AppJoint.gif)
 
-       d. Qualitis AppJoint —— Data quality management Tool
+       d. Qualitis AppConn —— Data quality management Tool
 
-         Qualitis AppJoint integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow
+         Qualitis AppConn integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow
 
 ![Qualitis](images/en_US/readme/Qualitis_AppJoint.gif)
 
-       e. Data Sender——Sender AppJoint
+       e. Data Sender——Sender AppConn
 
-         Sender AppJoint provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.
+         Sender AppConn provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.
 
          For example, the SendEmail node can directly send the screen shot of a display as an email.  
 
-       f. Signal AppJoint —— Signal Nodes
+       f. Signal AppConn —— Signal Nodes
 
-         Signal AppJoint is used to strengthen the correlation between business and process while keeping them decoupled.
+         Signal AppConn is used to strengthen the correlation between business and process while keeping them decoupled.
 
          DataChecker Node：Checks whether a table or partition exists.
 

assembly/bin/appconn-install.sh

@@ -0,0 +1,109 @@
+#!/bin/sh
+#Actively load user env
+source ~/.bashrc
+shellDir=`dirname $0`
+workDir=`cd ${shellDir}/..;pwd`
+
+SOURCE_ROOT=${workDir}
+
+#load config
+source ${SOURCE_ROOT}/conf/config.sh
+source ${SOURCE_ROOT}/conf/db.sh
+
+APPCONN_NAME=''
+APPCONN_INSTALL_IP=127.0.0.1
+APPCONN_INSTALL_PORT=8088
+
+#echo "Current path of init sql is ${DB_DML_PATH}"
+LOCAL_IP="`ifconfig | grep 'inet' | grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $2}'`"
+
+function isSuccess(){
+  if [ $? -ne 0 ]; then
+      echo "Failed to " + $1
+      exit 1
+  else
+      echo "Succeed to" + $1
+  fi
+}
+
+PROC_NAME=DSSProjectServerApplication
+ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
+if [ $ProcNumber -le 0 ];then
+   echo "${PROC_NAME} is not running,Please check whether DSS is installed"
+   exit 1000
+else
+   echo "Begine to install appconn"
+fi
+
+##choose install mysql mode
+function initInstallAppConn() {
+  echo "Please select the type of installation component?"
+  echo " 1: schedulis"
+  echo " 2: visualis"
+  echo " 3：Your AppConn Name"
+  echo " 4：exit"
+  read -p "Please input the choice:"  idx
+  if [[ '1' = "$idx" ]];then
+   APPCONN_NAME="schedulis"
+  elif [[ '2' = "$idx" ]];then
+    APPCONN_NAME="visualis"
+  elif [[ '4' = "$idx" ]];then
+    echo "no choice,exit!"
+    exit 1
+  else
+     APPCONN_NAME=$idx
+  fi
+  echo "Current installation component is ${APPCONN_NAME}"
+
+  echo ""
+  echo "If this machine(127.0.0.1) is installed, enter 1"
+  echo "For others, you need to enter a complete IP address."
+  read -p "Please enter the ip of appconn: "  ip
+  APPCONN_INSTALL_IP=$ip
+  if [[ '1' = "$ip" ]];then
+   APPCONN_INSTALL_IP="127.0.0.1"
+  fi
+  echo "You input ip is ${APPCONN_INSTALL_IP}"
+
+  echo ""
+  read -p "Please enter the port of appconn:"  port
+  APPCONN_INSTALL_PORT=$port
+  echo "You input ip is ${APPCONN_INSTALL_PORT}"
+}
+
+function replaceCommonIp() {
+ if [[ $APPCONN_INSTALL_IP == "127.0.0.1" ]] || [[ $APPCONN_INSTALL_IP == "0.0.0.0" ]];then
+    echo "APPCONN_INSTALL_IP is equals $APPCONN_INSTALL_IP, we will change it to ip address"
+    APPCONN_INSTALL_IP=$LOCAL_IP
+ fi
+}
+
+##choose execute mysql mode
+function executeSQL() {
+  TEMP_DB_DML_PATH=${SOURCE_ROOT}/dss-appconns/${APPCONN_NAME}/db
+  DB_DML_PATH=$TEMP_DB_DML_PATH/init_real.sql
+  cp -rf $TEMP_DB_DML_PATH/init.sql $DB_DML_PATH
+  sed -i "s/APPCONN_INSTALL_IP/$APPCONN_INSTALL_IP/g"       $DB_DML_PATH
+  sed -i "s/APPCONN_INSTALL_PORT/$APPCONN_INSTALL_PORT/g"   $DB_DML_PATH
+  sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH
+  mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH"
+  isSuccess "source $DB_DML_PATH"
+  echo "the table update finished"
+}
+
+echo ""
+echo "step1:Initialize installation settings"
+initInstallAppConn
+echo ""
+
+echo "step2:replaceIp"
+replaceCommonIp
+echo ""
+
+echo "step3:update database"
+executeSQL
+echo ""
+
+echo "step4:refresh appconn load"
+curl -H "Token-Code:BML-AUTH" -H "Token-User:hadoop" -X GET http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}/api/rest_j/v1/dss/framework/project/appconn/${APPCONN_NAME}/load
+echo ""

bin/checkEnv.sh → assembly/bin/checkEnv.sh

@@ -36,8 +36,6 @@ echo "<-----start to check used cmd---->"
 need_cmd yum
 need_cmd java
 need_cmd mysql
-need_cmd unzip
-need_cmd expect
 need_cmd telnet
 need_cmd tar
 need_cmd sed

assembly/bin/excecuteSQL.sh

@@ -0,0 +1,102 @@
+#!/bin/sh
+
+function checkExternalServer(){
+  echo "telnet check for your $SERVER_NAME, if you wait for a long time,may be your $SERVER_NAME does not prepared"
+  result=`echo -e "\n" | telnet $EXTERNAL_SERVER_IP $EXTERNAL_SERVER_PORT 2>/dev/null | grep Connected | wc -l`
+  if [ $result -eq 1 ]; then
+   echo "$SERVER_NAME is OK."
+  else
+    echo "$SERVER_NAME is Bad. You need to prepare the' $SERVER_NAME ' environment in advance"
+    exit 1
+  fi
+}
+
+## choose install mode
+function chooseInstallMode() {
+    echo "Simple installation mode"
+    #check for Java
+    checkJava
+    #check for mysql
+    SERVER_NAME=MYSQL
+    EXTERNAL_SERVER_IP=$MYSQL_HOST
+    EXTERNAL_SERVER_PORT=$MYSQL_PORT
+    checkExternalServer
+}
+
+##choose install mysql mode
+function chooseInstallMySQLMode() {
+  echo "Do you want to clear Dss table information in the database?"
+  echo " 1: Do not execute table-building statements"
+  echo " 2: Dangerous! Clear all data and rebuild the tables."
+  echo ""
+  MYSQL_INSTALL_MODE=1
+  read -p "Please input the choice:"  idx
+  if [[ '2' = "$idx" ]];then
+    MYSQL_INSTALL_MODE=2
+    echo "You chose Rebuild the table"
+  elif [[ '1' = "$idx" ]];then
+    MYSQL_INSTALL_MODE=1
+    echo "You chose not execute table-building statements"
+  else
+    echo "no choice,exit!"
+    exit 1
+  fi
+
+  ##init db
+  if [[ '2' = "$MYSQL_INSTALL_MODE" ]];then
+    ENV_FLAG="dev"
+    DB_CONF_PATH=${workDir}/db
+    DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql
+    replaceAppConnInstanceSQL
+    executeSQL
+  fi
+}
+
+##choose execute mysql mode
+function executeSQL() {
+  chooseInstallMode
+
+  sed -i "s/GATEWAY_INSTALL_IP/$GATEWAY_INSTALL_IP/g" $DB_DML_PATH
+  sed -i "s/GATEWAY_PORT/$GATEWAY_PORT/g" $DB_DML_PATH
+
+  sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH
+
+  mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_CONF_PATH/dss_ddl.sql"
+  isSuccess "source dss_ddl.sql"
+  mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH"
+  isSuccess "source dss_dml_real.sql"
+  echo "Rebuild the table"
+}
+
+function replaceAppConnInstanceSQL() {
+  DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql
+  cp -rf $DB_CONF_PATH/dss_dml.sql $DB_DML_PATH
+  sed -i "s#ORCHESTRATOR_IP#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP#g" $DB_DML_PATH
+  sed -i "s#ORCHESTRATOR_PORT#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT#g" $DB_DML_PATH
+
+  sed -i "s#WORKFLOW_IP#$DSS_WORKFLOW_SERVER_INSTALL_IP#g" $DB_DML_PATH
+  sed -i "s#WORKFLOW_PORT#$DSS_WORKFLOW_SERVER_PORT#g" $DB_DML_PATH
+
+  sed -i "s#EVENTCHECKER_JDBC_URL#$EVENTCHECKER_JDBC_URL#g" $DB_DML_PATH
+  sed -i "s#EVENTCHECKER_JDBC_USERNAME#$EVENTCHECKER_JDBC_USERNAME#g" $DB_DML_PATH
+  sed -i "s#EVENTCHECKER_JDBC_PASSWORD#$EVENTCHECKER_JDBC_PASSWORD#g" $DB_DML_PATH
+
+  sed -i "s#DATACHECKER_JOB_JDBC_URL#$DATACHECKER_JOB_JDBC_URL#g" $DB_DML_PATH
+  sed -i "s#DATACHECKER_JOB_JDBC_USERNAME#$DATACHECKER_JOB_JDBC_USERNAME#g" $DB_DML_PATH
+  sed -i "s#DATACHECKER_JOB_JDBC_PASSWORD#$DATACHECKER_JOB_JDBC_PASSWORD#g" $DB_DML_PATH
+
+  sed -i "s#DATACHECKER_BDP_JDBC_URL#$DATACHECKER_BDP_JDBC_URL#g" $DB_DML_PATH
+  sed -i "s#DATACHECKER_BDP_JDBC_USERNAME#$DATACHECKER_BDP_JDBC_USERNAME#g" $DB_DML_PATH
+  sed -i "s#DATACHECKER_BDP_JDBC_PASSWORD#$DATACHECKER_BDP_JDBC_PASSWORD#g" $DB_DML_PATH
+
+  sed -i "s#BDP_MASK_IP#127.0.0.1#g" $DB_DML_PATH
+  sed -i "s#BDP_MASK_PORT#8087#g" $DB_DML_PATH
+
+  sed -i "s#EMAIL_HOST#${EMAIL_HOST}#g" $DB_DML_PATH
+  sed -i "s#EMAIL_PORT#${EMAIL_PORT}#g" $DB_DML_PATH
+  sed -i "s#EMAIL_USERNAME#${EMAIL_USERNAME}#g" $DB_DML_PATH
+  sed -i "s#EMAIL_PASSWORD#${EMAIL_PASSWORD}#g" $DB_DML_PATH
+  sed -i "s#EMAIL_PROTOCOL#${EMAIL_PROTOCOL}#g" $DB_DML_PATH
+}
+
+chooseInstallMySQLMode