Single Response Address Support (#877)

* Encapsulate response in ResponseManager (DRAFT)

* Use key pair for crypto

* Process responses via periodic callback

* Minor touch ups - remove commented-out blocks

* Use AES & RSA algs, convey version from launcher

* Adjust connection poll interval, add some docstrings

* Fix lint issues

* Refactor response event framework

* Add support for legacy (version 0) launchers

* Update Python/Scala kernelspecs, revert R launcher changes

* Missed some kernelspec updates previously

* Update R kernel-launcher and specs to Version 1

* Update container-based configurations

* Add response-port to helm deployment

* Enable ability to handle multiple response IPs

* Apply doc updates

Author: kevin-bates

Makefile

@@ -84,7 +84,7 @@ install: ## Make a conda env with dist/*.whl and dist/*.tar.gz installed
 			pip uninstall -y jupyter_enterprise_gateway
 	conda env remove -y -n $(ENV)-install
 
-bdist:
+bdist: lint
 	make $(WHEEL_FILE)
 
 $(WHEEL_FILE): $(WHEEL_FILES)

docs/source/config-options.md

@@ -432,6 +432,16 @@ The following environment variables can be used to influence functionality and a
     Attempts to launch a kernel where KERNEL_UID's value is in this list will result
     in an exception indicating error 403 (Forbidden).  See also EG_PROHIBITED_GIDS.
 
+  EG_RESPONSE_IP=None
+    Experimental.  The IP address to use to formulate the response address (with 
+    `EG_RESPONSE_PORT`).  By default, the server's IP is used.  However, we may find 
+    it necessary to use a different IP in cases where the target kernels are external
+    to the Enterprise Gateway server (for example).  It's value may also need to be 
+    set in cases where the computed (default) is not correct for the current topology.
+
+  EG_RESPONSE_PORT=8877
+    The single response port used to receive connection information from launched kernels.
+
   EG_SHARED_NAMESPACE=False
     Kubernetes only. This value indicates whether (True) or not (False) all kernel pods
     should reside in the same namespace as Enterprise Gateway.  This is not a recommended
@@ -504,8 +514,8 @@ The following environment variables are managed by Enterprise Gateway and listed
     This value is set during each kernel launch and resides in the environment of
     the kernel launch process. Its value represents the address to which the remote
     kernel's connection information should be sent.  Enterprise Gateway is listening
-    on that socket and will close the socket once the remote kernel launcher has
-    conveyed the appropriate information.
+    on that socket and will associate that connnection information with the responding
+    kernel.
 ```
 
 ### Per-kernel Configuration Overrides

docs/source/docker.md

@@ -71,7 +71,7 @@ Custom kernel images require some support files from the Enterprise Gateway repo
 Enterprise Gateway provides a single [bootstrap-kernel.sh](https://github.com/jupyter/enterprise_gateway/blob/master/etc/kernel-launchers/bootstrap/bootstrap-kernel.sh) script that handles the three kernel languages supported out of the box - Python, R, and Scala.  When a kernel image is started by Enterprise Gateway, parameters used within the bootstrap-kernel.sh script are conveyed via environment variables.  The bootstrap script is then responsible for validating and converting those parameters to meaningful arguments to the appropriate launcher.
 
 ##### Kernel Launcher
-The kernel launcher, as discussed [here](system-architecture.html#kernel-launchers) does a number of things.  In paricular, it creates the connection ports and conveys that connection information back to Enterprise Gateway via the socket identified by the response address parameter.  Although not a requirement for container-based usage, it is recommended that the launcher be written in the same language as the kernel.  (This is more of a requirement when used in applications like YARN.)
+The kernel launcher, as discussed [here](system-architecture.html#kernel-launchers) does a number of things.  In particular, it creates the connection ports and conveys that connection information back to Enterprise Gateway via the socket identified by the response address parameter.  Although not a requirement for container-based usage, it is recommended that the launcher be written in the same language as the kernel.  (This is more of a requirement when used in applications like YARN.)
 
 #### About Jupyter Docker-stacks Images
 Most of what is presented assumes the base image for your custom image is derived from the [Jupyter Docker-stacks](https://github.com/jupyter/docker-stacks) repository.  As a result, it's good to cover what makes up those assumptions so you can build your own image independently from the docker-stacks repository.
@@ -189,7 +189,9 @@ cp -r python_kubernetes python_myCustomKernel
     "--RemoteProcessProxy.kernel-id",
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
-    "{response_address}"
+    "{response_address}",
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```

docs/source/getting-started.md

@@ -124,7 +124,7 @@ In that case, the image should ensure the availability of the kernel libraries a
 The launch of containerized kernels via Enterprise Gateway is two-fold.
 
 1. First, there's the argv section in the kernelspec that is processed by the server. In these cases, the command that is invoked is a python script using the target container's api (kubernetes, docker, or docker swarm) that is responsible for converting any necessary "parameters" to environment variables, etc. that are used during the actual container creation.
-2. The command that is run in the container is the actual kernel launcher script. This launcher is responsible for taking the response address (which is now an env variable) and returning the kernel's connection information back on that response address to Enterprise Gateway. The kernel launcher does additional things - but primarily listens for interrupt and shutdown requests, which it then passes along to the actual (embedded) kernel.
+2. The command that is run in the container is actually the kernel launcher script. This script is responsible for taking the response address (which is now an env variable) and returning the kernel's connection information back on that response address to Enterprise Gateway. The kernel launcher does additional things - but primarily listens for interrupt and shutdown requests, which it then passes along to the actual (embedded) kernel.
 
 So container environments have two launches - one to launch the container itself, the other to launch the kernel (within the container).
 

docs/source/kernel-docker.md

@@ -92,8 +92,8 @@ As always, kernels are launched by virtue of the `argv:` stanza in their respect
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
     "{response_address}",
-    "--RemoteProcessProxy.spark-context-initialization-mode",
-    "none"
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```
@@ -119,8 +119,8 @@ Running containers in Docker Swarm versus traditional Docker are different enoug
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
     "{response_address}",
-    "--RemoteProcessProxy.spark-context-initialization-mode",
-    "none"
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```

docs/source/kernel-kubernetes.md

@@ -36,6 +36,9 @@ spec:
   - name: http
     port: 8888
     targetPort: 8888
+  - name: response
+    port: 8877
+    targetPort: 8877
   selector:
     gateway-selector: enterprise-gateway
   sessionAffinity: ClientIP
@@ -69,6 +72,12 @@ spec:
       serviceAccountName: enterprise-gateway-sa
       containers:
       - env:
+        - name: EG_PORT
+          value: "8888"
+
+        - name: EG_RESPONSE_PORT
+          value: "8877"
+
           # Created above.
         - name: EG_NAMESPACE
           value: "enterprise-gateway"
@@ -105,6 +114,7 @@ spec:
         args: ["--gateway"]
         ports:
         - containerPort: 8888
+        - containerPort: 8877
 ```
 #### Namespaces
 A best practice for Kubernetes applications running in an enterprise is to isolate applications via namespaces.  Since Enterprise Gateway also requires isolation at the kernel level, it makes sense to use a namespace for each kernel, by default.
@@ -348,41 +358,50 @@ There are essentially two kinds of kernels (independent of language) launched wi
 
 When _vanilla_ kernels are launched, Enterprise Gateway is responsible for creating the corresponding pod. On the other hand, _spark-on-kubernetes_ kernels are launched via `spark-submit` with a specific `master` URI - which then creates the corresponding pod(s) (including executor pods).  Images can be launched using both forms provided they have the appropriate support for Spark installed.
 
-Here's the yaml configuration used when _vanilla_ kernels are launched. As noted in the `KubernetesProcessProxy` section below, this file ([kernel-pod.yaml](https://github.com/jupyter/enterprise_gateway/blob/master/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml)) serves as a template where each of the tags surrounded with `${}` represent variables that are substituted at the time of the kernel's launch.  All `${kernel_xxx}` parameters correspond to `KERNEL_XXX` environment variables that can be specified from the client in the kernel creation request's json body.
+Here's the yaml configuration used when _vanilla_ kernels are launched. As noted in the `KubernetesProcessProxy` section below, this file ([kernel-pod.yaml.j2](https://github.com/jupyter/enterprise_gateway/blob/master/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2)) serves as a template where each of the tags surrounded with `{{` and `}}` represent variables that are substituted at the time of the kernel's launch.  All `{{ kernel_xxx }}` parameters correspond to `KERNEL_XXX` environment variables that can be specified from the client in the kernel creation request's json body.
 ```yaml
 apiVersion: v1
 kind: Pod
 metadata:
-  name: ${kernel_username}-${kernel_id}
-  namespace: ${kernel_namespace}
+  name: "{{ kernel_pod_name }}"
+  namespace: "{{ kernel_namespace }}"
   labels:
-    kernel_id: ${kernel_id}
+    kernel_id: "{{ kernel_id }}"
     app: enterprise-gateway
     component: kernel
 spec:
   restartPolicy: Never
-  serviceAccountName: ${kernel_service_account_name}
+  serviceAccountName: "{{ kernel_service_account_name }}"
+  {% if kernel_uid is defined or kernel_gid is defined %}
   securityContext:
-    runAsUser: ${kernel_uid}
-    runAsGroup: ${kernel_gid}
+    {% if kernel_uid is defined %}
+    runAsUser: {{ kernel_uid | int }}
+    {% endif %}
+    {% if kernel_gid is defined %}
+    runAsGroup: {{ kernel_gid | int }}
+    {% endif %}
+    fsGroup: 100
+  {% endif %}
   containers:
   - env:
     - name: EG_RESPONSE_ADDRESS
-      value: ${eg_response_address}
+      value: "{{ eg_response_address }}"
+    - name: EG_PUBLIC_KEY
+      value: "{{ eg_public_key }}"
     - name: KERNEL_LANGUAGE
-      value: ${kernel_language}
+      value: "{{ kernel_language }}"
     - name: KERNEL_SPARK_CONTEXT_INIT_MODE
-      value: ${kernel_spark_context_init_mode}
+      value: "{{ kernel_spark_context_init_mode }}"
     - name: KERNEL_NAME
-      value: ${kernel_name}
+      value: "{{ kernel_name }}"
     - name: KERNEL_USERNAME
-      value: ${kernel_username}
+      value: "{{ kernel_username }}"
     - name: KERNEL_ID
-      value: ${kernel_id}
+      value: "{{ kernel_id }}"
     - name: KERNEL_NAMESPACE
-      value: ${kernel_namespace}
-    image: ${kernel_image}
-    name: ${kernel_username}-${kernel_id}
+      value: "{{ kernel_namespace }}"
+    image: "{{ kernel_image }}"
+    name: "{{ kernel_pod_name }}"
 ```
 There are a number of items worth noting:
 1. Kernel pods can be identified in three ways using `kubectl`:
@@ -393,7 +412,7 @@ There are a number of items worth noting:
     Note that since kernels run in isolated namespaces by default, it's often helpful to include the clause `--all-namespaces` on commands that will span namespaces.  To isolate commands to a given namespace, you'll need to add the namespace clause `--namespace <namespace-name>`.
 1. Each kernel pod is named by the invoking user (via the `KERNEL_USERNAME` env) and its kernel_id (env `KERNEL_ID`).  This identifier also applies to those kernels launched within `spark-on-kubernetes`.
 1. Kernel pods use the specified `securityContext`.  If env `KERNEL_UID` is not specified in the kernel creation request a default value of `1000` (the jovyan user) will be used.  Similarly for `KERNEL_GID`, whose default is `100` (the users group).  In addition, Enterprise Gateway enforces a lists of prohibited UID and GID values.  By default, this list is initialized to the 0 (root) UID and GID.  Administrators can configure the `EG_PROHIBITED_UIDS` and `EG_PROHIBITED_GIDS` environment variables via the enterprise-gateway.yaml file with comma-separated values to alter the set of user and group ids to be prevented.
-1. As noted above, if `KERNEL_NAMESPACE` is not provided in the request, Enterprise Gateway will create a namespace using the same naming algorithm for the pod.  In addition, the `kernel-controller` cluster role will be bound to a namespace-scoped role binding of the same name using the namespace's default service account as its subject.  Users wishing to use their own kernel namespaces must provide **both** `KERNEL_NAMESPACE` and `KERNEL_SERVICE_ACCOUNT_NAME` as these are both used in the `kernel-pod.yaml` as `${kernel_namespace}` and `${kernel_service_account_name}`, respectively.
+1. As noted above, if `KERNEL_NAMESPACE` is not provided in the request, Enterprise Gateway will create a namespace using the same naming algorithm for the pod.  In addition, the `kernel-controller` cluster role will be bound to a namespace-scoped role binding of the same name using the namespace's default service account as its subject.  Users wishing to use their own kernel namespaces must provide **both** `KERNEL_NAMESPACE` and `KERNEL_SERVICE_ACCOUNT_NAME` as these are both used in the `kernel-pod.yaml.j2` as `{{ kernel_namespace }}` and `{{ kernel_service_account_name }}`, respectively.
 1. Kernel pods have restart policies of `Never`.  This is because the Jupyter framework already has built-in logic for auto-restarting failed kernels and any other restart policy would likely interfere with the built-in behaviors.
 1. The parameters to the launcher that is built into the image are communicated via environment variables as noted in the `env:` section above.
 
@@ -514,8 +533,8 @@ As always, kernels are launched by virtue of the `argv:` stanza in their respect
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
     "{response_address}",
-    "--RemoteProcessProxy.spark-context-initialization-mode",
-    "none"
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```
@@ -536,6 +555,8 @@ For these invocations, the `argv:` is nearly identical to non-kubernetes configu
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
     "{response_address}",
+    "--RemoteProcessProxy.public-key",
+    "{public_key}",
     "--RemoteProcessProxy.spark-context-initialization-mode",
     "lazy"
   ]

docs/source/kernel-spark-standalone.md

@@ -67,7 +67,9 @@ After that, you should have a kernel.json that looks similar to the one below:
      "--RemoteProcessProxy.kernel-id",
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
-    "{response_address}"
+    "{response_address}",
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```

docs/source/kernel-yarn-client-mode.md

@@ -64,7 +64,9 @@ After that, you should have a kernel.json that looks similar to the one below:
      "--RemoteProcessProxy.kernel-id",
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
-    "{response_address}"
+    "{response_address}",
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```

docs/source/kernel-yarn-cluster-mode.md

@@ -64,7 +64,9 @@ After that, you should have a `kernel.json` that looks similar to the one below:
      "--RemoteProcessProxy.kernel-id",
     "{kernel_id}",
     "--RemoteProcessProxy.response-address",
-    "{response_address}"
+    "{response_address}",
+    "--RemoteProcessProxy.public-key",
+    "{public_key}"
   ]
 }
 ```