Skip to content

Implement shared session capability #1489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions docs/book/vc_shared_sessions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# vSphere Shared Session capability

One problem that can be found when provisioning a large amount of clusters using
vSphere Cloud Provider is vCenter session exhaustion. This happens because every
workload cluster needs to request a new session to vSphere to do proper reconciliation.

vSphere 8.0U3 and up uses a new approach of session management, that allows the
creation and sharing of the sessions among different clusters.

A cluster admin can implement a rest API that, once called, requests a new vCenter
session and shares with CPI. This session will not count on the total generated
sessions of vSphere, and instead will be a child derived session.

This configuration can be applied on vSphere Cloud Provider with the usage of
the following secret/credentials, instead of vSphere Username/password:

```yaml
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: vsphere-cloud-secret
stringData:
your-vcenter-host.vc-session-manager-url: "https://shared-session-service.tld/session"
your-vcenter-host.vc-session-manager-token: "authenticationtoken"
```

The configuration above will make CPI call the shared session rest API and use the
provided token to authenticate against vSphere, instead of using a username/password.

The parameter provider at `vc-session-manager-token` is sent as a `Authorization: Bearer` token
to the session manager, and in case this directive is not configured CPI will send the
Pod Service Account token instead.

Below is an example implementation of a shared session manager rest API. Starting the
program below and calling `http://127.0.0.1:18080/session` should return a JSON that is expected
by CPI using session manager to work:

```shell
$ curl 127.0.0.1:18080/session
{"token":"cst-VCT-52f8d061-aace-4506-f4e6-fca78293a93f-....."}
```

**NOTE**: Below implementation is **NOT PRODUCTION READY** and does not implement
any kind of authentication!

```go
package main

import (
"context"
"encoding/json"
"log"
"net/http"
"net/url"

"github.com/vmware/govmomi"
"github.com/vmware/govmomi/session"
"github.com/vmware/govmomi/vim25"
"github.com/vmware/govmomi/vim25/soap"
)

const (
vcURL = "https://my-vc.tld"
vcUsername = "[email protected]"
vcPassword = "somepassword"
)

var (
userPassword = url.UserPassword(vcUsername, vcPassword)
)

// SharedSessionResponse is the expected response of CPI when using Shared session manager
type SharedSessionResponse struct {
Token string `json:"token"`
}

func main() {
ctx := context.Background()
vcURL, err := soap.ParseURL(vcURL)
if err != nil {
panic(err)
}
soapClient := soap.NewClient(vcURL, false)
c, err := vim25.NewClient(ctx, soapClient)
if err != nil {
panic(err)
}
client := &govmomi.Client{
Client: c,
SessionManager: session.NewManager(c),
}
if err := client.SessionManager.Login(ctx, userPassword); err != nil {
panic(err)
}

vcsession := func(w http.ResponseWriter, r *http.Request) {
clonedtoken, err := client.SessionManager.AcquireCloneTicket(ctx)
if err != nil {
w.WriteHeader(http.StatusForbidden)
return
}
token := &SharedSessionResponse{Token: clonedtoken}
jsonT, err := json.Marshal(token)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
w.Write(jsonT)
}

http.HandleFunc("/session", vcsession)
log.Printf("starting webserver on port 18080")
http.ListenAndServe(":18080", nil)
}
```
2 changes: 1 addition & 1 deletion pkg/common/connectionmanager/connectionmanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ func (connMgr *ConnectionManager) Connect(ctx context.Context, vcInstance *VSphe
klog.Error("Failed to get credentials from Secret Credential Manager with err:", err)
return err
}
vcInstance.Conn.UpdateCredentials(credentials.User, credentials.Password)
vcInstance.Conn.UpdateCredentials(credentials.User, credentials.Password, credentials.VCSessionManagerURL, credentials.VCSessionManagerToken)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can just pass credential obj.

return vcInstance.Conn.Connect(ctx)
}

Expand Down
10 changes: 10 additions & 0 deletions pkg/common/connectionmanager/zones.go
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,11 @@ func (cm *ConnectionManager) getDIFromMultiVCorDC(ctx context.Context,

func withTagsClient(ctx context.Context, connection *vclib.VSphereConnection, f func(c *rest.Client) error) error {
c := rest.NewClient(connection.Client)
if connection.SessionManagerURL != "" {
c.SessionID(connection.Client.SessionCookie().Value)
return nil
}

signer, err := connection.Signer(ctx, connection.Client)
if err != nil {
return err
Expand All @@ -307,6 +312,11 @@ func withTagsClient(ctx context.Context, connection *vclib.VSphereConnection, f
}

defer func() {
// When using shared session manager we don't need to logout
if connection.SessionManagerURL != "" {
return
}

if err := c.Logout(ctx); err != nil {
klog.Errorf("failed to logout: %v", err)
}
Expand Down
28 changes: 23 additions & 5 deletions pkg/common/credentialmanager/credentialmanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,19 +193,35 @@ func parseConfig(data map[string][]byte, config map[string]*Credential) error {
}
unknownKeys := map[string][]byte{}
for credentialKey, credentialValue := range data {
if strings.HasSuffix(credentialKey, "password") {
switch {
case strings.HasSuffix(credentialKey, "password"):
vcServer := strings.Split(credentialKey, ".password")[0]
if _, ok := config[vcServer]; !ok {
config[vcServer] = &Credential{}
}
config[vcServer].Password = strings.TrimSuffix(string(credentialValue), "\n")
} else if strings.HasSuffix(credentialKey, "username") {

case strings.HasSuffix(credentialKey, "username"):
vcServer := strings.Split(credentialKey, ".username")[0]
if _, ok := config[vcServer]; !ok {
config[vcServer] = &Credential{}
}
config[vcServer].User = strings.TrimSuffix(string(credentialValue), "\n")
} else {

case strings.HasSuffix(credentialKey, "vc-session-manager-url"):
vcServer := strings.Split(credentialKey, ".vc-session-manager-url")[0]
if _, ok := config[vcServer]; !ok {
config[vcServer] = &Credential{}
}
config[vcServer].VCSessionManagerURL = strings.TrimSuffix(string(credentialValue), "\n")

case strings.HasSuffix(credentialKey, "vc-session-manager-token"):
vcServer := strings.Split(credentialKey, ".vc-session-manager-token")[0]
if _, ok := config[vcServer]; !ok {
config[vcServer] = &Credential{}
}
config[vcServer].VCSessionManagerToken = strings.TrimSuffix(string(credentialValue), "\n")
default:
unknownKeys[credentialKey] = credentialValue
}
}
Expand Down Expand Up @@ -287,10 +303,12 @@ func parseConfig(data map[string][]byte, config map[string]*Credential) error {
}

for vcServer, credential := range config {
if credential.User == "" || credential.Password == "" {
klog.Errorf("Username/Password is missing for server %s", vcServer)
if (credential.User == "" || credential.Password == "") && credential.VCSessionManagerURL == "" {

klog.Errorf("Username/Password or shared session manager URL/Token directives are missing for server %s", vcServer)
return ErrCredentialMissing
}

}
return nil
}
99 changes: 84 additions & 15 deletions pkg/common/credentialmanager/credentialmanager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,20 @@ import (

func TestSecretCredentialManagerK8s_GetCredential(t *testing.T) {
var (
userKey = "username"
passwordKey = "password"
testUser = "user"
testPassword = "password"
testServer = "0.0.0.0"
testServer2 = "0.0.1.1"
testIPv6Server = "fd01::1"
testUserServer2 = "user1"
testPasswordServer2 = "password1"
testIncorrectServer = "1.1.1.1"
userKey = "username"
passwordKey = "password"
vcSessionURL = "vc-session-manager-url"
vcSessionToken = "vc-session-manager-token"
testUser = "user"
testPassword = "password"
testServer = "0.0.0.0"
testServer2 = "0.0.1.1"
testIPv6Server = "fd01::1"
testUserServer2 = "user1"
testPasswordServer2 = "password1"
testIncorrectServer = "1.1.1.1"
testSessionManagerURL = "https://somemanager.tld/session"
testSessionManagerToken = "token"
)
var (
secretName = "vsconf"
Expand All @@ -50,10 +54,12 @@ func TestSecretCredentialManagerK8s_GetCredential(t *testing.T) {
deleteSecretOp = "DELETE_SECRET_OP"
)
type GetCredentialsTest struct {
server string
username string
password string
err error
server string
username string
password string
vcSessionURL string
vcSessionToken string
err error
}
type OpSecretTest struct {
secret *corev1.Secret
Expand Down Expand Up @@ -88,6 +94,16 @@ func TestSecretCredentialManagerK8s_GetCredential(t *testing.T) {
},
}

multiVCSecretMixedWithSessionManager := &corev1.Secret{
ObjectMeta: metaObj,
Data: map[string][]byte{
testServer + "." + userKey: []byte(testUser),
testServer + "." + passwordKey: []byte(testPassword),
testServer2 + "." + vcSessionURL: []byte(testSessionManagerURL),
testServer2 + "." + vcSessionToken: []byte(testSessionManagerToken),
},
}

ipv6CompatSecret := &corev1.Secret{
ObjectMeta: metaObj,
Data: map[string][]byte{
Expand Down Expand Up @@ -194,6 +210,20 @@ func TestSecretCredentialManagerK8s_GetCredential(t *testing.T) {
},
},
},
{
testName: "GetCredential for multi-vc with session manager",
ops: []string{addSecretOp, getCredentialsOp},
expectedValues: []interface{}{
OpSecretTest{
secret: multiVCSecretMixedWithSessionManager,
},
GetCredentialsTest{
server: testServer2,
vcSessionURL: testSessionManagerURL,
vcSessionToken: testSessionManagerToken,
},
},
},
{
testName: "GetCredential for alternative IPv6 server address compatable format",
ops: []string{addSecretOp, getCredentialsOp},
Expand Down Expand Up @@ -259,7 +289,9 @@ func TestSecretCredentialManagerK8s_GetCredential(t *testing.T) {
}
if expected.err == nil {
if expected.username != credential.User ||
expected.password != credential.Password {
expected.password != credential.Password ||
expected.vcSessionToken != credential.VCSessionManagerToken ||
expected.vcSessionURL != credential.VCSessionManagerURL {
t.Fatalf("Received credentials %v "+
"are different than actual credential user:%s password:%s", credential, expected.username,
expected.password)
Expand Down Expand Up @@ -352,6 +384,43 @@ func TestParseSecretConfig(t *testing.T) {
},
expectedError: ErrCredentialMissing,
},
{
testName: "Missing session manager token",
data: map[string][]byte{
"10.20.30.40.vc-session-manager-url": []byte("https://something.tld/session"),
},
config: map[string]*Credential{
testIP: {
VCSessionManagerURL: "https://something.tld/session",
},
},
expectedError: nil,
},
{
testName: "Missing session manager url",
data: map[string][]byte{
"10.20.30.40.vc-session-manager-token": []byte("token"),
},
config: map[string]*Credential{
testIP: {
VCSessionManagerToken: "token",
},
},
expectedError: ErrCredentialMissing,
},
{
testName: "Valid session manager configuration",
data: map[string][]byte{
"10.20.30.40.vc-session-manager-url": []byte("https://something.tld/session"),
"10.20.30.40.vc-session-manager-token": []byte("token"),
},
config: map[string]*Credential{
testIP: {
VCSessionManagerURL: "https://something.tld/session",
VCSessionManagerToken: "token",
},
},
},
{
testName: "IP with unknown key",
data: map[string][]byte{
Expand Down
3 changes: 3 additions & 0 deletions pkg/common/credentialmanager/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ type SecretCache struct {
type Credential struct {
User string `gcfg:"user"`
Password string `gcfg:"password"`
// VC shared session manager directives
VCSessionManagerURL string `gcfg:"vc-session-manager-url"`
VCSessionManagerToken string `gcfg:"vc-session-manager-token"`
}

// CredentialManager is used to manage vCenter credentials stored as
Expand Down
Loading