Skip to content

[Antithesis] Add the client part for k8s setup #20668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+155 −15
Open

[Antithesis] Add the client part for k8s setup #20668

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions tests/antithesis/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,33 @@ If you use something other than `kind`, please make sure the images are accessib
kubectl apply -f ./config/manifests
```

#### 4. Run the driver(s)

First, locate your client pod, which should be named with prefix client when you execute `kubectl get pods`.

```bash
> kubectl get pods
NAME READY STATUS RESTARTS AGE
client-6b9885f69b-dcndw 1/1 Running 0 10m
etcd-0 1/1 Running 0 10m
etcd-1 1/1 Running 0 10m
etcd-2 1/1 Running 0 10m
```

With the pod name, you can execute the traffic and validation commands:

```bash
# Traffic generation
kubectl exec -it client-6b9885f69b-dcndw -- /opt/antithesis/test/v1/robustness/single_driver_traffic

# Collecting results and compiling the reports
kubectl exec -it client-6b9885f69b-dcndw -- /opt/antithesis/test/v1/robustness/finally_validation
```

##### Making changes and retests

If you make changes to the client or server, you will have to rebuild the images and load them into the cluster again. You can use the same commands as in step 2. Once that the images are loaded, you can delete the existing pods to force kubernetes to create new ones with the updated images:

### Tearing down the cluster

```bash
Expand Down
6 changes: 3 additions & 3 deletions tests/antithesis/config/docker-compose-1-node.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ services:
group_add:
- '${GROUP_ID:-root}'
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the value of this rename? This only complicates the review. Please move it to separate PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See below. It's for distinguishing the existing local data path and all paths.

- ${ETCD_ROBUSTNESS_REPORT_PATH:-/tmp/etcdreport}:/var/report
command:
- /bin/sh
Expand Down Expand Up @@ -43,7 +43,7 @@ services:
ports:
- 12379:2379
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}0:/var/etcd/data
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}0:/var/etcd/data

client:
image: 'etcd-client:${IMAGE_TAG:-latest}'
Expand All @@ -54,5 +54,5 @@ services:
etcd0:
condition: service_started
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_REPORT_PATH:-/tmp/etcdreport}:/var/report
18 changes: 9 additions & 9 deletions tests/antithesis/config/docker-compose-3-node.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ services:
group_add:
- '${GROUP_ID:-root}'
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}1:/var/etcddata1
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}2:/var/etcddata2
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}1:/var/etcddata1
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}2:/var/etcddata2
- ${ETCD_ROBUSTNESS_REPORT_PATH:-/tmp/etcdreport}:/var/report
command:
- /bin/sh
Expand Down Expand Up @@ -45,7 +45,7 @@ services:
ports:
- 12379:2379
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}0:/var/etcd/data
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}0:/var/etcd/data

etcd1:
image: 'etcd-server:${IMAGE_TAG:-latest}'
Expand All @@ -72,7 +72,7 @@ services:
ports:
- 22379:2379
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}1:/var/etcd/data
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}1:/var/etcd/data

etcd2:
image: 'etcd-server:${IMAGE_TAG:-latest}'
Expand All @@ -99,7 +99,7 @@ services:
ports:
- 32379:2379
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}2:/var/etcd/data
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}2:/var/etcd/data

client:
image: 'etcd-client:${IMAGE_TAG:-latest}'
Expand All @@ -114,7 +114,7 @@ services:
etcd2:
condition: service_started
volumes:
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}1:/var/etcddata1
- ${ETCD_ROBUSTNESS_DATA_PATH:-/tmp/etcddata}2:/var/etcddata2
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}0:/var/etcddata0
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}1:/var/etcddata1
- ${ETCD_ROBUSTNESS_LOCAL_DATA_PATH:-/tmp/etcddata}2:/var/etcddata2
- ${ETCD_ROBUSTNESS_REPORT_PATH:-/tmp/etcdreport}:/var/report
52 changes: 52 additions & 0 deletions tests/antithesis/config/manifests/default-etcd-3-replicas.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,55 @@ spec:
resources:
requests:
storage: 200Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: client
labels:
app: client
spec:
replicas: 1
selector:
matchLabels:
app: client
template:
metadata:
labels:
app: client
spec:
containers:
- name: etcd-client
image: etcd-client:latest
command: ["/opt/antithesis/entrypoint/entrypoint"]
imagePullPolicy: Never
env:
- name: ETCD_ROBUSTNESS_SERVER_ENDPOINTS
value: "etcd-0.etcd.default.svc.cluster.local:2379,etcd-1.etcd.default.svc.cluster.local:2379,etcd-2.etcd.default.svc.cluster.local:2379"
- name: ETCD_ROBUSTNESS_DATA_PATHS
value: "/var/etcddata-0,/var/etcddata-1,/var/etcddata-2"
- name: ETCD_ROBUSTNESS_REPORT_PATH
value: "/var/report/"
volumeMounts:
- name: etcd-0
mountPath: /var/etcddata-0
- name: etcd-1
mountPath: /var/etcddata-1
- name: etcd-2
mountPath: /var/etcddata-2
- name: report
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use hostvolume for report? What's the benefit of PVC here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking it might be better to keep things consistent than mixing between hostPath and PVC. Let me know if you want it to be hostPath instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer clear and understandable. While we need to use PVCs for etcd as there are 3 replicas, I don't see a reason why we cannot have a static path for client report.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to hostPath for report now.

mountPath: /var/report/
volumes:
- name: etcd-0
persistentVolumeClaim:
claimName: data-etcd-0
- name: etcd-1
persistentVolumeClaim:
claimName: data-etcd-1
- name: etcd-2
persistentVolumeClaim:
claimName: data-etcd-2
- name: report
hostPath:
path: /var/report/
type: DirectoryOrCreate
15 changes: 13 additions & 2 deletions tests/antithesis/test-template/entrypoint/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,23 @@ var NodeCount = "3"
func CheckHealth() bool {
cfg := common.MakeConfig(NodeCount)

var hosts []string
nodeOptions := []string{"etcd0", "etcd1", "etcd2"}[:cfg.NodeCount]
for _, n := range nodeOptions {
hosts = append(hosts, fmt.Sprintf("%s:2379", n))
}

// override hosts if environment variable is set
h, _, _ := common.PathsFromEnv()
if len(h) > 0 {
hosts = h
}

// iterate over each node and check health
for _, node := range nodeOptions {
for _, node := range hosts {
fmt.Printf("Client [entrypoint]: checking connection with %s\n", node)
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{fmt.Sprintf("http://%s:2379", node)},
Endpoints: []string{node},
DialTimeout: 5 * time.Second,
})
if err != nil {
Expand Down
29 changes: 28 additions & 1 deletion tests/antithesis/test-template/robustness/common/path.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package common
import (
"fmt"
"os"
"strings"
)

const (
Expand All @@ -34,8 +35,12 @@ const (
localetcd2 = "127.0.0.1:32379"
// used by default when running the client locally
defaultetcdLocalDataPath = "/tmp/etcddata%d"
localetcdDataPathEnv = "ETCD_ROBUSTNESS_DATA_PATH"
localetcdDataPathEnv = "ETCD_ROBUSTNESS_LOCAL_DATA_PATH"
localReportPath = "report"

serverEndpointEnv = "ETCD_ROBUSTNESS_SERVER_ENDPOINTS"
dataPathEnv = "ETCD_ROBUSTNESS_DATA_PATHS"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the benefit of separating ETCD_ROBUSTNESS_DATA_PATH and ETCD_ROBUSTNESS_LOCAL_DATA_PATH? From binary point of view both are local.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How the existing local data path is used is different from the new one. The existing one takes a path (more like a prefix) and then add counts to it. The new one no longer assumes such patterns and requires that all paths are provided as a comma separated value.

Also, I do not want to rework how the existing data path works in this PR to keep things simple. I can send a follow up PR to refactor and consolidate them.

Copy link
Member

@serathius serathius Sep 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid over-complicating the existing code, especially avoid introducing two mechanism for the same thing. If the PR is too complicated it means it should be split.

Copy link
Member Author

@nwnt nwnt Sep 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be reworking how the existing code currently handles the paths it uses as I mentioned here then. Right now all the paths are baked into the code. I will be sending a PR for that change before continuing this one.

reportPathEnv = "ETCD_ROBUSTNESS_REPORT_PATH"
)

func DefaultPaths(cfg *Config) (hosts []string, reportPath string, dataPaths map[string]string) {
Expand Down Expand Up @@ -64,3 +69,25 @@ func etcdDataPaths(dir string, amount int) map[string]string {
}
return dataPaths
}

func PathsFromEnv() (hosts []string, reportPath string, dataPaths map[string]string) {
if h, ok := os.LookupEnv(serverEndpointEnv); ok {
hosts = strings.Split(h, ",")
}
var data []string
if d, ok := os.LookupEnv(dataPathEnv); ok {
data = strings.Split(d, ",")
}

if len(hosts) != len(data) {
panic("The number of endpoints and data paths must be equal")
}

dataPaths = make(map[string]string)
for i := range hosts {
dataPaths[hosts[i]] = data[i]
}
reportPath = os.Getenv(reportPathEnv)

return hosts, reportPath, dataPaths
}
10 changes: 10 additions & 0 deletions tests/antithesis/test-template/robustness/finally/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,16 @@ func main() {
cfg := common.MakeConfig(NodeCount)

_, reportPath, dirs := common.DefaultPaths(cfg)

// override paths if environment variables are set
_, rp, d := common.PathsFromEnv()
if rp != "" {
reportPath = rp
}
if len(d) > 0 {
dirs = d
}

if *local {
_, reportPath, dirs = common.LocalPaths(cfg)
}
Expand Down
13 changes: 13 additions & 0 deletions tests/antithesis/test-template/robustness/traffic/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,19 @@ func main() {
cfg := common.MakeConfig(NodeCount)

hosts, reportPath, etcdetcdDataPaths := common.DefaultPaths(cfg)

// override paths if environment variables are set
h, rp, d := common.PathsFromEnv()
if len(h) > 0 {
hosts = h
}
if rp != "" {
reportPath = rp
}
if len(d) > 0 {
etcdetcdDataPaths = d
}

if *local {
hosts, reportPath, etcdetcdDataPaths = common.LocalPaths(cfg)
}
Expand Down