Skip to content

Commit b9ec923

Browse files
pspoerripspoerri
committed
WIP: GEDS Daemon Mode.
1 parent ad34f6c commit b9ec923

File tree

1 file changed

+146
-0
lines changed

1 file changed

+146
-0
lines changed

‎doc/GEDS Daemon Mode.md

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# GEDS Daemon Mode Redesign.
2+
3+
GEDS shall be redesigned to support multiple local instances. A central daemon is responsible
4+
for managing files and sharing files between other daemons.
5+
GEDS clients write and read to a mounted CSI hostpath which shares the data between pods.
6+
NVMeoF and GPU Direct RDMA should be supported in a second step.
7+
8+
Goal:
9+
- Enable data-sharing of GEDS objects between multiple Python processes.
10+
- Enable data-sharing of GEDS objects across Kubernetes pods using a CSI hostpath.
11+
- Data should survive crashes.
12+
- Facilitate encryption.
13+
- Enable reading/writing data to/from GPU and/or NVMe directly
14+
- Native integration with Kubernetes
15+
16+
17+
## Overview
18+
19+
The diagram shows a high-level overview.
20+
```
21+
┌────────────────────────────────────────────────────────────────────────────┐
22+
│ Kubernetes Node │
23+
│ ┌────────────────────────────────────────┐ │
24+
│ │ Kubernetes Pod │ │
25+
│ IPC │ ┌─────────────┐ ┌─────────────────┐ │ │
26+
│ ┌───────────┼───► GEDS Client │ │ Apache Spark │ │ │
27+
│ │ │ └─────────────┘ └─────────────────┘ │ │
28+
│ │ │ │ │
29+
│ ┌─────────────────┼────┐ └────────────────────────────────────────┘ │
30+
│ │ Kubernetes Pod │ │ │
31+
│ │ ┌──────────────▼──┐ │ ┌────────────────────────────────────────┐ │
32+
│ │ │ GEDS Daemon ◄─┼──┐ │ Kubernetes Pod │ │
33+
│ │ └─────────────────┘ │ │ │ ┌─────────────┐ ┌──────────────────┐ │ │
34+
│ │ │ ├───┼───► GEDS Client │ │ Python + Pytorch │ │ │
35+
│ │ ┌─────────────────┐ │ │ │ └─────────────┘ └──────────────────┘ │ │
36+
│ │ │ PVC / Local SSD │ │ │ │ │ │
37+
│ │ └─────────────────┘ │ │ │ ┌─────────────┐ ┌──────────────────┐ │ │
38+
│ │ │ └───┼───► GEDS Client │ │ Python + Pytorch │ │ │
39+
│ └────────────────▲─────┘ │ └─────────────┘ └──────────────────┘ │ │
40+
│ │ │ │ │
41+
│ │ └────────────────────────────────────────┘ │
42+
│ │ TCP/IP │
43+
│ │ │
44+
└───────────────────┼────────────────────────────────────────────────────────┘
45+
│
46+
┌───────────────────┼────────────────────────────────────────────────────────┐
47+
│ Kubernetes Node │ │
48+
│ │ │
49+
│ ┌────────────────▼─────┐ ┌────────────────────────────────────┐ │
50+
│ │ Kubernetes Pod │ │ Kubernetes Pod │ │
51+
│ │ ┌─────────────────┐ │ │ ┌───────────────────────────────┐ │ │
52+
│ │ │ GEDS Daemon │ │ │ │ GEDS Client │ │ │
53+
│ │ └─────────────────┘ │ │ └─────▲────────────────▲────▲───┘ │ │
54+
│ │ │ │ │ I/O + MMAP │ │ │ │
55+
│ │ ┌─────────────────┐ │Mounts│ ┌─────▼──────────┐ │ │ │ │
56+
│ │ │ PVC / Local SSD ◄─┼──────┼──┤ Hostpath CSI │ │ │ │ │
57+
│ │ └─────────────────┘ │ │ └────────────────┘ │ │ │ │
58+
│ │ │ │ │ │ │ │
59+
│ └──────────────────────┘ └─────────────────────────┼────┼─────┘ │
60+
│ │ │ │
61+
│ GPU Direct │ │NVMeOF │
62+
│ ┌─────────────────────────▼┐ │ │
63+
│ │ GPU │ │ │
64+
│ └───────────────────▲──────┘ │ │
65+
│ │ │ │
66+
└────────────────────────────────────────────────────┼──────────┼────────────┘
67+
│ │
68+
GPU Direct RDMA │ │
69+
┌────────────────────────────────────────────────────▼──────────▼────────────┐
70+
│ NVMe JBOD │
71+
│ │
72+
│ │
73+
│ │
74+
│ │
75+
│ │
76+
└────────────────────────────────────────────────────────────────────────────┘
77+
```
78+
79+
## Operations
80+
81+
### Creating a file
82+
83+
1. GEDS Creates file with `BUCKET/PATH` and `UUID`.
84+
2. GEDS Client creates a temporary file in the hostpath volume.
85+
3. GEDS Client writes to the file.
86+
4. GEDS Client seals the file. This transfers the ownership of the file to the
87+
GEDS daemon. FD is kept open and the daemon does a rename.
88+
5. GEDS Daemon tells the metadata server that a new file has been
89+
created.
90+
91+
Conflict resolution:
92+
1. If no file with `BUCKET/PATH` does exist, the entry is created.
93+
2. If `BUCKET/PATH` already exists, then the existing entry is overridden.
94+
95+
6. GEDS Client reopens the file `Read-Only` and opens a file-descriptor.
96+
7. Other GEDS Clients are now able to open this file.
97+
98+
### Opening a file
99+
100+
1. GEDS Client tells the daemon that it wants to open `FILE`.
101+
2. GEDS Daemon:
102+
1. Checks with Metadata Server if the `BUCKET/PATH` exists. Receives `UUID` and locations.
103+
2. Increases the usage-count count of the filehandle to prevent deletion.
104+
3. Tells the `PATH` and size of `FILE` to the `Client`.
105+
106+
### Reading from a file
107+
108+
A file can be on one of the following locations:
109+
110+
- **Local**, if the file was created by a client on the same node, or if it is cached.
111+
- **DFS**, if the file was stored on a distributed file system.
112+
- **S3**, if the file is stored in a bucket on a S3-like service.
113+
- **Remote**, if the file is stored on a different GEDS Daemon.
114+
115+
The GEDS Client will forward all reads to the GEDS Dameon for **Remote** files. All other file-types can be read directly by the GEDS Client.
116+
117+
118+
### Closing a file
119+
120+
1. GEDS Client tells the daemon that it closed the `BUCKET/PATH`.
121+
2. GEDS Daemon:
122+
1. Decreases the file-counter.
123+
2. If the file-counter is zero and the file is marked as deleted, delete the file.
124+
125+
### Deleting a file
126+
127+
1. GEDS Client tells the daemon that it wants to delete `BUCKET/PATH`.
128+
2. GEDS Client closes local FD.
129+
2. GEDS Daemon:
130+
1. Decreases reference counter for open `UUID`.
131+
2. Tells the metadata server that the path `BUCKET/PATH` has been deleted.
132+
3. Marks file as deleted. Delete the file if the reference counter is zero.
133+
134+
### Garbage collection
135+
136+
1. A daemon process automatically checks local file-usage.
137+
2. If the defined quota is reached no files can be created.
138+
3. Local copies can be deleted if:
139+
1. The file is marked as deleted.
140+
2. The file is not open and other copies exist (UUID matches).
141+
3. A previous version of the file is not referenced any more.
142+
143+
## Technical Implementations
144+
145+
- The TCP/IP transport is a GEDS Client as well. This way the transport requires no extra
146+
file/open/close mechanisms.

0 commit comments

Comments
 (0)