Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
6a212e4
Adding extra help message to initbuckets.py
StefanoSperetta Mar 1, 2024
fa9edd5
Set up the InfluxDB organization for grafana from environment variable
StefanoSperetta Mar 1, 2024
977c902
Removing extra log messages
StefanoSperetta Mar 1, 2024
68d8d2e
Setting display dates for Grafana dashboards
StefanoSperetta Mar 1, 2024
3a6bd47
Reduce the crowdsec log file size
StefanoSperetta Mar 3, 2024
1602483
Solving the Django 4 deprecation warnings
StefanoSperetta Mar 3, 2024
b414422
Printing launch date and last received data in the main page
StefanoSperetta Mar 3, 2024
55037bd
Increase the hasked_key size to 150 characters following djangorestfr…
StefanoSperetta Mar 4, 2024
55a68c9
Store the user name with frames, not the Django ID
StefanoSperetta Mar 4, 2024
1bd0bd0
Start processign job imediatly when frames is received
StefanoSperetta Mar 14, 2024
7a15074
Map zoom adjusted
StefanoSperetta Mar 22, 2024
c52d490
Cleanup debug messages
StefanoSperetta Mar 22, 2024
3955e46
Added norad id to satellite location api
StefanoSperetta Mar 22, 2024
956a5b1
Adding satellite icon to map
StefanoSperetta Mar 22, 2024
1471898
Changed homepage main picture
StefanoSperetta Mar 22, 2024
70f8e50
Removing debug messages
StefanoSperetta Mar 22, 2024
ad783e7
Allow staff users to submit frames on behalf of other users
StefanoSperetta Mar 23, 2024
f067eba
Added buffer reprocessing task
StefanoSperetta Mar 23, 2024
263d19b
Skip already processed messages in the query
StefanoSperetta Mar 23, 2024
d61339e
Added debug messages in scraper
StefanoSperetta Mar 23, 2024
68d2b89
Speeding up buffer processing by ~4x
StefanoSperetta Mar 24, 2024
f80cbb7
Removing debug logger calls
StefanoSperetta Mar 25, 2024
a710be3
Start all tasks once frames are received to allow multi-threaded proc…
StefanoSperetta Mar 25, 2024
9d46229
Processing the raw bucket iteratively to allow multi-threaded processing
StefanoSperetta Mar 25, 2024
a289b60
Loading XTCETools only when starting the scheduler
StefanoSperetta Mar 29, 2024
581440c
Removing debug logger call
StefanoSperetta Mar 29, 2024
02fe72c
Processing frames in chunks to be more responsive
StefanoSperetta Mar 29, 2024
d4f74e5
Handle exceptions in the processign threads
StefanoSperetta Mar 30, 2024
9cbd5a6
Removing bookkeeping files, frames to process fully handled via the d…
StefanoSperetta Mar 30, 2024
eba82d7
Removing debug logger messages
StefanoSperetta Mar 30, 2024
303bdc0
Fix pylint warnings
StefanoSperetta Mar 30, 2024
721ce49
Fixing pylint warnings
StefanoSperetta Mar 30, 2024
56846d5
Fixing pyling warnings
StefanoSperetta Mar 31, 2024
767035b
Fixing remaining pylint warnings
StefanoSperetta Mar 31, 2024
e24cbc6
Addid callback after raw bucket processing
StefanoSperetta Apr 1, 2024
aec8bd8
Re-adding first row deleted by mistake and adding more comments
StefanoSperetta Apr 1, 2024
3b346b6
Fixing the map wrap around
StefanoSperetta Apr 2, 2024
074403f
Only add a satellite to the list if the frame was processd succesfully
StefanoSperetta Apr 3, 2024
1739329
Restructuring telemetry stored in influxdb
StefanoSperetta Apr 6, 2024
703f4cb
Removing dashboard no more used
StefanoSperetta Apr 6, 2024
ea58d36
Changing influxdb_api to class to simplify code
StefanoSperetta Apr 13, 2024
0a8a055
Fixing popup not moving with satellite
StefanoSperetta Apr 13, 2024
ef91122
Bumping pgAdmin to version 8.5
StefanoSperetta Apr 13, 2024
b04f5f8
Merge branch 'main' into update_processing
StefanoSperetta Apr 13, 2024
a65ee80
Changing thwe influxDB api to contain all database related code
StefanoSperetta Apr 22, 2024
27ac4e7
Improving the Delfi-PQ dashboard
StefanoSperetta Apr 22, 2024
8e7422f
Removing legend from graphs with a single value
StefanoSperetta Apr 24, 2024
6ecf3b6
Added total number of received frames
StefanoSperetta Apr 24, 2024
8085a01
Restructured grafana dashboards
StefanoSperetta Apr 29, 2024
e0a0fbe
Addind microseconds to the frames tables
StefanoSperetta May 11, 2024
0ec7f81
Ignore fields marked as to be ignored
StefanoSperetta May 11, 2024
c9d53ef
Fix pylint warnings
StefanoSperetta May 11, 2024
e991da7
Remove debug log call
StefanoSperetta May 11, 2024
80cc534
Merge pull request #165 from iiacoban42/main
StefanoSperetta May 11, 2024
3e5e2ce
Update crowdsec and pgadmin
StefanoSperetta May 19, 2024
2efb7f5
Set the user and application as tags in the raw frames table
StefanoSperetta May 19, 2024
ee6dda2
Merge branch 'main' into update_processing
StefanoSperetta Jun 29, 2024
56feef1
Fixex copy-paste error in documentation
StefanoSperetta Jun 29, 2024
aa7f624
Fixed copy-paste error in documentation
StefanoSperetta Jun 29, 2024
3da3c9e
Added missing step to local setup guide
StefanoSperetta Jun 29, 2024
f6c9fca
Added missing step to local setup guide
StefanoSperetta Jun 29, 2024
c16e23c
Merge branch 'main' into update_processing
StefanoSperetta Jul 5, 2024
8570ad9
Do not create a Delfispace bucket anymore
StefanoSperetta Jul 7, 2024
5388f9f
Update crowdsec to latest version
StefanoSperetta Jul 7, 2024
1ed9203
Merge branch 'main' into update_processing
StefanoSperetta Jul 15, 2024
4a05e1b
Merge branch 'main' into update_processing
StefanoSperetta Jul 17, 2024
fc88604
Fixing unit test username / usi swap
StefanoSperetta Jul 17, 2024
07114a2
Fixing unit test username / usi swap
StefanoSperetta Jul 17, 2024
781b56d
Merge branch 'main' into update_processing
StefanoSperetta Aug 5, 2024
d7dbd13
Merge branch 'main' into update_processing
StefanoSperetta Oct 3, 2024
4c89249
Merge branch 'main' into update_processing
StefanoSperetta Dec 29, 2024
6781910
Pushing old commits to branch. To be cleaned up.
StefanoSperetta Jun 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,6 @@ RUN chown -R user:user /app
RUN chmod -R 755 /var/log/django
RUN chown -R user:user /var/log/django

RUN chown -R user:user /app/transmission/processing/temp
RUN chmod -R 755 /app/transmission/processing/temp
RUN chown -R user:user /app/home/temp
RUN chmod -R 755 /app/home/temp

Expand Down
15 changes: 9 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,25 @@ The dependencies are generated by running `./create_python_requirements`. This a
3. Install the requirements (one time instruction):
`pip install -r requirements.txt -r requirements_dev.txt`

4. Set up the database via docker or connect your own Postgres instance
4. Set up the database via docker
`docker compose up db`

5. Run the migrations from the root folder:
`python src/manage.py migrate`

6. Create the InfluxDB buckets:
6. Set up InfluxDB via docker
`docker compose up influxdb`

7. Create the InfluxDB buckets:
`python src/manage.py initbuckets`

7. Run the server from the root folder:
8. Run the server from the root folder:
`python src/manage.py runserver` The server runs on http://127.0.0.1:8000/

8. To run the tests:
9. To run the tests:
`python src/manage.py test`

9. To run pylint:
10. To run pylint:
`find src -name "*.py" | xargs pylint`

## Setup (Run via docker)
Expand Down Expand Up @@ -115,7 +118,7 @@ CROWDSEC_LAPI=

8. Generate a django keys with `python manage.py djecrety` and copy it to the .env file.

9. Create the InfluxDB buckets: `python src/manage.py initbuckets`
9. Create the InfluxDB buckets: `python manage.py initbuckets`

Note: remove `--build` to skip building the container, will use the cached one (last build)

Expand Down
8 changes: 6 additions & 2 deletions docker-compose-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ services:
driver: "json-file"
options:
max-file: "5" # file count
max-size: "10m" # file size
max-size: "1m" # file size


proxy:
Expand Down Expand Up @@ -51,7 +51,11 @@ services:

pgadmin:
container_name: pgadmin4_container
<<<<<<< HEAD
image: dpage/pgadmin4:8.6
=======
image: dpage/pgadmin4:8.9
>>>>>>> 4c892495399d42d7e674640b738a495bfe09dc8b
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL:[email protected]}
Expand All @@ -68,7 +72,7 @@ services:


crowdsec:
image: crowdsecurity/crowdsec:v1.6.0-1
image: crowdsecurity/crowdsec:v1.6.2
restart: always
ports:
- "8080:8080"
Expand Down
7 changes: 3 additions & 4 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ services:
build:
context: .
volumes:
- ./src/transmission/processing/temp/:/app/transmission/processing/temp/
- ./src/home/temp/:/app/home/temp/
environment:
- ALLOWED_HOSTS=${ALLOWED_HOSTS:-localhost,127.0.0.1}
Expand All @@ -32,7 +31,6 @@ services:
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-postgres}
- POSTGRES_HOST=${POSTGRES_HOST:-db}
- POSTGRES_PORT=${POSTGRES_PORT:-5432}
- INFLUXDB_V2_ORG=${INFLUXDB_V2_ORG:-DelfiSpace}
- INFLUXDB_V2_TOKEN=${INFLUXDB_V2_TOKEN:-adminpwd}
restart: always
depends_on:
Expand All @@ -56,8 +54,8 @@ services:
- DOCKER_INFLUXDB_INIT_MODE=setup
- DOCKER_INFLUXDB_INIT_USERNAME=${INFLUX_USERNAME:-admin}
- DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUX_PASSWORD:-adminpwd}
- DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_V2_ORG:-DelfiSpace}
- DOCKER_INFLUXDB_INIT_BUCKET=${INFLUX_BUCKET:-default}
- DOCKER_INFLUXDB_INIT_ORG=default
- DOCKER_INFLUXDB_INIT_BUCKET=default
- DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${INFLUXDB_V2_TOKEN:-adminpwd}
restart: always
logging:
Expand All @@ -80,6 +78,7 @@ services:
environment:
- GF_SERVER_DOMAIN=${GF_SERVER_DOMAIN:-localhost}
- GF_INFLUXDB_V2_TOKEN=${INFLUXDB_V2_TOKEN:-adminpwd}
- INFLUXDB_V2_ORG=${INFLUXDB_V2_ORG:-DelfiSpace}
restart: always
logging:
driver: "json-file"
Expand Down
15 changes: 9 additions & 6 deletions docs/deployment_intructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,22 +32,25 @@ The dependencies are generated by running `./create_python_requirements`. This a
3. Install the requirements (one time instruction):
`pip install -r requirements.txt -r requirements_dev.txt`

4. Set up the database via docker or connect your own Postgres instance
4. Set up the database via docker
`docker compose up db`

5. Run the migrations from the root folder:
`python src/manage.py migrate`

6. Create the InfluxDB buckets:
6. Set up InfluxDB via docker
`docker compose up influxdb`

7. Create the InfluxDB buckets:
`python src/manage.py initbuckets`

7. Run the server from the root folder:
8. Run the server from the root folder:
`python src/manage.py runserver` The server runs on http://127.0.0.1:8000/

8. To run the tests:
9. To run the tests:
`python src/manage.py test`

9. To run pylint:
10. To run pylint:
`find src -name "*.py" | xargs pylint`

## Setup (Run via docker)
Expand Down Expand Up @@ -118,7 +121,7 @@ CROWDSEC_LAPI=

8. Generate a django keys with `python manage.py djecrety` and copy it to the .env file.

9. Create the InfluxDB buckets: `python src/manage.py initbuckets`
9. Create the InfluxDB buckets: `python manage.py initbuckets`

Note: remove `--build` to skip building the container, will use the cached one (last build)

Expand Down
Loading
Loading