Skip to content

Update processing #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 72 commits into
base: main
Choose a base branch
from
Draft

Update processing #159

wants to merge 72 commits into from

Conversation

StefanoSperetta
Copy link
Member

@StefanoSperetta StefanoSperetta commented Mar 1, 2024 •
edited
Loading

Update the processing and displaying of the data:

  • reduce the number of allocation for XTCEParsers: now it happens only once per thread
  • only call the processing thread of satellites that actually have a processor
  • avoid calling processing threads for satellites that have no frame to process
  • reduce the number of connections to influxDB: now it happens twice per raw frame to be processed, maxing ~10 connections per second
  • cleanup Grafana dashboards

@StefanoSperetta
Adding extra help message to initbuckets.py
6a212e4
@StefanoSperetta
Set up the InfluxDB organization for grafana from environment variable
fa9edd5
@StefanoSperetta
Removing extra log messages
977c902
@StefanoSperetta
Setting display dates for Grafana dashboards
68d8d2e
@StefanoSperetta
Reduce the crowdsec log file size
3a6bd47
@StefanoSperetta
Solving the Django 4 deprecation warnings
1602483
@StefanoSperetta
Printing launch date and last received data in the main page
b414422
@StefanoSperetta
Increase the hasked_key size to 150 characters following djangorestfr…
55037bd 
…amework-api-key version 2.2.0
@StefanoSperetta
Store the user name with frames, not the Django ID
55a68c9
@StefanoSperetta
Start processign job imediatly when frames is received
1bd0bd0
@StefanoSperetta
Map zoom adjusted
7a15074
@StefanoSperetta
Cleanup debug messages
c52d490
@StefanoSperetta
Added norad id to satellite location api
3955e46
@StefanoSperetta
Adding satellite icon to map
956a5b1
@StefanoSperetta
Changed homepage main picture
1471898
@StefanoSperetta
Removing debug messages
70f8e50
@StefanoSperetta
Allow staff users to submit frames on behalf of other users
ad783e7
@StefanoSperetta
Added buffer reprocessing task
f067eba
@StefanoSperetta
Skip already processed messages in the query
263d19b
@StefanoSperetta
Added debug messages in scraper
d61339e
@StefanoSperetta
Speeding up buffer processing by ~4x
68d2b89
@StefanoSperetta
Removing debug logger calls
f80cbb7
@StefanoSperetta
Start all tasks once frames are received to allow multi-threaded proc…
a710be3 
…essing
@StefanoSperetta
Processing the raw bucket iteratively to allow multi-threaded processing
9d46229
@StefanoSperetta
Loading XTCETools only when starting the scheduler
a289b60
@StefanoSperetta
Removing debug logger call
581440c
@StefanoSperetta
Processing frames in chunks to be more responsive
02fe72c
@StefanoSperetta
Handle exceptions in the processign threads
d4f74e5
@StefanoSperetta
Removing bookkeeping files, frames to process fully handled via the d…
9cbd5a6 
…atabase
@StefanoSperetta
Removing debug logger messages
eba82d7
@StefanoSperetta
Bumping pgAdmin to version 8.5
ef91122
@StefanoSperetta
Merge branch 'main' into update_processing
b04f5f8
@StefanoSperetta
Changing thwe influxDB api to contain all database related code
a65ee80
@StefanoSperetta
Improving the Delfi-PQ dashboard
27ac4e7
@StefanoSperetta
Removing legend from graphs with a single value
8e7422f
@StefanoSperetta
Added total number of received frames
6ecf3b6
@StefanoSperetta
Restructured grafana dashboards
8085a01
@StefanoSperetta
Addind microseconds to the frames tables
e0a0fbe
@StefanoSperetta
Ignore fields marked as to be ignored
0ec7f81
@StefanoSperetta
Fix pylint warnings
c9d53ef
@StefanoSperetta
Remove debug log call
e991da7
@StefanoSperetta
Merge pull request #165 from iiacoban42/main
80cc534 
Pull from main
@StefanoSperetta
Update crowdsec and pgadmin
3e5e2ce
@StefanoSperetta
Set the user and application as tags in the raw frames table
2efb7f5
@StefanoSperetta
Merge branch 'main' into update_processing
ee6dda2
@StefanoSperetta
Fixex copy-paste error in documentation
56feef1
@StefanoSperetta
Fixed copy-paste error in documentation
aa7f624
@StefanoSperetta
Added missing step to local setup guide
3da3c9e
@StefanoSperetta
Added missing step to local setup guide
f6c9fca
@StefanoSperetta
Merge branch 'main' into update_processing
c16e23c
@StefanoSperetta
Do not create a Delfispace bucket anymore
8570ad9
@StefanoSperetta
Update crowdsec to latest version
5388f9f
@StefanoSperetta
Merge branch 'main' into update_processing
1ed9203
@StefanoSperetta
Merge branch 'main' into update_processing
4a05e1b
@StefanoSperetta
Fixing unit test username / usi swap
fc88604
@StefanoSperetta
Fixing unit test username / usi swap
07114a2
@StefanoSperetta
Merge branch 'main' into update_processing
781b56d
@StefanoSperetta
Merge branch 'main' into update_processing
d7dbd13
@StefanoSperetta
Merge branch 'main' into update_processing
4c89249
@StefanoSperetta
Pushing old commits to branch. To be cleaned up.
6781910
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@StefanoSperetta