[BLE] Add support for decrypting PVVX, BTHome v2 and Victron BLE frames

[plambrechtsen]

Add support for decrypting PVVX, BTHome v2 and Victron BLE encrypted frames

Description:

This adds quite a few new features

• Decrypting the 3 encrypted formats supported by decoder - PVVX, BTHome v2 and Victron BLE
• Updated UI for BLE to have a default AES key with field validation to ensure it's hex
• Updated UI for BLE support for multiple per-device macaddress:aeskey mapping with field validation

Checklist:

• The pull request is done against the latest development branch
• Only one feature/fix was added per PR and the code change compiles without warnings
• I accept the DCO.

[DigiH]

I'll leave the final review to @1technophile and @h2zero . Just a few comments after briefly going over the PR.

.I'm not a fan of the introduction of tempChar. And think the Metric and Imperial properties should be kept as they are for auto-discovery and JSON publishing. Explained in more detail in

#2218 (comment)

Currently it looks as if it's only checking for the presence of the "encr" property in the received JSON

if BLEDecryptor
  if (BLEdata["encr"]) {
    // Decrypting BTHome v2 payload
…

but without checking for the actual encryption scheme number - "encr":2 for BTHome v2 encryption, but currently also all LYWSD03MMC with PVVX native encryption "encr":1, and Victron Energy devices "encr":3 would trigger this and fail miserably.

I also see that you only have defined a single AES encryption bindkey as a build definition.
his might be fine for BTHome v2 only devices which are all set the the very same encryption key, but won't really work for a mix of encrypted devices, where different MAC address devices have different bindkeys, especially once the Victron devices are added which each have their own hard coded bindkey.

What really is need there is the possibility to define MACaddress:Bindkey pairs, as implemented in Theengs Gateway

https://gateway.theengs.io/use/use.html#reading-encrypted-advertisements

Even then, defining it as a build definition would restrict this to users being able to create their won builds. Not and issue for you and me, but for decryption to be included in an OMG release build I think this would need to be usable for all users, also the ones only able to install pre-built binaries.

So storing these MAC address:Bindkey pairs in flash would be one possibility, by sending them as an MQTT command and/or through the WebUI. Or someone else might have a better idea for it.

I greatly appreciate your effort to bring decryption to OMG, and I'm sure this is working great for you and others only using BTHome v2 devices from your comments in other closed issues - if they don't come back stating that they cannot make their own build ;)

For a releasable version though I think there are still some refinements required addressing the issues mentioned above.

[plambrechtsen]

Thanks for this, I completely agree with all your feedback above about the Metric vs Imperial and will adjust that. To me the "Display Metric" just didn't make any sense without going to the documentation so a select drop down makes more sense in my view.

Agree all three algorithms should be supported and it is something I will spend some time probably next weekend. I hadn't found the algorithms until you pointed me to the python code and now I know what is needed. https://github.com/theengs/gateway/blob/development/TheengsGateway/decryption.py

Was just looking for where the when I have some time. The PVVX build modules can have their bindkey updated so having a shared encryption key is possible. But since the Victron have a static key then I will add a default + MACaddress:Bindkey option in the UI plus in the code would be the way to implement it.

[plambrechtsen]

Hi @DigiH I took onboard your feedback and have added support for PVVX and Victron.

I have commented out my client side JavaScript which I created for field validation.

The one issue I do see is the theengs-bridge-v11 as it's right on the limit for flash.

Checking size .pio\build\theengs-bridge-v11\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  23.6% (used 77332 bytes from 327680 bytes)
Flash: [==========]  100.0% (used 1966076 bytes from 1966080 bytes)
Building .pio\build\theengs-bridge-v11\firmware.bin

[1technophile]

Thanks for your efforts on this.

The one issue I do see is the theengs-bridge-v11 as it's right on the limit for flash.

We can work on this in a separate PR

[1technophile]

Could you make this PR only about decryption and move the commits related to:

• UI support for changing the default naming from the BLE advertised name or model_id
• Changing the UI from the checkbox "Display metric" to a drop down select between Metric and Imperial
• Updated gatewayBT so that the Home Assistant entitles display as °C or °F
  In one or several separate PR ?

If you need help let me know.

[plambrechtsen]

Checking the logic for the Victron decoder and I found out that the Theengs gateway code is wrong for

https://github.com/theengs/gateway/blob/development/TheengsGateway/decryption.py#L145

It's actually needs to be 16 bytes as per:

https://github.com/keshavdv/victron-ble/blob/main/victron_ble/devices/base.py#L1035

Tested with the example messages in the tests from victron_ble and was getting the expected decrypted value.

Also shortened the logic for the client side Javascript code for the BLE Keys. This is commented out due to size. Also simplified he logic in there to reduce the size. Did you want me to commit the original html page I built to test?

[DigiH]

@plambrechtsen Doesn't seem to be a byte count issue after all ;)

As the Theengs Gateway code is running and decrypting actual Vitron devices fine in the wild, with nonce padding like

12ab should be padded to 12ab000000000000

I'm not quite sure what you mean with padding to 8 bytes is wrong and it should be 16 bytes.

[1technophile]

Great add, let me know if ready

[plambrechtsen]

@DigiH when I got back home yesterday I thought that I should find some example encrypted Victron frames and make sure they decrypt correctly using mbedtls.
When I tested with these keys: https://github.com/keshavdv/victron-ble/blob/main/tests/test_battery_monitor.py#L14-L15

    uint8_t key[16] = { 0xaf, 0xf4, 0xd0, 0x99, 0x5b, 0x7d, 0x1e, 0x17, 0x6c, 0x0c, 0x33, 0xec, 0xb9, 0xe7, 0x0d, 0xcd };
    uint8_t nonce[8] = { 0xb0, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
    uint8_t ciphertext[] = { 0x92, 0x5d, 0x09, 0xa4, 0xd8, 0x9a, 0xa0, 0x12, 0x8b, 0xde, 0xf4, 0x8c, 0x62, 0x98, 0xa9 );

Then I should have got this plaintext

ffffe50400000000030000f40140df

But I didn't.

However when I padded the nonce to 16 bytes.

    uint8_t key[16] = { 0xaf, 0xf4, 0xd0, 0x99, 0x5b, 0x7d, 0x1e, 0x17, 0x6c, 0x0c, 0x33, 0xec, 0xb9, 0xe7, 0x0d, 0xcd };
    uint8_t nonce[16] = { 0xb0, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
    uint8_t ciphertext[] = { 0x92, 0x5d, 0x09, 0xa4, 0xd8, 0x9a, 0xa0, 0x12, 0x8b, 0xde, 0xf4, 0x8c, 0x62, 0x98, 0xa9 );

Then I got the correct results. I checked out a number of other implementations and they all have a 16 and not 8 zero padded string.

I suspect it's because of the python library used in Theengs Gateway's that the decryption works so it is good luck rather than working correctly.

That's why I needed to change the nonce length to be 2 bytes of he IV + 14 zero padded bytes for the mbedtls function to work correctly.

I think it's good to merge @1technophile as I don't think there is anything further that needs to be added. The only thing would be around if you want the client side Javascript code included as part of the build or not. It adds 600 bytes or so but it does then make sure that the user enters the AES keypairs in the correct format.
https://github.com/1technophile/OpenMQTTGateway/pull/2219/files#diff-63c546bd51817ee14827dd8b6e3dd8c475942a00e1171f7e368da500d50b4a12R230

That is the only thing that you might need to think about the pros and cons of having that in the build or not. I also have a HTML page I built and tested the javascript in before I minified it. I didn't add that to the PR as I wasn't sure where it should go or if you wanted it at all. Let me know and I can add that in too if you want.

[DigiH]

I suspect it's because of the python library used in Theengs Gateway's that the decryption works so it is good luck rather than working correctly.

I also used this online decryption tool to verify some of the examples given by the Victron user

https://www.lddgo.net/en/encrypt/aes

and this Decryptor there worked fine with the 8 bytes padding I got from the Victron forum. Testing it again now with 16 bytes it also works. Only when padding to less than 8 bytes does it come up with

So different libraries indeed seem to behave slightly differently, especially since the padding, be it 8 bytes or 16 bytes or actually any byte number in between, is all with 0s ;)

For bindkey matching and full consistency we could also extend the Theengs gateway Python padding to 16 bytes. It will work either way, as with the online decryptor
.
With your example above, but with an 8 byte nonce/IV in the online decryption.

[DigiH]

Either way, thanks a lot for bringing this to OpenMQTTGateway! 👍

C++, Pythion, Java … as long as it works correctly on each platform :)

FYI - theengs/gateway#308

I think it'll be a great addition to the next release, as some users have already been wishing for being able to use encrypted devices on OMG.

[DigiH]

@1technophile - You want to create a separate test dev build for this for the Victron user to also try out, as @plambrechtsen has already verified with his LYWSD03MMC, or just merge it and then the nightly build can be used for Victron verification?

I've already contacted the Victron user.

[plambrechtsen]

Thanks for that @DigiH confirming the nonce length issue in AES-CTR I have also started separating the previous changes around the display name into separate PRs.
The first was a cosmetic issue where devices like the temperature sensor when they are added to Home Assistant don't link to OMG, and it creates a random orphaned entity as the OMG device id is the mac address and not the gateway_name.
That is in #2220

[DigiH]

The first was a cosmetic issue where devices like the temperature sensor when they are added to Home Assistant don't link to OMG, and it creates a random orphaned entity as the OMG device id is the mac address and not the gateway_name.
That is in #2220

Yes, this has been mentioned before.

I also want to look at the mix of written out and abbreviated property names in the discovery messages. Which apparently isn't officially supported, bu in HA itself seems to pass through, wile in other controllers, which (partially) support HA auto-discovery, actually breaks the discovery. That'll have to wait until I'm back at my proper dev station though.

[1technophile]

Thanks !

[1technophile]

@DigiH I restarted the nightly build so that you have the development ready with the PR tomorrow morning

[DigiH]

@plambrechtsen @1technophile

I just installed the latest development build on one of my production BLE gateways with all the recent additions.

I really do like the WebUI interface for entering the BLE AES Default Key and MAC:AES Key pairs 👍

Anything entered however is volatile and not persistent across restarts, which do happens occasionally on BLE gateways, more or less frequent depending on the individual setup.

I mentioned before that these would be stored in flash (similar to all the other settings) and then reloaded if and when a restart happens, as otherwise the decryption functionality and the related devices will get lost at any time due to a random restart.

[plambrechtsen]

This is where the saveConfig() function is called @DigiH so as long as you don't have ESPWifiManualSetup then it should work fine.

[DigiH]

@plambrechtsen I overlooked that comment before, but that is what Iand many other users have, plus me also the parent MQTT_HTTPS_FW_UPDATE undefined ;)

[plambrechtsen]

And this is where the config gets loaded back in @DigiH from the /config.json

[plambrechtsen]

Hi @DigiH it should be saving config.

@plambrechtsen @1technophile

I just installed the latest development build on one of my production BLE gateways with all the recent additions.

I really do like the WebUI interface for entering the BLE AES Default Key and MAC:AES Key pairs 👍

Anything entered however is volatile and not persistent across restarts, which do happens occasionally on BLE gateways, more or less frequent depending on the individual setup.

I mentioned before that these would be stored in flash (similar to all the other settings) and then reloaded if and when a restart happens, as otherwise the decryption functionality and the related devices will get lost at any time due to a random restart.

It should be working as I have tested saving the config to the config.json on SPIFFS

Try uncommenting the serializeJsonPretty(json, Serial); at: https://github.com/1technophile/OpenMQTTGateway/blob/development/main/main.cpp#L2073

And add a serializeJsonPretty(json, Serial); just before saving the config at: https://github.com/1technophile/OpenMQTTGateway/blob/development/main/main.cpp#L2042

Both should show the config being loaded and saved.

[DigiH]

It should be working as I have tested saving the config to the config.json on SPIFFS

Are they boing loaded at startup again though?

Does it work for you when you enter a BLE AES Default Key different to the default one – I just changed the leading three octets to aabbcc

and a MAC:AES Key pair – I used a default MAC address without colons:Victron sample key.

Then hit Save
They are nicely still there when returning to the mani Manu or else and coming back.

But issuing a restart command to my gateway, the BLE config page was all reset to the default.

But as I do not have any encryption capable devices I couldn't try and see if the functionality might still be there and it is just a UI issue, which might be a possibility as well :)

[plambrechtsen]

The default needs to be a 32 character hex string. And then key pairs needs to be 12 characters hex for the mac without colons and then a : as a delimiter, then 32 characters hex for the key. And a space (not new line) to delimit each entry.

If you try uncommenting line https://github.com/1technophile/OpenMQTTGateway/blob/development/main/config_WebContent.h#L230 as that has client side validation javascript of both values in the browser for the default key and key pairs. Which I had to comment out due to size restrictions on the theengs plug, but it works on most other ESP32s.

[DigiH]

Code wise, all nice and good - not really looking at code tonight - in a Sunday evening TV series mode here right now ;)

So I only quickly installed the latest dev build earlier tonight, and tried entering something in both fields.

Fine when coming back from other UI pages – the entered details are still there, and I assume work correctly.

Issue a restart command to the gateway, log into the WebUI again and the BLE config page is reset to the defaults.

Is it a functionality issue or only a still confusing UI issue – I cannot tell without any decryption capable devices.

How does this work for you on a gateway at runtime? Or have you only tested it with the default build coded default AES key, which I assume you are using for all your devices?

You are submitting great new functionalities and fixes at incredible speed over the last days!
I just think we also need to take the time to make sure things are working fine for other users with different setup and configs as well.

So how does it behave at runtime for you with different details?

[plambrechtsen]

When I set the correct values it is working for me. Just built on a new ESP32 (M5Stack Core2) and it works as expected.

Hitting save on the BLE menu.

T: saving configs
N: Total size: 1894
{
  "ota_server_cert": "
...
  "ota_pass": "password",
  "ble_aes": "00000000000000000000000000000000",
  "ble_aes_keys": {}
}T: Screen wake up

And then restarting and seeing it being loaded back in.

************* WELCOME TO OpenMQTTGateway **************
N: SYS config not found
..
T: Intro display on screen
{
  "ota_server_cert": "
...
  "ota_pass": "password",
  "ble_aes": "00000000000000000000000000000000",
  "ble_aes_keys": {}
}

And adding AES Keys also get saved.

T: saving configs
N: Total size: 1894
{
  "ota_server_cert": "
...
  "ota_pass": "password",
  "ble_aes": "00000000000000000000000000000000",
  "ble_aes_keys": {
    "000000000001": "00000000000000000000000000000001",
    "000000000002": "00000000000000000000000000000002"
  }
}T: Screen wake up

And shows on screen post reboot.

[DigiH]

Hmmmm - very strange.

I'm still seeing it reset to defaults every single time.

The only difference I can think of now is that I updated over an existing setup running gateway, whereas you might be running your development version with already dditional changes not merged into OMG development yet?

Or did you also test with the current OMG development state – 6d1b793?

I will look at this again tomorrow. Maybe @1technophile can also see which behaviour he is getting.

[plambrechtsen]

I built from the latest development branch. Making only the tweaks I talked about enabling the client side JavaScript validation and logging the config file on save and load.

git log -1
commit 6d1b7935c4ab2bc73a0fb6e8abcacfb5b8b670d1 (HEAD -> development, origin/development, origin/HEAD)
Author: Peter Lambrechtsen <[email protected]>
Date:   Mon Aug 11 03:34:17 2025 +1200

    [DISC] Replace Home Assistant Auto Discovery MQTT keys with shortened abbreviations (#2221)

Update config_WebContent.h and enable the WebUI JavaScript validation by uncommenting the line:

-//const char ble_script[] = "function bkv....
+const char ble_script[] = "function bkv....
-const char ble_script[] = "";
+// const char ble_script[] = "";

Update main.cpp to log the config.json to log the output of load and save.

git diff
diff --git a/main/main.cpp b/main/main.cpp
index 8f75b30..29392e0 100644
--- a/main/main.cpp
+++ b/main/main.cpp
@@ -2033,6 +2033,7 @@ void saveConfig() {
   json["ble_aes_keys"] = ble_aes_keys;
 #  endif

+  serializeJsonPretty(json, Serial);
   File configFile = SPIFFS.open("/config.json", "w");
   if (!configFile) {
     Log.error(F("failed to open config file for writing" CR));
@@ -2070,7 +2071,7 @@ bool loadConfigFromFlash() {
       if (!json.isNull()) {
         Log.trace(F("\nparsed json, size: %u" CR), json.memoryUsage());
         // Print json to serial port
-        //serializeJsonPretty(json, Serial);
+        serializeJsonPretty(json, Serial);

 #  if !MQTT_BROKER_MODE
         for (int i = 0; i < 3; ++i) {

Then it worked fine for me.

[DigiH]

My misunderstanding then.

I assumed the merged state was functional, and that the disabled JavaScript validation was only that, an optional validation of the entered keys, but not necessary for functionality.

[DigiH]

As the guy with the Victron devices wanted to install and run the development test build binaries I told him to hold off for the time being.

@plambrechtsen Let us know when the development binaries will have the additions included to address this.

[DigiH]

Ok, retested this issue I was/am seeing, and pinned it down to the following.

A default esp32dev-ble environment build of the latest OMG development state does actually work fine and keeps the entered Default Key and MAC:Binkey pair stored over restarts,

BUT installing the same latest OMG development with my custom environment setup, as I did for the initial testing on one of my production BLE gateways, the volatility of the keys is reproducible.

FYI I use
'-DESPWifiManualSetup=true' - a very common occurrence for anyone with multiple gateways, being able to define the WiFi and MQTT broker credentials for all gateways without having to go through any initial provisioning after flashing the firmware.
See the condition for the config save

#  ifndef ESPWifiManualSetup
          saveConfig();
#  endif

with that ifndef also being part of the larger

#ifdef MQTT_HTTPS_FW_UPDATE
…

but I also undefine this for all my custom environments
'-UMQTT_HTTPS_FW_UPDATE' ;We remove this function to have sufficient FLASH available for OTA, you should also use ESPWifiManualSetup and 'board_build.ldscript = eagle.flash.1m64.ld' to save flash memory and have OTA working
simply for the fact which is explained in the comment ;)

I'm also using '-DNetworkAdvancedSetup=true' with all its network definitions and several other build flags, but haven't looked any further where these might also affect this issue.

The conclusion being - the keys will be saved and loaded fine for any user installing the pre-built binaries or building from the pre-defined environments, but for this to work for all possible custom environments the saving and loading of the details needs to be placed somewhere where it cannot mistakenly be disable by some unrelated build flag.

Just for clarification ;) the save and load for the WiFi/MQTT Broker details will obviously need to stay as they are, just the new AES key/.pairs will need to be saved and loaded independently of any build flag setting.

[DigiH]

I'm sure the save/load issue will be addressed at some stage so that everyone will be able to use decryption.

As the Victron user is using pre-built binaries I asked him to start testing today.

With Trace logging it became apparent that there isn't even an attempt to try to decrypt the Victron devices ;) Small typo for which I submitted

#2226