[Questions] Cannot upgrade from v3.13 to v4.0 "bad_config_entry_decoder,missing_passphrase" due to a colon in a generated password #14233
Closed
Answered
by
Joren-Thijs-KasparSolutions
Joren-Thijs-KasparSolutions
asked this question in
Questions
-
Community Support Policy
RabbitMQ version used4.0.9 Erlang version used27.3.x Operating system (distribution) usedLinux How is RabbitMQ deployed?Kubernetes Operator(s) from Team RabbitMQ rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics I couldn't run this on node 3 since it keeps crashin so I ran these commands on node 1. rabbitmq-diagnostics statusrabbitmq-diagnostics cluster_statusLogs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs =INFO REPORT==== 15-Jul-2025::07:18:23.665360 ===
alarm_handler: {set,{system_memory_high_watermark,[]}}
BOOT FAILED
===========
Exception during startup:
throw:{bad_config_entry_decoder,missing_passphrase}
rabbit_prelaunch_conf:decrypt_app/3, line 514
rabbit_prelaunch_conf:decrypt_config/2, line 494
rabbit_prelaunch_conf:setup/1, line 59
rabbit_prelaunch:do_run/0, line 113
rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
supervisor:do_start_child_i/3, line 959
supervisor:do_start_child/3, line 945
supervisor:-start_children/2-fun-0-/3, line 929
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0>
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> BOOT FAILED
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> ===========
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> Exception during startup:
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0>
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> throw:{bad_config_entry_decoder,missing_passphrase}
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0>
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch_conf:decrypt_app/3, line 514
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch_conf:decrypt_config/2, line 494
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch_conf:setup/1, line 59
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch:do_run/0, line 113
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> supervisor:do_start_child_i/3, line 959
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> supervisor:do_start_child/3, line 945
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> supervisor:-start_children/2-fun-0-/3, line 929
2025-07-15 07:18:24.599845+00:00 [error] <0.153.0>
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> supervisor: {local,rabbit_prelaunch_sup}
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> errorContext: start_error
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> reason: {bad_config_entry_decoder,missing_passphrase}
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> offender: [{pid,undefined},
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {id,prelaunch},
2025-07-15 07:18:25.614599+00:00 [error[] <0.153.0> {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,]}},
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {restart_type,transient},
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {significant,false},
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {shutdown,5000},
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {child_type,worker}]
2025-07-15 07:18:25.614599+00:00 [error] <0.153.0>
2025-07-15 07:18:25.615180+00:00 [notice[] <0.45.0> Application rabbitmq_prelaunch exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{bad_config_entry_decoder,missing_passphrase}}},{rabbit_prelaunch_app,start,[normal,[]]}}
{exit,terminating,[{application_controller,call,2,[{file,"application_controller.erl"},{line,511}]},{application,'-ensure_all_started/3-lc$^0/1-0-',1,[{file,"application.erl"},{line,367}]},{application,ensure_all_started,3,[{file,"application.erl"},{line,367}]},{rabbit,'-start_it/1-fun-0-',1,[{file,"rabbit.erl"},{line,423}]},{timer,tc,2,[{file,"timer.erl"},{line,595}]},{rabbit,start_it,1,[{file,"rabbit.erl"},{line,421}]},{init,start_it,1,[]},{init,start_em,1,]}]}
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> crasher:
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> initial call: application_master:init/3
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> pid: <0.151.0>
2025-07-15 07:18:25.615961+00:00 [error[] <0.151.0> registered_name: ]
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> exception exit: {{shutdown,
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> {failed_to_start_child,prelaunch,
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> {bad_config_entry_decoder,missing_passphrase}}},
2025-07-15 07:18:25.615961+00:00 [error[] <0.151.0> {rabbit_prelaunch_app,start,[normal,[]]}}
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> in function application_master:init/3 (application_master.erl, line 143)
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> ancestors: [application_controller,<0.10.0>]
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> message_queue_len: 1
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> messages: [{'EXIT',<0.152.0>,normal}]
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> links: [<0.45.0>]
2025-07-15 07:18:25.615961+00:00 [error[] <0.151.0> dictionary: ]
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> trap_exit: true
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> status: running
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> heap_size: 233
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> stack_size: 29
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> reductions: 68
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> neighbours:
2025-07-15 07:18:25.615961+00:00 [error] <0.151.0>
2025-07-15 07:18:25.629024+00:00 [notice] <0.84.0> alarm_handler: {clear,system_memory_high_watermark}
Kernel pid terminated (application_controller) ("{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{bad_config_entry_decoder,missing_passphrase}}},{rabbit_prelaunch_app,start,[normal,[]]}}}")
stream closed EOF for kaspar123-test/rabbit-mq-server-2 (rabbitmq)rabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location 10-defaults.conf10-operatorDefaults.conf11-default_user.conf90-userDefinedConfiguration.confSteps to deploy RabbitMQ clusterOur RabbitMQ cluster is controlled by A rabbitMQ operator on kubernetes. We specify the version of RabbitMQ we want and the required plugins. Then the operator will perform a Rolling upgrade of each Rabbitmq node until each node in the cluster has the same version. The operator config is a yaml file that is applied to the AKS cluster by a Azure Release pipeline. The pipeline uses the kubectl apply command. Steps to reproduce the behavior in questionCurrrently the cluster is running v3.13. When I upgrade the cluster operator config to use Rabbitmq:4.0-management-alpine, the first node is upgraded but fails to start due to a advanced.configN/A Application codeN/A Kubernetes deployment fileapiVersion: v1
metadata:
name: rabbit-mq-default-user
data:
default_user.conf: ${default_user_conf_b64}$
username: ${rabbitmq_username_b64}$
password: ${rabbitmq_password_b64}$
provider: cmFiYml0bXE= # base64 encoded 'rabbitmq' is always the same
type: cmFiYml0bXE= # base64 encoded 'rabbitmq' is always the same
kind: Secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rabbit-mq
labels:
app: rabbit-mq
annotations:
kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- ${HOSTNAME}$
secretName: common-tls-secret
rules:
- host: ${HOSTNAME}$
http:
paths:
- path: /rmq-mgmt
pathType: Prefix
backend:
service:
name: rabbit-mq
port:
number: 15672
---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbit-mq
spec:
resources:
requests:
cpu: 1
memory: 4Gi
limits:
cpu: 2
memory: 4Gi
image: rabbitmq:4.0-management-alpine # We pin the minor version to avoid feature-flags conflicts between nodes in a cluster. See https://www.rabbitmq.com/feature-flags.html
replicas: 3 # Must be an odd number, see https://www.rabbitmq.com/clustering.html#node-count
override:
statefulSet:
spec:
template:
metadata:
annotations:
config.linkerd.io/default-inbound-policy: all-unauthenticated
co.elastic.metrics/module: rabbitmq
co.elastic.metrics/hosts: "${data.host}:15672/rmq-mgmt"
co.elastic.metrics/metricsets: "node, queue"
co.elastic.metrics/period: 180s
co.elastic.metrics/username: ${kubernetes.${k8s-namespace}$.rabbit-mq-default-user.username}
co.elastic.metrics/password: ${kubernetes.${k8s-namespace}$.rabbit-mq-default-user.password}
rabbitmq:
additionalConfig: |
management.path_prefix = /rmq-mgmt
additionalPlugins:
- rabbitmq_shovel
- rabbitmq_shovel_management
- rabbitmq_delayed_message_exchangeWhat problem are you trying to solve?I am currently running upgrades to update our RabbitMQ clusters on k8s from 3.11 to the latest 4.1. I upgraded RabbitMQ on our dev environment from version v3.11 to v4.1 without any issues. Upgrading one version at a time and enabling all stable feature flags in the management UI before proceeding. Now I am repeating the same steps on our Test environement using the same configuration.
I looked around but couldn't find any other user who ran into that error. 
Can anyone please help? |
Beta Was this translation helpful? Give feedback.
Answered by
Joren-Thijs-KasparSolutions
Aug 9, 2025
Replies: 2 comments 8 replies
-
That error means that a configuration file is using an encrypted secret, and when RabbitMQ starts, the passphrase is not being provided. Since you have one environment that upgraded successfully, and one that didn't, you must very carefully determine what is different between the two environments. I would start by examining / diffing the configuration in each cluster's running containers. I'm not 100% certain on k8s, but RabbitMQ configs usually live in |
Beta Was this translation helpful? Give feedback.
8 replies
-
confirms that the passphrase for encrypted value decoding is not configured. We will not suggest anything more for |
Beta Was this translation helpful? Give feedback.
-
|
I am not asking for 3.13.x. The error on startup happens on a supported 4.x version that I am trying to upgrade too. And my currently supported 4.1 dev cluster that I did upgrade also has the passphrase for encrypted value decoding not configured. So why does it work on A but not on B? Is there a way to tell rabbitmq to ignore it? And can I add a passphrase later to a previously unencrypted version or will that cause problems? |
Beta Was this translation helpful? Give feedback.
-
|
Not using decrypt({encrypted, _} = EncValue,
{Cipher, Hash, Iterations, PassPhrase} = Algo) ->
{rabbit_pbe:decrypt_term(Cipher, Hash, Iterations, PassPhrase, EncValue),
Algo};
decrypt({encrypted, _} = EncValue,
ConfigEntryDecoder)
when is_list(ConfigEntryDecoder) ->
Algo = config_entry_decoder_to_algo(ConfigEntryDecoder),
decrypt(EncValue, Algo);
decrypt(List, Algo) when is_list(List) ->
decrypt_list(List, Algo, []);
decrypt(Value, Algo) ->
{Value, Algo}.In |
Beta Was this translation helpful? Give feedback.
-
|
Found it! It turned out the Randomly generated password we use for RabbitMQ had a |
Beta Was this translation helpful? Give feedback.
Answer selected by
Joren-Thijs-KasparSolutions
-
|
Thank you, even value escaping wouldn't have helped here (the parser considers the single quotes after parsing the |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
This limitation will addressed in Cuttlefish |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Found it! It turned out the Randomly generated password we use for RabbitMQ had a
:character in it (MyPass:w0rd). Since v4.0 this causes RabbitMQ to think it has encountered an encrypted value. Rotating the passwords without special characters fixed the issue.