Container docker_rabbitmq_1 is unhealthy

[superbully]

Hi, I'm testing the new version of georchestra (24) and I can't get the rabbitmq container up, it stays in unhealthy state (attached screenshot). I have read carefully all the documentation but I have not found anything to solve it.
The environment is up fine and practically everything works except the user administration part (attached screenshot). I don't know if it is related to the unhealthy container problem.
If someone can help me or knows how to solve the problem I will be very grateful.
Greetings.

[f-necas]

Hi @superbully ,

Sorry for the late response, is everything up to date ? (submodules too)

On which branch are you ?

[superbully]

Hi @superbully ,

Sorry for the late response, is everything up to date ? (submodules too)

On which branch are you ?
Hello, thank you very much for replying.
I am using the branch :

On branch 24.0
Your branch is up to date with 'origin/24.0'.

I have also tried to force to update all submodules with the following command:
git submodule update --recursive --remote

Unfortunately, I keep getting the same error.
thank you very much for your interest

[f-necas]

Can you paste the logs of rbmq container ?

[superbully]

Can you paste the logs of rbmq container ?

The log of rbmq contanier:

2024-10-29 10:18:52.048333+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2024-10-29 10:18:52.052478+00:00 [notice] <0.235.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2024-10-29 10:18:52.053083+00:00 [notice] <0.235.0> Logging: configured log handlers are now ACTIVE
2024-10-29 10:18:52.536599+00:00 [info] <0.235.0> ra: starting system quorum_queues
2024-10-29 10:18:52.536705+00:00 [info] <0.235.0> starting Ra system: quorum_queues in directory: /bitnami/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost
2024-10-29 10:18:52.554136+00:00 [info] <0.318.0> ra system 'quorum_queues' running pre init for 0 registered servers
2024-10-29 10:18:52.561951+00:00 [info] <0.339.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2024-10-29 10:18:52.572082+00:00 [notice] <0.358.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2024-10-29 10:18:52.608747+00:00 [info] <0.235.0> ra: starting system coordination
2024-10-29 10:18:52.608813+00:00 [info] <0.235.0> starting Ra system: coordination in directory: /bitnami/rabbitmq/mnesia/rabbit@localhost/coordination/rabbit@localhost
2024-10-29 10:18:52.610229+00:00 [info] <0.365.0> ra system 'coordination' running pre init for 0 registered servers
2024-10-29 10:18:52.611194+00:00 [info] <0.366.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2024-10-29 10:18:52.611396+00:00 [notice] <0.371.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2024-10-29 10:18:52.652670+00:00 [info] <0.235.0> 
2024-10-29 10:18:52.652670+00:00 [info] <0.235.0>  Starting RabbitMQ 3.12.14 on Erlang 26.2.5 [jit]
2024-10-29 10:18:52.652670+00:00 [info] <0.235.0>  Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries
2024-10-29 10:18:52.652670+00:00 [info] <0.235.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2024-10-29 10:18:52.652839+00:00 [error] <0.235.0> This release series has reached end of life and is no longer supported. Please visit https://rabbitmq.com/versions.html to learn more and upgrade

  ##  ##      RabbitMQ 3.12.14
  ##  ##
  ##########  Copyright (c) 2007-2024 Broadcom Inc and/or its subsidiaries
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Erlang:      26.2.5 [jit]
  TLS Library: OpenSSL - OpenSSL 3.0.13 30 Jan 2024
  Release series support status: out of support

  Doc guides:  https://rabbitmq.com/documentation.html
  Support:     https://rabbitmq.com/contact.html
  Tutorials:   https://rabbitmq.com/getstarted.html
  Monitoring:  https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf

  Starting broker...2024-10-29 10:18:52.654335+00:00 [info] <0.235.0> 
2024-10-29 10:18:52.654335+00:00 [info] <0.235.0>  node           : rabbit@localhost
2024-10-29 10:18:52.654335+00:00 [info] <0.235.0>  home dir       : /opt/bitnami/rabbitmq/.rabbitmq
2024-10-29 10:18:52.654335+00:00 [info] <0.235.0>  config file(s) : /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
2024-10-29 10:18:52.654335+00:00 [info] <0.235.0>  cookie hash    : WISHFx/7b8Bm/QCfphXpGQ==
2024-10-29 10:18:52.654335+00:00 [info] <0.235.0>  log(s)         : <stdout>
2024-10-29 10:18:52.654335+00:00 [info] <0.235.0>  data dir       : /bitnami/rabbitmq/mnesia/rabbit@localhost
2024-10-29 10:18:54.874243+00:00 [info] <0.235.0> Running boot step pre_boot defined by app rabbit
2024-10-29 10:18:54.874317+00:00 [info] <0.235.0> Running boot step rabbit_global_counters defined by app rabbit
2024-10-29 10:18:54.874666+00:00 [info] <0.235.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2024-10-29 10:18:54.874761+00:00 [info] <0.235.0> Running boot step rabbit_core_metrics defined by app rabbit
2024-10-29 10:18:54.876239+00:00 [info] <0.235.0> Running boot step rabbit_alarm defined by app rabbit
2024-10-29 10:18:54.883154+00:00 [info] <0.390.0> Memory high watermark set to 141664 MiB (148545880064 bytes) of 354160 MiB (371364700160 bytes) total
2024-10-29 10:18:54.888180+00:00 [info] <0.392.0> Enabling free disk space monitoring (disk free space: 44143738880, total memory: 371364700160)
2024-10-29 10:18:54.888906+00:00 [info] <0.392.0> Disk free limit set to 371364MB
2024-10-29 10:18:54.891716+00:00 [info] <0.392.0> Free disk space is insufficient. Free bytes: 44143738880. Limit: 371364700160
2024-10-29 10:18:54.891873+00:00 [warning] <0.388.0> disk resource limit alarm set on node rabbit@localhost.
2024-10-29 10:18:54.891873+00:00 [warning] <0.388.0> 
2024-10-29 10:18:54.891873+00:00 [warning] <0.388.0> **********************************************************
2024-10-29 10:18:54.891873+00:00 [warning] <0.388.0> *** Publishers will be blocked until this alarm clears ***
2024-10-29 10:18:54.891873+00:00 [warning] <0.388.0> **********************************************************
2024-10-29 10:18:54.891873+00:00 [warning] <0.388.0> 
2024-10-29 10:18:54.892231+00:00 [info] <0.235.0> Running boot step code_server_cache defined by app rabbit
2024-10-29 10:18:54.892330+00:00 [info] <0.235.0> Running boot step file_handle_cache defined by app rabbit
2024-10-29 10:18:54.892681+00:00 [info] <0.395.0> Limiting to approx 1048479 file handles (943629 sockets)
2024-10-29 10:18:54.892870+00:00 [info] <0.396.0> FHC read buffering: OFF
2024-10-29 10:18:54.892925+00:00 [info] <0.396.0> FHC write buffering: ON
2024-10-29 10:18:54.893213+00:00 [info] <0.235.0> Running boot step worker_pool defined by app rabbit
2024-10-29 10:18:54.893261+00:00 [info] <0.373.0> Will use 32 processes for default worker pool
2024-10-29 10:18:54.893296+00:00 [info] <0.373.0> Starting worker pool 'worker_pool' with 32 processes in it
2024-10-29 10:18:54.894536+00:00 [info] <0.235.0> Running boot step database defined by app rabbit
2024-10-29 10:18:54.896605+00:00 [info] <0.235.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-10-29 10:18:54.910950+00:00 [info] <0.235.0> Successfully synced tables from a peer
2024-10-29 10:18:54.969188+00:00 [info] <0.235.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2024-10-29 10:18:54.969494+00:00 [info] <0.235.0> Successfully synced tables from a peer
2024-10-29 10:18:54.969583+00:00 [info] <0.235.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2024-10-29 10:18:54.969890+00:00 [info] <0.235.0> Running boot step tracking_metadata_store defined by app rabbit
2024-10-29 10:18:54.970123+00:00 [info] <0.438.0> Setting up a table for connection tracking on this node: tracked_connection
2024-10-29 10:18:54.970286+00:00 [info] <0.438.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2024-10-29 10:18:54.970421+00:00 [info] <0.438.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2024-10-29 10:18:54.970515+00:00 [info] <0.438.0> Setting up a table for channel tracking on this node: tracked_channel
2024-10-29 10:18:54.970608+00:00 [info] <0.438.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2024-10-29 10:18:54.970731+00:00 [info] <0.235.0> Running boot step networking_metadata_store defined by app rabbit
2024-10-29 10:18:54.970917+00:00 [info] <0.235.0> Running boot step feature_flags defined by app rabbit
2024-10-29 10:18:54.971157+00:00 [info] <0.235.0> Running boot step codec_correctness_check defined by app rabbit
2024-10-29 10:18:54.971226+00:00 [info] <0.235.0> Running boot step external_infrastructure defined by app rabbit
2024-10-29 10:18:54.971292+00:00 [info] <0.235.0> Running boot step rabbit_event defined by app rabbit
2024-10-29 10:18:54.971486+00:00 [info] <0.235.0> Running boot step rabbit_registry defined by app rabbit
2024-10-29 10:18:54.971614+00:00 [info] <0.235.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2024-10-29 10:18:54.971753+00:00 [info] <0.235.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2024-10-29 10:18:54.971862+00:00 [info] <0.235.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2024-10-29 10:18:54.972052+00:00 [info] <0.235.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2024-10-29 10:18:54.972205+00:00 [info] <0.235.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2024-10-29 10:18:54.972322+00:00 [info] <0.235.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2024-10-29 10:18:54.972432+00:00 [info] <0.235.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2024-10-29 10:18:54.972522+00:00 [info] <0.235.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2024-10-29 10:18:54.972628+00:00 [info] <0.235.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2024-10-29 10:18:54.972734+00:00 [info] <0.235.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2024-10-29 10:18:54.972826+00:00 [info] <0.235.0> Running boot step rabbit_priority_queue defined by app rabbit
2024-10-29 10:18:54.972891+00:00 [info] <0.235.0> Priority queues enabled, real BQ is rabbit_variable_queue
2024-10-29 10:18:54.973012+00:00 [info] <0.235.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2024-10-29 10:18:54.973127+00:00 [info] <0.235.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2024-10-29 10:18:54.973234+00:00 [info] <0.235.0> Running boot step rabbit_queue_location_random defined by app rabbit
2024-10-29 10:18:54.973333+00:00 [info] <0.235.0> Running boot step kernel_ready defined by app rabbit
2024-10-29 10:18:54.973397+00:00 [info] <0.235.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2024-10-29 10:18:54.973656+00:00 [info] <0.235.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2024-10-29 10:18:54.974927+00:00 [info] <0.446.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2024-10-29 10:18:54.975132+00:00 [info] <0.235.0> Running boot step guid_generator defined by app rabbit
2024-10-29 10:18:54.998009+00:00 [info] <0.235.0> Running boot step rabbit_node_monitor defined by app rabbit
2024-10-29 10:18:54.998491+00:00 [info] <0.450.0> Starting rabbit_node_monitor (in ignore mode)
2024-10-29 10:18:54.998725+00:00 [info] <0.235.0> Running boot step delegate_sup defined by app rabbit
2024-10-29 10:18:54.999748+00:00 [info] <0.235.0> Running boot step rabbit_memory_monitor defined by app rabbit
2024-10-29 10:18:55.000156+00:00 [info] <0.235.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2024-10-29 10:18:55.000338+00:00 [info] <0.235.0> Running boot step core_initialized defined by app rabbit
2024-10-29 10:18:55.000420+00:00 [info] <0.235.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2024-10-29 10:18:55.000565+00:00 [info] <0.235.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2024-10-29 10:18:55.000670+00:00 [info] <0.235.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2024-10-29 10:18:55.000812+00:00 [info] <0.235.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2024-10-29 10:18:55.001300+00:00 [info] <0.235.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2024-10-29 10:18:55.002041+00:00 [info] <0.235.0> Running boot step rabbit_policies defined by app rabbit
2024-10-29 10:18:55.002737+00:00 [info] <0.235.0> Running boot step rabbit_policy defined by app rabbit
2024-10-29 10:18:55.002864+00:00 [info] <0.235.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2024-10-29 10:18:55.002973+00:00 [info] <0.235.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2024-10-29 10:18:55.003071+00:00 [info] <0.235.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2024-10-29 10:18:55.003235+00:00 [info] <0.235.0> Running boot step rabbit_vhost_limit defined by app rabbit
2024-10-29 10:18:55.003404+00:00 [info] <0.235.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2024-10-29 10:18:55.003531+00:00 [info] <0.235.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2024-10-29 10:18:55.003615+00:00 [info] <0.235.0> Management plugin: using rates mode 'basic'
2024-10-29 10:18:55.004050+00:00 [info] <0.235.0> Running boot step recovery defined by app rabbit
2024-10-29 10:18:55.005687+00:00 [info] <0.487.0> Making sure data directory '/bitnami/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2024-10-29 10:18:55.044903+00:00 [info] <0.487.0> Starting message stores for vhost '/'
2024-10-29 10:18:55.045346+00:00 [info] <0.497.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2024-10-29 10:18:55.048526+00:00 [info] <0.487.0> Started message store of type transient for vhost '/'
2024-10-29 10:18:55.048958+00:00 [info] <0.501.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2024-10-29 10:18:55.052473+00:00 [info] <0.487.0> Started message store of type persistent for vhost '/'
2024-10-29 10:18:55.052743+00:00 [info] <0.487.0> Recovering 0 queues of type rabbit_classic_queue took 45ms
2024-10-29 10:18:55.052864+00:00 [info] <0.487.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2024-10-29 10:18:55.052982+00:00 [info] <0.487.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2024-10-29 10:18:55.057510+00:00 [info] <0.235.0> Running boot step empty_db_check defined by app rabbit
2024-10-29 10:18:55.057606+00:00 [info] <0.235.0> Will not seed default virtual host and user: have definitions to load...
2024-10-29 10:18:55.057706+00:00 [info] <0.235.0> Running boot step rabbit_observer_cli defined by app rabbit
2024-10-29 10:18:55.057900+00:00 [info] <0.235.0> Running boot step rabbit_looking_glass defined by app rabbit
2024-10-29 10:18:55.058052+00:00 [info] <0.235.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2024-10-29 10:18:55.058313+00:00 [info] <0.235.0> Running boot step background_gc defined by app rabbit
2024-10-29 10:18:55.058585+00:00 [info] <0.235.0> Running boot step routing_ready defined by app rabbit
2024-10-29 10:18:55.058670+00:00 [info] <0.235.0> Running boot step pre_flight defined by app rabbit
2024-10-29 10:18:55.058763+00:00 [info] <0.235.0> Running boot step notify_cluster defined by app rabbit
2024-10-29 10:18:55.058871+00:00 [info] <0.235.0> Running boot step networking defined by app rabbit
2024-10-29 10:18:55.058981+00:00 [info] <0.235.0> Running boot step definition_import_worker_pool defined by app rabbit
2024-10-29 10:18:55.059115+00:00 [info] <0.373.0> Starting worker pool 'definition_import_pool' with 32 processes in it
2024-10-29 10:18:55.061204+00:00 [info] <0.235.0> Running boot step cluster_name defined by app rabbit
2024-10-29 10:18:55.061294+00:00 [info] <0.235.0> Running boot step direct_client defined by app rabbit
2024-10-29 10:18:55.061425+00:00 [info] <0.235.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
2024-10-29 10:18:55.061483+00:00 [info] <0.235.0> Creating table rabbit_node_maintenance_states for maintenance mode status
2024-10-29 10:18:55.062041+00:00 [info] <0.235.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2024-10-29 10:18:55.062176+00:00 [info] <0.568.0> Resetting node maintenance status
2024-10-29 10:18:55.077063+00:00 [info] <0.627.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2024-10-29 10:18:55.077199+00:00 [info] <0.655.0> Statistics database started.
2024-10-29 10:18:55.077286+00:00 [info] <0.654.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2024-10-29 10:18:55.077651+00:00 [info] <0.568.0> Ready to start client connection listeners
2024-10-29 10:18:55.079258+00:00 [info] <0.679.0> started TCP listener on [::]:5672
 completed with 3 plugins.
2024-10-29 10:18:55.136354+00:00 [info] <0.568.0> Server startup complete; 3 plugins started.
2024-10-29 10:18:55.136354+00:00 [info] <0.568.0>  * rabbitmq_management
2024-10-29 10:18:55.136354+00:00 [info] <0.568.0>  * rabbitmq_management_agent
2024-10-29 10:18:55.136354+00:00 [info] <0.568.0>  * rabbitmq_web_dispatch
2024-10-29 10:18:55.263234+00:00 [info] <0.9.0> Time to start RabbitMQ: 5231066 us

[f-necas]

And what's the response if you execute this command inside the container ?

rabbitmq-diagnostics -q ping && rabbitmq-diagnostics -q check_local_alarms

By my side :

I have no name!@d1bee54a0b40:/$ rabbitmq-diagnostics -q ping && rabbitmq-diagnostics -q check_local_alarms
Ping succeeded
Node rabbit@localhost reported no local alarms

[superbully]

And what's the response if you execute this command inside the container ?

rabbitmq-diagnostics -q ping && rabbitmq-diagnostics -q check_local_alarms

By my side :

I have no name!@d1bee54a0b40:/$ rabbitmq-diagnostics -q ping && rabbitmq-diagnostics -q check_local_alarms
Ping succeeded
Node rabbit@localhost reported no local alarms

In my side:

$ rabbitmq-diagnostics -q ping && rabbitmq-diagnostics -q check_local_alarms
Ping succeeded
Error:
Free disk space alarm on node rabbit@localhost
$

[superbully]

Hello, i solved the problem. In the file .envs-rabbitmq i have added this line:

RABBITMQ_DISK_FREE_ABSOLUTE_LIMIT = 2GB

With this work fine. I have not errors.
Thank you so much. Best regards !

[edevosc2c]

In #305 we are going to disable rabbitmq by default so this should be less of an issue.