RedisClient::ReadOnlyError (READONLY You can't write against a read only replica

[javiermartingonzalez]

Before you start (required)

General checklist

• I have removed personal / sensitive data from screenshots and logs
• I have searched existing issues and discussions to ensure this is not a duplicate issue

How are you using Maybe?

• I am a paying Maybe customer (hosted version)
  • Paying Maybe users can also open requests in Intercom (if there is sensitive info involved)
• I am a self-hosted user

Self hoster checklist

Paying, hosted users should delete this entire section.

If you are a self-hosted user, please complete all of the information below. Issues with incomplete information will be marked as Needs Info to help our small team prioritize bug fixes.

• Self hosted app commit SHA (find in user menu): 02bfa9f
  • I have confirmed that my app's commit is the latest version of Maybe
• Where are you hosting?
  • Render
  • Docker Compose
  • Umbrel
  • Other (please specify)

---

Bug description

Sometimes, the sync process is not working. I'm getting "error application" page and after checking the logs the error is:

E, [2025-04-04T07:55:20.983594 #1] ERROR -- : [ActiveJob] Failed enqueuing SyncJob to Sidekiq(high_priority): RedisClient::ReadOnlyError (READONLY You can't write against a read only replica. (redis://redis:6379/1))

So, sync is not finished and I have to remove pending status jobs from database, restart the containers and try again.

Redis service is up and giving the error:

1:S 04 Apr 2025 07:57:36.849 # Error condition on socket for SYNC: Connection refused
1:S 04 Apr 2025 07:57:37.655 * Connecting to MASTER 124.222.224.112:21261
1:S 04 Apr 2025 07:57:37.656 * MASTER <-> REPLICA sync started
1:S 04 Apr 2025 07:57:37.871 # Error condition on socket for SYNC: Connection refused
1:S 04 Apr 2025 07:57:38.684 * Connecting to MASTER 124.222.224.112:21261
1:S 04 Apr 2025 07:57:38.686 * MASTER <-> REPLICA sync started
1:S 04 Apr 2025 07:57:38.909 # Error condition on socket for SYNC: Connection refused
1:S 04 Apr 2025 07:57:39.740 * Connecting to MASTER 124.222.224.112:21261
1:S 04 Apr 2025 07:57:39.741 * MASTER <-> REPLICA sync started

Trying to access to Redis contaner and set a test value, I'm getting same error:

redis-cli -h redis -p 6379
SET test_key "hello"

Restarting the containers and trying again, is working again. So I don't know if it's a lock due something Maybe is sending to Redis or a Redis problem keeping the services on. Also, the need to delete manually the jobs from database is not a good solution. Can you check? Could be possible a button in the interface to delete the pending jobs clicking on it without ssh to the machine?

To Reproduce

Trying to sync different kinds of accounts manually or automatically by the cron

Expected behavior

To finish the job in the background

[javiermartingonzalez]

Could be something in Maybe trying to use REPLICAOF command and then trying to write to a replica (not allowed - replicas are read only) instead of writing to master? I don't know if your using it on Maybe or what happened in Redis to have a replicate created.

I don't know the expected behaviour of jobs and workers and how are handling it on Readis, but I'm reading it's possible to set a redis configuration to force to avoid creating replicas and use always master, and in this case the REPLICAOF command won't do anyting and writing problem won't occur.

[zachgoll]

@javiermartingonzalez this does not appear to be related to Maybe. We're not seeing this error in production, which tells me that this is some sort of resource contention issue on the host machine (out of memory?? causing Redis to attempt some sort of failover behavior).

Looks like this has recently come up for others as well:

redis/docker-library-redis#434

I'm going to move this over to our Docker discussions as I don't believe there is any direct action we can take on Maybe's end to fix this (other than potentially updating our "example" compose file with a different Redis image)

[javiermartingonzalez]

Thank you! @zachgoll I will check the reason of creating replicas if the Maybe behaviour is not using REPLICAOF command. I will also try to force Redis to avoid creating replicas and using always master node.

But in another hand, a troubleshoot / maintenance to clean pending stucked jobs will be useful without sh to the machine. Because if old jobs are not cancelled, the information is not accurate and now there is no easy way to retry without removing manually from database

[javiermartingonzalez]

https://bobcares.com/blog/readonly-you-cant-write-against-a-read-only-replica/

https://laracasts.com/discuss/channels/servers/docker-on-production-error-redisexception-readonly-you-cant-write-against-a-read-only-replica

Some users suggested to add redis-server --appendonly yes --replica-read-only command to Docker Compose file to allow writing to replicas (but I don't know if it's safe)

However, if the setup intentionally includes read-only replicas for scaling or high availability, avoid making this change to prevent disrupting the architecture.

Trying adding (under volumes and the same level):

    command: >
      redis-server
      --appendonly yes
      --replica-read-only no

Another solution is to use redis:alpine instead of redis:latest when using it with Docker.

Will try both solutions and give feedback if they are working or not