Something seems wrong with the healthcheck

[ovizii]

Looking at your Dockerfile it seems reasonable. This was all working so nicely when I was using image: ghcr.io/rtuszik/photon-docker:dev but when you published version 1, I switched to latest. Any advice is welcome.

HEALTHCHECK --interval=30s --timeout=10s --start-period=240s --retries=3 \
  CMD curl -f http://localhost:2322/status || exit 1

I even copied it into my docker-compose.yml file to tweak it and slow it down:

      test: ["CMD", "curl", "-f", "http://localhost:2322/status"]
      interval: 30s
      timeout: 15s
      start_period: 300s
      retries: 3

but that didn't help as healthcheck error immediately start popping up as soon as I start the container.
It feels kind of random, as after a few restarts the container just works. And these healthchecks occur way more often than the above indicated 3 retries.

docker compose logs -f photon
photon  | 2025-09-13 16:09:42,035 - root - INFO - Photon Manager starting...
photon  | 2025-09-13 16:09:42,037 - root - INFO - Existing index found, skipping initial setup
photon  | 2025-09-13 16:09:42,040 - root - INFO - Starting Photon (attempt 1/3)...
photon  | 2025-09-13 16:09:42,052 - root - INFO - Photon started with PID: 18
photon  | 2025-09-13 16:09:42,055 - root - INFO - Waiting for Photon to become ready...
photon  | 2025-09-13 16:09:42,064 - root - ERROR - Photon health check failed after 1 attempts
photon  | WARNING: Using incubator modules: jdk.incubator.vector
photon  | 2025-09-13 16:09:43,812 [main] INFO  de.komoot.photon.App - Start up database cluster, this might take some time.
photon  | Cluster Name: photon
photon  | Base Path:    /photon/data/photon_data
photon  | Num Of Node:  1
.......
photon  | org.opensearch.transport.RemoteTransportException: [Node 1][127.0.0.1:9300][internal:cluster/coordination/join]
photon  | Caused by: org.opensearch.cluster.coordination.CoordinationStateRejectedException: incoming term 51 does not match current term 52
photon  |       at org.opensearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:270)
photon  |       at org.opensearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1262)
photon  |       at java.base/java.util.Optional.ifPresent(Unknown Source)
photon  |       at org.opensearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:709)
photon  |       at org.opensearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:672)
photon  |       at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
photon  |       at org.opensearch.transport.TransportService.connectToNode(TransportService.java:502)
photon  |       at org.opensearch.transport.TransportService.connectToNode(TransportService.java:485)
photon  |       at org.opensearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:655)
photon  |       at org.opensearch.cluster.coordination.JoinHelper.lambda$new$1(JoinHelper.java:209)
photon  |       at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63)
photon  |       at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108)
photon  |       at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1071)
photon  |       at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014)
photon  |       at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
photon  |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
photon  |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
photon  |       at java.base/java.lang.Thread.run(Unknown Source)
......
photon  | [2025-09-13T16:11:16,210][WARN ][stderr                   ] ---- unassigned
photon  | [2025-09-13T16:11:16,214][WARN ][stderr                   ] --------[photon][0], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-09-13T16:10:45.363Z], delayed=false, allocation_status[deciders_throttled]]
photon  | [2025-09-13T16:11:16,217][WARN ][stderr                   ] --------[photon][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-09-13T16:10:45.363Z], delayed=false, allocation_status[no_attempt]]
photon  | [2025-09-13T16:11:16,226][WARN ][stderr                   ] --------[photon][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-09-13T16:10:45.363Z], delayed=false, allocation_status[no_attempt]]
photon  | [2025-09-13T16:11:16,228][WARN ][stderr                   ] --------[photon][2], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-09-13T16:10:45.363Z], delayed=false, allocation_status[no_attempt]]
photon  | [2025-09-13T16:11:16,236][WARN ][stderr                   ] --------[photon][3], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-09-13T16:10:45.363Z], delayed=false, allocation_status[no_attempt]]
photon  | [2025-09-13T16:11:16,239][WARN ][stderr                   ] --------[photon][4], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2025-09-13T16:10:45.363Z], delayed=false, allocation_status[no_attempt]]
photon  | [2025-09-13T16:11:16,245][WARN ][stderr                   ] tasks: (0):
photon  | [2025-09-13T16:11:16,252][WARN ][stderr                   ]   at org.codelibs.opensearch.runner.OpenSearchRunner.onFailure(OpenSearchRunner.java:1101)
photon  | [2025-09-13T16:11:16,257][WARN ][stderr                   ]   at org.codelibs.opensearch.runner.OpenSearchRunner.ensureYellow(OpenSearchRunner.java:758)
photon  | [2025-09-13T16:11:16,264][WARN ][stderr                   ]   at de.komoot.photon.Server.startInternal(Server.java:103)
photon  | [2025-09-13T16:11:16,271][WARN ][stderr                   ]   at de.komoot.photon.Server.start(Server.java:63)
photon  | [2025-09-13T16:11:16,280][WARN ][stderr                   ]   at de.komoot.photon.App.runPhoton(App.java:74)
photon  | [2025-09-13T16:11:16,298][WARN ][stderr                   ]   at de.komoot.photon.App.main(App.java:38)
photon  | 2025-09-13 16:11:20,968 - root - ERROR - Photon health check failed after 1 attempts
photon  | 2025-09-13 16:11:25,973 - root - ERROR - Photon health check failed after 1 attempts
photon  | 2025-09-13 16:11:30,979 - root - ERROR - Photon health check failed after 1 attempts
photon  | 2025-09-13 16:11:35,982 - root - ERROR - Photon health check failed after 1 attempts
photon  | 2025-09-13 16:11:40,999 - root - ERROR - Photon health check failed after 1 attempts
photon  | 2025-09-13 16:11:46,004 - root - ERROR - Photon failed to become ready within 120 seconds
photon  | 2025-09-13 16:11:46,004 - root - ERROR - Photon health check failed on attempt 1
photon  | 2025-09-13 16:11:46,004 - root - INFO - Stopping Photon...
photon  | 2025-09-13 16:11:48,019 - root - INFO - Retrying Photon startup...

[rtuszik]

I don't think the health check is the issue here. It seems like photon didn't start properly at all?

What do the photon logs say?

The health check will only show the container as healthy when the photon api is available.

[ovizii]

I did some more digging and somehow, my PUID and PGID seems to have been ignored in favor of the default 9011 :-/

      - PUID=1000
      - PGID=1000

ls -al ./photon_data/
total 11
drwxr-xr-x  5 9011 9011  5 Aug  5 19:30 .
drwxr-xr-x 16 root root 17 Jun 26 13:35 ..
drwxr-xr-x  2 9011 9011  3 Aug  5 19:30 logs
drwxr-xr-x  3 9011 9011  3 Aug  5 19:30 photon_data
drwxr-xr-x  2 9011 9011  2 Jun 26 15:15 temp

I stopped photon, did a chown -R 1000:1000 ./photon_data/ and restarted photon and it reset the perms back to 9011 just like they were before :-(

[ovizii]

Here is my compose:

# https://github.com/rtuszik/photon-docker/tree/main
  photon:
    image: ghcr.io/rtuszik/photon-docker:dev #latest # set to dev to test user/group settings in ENV variable below
    container_name: photon
#    hostname: photon
    environment:
      - UPDATE_STRATEGY=PARALLEL #DISABLED #SEQUENTIAL
      - UPDATE_INTERVAL=90d
      - REGION=europe # switch to planet instead of europe before travelling
#      - FORCE_UPDATE=TRUE # can force an update upon restart
      - LOG_LEVEL=INFO # Options: DEBUG, INFO, ERROR
      - APPRISE_URLS=apprise://192.168.191.90:8000/apprise/?tags=tag1,tag2
      # - BASE_URL=https://r2.koalasec.org/public
      - PUID=1000
      - PGID=1000
    volumes:
      - ./photon_data:/photon/data
    restart: "no"
    network_mode: "service:reitti"
#        ports:
#            - "2322:2322"
    healthcheck:
      disable: true
#      test: ["CMD", "curl", "-f", "http://localhost:2322/status"]
#      interval: 30s
#      timeout: 10s
#      start_period: 300s
#      retries: 3
    cpus: 6
    mem_limit: 2G

Host OS: Debian
Host Type: Bare Metal
Hardware details:
    CPU: AMD something something can't remember
    Available RAM: 64GB
    Storage Type: HD
    Storage Size: 18TB

[D3r3k23]

Were you able to fix this?

[ovizii]

Were you able to fix this?

Unfortunately not and there was no feedback over here on this issue yet.