Maintenance mode issue

[RoSk0]

While migrating to the Kubernetes installation ( using Helm chart mentioned in #22 ) I faced an issue. When application is in the maintenance mode and new version is being deployed, new containers, specifically daemon and websocket, would not start as they have the maintenance mode check.

I believe this check should rather exist in the application code and prevent both daemon and websocket code from operating. This way, deployment of the new code version would be possible and successful while the application is in the maintenance mode.

Keen to hear maintainers feedback before attempting to implement suggestion.

[tmachyshyn]

@RoSk0 It should work as you have described.
What problem are you facing?

[niekvleeuwen]

@tmachyshyn

I'm running into the same issue in a Kubernetes environment.

To elaborate on the root cause: the problem is a circular dependency between the sidecar containers and the main application container when maintenance mode is active.

Here's what happens during a deployment while maintenance mode is enabled:

1. The daemon and websocket sidecar containers fail to start because of the maintenance mode check in their entrypoint scripts (e.g. docker-websocket.sh). This check causes them to exit early, so they never reach a running/ready state.
2. Kubernetes considers the Pod as not fully ready because the sidecar containers are not healthy/ready.
3. The readiness probe of the EspoCRM application container fails as a result, causing the load balancer/ingress to stop routing traffic to it, returning a 503 Service Unavailable.
4. Because the application is returning 503, it is not possible to access the Web UI to disable maintenance mode.
5. This leaves the deployment in a deadlock. Maintenance mode can't be disabled via the UI because the app is returning 503, and the sidecars won't start because maintenance mode is active.

A possible solution would be to replace the hard exit in the entrypoint of the deamon and websocket containers with a retry loop. This keeps the container alive, so Kubernetes no longer sees it as crashed. The Pod reaches a ready state, the readiness probe on the app container succeeds, and the Web UI becomes accessible again to disable maintenance mode.

Would love to see this addressed. Happy to contribute a PR if the maintainers agree on the approach.

[tmachyshyn]

@niekvleeuwen thank you for describing the issue.

Please implement a health check as described in the documentation: https://docs.espocrm.com/administration/docker/installation/#docker-compose. This should resolve the problem.

Also, could you clarify why Kubernetes does not attempt to start or restart the container after exit 0 is executed?

[niekvleeuwen]

@tmachyshyn thanks for the pointer to the health check docs. I'll take a look, though I don't think it addresses the deadlock itself.

Kubernetes does restart the container. The maintenance mode check doesn't exit 0, it exits 1. With the default restartPolicy: Always, kubelet treats any non-zero exit as a crash and restarts the container automatically. So it's not that Kubernetes fails to restart it, it's that Kubernetes restarts it correctly, over and over, because the exit condition (maintenance mode being on) never goes away on its own. Due to this container not being marked as ready, the main container will never be available.

However, as of EspoCRM v10, you don't need the Web UI to disable maintenance mode anymore. Opening a shell and executing the following will disable the maintenance mode.

bin/command config:set maintenanceMode false

It doesn't fix the underlying issue (the daemon/websocket still exit 1 on every restart while maintenance mode is on), but it does give a reliable way to break the deadlock.

[tmachyshyn]

@niekvleeuwen it seems we may be referring to different docker versions of EspoCRM. Starting with EspoCRM v10, several major improvements were introduced. As a result, the daemon/websocket process now returns exit code 0 when maintenance mode is enabled or when the system is not functioning properly, https://github.com/espocrm/espocrm-docker/blob/master/entrypoint-utils.sh#L16

Could you please check this on your side?

[niekvleeuwen]

@tmachyshyn thanks, I did check on my side, and noticed the daemon/websocket containers do exit 0 in v10.

Small correction to my earlier comment: I said the check "exits 1, not 0," implying exit 0 would fix things. That was mistaken, the exit code doesn't actually matter here.

The real issue is the policy of a sidecar container. Whether it exits 0 or 1, Kubernetes restarts it. Since maintenance mode is still on at the next start, it exits again immediately, so the sidecar never stays up long enough for the pod to be ready. That means the main app container is never reachable, and we're back in the same deadlock.

I still think a retry loop (keep the sidecar alive and polling instead of exiting) is the right fix so this resolves itself. Happy to send a PR if needed.