Looks like old version of OneDev is not terminating gracefully upon upgrading. Can you please let me know the reproducing steps?

@robin

cd onedev-k8s/
kubectl apply -k .

I set up a k8s cluster running OneDev 5.3.0, and then upgrade to 5.3.1 by changing the image version, and everything works fine. If you can comes with a detailed reproducible steps for this issues, it will save me a lot of time to solve the issue.

@robin

I've tried to debug by the following

cd onedev-k8s/maintenance;
kubectl apply -k .
kubectl exec -it onedev-65675f5b6f-4l7rs -- bash
/app/boot/wrapper-linux-x86-64 /app/conf/wrapper.conf wrapper.pidfile=/app/status/onedev_upgrade.pid -- upgrade /opt/onedev
root@onedev-65675f5b6f-4l7rs:~/bin# /app/boot/wrapper-linux-x86-64 /app/conf/wrapper.conf wrapper.pidfile=/app/status/onedev_upgrade.pid -- upgrade /opt/onedev
--> Wrapper Started as Console
Java Service Wrapper Standard Edition 64-bit 3.5.44
  Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping

Launching a JVM...
WrapperManager: Initializing...
INFO  - Launching application from '/app'...
INFO  - Cleaning temp directory...
INFO  - Initializing dependency injection container...
INFO  - Starting plugin manager...
INFO  - Upgrading /opt/onedev...
ERROR - Please stop server running at "/opt/onedev" before upgrading
<-- Wrapper Stopped
root@onedev-65675f5b6f-4l7rs:~/bin# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  28248 11588 ?        Ss   13:37   0:00 /bin/bash /root/bin/idle.sh
root           7  0.0  0.0  20280  3952 pts/0    Ss   13:37   0:00 bash
root          65  0.0  0.0   6180   736 ?        S    13:38   0:00 sleep 60
root         106  0.0  0.0  36080  3208 pts/0    R+   13:39   0:00 ps aux

When OneDev runs, it will create file /opt/onedev/status/onedev.pid, and this file will be removed if the container is stopped gracefully. The upgrade process will check existence of this file to determine whether or not the server is running. Obviously OneDev is forcibly stopped for some reason and that file is left behind. You may remove this file manually to make upgrade working.

So the problem is how to reproduce the situation OneDev container is forcibly stopped. At my side, whether I perform an image update, or entering into maintenance mode, k8s stops the container gracefully (with a 30s grace shutdown period).

Ok, it looks like there was a stray pid file, after removing it, it seems to work now :)

OK. Will improve the upgrade logic to make it more robust.