Shamil
opened 4 years ago
-
Looks like old version of OneDev is not terminating gracefully upon upgrading. Can you please let me know the reproducing steps?
-
I set up a k8s cluster running OneDev 5.3.0, and then upgrade to 5.3.1 by changing the image version, and everything works fine. If you can comes with a detailed reproducible steps for this issues, it will save me a lot of time to solve the issue.
-
I've tried to debug by the following
cd onedev-k8s/maintenance; kubectl apply -k . kubectl exec -it onedev-65675f5b6f-4l7rs -- bash /app/boot/wrapper-linux-x86-64 /app/conf/wrapper.conf wrapper.pidfile=/app/status/onedev_upgrade.pid -- upgrade /opt/onedev root@onedev-65675f5b6f-4l7rs:~/bin# /app/boot/wrapper-linux-x86-64 /app/conf/wrapper.conf wrapper.pidfile=/app/status/onedev_upgrade.pid -- upgrade /opt/onedev --> Wrapper Started as Console Java Service Wrapper Standard Edition 64-bit 3.5.44 Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved. http://wrapper.tanukisoftware.com Licensed to OneDev for Service Wrapping Launching a JVM... WrapperManager: Initializing... INFO - Launching application from '/app'... INFO - Cleaning temp directory... INFO - Initializing dependency injection container... INFO - Starting plugin manager... INFO - Upgrading /opt/onedev... ERROR - Please stop server running at "/opt/onedev" before upgrading <-- Wrapper Stopped root@onedev-65675f5b6f-4l7rs:~/bin# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 28248 11588 ? Ss 13:37 0:00 /bin/bash /root/bin/idle.sh root 7 0.0 0.0 20280 3952 pts/0 Ss 13:37 0:00 bash root 65 0.0 0.0 6180 736 ? S 13:38 0:00 sleep 60 root 106 0.0 0.0 36080 3208 pts/0 R+ 13:39 0:00 ps aux -
When OneDev runs, it will create file
/opt/onedev/status/onedev.pid, and this file will be removed if the container is stopped gracefully. The upgrade process will check existence of this file to determine whether or not the server is running. Obviously OneDev is forcibly stopped for some reason and that file is left behind. You may remove this file manually to make upgrade working.So the problem is how to reproduce the situation OneDev container is forcibly stopped. At my side, whether I perform an image update, or entering into maintenance mode, k8s stops the container gracefully (with a 30s grace shutdown period).
-
Ok, it looks like there was a stray pid file, after removing it, it seems to work now :)
-
OK. Will improve the upgrade logic to make it more robust.
-
Previous Value Current Value Open
Closed
| Type |
Question
|
| Priority |
Normal
|
| Assignee |
Logs from the container suggest that the upgrade isn't happening properly...