#1552  After Helm Upgrade to 9.1.5 onedev crashloop
Closed
Daniel Kollmannsberger opened 8 months ago

Hey there!

I upgraded from 9.0.4 to 9.1.5

I get now following error: https://hastebin.com/share/axuwiketaz.bash

Is this about the database? I checked the connection with another pod in the same namespace & the connection is working.

Robin Shen commented 8 months ago

OneDev tries to connect to database server to determine its own ip address. And the database host looks unreachable from OneDev. Please let me know below values you are using for helm install:

database.type
database.host
database.port
Daniel Kollmannsberger commented 8 months ago

This is how my values.yaml - database section looks like

database:
  external: true
  host: onedev-mysql.onedev.svc.cluster.local
  maximumPoolSize: "25"
  name: onedev
  password: changeit
  port: "3306"
  type: mysql
  user: onedev
Robin Shen commented 8 months ago

Is OneDev server installed also in namespace onedev? Can you please run a test pod (ubuntu for instance) in this namespace, and run below command to see if it works?

telnet onedev-mysql.onedev.svc.cluster.local 3306
Daniel Kollmannsberger commented 8 months ago

Yes, there both in the same namespace:

root@ubuntu:/# telnet onedev-mysql.onedev.svc.cluster.local 3306
Trying 10.43.85.63...
Connected to onedev-mysql.onedev.svc.cluster.local.
Escape character is '^]'.
J
8.0.34s4
PTv�!J.ZV=N>,/0mysql_native_password2#08S01Got timeout reading communication packetsConnection closed by foreign host.
root@ubuntu:/#
Robin Shen commented 8 months ago

It is odd that OneDev container can not connect to this address. Please change the db host as cluster ip of the mysql service, and test again to see if it works.

Daniel Kollmannsberger commented 8 months ago

I get a JDBC Class not found error.

Daniel Kollmannsberger commented 8 months ago

Maybe i try it with a clean install

Robin Shen commented 8 months ago

This is quite odd. I tested upgrading from 9.0.4 to 9.1.5 on GKS and it works fine. If you have any findings, please let me know.

Daniel Kollmannsberger commented 8 months ago

I finally found the reason:

I need to declare these keys:

database:
  external: true
  dbHost: onedev-mysql.onedev.svc.cluster.local
  dbMaximumPoolSize: "25"
  dbName: onedev
  dbPassword: changeit
  dbPort: "3306"
  dbType: mysql
  dbUser: root
Daniel Kollmannsberger commented 8 months ago

I think you forgot to update which keys to use inside the application

Daniel Kollmannsberger commented 8 months ago

Ok, my fault. I was on the wrong cluster. I tried a clean onedev install on a clean minikube cluster. I have the same issue.

Something is wrong on the application site. With this version. I will have a look on the latest changes later.

Robin Shen commented 8 months ago

Seems that you did not quote values. Mine is:

database:
  external: true
  type: "mysql"
  host: "onedev-mysql.onedev.svc.cluster.local"
  port: "3306"
  name: "onedev"
  user: "root"
  password: "changeit"
  maximumPoolSize: "25"
Robin Shen commented 8 months ago

OK. You are using the wrong key. Please remove the "db" prefix from various keys.

Robin Shen commented 8 months ago

This change was made in 9.0.0. See incompatibiltity note here:

https://code.onedev.io/onedev/server/~builds/4072/markdown/Incompatibilities/server-product/system/incompatibilities/incompatibilities.md#900

Since you are mentioning that you are upgrading from 9.0.4 to 9.1.5. I thought you already use the correct keys now...

Robin Shen commented 8 months ago

Before each upgrade, I'd suggest to check the update from OneDev help menu (bottom left of the screen). It will show all changes since your current version, as well as any incompatibilities.

Daniel Kollmannsberger commented 8 months ago

Okay, I found another thing.

database.external has to be false. Otherwise it comes to the NoRouteToHost Error. But if its false. My installation is stuck in "...Database not populated yet"

I did some connection checks from another pod in the same namespace. And its working

Daniel Kollmannsberger commented 8 months ago

I found the main issue. I migrated to a newer k8s version. The dns is not working as expected. Specifying onedev-mysql in host. Solved the issue.

Thanks anyway! & Big sorry for the circumstances

Robin Shen changed state to 'Closed' 8 months ago
Previous Value Current Value
Open
Closed
Robin Shen commented 8 months ago

No problem. Closing the issue now.

issue 1 of 1
Type
Bug
Priority
Normal
Assignee
Affected Versions
9.1.5
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
onedev/server#1552
Please wait...
Page is in error, reload to recover