After Helm Upgrade to 9.1.5 onedev crashloop (OD-1552)
Daniel Kollmannsberger opened 2 years ago

Hey there!

I upgraded from 9.0.4 to 9.1.5

I get now following error: https://hastebin.com/share/axuwiketaz.bash

Is this about the database? I checked the connection with another pod in the same namespace & the connection is working.

  • Robin Shen commented 2 years ago

    OneDev tries to connect to database server to determine its own ip address. And the database host looks unreachable from OneDev. Please let me know below values you are using for helm install:

    database.type
    database.host
    database.port
    
  • Daniel Kollmannsberger commented 2 years ago

    This is how my values.yaml - database section looks like

    database:
      external: true
      host: onedev-mysql.onedev.svc.cluster.local
      maximumPoolSize: "25"
      name: onedev
      password: changeit
      port: "3306"
      type: mysql
      user: onedev
    
  • Robin Shen commented 2 years ago

    Is OneDev server installed also in namespace onedev? Can you please run a test pod (ubuntu for instance) in this namespace, and run below command to see if it works?

    telnet onedev-mysql.onedev.svc.cluster.local 3306
    
  • Daniel Kollmannsberger commented 2 years ago

    Yes, there both in the same namespace:

    root@ubuntu:/# telnet onedev-mysql.onedev.svc.cluster.local 3306
    Trying 10.43.85.63...
    Connected to onedev-mysql.onedev.svc.cluster.local.
    Escape character is '^]'.
    J
    8.0.34s4
    PTv�!J.ZV=N>,/0mysql_native_password2#08S01Got timeout reading communication packetsConnection closed by foreign host.
    root@ubuntu:/#
    
  • Robin Shen commented 2 years ago

    It is odd that OneDev container can not connect to this address. Please change the db host as cluster ip of the mysql service, and test again to see if it works.

  • Daniel Kollmannsberger commented 2 years ago

    I get a JDBC Class not found error.

  • Daniel Kollmannsberger commented 2 years ago

    Maybe i try it with a clean install

  • Robin Shen commented 2 years ago

    This is quite odd. I tested upgrading from 9.0.4 to 9.1.5 on GKS and it works fine. If you have any findings, please let me know.

  • Daniel Kollmannsberger commented 2 years ago

    I finally found the reason:

    I need to declare these keys:

    database:
      external: true
      dbHost: onedev-mysql.onedev.svc.cluster.local
      dbMaximumPoolSize: "25"
      dbName: onedev
      dbPassword: changeit
      dbPort: "3306"
      dbType: mysql
      dbUser: root
    
  • Daniel Kollmannsberger commented 2 years ago

    I think you forgot to update which keys to use inside the application

  • Daniel Kollmannsberger commented 2 years ago

    Ok, my fault. I was on the wrong cluster. I tried a clean onedev install on a clean minikube cluster. I have the same issue.

    Something is wrong on the application site. With this version. I will have a look on the latest changes later.

  • Robin Shen commented 2 years ago

    Seems that you did not quote values. Mine is:

    database:
      external: true
      type: "mysql"
      host: "onedev-mysql.onedev.svc.cluster.local"
      port: "3306"
      name: "onedev"
      user: "root"
      password: "changeit"
      maximumPoolSize: "25"
    
  • Robin Shen commented 2 years ago

    OK. You are using the wrong key. Please remove the "db" prefix from various keys.

  • Robin Shen commented 2 years ago

    This change was made in 9.0.0. See incompatibiltity note here:

    https://code.onedev.io/onedev/server/~builds/4072/markdown/Incompatibilities/server-product/system/incompatibilities/incompatibilities.md#900

    Since you are mentioning that you are upgrading from 9.0.4 to 9.1.5. I thought you already use the correct keys now...

  • Robin Shen commented 2 years ago

    Before each upgrade, I'd suggest to check the update from OneDev help menu (bottom left of the screen). It will show all changes since your current version, as well as any incompatibilities.

  • Daniel Kollmannsberger commented 2 years ago

    Okay, I found another thing.

    database.external has to be false. Otherwise it comes to the NoRouteToHost Error. But if its false. My installation is stuck in "...Database not populated yet"

    I did some connection checks from another pod in the same namespace. And its working

  • Daniel Kollmannsberger commented 2 years ago

    I found the main issue. I migrated to a newer k8s version. The dns is not working as expected. Specifying onedev-mysql in host. Solved the issue.

    Thanks anyway! & Big sorry for the circumstances

  • Robin Shen changed state to 'Closed' 2 years ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen commented 2 years ago

    No problem. Closing the issue now.

issue 1/1
Type
Bug
Priority
Normal
Assignee
Affected Versions
9.1.5
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
OD-1552
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover