OneDev
Dashboards
Projects
Global Views
Code Search
server
Code
Pull Requests
Issues
List
Boards
Iterations
Builds
Packages
Statistics
Child Projects
Audit Log
OneDev 14.0.5
Documentation Support & Bug Report RESTful API Incompatibilities License Agreement Check Update
Projects onedev server Issues OD-2611
Stuck in maintenance mode after update (OD-2611)
Glenn D. opened 1 month ago

After updating from 11.x to the latest 13.1.0 it seems that the database is stuck in maintenance mode:

image.png

The directory where all the data is stored grew with 200 MB, but it's not changing anymore. Any idea how I can get some more insights in what is going on?

  • Glenn D. commented 1 month ago

    I found the following in the logs:

    2024-08-30 22:14:29,508 INFO  [hz.127.0.0.1:5710.cached.thread-2] i.o.s.e.impl.DefaultProjectManager Initializing git repository in '/opt/onedev/site/projects/1/git'...
2024-08-31 20:00:30,013 ERROR [DefaultQuartzScheduler_Worker-5] o.h.e.jdbc.spi.SqlExceptionHelper HikariPool-1 - Connection is not available, request timed out after 30003ms.
2024-08-31 19:36:25,236 ERROR [Thread-15] o.h.e.jdbc.spi.SqlExceptionHelper HikariPool-1 - Connection is not available, request timed out after 30022ms.
2024-08-31 22:15:42,307 ERROR [DefaultQuartzScheduler_Worker-5] i.o.s.t.DefaultTaskScheduler Error executing scheduled task
org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
	at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:273)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:281)
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
	at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
	at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:471)
	at io.onedev.server.persistence.DefaultTransactionManager.lambda$call$0(DefaultTransactionManager.java:63)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
	at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:57)
	at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
	at io.onedev.server.taskschedule.DefaultTaskScheduler$1.execute(DefaultTaskScheduler.java:76)
	at io.onedev.server.taskschedule.DefaultTaskScheduler$HelperTask.execute(DefaultTaskScheduler.java:122)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30003ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:583)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
	at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.getConnection(HikariCPConnectionProvider.java:77)
	at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
	... 14 common frames omitted
2024-08-31 22:15:42,333 ERROR [Thread-15] i.o.server.job.DefaultJobManager Error checking unfinished builds
javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
	at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1602)
	at io.onedev.server.entitymanager.impl.DefaultBuildManager.queryUnfinished(DefaultBuildManager.java:292)
	at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
	at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
	at io.onedev.server.job.DefaultJobManager.run(DefaultJobManager.java:1155)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
	at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:50)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2104)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2041)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2019)
	at org.hibernate.loader.Loader.doQuery(Loader.java:948)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
	at org.hibernate.loader.Loader.doList(Loader.java:2850)
	at org.hibernate.loader.Loader.doList(Loader.java:2832)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664)
	at org.hibernate.loader.Loader.list(Loader.java:2659)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
	at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
	at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
	at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
	... 6 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30022ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:583)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
	at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.getConnection(HikariCPConnectionProvider.java:77)
	at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
	at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
	... 26 common frames omitted
2024-08-31 22:19:50,725 WARN  [sshd-SshServer[9659e51](port=6611)-nio2-thread-1] o.a.s.s.session.ServerSessionImpl exceptionCaught(ServerSessionImpl[null@/10.0.0.2:42412])[state=Opened] IOException: Connection reset by peer
  • Robin Shen commented 1 month ago

    Please delete file maintenance from OneDev's mount directory and try again.

  • Robin Shen commented 1 month ago

    Feel free to reopen if there is more info

  • Robin Shen changed state to 'Closed' 1 month ago
    Previous Value Current Value 
    Open
    		 
    Closed
  • Glenn D. commented 1 month ago

    Sorry it took so long. I deleted the "maintenance" file and restarted. It asked me to create an admin account, which I reluctantly did, but all my projects are gone.

  • Robin Shen commented 1 month ago

    This is odd. Please check inside directory /opt/onedev/site/db-backup/ to see if database backup is there. Also check /opt/onedev/site/projects, git repositories of all projects should be there.

  • Glenn D. commented 1 month ago

    The folders are there. The db backup .zip is from the 26th which should be correct.

    I can't look into the git folders to see if everything is there. I will have to load it as repo I assume.

    History
  • Robin Shen commented 1 month ago

    Which database type are you using?

  • Glenn D. commented 1 month ago

    Just the internal database

  • Robin Shen commented 1 month ago

    Please do the following:

    1. Backup the directory mounting to /opt/onedev
    2. Keep OneDev container running
    3. Run command docker exec -it <OneDev container id> bash to exec into the container
    4. While in container bash, run command /opt/onedev/bin/restore-db.sh /path/to/<db backup>.zip

    After restore, OneDev will restart. Please check if projects are restored after that.

  • Glenn D. commented 1 month ago

    That seems to have restored everything! Anything I need to check to be sure everything is OK?

  • Robin Shen commented 1 month ago

    Please check if OneDev is on 13.1.0 now. Also I'd suggest to make another upgrade to 13.1.2 (have some bug fixes) to make sure upgrade is working (backup mount directory before doing this) as expected.

  • Glenn D. commented 1 month ago

    Everything seems to be working fine again. Thank you very much!

issue 1/1
Type
Question
Priority
Normal
Assignee
Robin Shen
Labels
No labels
Issue Votes (0)
Login to vote
Watchers (2)
Reference
OD-2611
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover