#727  Failed to upgrade - Table onedev.o_UnitTestMetric doesn't exist
Closed
John opened 2 weeks ago

I tried upgrading to 7.2.0 (docker image), and the upgrade process fails due to a table not existing:

INFO  - Launching application from '/app'...
INFO  - Starting server...
INFO  - Upgrading /opt/onedev...
INFO  - >>> INFO  - Launching application from '/opt/onedev'...
INFO  - >>> INFO  - Cleaning temp directory...
INFO  - >>> INFO  - Starting server...
INFO  - Old data version: 84
INFO  - Backing up old program files as /opt/onedev/site/program-backup/2022-05-14_21-42-19...
INFO  - Backing up database as /opt/onedev/site/db-backup/2022-05-14_21-42-19.zip...
INFO  - >>> INFO  - Launching application from '/opt/onedev'...
INFO  - >>> INFO  - Cleaning temp directory...
INFO  - >>> INFO  - Starting server...
INFO  - >>> INFO  - Backing up database to /opt/onedev/site/db-backup/2022-05-14_21-42-19.zip...
INFO  - >>> INFO  - Exporting table 'UnitTestMetric'...
INFO  - >>> INFO  - Querying table ids...
INFO  - >>> WARN  - Error: 1146-42S02: Table 'onedev.o_UnitTestMetric' doesn't exist
INFO  - >>> ERROR - (conn=45693) Table 'onedev.o_UnitTestMetric' doesn't exist
INFO  - >>> ERROR - Error booting application
INFO  - >>> javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
INFO  - >>>     at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
INFO  - >>>     at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1602)
INFO  - >>>     at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.list(CriteriaQueryTypeQueryAdapter.java:96)
INFO  - >>>     at io.onedev.server.persistence.DefaultPersistManager.exportData(DefaultPersistManager.java:422)
INFO  - >>>     at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
INFO  - >>>     at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
INFO  - >>>     at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
INFO  - >>>     at io.onedev.server.persistence.DefaultPersistManager.exportData(DefaultPersistManager.java:405)
INFO  - >>>     at io.onedev.server.maintenance.BackupDatabase.start(BackupDatabase.java:69)
INFO  - >>>     at io.onedev.server.OneDev.start(OneDev.java:109)
INFO  - >>>     at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44)
INFO  - >>>     at io.onedev.commons.loader.AppLoader.start(AppLoader.java:73)
INFO  - >>>     at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:189)
INFO  - >>> Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
INFO  - >>>     at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63)
INFO  - >>>     at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
INFO  - >>>     at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
INFO  - >>>     at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
INFO  - >>>     at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:67)
INFO  - >>>     at org.hibernate.loader.Loader.getResultSet(Loader.java:2304)
INFO  - >>>     at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2057)
INFO  - >>>     at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2019)
INFO  - >>>     at org.hibernate.loader.Loader.doQuery(Loader.java:948)
INFO  - >>>     at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
INFO  - >>>     at org.hibernate.loader.Loader.doList(Loader.java:2850)
INFO  - >>>     at org.hibernate.loader.Loader.doList(Loader.java:2832)
INFO  - >>>     at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664)
INFO  - >>>     at org.hibernate.loader.Loader.list(Loader.java:2659)
INFO  - >>>     at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
INFO  - >>>     at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
INFO  - >>>     at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
INFO  - >>>     at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
INFO  - >>>     at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
INFO  - >>>     at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
INFO  - >>>     ... 11 common frames omitted
INFO  - >>> Caused by: java.sql.SQLSyntaxErrorException: (conn=45693) Table 'onedev.o_UnitTestMetric' doesn't exist
INFO  - >>>     at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:270)
INFO  - >>>     at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:358)
INFO  - >>>     at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:133)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:807)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:746)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:665)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:608)
INFO  - >>>     at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:93)
INFO  - >>>     at org.mariadb.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:276)
INFO  - >>>     at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
INFO  - >>>     at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
INFO  - >>>     at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
INFO  - >>>     ... 26 common frames omitted
ERROR - Failed to upgrade /opt/onedev
INFO  - Restoring old program files due to upgrade failure...
INFO  - Old program files restored
WARN  - The database might be in inconsistent state due to upgrade failure. In that case, you need to restore the database by first cleaning it, and then running below command:
INFO  - /opt/onedev/bin/restore-db.sh /opt/onedev/site/db-backup/2022-05-14_21-42-19.zip
Robin Shen commented 2 weeks ago

From which version are you performing the upgrade?

John commented 2 weeks ago

Is there an easy way to tell? I can't start the app to check the version because it just tries to upgrade again and fails/crashes

Robin Shen commented 2 weeks ago

You may stop OneDev container, and run below command:

docker run --rm -v <onedev data dir>:/opt/onedev 1dev/server cat /opt/onedev/version.txt

Also please let me know your MariaDB details, including OS and version, jdbc driver version, onedev database character set, etc.

John commented 2 weeks ago

Running the command returned "7.1.8"

MariaDB Details:

OS: Official MariaDB docker image
JDBC driver version: 3.0.3

hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect
hibernate.connection.driver_class=org.mariadb.jdbc.Driver
hibernate.connection.url=jdbc:mariadb://<removed>:3306/onedev
hibernate.connection.username=<removed>
hibernate.connection.password=<removed>

Database character set - Not sure how to get this, but I can provide the following:

collation: utf8mb4_unicode_ci
engine: InnoDB

Robin Shen commented 2 weeks ago

Just tested and it works at my side. I am starting MariaDB with below command:

docker run --name mariadb -p 3306:3306 --env MARIADB_PASSWORD=root --env MARIADB_ROOT_PASSWORD=root --env MARIADB_DATABASE=onedev -v $(pwd)/db:/var/lib/mysql --network mynet mariadb:latest

Please pull your mariadb to update to latest version to see if it works. If still not work, please tell me detailed steps to reproduce this issue.

Robin Shen commented 2 weeks ago

BTW: you may still get your OneDev up and running with 7.1.8 image, although upgrade fails.

John commented 2 weeks ago

I've made sure the mariadb docker image I'm using has been updated to the latest version, also
I tried starting the app on both 7.1.8 and 7.1.7 after restoring the database from backup, and I was met with the following error:

INFO  - Launching application from '/app'...
INFO  - Starting server...
INFO  - Upgrading /opt/onedev...
INFO  - >>> INFO  - Launching application from '/opt/onedev'...
INFO  - >>> INFO  - Cleaning temp directory...
INFO  - >>> INFO  - Starting server...
INFO  - Old data version: 84
INFO  - Backing up old program files as /opt/onedev/site/program-backup/2022-05-16_03-16-15...
INFO  - Copying new program files into /opt/onedev...
INFO  - Successfully upgraded /opt/onedev
03:16:16 INFO  i.onedev.commons.bootstrap.Bootstrap - Launching application from '/opt/onedev'...
03:16:16 INFO  i.onedev.commons.bootstrap.Bootstrap - Cleaning temp directory...
03:16:16 INFO  io.onedev.commons.loader.AppLoader - Starting server...
03:16:23 ERROR i.onedev.commons.bootstrap.Bootstrap - Error booting application
org.hibernate.type.SerializationException: could not deserialize
        at org.hibernate.internal.util.SerializationHelper.doDeserialize(SerializationHelper.java:243)
        at org.hibernate.internal.util.SerializationHelper.deserialize(SerializationHelper.java:287)
        at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.fromBytes(SerializableTypeDescriptor.java:138)
        at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:120)
        at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:29)
        at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$1.doExtract(BlobTypeDescriptor.java:48)
        at org.hibernate.type.descriptor.sql.BasicExtractor.extract(BasicExtractor.java:47)
        at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:257)
        at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:253)
        at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:243)
        at org.hibernate.type.AbstractStandardBasicType.hydrate(AbstractStandardBasicType.java:329)
        at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:3134)
        at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.loadFromResultSet(EntityReferenceInitializerImpl.java:342)
        at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.hydrateEntityState(EntityReferenceInitializerImpl.java:269)
        at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:102)
        at org.hibernate.loader.plan.exec.internal.EntityLoadQueryDetails$EntityLoaderRowReader.readRow(EntityLoadQueryDetails.java:288)
        at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractRows(ResultSetProcessorImpl.java:157)
        at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:94)
        at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:105)
        at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:285)
        at org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:4441)
        at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4431)
        at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:569)
        at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:537)
        at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:208)
        at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:332)
        at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:108)
        at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:74)
        at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:113)
        at org.hibernate.internal.SessionImpl.fireLoadNoChecks(SessionImpl.java:1186)
        at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1175)
        at org.hibernate.internal.SessionImpl.access$2100(SessionImpl.java:193)
        at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.doLoad(SessionImpl.java:2786)
        at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.lambda$load$1(SessionImpl.java:2767)
        at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.perform(SessionImpl.java:2723)
        at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2767)
        at org.hibernate.internal.SessionImpl.get(SessionImpl.java:983)
        at io.onedev.server.persistence.dao.DefaultDao.get(DefaultDao.java:42)
        at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
        at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
        at io.onedev.server.persistence.dao.BaseEntityManager.get(BaseEntityManager.java:77)
        at io.onedev.server.maintenance.DefaultDataManager.init(DefaultDataManager.java:118)
        at io.onedev.server.persistence.TransactionInterceptor$1.call(TransactionInterceptor.java:23)
        at io.onedev.server.persistence.DefaultTransactionManager$1.call(DefaultTransactionManager.java:72)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
        at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60)
        at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
        at io.onedev.server.OneDev.start(OneDev.java:111)
        at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44)
        at io.onedev.commons.loader.AppLoader.start(AppLoader.java:73)
        at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:189)
Caused by: java.io.StreamCorruptedException: invalid stream header: C2ACED9C
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:938)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
        at org.hibernate.internal.util.SerializationHelper$CustomObjectInputStream.<init>(SerializationHelper.java:309)
        at org.hibernate.internal.util.SerializationHelper$CustomObjectInputStream.<init>(SerializationHelper.java:299)
        at org.hibernate.internal.util.SerializationHelper.doDeserialize(SerializationHelper.java:218)
        ... 51 common frames omitted

This seems to be some sort of deserialization issue in the database possibly? I'm not sure as java isn't my thing

Robin Shen commented 2 weeks ago

Looks that some database record is corrupted for some reason. You might need to restore from an old backup to get it working.

Robin Shen commented 2 weeks ago

Or you may send the database backup (you are using to restore database) to me [robin AT onedev DOT io] and I will check what might be wrong.

John commented 2 weeks ago

Thank you, I sent an email. Would it be easier to restart from scratch? Would I (easily) be able to grab the git repositories I have setup and just migrate them over to another instance?

Robin Shen commented 2 weeks ago

All git repositories are stored under <onedev data dir>/site/projects identified by project id.

I have not got the email yet. If attachment is too large, please share a dropbox link.

John commented 2 weeks ago

Hi Robin,

The email subject I sent you is "Issue #727 - Database Corruption", I've confirmed its not too big and was able to send it on my end. I took a look at the projects directory but I am unable to find the source code of my projects. Could you elaborate within a project folder how I can find/export the files within the project

John commented 2 weeks ago

I ended up restoring from a waaaay older database backup taken by onedev on an older version, swapping to the latest (7.2.3) and letting the upgrade migration process take place. It took a while, but eventually succeeded. Thanks, closing.

John changed state to 'Closed' 2 weeks ago
Previous Value Current Value
Open
Closed
Robin Shen commented 1 week ago

This upgrade failure is odd. Please take a manual backup of your database (via OneDev's database backup menu) before next upgrade.

issue 1 of 1
Type
Bug
Priority
Normal
Assignee
Seen Builds
Not Found
Issue Votes (0)
Watchers (4)
Reference
issue onedev/server#727
Please wait...
Page is in error, reload to recover