Failed to upgrade - Table onedev.o_UnitTestMetric doesn't exist (OD-727)
John opened 4 years ago

I tried upgrading to 7.2.0 (docker image), and the upgrade process fails due to a table not existing:

INFO  - Launching application from '/app'...
INFO  - Starting server...
INFO  - Upgrading /opt/onedev...
INFO  - >>> INFO  - Launching application from '/opt/onedev'...
INFO  - >>> INFO  - Cleaning temp directory...
INFO  - >>> INFO  - Starting server...
INFO  - Old data version: 84
INFO  - Backing up old program files as /opt/onedev/site/program-backup/2022-05-14_21-42-19...
INFO  - Backing up database as /opt/onedev/site/db-backup/2022-05-14_21-42-19.zip...
INFO  - >>> INFO  - Launching application from '/opt/onedev'...
INFO  - >>> INFO  - Cleaning temp directory...
INFO  - >>> INFO  - Starting server...
INFO  - >>> INFO  - Backing up database to /opt/onedev/site/db-backup/2022-05-14_21-42-19.zip...
INFO  - >>> INFO  - Exporting table 'UnitTestMetric'...
INFO  - >>> INFO  - Querying table ids...
INFO  - >>> WARN  - Error: 1146-42S02: Table 'onedev.o_UnitTestMetric' doesn't exist
INFO  - >>> ERROR - (conn=45693) Table 'onedev.o_UnitTestMetric' doesn't exist
INFO  - >>> ERROR - Error booting application
INFO  - >>> javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
INFO  - >>>     at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
INFO  - >>>     at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1602)
INFO  - >>>     at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.list(CriteriaQueryTypeQueryAdapter.java:96)
INFO  - >>>     at io.onedev.server.persistence.DefaultPersistManager.exportData(DefaultPersistManager.java:422)
INFO  - >>>     at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
INFO  - >>>     at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
INFO  - >>>     at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
INFO  - >>>     at io.onedev.server.persistence.DefaultPersistManager.exportData(DefaultPersistManager.java:405)
INFO  - >>>     at io.onedev.server.maintenance.BackupDatabase.start(BackupDatabase.java:69)
INFO  - >>>     at io.onedev.server.OneDev.start(OneDev.java:109)
INFO  - >>>     at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44)
INFO  - >>>     at io.onedev.commons.loader.AppLoader.start(AppLoader.java:73)
INFO  - >>>     at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:189)
INFO  - >>> Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
INFO  - >>>     at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63)
INFO  - >>>     at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
INFO  - >>>     at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
INFO  - >>>     at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
INFO  - >>>     at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:67)
INFO  - >>>     at org.hibernate.loader.Loader.getResultSet(Loader.java:2304)
INFO  - >>>     at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2057)
INFO  - >>>     at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2019)
INFO  - >>>     at org.hibernate.loader.Loader.doQuery(Loader.java:948)
INFO  - >>>     at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
INFO  - >>>     at org.hibernate.loader.Loader.doList(Loader.java:2850)
INFO  - >>>     at org.hibernate.loader.Loader.doList(Loader.java:2832)
INFO  - >>>     at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664)
INFO  - >>>     at org.hibernate.loader.Loader.list(Loader.java:2659)
INFO  - >>>     at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
INFO  - >>>     at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
INFO  - >>>     at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
INFO  - >>>     at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
INFO  - >>>     at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
INFO  - >>>     at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
INFO  - >>>     ... 11 common frames omitted
INFO  - >>> Caused by: java.sql.SQLSyntaxErrorException: (conn=45693) Table 'onedev.o_UnitTestMetric' doesn't exist
INFO  - >>>     at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:270)
INFO  - >>>     at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:358)
INFO  - >>>     at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:133)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:807)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:746)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:665)
INFO  - >>>     at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:608)
INFO  - >>>     at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:93)
INFO  - >>>     at org.mariadb.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:276)
INFO  - >>>     at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
INFO  - >>>     at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
INFO  - >>>     at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
INFO  - >>>     ... 26 common frames omitted
ERROR - Failed to upgrade /opt/onedev
INFO  - Restoring old program files due to upgrade failure...
INFO  - Old program files restored
WARN  - The database might be in inconsistent state due to upgrade failure. In that case, you need to restore the database by first cleaning it, and then running below command:
INFO  - /opt/onedev/bin/restore-db.sh /opt/onedev/site/db-backup/2022-05-14_21-42-19.zip
  • Robin Shen commented 4 years ago

    From which version are you performing the upgrade?

  • John commented 4 years ago

    Is there an easy way to tell? I can't start the app to check the version because it just tries to upgrade again and fails/crashes

  • Robin Shen commented 4 years ago

    You may stop OneDev container, and run below command:

    docker run --rm -v <onedev data dir>:/opt/onedev 1dev/server cat /opt/onedev/version.txt
    

    Also please let me know your MariaDB details, including OS and version, jdbc driver version, onedev database character set, etc.

  • John commented 4 years ago

    Running the command returned "7.1.8"

    MariaDB Details:

    OS: Official MariaDB docker image JDBC driver version: 3.0.3

    hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect
    hibernate.connection.driver_class=org.mariadb.jdbc.Driver
    hibernate.connection.url=jdbc:mariadb://<removed>:3306/onedev
    hibernate.connection.username=<removed>
    hibernate.connection.password=<removed>
    

    Database character set - Not sure how to get this, but I can provide the following:

    collation: utf8mb4_unicode_ci engine: InnoDB

  • Robin Shen commented 4 years ago

    Just tested and it works at my side. I am starting MariaDB with below command:

    docker run --name mariadb -p 3306:3306 --env MARIADB_PASSWORD=root --env MARIADB_ROOT_PASSWORD=root --env MARIADB_DATABASE=onedev -v $(pwd)/db:/var/lib/mysql --network mynet mariadb:latest
    

    Please pull your mariadb to update to latest version to see if it works. If still not work, please tell me detailed steps to reproduce this issue.

  • Robin Shen commented 4 years ago

    BTW: you may still get your OneDev up and running with 7.1.8 image, although upgrade fails.

  • John commented 4 years ago

    I've made sure the mariadb docker image I'm using has been updated to the latest version, also I tried starting the app on both 7.1.8 and 7.1.7 after restoring the database from backup, and I was met with the following error:

    INFO  - Launching application from '/app'...
    INFO  - Starting server...
    INFO  - Upgrading /opt/onedev...
    INFO  - >>> INFO  - Launching application from '/opt/onedev'...
    INFO  - >>> INFO  - Cleaning temp directory...
    INFO  - >>> INFO  - Starting server...
    INFO  - Old data version: 84
    INFO  - Backing up old program files as /opt/onedev/site/program-backup/2022-05-16_03-16-15...
    INFO  - Copying new program files into /opt/onedev...
    INFO  - Successfully upgraded /opt/onedev
    03:16:16 INFO  i.onedev.commons.bootstrap.Bootstrap - Launching application from '/opt/onedev'...
    03:16:16 INFO  i.onedev.commons.bootstrap.Bootstrap - Cleaning temp directory...
    03:16:16 INFO  io.onedev.commons.loader.AppLoader - Starting server...
    03:16:23 ERROR i.onedev.commons.bootstrap.Bootstrap - Error booting application
    org.hibernate.type.SerializationException: could not deserialize
            at org.hibernate.internal.util.SerializationHelper.doDeserialize(SerializationHelper.java:243)
            at org.hibernate.internal.util.SerializationHelper.deserialize(SerializationHelper.java:287)
            at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.fromBytes(SerializableTypeDescriptor.java:138)
            at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:120)
            at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:29)
            at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$1.doExtract(BlobTypeDescriptor.java:48)
            at org.hibernate.type.descriptor.sql.BasicExtractor.extract(BasicExtractor.java:47)
            at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:257)
            at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:253)
            at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:243)
            at org.hibernate.type.AbstractStandardBasicType.hydrate(AbstractStandardBasicType.java:329)
            at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:3134)
            at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.loadFromResultSet(EntityReferenceInitializerImpl.java:342)
            at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.hydrateEntityState(EntityReferenceInitializerImpl.java:269)
            at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:102)
            at org.hibernate.loader.plan.exec.internal.EntityLoadQueryDetails$EntityLoaderRowReader.readRow(EntityLoadQueryDetails.java:288)
            at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractRows(ResultSetProcessorImpl.java:157)
            at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:94)
            at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:105)
            at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:285)
            at org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:4441)
            at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4431)
            at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:569)
            at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:537)
            at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:208)
            at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:332)
            at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:108)
            at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:74)
            at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:113)
            at org.hibernate.internal.SessionImpl.fireLoadNoChecks(SessionImpl.java:1186)
            at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1175)
            at org.hibernate.internal.SessionImpl.access$2100(SessionImpl.java:193)
            at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.doLoad(SessionImpl.java:2786)
            at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.lambda$load$1(SessionImpl.java:2767)
            at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.perform(SessionImpl.java:2723)
            at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2767)
            at org.hibernate.internal.SessionImpl.get(SessionImpl.java:983)
            at io.onedev.server.persistence.dao.DefaultDao.get(DefaultDao.java:42)
            at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
            at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
            at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
            at io.onedev.server.persistence.dao.BaseEntityManager.get(BaseEntityManager.java:77)
            at io.onedev.server.maintenance.DefaultDataManager.init(DefaultDataManager.java:118)
            at io.onedev.server.persistence.TransactionInterceptor$1.call(TransactionInterceptor.java:23)
            at io.onedev.server.persistence.DefaultTransactionManager$1.call(DefaultTransactionManager.java:72)
            at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
            at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60)
            at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
            at io.onedev.server.OneDev.start(OneDev.java:111)
            at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44)
            at io.onedev.commons.loader.AppLoader.start(AppLoader.java:73)
            at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:189)
    Caused by: java.io.StreamCorruptedException: invalid stream header: C2ACED9C
            at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:938)
            at java.io.ObjectInputStream.<init>(ObjectInputStream.java:396)
            at org.hibernate.internal.util.SerializationHelper$CustomObjectInputStream.<init>(SerializationHelper.java:309)
            at org.hibernate.internal.util.SerializationHelper$CustomObjectInputStream.<init>(SerializationHelper.java:299)
            at org.hibernate.internal.util.SerializationHelper.doDeserialize(SerializationHelper.java:218)
            ... 51 common frames omitted
    

    This seems to be some sort of deserialization issue in the database possibly? I'm not sure as java isn't my thing

  • Robin Shen commented 4 years ago

    Looks that some database record is corrupted for some reason. You might need to restore from an old backup to get it working.

  • Robin Shen commented 4 years ago

    Or you may send the database backup (you are using to restore database) to me [robin AT onedev DOT io] and I will check what might be wrong.

  • John commented 4 years ago

    Thank you, I sent an email. Would it be easier to restart from scratch? Would I (easily) be able to grab the git repositories I have setup and just migrate them over to another instance?

  • Robin Shen commented 4 years ago

    All git repositories are stored under <onedev data dir>/site/projects identified by project id.

    I have not got the email yet. If attachment is too large, please share a dropbox link.

  • John commented 4 years ago

    Hi Robin,

    The email subject I sent you is "Issue #727 - Database Corruption", I've confirmed its not too big and was able to send it on my end. I took a look at the projects directory but I am unable to find the source code of my projects. Could you elaborate within a project folder how I can find/export the files within the project

  • John commented 4 years ago

    I ended up restoring from a waaaay older database backup taken by onedev on an older version, swapping to the latest (7.2.3) and letting the upgrade migration process take place. It took a while, but eventually succeeded. Thanks, closing.

  • John changed state to 'Closed' 4 years ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen commented 4 years ago

    This upgrade failure is odd. Please take a manual backup of your database (via OneDev's database backup menu) before next upgrade.

issue 1/1
Type
Bug
Priority
Normal
Assignee
Affected Versions
Not Found
Issue Votes (0)
Watchers (4)
Reference
OD-727
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover