#1583  Failing to start correctly due to Error checking unfinished builds on Windows
Closed
aage opened 7 months ago

Tried to restart onedev Windows service (wrapper service) on Windows Server 2022 Standard, the Windows service will start (i.e. is in running state) but the GUI will return an HTTP 500 error.

Troubleshooting Tried to do the following (based on https://code.onedev.io/onedev/server/~issues/1055):

  • Stop windows service
  • Delete everything in /temp/server
  • Start Windows

But files keep returning in this directory after restart and the error remains. Any other workaround suggestions would be very welcome.

Error in browser

image.png

Stacktrace

ERROR [Thread-14] i.o.server.job.DefaultJobManager Error checking unfinished builds java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.hibernate.type.SerializationException: could not deserialize at io.onedev.server.job.DefaultJobManager.run(DefaultJobManager.java:1185) at java.base/java.lang.Thread.run(Thread.java:1589) Caused by: java.util.concurrent.ExecutionException: org.hibernate.type.SerializationException: could not deserialize at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:278) at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1589) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: org.hibernate.type.SerializationException: could not deserialize at org.hibernate.internal.util.SerializationHelper.doDeserialize(SerializationHelper.java:231) at org.hibernate.internal.util.SerializationHelper.deserialize(SerializationHelper.java:287) at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.fromBytes(SerializableTypeDescriptor.java:138) at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:120) at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:29) at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$1.doExtract(BlobTypeDescriptor.java:48) at org.hibernate.type.descriptor.sql.BasicExtractor.extract(BasicExtractor.java:47) at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:257) at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:253) at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:243) at org.hibernate.type.AbstractStandardBasicType.hydrate(AbstractStandardBasicType.java:329) at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:3134) at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.loadFromResultSet(EntityReferenceInitializerImpl.java:342) at org.hibernate.loader.plan.exec.process.internal.EntityReferenceInitializerImpl.hydrateEntityState(EntityReferenceInitializerImpl.java:269) at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:102) at org.hibernate.loader.plan.exec.internal.EntityLoadQueryDetails$EntityLoaderRowReader.readRow(EntityLoadQueryDetails.java:288) at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractRows(ResultSetProcessorImpl.java:157) at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:94) at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:105) at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:285) at org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:4441) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4431) at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:569) at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:537) at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:208) at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:103) at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:74) at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:113) at org.hibernate.internal.SessionImpl.fireLoadNoChecks(SessionImpl.java:1186) at org.hibernate.internal.SessionImpl.immediateLoad(SessionImpl.java:1005) at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:179) at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:310) at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyInterceptor.intercept(ByteBuddyInterceptor.java:45) at org.hibernate.proxy.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:95) at io.onedev.server.model.Build$HibernateProxy$aX9WmXmz.getStatus(Unknown Source) at io.onedev.server.job.DefaultJobManager.lambda$run$10(DefaultJobManager.java:1135) at io.onedev.server.persistence.DefaultTransactionManager.lambda$run$1(DefaultTransactionManager.java:95) at io.onedev.server.persistence.DefaultTransactionManager.lambda$call$0(DefaultTransactionManager.java:69) at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90) at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60) at io.onedev.server.persistence.DefaultTransactionManager.run(DefaultTransactionManager.java:94) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) at java.base/java.lang.reflect.Method.invoke(Method.java:578) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at jdk.proxy2/jdk.proxy2.$Proxy24.run(Unknown Source) at io.onedev.server.job.DefaultJobManager.lambda$run$13507acc$1(DefaultJobManager.java:1132) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:276) ... 6 common frames omitted Caused by: java.io.EOFException: null at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2940) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3717) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3505) at java.base/java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1262) at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:673) at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:1012) at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2048) at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1925) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2248) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1760) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:538) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:496) at org.hibernate.internal.util.SerializationHelper.doDeserialize(SerializationHelper.java:225) ... 53 common frames omitted

Robin Shen commented 7 months ago

Some records of running builds in database are corrupted for some reason. Which database are you using?

aage commented 7 months ago

Hey @robin

From what I can tell from the hibernate.properties it uses HSQLDB (I haven't done the install). Does this answer your question?

Thank you

Robin Shen commented 7 months ago

For production use, please switch to one of the supported external database both for reliability and performance reason. To solve this specific issue, you will need to download hsqldb tool (https://hsqldb.org/), open the database (the "internaldb" folder under OneDev's installation directory), and then delete all records in table O_BUILD with pending/waiting/runing status.

aage commented 7 months ago

Thank you @robin , this helped me out. Will look into using a different type of database.

My steps to fix this issue (might help some others)

  • Download tool via https://hsqldb.org/
  • Stop the onedev Windows Service
  • Run this on the server where OneDev is installed by:
  • Running the runManagerSwing.bat file in the bin dir
  • Pointing it to the database by using the connection string found in hibernate.properties in the OneDev conf dir
  • Now deleting all builds (O_BUILD) with (in my case only) the waiting status (O_STATUS) using a SQL statement (had to look into the source of OneDev to find out integer maps to which status, which I could find here: src). These are the statusses and accompanying integers for those interested: WAITING 0, PENDING 1, RUNNING 2, FAILED 3, CANCELLED 4, TIMED_OUT 5, SUCCESSFUL 6.
  • Now starting the Windows Service had OneDev up and running again
Robin Shen changed state to 'Closed' 7 months ago
Previous Value Current Value
Open
Closed
Robin Shen commented 7 months ago

@aage thank you for the great detail. 👍

Robin Shen commented 7 months ago

Please follow below on how to switch database: https://docs.onedev.io/administration-guide/switch-database

Also make sure to schedule auto backup in administration / system maintenance / database backup.

issue 1 of 1
Type
Bug
Priority
Normal
Assignee
Affected Versions
9.1.5
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
onedev/server#1583
Please wait...
Page is in error, reload to recover