OD-3180: One corrupt build blob (after disk full) makes other blobs return wrong data; SSO login disappears

One corrupt build blob (after disk full) makes other blobs return wrong data; SSO login disappears #3180

Hille opened 2 hours ago

Environment: OneDev 16.8.7, embedded HSQLDB 2.7.1 (LOB COMPRESSED TRUE), Temurin JDK 21, Debian LXC. (Proxmox CT)

What happened:

The disk filled up during a "Pull from GitHub" job (large repository fetch). The build had been created about a minute earlier.
After that, the log showed this every 5 seconds: DefaultJobService Error checking unfinished builds: java.lang.ClassCastException: Cannot cast java.util.ArrayList to java.util.LinkedHashMap
After freeing space and restarting, the server started normally ("Server is ready"), but the login page no longer showed the SSO button. Log: GeneralErrorPage Error processing wicket request: java.lang.ClassCastException: Cannot cast java.util.ArrayList to io.onedev.server.model.support.administration.sso.SsoConnector

Analysis:

Decompressing every LOB offline showed exactly one unreadable blob: a blob column of the build created just before the disk filled.
The SSO provider's blob was intact on disk (it decodes to OpenIdConnector, byte-identical to a backup), yet at runtime it was returned as an ArrayList.
Suspicion: after the failed inflate of the corrupt LOB, HSQLDB returns stale buffer contents for subsequent LOB reads. Because the unfinished-build check re-reads the corrupt build every 5 seconds, all other blob reads were affected.

Workaround: With the server stopped, I overwrote the corrupt LOB block with a valid one from another build and fixed its PARTS/LOB_IDS lengths in onedev.script. All errors stopped and SSO login returned.

Expected: A corrupt blob on one build should fail only that build (ideally marking it failed), not affect unrelated entities or authentication.

2026-10-12 21:58:28,269 ERROR [hz.127.0.0.1:5710.cached.thread-6] i.o.server.job.DefaultJobService Error checking unfinished builds
java.lang.ClassCastException: Cannot cast java.util.ArrayList to java.util.LinkedHashMap
        at java.base/java.lang.Class.cast(Class.java:4069)
        at org.hibernate.type.descriptor.java.JavaType.cast(JavaType.java:125)
        at org.hibernate.type.descriptor.java.SerializableJavaType.fromBytes(SerializableJavaType.java:146)
        at org.hibernate.type.descriptor.java.SerializableJavaType.wrap(SerializableJavaType.java:129)
        at org.hibernate.type.descriptor.java.SerializableJavaType.wrap(SerializableJavaType.java:30)
        at org.hibernate.type.descriptor.jdbc.BlobJdbcType$1.doExtract(BlobJdbcType.java:62)
        at org.hibernate.type.descriptor.jdbc.BasicExtractor.extract(BasicExtractor.java:43)
        at org.hibernate.sql.results.jdbc.internal.JdbcValuesResultSetImpl.getCurrentRowValue(JdbcValuesResultSetImpl.java:381)
        at org.hibernate.sql.results.internal.RowProcessingStateStandardImpl.getJdbcValue(RowProcessingStateStandardImpl.java:148)
        at org.hibernate.sql.results.graph.basic.BasicResultAssembler.extractRawValue(BasicResultAssembler.java:52)
        at org.hibernate.sql.results.graph.basic.BasicResultAssembler.assemble(BasicResultAssembler.java:57)
        at org.hibernate.sql.results.graph.entity.internal.EntityInitializerImpl.extractConcreteTypeStateValues(EntityInitializerImpl.java:2030)
        at org.hibernate.sql.results.graph.entity.internal.EntityInitializerImpl.initializeEntityInstance(EntityInitializerImpl.java:1691)
        at org.hibernate.sql.results.graph.entity.internal.EntityInitializerImpl.initializeInstance(EntityInitializerImpl.java:1597)
        at org.hibernate.sql.results.graph.entity.internal.EntityInitializerImpl.initializeInstance(EntityInitializerImpl.java:88)
        at org.hibernate.sql.results.internal.StandardRowReader.coordinateInitializers(StandardRowReader.java:239)
        at org.hibernate.sql.results.internal.StandardRowReader.readRow(StandardRowReader.java:137)
        at org.hibernate.sql.results.spi.ListResultsConsumer.readUniqueAssert(ListResultsConsumer.java:258)
        at org.hibernate.sql.results.spi.ListResultsConsumer.readRows(ListResultsConsumer.java:230)
        at org.hibernate.sql.results.spi.ListResultsConsumer.consume(ListResultsConsumer.java:164)
        at org.hibernate.sql.results.spi.ListResultsConsumer.consume(ListResultsConsumer.java:31)
        at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.doExecuteQuery(JdbcSelectExecutorStandardImpl.java:213)
        at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.executeQuery(JdbcSelectExecutorStandardImpl.java:100)
        at org.hibernate.sql.exec.spi.JdbcSelectExecutor.executeQuery(JdbcSelectExecutor.java:63)
        at org.hibernate.sql.exec.spi.JdbcSelectExecutor.list(JdbcSelectExecutor.java:137)
        at org.hibernate.loader.ast.internal.SingleIdLoadPlan.load(SingleIdLoadPlan.java:139)
        at org.hibernate.loader.ast.internal.SingleIdLoadPlan.load(SingleIdLoadPlan.java:111)
        at org.hibernate.loader.ast.internal.SingleIdEntityLoaderStandardImpl.load(SingleIdEntityLoaderStandardImpl.java:64)
        at org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:3676)

Copy

Activities

Robin Shen commented 54 minutes ago

You have to restore the database from a backup. For production use, please switch to use external database.

Copy
Login to comment

Type	Bug
Priority	Normal
Assignee	Robin Shen
Affected Versions	16.8.7
Labels	No labels

Issue Votes (0)

Watchers (2)

Reference

OD-3180