OneDev
opened 2 days ago
(on behalf of onedev.io)
Details of support request from Jake Ceballos
I was running a job that builds and publishes a container image. When I attempt to access the builds view for the project I get an exception page with the output pasted below. I can access the builds view for other projects but the global builds view shows the exception. Also, when trying to fetch the build via the REST API I get a 500 error. Fetching other builds works. I recently upgraded from 16.3.x to 16.4.2. App and builds was working fine post-upgrade. Don't think it's the root cause but may be something worth noting. ``` org.apache.wicket.WicketRuntimeException: Error attaching this container for rendering: [WebMarkupContainer [Component id = body]] at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1838) at org.apache.wicket.Component.onBeforeRender(Component.java:3934) at org.apache.wicket.Component.internalBeforeRender(Component.java:958) at org.apache.wicket.Component.beforeRender(Component.java:1028) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1826) at org.apache.wicket.Component.onBeforeRender(Component.java:3934) at org.apache.wicket.Component.internalBeforeRender(Component.java:958) at org.apache.wicket.Component.beforeRender(Component.java:1028) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1826) at org.apache.wicket.Component.onBeforeRender(Component.java:3934) at org.apache.wicket.Component.internalBeforeRender(Component.java:958) at org.apache.wicket.Component.beforeRender(Component.java:1028) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1826) at org.apache.wicket.Component.onBeforeRender(Component.java:3934) at org.apache.wicket.Component.internalBeforeRender(Component.java:958) at org.apache.wicket.Component.beforeRender(Component.java:1028) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1826) at org.apache.wicket.Component.onBeforeRender(Component.java:3934) at org.apache.wicket.Page.onBeforeRender(Page.java:801) at io.onedev.server.web.page.base.BasePage.onBeforeRender(BasePage.java:396) at org.apache.wicket.Component.internalBeforeRender(Component.java:958) at org.apache.wicket.Component.beforeRender(Component.java:1028) at org.apache.wicket.Component.internalPrepareForRender(Component.java:2249) at org.apache.wicket.Page.internalPrepareForRender(Page.java:242) at org.apache.wicket.Component.render(Component.java:2342) at org.apache.wicket.Page.renderPage(Page.java:1018) at org.apache.wicket.request.handler.render.WebPageRenderer.renderPage(WebPageRenderer.java:124) at org.apache.wicket.request.handler.render.WebPageRenderer.respond(WebPageRenderer.java:236) at org.apache.wicket.core.request.handler.RenderPageRequestHandler.respond(RenderPageRequestHandler.java:175) at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:892) at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64) at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:261) at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:218) at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:289) at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70) at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203) at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.onedev.server.web.DefaultWicketServlet.access$001(DefaultWicketServlet.java:18) at io.onedev.server.web.DefaultWicketServlet.lambda$service$0(DefaultWicketServlet.java:50) at io.onedev.server.persistence.DefaultSessionService.lambda$run$0(DefaultSessionService.java:103) at io.onedev.server.persistence.DefaultSessionService.call(DefaultSessionService.java:85) at io.onedev.server.persistence.DefaultSessionService.run(DefaultSessionService.java:102) at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:44) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:90) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:499) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:396) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.security.CorsFilter.doFilter(CorsFilter.java:47) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: IOException occurred reading a binary value at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1602) at org.hibernate.query.Query.getResultList(Query.java:165) at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:76) at io.onedev.server.service.impl.DefaultBuildService.query(DefaultBuildService.java:638) at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23) at io.onedev.server.persistence.DefaultSessionService.call(DefaultSessionService.java:85) at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18) at io.onedev.server.web.component.build.list.BuildListPanel$16.iterator(BuildListPanel.java:1051) at org.apache.wicket.markup.repeater.data.DataViewBase$ModelIterator.<init>(DataViewBase.java:107) at org.apache.wicket.markup.repeater.data.DataViewBase.getItemModels(DataViewBase.java:74) at org.apache.wicket.markup.repeater.AbstractPageableView.getItemModels(AbstractPageableView.java:101) at org.apache.wicket.markup.repeater.RefreshingView.onPopulate(RefreshingView.java:93) at org.apache.wicket.markup.repeater.AbstractRepeater.onBeforeRender(AbstractRepeater.java:124) at org.apache.wicket.markup.repeater.AbstractPageableView.onBeforeRender(AbstractPageableView.java:115) at org.apache.wicket.Component.internalBeforeRender(Component.java:958) at org.apache.wicket.Component.beforeRender(Component.java:1028) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1826) ... 116 more Caused by: org.hibernate.HibernateException: IOException occurred reading a binary value at org.hibernate.type.descriptor.java.DataHelper.extractBytes(DataHelper.java:174) at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:120) at org.hibernate.type.descriptor.java.SerializableTypeDescriptor.wrap(SerializableTypeDescriptor.java:29) at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$1.doExtract(BlobTypeDescriptor.java:48) at org.hibernate.type.descriptor.sql.BasicExtractor.extract(BasicExtractor.java:47) at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:257) at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:253) at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:243) at org.hibernate.type.AbstractStandardBasicType.hydrate(AbstractStandardBasicType.java:329) at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:3134) at org.hibernate.loader.Loader.loadFromResultSet(Loader.java:1870) at org.hibernate.loader.Loader.hydrateEntityState(Loader.java:1798) at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1771) at org.hibernate.loader.Loader.getRow(Loader.java:1623) at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:740) at org.hibernate.loader.Loader.getRowsFromResultSet(Loader.java:1039) at org.hibernate.loader.Loader.processResultSet(Loader.java:990) at org.hibernate.loader.Loader.doQuery(Loader.java:959) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349) at org.hibernate.loader.Loader.doList(Loader.java:2850) at org.hibernate.loader.Loader.doList(Loader.java:2832) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664) at org.hibernate.loader.Loader.list(Loader.java:2659) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414) at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593) ... 132 more Caused by: java.io.IOException: org.hsqldb.HsqlException: java.io.EOFException at org.hsqldb.lib.java.JavaSystem.toIOException(Unknown Source) at org.hsqldb.types.BlobInputStream.read(Unknown Source) at java.base/java.io.InputStream.read(InputStream.java:284) at java.base/java.io.InputStream.read(InputStream.java:218) at org.hibernate.type.descriptor.java.DataHelper.extractBytes(DataHelper.java:166) ... 160 more Caused by: org.hsqldb.HsqlException: java.io.EOFException at org.hsqldb.error.Error.error(Unknown Source) at org.hsqldb.types.BlobDataID.getBytes(Unknown Source) at org.hsqldb.types.BlobInputStream.readIntoBuffer(Unknown Source) ... 164 more ```
Basic Info
| Product Version | Agent Count | User Count | Licensed To | Remaining User Months |
|---|---|---|---|---|
| 16.4.2 | 2 | 1 | ceballos technologies | 10 |
Cluster Members
| Install Type | OS | JVM | Total Memory | Used Memory | System Date |
|---|---|---|---|---|---|
| Docker | Linux 6.12.91-fly, amd64 | OpenJDK 64-Bit Server VM 17.0.19, Ubuntu | 986 MB | 312 MB | 2026-08-08 00:33:23 +0000 |
Subscription Key UUIDs
Copy
-
This stack trace points to corrupted LOB data in the embedded HSQLDB for one (or more)
Buildrows in that project — not a general UI/API bug, and not something the 16.3 → 16.4.2 upgrade itself typically causes.What is failing
When OneDev lists or loads builds, Hibernate reads a binary column on
o_Build(o_checkoutPaths). For a bad row, HSQLDB throwsEOFExceptionwhile reading the blob, which surfaces as:- project Builds page exception
- global Builds page exception (because that list includes the bad build)
- REST
GETof that build → HTTP 500
Other projects work because their build rows are still readable. That matches what you reported after the container-image publish job.
Your host also shows only ~1 GB RAM for the OneDev container. Publishing a container image on the embedded DB with that little memory can OOM / hard-kill the process and leave HSQLDB LOB files truncated. That is a common way to get exactly this failure.
Recovery (Docker + embedded HSQLDB)
UI/API delete will also try to load the bad row, so clean it up with SQL while OneDev is stopped.
Replace
YOUR_PROJECT_PATHandYOUR_BUILD_NUMBERwith the project path and the build number that returns 500 from the REST API. Run these from the directory that contains your compose file and data volume (often./onedevor./onedev_datamounted at/opt/onedev).0. Stop and back up
docker compose down # adjust the data dir name if yours differs cp -a ./onedev "./onedev.bak-$(date +%Y%m%d%H%M%S)"1. Download SqlTool (once)
curl -fsSL -o /tmp/sqltool-2.7.1.jar \ https://repo1.maven.org/maven2/org/hsqldb/sqltool/2.7.1/sqltool-2.7.1.jar2. Resolve the build id (does not read the bad LOB)
cat > /tmp/onedev-find-build.sql <<'SQL' SELECT p.o_id AS project_id, p.o_path, b.o_id AS build_id, b.o_number, b.o_jobName FROM o_Build b JOIN o_Project p ON p.o_id = b.o_project_id WHERE p.o_path = 'YOUR_PROJECT_PATH' AND b.o_number = YOUR_BUILD_NUMBER; COMMIT; SHUTDOWN; SQL docker run --rm \ -v "$(pwd)/onedev:/data" \ -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \ -v /tmp/onedev-find-build.sql:/find.sql:ro \ --entrypoint bash \ 1dev/server:16.4.2 \ -c 'set -e HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1) echo "Using HSQLDB jar: $HSQL" java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \ --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \ /find.sql'Note the
build_idfrom that output.3. Delete the bad build (keep published packages)
This removes the broken build row and its dependent rows, but keeps package records by clearing their build link first (so the published container image metadata can remain).
# set BUILD_ID from step 2 BUILD_ID=REPLACE_WITH_BUILD_ID cat > /tmp/onedev-delete-bad-build.sql <<SQL -- keep packages; only detach them from the bad build UPDATE o_Pack SET o_build_id = NULL WHERE o_build_id = ${BUILD_ID}; DELETE FROM o_BuildLabel WHERE o_build_id = ${BUILD_ID}; DELETE FROM o_BuildParam WHERE o_build_id = ${BUILD_ID}; DELETE FROM o_UnitTestMetric WHERE o_build_id = ${BUILD_ID}; DELETE FROM o_CoverageMetric WHERE o_build_id = ${BUILD_ID}; DELETE FROM o_ProblemMetric WHERE o_build_id = ${BUILD_ID}; DELETE FROM o_BuildDependence WHERE o_dependent_id = ${BUILD_ID} OR o_dependency_id = ${BUILD_ID}; DELETE FROM o_Build WHERE o_id = ${BUILD_ID}; COMMIT; SHUTDOWN; SQL docker run --rm \ -v "$(pwd)/onedev:/data" \ -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \ -v /tmp/onedev-delete-bad-build.sql:/cleanup.sql:ro \ --entrypoint bash \ 1dev/server:16.4.2 \ -c 'set -e HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1) java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \ --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \ /cleanup.sql'If your data volume directory is not named
./onedev, change the-v "$(pwd)/onedev:/data"mount accordingly (it must be the host dir that maps to/opt/onedev).4. Start again
docker compose up -dThen reopen the project Builds page and the global Builds page. If either still fails, another build row is also corrupted — repeat steps 2–3 for the next build that returns REST 500 (or tell us the project path / build numbers and we can refine the SQL).
If you have a recent DB backup
Restoring a backup taken before the bad build is also fine:
https://docs.onedev.io/administration-guide/backup-restore
Hardening (recommended)
- Give the OneDev container more memory (2 GB+ is much safer when building/publishing images).
- For production, use an external database (PostgreSQL/MySQL) instead of embedded HSQLDB — see
conf/hibernate.properties/ the Docker Compose example that ships with OneDev. - Enable periodic database backup under Administration → Database Backup.
Please reply with whether the Builds pages load again after deleting the bad build, or paste the SqlTool output if something fails. If this resolves it, we can close the issue.
-
Previous Value Current Value Open
Closed
| Type |
Question
|
| Priority |
Normal
|
| Assignee | |
| Labels |
No labels
|
Issue Votes (0)
Watchers (2)
External Participants
-
onedev.io
External participants do not have accounts and involve in the issue via email