#1623  Lost issue
Closed
Jerome St-Louis opened 6 months ago

I just noticed that an issue no longer exists on our instance. I get the 500 Panda:

OOPS! There Is An Error
Unable to find commit associated with object id: AnyObjectId[fbee0d4c1bd947315b0db3d81fa639525e34ed39]

This only happens for this specific issue, but it is worrisome, because this is already a significant loss of useful information in this one issue. We are using PostgreSQL as a backend. How could we assess what went wrong? I do suspect a critical OneDev bug.

The message mentions that it cannot find a commit.

Is the issue still there and fine, and it just has a reference to a commit that perhaps no longer exists, but as a result even the issue cannot be displayed? (this should not result in a 500).

Robin Shen commented 6 months ago

What url you are accessing?

Jerome St-Louis commented 6 months ago
Jerome St-Louis commented 6 months ago

@robin Interestingly (and this is reassuring for the issue data still being there), the mentioned object ID definitely is a commit that I still have in my local workspace.

Robin Shen commented 6 months ago

When displaying the issue, OneDev also checks git commit it associated with. Maybe that commit is missing. Please show me the full stack trace.

Jerome St-Louis commented 6 months ago

@robin That is the only information that is displayed in the 500, not any other detail.

I tried to push the commit again to a branch to the server and I can see the commit fine at:

https://example.com/myproject/~commits/fbee0d4c1bd947315b0db3d81fa639525e34ed39

but I still get the error at the issue URL (with no details).

Is there a way to get a more detailed stack trace?

Robin Shen commented 6 months ago

Please check the server log to see if there is more info there.

Jerome St-Louis commented 6 months ago

@robin I don't see anything in the server log, but there seems to be nothing there for the last 5 days which is odd.

I tried to restart the server, and it does write a new server.log, but it doesn't write anything there when trying to access the issue.

Robin Shen commented 6 months ago

Please make sure you are on 9.2.3, then copy attached file to <OneDev server install dir>/lib. This will give more info of the error.

io.onedev.server-core-9.2.3.jar

Jerome St-Louis commented 6 months ago

@robin

org.apache.wicket.WicketRuntimeException: Error attaching this container for rendering: [Page class = io.onedev.server.web.page.project.issues.detail.IssueActivitiesPage, id = 0, render count = 1]
	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1838)
	at org.apache.wicket.Component.onBeforeRender(Component.java:3921)
	at org.apache.wicket.Page.onBeforeRender(Page.java:801)
	at io.onedev.server.web.page.base.BasePage.onBeforeRender(BasePage.java:304)
	at org.apache.wicket.Component.internalBeforeRender(Component.java:950)
	at org.apache.wicket.Component.beforeRender(Component.java:1018)
	at org.apache.wicket.Component.internalPrepareForRender(Component.java:2236)
	at org.apache.wicket.Page.internalPrepareForRender(Page.java:242)
	at org.apache.wicket.Component.render(Component.java:2327)
	at org.apache.wicket.Page.renderPage(Page.java:1018)
	at org.apache.wicket.request.handler.render.WebPageRenderer.renderPage(WebPageRenderer.java:124)
	at org.apache.wicket.request.handler.render.WebPageRenderer.respond(WebPageRenderer.java:236)
	at org.apache.wicket.core.request.handler.RenderPageRequestHandler.respond(RenderPageRequestHandler.java:175)
	at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:917)
	at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
	at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:274)
	at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:231)
	at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:302)
	at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70)
	at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
	at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:43)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$177280726.GUICE$TRAMPOLINE(<generated>)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
	at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
	at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
	at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$177280726.service(<generated>)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:458)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:372)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.onedev.server.git.exception.ObjectNotFoundException: Unable to find commit associated with object id: AnyObjectId[a92a406276ee8fbba81864d372b0450e62d608bb]
	at io.onedev.server.model.Project.getRevCommit(Project.java:878)
	at io.onedev.server.model.Project.getRevCommit(Project.java:862)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyInterceptor.intercept(ByteBuddyInterceptor.java:56)
	at org.hibernate.proxy.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:95)
	at io.onedev.server.model.Project$HibernateProxy$9ZNo7HP9.getRevCommit(Unknown Source)
	at io.onedev.server.web.DefaultUrlManager.urlFor(DefaultUrlManager.java:101)
	at io.onedev.server.web.DefaultUrlManager.urlFor(DefaultUrlManager.java:72)
	at io.onedev.server.notification.ActivityDetail.referencedFrom(ActivityDetail.java:146)
	at io.onedev.server.model.support.issue.changedata.IssueReferencedFromCodeCommentData.getActivityDetail(IssueReferencedFromCodeCommentData.java:56)
	at io.onedev.server.web.component.issue.activities.activity.IssueChangePanel.onInitialize(IssueChangePanel.java:42)
	at org.apache.wicket.Component.fireInitialize(Component.java:878)
	at org.apache.wicket.MarkupContainer$3.component(MarkupContainer.java:1077)
	at org.apache.wicket.MarkupContainer$3.component(MarkupContainer.java:1073)
	at org.apache.wicket.util.visit.Visits.visitChildren(Visits.java:144)
	at org.apache.wicket.util.visit.Visits.visitChildren(Visits.java:123)
	at org.apache.wicket.util.visit.Visits.visitChildren(Visits.java:192)
	at org.apache.wicket.MarkupContainer.visitChildren(MarkupContainer.java:984)
	at org.apache.wicket.MarkupContainer.internalInitialize(MarkupContainer.java:1072)
	at org.apache.wicket.MarkupContainer.addedComponent(MarkupContainer.java:1048)
	at org.apache.wicket.MarkupContainer.add(MarkupContainer.java:242)
	at io.onedev.server.web.component.issue.activities.IssueActivitiesPanel.onBeforeRender(IssueActivitiesPanel.java:99)
	at org.apache.wicket.Component.internalBeforeRender(Component.java:950)
	at org.apache.wicket.Component.beforeRender(Component.java:1018)
	at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1826)
	... 91 more
Jerome St-Louis changed fields 6 months ago
Name Previous Value Current Value
Type
Support Request
Bug
Affected Versions
empty
9.2.3
Robin Shen commented 6 months ago

Seems that this issue is referenced from a code comment on commit a92a406276ee8fbba81864d372b0450e62d608bb, which no longer exists (possibly due to GC). Please check if this is the case. If so, you may remove that code comment (since commit is removed and the code comment can no longer be displayed) to get around the issue before the release fixing the issue is available.

Robin Shen changed fields 6 months ago
Name Previous Value Current Value
Priority
Critical
Major
Jerome St-Louis commented 6 months ago

@robin Thanks I didn't notice that the detailed error log had a different commit ID (I had pushed a branch with first lost commit).

I pushed this new commit again (which luckily I still had locally) and that did solve the problem and my issue 238 is now back!!

However, it is a major bug on a number of levels that compounded into completely losing access to the issue, and all of these should be fixed:

  • Missing references should never cause an all out failure to show anything at all for the issue (no 500, but "invalid" something, like it does right now for code comments)
  • It should still be possible to see those code comments even if the commit was somehow garbage collected (right now they get an invalid link, so I cannot even see which commit is the problem from the code comments). You could still show the actual comment and file / line numbers, even if the code cannot be shown alongside it, and also the commit ID that used to exist.
  • Some tag or something should hold on to commits for which there are code comments, so that they do not get garbage collected

Code comments are almost always on draft commits that will not end up integrated, so when the branches are gone because the improved version gets merged (we started to try cleaning up branches upon merging), this will always happen, so this is a very common scenario, not a special rare one. However, the code comments are still useful documentation, so a setting could let the administrator decide to either:

  • automatically delete code comments on garbage collected commits (no longer referenced by any branch), or
  • automatically tag commits on which there are comments so as to preserve them.

How can I know which commit this is on:

https://example.com/myproject/~code-comments/4/invalid ?

I would like to see the commit ID so that I can revive it. But I don't really want to keep branches around only to keep those code comments alive.

Robin Shen commented 6 months ago

Filed an improvement request (#1624) to show code comment detail upon commit GC.

As to an option to tag commit to avoid being GCed, it may bloat the ref database.

Robin Shen changed state to 'Closed' 6 months ago
Previous Value Current Value
Open
Closed
Robin Shen commented 6 months ago

Replaced with #1625 which describes the problem more accurately.

issue 1 of 1
Type
Bug
Priority
Major
Assignee
Affected Versions
9.2.3
Labels
No labels
Issue Votes (0)
Watchers (4)
Reference
onedev/server#1623
Please wait...
Page is in error, reload to recover