OneDev
Dashboards
Projects
Pull Requests
Issues
Builds
Packages
Code Search
server
Code
Pull Requests
Issues
List
Boards
Milestones
Builds
Packages
Statistics
Child Projects
OneDev 10.5.2
Documentation Support & Bug Report RESTful API Incompatibilities License Agreement Check Update
Projects onedev server Issues #1769
#1769  Agents fail to update after server upgrade to 10.1.6
Closed
narandill opened 2 months ago

Hi Robin,

I just updated to fix the issue #1765 with Publishing Artifacts, but suddenly all my Agents went offline (but I see them running in host machines or docker):

Screenshot 2024-02-22 at 23.25.21.png

In OneDev Server docker container I got this:

--> Wrapper Started as Console
]0;OneDevJava Service Wrapper Standard Edition 64-bit 3.5.51
  Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping
Launching a JVM...
WrapperManager: Initializing...
INFO  - Launching application from '/app'...
INFO  - Starting application...
INFO  - Upgrading /opt/onedev...
INFO  - >>> INFO  - Launching application from '/opt/onedev'...
INFO  - >>> INFO  - Cleaning temp directory...
INFO  - >>> INFO  - Starting application...
INFO  - >>> INFO  - Waiting for server to stop...
INFO  - >>> INFO  - Stopping application...
INFO  - Backing up old program files as /opt/onedev/site/program-backup/2024-02-22_23-07-53...
INFO  - Copying new program files into /opt/onedev...
INFO  - Successfully upgraded /opt/onedev
INFO  - Stopping application...
<-- Wrapper Stopped
--> Wrapper Started as Console
]0;OneDevJava Service Wrapper Standard Edition 64-bit 3.5.51
  Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping
Launching a JVM...
WrapperManager: Initializing...
23:07:58 INFO  i.onedev.commons.bootstrap.Bootstrap - Launching application from '/opt/onedev'...
23:07:58 INFO  i.onedev.commons.bootstrap.Bootstrap - Cleaning temp directory...
23:07:58 INFO  io.onedev.commons.loader.AppLoader - Starting application...
23:08:32 INFO  i.o.s.e.i.DefaultAgentAttributeManager - Caching agent attribute info...
23:08:32 INFO  i.o.s.e.impl.DefaultIssueManager - Caching issue info...
23:08:33 INFO  i.o.s.e.impl.DefaultProjectManager - Checking projects...
23:08:37 INFO  i.o.s.e.impl.DefaultBuildManager - Caching build info...
23:08:37 INFO  i.o.s.e.i.DefaultBuildParamManager - Caching build param info...
23:08:38 INFO  i.o.s.e.i.DefaultBuildMetricManager - Caching build metric info...
23:08:45 ERROR i.o.s.w.ResourceErrorRequestHandler - Error processing resource request
org.apache.wicket.protocol.http.servlet.ResponseIOException: org.eclipse.jetty.io.EofException
	at org.apache.wicket.protocol.http.servlet.ServletWebResponse.write(ServletWebResponse.java:140)
	at org.apache.wicket.protocol.http.HeaderBufferingWebResponse.write(HeaderBufferingWebResponse.java:196)
	at org.apache.wicket.request.Response$StreamAdapter.write(Response.java:148)
	at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:127)
	at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:147)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:93)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:266)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:176)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:213)
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:669)
	at io.onedev.commons.utils.TarUtils.addTarEntry(TarUtils.java:102)
	at io.onedev.commons.utils.TarUtils$1.visitFile(TarUtils.java:53)
	at io.onedev.commons.utils.TarUtils$1.visitFile(TarUtils.java:36)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
	at io.onedev.commons.utils.TarUtils.tar(TarUtils.java:36)
	at io.onedev.server.web.resource.AgentLibResource$1.writeData(AgentLibResource.java:55)
	at org.apache.wicket.request.resource.AbstractResource.respond(AbstractResource.java:660)
	at org.apache.wicket.request.handler.resource.ResourceRequestHandler.respond(ResourceRequestHandler.java:105)
	at org.apache.wicket.request.handler.resource.ResourceReferenceRequestHandler.respond(ResourceReferenceRequestHandler.java:108)
	at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:891)
	at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
	at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:260)
	at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:217)
	at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:288)
	at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70)
	at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
	at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at io.onedev.server.web.DefaultWicketServlet.lambda$service$0(DefaultWicketServlet.java:45)
	at io.onedev.server.persistence.DefaultSessionManager.lambda$run$0(DefaultSessionManager.java:108)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
	at io.onedev.server.persistence.DefaultSessionManager.run(DefaultSessionManager.java:107)
	at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:42)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:458)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:372)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.security.CorsFilter.doFilter(CorsFilter.java:47)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
	Suppressed: org.apache.wicket.protocol.http.servlet.ResponseIOException: org.eclipse.jetty.io.EofException: Closed
		at org.apache.wicket.protocol.http.servlet.ServletWebResponse.write(ServletWebResponse.java:140)
		at org.apache.wicket.protocol.http.HeaderBufferingWebResponse.write(HeaderBufferingWebResponse.java:196)
		at org.apache.wicket.request.Response$StreamAdapter.write(Response.java:148)
		at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
		at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
		at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
		at org.apache.commons.io.IOUtils.close(IOUtils.java:444)
		at org.apache.commons.io.output.ProxyOutputStream.close(ProxyOutputStream.java:90)
		at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.close(FixedLengthBlockOutputStream.java:72)
		at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.close(FixedLengthBlockOutputStream.java:149)
		at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:307)
		at io.onedev.commons.utils.TarUtils.tar(TarUtils.java:33)
		... 90 common frames omitted
	Caused by: org.eclipse.jetty.io.EofException: Closed
		at org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:771)
		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:795)
		at org.apache.wicket.protocol.http.servlet.ServletWebResponse.write(ServletWebResponse.java:136)
		... 101 common frames omitted
Caused by: org.eclipse.jetty.io.EofException: null
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:280)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:831)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:248)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:229)
	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:555)
	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1014)
	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1086)
	at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:111)
	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285)
	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:269)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:896)
	at org.apache.wicket.protocol.http.servlet.ServletWebResponse.write(ServletWebResponse.java:136)
	... 106 common frames omitted
Caused by: java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493)
	at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:507)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:274)
	... 120 common frames omitted

One of the Agents (Docker Executor):

23:12:17 INFO  io.onedev.agent.Agent - Connecting to http://192.168.0.XXX:6610...
23:12:17 INFO  io.onedev.agent.AgentSocket - Connected to server
23:12:17 INFO  io.onedev.agent.AgentSocket - Updating agent to version 1.10.8...
23:12:17 ERROR io.onedev.agent.AgentSocket - Websocket error
java.lang.RuntimeException: Cannot call method public void io.onedev.agent.AgentSocket#onMessage(byte[], int, int) with args: [[B, java.lang.Integer, java.lang.Integer]
	at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.unwrapRuntimeException(CallableMethod.java:93)
	at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:75)
	at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
	at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryMessage(JettyAnnotatedEventDriver.java:143)
	at org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.messageComplete(SimpleBinaryMessage.java:75)
	at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:67)
	at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryFrame(JettyAnnotatedEventDriver.java:130)
	at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:147)
	at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
	at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
	at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
	at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/io/file/attribute/FileTimes
	at org.apache.commons.compress.archivers.tar.TarArchiveEntry.getModTime(TarArchiveEntry.java:986)
	at io.onedev.commons.utils.TarUtils.untar(TarUtils.java:177)
	at io.onedev.agent.AgentSocket.onMessage(AgentSocket.java:110)
	at jdk.internal.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:70)
	... 22 common frames omitted
narandill changed title 2 months ago
Previous Value Current Value 
Agents offline after upgrade to 10.1.6
 
Agents fail to update after server upgrade to 10.1.6
Robin Shen commented 2 months ago

Please update agent to get this around. If agent is running in docker, just pull latest, otherwise, redownload the agent package from server.
narandill commented 2 months ago

Thank you @robin !
narandill changed state to 'Closed' 2 months ago
Previous Value Current Value 
Open
 
Closed
narandill commented 2 months ago

Resolved by first reinstalling shell executors on host machines, next docker executors.
issue 1 of 1
Type
Bug
Priority
Critical
Assignee
Robin Shen
Affected Versions
10.1.6
Labels
No labels
Issue Votes (0)
Login to vote
Watchers (4)
Reference
onedev/server#1769
Please wait...
Page is in error, reload to recover