Git/ssh failures (OD-1762)

I have onedev's git setup to work with SSH. We have an external tool (GoCD) which polls the repository for each project to detect changes and kick off deployments. It is working but i am encountering an issue.

Periodically all git interactions through ssh begin to fail. It is only temporary and seems to resolve itself given enough time.

When we experience these issues we get the following error in the logs.

2024-02-21 08:09:40,244 ERROR [pool-3-thread-2427] io.onedev.server.git.SshCommand Error executing git command
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.sshd.common.channel.exception.SshChannelClosedException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[git@/172.21.9.81:58368]] SSH_MSG_CHANNEL_DATA) len=35 - channel already closed
	at io.onedev.commons.utils.command.StreamPumper.waitFor(StreamPumper.java:70)
	at io.onedev.commons.utils.command.PumpInputToOutput.waitFor(PumpInputToOutput.java:26)
	at io.onedev.commons.utils.command.Commandline.execute(Commandline.java:419)
	at io.onedev.commons.utils.command.Commandline.execute(Commandline.java:270)
	at io.onedev.server.git.command.UploadPackCommand.run(UploadPackCommand.java:65)
	at io.onedev.server.git.SshCommand$1.run(SshCommand.java:149)
	at io.onedev.server.util.concurrent.DefaultWorkExecutor$1.call(DefaultWorkExecutor.java:199)
	at io.onedev.server.util.concurrent.DefaultWorkExecutor$1.call(DefaultWorkExecutor.java:195)
	at io.onedev.server.security.SecurityUtils$2.call(SecurityUtils.java:407)
	at io.onedev.server.util.concurrent.DefaultWorkExecutor.lambda$call$0(DefaultWorkExecutor.java:70)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at io.onedev.server.security.SecurityUtils.lambda$inheritSubject$1(SecurityUtils.java:375)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.sshd.common.channel.exception.SshChannelClosedException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[git@/172.21.9.81:58368]] SSH_MSG_CHANNEL_DATA) len=35 - channel already closed
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at io.onedev.commons.utils.command.StreamPumper.waitFor(StreamPumper.java:68)
	... 14 common frames omitted
Caused by: java.lang.RuntimeException: org.apache.sshd.common.channel.exception.SshChannelClosedException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[git@/172.21.9.81:58368]] SSH_MSG_CHANNEL_DATA) len=35 - channel already closed
	at io.onedev.commons.utils.command.StreamPumper$1.run(StreamPumper.java:40)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	... 5 common frames omitted
Caused by: org.apache.sshd.common.channel.exception.SshChannelClosedException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[git@/172.21.9.81:58368]] SSH_MSG_CHANNEL_DATA) len=35 - channel already closed
	at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:146)
	at io.onedev.server.util.OutputStreamWrapper.write(OutputStreamWrapper.java:26)
	at io.onedev.commons.utils.com
mand.StreamPumper$1.run(StreamPumper.java:35)
	... 6 common frames omitted

Activities

Dylan Whiteford changed fields 2 years ago
Name Previous Value Current Value
Affected Versions

10.1.3

10.1.3 10.0.0
Robin Shen commented 2 years ago

It will help a lot if you can investigate to find out reproducing steps.
Robin Shen changed state to 'Closed' 2 years ago
Previous Value Current Value
Open

Closed
Robin Shen commented 2 years ago

Unable to reproduce. Please reopen if there are more info.
Login to comment

Name	Previous Value	Current Value
Affected Versions	10.1.3	10.1.3 10.0.0

Previous Value	Current Value
Open	Closed

Type	Bug
Priority	Normal
Assignee	Robin Shen
Affected Versions	10.1.3 10.0.0
Labels	No labels

Issue Votes (0)

Watchers (3)

Reference

OD-1762