Submodule checkout in builds hangs OneDev (OD-648)

devcharly opened 4 years ago

Having a major problem with git submodule checkout in builds, that hangs OneDev.

My setup:

OneDev 6.3.21 on bare metal machine
Mac mini M1 macOS 12.2.1
Zulu Java 11.0.11 (aarch64)
git 2.32.0 (Apple Git-132)
PostgreSQL 14.2

Have a git repo with two submodules and created a new .onedev-buildspec.yml that simply checks out the repo including submodules using step type "Checkout Code" (depth is 1).

Running the build shows following and then whole OneDev is dead:

...
Running step "Checkout"...
Checking out code...
Retrieving submodules...

In the Terminal.app, where I started OneDev with bin/server.sh console, it outputs:

... Server is ready at http://10.1.1.100:6610
Username for 'http://10.1.1.100:6610': JVM process was stopped.  It will be killed if the ping timeout expires.

There are still some git processes (used ps -ef | grep git):

  504 72366 72253   0  5:24PM ttys002    0:00.00 /Applications/Xcode.app/Contents/Developer/usr/bin/git submodule update --init --recursive --force --quiet --depth=1
  504 72367 72366   0  5:24PM ttys002    0:00.01 /bin/sh /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-submodule update --init --recursive --force --quiet --depth=1
  504 72399 72367   0  5:24PM ttys002    0:00.00 /bin/sh /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-submodule update --init --recursive --force --quiet --depth=1
  504 72400 72367   0  5:24PM ttys002    0:00.00 /bin/sh /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-submodule update --init --recursive --force --quiet --depth=1
  504 72401 72399   0  5:24PM ttys002    0:00.00 git submodule--helper update-clone --quiet --depth --depth=1 --
  504 72402 72401   0  5:24PM ttys002    0:00.00 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git submodule--helper clone --quiet --path SubmoduleA --name SubmoduleA --url http://10.1.1.100:6610/SubmoduleA --depth=1
  504 72403 72402   0  5:24PM ttys002    0:00.00 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git clone --no-checkout --quiet --depth 1 --separate-git-dir /Users/onedev-user/onedev-install/temp/server/onedev-build13045679749900539450/workspace/.git/modules/SubmoduleA -- http://10.1.1.100:6610/SubmoduleA /Users/onedev-user/onedev-install/temp/server/onedev-build13045679749900539450/workspace/SubmoduleA
  504 72404 72403   0  5:24PM ttys002    0:00.00 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git remote-http origin http://10.1.1.100:6610/SubmoduleA
  504 72405 72404   0  5:24PM ttys002    0:00.00 /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git-remote-http origin http://10.1.1.100:6610/SubmoduleA

Pressing Ctrl+C two times in the terminal kills OneDev.

When I try to restart OneDev using bin/server.sh console, same message as above appears after some seconds and OneDev is dead. Seems that OneDev tries to continue/restart the build at startup because it deletes an existing <onedev-install>/temp/server/onedev-build<number> directory and creates a new one.

Killing OneDev again with Ctrl+C and restarting it sometimes brings it back to life. If not I have to restore a database backup...

On the client I'm using SSH to access git. The url of the submodule is specified as relative path in .gitmodules. E.g. url = ../mySubmodule. This works fine on the client. BTW all git repos of submodules are stored in OneDev projects on same machine. The two log files console.log and server.log do not contain any exception or error message. Only standard messages.

Any idea how to fix this?

Activities

devcharly commented 4 years ago
Regarding the message:

Username for 'http://10.1.1.100:6610': JVM process was stopped. It will be killed if the ping timeout expires.

I think that the part Username for 'http://10.1.1.100:6610': is an output of git and it now waits for the username as input.
Robin Shen changed state to 'Closed' 4 years ago
Previous Value Current Value
Open

Closed
Robin Shen commented 4 years ago

When clone submodules, the default credential will not work, you will need to create a build secret with value set to an access token with permission to access relevant projects, and use that build secret as clone credential in checkout step.

This tutorial has details on how to use custom clone credential: https://code.onedev.io/projects/162/blob/main/pages/push-in-job.md

As to OneDev process being killed upon git prompting for password, this should be a bug of JSW (service wrapper library used by OneDev), and I filed an bug as below: https://code.onedev.io/projects/160/issues/649

To avoid this, you will need to run OneDev in service mode: https://code.onedev.io/projects/162/blob/main/pages/run-as-system-service.md
wojtek referenced from other issue 2 years ago

Hanging git submodule checkout (OD-1975)

Closed
Login to comment

Previous Value	Current Value
Open	Closed

Type	Bug
Priority	Major
Assignee	Robin Shen
Affected Versions	Not Found

Issue Votes (0)

Watchers (3)

Reference

OD-648