OneDev
Dashboards
Projects
Global Views
Code Search
server
Code
Pull Requests
Issues
List
Boards
Iterations
Builds
Packages
Statistics
Child Projects
Audit Log
OneDev 14.0.6
Documentation Support & Bug Report RESTful API Incompatibilities License Agreement Check Update
Projects onedev server Issues OD-2096
Repository pull sync results in 500 (OD-2096)
Simeon L. opened 1 year ago

Hey,

I am trying to setup a mirrored repository (pull only), but nearly every single build results in a 500. Not every build fails however, I have 2 Projects with a similar ci/cd config and both seem to only succeed once in 10 tries.

Attached you can find the build spec and the error

version: 35
jobs:
- name: Sync
  steps:
  - !PullRepository
    name: pull
    remoteUrl: https://github.com/FTBTeam/FTB-Chunks.git
    passwordSecret: Token
    refs: 1.20.1/main
    withLfs: false
    force: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  triggers:
  - !ScheduleTrigger
    cronExpression: 0 0 0 * * ?
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  timeout: 3600

java.lang.RuntimeException: Http request failed (status code: 500, error message: Command execution failed (command: git fetch https://*****@github.com/FTBTeam/FTB-Chunks.git 1.20.1/main:1.20.1/main, exit code: 1)) at io.onedev.k8shelper.KubernetesHelper.checkStatus(KubernetesHelper.java:342) at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:853) at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:812)

  • Robin Shen commented 1 year ago

    This seems like a github error instead of OneDev. Also it works now.

  • Simeon L. commented 1 year ago

    Unfortunately no, it doesn't.

    This is our Build Log from 26 min ago:

    01:59:41 java.lang.RuntimeException: Http request failed (status code: 500, error message: Command execution failed (command: git fetch https://*****@github.com/FTBTeam/FTB-Chunks.git 1.20.1/main:1.20.1/main, exit code: 1))
	at io.onedev.k8shelper.KubernetesHelper.checkStatus(KubernetesHelper.java:342)
	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:853)
	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:812)
	at io.onedev.agent.AgentSocket$4.doExecute(AgentSocket.java:805)
	at io.onedev.agent.AgentSocket$4.lambda$execute$1(AgentSocket.java:676)
	at io.onedev.agent.ExecutorUtils.runStep(ExecutorUtils.java:100)
	at io.onedev.agent.AgentSocket$4.execute(AgentSocket.java:673)
	at io.onedev.k8shelper.LeafFacade.execute(LeafFacade.java:12)
	at io.onedev.k8shelper.CompositeFacade.execute(CompositeFacade.java:35)
	at io.onedev.agent.AgentSocket.executeDockerJob(AgentSocket.java:586)
	at io.onedev.agent.AgentSocket.service(AgentSocket.java:981)
	at io.onedev.agent.AgentSocket.lambda$onMessage$2(AgentSocket.java:212)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

    Maybe for more details: We imported the project from github via URL (since its not our own project but rather a public one), we want to keep in sync with a specific branch since its for the version of the mod we use. Therefore I followed to docs to make a scheduled repository pul

  • Simeon L. commented 1 year ago

    Also here is an image from one of our build logs. as you can see all of em failed except one. They all have the exact same error message. The Scheduler is set to 00:00 Berlin time daily.

    image.png

  • Robin Shen commented 1 year ago

    Please create a project at code.onedev.io reproducing the issue. This will be a lot easier for me to find the problem.

  • Simeon L. commented 1 year ago

    Okay so I have created a project here and it works without issues. So now I am wondering whether or not there is an issue with my agent or its configuration. I only have one Job Executor and its a Remote Docker Executor. the agent selector is set to any agent.

  • Robin Shen commented 1 year ago

    Please check your agent log to see if there is more info of the error. Also run below command directly from your terminal on agent machine to see if it works:

    git fetch https://*****@github.com/FTBTeam/FTB-Chunks.git 1.20.1/main:1.20.1/main
  • Simeon L. commented 1 year ago

    The Agent.log doesn't even show the error I see in the job log. Also I can't run that command outside of a git repository, and I don't know where to find the repository on the agent machine

  • Robin Shen commented 1 year ago

    You may bare clone that repository on agent, then run that command inside the repository directory.

  • Simeon L. commented 1 year ago

    Jup, the fetch command works without issues both with and without an access token.

  • Robin Shen commented 1 year ago

    Please tell me detail steps to reproduce the issue (how you are installing OneDev server and agent, etc). Otherwise it will be difficult for me the find out the problem.

  • Simeon L. commented 1 year ago

    Okay so: Both the server and agent are installed on bare-metal. I have followed this guide: https://docs.onedev.io/installation-guide/run-on-bare-metal After that, I setup everything and added our first Agent. for that I went into the administration section and then "Agents" pressed the + and clicked on bare-metal. I downloaded the Zip, extracted it and ran agent.sh install to get the systemd configuration. Currently its running on the integrated HSQL All of this is running on: Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm

    then I made a Job Executor: Type: Remote Docker Executor Name: Pikachu-Containers (our Agent is called Pikachu) Privilege Settings: Default except "Always Pull Image" which is turned on Other Settings: Buildx Builder "onedev"

    the Server has the latest possible version (for that distro) installed: git version 2.39.5 This Java version is installed: openjdk version "21.0.4" 2024-07-16 LTS OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)

  • Robin Shen commented 1 year ago

    Did the same, but still works at my side. Not sure if this related to your environment. Are you able to reproduce this on a new EC2 box?

  • Simeon L. commented 1 year ago

    This isn’t hosted on aws but rather on a vps. There must be a way to figure out what’s going on.

  • Robin Shen commented 1 year ago

    This isn’t hosted on aws but rather on a vps. There must be a way to figure out what’s going on.

    Please reproduce the issue at your side, and check server log from "Adminstration / System Maintenance". It should contain more info of the error.

  • Simeon L. commented 1 year ago 
    2024-09-25 12:35:18,658 ERROR [qtp298088259-152] i.o.s.rest.JerseyExceptionMapper Error processing api request
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Cannot run program "git" (in directory "/opt/onedev/site/projects/34/git"): error=0, Failed to exec spawn helper: pid: 630905, exit value: 1
	at io.onedev.server.ee.clustering.DefaultClusterManager.getResult(DefaultClusterManager.java:291)
	at io.onedev.server.ee.clustering.DefaultClusterManager.runOnServer(DefaultClusterManager.java:297)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
	at jdk.proxy3/jdk.proxy3.$Proxy30.runOnServer(Unknown Source)
	at io.onedev.server.entitymanager.impl.DefaultProjectManager.runOnActiveServer(DefaultProjectManager.java:1197)
	at io.onedev.server.buildspec.step.PullRepository.lambda$run$0(PullRepository.java:158)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
	at io.onedev.server.buildspec.step.PullRepository.run(PullRepository.java:133)
	at io.onedev.server.job.DefaultJobManager.runServerStep(DefaultJobManager.java:1489)
	at io.onedev.server.plugin.executor.kubernetes.KubernetesResource.lambda$runServerStep$0(KubernetesResource.java:118)
	at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:55)
	at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:37)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at io.onedev.server.rest.DefaultServletContainer.service(DefaultServletContainer.java:28)
	at io.onedev.server.rest.DefaultServletContainer$$EnhancerByGuice$$192943560.GUICE$TRAMPOLINE(<generated>)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
	at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
	at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
	at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
	at io.onedev.server.rest.DefaultServletContainer$$EnhancerByGuice$$192943560.service(<generated>)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:454)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:363)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.security.CorsFilter.doFilter(CorsFilter.java:47)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.onedev.server.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Cannot run program "git" (in directory "/opt/onedev/site/projects/34/git"): error=0, Failed to exec spawn helper: pid: 630905, exit value: 1
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:278)
	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run program "git" (in directory "/opt/onedev/site/projects/34/git"): error=0, Failed to exec spawn helper: pid: 630905, exit value: 1
	at io.onedev.commons.utils.command.Commandline.execute(Commandline.java:282)
	at io.onedev.commons.utils.command.Commandline.execute(Commandline.java:244)
	at io.onedev.commons.utils.command.Commandline.execute(Commandline.java:232)
	at io.onedev.server.buildspec.step.PullRepository$PullTask.call(PullRepository.java:232)
	at io.onedev.server.buildspec.step.PullRepository$PullTask.call(PullRepository.java:167)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:276)
	... 6 common frames omitted
Caused by: java.io.IOException: Cannot run program "git" (in directory "/opt/onedev/site/projects/34/git"): error=0, Failed to exec spawn helper: pid: 630905, exit value: 1
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1170)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
	at io.onedev.commons.utils.command.Commandline.execute(Commandline.java:279)
	... 12 common frames omitted
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 630905, exit value: 1
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:295)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:225)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1126)
	... 14 common frames omitted
  • Simeon L. commented 1 year ago

    this is the only error that shows up

  • Simeon L. commented 1 year ago

    Is it maybe because the server is running with the "onedev" user and the agent is running as root?

  • Simeon L. commented 1 year ago

    I managed to fix the problem. It was caused by exactly that. Since the agent is running as root (so it can access the docker sock without me having to deal with rootless). I just had to run "git config --global --add safe.directory '*'" and it instantly started working.

  • Simeon L. commented 1 year ago

    Okay nevermind. That completely destroy the perms. of the git folder in the site/projects/34 folder. is there a way to tell the system "hey when you sync, use the same user as the server itself"

  • Robin Shen commented 1 year ago

    The repository sync operation actually happens on server. So it does not matter that agent is running as root while server is not.

    Seems that the problem is that the directory /opt/onedev/site/projects/34/git or some files under it is owned by root but OneDev server is running as non-root. I guess this might be because that you previously run OneDev server as root user, then switched to non-root user.

    Please run below command to fix permissions after changing RUN_AS user:

    chown -R <RUN_AS_USER> /opt/onedev

    After that, the pull should be working fine.

  • Simeon L. commented 1 year ago

    If the server is the one running the sync then why does an agent get assigned to the build ? And why does it use the remote docker executor as job executor for the sync?

    all the files are owned by the right user. And yet it wasnt working. Once i bypassed git‘s dubious ownership detection the agent was able to sync. However since the agent is running as root, once the build was done: some files that got updated during the sync are now owned by root rather rhen the onedev user.

  • Robin Shen commented 1 year ago

    Job executor coordinates different steps in the job, and repository sync step is deligated to server, while other steps may run on agent.

    My test here is that file ownership does not get changed even if server runs as non-root and agent process runs as root. All files are owned by the user running OneDev server process.

    So I suggested to reproduce the issue on a new EC2 instance (even if it is not the same as your current environment). If it can be reproduced, I can test with your EC2 image; otherwise, you might be aware of the environtal difference on your machine causing this.

    PS: My time is limited. So giving me more info will help me. For commercial users, please submit support request instead.

  • Robin Shen commented 1 year ago

    And why does it use the remote docker executor as job executor for the sync?

    OneDev will use the first eligible job executor if is is not specified when define the job. When determine eligibility of the job executor, only job requirement setting is checked.

  • Simeon L. commented 1 year ago

    As much as i would love to be able to provide an EC2 image or just test on EC2 we are a very small team with little to no budget to spare. I am more then happy to provide you with any information needed.

    if you want we can also move this conversation to slack or discord for easier communication. However please note: i live in Germany so its 03:33 for me right now. I can message you in a few hours once i wake up again.

    i know this is difficult with no information and we are not commercial customers. However we do really appreciate your help.

  • Robin Shen commented 1 year ago

    As much as i would love to be able to provide an EC2 image or just test on EC2 we are a very small team with little to no budget to spare. I am more then happy to provide you with any information needed.

    if you want we can also move this conversation to slack or discord for easier communication. However please note: i live in Germany so its 03:33 for me right now. I can message you in a few hours once i wake up again.

    i know this is difficult with no information and we are not commercial customers. However we do really appreciate your help.

    I understand it. Please have a good rest. I also hope to address any possible bug in OneDev to make it a better product.

    This is what I am testing on a Ubuntu box:

    1. Create an os user say "onedev"

    2. Download OneDev and extract to "/opt/onedev". Run command below to change ownership:

      sudo chown -R onedev /opt/onedev

    3. Edit "/opt/onedev/bin/server.sh" to specify RUN_AS_USER as "onedev"

    4. Run "sudo /opt/onedev/bin/server.sh install" to install the service

    5. Run "sudo service onedev start" to start OneDev, which will run as user "onedev"

    6. Set up OneDev and import from url "https://github.com/FTBTeam/FTB-Chunks.git"

    7. Now download agent from agents page, and extract it to "/opt/onedevagent".

    8. Run "sudo /opt/onedevagent/bin/agent.sh install" to install agent service, which will run as root

    9. Create a remote docker executor

    10. Open imported project "FTBTeam/FTB-Chunks.git", add a build spec with below content:

      version: 35
jobs:
- name: Sync
  steps:
  - !PullRepository
    name: pull
    remoteUrl: https://github.com/FTBTeam/FTB-Chunks.git
    passwordSecret: Token
    refs: 1.20.1/main
    withLfs: false
    force: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  triggers:
  - !ScheduleTrigger
    cronExpression: 0 0 0 * * ?
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  timeout: 3600

    11. Now run the job manually to sync repository.

    Are there any steps I missed? If not, can you do the same to try with a new OneDev instance on your machine to see if it works?

  • Simeon L. commented 1 year ago

    Hey, I will try and come back then.

  • Robin Shen changed state to 'Closed' 1 year ago
    Previous Value Current Value 
    Open
    		 
    Closed
  • Robin Shen commented 1 year ago

    Closing. Feel free to reopen if there is more info.

  • clay commented 8 months ago

    I am having this issue using the kubernetes executor

    00:42:53 Running step "Pull JuiceCloud"...
00:42:54 java.lang.RuntimeException: Http request failed (status code: 500, error message: Command execution failed (command: git fetch https://*****@github.com/clbx/JuiceCloud refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*, exit code: 1))
00:42:54     	at io.onedev.k8shelper.KubernetesHelper.checkStatus(KubernetesHelper.java:342)
00:42:54     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:853)
00:42:54     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:812)
00:42:54     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:798)
00:42:54     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:782)
00:42:54     	at io.onedev.k8shelper.RunServerSideStep.main(RunServerSideStep.java:19)
00:42:54     
00:42:54 Command exited with code 1
00:42:54 Step "Pull JuiceCloud" is failed
00:43:03 Job finished****

    More than happy to do some debug steps, I'm just getting started with OneDev and I'm not super familiar yet.

  • Robin Shen commented 8 months ago

    Please check server log in Administration / System Maintenance / Server Log to see if there are any errors printed.

  • clay commented 8 months ago

    Nothing in the server logs

  • Robin Shen commented 8 months ago

    Please attach an example project (git repository) to demonstrate the issue, as well as an example github repo that is syncing from.

  • clay commented 8 months ago

    I created a new repo in OneDev with only .onedev-buildspec.yml in it.

    version: 39
jobs:
- name: Pull from Github
  jobExecutor: kubernetes
  steps:
  - !PullRepository
    name: Pull from github
    remoteUrl: https://github.com/clbx/testrepo.git
    refs: refs/heads/* refs/tags/*
    withLfs: false
    force: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  timeout: 14400

    The test repo is here: https://github.com/clbx/testrepo

    This is the full job log

    20:43:26 Checking cluster access...
20:43:26 Preparing job (executor: kubernetes, namespace: kubernetes-5-1-1)...
20:43:26 Kubernetes: Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "init", "step-0", "sidecar" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "init", "step-0", "sidecar" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "init", "step-0", "sidecar" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "init", "step-0", "sidecar" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
20:43:26 Running job on node titan...
20:43:26 Starting job containers...
20:43:29 Retrieving job data from https://git.REDACTED.com...
20:43:29 Generating command scripts...
20:43:29 Downloading job dependencies from https://git.REDACTED.com...
20:43:29 Job workspace initialized
20:43:30 Running step "Pull from github"...
20:43:32 java.lang.RuntimeException: Http request failed (status code: 500, error message: Command execution failed (command: git fetch https://github.com/clbx/testrepo.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*, exit code: 1))
20:43:32     	at io.onedev.k8shelper.KubernetesHelper.checkStatus(KubernetesHelper.java:342)
20:43:32     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:853)
20:43:32     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:812)
20:43:32     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:798)
20:43:32     	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:782)
20:43:32     	at io.onedev.k8shelper.RunServerSideStep.main(RunServerSideStep.java:19)
20:43:32     
20:43:32 Command exited with code 1
20:43:32 Step "Pull from github" is failed
20:43:40 Job finished
  • Robin Shen commented 8 months ago

    Still works at my side. Please turn on debug logging by copying logback.xml to onedev container:

    kubectl cp ./logback.xml onedev/onedev-0:/opt/onedev/conf/logback.xml

    Then wait for 1 minute for the change to be picked up, and run the build again. The server log should contain all commands being executed with its output. NOTE that versions prior to 11.9.0 has a bug that the access token used in command line is not being masked in server log, please make sure to remove that before posting here.

  • clay commented 8 months ago

    Here's what it returned from my test repo

    2025-05-02 04:46:04,288 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl cluster-info
2025-05-02 04:46:04,376 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get namespaces --field-selector metadata.name=kubernetes-5-1-2 -o name --chunk-size=0
2025-05-02 04:46:04,501 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl create namespace kubernetes-5-1-2
2025-05-02 04:46:04,574 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl create -f /opt/onedev/temp/server/k8s10310414004877530501.yaml -o jsonpath={.metadata.name}
2025-05-02 04:46:04,718 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:04,790 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=4737, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=8e116b7d-afab-4c9d-b396-c4786e8d1a68}
2025-05-02 04:46:04,790 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=4737
2025-05-02 04:46:04,790 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=4737
2025-05-02 04:46:04,805 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:06,450 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=4765, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=f6e78e5d-ad58-4ecb-ac17-ebafc08d83a4}
2025-05-02 04:46:06,450 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=4765
2025-05-02 04:46:06,450 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=4765
2025-05-02 04:46:06,461 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl logs job -c init -n kubernetes-5-1-2 --follow --timestamps=true
2025-05-02 04:46:08,120 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:08,192 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=4825, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=7ac9d18a-b77d-48c8-966a-9a828472d915}
2025-05-02 04:46:08,192 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=4825
2025-05-02 04:46:08,192 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=4825
2025-05-02 04:46:08,203 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:08,463 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=4857, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=ccb15563-62da-4261-98a4-b070e9ceb2f6}
2025-05-02 04:46:08,464 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=4857
2025-05-02 04:46:08,464 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=4857
2025-05-02 04:46:08,474 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl logs job -c step-0 -n kubernetes-5-1-2 --follow --timestamps=true
2025-05-02 04:46:09,518 DEBUG [hz.127.0.0.1:5710.cached.thread-5] i.o.c.utils.command.Commandline Executing command: git fetch https://github.com/clbx/testrepo.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*
2025-05-02 04:46:10,756 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:10,833 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=4927, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=96a88fcf-a5ac-408e-a622-811c6b7d3d20}
2025-05-02 04:46:10,833 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=4927
2025-05-02 04:46:10,833 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=4927
2025-05-02 04:46:10,844 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:10,914 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=4956, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=74b5fab4-5798-43b5-8030-8bf2b759d690}
2025-05-02 04:46:10,915 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=4956
2025-05-02 04:46:10,915 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=4956
2025-05-02 04:46:10,925 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl logs job -c sidecar -n kubernetes-5-1-2 --follow --timestamps=true
2025-05-02 04:46:11,004 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl get pod job -n kubernetes-5-1-2 --watch -o json
2025-05-02 04:46:11,075 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree killAll: process=Process[pid=5013, exitValue="not exited"] and envs={ONEDEV_COMMAND_EXECUTION_UUID=0713ec23-bce8-41c5-ac50-b2221a06b582}
2025-05-02 04:46:11,075 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Recursively killing pid=5013
2025-05-02 04:46:11,075 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.ProcessTree Killing pid=5013
2025-05-02 04:46:11,085 DEBUG [hz.127.0.0.1:5710.cached.thread-7] i.o.c.utils.command.Commandline Executing command: kubectl delete namespace kubernetes-5-1-2
  • Robin Shen commented 8 months ago

    Please exec into onedev container to run below command to see if it works:

    cd /opt/onedev/site/projects/<project id>/git
git fetch https://github.com/clbx/testrepo.git "refs/heads/*:refs/heads/*" "refs/tags/*:refs/tags/*"
  • clay commented 8 months ago 
    root@onedev-0:/opt/onedev/site/projects/5/git# git fetch https://github.com/clbx/testrepo.git "refs/heads/*:refs/heads/*" "refs/tags/*:refs/tags/*"
From https://github.com/clbx/testrepo
 ! [rejected]        main       -> main  (non-fast-forward)
  • Robin Shen commented 8 months ago

    Thanks for the info. The error happens as the git repository at OneDev side has new commits compared to GitHub mirror, and a pull sync will override the new commits. OneDev by default rejects that unless the "force pull" option is specified in step setting.

    Normally in this case, you need to push new commits to GitHub side first. A repository push step also available to do this via a CI/CD job.

    OneDev will also be improved to capture error messages like this to make the sync failure easier to understand.

issue 1/1
Type
Bug
Priority
Normal
Assignee
Robin Shen
Affected Versions
11.1.3
Labels
No labels
Issue Votes (0)
Login to vote
Watchers (3)
Reference
OD-2096
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover