Steps tab reports checkout and cache steps as failed on the Kubernetes executor #3151
christoffers opened 3 days ago

Since upgrading to 16.8.0, every build shows the checkout step as failed in the new Steps tab, and the log ends the step with Step "checkout" is failed. The same happens to every SetupCacheStep. Command steps and kaniko steps are reported correctly. The job itself finishes SUCCESSFUL, and the steps that follow run although their condition is SUCCESSFUL, so the executor treats the checkout as successful. Only the reported step status is wrong.

Log excerpt (progress lines removed)

10:51:41 Job working directory initialized
10:51:43
10:51:54 From https://<server>/<project>
10:51:54  * branch            f1881adc99299e37d3cc0963f16f136d56907772 -> FETCH_HEAD
10:51:58
10:51:58 Step "checkout" is failed
10:51:58 Running step "dependency cache"...
10:52:01 Exact matched cache (key: gleam-server, checksum: b609a3a1673193bf67999698a60cdb46, path: .ci-cache/gleam-build)
10:52:02
10:52:02 Step "dependency cache" is failed
10:52:05 Running step "check"...
10:53:06 Step "check" is successful
10:53:26 Job finished

Steps in the build spec

- type: CheckoutStep
  name: checkout
  cloneCredential:
    type: DefaultCredential
  withLfs: false
  withSubmodules: false
  condition: SUCCESSFUL
  optional: false

- type: SetupCacheStep
  name: dependency cache
  key: gleam-server
  checksumFiles: packages/*/gleam.toml ...
  entries:
    - path: .ci-cache/gleam-build
  uploadStrategy: UPLOAD_IF_NOT_EXACT_MATCH

Suspected cause (from reading theonedev/k8s-helper at ca94abf; not confirmed)

JobHelper.generateCommandScript wraps each step in a shell script that runs the step, prints the ##onedev-step:SUCCESSFUL:step-N event, and then echoes LOG_END_MESSAGE (JobHelper.java, lines 206-219).

For a checkout, cache or server-side step, the wrapped command is a Java class, and each of them logs LOG_END_MESSAGE itself in a finally block before exiting:

  • CheckoutCode.java, line 37
  • SetupJobCache.java, line 30
  • RunServerSideStep.java, line 26

The server stops collecting the container's log at the first LOG_END_MESSAGE (KubernetesExecutor.java, collectContainerLog(..., new SeenMessage(LOG_END_MESSAGE), ...)), so the step event printed by the wrapper afterwards is never read, and a step with a start event but no outcome is shown as failed. A plain CommandStep prints nothing of the kind itself, which would explain why those are reported correctly.

If that is right, RunServerSideStep steps (publish artifacts, reports) are affected too.

1/1
Type
Bug
Priority
Normal
Assignee
Affected Versions
OneDev 16.8.0, Kubernetes executor, k8s-helper 4.3.7
Labels
No labels
Issue Votes (0)
Watchers (2)
Reference
OD-3151
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover