OneDev
Dashboards
Projects
Global Views
Code Search
server
Overview
Code
Pull Requests
Issues
List
Boards
Iterations
Builds
Packages
Statistics
OneDev 16.5.0
Documentation Support & Bug Report RESTful API Incompatibilities License Agreement Check Update
Projects onedev server onedev/server Issues OD-2972
Job Executor: Inherit onedev-trustcerts for TLS in Kubernetes #2972
christoffers opened 6 days ago

Summary
Fix SSL certificate trust for OneDev job executor in Kubernetes

Motivation
When running jobs with internal DNS names behind TLS, the executor fails with: javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: (certificate_unknown) The certificate chain is not trusted.

The OneDev StatefulSet has onedev-trustcerts mounted, but the executor job does not inherit these certificates when a job is created, causing SSL handshake failures.

Scope

  • Ensure executor jobs inherit the onedev-trustcerts volume or equivalent truststore.
  • Validate that the executor JVM trusts the same certificates as the main OneDev instance.
  • Apply minimal changes to the job executor configuration (e.g., Kubernetes manifests, JVM flags).

Out of Scope

  • Modifying the main OneDev StatefulSet configuration.
  • Disabling SSL verification globally (e.g., -Djavax.net.ssl.trustAll=true).
  • Server-side certificate generation or management.

Constraints

  • Avoid introducing new dependencies or significant overhead.
  • Reuse existing certificate mounting mechanisms (e.g., onedev-trustcerts).
  • Ensure backward compatibility with existing job configurations.
  • Do not use insecure workarounds like disabling SSL verification.
Copy
  • Robin Shen commented 5 days ago

    Is this happening when job executor talks to OneDev server (like downloading dependencies / report job status etc)? If so, the job executor should already trust the cert. If this happens for custom logic of the job talking to your own server, I am afraid there is not a general approach for trusting the cert, as it depends on specific tools or languages.

    Copy
  • christoffers commented 5 days ago

    Looks like it is a callback from the executor pod:

    

15:00:46 Checking cluster access...
15:00:46 Preparing job (executor: runner, namespace: runner-2-1-2)...
15:00:47 Running job on node <NODE>...
15:00:47 Starting job containers...
15:01:09 Retrieving job data from https://<internal_url>...
15:01:10 javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: (certificate_unknown) The certificate chain is not trusted
15:01:10     	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:270)
15:01:10     	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
15:01:10     	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
15:01:10     	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
15:01:10     	at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
15:01:10     	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413)
15:01:10     	at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313)
15:01:10     	at io.onedev.k8shelper.JobHelper.init(JobHelper.java:188)
15:01:10     	at io.onedev.k8shelper.JobInit.main(JobInit.java:17)
15:01:10     Caused by: javax.net.ssl.SSLHandshakeException: (certificate_unknown) The certificate chain is not trusted
15:01:10     	at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
15:01:10     	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
15:01:10     	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
15:01:10     	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
15:01:10     	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
15:01:10     	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
15:01:10     	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
15:01:10     	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
15:01:10     	at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:380)
15:01:10     	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:268)
15:01:10     	... 14 more
15:01:10     Caused by: java.security.cert.CertificateException: The certificate chain is not trusted
15:01:10     	at nl.altindag.ssl.trustmanager.CombinableX509TrustManager.checkTrusted(CombinableX509TrustManager.java:61)
15:01:10     	at nl.altindag.ssl.trustmanager.AggregatedX509ExtendedTrustManager.checkServerTrusted(AggregatedX509ExtendedTrustManager.java:86)
15:01:10     	... 34 more
15:01:10     	Suppressed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
15:01:10     		at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
15:01:10     		at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
15:01:10     		at java.base/sun.security.validator.Validator.validate(Unknown Source)
15:01:10     		at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
15:01:10     		at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
15:01:10     		at nl.altindag.ssl.trustmanager.DelegatingX509ExtendedTrustManager.checkServerTrusted(DelegatingX509ExtendedTrustManager.java:48)
15:01:10     		at nl.altindag.ssl.trustmanager.AggregatedX509ExtendedTrustManager.lambda$checkServerTrusted$4(AggregatedX509ExtendedTrustManager.java:86)
15:01:10     		at nl.altindag.ssl.trustmanager.CombinableX509TrustManager.checkTrusted(CombinableX509TrustManager.java:41)
15:01:10     		... 35 more
15:01:10     	Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
15:01:10     		at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
15:01:10     		at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
15:01:10     		at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
15:01:10     		... 43 more
15:01:10     	Suppressed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
15:01:10     		at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
15:01:10     		at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
15:01:10     		at java.base/sun.security.validator.Validator.validate(Unknown Source)
15:01:10     		at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
15:01:10     		at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
15:01:10     		at nl.altindag.ssl.trustmanager.AggregatedX509ExtendedTrustManager.lambda$checkServerTrusted$4(AggregatedX509ExtendedTrustManager.java:86)
15:01:10     		at nl.altindag.ssl.trustmanager.CombinableX509TrustManager.checkTrusted(CombinableX509TrustManager.java:41)
15:01:10     		... 35 more
15:01:10     	Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
15:01:10     		at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
15:01:10     		at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
15:01:10     		at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
15:01:10     		... 42 more
15:01:10     
15:01:25 Container 'init' exited with code 1
    Copy
  • Robin Shen commented 5 days ago

    Does the directory specified as param --from-file of the command creating secret onedev-trustcerts contain all certificates in the trust chain? I tested self-signed certificate with a local k8s and it works fine.

    Copy
  • christoffers commented 5 days ago

    Yes, this was a human error. The wrong TLS was added to the production pipeline :). Everything is working now.

    Closing issue as this was an error from our side.

    Copy
  • christoffers changed state to 'Closed' 5 days ago
    Previous Value Current Value 
    Open
    		 
    Closed
1/1
Type
Improvement
Priority
Normal
Assignee
Robin Shen
Labels
No labels
Issue Votes (1)
Login to vote
Watchers (3)
Reference
OD-2972
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover