#1768  How do I get the Onedev Kubernetes Executor working in a microk8s helm deployment?
Closed
Joshua Norman opened 2 months ago

I am quite new in the Linux side of things with setting up servers etc. I have set up a Kubernetes cluster in microk8s and have deployed Onedev in the cluster. I don't have an ingress or any certificates, and am accessing Onedev via an exposed service by IP.

The problem I'm encountering is when setting up a Kubernetes executor in Onedev to run jobs, I am faced with the following:

First it tries to start the job container:

image.png

and then it fails with the following error:

init: failed to pull and unpack image "docker.io/1dev/k8s-helper-linux:2.12.1": failed to resolve reference "docker.io/1dev/k8s-helper-linux:2.12.1": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://auth.docker.io/token?scope=repository%3A1dev%2Fk8s-helper-linux%3Apull&service=registry.docker.io: 401 Unauthorized

What I've tried so far:

  1. Removing the port in Onedev Administration System Settings.
  2. Changing the same port to the Kubernetes cluster IP.
  3. Installing kubectl and getting it to work without prefixing "microk8s".
  4. Adding a cluster role and using that in the kubernetes executor setup.

ps. It can't be my dockerhub credentials as those are working just fine with a docker executor in Onedev.

I've tried searching if anyone's encountered the same problem, but to no avail. Your help would be greatly appreciated.

Robin Shen commented 2 months ago

Seems that dockerhub now requires login even for public images. Please workaround this by creating a Kubernetes executor in Administration / Job Executors and specify your login to DockerHub.

PS: I created a k8s cluster on google cloud, and it does not require login. Not sure how gke handles dockerhub login requirement...

Joshua Norman commented 2 months ago

I have done this, but still getting the same error: image_2.png

Robin Shen commented 2 months ago

This is quite odd. Please login to the k8s node, and pull image 1dev/k8s-helper-linux:2.12.7 (assuming you are running latest OneDev version) manually to see what happens.

Robin Shen commented 2 months ago

Sorry ought to be 1dev/k8s-helper-linux:2.12.1 (no need to run latest OneDev version).

Joshua Norman commented 2 months ago

I've pulled the image from the node, and it seems to work just fine (I'm using the same dockerhub credentials on this server) image_3.png

Robin Shen commented 2 months ago

Believe this is an issue with dockerhub. Sometimes I get unauthorized error when pull public image from terminal. Maybe it is due to their rate limit control.

Robin Shen commented 2 months ago

Will host the helper image to some other places in future versions. For now, if you have some places to host the helper image, you may push it there, and map the helper image to your location in k8s executor image mapping setting.

Robin Shen changed state to 'Closed' 2 months ago
Previous Value Current Value
Open
Closed
Robin Shen commented 2 months ago

k8s-helper image now hosts on code.onedev.io in version 10.2.0. I am closing this issue.

Joshua Norman commented 2 months ago

Hi Robin, thanks so much for the assistance. However, I am still getting the same problem after updating to the latest onedev version: image_4.png

Joshua Norman changed state to 'Open' 2 months ago
Previous Value Current Value
Closed
Open
Robin Shen commented 2 months ago

Seems that your cluster has issues downloading images from docker. You may test with below steps:

  1. Create a test namespace by running kubectl create namespace test
  2. Download test.yaml
  3. Run command kubectl apply -f test.yaml
  4. Run command kubectl logs test -n test to see if it works
Joshua Norman commented 2 months ago

That seemed to work:

image_5.png

Robin Shen commented 2 months ago

Could it be possible that the registry login credentials are specified incorrectly in k8s executor? You may remove registry logins from the executor and test again.

Joshua Norman commented 2 months ago

Wow that worked! I completely removed the registry login and it works perfectly, thanks.

Could it be that the registry logins are not being passed correctly? Because I used the exact same details in a docker executor and it worked.

Robin Shen commented 2 months ago

Please create a file config.json with below content:

{
  "auths" : {
    "https://index.docker.io/v1/" : {
      "auth" : "<base64 encoding of docker credential>"
    }
  }
}

Here <base64 encoding of docker credential> should be replaced with output of below command:

echo -n "docker-user-name:docker-password" | base64

Then run below command to get base64 of config.json:

cat config.json | base64

Then create a file secret.yaml with below content and replace <base64 of config.json> with above output:

apiVersion: v1
kind: Secret
metadata: 
  name: image-pull-secret
  namespace: test
data:
  .dockerconfigjson: <base64 of config.json>
type: kubernetes.io/dockerconfigjson

Run command kubectl apply -f secret.yaml to create the secret into test namespace

Then modify test.yaml with below content, replacing <some private image> with a real image at dockerhub:

apiVersion: v1
kind: Pod
metadata: 
  name: test
  namespace: test
spec:
  containers:
  - args: ["-c", "echo it works"]
    image: <some private image>
    name: test
    command: [sh]
  restartPolicy: Never
  imagePullSecrets:
  - name: image-pull-secret

Now run kubectl apply -f test.yaml again and check the log to see if it works.

Joshua Norman commented 2 months ago

I get this error:

root@Milan7763-15103:/home/josh/onedev# kubectl logs test -f -n test
Error from server (BadRequest): container "test" in pod "test" is waiting to start: trying and failing to pull image

And the pod is failing to start: image_6.png

Joshua Norman commented 2 months ago

I get this error:

root@Milan7763-15103:/home/josh/onedev# kubectl logs test -f -n test
Error from server (BadRequest): container "test" in pod "test" is waiting to start: trying and failing to pull image

And the pod is failing to start: image_6.png

This was with the public postgres image

Robin Shen commented 2 months ago

This is a very simple test not related to OneDev. I guess something wrong with your k8s setup. Sorry that I am not a k8s expert. But you at least need to make this simple test working.

Robin Shen commented 2 months ago

kubectl get events -n test may give you detail info of why image pull are failing

Joshua Norman commented 2 months ago

My main problem was just not being able to run a job, so now that I can I'm happy. I will investigate the problem with my k8s cluster.

Thanks @robin

Joshua Norman changed state to 'Closed' 2 months ago
Previous Value Current Value
Open
Closed
issue 1 of 1
Type
Question
Priority
Minor
Assignee
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
onedev/server#1768
Please wait...
Page is in error, reload to recover