How do I get the Onedev Kubernetes Executor working in a microk8s helm deployment? (OD-1768)
Joshua Norman opened 2 years ago

I am quite new in the Linux side of things with setting up servers etc. I have set up a Kubernetes cluster in microk8s and have deployed Onedev in the cluster. I don't have an ingress or any certificates, and am accessing Onedev via an exposed service by IP.

The problem I'm encountering is when setting up a Kubernetes executor in Onedev to run jobs, I am faced with the following:

First it tries to start the job container:

image.png

and then it fails with the following error:

init: failed to pull and unpack image "docker.io/1dev/k8s-helper-linux:2.12.1": failed to resolve reference "docker.io/1dev/k8s-helper-linux:2.12.1": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://auth.docker.io/token?scope=repository%3A1dev%2Fk8s-helper-linux%3Apull&service=registry.docker.io: 401 Unauthorized

What I've tried so far:

  1. Removing the port in Onedev Administration System Settings.
  2. Changing the same port to the Kubernetes cluster IP.
  3. Installing kubectl and getting it to work without prefixing "microk8s".
  4. Adding a cluster role and using that in the kubernetes executor setup.

ps. It can't be my dockerhub credentials as those are working just fine with a docker executor in Onedev.

I've tried searching if anyone's encountered the same problem, but to no avail. Your help would be greatly appreciated.

  • Robin Shen commented 2 years ago

    Seems that dockerhub now requires login even for public images. Please workaround this by creating a Kubernetes executor in Administration / Job Executors and specify your login to DockerHub.

    PS: I created a k8s cluster on google cloud, and it does not require login. Not sure how gke handles dockerhub login requirement...

  • Joshua Norman commented 2 years ago

    I have done this, but still getting the same error: image_2.png

  • Robin Shen commented 2 years ago

    This is quite odd. Please login to the k8s node, and pull image 1dev/k8s-helper-linux:2.12.7 (assuming you are running latest OneDev version) manually to see what happens.

  • Robin Shen commented 2 years ago

    Sorry ought to be 1dev/k8s-helper-linux:2.12.1 (no need to run latest OneDev version).

  • Joshua Norman commented 2 years ago

    I've pulled the image from the node, and it seems to work just fine (I'm using the same dockerhub credentials on this server) image_3.png

  • Robin Shen commented 2 years ago

    Believe this is an issue with dockerhub. Sometimes I get unauthorized error when pull public image from terminal. Maybe it is due to their rate limit control.

  • Robin Shen commented 2 years ago

    Will host the helper image to some other places in future versions. For now, if you have some places to host the helper image, you may push it there, and map the helper image to your location in k8s executor image mapping setting.

  • Robin Shen changed state to 'Closed' 2 years ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen commented 2 years ago

    k8s-helper image now hosts on code.onedev.io in version 10.2.0. I am closing this issue.

  • Joshua Norman commented 2 years ago

    Hi Robin, thanks so much for the assistance. However, I am still getting the same problem after updating to the latest onedev version: image_4.png

  • Joshua Norman changed state to 'Open' 2 years ago
    Previous Value Current Value
    Closed
    Open
  • Robin Shen commented 2 years ago

    Seems that your cluster has issues downloading images from docker. You may test with below steps:

    1. Create a test namespace by running kubectl create namespace test
    2. Download test.yaml
    3. Run command kubectl apply -f test.yaml
    4. Run command kubectl logs test -n test to see if it works
  • Joshua Norman commented 2 years ago

    That seemed to work:

    image_5.png

  • Robin Shen commented 2 years ago

    Could it be possible that the registry login credentials are specified incorrectly in k8s executor? You may remove registry logins from the executor and test again.

  • Joshua Norman commented 2 years ago

    Wow that worked! I completely removed the registry login and it works perfectly, thanks.

    Could it be that the registry logins are not being passed correctly? Because I used the exact same details in a docker executor and it worked.

  • Robin Shen commented 2 years ago

    Please create a file config.json with below content:

    {
      "auths" : {
        "https://index.docker.io/v1/" : {
          "auth" : "<base64 encoding of docker credential>"
        }
      }
    }
    

    Here <base64 encoding of docker credential> should be replaced with output of below command:

    echo -n "docker-user-name:docker-password" | base64
    

    Then run below command to get base64 of config.json:

    cat config.json | base64
    

    Then create a file secret.yaml with below content and replace <base64 of config.json> with above output:

    apiVersion: v1
    kind: Secret
    metadata: 
      name: image-pull-secret
      namespace: test
    data:
      .dockerconfigjson: <base64 of config.json>
    type: kubernetes.io/dockerconfigjson
    

    Run command kubectl apply -f secret.yaml to create the secret into test namespace

    Then modify test.yaml with below content, replacing <some private image> with a real image at dockerhub:

    apiVersion: v1
    kind: Pod
    metadata: 
      name: test
      namespace: test
    spec:
      containers:
      - args: ["-c", "echo it works"]
        image: <some private image>
        name: test
        command: [sh]
      restartPolicy: Never
      imagePullSecrets:
      - name: image-pull-secret
    

    Now run kubectl apply -f test.yaml again and check the log to see if it works.

  • Joshua Norman commented 2 years ago

    I get this error:

    root@Milan7763-15103:/home/josh/onedev# kubectl logs test -f -n test
    Error from server (BadRequest): container "test" in pod "test" is waiting to start: trying and failing to pull image
    

    And the pod is failing to start: image_6.png

  • Joshua Norman commented 2 years ago

    I get this error:

    root@Milan7763-15103:/home/josh/onedev# kubectl logs test -f -n test
    Error from server (BadRequest): container "test" in pod "test" is waiting to start: trying and failing to pull image
    

    And the pod is failing to start: image_6.png

    This was with the public postgres image

  • Robin Shen commented 2 years ago

    This is a very simple test not related to OneDev. I guess something wrong with your k8s setup. Sorry that I am not a k8s expert. But you at least need to make this simple test working.

  • Robin Shen commented 2 years ago

    kubectl get events -n test may give you detail info of why image pull are failing

  • Joshua Norman commented 2 years ago

    My main problem was just not being able to run a job, so now that I can I'm happy. I will investigate the problem with my k8s cluster.

    Thanks @robin

  • Joshua Norman changed state to 'Closed' 2 years ago
    Previous Value Current Value
    Open
    Closed
issue 1/1
Type
Question
Priority
Minor
Assignee
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
OD-1768
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover