Periodic syncing GitHub to 1dev repo problem (OD-348)

Artur opened 5 years ago

Hi,

I created a build to periodically pull GitHub code repo and merge with repository on 1dev. The build seems to be working OK, but accessing remote repository returns an error, and all the process is unsuccessful.

Here is my build code:

- name: Pull Code from Git
  steps:
  - !CheckoutStep
    name: checkout
    cloneCredential: !HttpCredential
      accessTokenSecret: 1dev_token
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: merge
    image: alpine/git:latest
    commands:
    - git config --global user.name "@secrets:1dev_user@"
    - git config --global user.email "@secrets:1dev_email@"
    - ''
    - '# Fetch and checkout master as OneDev by default clones '
    - '# code up to current commit'
    - git fetch origin master
    - git checkout master
    - ''
    - git remote add upstream https://@secrets:github_user@:@secrets:github_token@@@github.com/@secrets:github_repo@
    - git fetch upstream
    - git merge upstream/master
    - git push origin master
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  triggers:
  - !ScheduleTrigger
    cronExpression: 0 15 1 ? * *
    projects: -templates
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  cpuRequirement: 250m
  memoryRequirement: 128m
  timeout: 3600
properties:
- name: branch
  value: master

And when this build is run, I get the following in the log output:

17:00:13Discovering job executor...
17:00:13Discovered job executor type: Docker Executor
17:00:13Executing job (executor: auto-discovered, network: auto-discovered-tigase-convene-2-0)...
17:00:13Allocating job caches...
17:00:17Copying job dependencies...
17:00:17Running step "checkout"...
17:00:17Checking out code...
17:00:21Running step "merge"...
17:00:23From https://1dev.just-4.dev/tigase-convene
17:00:23 * branch            master     -> FETCH_HEAD
17:00:23 * [new branch]      master     -> origin/master
17:00:23Branch 'master' set up to track remote branch 'master' from 'origin'.
17:00:23Switched to a new branch 'master'
17:00:23https://*****:*****@github.com/*****
17:00:24fatal: unable to access 'https://github.com/*****/': The requested URL returned error: 400
17:00:24merge: upstream/master - not something we can merge
17:00:24Everything up-to-date
17:00:24Reporting job caches...
17:00:28Job finished

See, the error:

17:00:24fatal: unable to access 'https://github.com/*****/': The requested URL returned error: 400

Looks like the url is malformed or secrets are not correctly resolved or something. When I run all these commands from my terminal, replacing secrets with correct values all is successful.

Activities

Robin Shen commented 5 years ago

This happens as OneDev configures http.extraHeader to use custom token when checking out code from OneDev. As long as this setting is added, git will always use this even when connect to other urls. The workaround is to checkout other repository (GitHub in this case) via ssh instead of https
Artur commented 5 years ago

I can checkout GitHub via ssh but how to setup a build step to use private ssh key for connecting to GitHub via ssh?

Artur commented 5 years ago

I am experimenting with script like this, but I cannot get it work.

- name: Merge from GitHub
  steps:
  - !CommandStep
    name: merge
    image: alpine/git:latest
    commands:
    - git config --global user.name "@secrets:1dev_user@"
    - git config --global user.email "@secrets:1dev_email@"
    - ''
    - '# Fetch and checkout master as OneDev by default clones '
    - '# code up to current commit'
    - git fetch origin master
    - git checkout master
    - ''
    - mkdir ~/.ssh
    - echo "@secrets:github_ssh_key@" > ~/.ssh/id_rsa
    - chmod  400 ~/.ssh/id_rsa
    - cat ~/.ssh/id_rsa
    - (ssh-keygen -F github.com || ssh-keyscan github.com) >> ~/.ssh/known_hosts
    - ''
    - git remote add upstream git@@github.com:@secrets:github_user@/@secrets:github_repo@
    - git fetch upstream
    - git merge -m "github merge" upstream/master
    - git push origin master
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
properties:
- name: branch
  value: master

Despite setting up ssh for non-interactive work it still fails getting code from github. The last version gives me the following error:

17:21:09.  Load key "/root/.ssh/id_rsa": invalid format
17:21:09.  [email protected]: Permission denied (publickey).
17:21:09.  fatal: Could not read from remote repository.

The 'cat' command above displays content of the private key file which looks correct to me.

Any suggestion would be ery much appreciated.

Artur commented 5 years ago
I am also getting an error at the end which seems like another issue:

fatal: could not read Username for 'https://1dev.just-4.dev': No such device or address

https://1dev.just-4.dev is the address of my 1dev installation where the code repository is located.
Robin Shen commented 5 years ago

Your script works fine at my side. It is odd...
Robin Shen commented 5 years ago

Please make sure to use private key for secret "github_ssh_key"
Artur commented 5 years ago
Yes, I added correct ssh key to my secrets for the project. I found on the net info that missing public ssh key may cause such an error. So I added following lines to include public ssh key and some debug:

ssh -V echo "Setting up private ssh key" mkdir ~/.ssh echo -e "@secrets:github_ssh_key@" > ~/.ssh/id_rsa chmod 400 ~/.ssh/id_rsa echo "Private ssh key:" cat ~/.ssh/id_rsa echo "Setting up public ssh key" echo -e "@secrets:github_ssh_pub@" > ~/.ssh/id_rsa.pub echo "Public ssh key:" cat ~/.ssh/id_rsa.pub

Now, strange thing. cat ~/.ssh/id_rsa prints correct content of the private ssh key but cat ~/.ssh/id_rsa.pub prints only this: *****

Do you have any ideas or suggestions? Both, private and public key are added to project secrets in the same way.

Robin Shen commented 5 years ago

It is printing "*****" as the secret value will be masked out in log for security reason. At my side, I am able to checkout from GitHub without any problem without public key. Since I am using the same image as yours (alpine/git), the only reason might be the ssh key format?

Nevertheless, there is a much simpler approach without using ssh checkout. Just write the logic as below:

- name: Merge from GitHub
  steps:
  - !CommandStep
    name: merge
    image: alpine/git:latest
    commands:
    - git config --global user.name "@secrets:1dev_user@"
    - git config --global user.email "@secrets:1dev_email@"
    - ''
    - '# Fetch and checkout master as OneDev by default clones '
    - '# code up to current commit'
    - git fetch origin master
    - git checkout master
    - ''
    - cp ~/.gitconfig ~/.gitconfig.bak
    - git config --global --unset http.extraHeader
    - ''
    - git remote add upstream https://@secrets:github_access_token@@@github.com/@secrets:github_user@/@secrets:github_repo@
    - git fetch upstream
    - git merge -m "github merge" upstream/master
    - cp ~/.gitconfig.bak ~/.gitconfig
    - git push origin master
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
properties:
- name: branch
  value: master

It backs up the git config which contains the http.extraHeader setting, then unset it to make it working with http protocol of other repositories. Then restore it before pushing merged code back to OneDev

Artur commented 5 years ago

Thank you. With this I was able to move forward past the GitHub connection.

Unfortunately, now the last step:

git push origin master

is giving me headache. It just does not work, it is giving me various errors like:

fatal: could not read Username for 'https://1dev.just-4.dev': No such device or address

or after trying various username/password/token combinations it sometimes gives me:

fatal: Authentication failed for 'https://1dev.just-4.dev/tigase-convene/'

instead.

Here is my current script, with some debug code added to investigate the issue:

- name: Merge from GitHub
  steps:
  - !CommandStep
    name: merge
    image: alpine/git:latest
    commands:
    - export GIT_TRACE=true
    - ''
    - echo -e "\n git config"
    - git config credential.helper store --file=~/.git-credentials
    - echo "https://@secret:1dev_user@:@secret:1dev_token@@@@property:1dev_host@" >> ~/.git-credentials
    - echo "https://@secret:github_user@:@secret:github_token@@@github.com" >> ~/.git-credentials
    - ''
    - cat ~/.git-credentials
    - ''
    - git config --global user.email "@secrets:github_email@"
    - ''
    - '# Fetch and checkout master as OneDev by default clones '
    - '# code up to current commit'
    - echo -e "\ngit fetch origin master"
    - git fetch origin master
    - echo -e "\ngit checkout master"
    - git checkout master
    - ''
    - echo -e "\ncp -fv ~/.gitconfig ~/.gitconfig.bak"
    - cp -fv ~/.gitconfig ~/.gitconfig.bak
    - ''
    - echo -e "\ngit config --global --unset http.extraHeader"
    - git config --global --unset http.extraHeader
    - ''
    - echo -e "\ngit remote add upstream https://github.com/@secrets:github_repo@"
    - git remote add upstream https://github.com/@secrets:github_repo@
    - ''
    - echo -e "\ngit fetch upstream"
    - git fetch upstream
    - ''
    - echo -e "\ngit merge -m "github merge" upstream/master"
    - git merge -m "github merge" upstream/master
    - ''
    - echo -e "\ncp -fv ~/.gitconfig.bak ~/.gitconfig"
    - cp -fv ~/.gitconfig.bak ~/.gitconfig
    - ''
    - echo -e "\ngit push origin master"
    - git push origin master
    - ''
    - ping -c 3 1dev.just-4.dev
    - ''
    - cat ~/.gitconfig
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
properties:
- name: branch
  value: master
- name: 1dev_host
  value: 1dev.just-4.dev

and here is the log output:

14:34:31Discovering job executor...
14:34:31Discovered job executor type: Docker Executor
14:34:31Executing job (executor: auto-discovered, network: auto-discovered-tigase-convene-26-0)...
14:34:31Allocating job caches...
14:34:33Copying job dependencies...
14:34:33Running step "checkout"...
14:34:33Checking out code...
14:34:34Running step "merge -> merge (1)"...
14:34:36
14:34:36 git config
14:34:3621:34:36.054847 git.c:444               trace: built-in: git config credential.helper store '--file=~/.git-credentials'
14:34:36https://*****:*****@1dev.just-4.dev
14:34:36https://*****:*****@github.com
14:34:3621:34:36.067869 git.c:444               trace: built-in: git config --global user.email 1848738+*****@users.noreply.github.com
14:34:36
14:34:36git fetch origin master
14:34:3621:34:36.069047 git.c:444               trace: built-in: git fetch origin master
14:34:3621:34:36.069438 run-command.c:664       trace: run_command: GIT_DIR=.git git remote-https origin https://1dev.just-4.dev/tigase-convene
14:34:3621:34:36.069981 git.c:730               trace: exec: git-remote-https origin https://1dev.just-4.dev/tigase-convene
14:34:3621:34:36.069991 run-command.c:664       trace: run_command: git-remote-https origin https://1dev.just-4.dev/tigase-convene
14:34:3621:34:36.122014 run-command.c:664       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
14:34:3621:34:36.123436 run-command.c:664       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
14:34:3621:34:36.124074 git.c:444               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
14:34:36From https://1dev.just-4.dev/tigase-convene
14:34:36 * branch            master     -> FETCH_HEAD
14:34:36 * [new branch]      master     -> origin/master
14:34:3621:34:36.127632 run-command.c:1625      run_processes_parallel: preparing to run up to 1 tasks
14:34:3621:34:36.127648 run-command.c:1657      run_processes_parallel: done
14:34:3621:34:36.127676 run-command.c:664       trace: run_command: git maintenance run --auto --no-quiet
14:34:3621:34:36.128375 git.c:444               trace: built-in: git maintenance run --auto --no-quiet
14:34:36
14:34:36git checkout master
14:34:3621:34:36.129593 git.c:444               trace: built-in: git checkout master
14:34:36Branch 'master' set up to track remote branch 'master' from 'origin'.
14:34:36Switched to a new branch 'master'
14:34:36
14:34:36cp -fv ~/.gitconfig ~/.gitconfig.bak
14:34:36'/root/.gitconfig' -> '/root/.gitconfig.bak'
14:34:36
14:34:36git config --global --unset http.extraHeader
14:34:3621:34:36.292086 git.c:444               trace: built-in: git config --global --unset http.extraHeader
14:34:36
14:34:36git remote add upstream https://github.com/*****
14:34:3621:34:36.301790 git.c:444               trace: built-in: git remote add upstream https://github.com/*****
14:34:36
14:34:36git fetch upstream
14:34:3621:34:36.314356 git.c:444               trace: built-in: git fetch upstream
14:34:3621:34:36.318945 run-command.c:664       trace: run_command: GIT_DIR=.git git remote-https upstream https://github.com/*****
14:34:3621:34:36.322231 git.c:730               trace: exec: git-remote-https upstream https://github.com/*****
14:34:3621:34:36.322248 run-command.c:664       trace: run_command: git-remote-https upstream https://github.com/*****
14:34:3621:34:36.699364 run-command.c:664       trace: run_command: 'git credential-store get'
14:34:3621:34:36.704568 git.c:444               trace: built-in: git credential-store get
14:34:3621:34:36.941742 run-command.c:664       trace: run_command: 'git credential-store store'
14:34:3621:34:36.944436 git.c:444               trace: built-in: git credential-store store
14:34:3821:34:38.405239 run-command.c:664       trace: run_command: git index-pack --stdin --fix-thin '--keep=fetch-pack 22 on 2457ab7b39dd' --pack_header=2,131
14:34:3821:34:38.406063 git.c:444               trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 22 on 2457ab7b39dd' --pack_header=2,131
14:34:4521:34:45.029978 run-command.c:664       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
14:34:4521:34:45.030711 git.c:444               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
14:34:45From https://github.com/*****
14:34:45 * [new branch]      development -> upstream/development
14:34:45 * [new branch]      master      -> upstream/master
14:34:45 * [new branch]      voip        -> upstream/voip
14:34:4521:34:45.034637 run-command.c:1625      run_processes_parallel: preparing to run up to 1 tasks
14:34:4521:34:45.034651 run-command.c:1657      run_processes_parallel: done
14:34:4521:34:45.034687 run-command.c:664       trace: run_command: git maintenance run --auto --no-quiet
14:34:4521:34:45.035407 git.c:444               trace: built-in: git maintenance run --auto --no-quiet
14:34:45
14:34:45git merge -m github merge upstream/master
14:34:4521:34:45.036533 git.c:444               trace: built-in: git merge -m 'github merge' upstream/master
14:34:45Merge made by the 'recursive' strategy.
14:34:4521:34:45.041325 run-command.c:664       trace: run_command: git maintenance run --auto --no-quiet
14:34:4521:34:45.042068 git.c:444               trace: built-in: git maintenance run --auto --no-quiet
14:34:45 README.md | 4 +---
14:34:45 1 file changed, 1 insertion(+), 3 deletions(-)
14:34:45
14:34:45cp -fv ~/.gitconfig.bak ~/.gitconfig
14:34:45'/root/.gitconfig.bak' -> '/root/.gitconfig'
14:34:45
14:34:45git push origin master
14:34:4521:34:45.044051 git.c:444               trace: built-in: git push origin master
14:34:4521:34:45.044528 run-command.c:664       trace: run_command: GIT_DIR=.git git remote-https origin https://1dev.just-4.dev/tigase-convene
14:34:4521:34:45.045853 git.c:730               trace: exec: git-remote-https origin https://1dev.just-4.dev/tigase-convene
14:34:4521:34:45.045869 run-command.c:664       trace: run_command: git-remote-https origin https://1dev.just-4.dev/tigase-convene
14:34:4521:34:45.068266 run-command.c:664       trace: run_command: 'git credential-store get'
14:34:4521:34:45.069982 git.c:444               trace: built-in: git credential-store get
14:34:4521:34:45.073356 run-command.c:664       trace: run_command: 'git credential-store erase'
14:34:4521:34:45.074240 git.c:444               trace: built-in: git credential-store erase
14:34:45fatal: Authentication failed for 'https://1dev.just-4.dev/tigase-convene/'
14:34:45PING 1dev.just-4.dev (70.95.146.126): 56 data bytes
14:34:4564 bytes from 70.95.146.126: seq=0 ttl=63 time=0.199 ms
14:34:4664 bytes from 70.95.146.126: seq=1 ttl=63 time=0.257 ms
14:34:4764 bytes from 70.95.146.126: seq=2 ttl=63 time=0.375 ms
14:34:47
14:34:47--- 1dev.just-4.dev ping statistics ---
14:34:473 packets transmitted, 3 packets received, 0% packet loss
14:34:47round-trip min/avg/max = 0.199/0.277/0.375 ms
14:34:47[http]
14:34:47	extraHeader = Authorization: Bearer *****
14:34:47[user]
14:34:47	email = 1848738+*****@users.noreply.github.com
14:34:47Reporting job caches...
14:34:50Job finished

I would appreciate any help as I am stuck now and I have no idea what can be wrong.

Artur commented 5 years ago
Finally got it working! I got an idea while re-reading my last comment and looking at the build log. Content of the /root/.gitconfig shows:

[http] extraHeader = Authorization: Bearer *****

And your documentation says that, if this is set, git credentials set in other way do not work. However, after connecting to GitHub, this Bearer based authorization also stopped working as git could not resolve Username for 1dev system. My provided credentials for 1dev did not work either because of the Bearer authorization in config file. So I just removed this part and decided to use token based authorization for 1dev as well.

Here is the final working script:

git config --global user.email "@secrets:github_email@" git config credential.helper store --file=~/.git-credentials echo "https://@secret:1dev_user@:@secret:1dev_token@@@@property:1dev_host@" >> ~/.git-credentials echo "https://@secret:github_user@:@secret:github_token@@@github.com" >> ~/.git-credentials # Fetch and checkout master as OneDev by default clones # code up to current commit git fetch origin master git checkout master git config --global --unset http.extraHeader git remote add upstream https://github.com/@secrets:github_repo@ git fetch upstream git merge -m "github merge" upstream/master git push origin master

It is working but if you have any suggestions on how to simplify it and get it working without setting 1dev credentials, that would be helpful as well.
Artur commented 5 years ago

Digging further I found information in 1dev that the Bearer credentials have read-only permissions... so this explains the issue completely.
Robin Shen commented 5 years ago

Updated to latest version of alpine/git and I have the same issue. Looks like http.extraHeader no longer working as expected even if no GitHub repository checkout is involved. Will look into this further.
Artur commented 5 years ago
From my understanding it is working as expected:

If Bearer authentication is active (extraHeader) no other authentication is working - this is expected and documented on your website

Bearer authentication - default credentials have read-only permissions, so pushing back to 1dev repo is not possible - this is also expected and documented
Robin Shen changed state to 'Closed' 5 years ago
Previous Value Current Value
Open

Closed
Robin Shen commented 5 years ago

Just have time to check this further. For OneDev authentication, I plan to use a custom header instead of using traditional "Authorization" header. This way, it will not interfere with other repository authentications.

This way, one no longer needs to do extra things such as unset http.exraHeader and backup/restore .gitconfig. Issue #354 has been created to track the improvement, and I am closing this one.
Login to comment

Previous Value	Current Value
Open	Closed

Type	Question
Priority	Normal
Assignee	Robin Shen

Issue Votes (0)

Watchers (3)

Reference

OD-348