job secret [help wanted] (OD-519)
bufferUnderrun opened 4 years ago

Hi,

i have to connect to one of my server over ssh inside a job with docker executor.

I don't want my secret/public keys in the git repo. I don't want do make a custom image with my secret/public keys inside.

So, i just use tools from onedev like the job secret :

echo "@secret:idsec@" >> .ssh/id_rsa
echo "@secret:idpub@" >> .ssh/id_rsa.pub
ssh -o StrictHostKeyChecking=no debian@@server

That's not working. The issue came from the secret:idpub were the value is replace by "*****". From the manual : "value less than 5 characters will not be masked in build".

Questions :

  • why my secret idpub is not echo ?
  • how can i do to connect over ssh ?

Thanks for your help !!

  • Robin Shen commented 4 years ago

    Try this:

    mkdir /root/.ssh
    cat <<EOT>> /root/.ssh/id_rsa
    @secret:private-key@
    EOT
    chmod 400 /root/.ssh/id_rsa
    ssh -o "StrictHostKeyChecking no" debian@@server echo hello world
    
  • bufferUnderrun commented 4 years ago

    not working

    honestly i don't know what i'm doing wrong : the secret is always replaced by MASK "*****"

    the secret:idsec image_2.png

    the secret:idpub image_3.png

    The command image.png

    The report log image_5.png

    why the idsec is correctly echoed but not the idpub ?

  • Robin Shen commented 4 years ago

    It only masks the secret occurrences in log, and will not affect actual behavior.

    Why you are echoing the secret to console? The secret content should be written to file id_rsa as I demonstrated above

  • bufferUnderrun commented 4 years ago

    The exact command i passed before opening the issue was :

    echo "@secret:idsec@" >> .ssh/id_rsa
    echo "@secret:idpub@" >> .ssh/id_rsa.pub
    

    Because it was not working, i try to cat the result files to see if it was correctly filled. To my surprise, the id_rsa.pub was filled with "*****" wheras the id_rsa was filled with the correct private Key.

    Like i said in the previous post, only the secret:idsec is correctly handle by onedev and not secret:idpub which is not consistent.

    Whether the both failed or whether they succeed.

    So there is something wrong

  • Robin Shen commented 4 years ago

    I guess one secret is of multiple lines, while another is single line. OneDev only masks secrets in log line by line.

    Echo multiple lines into a file might be problematic. The correct way is to use

    cat <<EOT>> /root/.ssh/id_rsa
    @secret:private-key@
    EOT
    
  • bufferUnderrun commented 4 years ago

    The command Screenshot_20220107-022559.png

    Not working, if i cat the file it is filled with "*****" Screenshot_20220107-022517.png

  • Robin Shen commented 4 years ago

    You can not examine content of the file by catting to console, it will again be masked. in console. You may copy the file out and examine the content, or publish the file via publish artifacts step, and check its content.

    Also I guess the comma character around the @secret:idpub@ should be removed.

    Remember that all @...@ will be replaced literally by OneDev before passing to shell for execution.

  • bufferUnderrun commented 4 years ago

    You are absolutely right. When publishing theses files as artifacts, they are correct.

    But i still get ssh connection error like if have i have wrong keys. I still investigate...

  • Robin Shen commented 4 years ago

    I'd suggest to run the container from terminal directly, played with ssh keys, to make sure that it works. Then make the script into OneDev build spec.

    Also I can use the commands suggested previously to login to remote server.

  • Robin Shen commented 4 years ago

    Another thing important: make sure to change file mode of id_rsa to 400

  • bufferUnderrun commented 4 years ago

    when i test

    docker run --rm -it --mount type=bind,src=/home/debian/test,dst=/opt/ alpine
    

    and then inside, execthe same command

    mkdir /root/.ssh
    cat /opt/id_rsa >>/root/.ssh/id_rsa
    chmod 400 /root/.ssh/id_rsa
    ssh -o StrictHostKeyChecking=no debian@server
    

    everything works

    but not with onedev, very frustrating.

    Moreover, i think there will be TTY problem running ssh command inside docker and onedev.

    So, do you have a way to connect to server, copy file and run some commands ?

  • bufferUnderrun commented 4 years ago

    update

    when running verbose image_6.png

    it seems my key is invalid. from stackoverflow some other have the same issue rapporting endofline format.

    So let's try something : adding a extra empty line in the job secret

    image_7.png

    After saving, i show the job at new, the extra line has disappear image_8.png

    Do you rewrite the job secret value with trim() or not handle append new line ?

  • Robin Shen commented 4 years ago

    Yes, input will be trimmed for normalization purpose. It is surprising me the EOL is required for your server. I tested with Ubuntu server and the EOL does not matter.

  • bufferUnderrun commented 4 years ago

    input will be trimmed for normalization purpose

    ?

    and what about the line return when pasting in the textarea ? my client pc in on windows. i try to force \n and then pasting but do not work either

  • Robin Shen commented 4 years ago

    All white spaces including line ending will be deleted

  • Robin Shen commented 4 years ago

    Will turn this off in next patch release. For now, you may manually append a line ending to your private key file

  • bufferUnderrun commented 4 years ago

    what it works :

    • adding the .ssh/id_rsa in the git repo and use-it for the ssh
    • adding the private key to a file "foo" in the git repo and cat foo >> .ssh/id_rsa (so no problem with cat command)

    what it does not work :

    • paste the key to job secret and cat or echo the value to .ssh/id_rsa

    you said you test it, but in the same conditions ? onedev is running on container and job run alpine image. i'm pasting the job secret from firefox on windows.

  • bufferUnderrun commented 4 years ago

    and what about having custom file inject inside the running job container ?

    we have artifact which are the result file of build why no having the opposite, file insert in the build without being in the repo ?

  • Robin Shen commented 4 years ago

    Why cat or echo directly does not work:

    As I mentioned earlier, @...@ will be replaced literally, so if you write

    cat @secrets:id_sec@ >> /root/.ssh/id_rsa
    

    It will be seen by shell as below after @secrets:id_sec@ being replaced:

    cat firstline of private key
    second line of private key
    ...
    last line of private key >> /root/.ssh/id_rsa
    

    This is the reason I am using below to write multiline string to a file in the first place:

    cat <<EOT>> /root/.ssh/id_rsa
    @secret:private-key@
    EOT
    
  • bufferUnderrun commented 4 years ago

    yes, that's why i use the code you propose

    cat <<EOT>> /root/.ssh/id_rsa
    @secret:private-key@
    EOT
    

    but does not work

    i attach the private/pub keys i used you can try (obviously not production keys...).

  • Robin Shen commented 4 years ago

    What image are you using for the job (alpine does not have ssh client available)? And what OS of your server you are trying ssh into?

  • bufferUnderrun commented 4 years ago

    i'm using alpine and connect to debian 11.

    this job command works as the id_rsa come from file in the git repo image_10.png

    and not working when using the job secret with the cat EOT...

  • Robin Shen commented 4 years ago

    Turns out that multi-line input in a text area will be saved using "\r\n" as line ending (this is html standard). And Debian is strict on line ending while Ubuntu not. The workaround is install dos2unix to your container, and then call below statement to convert "\r\n" to "\n"

    dos2unix /root/.ssh/id_rsa
    
  • bufferUnderrun commented 4 years ago

    And Debian is strict on line ending

    😬 rohhhhhh

    dos2unix /root/.ssh/id_rsa
    

    such a journey, but this workaround is perfect !

    thanks you Robin for all the help 👍

    you can close the thread

  • Robin Shen changed state to 'Closed' 4 years ago
    Previous Value Current Value
    Open
    Closed
issue 1/1
Type
Question
Priority
Normal
Assignee
Issue Votes (0)
Watchers (2)
Reference
OD-519
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover