#519  job secret [help wanted]
Closed
bufferUnderrun opened 2 years ago

Hi,

i have to connect to one of my server over ssh inside a job with docker executor.

I don't want my secret/public keys in the git repo. I don't want do make a custom image with my secret/public keys inside.

So, i just use tools from onedev like the job secret :

echo "@secret:idsec@" >> .ssh/id_rsa
echo "@secret:idpub@" >> .ssh/id_rsa.pub
ssh -o StrictHostKeyChecking=no debian@@server

That's not working. The issue came from the secret:idpub were the value is replace by "*****". From the manual : "value less than 5 characters will not be masked in build".

Questions :

  • why my secret idpub is not echo ?
  • how can i do to connect over ssh ?

Thanks for your help !!

Robin Shen commented 2 years ago

Try this:

mkdir /root/.ssh
cat <<EOT>> /root/.ssh/id_rsa
@secret:private-key@
EOT
chmod 400 /root/.ssh/id_rsa
ssh -o "StrictHostKeyChecking no" debian@@server echo hello world
bufferUnderrun commented 2 years ago

not working

honestly i don't know what i'm doing wrong : the secret is always replaced by MASK "*****"

the secret:idsec image_2.png

the secret:idpub image_3.png

The command image.png

The report log image_5.png

why the idsec is correctly echoed but not the idpub ?

Robin Shen commented 2 years ago

It only masks the secret occurrences in log, and will not affect actual behavior.

Why you are echoing the secret to console? The secret content should be written to file id_rsa as I demonstrated above

bufferUnderrun commented 2 years ago

The exact command i passed before opening the issue was :

echo "@secret:idsec@" >> .ssh/id_rsa
echo "@secret:idpub@" >> .ssh/id_rsa.pub

Because it was not working, i try to cat the result files to see if it was correctly filled. To my surprise, the id_rsa.pub was filled with "*****" wheras the id_rsa was filled with the correct private Key.

Like i said in the previous post, only the secret:idsec is correctly handle by onedev and not secret:idpub which is not consistent.

Whether the both failed or whether they succeed.

So there is something wrong

Robin Shen commented 2 years ago

I guess one secret is of multiple lines, while another is single line. OneDev only masks secrets in log line by line.

Echo multiple lines into a file might be problematic. The correct way is to use

cat <<EOT>> /root/.ssh/id_rsa
@secret:private-key@
EOT
bufferUnderrun commented 2 years ago

The command Screenshot_20220107-022559.png

Not working, if i cat the file it is filled with "*****" Screenshot_20220107-022517.png

Robin Shen commented 2 years ago

You can not examine content of the file by catting to console, it will again be masked. in console. You may copy the file out and examine the content, or publish the file via publish artifacts step, and check its content.

Also I guess the comma character around the @secret:idpub@ should be removed.

Remember that all @...@ will be replaced literally by OneDev before passing to shell for execution.

bufferUnderrun commented 2 years ago

You are absolutely right. When publishing theses files as artifacts, they are correct.

But i still get ssh connection error like if have i have wrong keys. I still investigate...

Robin Shen commented 2 years ago

I'd suggest to run the container from terminal directly, played with ssh keys, to make sure that it works. Then make the script into OneDev build spec.

Also I can use the commands suggested previously to login to remote server.

Robin Shen commented 2 years ago

Another thing important: make sure to change file mode of id_rsa to 400

bufferUnderrun commented 2 years ago

when i test

docker run --rm -it --mount type=bind,src=/home/debian/test,dst=/opt/ alpine

and then inside, execthe same command

mkdir /root/.ssh
cat /opt/id_rsa >>/root/.ssh/id_rsa
chmod 400 /root/.ssh/id_rsa
ssh -o StrictHostKeyChecking=no debian@server

everything works

but not with onedev, very frustrating.

Moreover, i think there will be TTY problem running ssh command inside docker and onedev.

So, do you have a way to connect to server, copy file and run some commands ?

bufferUnderrun commented 2 years ago

update

when running verbose image_6.png

it seems my key is invalid. from stackoverflow some other have the same issue rapporting endofline format.

So let's try something : adding a extra empty line in the job secret

image_7.png

After saving, i show the job at new, the extra line has disappear image_8.png

Do you rewrite the job secret value with trim() or not handle append new line ?

Robin Shen commented 2 years ago

Yes, input will be trimmed for normalization purpose. It is surprising me the EOL is required for your server. I tested with Ubuntu server and the EOL does not matter.

bufferUnderrun commented 2 years ago

input will be trimmed for normalization purpose

?

and what about the line return when pasting in the textarea ? my client pc in on windows. i try to force \n and then pasting but do not work either

Robin Shen commented 2 years ago

All white spaces including line ending will be deleted

Robin Shen commented 2 years ago

Will turn this off in next patch release. For now, you may manually append a line ending to your private key file

bufferUnderrun commented 2 years ago

what it works :

  • adding the .ssh/id_rsa in the git repo and use-it for the ssh
  • adding the private key to a file "foo" in the git repo and cat foo >> .ssh/id_rsa (so no problem with cat command)

what it does not work :

  • paste the key to job secret and cat or echo the value to .ssh/id_rsa

you said you test it, but in the same conditions ? onedev is running on container and job run alpine image. i'm pasting the job secret from firefox on windows.

bufferUnderrun commented 2 years ago

and what about having custom file inject inside the running job container ?

we have artifact which are the result file of build why no having the opposite, file insert in the build without being in the repo ?

Robin Shen commented 2 years ago

Why cat or echo directly does not work:

As I mentioned earlier, @...@ will be replaced literally, so if you write

cat @secrets:id_sec@ >> /root/.ssh/id_rsa

It will be seen by shell as below after @secrets:id_sec@ being replaced:

cat firstline of private key
second line of private key
...
last line of private key >> /root/.ssh/id_rsa

This is the reason I am using below to write multiline string to a file in the first place:

cat <<EOT>> /root/.ssh/id_rsa
@secret:private-key@
EOT
bufferUnderrun commented 2 years ago

yes, that's why i use the code you propose

cat <<EOT>> /root/.ssh/id_rsa
@secret:private-key@
EOT

but does not work

i attach the private/pub keys i used you can try (obviously not production keys...).

Robin Shen commented 2 years ago

What image are you using for the job (alpine does not have ssh client available)? And what OS of your server you are trying ssh into?

bufferUnderrun commented 2 years ago

i'm using alpine and connect to debian 11.

this job command works as the id_rsa come from file in the git repo image_10.png

and not working when using the job secret with the cat EOT...

Robin Shen commented 2 years ago

Turns out that multi-line input in a text area will be saved using "\r\n" as line ending (this is html standard). And Debian is strict on line ending while Ubuntu not. The workaround is install dos2unix to your container, and then call below statement to convert "\r\n" to "\n"

dos2unix /root/.ssh/id_rsa
bufferUnderrun commented 2 years ago

And Debian is strict on line ending

😬 rohhhhhh

dos2unix /root/.ssh/id_rsa

such a journey, but this workaround is perfect !

thanks you Robin for all the help 👍

you can close the thread

Robin Shen changed state to 'Closed' 2 years ago
Previous Value Current Value
Open
Closed
issue 1 of 1
Type
Question
Priority
Normal
Assignee
Issue Votes (0)
Watchers (2)
Reference
onedev/server#519
Please wait...
Page is in error, reload to recover