Deploy project build examples (#8)

Quin opened 4 years ago

I've got many WordPress sites that I like to automate.

I come from a Gitlab background, and their documentation for CI/CD isn't the best, and I managed to cobble together something that has managed to work. Basically, it's a bunch of SSH commands (so I don't have to manually copy and paste each line whenever I need to deploy). This ran whenever a designated branch was updated.

Now, what would be the best practice in order to do the same, and how to watch for those changes (i.e. push to Staging branch to update Staging server)? I generally don't use build or version numbers (knowing that is an option), because I don't really know how to use them effectively.

Below is what I used (and I used Gitlab variables too):

deploy_staging:
  stage: deploy
  script:
    - eval $(ssh-agent -s)
    - echo "$SSH_REPO_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh && touch ~/.ssh/known_hosts
    - echo "$SSH_STAGING_KNOWN_HOST" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts

    # Make a temp folder
    - ssh -p22 root@${STAGING_IP} "mkdir -p /var/www/html/${SITE_DIR}_tmp"
    # Copy the files from the repo to the temp folder
    - rsync -rav -e ssh --exclude='.git/' --exclude='.gitlab-ci.yml' --exclude='gulpfile.js' --delete-excluded ./ root@${STAGING_IP}:/var/www/html/${SITE_DIR}_tmp
    # Rename current site into backup
    - ssh -p22 root@${STAGING_IP} "mv /var/www/html/${SITE_DIR} /var/www/html/${SITE_DIR}_bak"
    # Copy uploads folder to temp
    - ssh -p22 root@${STAGING_IP} "cp -rf /var/www/html/${SITE_DIR}_bak/wp-content/uploads /var/www/html/${SITE_DIR}_tmp/wp-content/uploads"
    # Make the temp folder the actual site
    - ssh -p22 root@${STAGING_IP} "cp -rf /var/www/html/${SITE_DIR}_tmp /var/www/html/${SITE_DIR}"
    # Set the ownership and permissions
    - ssh -p22 root@${STAGING_IP} "find /var/www/html/${SITE_DIR} -type f -exec chmod 644 {} \;"
    - ssh -p22 root@${STAGING_IP} "find /var/www/html/${SITE_DIR} -type d -exec chmod 755 {} \;"
    - ssh -p22 root@${STAGING_IP} "chown -R www-data:www-data /var/www/html/${SITE_DIR}"
    # Remove the old site
    - ssh -p22 root@${STAGING_IP} "rm -rf /var/www/html/${SITE_DIR}_bak"
    # Remove the temp site
    - ssh -p22 root@${STAGING_IP} "rm -rf /var/www/html/${SITE_DIR}_tmp"
  only:
    - staging

Activities

Robin Shen commented 4 years ago

In OneDev, you can add a job and have it triggered automatically when a branch is updated (check the triggers and params section of a job). Then you can add necessary steps to do what you want, including checkout code, running commands to call ssh to deploy files to your servers etc.
Robin Shen changed state to 'Closed' 4 years ago
Previous Value Current Value
Open

Closed
Quin commented 4 years ago

Do you have any examples of how to do this? I'm a bit stuck in trying to figure this out.

Previous Value	Current Value
Open	Closed

Robin Shen commented 4 years ago

For instance I used below build spec to copy web site content to server via SCP, and then ssh to the server deploying the content:

version: 6
jobs:
- name: Publish
  steps:
  - !CheckoutStep
    name: 'checkout '
    cloneCredential: !DefaultCredential {}
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: build & deploy
    image: node:10.21.0
    commands:
    - set -e
    - ''
    - npm install
    - node_modules/.bin/gatsby build
    - ''
    - mkdir /root/.ssh
    - chmod 700 /root/.ssh
    - ''
    - cat << EOF > /root/.ssh/known_hosts
    - '<known host string>'
    - EOF
    - ''
    - cat << EOF > /root/.ssh/id_rsa
    - '@secrets:private_key@'
    - EOF
    - ''
    - chmod 400 /root/.ssh/id_rsa
    - ''
    - scp -r public myuser@@myhost:/home/myuser
    - ssh myuser@@myhost sudo rm -rf /var/www/html
    - ssh myuser@@myhost sudo mv /home/myuser/public /var/www/html
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  triggers:
  - !BranchUpdateTrigger
    branches: main
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  cpuRequirement: 250m
  memoryRequirement: 128m
  caches:
  - key: npm-cache
    path: /root/.npm
  timeout: 3600

Here <known host string> needs to be replaced by host fingerprint of the server, and you also need to define a build secret private_key holding a private SSH key able to login to the server

Quin commented 4 years ago

Thanks for this @robin. Is there a way to set known host string and private_key as a variable stored on the project, so they're not saved into version control?
Robin Shen commented 4 years ago

Here private_key is defined as a project secret (check project setting / build setting / job secret). The same can be done for known host.
Quin commented 4 years ago

Great. But how do I know what/where the directory is for the project? Here, you are copying the "public" directory to the remote/production server with SCP, right?
Robin Shen commented 4 years ago

When run the step command, the current directory is root of repository if checkout step runs previously. And my "public" directory is a sub folder stored in the repository.

Quin commented 4 years ago

I've tried editing the buildspec, and this is what comes back whenever I add something other than just the version number (and I cannot commit if the version number hasn't changed):

Error parsing build spec
io.onedev.server.buildspec.BuildSpecParseException: Malformed build spec
	at io.onedev.server.buildspec.BuildSpec$1.load(BuildSpec.java:82)
	at io.onedev.server.buildspec.BuildSpec$1.load(BuildSpec.java:72)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3444)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2193)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2152)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2042)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3850)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3874)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4799)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4805)
	at io.onedev.server.buildspec.BuildSpec.parse(BuildSpec.java:574)
	at io.onedev.server.web.page.project.blob.render.renderers.buildspec.BuildSpecBlobViewPanel.onInitialize(BuildSpecBlobViewPanel.java:80)
	at org.apache.wicket.Component.fireInitialize(Component.java:878)
	at org.apache.wicket.MarkupContainer.internalInitialize(MarkupContainer.java:1071)
	at org.apache.wicket.MarkupContainer.addedComponent(MarkupContainer.java:1048)
	at org.apache.wicket.MarkupContainer.replace(MarkupContainer.java:856)
	at io.onedev.server.web.page.project.blob.ProjectBlobPage.newBlobContent(ProjectBlobPage.java:845)
	at io.onedev.server.web.page.project.blob.ProjectBlobPage.onResolvedRevisionChange(ProjectBlobPage.java:1039)
	at io.onedev.server.web.page.project.blob.ProjectBlobPage.onCommitted(ProjectBlobPage.java:1467)
	at io.onedev.server.web.page.project.blob.render.commitoption.CommitOptionPanel.save(CommitOptionPanel.java:342)
	at io.onedev.server.web.page.project.blob.render.commitoption.CommitOptionPanel.access$100(CommitOptionPanel.java:67)
	at io.onedev.server.web.page.project.blob.render.commitoption.CommitOptionPanel$1.onSubmit(CommitOptionPanel.java:192)
	at org.apache.wicket.ajax.markup.html.form.AjaxButton$1.onSubmit(AjaxButton.java:113)
	at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:218)
	at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312)
	at org.apache.wicket.markup.html.form.Form.process(Form.java:976)
	at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:797)
	at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:174)
	at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155)
	at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:601)
	at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
	at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
	at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248)
	at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234)
	at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:955)
	at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
	at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:288)
	at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:245)
	at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:316)
	at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70)
	at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
	at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:43)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$b27a4b57.CGLIB$service$2(<generated>)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$b27a4b57$$FastClassByGuice$$122b7d63.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
	at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$b27a4b57.service(<generated>)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
	at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:440)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:330)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.util.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at io.onedev.commons.bootstrap.Bootstrap.unchecked(Bootstrap.java:363)
	at io.onedev.commons.utils.ExceptionUtils.unchecked(ExceptionUtils.java:35)
	at io.onedev.server.migration.MigrationHelper.migrate(MigrationHelper.java:151)
	at io.onedev.server.migration.VersionedYamlDoc.toBean(VersionedYamlDoc.java:61)
	at io.onedev.server.buildspec.BuildSpec$1.load(BuildSpec.java:80)
	... 122 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.onedev.server.migration.MigrationHelper.migrate(MigrationHelper.java:149)
	... 124 more
Caused by: java.lang.IllegalStateException
	at com.google.common.base.Preconditions.checkState(Preconditions.java:494)
	at io.onedev.server.buildspec.BuildSpec.migrate5(BuildSpec.java:720)
	... 128 more

I can still edit. Deleting the file and starting again does the same thing.

Robin Shen commented 4 years ago

Just copy the build spec as is to OneDev build spec editor (edit source), then switch to edit tab to edit it.
Quin commented 4 years ago

I honestly did, and it broke when I edited it. Even re-pasting it as is didn't work, nor pasting in a new file.

So I deleted it and started from scratch using the UI. There is little going in terms of documentation and understanding, because copying that made a job run, but following another guide you created involves creating an agent (which I didn't do before).
Robin Shen commented 4 years ago

Not sure what happens at your side. But I just copy the content, and then switch to "Edit" tab to change things. Everything works fine. Generally you should not edit the build spec source directly.
Quin commented 4 years ago

Okay, I've basically got this working again, and I have a better understanding. However, once broke, the file needs to be deleted again and then recreated.

Quin commented 4 years ago

@robin I am now facing an issue in creating a directory. I don't think it's a OneDev issue, but maybe you can help?

I can't scp . due to security issues, so I'm trying rsync, which I've used before. However, I get "permission denied" when try to create a directory. I tried echoing out $USER, but it returns as blanked out (as though a password); I had hoped to check the user and make sure they have the correct permissions. The dir secret also returns as the blanked string: rsync: mkdir "/var/www/html/*****_tmp" failed: No such file or directory (2)

It fails each time I get to the Upload step, due to the permissions issue.

version: 15
jobs:
- name: Deploy to Staging
  steps:
  - !CommandStep
    name: SSH Agent
    runInContainer: false
    interpreter: !DefaultInterpreter
      commands:
      - eval $(ssh-agent -s)
      - echo "@secrets:private_key@" | tr -d '\r' | ssh-add -
      - mkdir -p ~/.ssh
      - chmod 700 ~/.ssh
      - ''
      - echo "@secret:known_hosts@" >> ~/.ssh/known_hosts
      - chmod 644 ~/.ssh/known_hosts
      - ''
      - echo $USER
    useTTY: true
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CheckoutStep
    name: Checkout
    cloneCredential: !DefaultCredential {}
    withLfs: false
    withSubmodules: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Upload
    runInContainer: false
    interpreter: !DefaultInterpreter
      commands:
      - echo "$USER"
      - mkdir -p /var/www/html/@secret:dir@_tmp
      - rsync -rav -e ssh --exclude='.git/' --exclude='*.gitignore' --exclude='*.yml'
        --exclude='*.sql' --exclude='gulpfile.js' --delete-excluded ./ @secret:user@@@@secret:ip@:/var/www/html/@secret:dir@_tmp
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Backup
    runInContainer: false
    interpreter: !DefaultInterpreter
      commands:
      - echo Create a backup from current files
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Copy
    runInContainer: false
    interpreter: !DefaultInterpreter
      commands:
      - echo Move the files to the server
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Cleanup
    runInContainer: false
    interpreter: !DefaultInterpreter
      commands:
      - echo Remove the previous versions
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  cpuRequirement: 250
  memoryRequirement: 128
  timeout: 3600
  postBuildActions:
  - !SendNotificationAction
    condition: always
    receivers: user(classicniall)

Robin Shen commented 4 years ago

I am not familiar with rsync. It might be easier to make your script working out of OneDev first.
Quin commented 4 years ago

Okay, I am getting somewhere. I'm rebuilding it all, and testing out all the commands.

I do get this: sudo: unable to resolve host *****: Name or service not known When I run this command: ssh @secret:user@@@@secret:ip@ sudo mkdir -p /var/www/html/site_tmp However, the directory is created.
Login to comment

Type	Question
Priority	Normal
Assignee	Not assigned

Issue Votes (0)

Watchers (2)

Reference

onedev/manual#8