#443  Mirror remote repository locally
Released
Stanislav N. opened 2 years ago

Hello, I want to be able to pull remote repository into a project inside 1dev e.g. from github or gitlab. Why:

  • To use dependencies from local repository (in case of dependency's repository disappeared from remote location)
  • To deploy third party software using a runner that has no internet access.

Maybe there are more use cases, but these are mine.

Robin Shen commented 2 years ago

You may add a helper project, configure its build spec to run a job with a cron schedule. The job pulls from remote repository and push into local repository. I experimented mirroring a github repo and below is the example build spec:

version: 13
jobs:
- name: CI
  steps:
  - !CommandStep
    name: sync
    image: alpine/git
    commands:
    - git clone https://someuser:access_token@@github.com/someuser/test
    - cd test
    - git push http://someuser:access_token@@onedev_host/test
      main:main
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  triggers:
  - !ScheduleTrigger
    cronExpression: 0 0 12 * * ?
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  cpuRequirement: 500
  memoryRequirement: 256
  timeout: 3600

In production, you may want to replace access token with job secret for security reason.

Robin Shen commented 2 years ago

Tips: To copy a build spec, edit your build spec and switch to edit source mode, copy the content, and then switch back to edit mode for further modification.

Stanislav N. commented 2 years ago

Sounds like a crutch :)

I think it's better to have a separate setting for repository which will contain remote repository's URL. There are two ways I saw that in the wild:

  1. Enter URL when creating repository. This repository will be marked as mirror and all changes will be overwritten. This is gitea way.
  2. Enter URL in repository's settings. Every automatic pull will overwrite any local changes. This is Gitlab/Kallithea way.

I was speaking about something like that.

Robin Shen commented 2 years ago

Thanks for the info. Will add the mirror concept in future version.

Trevor Vance commented 2 years ago

This is probably the only missing feature that is keeping me from fully switching from Gitea. I self-host locally and on a Linode. I have backups of of the containers data volumes, but I like to have my two instances synchronized for redundancy that won't require a restore in a failure scenario. For instance, if the server hosting the git service fails. Another use-case is to backup repos from github/gitlab that are not my own. Basically, I mean backing up my starred repos. This might be more out of the scope of what onedev is meant for. If it was implemented though, I think I would fully move to onedev for backing up public repos and as my git server. I loved the import process from Gitea, and how it lets you do them in bulk. Gitea, for comparison, makes you import a single repo at a time. I'll try the build script above soon, but It looks time-consuming to do if you have a lot of repos.

I'll share how I would like to see this implemented.

  • Keep the import process in bulk like it is.
  • The /projects/import/Gitea page (image below) should offer to save the credentials. Preferably with a dropdown box with "Save API URL and Password", "Save API URL only", and "Don't Save". onedev-gitea-1.png
  • To help with the one above, it would be very nice to have an interface for users to manage Personal Access Token for other services. It would be nice to have one PAT on Github for instance called "onedev Token" and all private repos being mirrored would use that to access. The token should be secure though. This might be a case where you would want to not output it anywhere. But, give the option to change it easily.
  • The "Repositories to Import" menu (image below), should not list any repos that currently exist. Basically, check the URLs and make sure nothing has it as a remote repo. There might be something that stops it from importing duplicates, I haven't tested that. But, some that I have just imported are showing up in that list if I try the import again. onedev-gitea-2.png
  • Have a configurable cron schedule to pull all mirrored repositories.
  • Also, another "project setting" section or it could probably be added in "General Setting" (image below) of the project settings. The mirror options that I would add would be "Remote URL" textbox, "Mirror Enable" checkbox, "Mirror Cron Schedule" cron box (like is being used in the backup settings), and maybe a drop-down box with options for Pull, Push, or Pull and Push. That way you can have a backup model (pull) if you'd like. But, you could also have an actual "mirror" (Pull and Push).

onedev-settings.png

I may be thinking too much in a self-hosting mindset though. Having all of these implementations might add a lot of overhead and people hosting public instances might not want anyone/everyone to have access to this. There is also a benefit for self-hosters that it makes it easier to try onedev. There is zero risk because we could mirror our current git server until we decide to fully make the switch. :-)

Robin Shen commented 2 years ago

Thanks for the suggestions. For automated sync, I plan to add built-in step to support that. Then user can easily accomplish the task with current CI/CD job schedule facility. This will not make OneDev too complicated to solve different use cases.

I have a question though for repository mirror: how do you proceed if both sides changed the repository, do you want a merge in this case? And in case of merge conflict, do you want to create a pull request and notify repository owner to resolve it?

Trevor Vance commented 2 years ago

how do you proceed if both sides changed the repository, do you want a merge in this case? And in case of merge conflict, do you want to create a pull request and notify repository owner to resolve it?

I think you pretty much hit the nail on the head. A merge conflict would be best handled with a pull request and a notification to the owner.

Another thing, now that I am thinking about it. Using the CI/CD job schedule it would be best if, in addition to a cron schedule, the push/pull could be triggered whenever a change is made on either repo. Ideally, both would push to the other immediately, then merge conflicts would be pushed back into between the devices and an individual server, hopefully™. Even if this is the case you would want to do like you said above for whenever it does happen.

onedev-mirror.drawio (1).png

Thanks for your prompt response, by the way. 😁

Robin Shen added to milestone "7.1.0" 2 years ago
OneDev changed state to 'Closed' 2 years ago
Previous Value Current Value
Open
Closed
OneDev commented 2 years ago

State changed as code fixing the issue is committed

OneDev changed state to 'Released' 2 years ago
Previous Value Current Value
Closed
Released
OneDev commented 2 years ago

State changed as build #2600 is successful

dufernst changed state to 'Open' 1 year ago
Previous Value Current Value
Released
Open
dufernst commented 1 year ago

Thanks for the new functionality it works perfect for my own repositories (that I want to sync between OneDev and Github).

However, for public repositories not controlled by me, the current approach with a configured pipeline in the repo itself, conflicts with the fast forward (and in general with creating a clean mirror of a remote repo).

Your proposed syncing option in a helper repo would theoretically work, but in practice having to clone 20-30GB spread over +- 100 repositories every time syncing occurs is not sensible.

A couple of options include:

Allowing the build config to be specified in a secondary branch, and allowing the pull build step in that secondary branch. While it feels like a dirty solution, it seems like the easiest to implement.
Being able to configure in the parents repo pipeline the synchronization of it's child repo's.
Making pull mirrors first class citizens in OneDev and allowing users to configure mirroring without specifying pipelines.
dufernst referenced from pull request 1 year ago
OneDev changed state to 'Closed' 1 year ago
Previous Value Current Value
Open
Closed
OneDev commented 1 year ago

State changed as code fixing the issue is committed

OneDev changed state to 'Released' 1 year ago
Previous Value Current Value
Closed
Released
OneDev commented 1 year ago

State changed as build #2956 is successful

crash commented 1 year ago

Hi, on many projects which I want to have on my local onedev instance, I’m using PullRepository step to achieve a sort of gitea “repo-mirror” feature, but in this way I’m forced to keep the default branch to the one with the relative build-spec.
Probably with this merge I could achieve a workaround to this, in any case can you give me an example of using this new “parent projects pull syncing”?

thank you!

Robin Shen commented 1 year ago

The pull repository step now has a sync child option like below:

2022-10-10_10-11-49.png

issue 1 of 1
Type
New Feature
Priority
Normal
Assignee
Milestones
Issue Votes (2)
Watchers (7)
Reference
onedev/server#443
Please wait...
Page is in error, reload to recover