#762  Allow globally reusable definition of cache folders and build steps
Closed
jbauer opened 2 years ago

Hi,

Having a large number of Java projects you often end up defining the same things in build spec over and over again especially if all these projects are built using the same build tool, e.g. Gradle or Maven.

Common groups of build steps:

  • Initialize the build
    • checkout code
    • detect build version
    • set build version
  • Testing
    • run tests
    • publish test results

Common Java cache folders for such projects:

  • Gradle distributions downloaded by Gradle wrapper
  • JDKs downloaded by Gradle build toolchains
  • Gradle artifact cache
  • Maven artifact cache

It would be great if we could define global templates for build steps and cache configuration. So given the example above you would have global step templates named Initialize (having 3 steps) and Run Tests (having 2 steps) and I can reuse them in any job. I could even imagine that global step templates can include other global step templates, so that a CI step template could include the Initialize and Run tests templates. Having such global templates would also make it easier to change the behavior of many builds by simply updating the global step template.

Same with caches. The above example could be stored as a global cache config named "Gradle/Maven build cache" and then reused everywhere.

Robin Shen commented 2 years ago

How about importing step templates from a common project?

Another approach is to define a common job in common project with steps and caches parameterized, and then import the common job in different projects and have a downstream job depending on this common job with desired parameter values.

jbauer commented 2 years ago

How about importing step templates from a common project?

Hm that might work, although a bit hidden. I just tried it as administrator but OneDev complains about missing code read permissions on the project I want to import the steps from. So I selected the buildbot ssh private key (which has code read on everything and used by build jobs) but then OneDev says the private key is invalid AND prints the private key into the error message! So I guess something is broken?

Robin Shen commented 2 years ago

Even if you are login as administrator, the job itself has to run in the background requiring permission to import from other projects. This is something like root user still need to give a OS process permission explicitly to access system resources.

The secret has to contain value of an access token instead of private key, as explained in inline help of the build spec import page.

The access token value should not printed in error message though. I will create a bug report to track this.

Robin Shen commented 2 years ago
Robin Shen changed state to 'Closed' 2 years ago
Previous Value Current Value
Open
Closed
jbauer commented 2 years ago

Sorry for coming back late but I was busy with other things.

I have taken a look at importing build settings from a common project but I still feel like some things are hidden or missing.

  1. After reading the linked article and going through OneDev UI I had the impression that I need to create a GIT tag in the common project in order to import it. That would be a blocker because when I change something in the common project I do not want to go through ~60 projects and update the GIT tag in the import section. So I just tried entered HEAD and luckily it did work, so that I can basically batch update many projects by updating the common project. Maybe you should add latest or HEAD to the dropdown listing all GIT tags in the import UI. That would make this feature more obvious (and hopefully never disappear in future releases).
  2. When I define a job in a common project and import it into another project there is no way to adjust job settings. The only way to adjust anything is by recreating the job again by scratch for that project that needs adjustments. Common examples would be changing resource requirements for larger projects, changing job and project dependencies or post build actions if a project is more complex. If you have to adjust many things, starting from scratch is ok, but if you just want to change resource requirements it seems a bit overkill. Maybe allow parameters/properties in more job settings?
  3. While I have mostly Java + Gradle projects I could not figure out if I can have more than one build spec file in a common project, e.g. one for java+gradle projects, one for java+maven projects, one for go projects. IMHO it feels a bit messy to create projects for each toolchain with just a single build spec file in it.

So the main issue is that jobs in a common project are not adjustable. Allowing properties/parameters in more job settings can help but treating an imported job as a template that can be modified as needed would be great. However that would need to keep track of what has been modified so that modified settings are not overriden when updating the original, imported job.

jbauer changed state to 'Open' 2 years ago
Previous Value Current Value
Closed
Open
Robin Shen changed state to 'Closed' 2 months ago
Previous Value Current Value
Open
Closed
Robin Shen commented 2 months ago

Caches are now steps which can be shared via step templates. Resource requirements moved to job executors which is no longer applicable for this topic. And most other job settings can be adjusted via variables.

issue 1 of 1
Type
Improvement
Priority
Normal
Assignee
Issue Votes (0)
Watchers (4)
Reference
onedev/server#762
Please wait...
Page is in error, reload to recover