#1026  Many non existing dependencies
Closed
Tamas Cservenak opened 1 year ago

I know that onedev (parent, maven-plugin, server) is buildable "from scratch", did it locally. But, as I locally use Maven Repository Manager (MRM), tried to set up my env "by the book" (like never proxy a group/virtual reposiries), I found several artifacts NOT present anywhere, only on onedev repository (like the winp https://github.com/jenkinsci/winp), as it was available from now defunct JCenter, while newer versions are available from Jenkins GH. Consider some sort of "consolidation", as build is currently very dependant on onedev repository, maybe upgradfe these? I will try to collect the dependencies that are like winp and figure out something, unless there is objection...

Tamas Cservenak commented 1 year ago

Ideally all dependencies should come from:

  • ondedev hosted (parent POM, maven-plugin, common)
  • central
  • maybe jenkins repo

but they should have "available origin", the fact these are available ONLY from onedev repo (I guess cache?) is wrong...

Robin Shen commented 1 year ago

Thanks for the investigation. OneDev maven repository is setup to serve OneDev artifacts, as well as some other artifacts I did not manage to find any official repositories earlier, including several eclipse c/c++ parser libraries, mysql connector, and winp library. The repository can be publicly accessed here:

https://artifact.onedev.io/artifactory/webapp/#/artifacts/browse/tree/General/onedev

Some might have official repositories now, please feel free to submit pull request to fix them. As long as all components of OneDev can be built from scratch, I will merge the change.

Robin Shen changed fields 1 year ago
Name Previous Value Current Value
Type
Bug
Improvement
Robin Shen commented 1 year ago

Also seems that the repository (using JFrog Artifactory) works in a proxy mode: for all missing artifacts, it will resort to maven central. I followed some quick tutorial to make things working and then go forward. If you have ideas for optimizing, I will be happy to merge them as well.

Robin Shen commented 1 year ago

PS: you may submit pull request directly here. Repositories can be forked like below:

2022-12-06_08-45-59.png

Tamas Cservenak commented 1 year ago

Howdy, I believe I can sort out everything, EXCEPT the Eclipse CDT (that is used ONLY in common-jsymbol)... Eclipse seems a bit "walled garden" like, as I don't see them publishing anything available to non-Eclipse ecosystem, all they publish to is their P2 repository... Any possiblity to replace this with something else? Do you have any stats how much CppExtractor is actually used? (as at one point all I did is just kill off CppExtractor and CppExtractorTest and commented out eclipse artifacts as "hack")

Tamas Cservenak commented 1 year ago

Also, would be good, if possible, to set up following additional hosted repositories (so they should serve ONLY what they have, not "auto reach to central" or any other "magic" thing):

  • onedev-releases
  • onedev-snapshots
  • onedev-thirdparty

If this is possible, would be very good (thirdparty could host things not available currently, like Eclipse CDT, while former two are obvious). Currently the onedev repo is "dangerous" as not only is mixed bag, but is also "auto reaching to cenrtal". This means, maven completely looses it's notion of artifact origin....

Tamas Cservenak commented 1 year ago

Aside of Eclipse CDT, this is what we have of repositories currently:

  • central
  • onedev
  • jenkins (for winp, 1 artifact)
  • jetbrains (for pty4j, 1 artifact)

So parent is modified to add these 3 (as central is already there). Btw, I used locally Maven 3.9.0-SNAPSHOT, as build gets a huge boost from new Maven Resolver feature https://maven.apache.org/resolver/remote-repository-filtering.html (will be out with Maven 3.9.0).

My groupId filters for onedev are:

groupId-jenkins.txt:

org.jvnet.winp

groupId-jetbrains.txt:

org.jetbrains.pty4j

groupId-onedev.txt:

io.onedev
Tamas Cservenak commented 1 year ago

And one question: is there any peculiar reason why Maven SNAPSHOT versions are not used?

Robin Shen commented 1 year ago

Eclipse CDT is used in CPPExtractor and this is required to parse symbols of C/C++ files, which is a core feature of OneDev.

I've created repository onedev-release, onedev-snapshot and onedev-thirdparty as suggested and moved necessary artifacts to them.

Please suggest necessary changes for pom. I'd suggest to use Maven 3.8 stable release instead of 3.9 snapshot for OneDev build. With Maven local repository cache, dependency resolving performance is not a problem for OneDev build.

As to not using SNAPSHOT versions, I tend to use release version for stability reason, unless there is something important only available in snapshots.

Tamas Cservenak commented 1 year ago

Cool.

Of course, Maven stable (currently 3.8.6) is to be used, I just tested upcoming 3.9 features, that's all.

Will ping in with changes a bit later today.

Tamas Cservenak commented 1 year ago

Sorry but got side tracked... will ping back once holidays off, probably in week or two

Robin Shen commented 1 year ago

@cstamas no problem at all. Just take your time. Wish a wonderful new year!

Unknown commented 1 year ago

@cstamas Why would jenkins be needed, onedev already has CI/CD built into it thus jenkins would not be needed, all builds for onedev can be found here

As for the dependencies, onedev has its own artifacts hosted on code.onedev.io, if you are talking about the other dependencies used, it would be very annoying to keep syncing up with maven central, just to not depend on anything other than code.onedev.io, plus it would increase the resource needs for onedev's development.

Tamas Cservenak commented 1 year ago

@polarian am aware of CI/CD capabilities of onedev, and I never mentioned Jenkins, so unsure where it comes from 😄

Also am aware of hosted repositories onedev use/has, all I wanted to clean up things a bit.

And finally, Maven releases happened so far (latest stable 3.8.7) is a bit "hostile" to projects NOT using Central only, like onedev is (well, it uses Central for deps but also gets some stuff from onedev hosted like deps mentioned above). So my goal was also to "exercise" some upcoming features of Maven 3.9 (and 4) with real life project like onedev is (see https://maven.apache.org/resolver/remote-repository-filtering.html for details and https://github.com/cstamas/rrf-demo for demo).

All in all, everything is fine, thanks for repo modifications (onedev-release, onedev-snapshot and onedev-thirdparty, they really serve just the "best practice" for artifact "hygiene"), and will come back soon.

Nothing is missing from onedev project, is really cool stuff from my POV, everything is here, it's just me and my time.... sorry for that,

Unknown commented 1 year ago

You did mention Jenkins: image.png

So this was more housekeeping that anything else?

Tamas Cservenak commented 1 year ago

@polarian sorry, was lapsus. I meant "CI system" whatever it is...

Well, there IS artifact that is hosted in jenkins repo only (is not on Central), so I was talking about this https://www.jenkins.io/doc/developer/publishing/artifact-repository/ and not as "Jenkins CI instance"

Tamas Cservenak commented 1 year ago

Yes, we can say this is "ongoing housekeeping" related to best practices applied for project, it's dependencies and hosted stuff.

Unknown commented 1 year ago

Personally I do not like the idea of the word "snapshot" because it causes issues with versioning, although it breaks java convention I personally like to stick with semantic versioning, which it appears onedev already goes by. I guess appending -snapshot to the end of the version works too :)

Were you requesting that Robin builds snapshots of the codebase before the actual releases?

Unknown commented 1 year ago
Tamas Cservenak commented 1 year ago

I was asking Robin why he does not produce snapshots, as one would expect in maven project, yes. I did not initiate any change in that area. Once I convince you guys why snapshot is good (or CD instead of it), then I may propose some changes 😄

Re Jenkins confusion: So Jenkins project delivers Jenkins CI as we know, but to achieve that, they also host a maven repository (in same manner as onedev hosts its own repository) for "intermediate artifacts" produced by Jenkins project, and that repository is no synced to Central. Onedev does depend on artifacts from this repository. So I meant the "use of Jenkins hosted repository" and not "use of Jenkins CI instance".

Unknown commented 1 year ago
The thing is at the moment, snapshots are not needed because the project is still quite small (the codebase would say otherwise) but member sizes there is only Robin developing it with a few contributors every now and again, and thus development is slow and steady.

Snapshots are often used for public testing, to allow people to test new features, so unless Robin needs to test a new feature, why should a snapshot be built, as it would just waste cpu time on the onedev server.


Robin Shen changed state to 'Closed' 8 months ago
Previous Value Current Value
Open
Closed
issue 1 of 1
Type
Improvement
Priority
Normal
Assignee
Issue Votes (0)
Watchers (4)
Reference
onedev/server#1026
Please wait...
Page is in error, reload to recover