[cache] Expose paths from within job agent container to host machine (OD-2632)
wojtek opened 4 weeks ago

Hi, is there any way to mount host directory to job container during build run? The idea is to have same .gradle and .m2 directories to avoid constantly downloading dependencies (which in case of Kotlin/Android project can be a lot…)

I know there is a "Cache step" (https://docs.onedev.io/tutorials/cicd/job-cache) but from what I tried/check it basically involves moving directories configured as cacheable back and forth to the server. It has a couple of downsides: it still involves transferring rather larger amount of which both wastes time as well as often resources if the agent is in the different cloud provider (ingress/egress cost)

Having simple way to configure local host directory as shared each time job is run would be great.

  • Robin Shen commented 4 weeks ago

    OneDev stored cache on host machine initially, and changed to store on server according to feedback from several users, mentioning that host machine should not store any application data, as it can affect stability of host machine.

  • wojtek commented 4 weeks ago

    host machine should not store any application data, as it can affect stability of host machine.

    Erm. I would say this argument is flawed. A lot of things can affect stability of host machine - for example running more memory-demanding job. While I agree that more cache on local machine would be another point adding to the list it's not the only one and as always all machines/deployments require monitoring of it's state (and probably use some orchestration to restore it to working state if the machine goes down/is unreachable).

    On the other hand - cache on the main server can "affect it's stability" as well (and again - requires monitoring/orchestration just the same).

    You added "cache management" (on the server) to somewhat mitigate it but the same solution could have been applied to caches stored on agent machine - it would simply be shown in the administrator/agents section.

    And it would make running the jobs waaaaaaay faster as it would avoid transferring giganting amounts of data back and forth…

    Also - it would be convenient to have single cache shared with all projects as having same .gradle and .m2 directory would avoid repeatitive downloading of dependencies, which impacts maven central, see: https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons https://blog.gradle.org/maven-central-mirror

    Problematic are most probably more CI/CD pipeplines with ephemeral build agents and neither a repository mirror nor a shared read-only dependency cache, which means that each and every build by that CI/CD pipeling will re-download the world.

  • Robin Shen commented 3 weeks ago

    This is an obvious bad practice, and will not be considered. Not to mention that there are other limitations such as cache not sharable with other nodes, unable to remove obsolete caches etc.

  • Robin Shen changed state to 'Closed' 3 weeks ago
    Previous Value Current Value
    Open
    Closed
issue 1/1
Type
Question
Priority
Normal
Assignee
Labels
No labels
Issue Votes (0)
Watchers (2)
Reference
OD-2632
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover