NixOS support for agent (OD-1080)
bufferUnderrun opened 3 years ago

Hi,

In my journey to make onedev running under Nixos/podman-rooless, the last not working thing is the bare metal agent.

Some command line tools are not found

> ./agent.sh console
Unable to locate ps.
Please report this message along with the location of the command on your system.

In nixos, all core-utilities are not located in /usr/bin but in another dimension 😅 so the agent-sh-3000-bunch-of-line is "mission abandoned".

Before trying to patch, i wonder if you use a sort of generator to make the agent.sh ?

At last, to get better portability across posix OSess, it's recommanded to use /usr/bin/env command instead of /usr/bin/command.

So using /usr/bin/env ps in the script should resolve some issues...

Thanks for you help

  • Robin Shen changed fields 3 years ago
    Name Previous Value Current Value
    Type
    Bug
    Improvement
  • Robin Shen commented 3 years ago

    NixOS is not in the list of OS supported by OneDev. So I am turning this into an improvement request.

    OneDev under the hood relies on java service wrapper (https://wrapper.tanukisoftware.com/doc/english/home.html) for OS and service integration. I will contact their support to see if there is a chance to support NixOS.

  • Robin Shen changed title 3 years ago
    Previous Value Current Value
    agent.sh and cmd dependencies
    NixOS support for agent
  • Robin Shen commented 3 years ago

    Contacted JSW support, and they want some more info:

    1. run which ps to send the output
    2. open an terminal and change to directory <agent install dir>/boot, run command ./wrapper-linux-x86-64 -h and send the output (you may need to run other wrapper-linux-xxx binary if x86 64bit is not the right one)
    3. If step 2 works, please run './wrapper-linux-x86-64 ../conf/wrapper.conf` to see if it works.
  • bufferUnderrun commented 3 years ago

    ~$ which ps

    /run/current-system/sw/bin/ps
    

    ~/agent/boot/$ ./wrapper-linux-x86-64 -h

    zsh: no such file or directory: ./wrapper-linux-x86-64
    

    it's related to nixos special architecture https://nixos.wiki/wiki/Packaging/Binaries. So some infos

    ~/agent/boot $ file wrapper-linux-x86-64

    wrapper-linux-x86-64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.26, BuildID[sha1]=743d3d07c2d5bb8e5ece878d2e5c5c99c0afd7fc, stripped
    

    ~/agent/boot $ ldd wrapper-linux-x86-64

    linux-vdso.so.1 (0x00007fff6e934000)
    libm.so.6 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libm.so.6 (0x00007f6b29a2e000)
    libpthread.so.0 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libpthread.so.0 (0x00007f6b29a29000)
    libc.so.6 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libc.so.6 (0x00007f6b29820000)
    /lib64/ld-linux-x86-64.so.2 => /nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f6b29b10000)
    
  • bufferUnderrun commented 3 years ago

    I am curious about the interest of the wrapper ?

    At last, do you have a launching workaound like java *.jar --parm1 --part2... so i could write a specific systemd unit for nixos ?

  • Robin Shen commented 3 years ago

    It wraps OneDev as OS services on different OS. Also it can restart agent VM after upgraded. It is not trival to support this on Windows, Mac, Linux, FreeBSD, and many other OSes.

    You can run agent without JSW like below:

    cd <agent install dir>/boot
    java -jar "../lib/<version>/*" io.onedev.agent.Agent
    

    However you lost the ability to upgrade agent automatically.

  • bufferUnderrun commented 3 years ago

    trying

    ~/agent/boot> java -jar "../lib/1.7.15/*" io.onedev.agent.Agent
     Error: Unable to access jarfile ../lib/1.7.15/*
    

    but

    ~/agent/boot>  java -jar ../lib/1.7.15/* io.onedev.agent.Agent
    no main manifest attribute, in ../lib/1.7.15/ch.qos.logback.logback-classic-1.2.11.jar
    
  • Robin Shen commented 3 years ago

    Sorry ought to be java -cp "../lib/<version>/*" io.onedev.agent.Agent

  • bufferUnderrun commented 3 years ago

    Perfect, i will write my own systemd unit even if autoupdate is not handle.

    So, i can wait for JWS support nixos as in my point of view everything is working for production level.

    Thanks robin for your support

  • Robin Shen commented 3 years ago

    From JSW support:

    I didn't know NixOS, but my understanding is that it's a system that is suited to software with source code available, as it first has to build a binary locally before being able to execute it (or use some binary caching). If that's the case, then the Wrapper Standard and Professional Editions will not work as-is on NixOS.

    The link you sent (https://nixos.wiki/wiki/Packaging/Binaries), as well as the following page, describe several methods to execute binaries on NixOS. Have you tried any of them? In particular, have you tried autoPatchelfHook to fix the dependency paths?

    If that would work for you, we could think of ways to edit the wrapper script in order to ease the patching of the Wrapper binaries, for example with an additional setup step, either called during the installation of the service or with the help of a new command. Please feel free to share your thoughts on this.

    These would be very platform-specific changes, and we had no other requests to support NixOS yet. So we would need to consider if it should be implemented as a custom edit, or if it would be included in the official Wrapper distributions.

  • bufferUnderrun commented 3 years ago

    Sorry for late, thanks for the response.

    I setup a custom solution which not handle agent update but works great since 1 week. I copy/paste here the steps to follow if someone is experiencing same difficulties.

    1. For an unprivilege user <myuser>
    2. Extract the bare metal agent under the home dir /home/<myuser>/onedev-agent.
    3. Configure the agent following the official onedev guide (agentName...)
    4. Then edit the Nixos /etc/nixos/configuration.nix
        systemd.user.services."onedev-agent-<myuser>" = {
            enable = true;
            unitConfig = { ConditionUser = "<myuser>"; };
            wantedBy = [ "default.target" ];
            after = [ "network.target" ];
            description = "onedev-agent-<myuser>";
            path = [ config.system.path ];
            serviceConfig =
            let
                java = "${pkgs.jdk11_headless}/bin/java";
            in
            {
                ExecStart = "${java} -cp '%h/onedev-agent/lib/1.7.16/*' io.onedev.agent.Agent";
                Type = "simple";
            };
        };
    

    5 . update Nixos with nixos-rebuild switch 6 . in a terminal under the <myuser> user session use the systemd command to start the newly created service systemctl --user start onedev-agent-<myuser> 7 . if correct, enable the service at boot up with systemctl --user enable onedev-agent-<myuser> and allow lingering with this user loginctl enable-linger <myuser>

    Remarks :

    • the path to lib/1.7.16 is corresponding to onedev v7.9.4. Adapt depending your version.
    • i prefer running agent with an unprivilege dedidacted user. This way allows running multiple separated agents cleanly in the futur. In fact, the user running the agent is the same running my production app. So yes, i use the bare metal agent for the CD job... and also for CI job build IOS app on MacOS.

    I close the issue and as soon as i have time, i will test the packaging binaries proposed by nixos team and reopen this issue.

    Thanks for your help

  • bufferUnderrun changed state to 'Closed' 3 years ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen commented 3 years ago

    Thanks for sharing @bufferunderrun

issue 1/1
Type
Improvement
Priority
Normal
Assignee
Issue Votes (0)
Watchers (3)
Reference
OD-1080
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover