Set up Test Server

1. Launch a AWS EC2 c5a.xlarge instance with Ubuntu Server 18.04 image and 50G GP2 disk. Configure security group to allow all incoming TCP ports for our test convenience.
2. Login to launched instance as user ubuntu, edit /etc/hostname to set host name as server, and then reboot
3. Install GitLab EE follow its installation guide (this installs 14.10.2-ee at the time of testing). Specify environment variable EXTERNAL_URL as http://server during install
4. After install, login to GitLab and disable gravatar to avoid out-of-server resource loading from Admin/Settings/General/Account and Limit. Also disable auto devops to save cpu cycles from Admin/Settings/CICD/Continuous Integration and Deployment.
5. Add a blank public project kubernetes under root user, without ticking the option Initialize repository with a README
6. Install OneDev by running below commands

ubuntu@server:~$ sudo apt update
ubuntu@server:~$ sudo apt install -y unzip openjdk-8-jdk
ubuntu@server:~$ wget https://code.onedev.io/downloads/projects/160/builds/2730/artifacts/onedev-7.3.2.zip
ubuntu@server:~$ unzip onedev-7.3.2.zip
ubuntu@server:~$ mv onedev-7.3.2 onedev  

6. Continue to edit file /home/ubuntu/onedev/conf/wrapper.conf to change wrapper.java.maxmemory.percent to 20, which means to use 20% of physical memory as the default 50% is too deluxe for our tests
7. Start OneDev from terminal by running /home/ubuntu/onedev/bin/server.sh console
8. Login to OneDev to set it up. Disable gravatar from menu Administration / System Setting
9. Add a new project kubernetes. Edit general setting of the new project to change default role as Code Read. Also switch to menu Administration / Security Setting to enable anonymous access

Test Repository Push/Clone Performance

1. Launch another EC2 instance with same subnet, hardware spec and OS as test server. Configure security group to allow all incoming TCP ports for our test convenience.
2. Login to new launched instance, name it test by editing /etc/hostname and reboot
3. Edit /etc/hosts to add an entry server pointing to private ip address of server machine
4. Login to test as user ubuntu and get kubernetes source from GitHub:

ubuntu@test:~$ git clone https://github.com/kubernetes/kubernetes
ubuntu@test:~$ cd kubernetes
ubuntu@test:~/kubernetes$ git reset --hard 92285fd74e8906bc4910d6fb6729b7ec0e74f427

Here reset master branch of the repository to same commit as I am doing the test 3. Stop GitLab on server by running sudo gitlab-ctl stop), and keep OneDev running 4. Measure repository push time to OneDev by running:

ubuntu@test:~$ cd ~/kubernetes
ubuntu@test:~/kubernetes$ time git push http://<user>:<pass>@server:6610/kubernetes master:master

Here <user> and <pass> should be replaced with OneDev administrator user/pass 5. Continue to measure repository clone time from OneDev by running:

ubuntu@test:~/kubernetes$ cd ~
ubuntu@test:~$ time git clone http://server:6610/kubernetes kubernetes-onedev

6. Now stop OneDev on server by pressing ctrl-c from its console window, and start GitLab again by running sudo gitlab-ctl start
7. Measure repository push time to GitLab by running:

ubuntu@test:~$ cd ~/kubernetes
ubuntu@test:~/kubernetes$ time git push http://<user>:<pass>@server/root/kubernetes.git master:master

Here <user> and <pass> should be replaced with GitLab administrator user/pass 5. Continue to measure repository clone time from GitLab by running:

ubuntu@test:~/kubernetes$ cd ~
ubuntu@test:~$ time git clone http://server/root/kubernetes.git kubernetes-gitlab

Test Web UI Performance

1. Login to server machine to start both OneDev and GitLab
2. Login to test machine to run below commands:

ubuntu@test:~$ cd kubernetes
ubuntu@test:~/kubernetes$ git push http://<OneDev user>:<OneDev pass>@server:6610/kubernetes eb88daeeae4a53a20579620e1791e30416517223:refs/heads/prod
ubuntu@test:~/kubernetes$ git push http://<GitLab user>:<GitLab pass>@server/root/kubernetes.git eb88daeeae4a53a20579620e1791e30416517223:refs/heads/prod

Here <OneDev user>, <OneDev pass>, <GitLab user> and <GitLab pass> should be replaced by OneDev/GitLab administrator user/pass respectively

3. Login to OneDev to create a pull request from branch master to prod for kubernetes project. Do the same for GitLab
4. Login to test machine, run below command to install chrome and jmeter:

ubuntu@test:~$ sudo apt update
ubuntu@test:~$ sudo apt install -y openjdk-8-jdk
ubuntu@test:~$ wget https://code.onedev.io/projects/406/raw/master/jmeter.tar.gz
ubuntu@test:~$ tar zxvf jmeter.tar.gz
ubuntu@test:~$ wget https://code.onedev.io/projects/406/raw/master/google-chrome-stable_current_amd64.deb
ubuntu@test:~$ sudo apt install ./google-chrome-stable_current_amd64.deb

Here we download a pre-configured JMeter package from code.onedev.io. If you want to start from scratch with official JMeter package, you will need to:

• Install JMeter plugin manager
• Install Selenium webdrive plugin
• Install chrome webdrive for Linux AMD 64
• Generate rmi keystore
• Copy folder bin/tests from pre-configured JMeter package to your own JMeter directory

We also use Chrome package from code.onedev.io as its version is 101, which is the same as the webdrive executable. You may download from Google directly, but make sure the version matches.

5. Run below command to make sure our tests can run without any errors:

ubuntu@test:~$ cd apache-jmeter-5.4.3/bin
ubuntu@test:~/apache-jmeter-5.4.3/bin$ ./jmeter -n -t tests/onedev/tree.jmx 

6. Press ctrl-c to stop tests. Now we need to launch more EC2 instances to run chrome as a single test machine is not powerful enough to drive OneDev to use nearly 100% cpu
7. Launch a AWS EC2 c5a.2xlarge instance with Ubuntu Server 18.04 image and 16G GP2 disk. Configure security group to allow all incoming TCP ports for our test convenience
8. Login to newly launched instance, edit /etc/hosts to map name server to private ip address of the server running OneDev and GitLab
9. Install JMeter and Chrome on this instance following instructions in step 4 above
10. Run below command to run JMeter server:

ubuntu@ip-xxx-xxx-xxx-xxx:~$ cd apache-jmeter-5.4.3/bin
ubuntu@ip-xxx-xxx-xxx-xxx:~/apache-jmeter-5.4.3/bin$ ./jmeter-server

11. Repeat step 7~10 to run JMeter server on another three instances. You may consider creating AWS image to start JMeter server instances easier
12. Login to test machine, edit property remote_hosts of file apache-jmeter-5.4.3/bin/jmeter.properties to include private addresses (separated by comma) of all JMeter server instances launched above
13. Now you can run distributed tests from the test machine with below command:

ubuntu@test:~$ cd apache-jmeter-5.4.3/bin
ubuntu@test:~/apache-jmeter-5.4.3/bin$ ./jmeter -n -t <path to JMeter test plan>

Here <path to JMeter test plan> should be replaced by path to various .jmx file under tests directory. Make sure to stop GitLab when testing OneDev, and vice versa. Each test plan will last for 10 minutes, and will print test summary when it finishes. Make sure to run each test plan for 1 minute to warm up the server before formal test. Running test can be stopped by executing ~/apache-jmeter-5.4.3/bin/stoptest.sh.

While test is running, you may run command htop on the server machine to monitor cpu and memory usage. For test plan tests/onedev/tree.jmx, one more instance needs to be launched to run JMeter server (and remember to add its private ip address to jmeter.properties on test machine as explained above), otherwise server cpu can not reach 100%

Test Result

https://robinshen.medium.com/performance-compasion-of-onedev-and-gitlab-c11fc27b25be

Any question, please contact me at [robin AT onedev DOT io]