Do you mean to use more memory in server shell job executor? The default docker command to start OneDev does not have any limit on memory used by the container, so I guess you should use as much memory as the system has in server shell executor. 

The thing is that I have the impression the server Shell does not take full advantage of the server configuration. I checked the Administration > System Maintenance > Server information section while the server shell executor was working (slowly) and the used memory was minimal. And yet I had allocated a lot of RAM and processor power to the Server Shell Executor:

A documentation on how to optimize the build is welcome.

The cpu/memory requirement of job is only used by job scheduler to determine which node (when using Kubernetes executor or remote docker/shell executor) should be used to run the job. It has no effect on real assignment of resources.

I started a Ubuntu VM with 10G physical memory, starting OneDev via docker, and run a job with server shell executor, and the job can run command requesting more than 6G memory. So there is nothing at OneDev side limiting memory of your script. Slowness of your build might be due to some other reason, such as docker file system etc. You may start OneDev without docker and run your build to see if it is still slow.