Artur
opened 9 months ago
-
I faced the same issue before, and solved it by putting this instance behind cloudflare and configured below waf rule:
-
Previous Value Current Value Open
Closed
-
Thank you for the suggestion on how to remedy the problem.
However, this seems more like a workaround rather than a proper solution. It seems it requires use of CloudFlare but not everybody uses CloudFLare.
I am reopening the ticket hoping that you reconsider implementation of additional permissions to make it possible to hide these heavy functions from the public.
Having these permissions does not take any functionality away from OneDev. Whoever wants may continue to allow to expose commits and compare publicly and use CloudFlare to block AI scrappers but there would be another solution. And, I strongly believe, at least in our case, exposing compare and commits to public does not bring much value.
-
Previous Value Current Value Closed
Open
-
We had some more talk about this within our team and there was suggestion/proposition to maybe implement some kind of limits for a number of queries for non-logged in users, anonymous users or users with low trust level. Because, some day, AI crawlers will learn how to create accounts on the website and will be accessing system from logged in user accounts.
On the other hand, if we have separate permissions for the resource heavy actions we can assign them to specific groups for users who we trust.
-
Name Previous Value Current Value Type
Security Vulnerability
Improvement
-
I don't think this is a security vulnerability, as this is common for every system under heavy load. You either need to turn off the public access or implement rate-limited access by configuring the front end.
-
Yes, we are going to implement some kind of rate limitations on our front-end as a temp solution. We do not want to completely disable public access to our open source projects as it would be counterproductive. We are in the open source software business. However, as stated at the beginning, limiting public access to certain functionality would be an ideal solution.
-
-
-
Maybe you can add a UI to provide a robots.txt file so people can generally disable crawling of certain urls. Assuming that crawlers respect the file.
-
The default robots.txt already disallows accessing these pages.
-
AI scrappers ignore robots.txt
-
It is very cubersome to judge resource heavy actions. Even if it does, blocking some actions can affect other actions.
-
Previous Value Current Value Open
Closed
| Type |
Improvement
|
| Priority |
Normal
|
| Assignee | |
| Labels |
No labels
|
Hi,
W have a bunch of open source and public projects available for read-only to non-logged in users. That is to everybody.
We have recently started to encounter problems with something killing our OneDev server. It looks like some AI bots and scrappers and potentially other bots go through our OneDev system and click on everything that is possible to click. It seems like the most called function is
Code Comparewhich is called with every possible combination of commits and tags. This seems like putting a lot of load on our system until OneDev fails with OOM.For this reason it would be useful to be able to disable some functionality for anonymous users, at least this
Code Comparefunction. There might be some other functionality heavy on resources worth considering.Or maybe, you have other suggestions on how to deal with problem like this.