Index trigger on Files view (viewing repo on website) can cause CPU spike, then OOM #2715
mc26paradox opened 3 months ago

(Title is an accurate summary, adjust Priority accordingly.)

With a particular, small folder of files uploaded, OneDev will show the following banner when viewing the relevant repository's files in a browser: Revision indexing in progress... (search in this revision will be accurate after indexed)

After doing so, OneDev will start maxing out a few cores until every thread hits OOM, with no other log entries.

Example output:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-24"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.127.0.0.1:5710.IO.BalancerThread"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-23"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.127.0.0.1:5710.MetricsRegistry.thread-1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.127.0.0.1:5710.scheduled.thread-"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandlr in thread "DefaultQuartzScheduler_QuartzSchedulerThread"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.127.0.0.1:5710.MetricsRegistry.thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Exodus shared timer thread"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "hz.127.0.0.1:5710.HealthMonitor"

As expected, OneDev remains locked up and mostly unresponsive until restart, with any functional feature breaking down, like new logins citing incorrect credentials, or listing other repo files in the browser showing a blank repo entirely. (You can imagine this gave me quite a spook at first, thinking the entire data directory nuked itself)

This is repeatable and consistent, thankfully, and I have the problematic files on hand if you'd like a copy for testing. (Triggering data has zero sensitive information, it's a local copy of some plugins for note-taking software, so they can be re-acquired directly.)

Full details of the setup and environment on discovery:

  • OneDev was accidentally 3 months outdated, exact version unknown other than said timestamp and version 13.x.x
    • Issue was replicated after successful upgrade to the currently latest version, 14.1.8
  • Issue does not trigger unless a user's browser navigates to Projects > (repo) > Code > Files
    • Interacting with the repo via git HTTPS push/pull works as expected. (before lockup, of course)
  • Tested client browser is Firefox v148, on Linux (Fedora 43)
  • Server is running via Podman on the official Docker container, 1dev/server
    • (Server container and client browser on the same machine)
    • CPU spike comes from server process in container, not the browser. I cannot measure the RAM spike via OneDev admin panel, as the lockup triggers within a second.
    • Server's settings are more-or-less default, including the container's default of 512 MB RAM. During normal use, OneDev reports RAM use at approx. 140 MB.
    • Only a few small repos exist on the server, site/ is only 53 MB total. (Entire data dir is 1.1 GB total)
  • Java version used is JDK 25 via GraalVM v37.1 on Linux x64, downloaded at either:
  • Issue occurs both by local HTTP access with an administrator account, and a normal user account on HTTPS and a public DNS address (looped back by ISP)
  • Removing the files in question stops the problem from triggering, and adding them back will make it possible to trigger again.
    • Viewing a previous commit where the files were added (Projects > (repo)> Code > Commits) is also a valid trigger.
  • The repo this was discovered on was brand new, made as child of a year-old active repo with 2 users, and only contains a single .md text file otherwise.

If you'd like the problematic files without downloading from a stranger, it's quick to create them:

  • Download and install Obsidian, a popular community-made lightweight note-taking software
  • Make a new Vault in a folder somewhere
  • Hit the gear ⚙️ icon in the bottom left
  • Pick "Community Plugins" from the left, accept the warning, and search for and install: (still downloaded if left disabled, the default)
  • The chosen folder now has a nearly-identical set of files, and Obsidian can be uninstalled from that point if you wish.
    • Note that only Obsidian is executed in this example, as far as "running mystery software" is considered

Triggering folder structure breakdown, via tree .obsidian/: (Total folder size is 1.0 MB)

.obsidian/
├── appearance.json
├── app.json
├── community-plugins.json
├── core-plugins.json
├── graph.json
├── hotkeys.json
├── plugins
│   ├── git-changelog
│   │   ├── data.json
│   │   ├── main.js
│   │   ├── manifest.json
│   │   └── styles.css
│   └── obsidian-git
│       ├── data.json
│       ├── main.js
│       ├── manifest.json
│       └── styles.css
└── workspace.json

4 directories, 15 files

Ignoring this directory via .gitignore, and hence keeping it off the repo, avoids triggering the issue for now, but I'd prefer to be able to upload it as all (private, authenticated) recipients should have these files, either via the repo or manually recreating them.

If I missed an important detail, or there's something you'd like me to check, feel free to ask.

  • Robin Shen commented 3 months ago

    I do not mind testing with your files. Please share them directly here.

  • mc26paradox commented 3 months ago

    Didn't notice at first that "Link" in the editor here included an upload option, i've double-checked that it's safe to export, and tested again as the files changed slightly.

    Confirmed the latest version of the files still triggers it, here's the original version: .obsidian.zip

  • Robin Shen commented 3 months ago

    Turns out that there is a huge js file (plugins/obsidian-git/main.js) causing the JavaScript symbol extractor eating too much memory. At my side, it works until I set heap size as 1G.

    I filed an improvement request for this and closing this one: OD-2716

  • Robin Shen changed state to 'Closed' 3 months ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen commented 3 months ago

    You may also configure project settings > code > code analysis to only analyze desired files. For instance, specifying files to be analyzed as -**/*.js there will prevent OneDev from analyzing all js files.

  • mc26paradox commented 3 months ago

    That'd actually work perfectly, since the parent of the project in question has zero JS in it, and the sub-project I found this on was pure text notes for that project, so there's no functional code anyways.

    Thank you for the fast find on this (a few hours!), OneDev's been improving faster than i've been keeping track of it, even when under limited resources like a Raspberry Pi 4, while staying surprisingly easy to use!

  • Robin Shen commented 3 months ago

    No problem. Thx for using the project. Any further feedbacks are very welcomed.

1/1
Type
Bug
Priority
Major
Assignee
Affected Versions
13.?.?, 14.1.8 (latest)
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
OD-2715
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover