Wine repository clone's index weighing 40 GB #3117
Davide Beatrici opened 4 weeks ago

I just noticed the projects directory was occupying the entire disk space and the culprits turned out to be these:

3.8G    7
8.7G    25900
12.2G   25793
40.8G   5
 o_id  |  o_name   |           o_path           |
-------+-----------+----------------------------+
     5 | Wine      | WineHQ/Wine                | 
     7 | Wine      | davidebeatrici/WineHQ/Wine | 
 25793 | drakx     | Velanix/Software/drakx     | 
 25900 | linux     | torvalds/linux             | 

The index directory is the cause behind the huge size:

32.0K   attachment
32.0K   builds
4.0K    cache
484.5M  git
40.1G   index
183.9M  info
4.0K    site
4.0K    type
4.0K    uploads

More specifically:

224.5M  index/_4wh_Lucene84_0.doc
225.6M  index/_brw_Lucene84_0.doc
240.8M  index/_37m_Lucene84_0.doc
242.6M  index/_oq_Lucene84_0.doc
250.9M  index/_2ft_Lucene84_0.doc
268.6M  index/_1w3_Lucene84_0.doc
300.0M  index/_186_Lucene84_0.doc
517.7M  index/_6wz.cfs
1.0G    index/_c26.cfs
1.1G    index/_by8.cfs
1.1G    index/_c6m.cfs
4.4G    index/_186_Lucene84_0.pos
4.4G    index/_oq_Lucene84_0.pos
4.5G    index/_2ft_Lucene84_0.pos
4.6G    index/_1w3_Lucene84_0.pos
4.6G    index/_37m_Lucene84_0.pos
4.6G    index/_4wh_Lucene84_0.pos
4.6G    index/_brw_Lucene84_0.pos
  • Robin Shen commented 3 weeks ago

    OneDev performs full text index of commit of the repository when the commit is visited. In normal cases, the index should not be so large if the repository itself is not very large. I can test the repository if this is publicly available.

  • Robin Shen changed state to 'Closed' 3 weeks ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen changed state to 'Open' 3 weeks ago
    Previous Value Current Value
    Closed
    Open
  • Davide Beatrici commented 3 weeks ago

    Thank you!

    Wine repository: https://git.houseof.software/WineHQ/Wine Linux repository: https://git.houseof.software/torvalds/linux

    I kept them mostly for benchmarking purposes, since the creation of the OneDev instance last year.

    Did perhaps something change regarding the index in OneDev 16.6? Because the virtual disk I assigned to the container became full two days after updating from 16.5.8 to 16.6.1.

  • Robin Shen commented 3 weeks ago

    Nothing changed regarding repository indexing since 16.5.8. I checked Wine and the index on tip commit is only 450k. I guess your large index is because it indexed lots of history commits which may happen if someone or some UI automation tools visited many history commits triggering the index.

    Nevertheless, you may stop OneDev and remove the index folder. It will not hurt OneDev functionality, just that it will be re-indexed when someone visited some commit of that repo later.

  • Robin Shen changed state to 'Closed' 3 weeks ago
    Previous Value Current Value
    Open
    Closed
  • Davide Beatrici commented 3 weeks ago

    Makes perfect sense, thank you!

    Could you perhaps add a per-repository and/or global index size limit?

  • Robin Shen commented 3 weeks ago

    It is difficult to limit index size, but we can add settings to control whether or not create index, and what type of index to create: OD-3124

  • Davide Beatrici commented 3 weeks ago

    Sounds good!

1/1
Type
Bug
Priority
Normal
Assignee
Affected Versions
16.6.2
Labels
No labels
Issue Votes (0)
Watchers (2)
Reference
OD-3117
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover