OneDev
Dashboards
Projects
Pull Requests
Issues
Builds
Packages
Code Search
server
Code
Pull Requests
Issues
List
Boards
Milestones
Builds
Packages
Statistics
Child Projects
OneDev 10.5.2
Documentation Support & Bug Report RESTful API Incompatibilities License Agreement Check Update
Projects onedev server Issues #1412
#1412  git project repository corrupt
Closed
bufferUnderrun opened 11 months ago

Hi,

one of my onedev instances stop few minutes after upgrade to 8.3.6. it refuses to start again.

see log


--> Wrapper Started as Console
Java Service Wrapper Standard Edition 64-bit 3.5.51
  Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping

Launching a JVM...
WrapperManager: Initializing...
09:09:40 INFO  i.onedev.commons.bootstrap.Bootstrap - Launching application from '/opt/onedev'...
09:09:40 INFO  i.onedev.commons.bootstrap.Bootstrap - Cleaning temp directory...
09:09:40 INFO  io.onedev.commons.loader.AppLoader - Starting application...
09:09:50 INFO  i.o.s.e.i.DefaultBuildParamManager - Caching build param info...
09:09:51 INFO  i.o.s.e.impl.DefaultIssueManager - Caching issue info...
09:09:51 INFO  i.o.s.e.i.DefaultBuildMetricManager - Caching build metric info...
09:09:51 INFO  i.o.s.e.impl.DefaultBuildManager - Caching build info...
09:09:52 INFO  i.o.s.e.impl.DefaultProjectManager - Checking projects...
09:09:55 ERROR i.onedev.commons.bootstrap.Bootstrap - Error booting application
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at io.onedev.server.event.Listener.notify(Listener.java:21)
        at io.onedev.server.event.DefaultListenerRegistry.invokeListeners(DefaultListenerRegistry.java:103)
        at io.onedev.server.event.DefaultListenerRegistry.post(DefaultListenerRegistry.java:153)
        at io.onedev.server.persistence.TransactionInterceptor$1.call(TransactionInterceptor.java:23)
        at io.onedev.server.persistence.DefaultTransactionManager.lambda$call$0(DefaultTransactionManager.java:69)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
        at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60)
        at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
        at io.onedev.server.OneDev.lambda$start$1(OneDev.java:147)
        at io.onedev.server.persistence.DefaultSessionManager.lambda$run$0(DefaultSessionManager.java:108)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
        at io.onedev.server.persistence.DefaultSessionManager.run(DefaultSessionManager.java:107)
        at io.onedev.server.OneDev.start(OneDev.java:147)
        at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44)
        at io.onedev.commons.loader.AppLoader.start(AppLoader.java:60)
        at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:196)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException: null
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.onedev.server.event.Listener.notify(Listener.java:19)
        ... 21 common frames omitted
Caused by: java.lang.RuntimeException: java.io.EOFException: Short read of block.
        at io.onedev.server.git.GitUtils.parseCommit(GitUtils.java:373)
        at io.onedev.server.git.GitUtils.getLastCommit(GitUtils.java:149)
        at io.onedev.server.entitymanager.impl.DefaultProjectManager.on(DefaultProjectManager.java:761)
        at io.onedev.server.persistence.TransactionInterceptor$1.call(TransactionInterceptor.java:23)
        at io.onedev.server.persistence.DefaultTransactionManager.lambda$call$0(DefaultTransactionManager.java:62)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
        at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60)
        at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
        ... 26 common frames omitted
Caused by: java.io.EOFException: Short read of block.
        at org.eclipse.jgit.util.IO.readFully(IO.java:203)
        at org.eclipse.jgit.internal.storage.file.UnpackedObject.open(UnpackedObject.java:72)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.getObjectLoader(LooseObjects.java:188)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.open(LooseObjects.java:149)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseObject(ObjectDirectory.java:396)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseFromSelfOrAlternate(ObjectDirectory.java:373)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:349)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:1069)
        at io.onedev.server.git.GitUtils.parseCommit(GitUtils.java:365)
        ... 33 common frames omitted
09:09:55 INFO  io.onedev.commons.loader.AppLoader - Stopping application...
<-- Wrapper Stopped
Robin Shen commented 11 months ago

Seems that git repository of some project is corrupted for some reason. If this error persists after restart, please upgrade to 8.3.7 (the upgrade should be working even if with this error) which will print the project being checked on start up by changing logger io.onedev.server to DEBUG in conf/logback.xml.

Then you run git fsck in git sub directory of problematic project to see if there is any findings.
bufferUnderrun commented 11 months ago

Found the culprit project the way you told.

The git repository was corrupt which i don't know how... first time it happens.

I had to rollback the backup i always do before any upgrade and everything is online on v8.3.7.

Thanks for you support
bufferUnderrun changed title 11 months ago
Previous Value Current Value 
instance down 2min after upgrade from 8.2.7 to 8.3.6
 
git project repository corrupt
bufferUnderrun changed state to 'Closed' 11 months ago
Previous Value Current Value 
Open
 
Closed
Robin Shen commented 11 months ago

This is odd. Have you examined with git fsck to see which part is corrupted?
bufferUnderrun commented 11 months ago

i'm not. After understand the fastest and safest was to rollback, i just did it.

onedev down a morning, get some pressure to getting things back to work (CI/CD)...
Robin Shen commented 11 months ago

OK. I will pay attention to the upgrade procedure to see if there is any possiblity of breaking things.
issue 1 of 1
Type
Bug
Priority
Critical
Assignee
Robin Shen
Affected Versions
8.3.6
Issue Votes (0)
Login to vote
Watchers (3)
Reference
onedev/server#1412
Please wait...
Page is in error, reload to recover