git project repository corrupt (OD-1412)

bufferUnderrun opened 3 years ago

Hi,

one of my onedev instances stop few minutes after upgrade to 8.3.6. it refuses to start again.

see log


--> Wrapper Started as Console
Java Service Wrapper Standard Edition 64-bit 3.5.51
  Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
    http://wrapper.tanukisoftware.com
  Licensed to OneDev for Service Wrapping

Launching a JVM...
WrapperManager: Initializing...
09:09:40 INFO  i.onedev.commons.bootstrap.Bootstrap - Launching application from '/opt/onedev'...
09:09:40 INFO  i.onedev.commons.bootstrap.Bootstrap - Cleaning temp directory...
09:09:40 INFO  io.onedev.commons.loader.AppLoader - Starting application...
09:09:50 INFO  i.o.s.e.i.DefaultBuildParamManager - Caching build param info...
09:09:51 INFO  i.o.s.e.impl.DefaultIssueManager - Caching issue info...
09:09:51 INFO  i.o.s.e.i.DefaultBuildMetricManager - Caching build metric info...
09:09:51 INFO  i.o.s.e.impl.DefaultBuildManager - Caching build info...
09:09:52 INFO  i.o.s.e.impl.DefaultProjectManager - Checking projects...
09:09:55 ERROR i.onedev.commons.bootstrap.Bootstrap - Error booting application
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at io.onedev.server.event.Listener.notify(Listener.java:21)
        at io.onedev.server.event.DefaultListenerRegistry.invokeListeners(DefaultListenerRegistry.java:103)
        at io.onedev.server.event.DefaultListenerRegistry.post(DefaultListenerRegistry.java:153)
        at io.onedev.server.persistence.TransactionInterceptor$1.call(TransactionInterceptor.java:23)
        at io.onedev.server.persistence.DefaultTransactionManager.lambda$call$0(DefaultTransactionManager.java:69)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
        at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60)
        at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
        at io.onedev.server.OneDev.lambda$start$1(OneDev.java:147)
        at io.onedev.server.persistence.DefaultSessionManager.lambda$run$0(DefaultSessionManager.java:108)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
        at io.onedev.server.persistence.DefaultSessionManager.run(DefaultSessionManager.java:107)
        at io.onedev.server.OneDev.start(OneDev.java:147)
        at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44)
        at io.onedev.commons.loader.AppLoader.start(AppLoader.java:60)
        at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:196)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException: null
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.onedev.server.event.Listener.notify(Listener.java:19)
        ... 21 common frames omitted
Caused by: java.lang.RuntimeException: java.io.EOFException: Short read of block.
        at io.onedev.server.git.GitUtils.parseCommit(GitUtils.java:373)
        at io.onedev.server.git.GitUtils.getLastCommit(GitUtils.java:149)
        at io.onedev.server.entitymanager.impl.DefaultProjectManager.on(DefaultProjectManager.java:761)
        at io.onedev.server.persistence.TransactionInterceptor$1.call(TransactionInterceptor.java:23)
        at io.onedev.server.persistence.DefaultTransactionManager.lambda$call$0(DefaultTransactionManager.java:62)
        at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
        at io.onedev.server.persistence.DefaultTransactionManager.call(DefaultTransactionManager.java:60)
        at io.onedev.server.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:18)
        ... 26 common frames omitted
Caused by: java.io.EOFException: Short read of block.
        at org.eclipse.jgit.util.IO.readFully(IO.java:203)
        at org.eclipse.jgit.internal.storage.file.UnpackedObject.open(UnpackedObject.java:72)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.getObjectLoader(LooseObjects.java:188)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.open(LooseObjects.java:149)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseObject(ObjectDirectory.java:396)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseFromSelfOrAlternate(ObjectDirectory.java:373)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:349)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:1069)
        at io.onedev.server.git.GitUtils.parseCommit(GitUtils.java:365)
        ... 33 common frames omitted
09:09:55 INFO  io.onedev.commons.loader.AppLoader - Stopping application...
<-- Wrapper Stopped

Activities

Robin Shen commented 3 years ago

Seems that git repository of some project is corrupted for some reason. If this error persists after restart, please upgrade to 8.3.7 (the upgrade should be working even if with this error) which will print the project being checked on start up by changing logger io.onedev.server to DEBUG in conf/logback.xml.

Then you run git fsck in git sub directory of problematic project to see if there is any findings.
bufferUnderrun commented 3 years ago

Found the culprit project the way you told.

The git repository was corrupt which i don't know how... first time it happens.

I had to rollback the backup i always do before any upgrade and everything is online on v8.3.7.

Thanks for you support

bufferUnderrun changed title 3 years ago

Previous Value	Current Value
instance down 2min after upgrade from 8.2.7 to 8.3.6	git project repository corrupt

bufferUnderrun changed state to 'Closed' 3 years ago
Previous Value Current Value
Open

Closed
Robin Shen commented 3 years ago

This is odd. Have you examined with git fsck to see which part is corrupted?
bufferUnderrun commented 3 years ago

i'm not. After understand the fastest and safest was to rollback, i just did it.

onedev down a morning, get some pressure to getting things back to work (CI/CD)...
Robin Shen commented 3 years ago

OK. I will pay attention to the upgrade procedure to see if there is any possiblity of breaking things.
Login to comment

Previous Value	Current Value
Open	Closed

Type	Bug
Priority	Critical
Assignee	Robin Shen
Affected Versions	8.3.6

Issue Votes (0)

Watchers (3)

Reference

OD-1412