jbauer
opened 2 years ago
-
OneDev
changed state to 'Closed' 2 years ago
Previous Value Current Value Open
Closed
-
State changed as code fixing the issue is committed (9135668a)
-
I guess I have to copy back the contents of
/opt/onedev/sites/program-backup/<date>to/opt/onedev/and then figure out a way how to call/opt/onedev/bin/restore-db.sh /opt/onedev/site/db-backup/2023-09-18_12-12-31.zipfrom a docker container with mounted onedev? -
Also note
Failed to delete file /opt/onedev/conf/trust-certswhich contains aca.crtfile for OneDev. Maybe that is a second issue -
Yes, please restore the program backup manually. Then you can start OneDev container (database may be empty now). Then exec into the container, and run "bin/restore-db.sh" to restore database.
-
But I would use the 9.1.5 container for it, right?
-
Yes, that is right.
-
OneDev
changed state to 'Released' 2 years ago
Previous Value Current Value Closed
Released
-
State changed as build #4106 is successful
-
Also note Failed to delete file /opt/onedev/conf/trust-certs which contains a ca.crt file for OneDev. Maybe that is a second issue
Is
/opt/onedev/conf/trust-certsa separate mount path? -
Also note Failed to delete file /opt/onedev/conf/trust-certs which contains a ca.crt file for OneDev. Maybe that is a second issue
Is
/opt/onedev/conf/trust-certsa separate mount path?Yes it is. Reason is that others (e.g. OneDev agent, other services) also need the
ca.crt. So I have a docker volume mounted with just theca.crt.- type: volume source: ca-cert target: /opt/onedev/conf/trust-certs - type: volume source: ca-cert target: /agent/conf/trust-certs -
@robin Executing the restore command in console of 9.1.5 container results in
INFO - Validating data file 'LinkSpecs.xml'... ERROR - Error booting application java.lang.RuntimeException: Error validating entity (entity class: class io.onedev.server.model.LinkSpec, entity id: 3, entity property: issueQuery, error message: Malformed query) at io.onedev.server.data.DefaultDataManager.reportError(DefaultDataManager.java:511) at io.onedev.server.data.DefaultDataManager.validateData(DefaultDataManager.java:498) at io.onedev.server.commandhandler.RestoreDatabase.doRestore(RestoreDatabase.java:108) at io.onedev.server.commandhandler.RestoreDatabase.lambda$start$0(RestoreDatabase.java:79) at io.onedev.server.commandhandler.CommandHandler.doMaintenance(CommandHandler.java:31) at io.onedev.server.commandhandler.RestoreDatabase.start(RestoreDatabase.java:72) at io.onedev.commons.loader.DefaultPluginManager.start(DefaultPluginManager.java:44) at io.onedev.commons.loader.AppLoader.start(AppLoader.java:60) at io.onedev.commons.bootstrap.Bootstrap.main(Bootstrap.java:198) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349) at java.base/java.lang.Thread.run(Thread.java:829) INFO - Stopping application...Any idea? Does it validate against the database tables? I have not deleted the db as I thought it is empty anyways.
-
The relevant XML of LinkSpecs.xml is
<io.onedev.server.model.LinkSpec revision="0.0"> <id>3</id> <name>Task</name> <multiple>true</multiple> <issueQuery>"Typ" is "Aufgabe"</issueQuery> <opposite> <name>Parent</name> <multiple>false</multiple> </opposite> <order>3</order> </io.onedev.server.model.LinkSpec> -
"Typ" is "Aufgabe"is invalid. Should be"Type" is "Aufgabe"instead. Please fix this manually and repack the zip file to restore -
"Typ" is "Aufgabe"is invalid. Should be"Type" is "Aufgabe"instead. Please fix this manually and repack the zip file to restoreIn my installation I have renamed
Typeto germanTyp. I do not have any issue field namedType, soTypis generally correct. -
That is odd. Nevertheless, you may run
restore-db.sh /path/to/backup.zip falseto bypass validation. After restore, you may check that query further. -
As an example taken from IssueFields.xml
<io.onedev.server.model.IssueField revision="0.0"> <id>744</id> <issue>74</issue> <name>Typ</name> <value>Verbesserung</value> <type>Enumeration</type> <ordinal>2</ordinal> </io.onedev.server.model.IssueField>That is basically translated from English
Type -> Improvementto GermanTyp -> Verbesserung. -
Seems like there is another bug here. Please remove file
/opt/onedev/maintenanceand then restart the container to see if it works. -
@robin Yes that file exists. After deleting it OneDev has started.
I also checked the issue links in OneDev UI and it seems totally correct. OneDev even suggests
"Typ" is "Aufgabe"via auto completion. There also some issues in the DB which use this issue link. So I guess there is some issue in the validation code. -
I tested locally and it works. Maybe there is some different charset? If you reproduce the issue with a sample backup, it will help me to address the issue.
-
There are multiple issues here when OneDev fails to ugprade. I will investigate them hoping to make the upgrade experience smoother, even for a failed upgrade.
-
Successfully upgrade from v9.1.2 to v9.1.10 but there a step which is slower than before.
-
Successfully upgrade from v9.1.2 to v9.1.10 but there a step which is slower than before.
No changes affecting performance as far as I know.
-
I tested locally and it works. Maybe there is some different charset? If you reproduce the issue with a sample backup, it will help me to address the issue.
@robin I have a git clone of onedev/server. I updated it to 9.1.5 and now I am wondering how I could execute
restore-db.shagainst it locally in IDE so I can debug it? Seems like the script does not exist in git? Do I need to generate it somehow? -
You may create a
Run Configurationin IntelliJ the same as running OneDev server, but with command paramrestore-db /path/to/backup.zip -
@robin Grrr just typed an answer and OneDev had reloaded because you updated it to 9.1.12 :)
So the issue with
restore-dbcommand/shell script is that the DB was empty. If the restore command validates a backup XML file that contains an issue query the code ends up inIssueQuery.checkField(fieldName, operator, options)and tries to load global issue properties from the DB. Because DB is emptygetGlobalIssueSetting()returnsnulland an NPE is thrown. That NPE is catched inIssueQueryValidator.isValid(value, constraintContext)however the stack trace is never logged. Instead the NPE message is checked and because there isn't any message it is replaced withMalformed queryand then converted to a misleading constraint violation.To me it looks like you can not restore a DB backup if that backup contains any XML file with some issue query. It does not need to be a
LinkSpec(which is my case). I think if you want to validate the backup you need to insert some data while validating. So maybe you need a dependency tree of your entities to validate them in correct order and insert validated data right away. Otherwise validating on empty DB doesn't really work. Or you skip validating issue queries containing custom field names. But then validation is only half-baked.There might be other validators that require settings from the DB.
-
To me it looks like you can not restore a DB backup if that backup contains any XML file with some issue query. It does not need to be a LinkSpec (which is my case). I think if you want to validate the backup you need to insert some data while validating. So maybe you need a dependency tree of your entities to validate them in correct order and insert validated data right away. Otherwise validating on empty DB doesn't really work. Or you skip validating issue queries containing custom field names. But then validation is only half-baked.
Thanks for the investigation.
Oh and while testing around I noticed that the .Bootstrap clean-db command does not delete all tables. After executing there were still o_clusterserver and o_databaselock tables in the database.
These two are special tables and will never change. It wont't be cleaned also
-
Data validation will be turned off completely for data restore. It does not provide much value as data file itself is exported from OneDev. And it causes much trouble instead.
| Type |
Bug
|
| Priority |
Critical
|
| Assignee | |
| Affected Versions |
9.1.5, 9.1.9
|
| Labels |
No labels
|
