Thousands of comments created after removing AI user entitlement permissions (OD-2894)

Thousands of comments created after removing AI user entitlement permissions #2894

Zak Siddiqui opened 3 days ago

I just encountered a really bad bug where thousands and thousands of comments were created after I removed permissions for an AI user entitlement to have access to all projects and be accessed by all users.

I got thousands of comments that did not stop appearing, that the AI user agent is not entitled.

My server hung, and I am currently rebooting it.

Zak Siddiqui changed title 3 days ago

Previous Value	Current Value
Thousands of comments created after removing AI user entit lement permissions	Thousands of comments created after removing AI user entitlement permissions

Service Desk commented 3 days ago
This is a real bug triggered by turning off Entitle to All Users and Projects while AI users still @mention each other.

Cause

With “entitle to all” disabled, AI users are no longer entitled to interact with each other unless you explicitly entitle them.

In a multi-AI setup (e.g. leadmechanic mentioning @leadreviewer, planners/mechanics mentioning other agents), this can loop:

AI-A posts a comment that @mentions AI-B

Entitlement fails → OneDev posts as AI-B: @AI-A you are not entitled to interact with me

That reply @mentions AI-A → AI-A posts the same message back at AI-B

The two AIs ping-pong without bound and flood the issue/PR until the server is overloaded

So the thousands of “not entitled” comments are an entitlement-check feedback loop between AI users, not a one-off denial.

Stop the flood / recover

Stop or reboot the server if it is still thrashing (as you did).

Before normal traffic returns, disable the AI users involved (or remove their ability to post / turn off proactive behavior) so the loop cannot resume.

Clean up spam comments on affected issues/PRs if needed (very large floods may need DB cleanup).

Re-enable AIs only after entitlements (below) are corrected.

Safer entitlement setup for multi-AI orchestration

With “entitle to all” off:

Entitle the human users/groups who should chat with / mention each AI.

Entitle the projects where each AI may be assigned as assignee/reviewer.

If AIs must @mention each other, also entitle those AI users to one another (user entitlement on each AI), or remove cross-AI @mentions from prompts/rules until a fix is available.

Until fixed, the safest pattern is: only humans @mention AIs; add AI reviewers/assignees via assignment rules or human action instead of AI-to-AI @mentions.

Product fix

@robin Bug in issue/PR notification handling: when entitlement fails for an AI mention, the “not entitled” reply @mentions the caller and re-enters mention processing. If the caller is another AI (common in multi-agent setups), this can recurse forever. The denial comment should not re-trigger AI mention handling (for example, do not @mention in that reply, or ignore mentions from AI authors / from these system denial comments).

@zaakiy Does this match what you saw after disabling “entitle to all”? If the recovery steps are enough for your instance, we can close this question; the loop itself still needs the product fix above.
Robin Shen changed fields 3 days ago
Name Previous Value Current Value
Type

Question

Bug

Assignee

servicedesk

robin

Affected Versions
empty
<=16.2.3
Zak Siddiqui commented 3 days ago

@robin the caller was another AI. I'm on the run right now, so I can't understand everything as I read it. I hope that's enough to help for now.

The other part of the problem is the cleanup. I initiated a script to clean up the comments by deleting the comments using the API. Well, it did delete the comments, but it just created another comment that's saying that the previous comment was deleted. So instead of 8,000 comments, I now have 8,000 deleted messages, which made my browser crash! So it would be nice to be able to have these be cleaned up as well, not only when the bug happens but also retroactively for circumstances like mine when I upgrade to the fixed version.
Zak Siddiqui commented 3 days ago

@servicedesk need better recovery steps for the following part of the recovery:

The other part of the problem is the cleanup. I initiated a script to clean up the comments by deleting the comments using the API. Well, it did delete the comments, but it just created another comment that's saying that the previous comment was deleted. So instead of 8,000 comments, I now have 8,000 deleted messages, which made my browser crash! So it would be nice to be able to have these be cleaned up as well, not only when the bug happens but also retroactively for circumstances like mine when I upgrade to the fixed version.

History
Service Desk commented 3 days ago
@zaakiy You're right — deleting comments via the REST API/UI always creates a “removed comment” activity on the issue/PR timeline. That is why ~8k deletes became ~8k deletion notices and crashed the browser. There is currently no API to purge those activities.

Do not use the comment API for mass cleanup

Use direct SQL with OneDev stopped. API/UI delete will keep adding “removed comment” rows.

Recovery for your current state (comments already deleted)

Stop OneDev (keep the involved AI users disabled before you start it again).

Backup the database first: https://docs.onedev.io/administration-guide/backup-restore

Delete the “removed comment” activities (pick the variant for your DB):

PostgreSQL:

DELETE FROM o_IssueChange WHERE position(convert_to('IssueCommentRemoveData', 'UTF8') IN o_data) > 0; DELETE FROM o_PullRequestChange WHERE position(convert_to('PullRequestCommentRemovedData', 'UTF8') IN o_data) > 0;

MySQL / MariaDB:

DELETE FROM o_IssueChange WHERE o_data LIKE CONCAT('%', 'IssueCommentRemoveData', '%'); DELETE FROM o_PullRequestChange WHERE o_data LIKE CONCAT('%', 'PullRequestCommentRemovedData', '%');

Start OneDev and reopen the affected issues/PRs — the activity flood should be gone.

If any “not entitled” spam comments are still present

Delete those with SQL too (not the API), then recalculate comment counts:

-- preview first SELECT COUNT(*) FROM o_IssueComment WHERE o_content LIKE '%you are not entitled to interact with me%'; DELETE FROM o_IssueCommentReaction WHERE o_comment_id IN ( SELECT o_id FROM o_IssueComment WHERE o_content LIKE '%you are not entitled to interact with me%' ); DELETE FROM o_IssueCommentRevision WHERE o_comment_id IN ( SELECT o_id FROM o_IssueComment WHERE o_content LIKE '%you are not entitled to interact with me%' ); DELETE FROM o_IssueComment WHERE o_content LIKE '%you are not entitled to interact with me%'; UPDATE o_Issue i SET o_commentCount = ( SELECT COUNT(*) FROM o_IssueComment c WHERE c.o_issue_id = i.o_id );

(Same idea for o_PullRequestComment* / o_PullRequest.o_commentCount if PRs were affected.)

Product follow-up

@robin Please fold into the product fix: (1) entitlement denial must not re-enter mention handling, and (2) a safe way to purge spam comments/activities without leaving “removed comment” noise — ideally also a one-shot cleanup usable on upgrade for instances already flooded.

@zaakiy If these SQL steps clear your instance, reply here. If the UI is still broken after the deletes, tell us which database you use and we can refine the statements.

Name	Previous Value	Current Value
Type	Question	Bug
Assignee	servicedesk	robin
Affected Versions	empty	<=16.2.3

Zak Siddiqui commented 3 days ago

@servicedesk I'm using a native data store and I'm not sure if it's my, mySQL or Postgres or something else.

I do not have any database configuration, so I believe it's part of the container.

version: '3.8'

services:
  onedev:
    image: 1dev/server:latest
    container_name: onedev
    restart: unless-stopped
    ports:
      - "6610:6610"
      - "6611:6611"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./onedev_data:/opt/onedev
    networks:
      - onedev-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "2"

networks:
  onedev-network:
    driver: bridge

When i tried looking into it (the data store), it said it had something starting with H.

Service Desk commented 3 days ago

@zaakiy That “something starting with H” is HSQLDB (HyperSQL) — OneDev’s default embedded database. Your Compose file has no external DB settings and mounts ./onedev_data to /opt/onedev, so the DB files live at:

./onedev_data/internaldb/

(inside the container: /opt/onedev/internaldb/)

The earlier MySQL/Postgres SQL still applies in spirit; below is the Docker + HSQLDB variant.

1. Stop OneDev and back up the volume

Keep the involved AI users disabled. Then on the host (directory with your docker-compose.yml):

docker compose stop

# full volume backup (important)
cp -a ./onedev_data "./onedev_data.bak-$(date +%Y%m%d%H%M%S)"

2. Delete the “removed comment” activities via HSQLDB

Run SqlTool against the stopped data directory (do not start OneDev while this runs):

docker run --rm \
  -v "$(pwd)/onedev_data:/opt/onedev" \
  --entrypoint bash \
  1dev/server:latest \
  -c '
set -e
JAR=$(find /opt/onedev /app -name "hsqldb*.jar" | head -1)
echo "Using jar: $JAR"
test -n "$JAR"
java -cp "$JAR" org.hsqldb.cmdline.SqlTool --inlineRc="url=jdbc:hsqldb:file:/opt/onedev/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" <<'\''SQL'\''
-- preview counts first
SELECT COUNT(*) AS issue_removed_comment_activities
FROM o_IssueChange
WHERE POSITION(CAST('\''IssueCommentRemoveData'\'' AS VARBINARY(64)) IN o_data) > 0;

SELECT COUNT(*) AS pr_removed_comment_activities
FROM o_PullRequestChange
WHERE POSITION(CAST('\''PullRequestCommentRemovedData'\'' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_IssueChange
WHERE POSITION(CAST('\''IssueCommentRemoveData'\'' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_PullRequestChange
WHERE POSITION(CAST('\''PullRequestCommentRemovedData'\'' AS VARBINARY(64)) IN o_data) > 0;

COMMIT;
SHUTDOWN;
SQL
'

3. If “not entitled” spam comments are still present

Still with OneDev stopped, run the same docker run ... SqlTool pattern with:

SELECT COUNT(*) FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

DELETE FROM o_IssueCommentReaction
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueCommentRevision
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

UPDATE o_Issue i
SET o_commentCount = (
  SELECT COUNT(*) FROM o_IssueComment c WHERE c.o_issue_id = i.o_id
);

COMMIT;
SHUTDOWN;

(Same idea for o_PullRequestComment* / o_PullRequest.o_commentCount if PRs were hit.)

4. Start again

docker compose start

Reopen the affected issues/PRs — the deletion-notice flood should be gone.

Notes

Do not mass-delete via the REST API/UI; that recreates “removed comment” activities.
Official backup/restore docs (also usable later via /opt/onedev/bin/backup-db.sh inside the container): https://docs.onedev.io/administration-guide/backup-restore
For production, consider switching from embedded HSQLDB to PostgreSQL/MySQL when you can: https://docs.onedev.io/installation-guide/run-on-bare-metal

@zaakiy After step 4, does the issue/PR timeline load again without crashing? If SqlTool cannot find hsqldb*.jar or a statement errors, paste the exact command output and we will adjust.

Service Desk commented 3 days ago

@zaakiy Correction: the SqlTool block in my previous comment had broken quoting from markdown escaping. Use this copy-pasteable version instead.

Delete “removed comment” activities (Docker + HSQLDB)

With OneDev stopped and ./onedev_data already backed up:

docker run --rm \
  -v "$(pwd)/onedev_data:/opt/onedev" \
  --entrypoint bash \
  1dev/server:latest \
  -c "set -e
JAR=\$(find /opt/onedev /app -name 'hsqldb*.jar' | head -1)
echo Using jar: \$JAR
test -n \"\$JAR\"
java -cp \"\$JAR\" org.hsqldb.cmdline.SqlTool --inlineRc='url=jdbc:hsqldb:file:/opt/onedev/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=' <<'SQL'
SELECT COUNT(*) AS issue_removed_comment_activities
FROM o_IssueChange
WHERE POSITION(CAST('IssueCommentRemoveData' AS VARBINARY(64)) IN o_data) > 0;

SELECT COUNT(*) AS pr_removed_comment_activities
FROM o_PullRequestChange
WHERE POSITION(CAST('PullRequestCommentRemovedData' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_IssueChange
WHERE POSITION(CAST('IssueCommentRemoveData' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_PullRequestChange
WHERE POSITION(CAST('PullRequestCommentRemovedData' AS VARBINARY(64)) IN o_data) > 0;

COMMIT;
SHUTDOWN;
SQL"

For leftover “not entitled” comments, use the same wrapper and replace the SQL body with the statements from step 3 in the previous comment (the SQL itself was fine).

Please reply with whether the timeline loads cleanly after this, or with any command error output.

Zak Siddiqui commented 3 days ago

@servicedesk that did not work at all.

The comments in question and the issue comment removed messages persist. details below.

docker compose down

󰕈 ubuntu@devmachine  ~/onedev   main   18:49:13 
❯ docker run --rm \
  -v "$(pwd)/onedev_data:/opt/onedev" \
  --entrypoint bash \
  1dev/server:latest \
  -c '
set -e
JAR=$(find /opt/onedev /app -name "hsqldb*.jar" | head -1)
echo "Using jar: $JAR"
test -n "$JAR"
java -cp "$JAR" org.hsqldb.cmdline.SqlTool --inlineRc="url=jdbc:hsqldb:file:/opt/onedev/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" <<'\''SQL'\''
-- preview counts first
SELECT COUNT(*) AS issue_removed_comment_activities
FROM o_IssueChange
WHERE POSITION(CAST('\''IssueCommentRemoveData'\'' AS VARBINARY(64)) IN o_data) > 0;

SELECT COUNT(*) AS pr_removed_comment_activities
FROM o_PullRequestChange
WHERE POSITION(CAST('\''PullRequestCommentRemovedData'\'' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_IssueChange
WHERE POSITION(CAST('\''IssueCommentRemoveData'\'' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_PullRequestChange
WHERE POSITION(CAST('\''PullRequestCommentRemovedData'\'' AS VARBINARY(64)) IN o_data) > 0;

COMMIT;
SHUTDOWN;
SQL
'

Using jar:

󰕈 ubuntu@devmachine  ~/onedev   main   18:49:30   in 6s50ms
❯ docker run --rm \
  -v "$(pwd)/onedev_data:/opt/onedev" \
  --entrypoint bash \
  1dev/server:latest \
  -c '
set -e
JAR=$(find /opt/onedev /app -name "hsqldb*.jar" | head -1)
echo "Using jar: $JAR"
test -n "$JAR"
java -cp "$JAR" org.hsqldb.cmdline.SqlTool --inlineRc="url=jdbc:hsqldb:file:/opt/onedev/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" <<'\''SQL'\''
SELECT COUNT(*) FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

DELETE FROM o_IssueCommentReaction
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueCommentRevision
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

UPDATE o_Issue i
SET o_commentCount = (
  SELECT COUNT(*) FROM o_IssueComment c WHERE c.o_issue_id = i.o_id
);

COMMIT;
SHUTDOWN;
SQL
'
Using jar:

docker compose up -d

Service Desk commented 3 days ago

@zaakiy Thanks for the exact output — that explains it. No SQL ran at all.

Why it failed

Using jar: was empty because the driver is named org.hsqldb.hsqldb-2.7.1.jar, not hsqldb*.jar. The find … -name "hsqldb*.jar" pattern never matched, then set -e exited before SqlTool started.
Even with the right jar, SqlTool is not inside org.hsqldb.hsqldb-*.jar (HSQLDB 2.7 split it into a separate sqltool artifact). The previous steps would have failed on the next line anyway.

So the “not entitled” rows and “removed comment” activities were never deleted.

Corrected recovery (Docker + embedded HSQLDB)

Run these from the directory that contains your docker-compose.yml and ./onedev_data. Keep the involved AI users disabled.

0. Stop and back up

docker compose down

cp -a ./onedev_data "./onedev_data.bak-$(date +%Y%m%d%H%M%S)"

1. Download SqlTool on the host (once)

curl -fsSL -o /tmp/sqltool-2.7.1.jar \
  https://repo1.maven.org/maven2/org/hsqldb/sqltool/2.7.1/sqltool-2.7.1.jar

2. Write the activity-cleanup SQL

cat > /tmp/onedev-cleanup-activities.sql <<'SQL'
SELECT COUNT(*) AS issue_removed_comment_activities
FROM o_IssueChange
WHERE POSITION(CAST('IssueCommentRemoveData' AS VARBINARY(64)) IN o_data) > 0;

SELECT COUNT(*) AS pr_removed_comment_activities
FROM o_PullRequestChange
WHERE POSITION(CAST('PullRequestCommentRemovedData' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_IssueChange
WHERE POSITION(CAST('IssueCommentRemoveData' AS VARBINARY(64)) IN o_data) > 0;

DELETE FROM o_PullRequestChange
WHERE POSITION(CAST('PullRequestCommentRemovedData' AS VARBINARY(64)) IN o_data) > 0;

COMMIT;
SHUTDOWN;
SQL

3. Run it (mount data at `/data`, keep image `/app` intact)

docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-activities.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
echo "Using HSQLDB jar: $HSQL"
test -n "$HSQL"
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

You should see non-zero preview counts (likely thousands), then the deletes. If both counts print 0, stop and paste the full output — do not proceed to step 4 yet.

4. Optional: purge leftover “not entitled” comments

Only if those comment bodies are still present (many were already removed via API):

cat > /tmp/onedev-cleanup-spam-comments.sql <<'SQL'
SELECT COUNT(*) AS issue_not_entitled_comments
FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

DELETE FROM o_IssueCommentReaction
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueCommentRevision
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

UPDATE o_Issue i
SET o_commentCount = (
  SELECT COUNT(*) FROM o_IssueComment c WHERE c.o_issue_id = i.o_id
);

COMMIT;
SHUTDOWN;
SQL

docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-spam-comments.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

(Same idea for o_PullRequestComment* / o_PullRequest.o_commentCount if PRs were hit.)

5. Start again

docker compose up -d

Reopen the affected issues/PRs and check whether the timeline loads without crashing.

Please paste the full output of step 3 (especially the two COUNT(*) lines and any Java/SqlTool errors). If the jar path or JDBC URL is wrong we can adjust immediately from that.

Robin Shen commented 3 days ago

@zaakiy You may delete all issuechange entries created today.
Robin Shen commented 3 days ago

Do not try to inspect the changedata as it is serialized java data and is difficult to match via sql command

Zak Siddiqui commented 3 days ago

@servicedesk

❯ docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-activities.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
echo "Using HSQLDB jar: $HSQL"
test -n "$HSQL"
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

Using HSQLDB jar: /app/lib/org.hsqldb.hsqldb-2.7.1.jar
Jul 23, 2026 9:03:56 AM org.hsqldb.cmdline.SqlTool objectMain
SEVERE: SQL Error at 'file:/cleanup.sql' line 3:
"SELECT COUNT(*) AS issue_removed_comment_activities
FROM o_IssueChange
WHERE POSITION(CAST('IssueCommentRemoveData' AS VARBINARY(64)) IN o_data) > 0"
data exception: invalid character value for cast
org.hsqldb.cmdline.SqlTool$SqlToolException

Service Desk commented 3 days ago

@zaakiy Updating cleanup per @robin: do not filter on o_data / change type. That column is serialized Java and is hard to match in SQL. Delete all IssueChange rows created today instead.

Corrected cleanup (Docker + HSQLDB)

Keep the involved AI users disabled. Run from the directory with your docker-compose.yml and ./onedev_data.

0. Stop and back up

docker compose down

cp -a ./onedev_data "./onedev_data.bak-$(date +%Y%m%d%H%M%S)"

1. Download SqlTool (once)

curl -fsSL -o /tmp/sqltool-2.7.1.jar \
  https://repo1.maven.org/maven2/org/hsqldb/sqltool/2.7.1/sqltool-2.7.1.jar

2. Delete today’s issue change activities

cat > /tmp/onedev-cleanup-activities.sql <<'SQL'
SELECT COUNT(*) AS issue_changes_today
FROM o_IssueChange
WHERE CAST(o_date AS DATE) = CURRENT_DATE;

DELETE FROM o_IssueChange
WHERE CAST(o_date AS DATE) = CURRENT_DATE;

COMMIT;
SHUTDOWN;
SQL

docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-activities.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
echo "Using HSQLDB jar: $HSQL"
test -n "$HSQL"
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

Expect a large non-zero count. This removes all issue timeline changes from today (not only “removed comment”), which is the intended recovery approach.

3. Optional: leftover “not entitled” comments

Only if those comment bodies are still present:

cat > /tmp/onedev-cleanup-spam-comments.sql <<'SQL'
SELECT COUNT(*) AS issue_not_entitled_comments
FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

DELETE FROM o_IssueCommentReaction
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueCommentRevision
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

UPDATE o_Issue i
SET o_commentCount = (
  SELECT COUNT(*) FROM o_IssueComment c WHERE c.o_issue_id = i.o_id
);

COMMIT;
SHUTDOWN;
SQL

docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-spam-comments.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

4. Start again

docker compose up -d

Please paste the full output of step 2 (especially the COUNT(*) line). If the timeline still crashes after that, reply with the error details.

Zak Siddiqui commented 3 days ago

Thanks @robin

󰕈 ubuntu@devmachine  ~/onedev   main   19:05:31 
❯ cat > /tmp/onedev-cleanup-activities.sql <<'SQL'
SELECT COUNT(*) AS issue_changes_today
FROM o_IssueChange
WHERE CAST(o_date AS DATE) = CURRENT_DATE;

DELETE FROM o_IssueChange
WHERE CAST(o_date AS DATE) = CURRENT_DATE;

COMMIT;
SHUTDOWN;
SQL

docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-activities.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
echo "Using HSQLDB jar: $HSQL"
test -n "$HSQL"
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

Using HSQLDB jar: /app/lib/org.hsqldb.hsqldb-2.7.1.jar
               8442

󰕈 ubuntu@devmachine  ~/onedev   main   19:05:43   in 2s692ms
❯ cat > /tmp/onedev-cleanup-spam-comments.sql <<'SQL'
SELECT COUNT(*) AS issue_not_entitled_comments
FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

DELETE FROM o_IssueCommentReaction
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueCommentRevision
WHERE o_comment_id IN (
  SELECT o_id FROM o_IssueComment
  WHERE o_content LIKE '%you are not entitled to interact with me%'
);

DELETE FROM o_IssueComment
WHERE o_content LIKE '%you are not entitled to interact with me%';

UPDATE o_Issue i
SET o_commentCount = (
  SELECT COUNT(*) FROM o_IssueComment c WHERE c.o_issue_id = i.o_id
);

COMMIT;
SHUTDOWN;
SQL

docker run --rm \
  -v "$(pwd)/onedev_data:/data" \
  -v /tmp/sqltool-2.7.1.jar:/sqltool.jar:ro \
  -v /tmp/onedev-cleanup-spam-comments.sql:/cleanup.sql:ro \
  --entrypoint bash \
  1dev/server:latest \
  -c 'set -e
HSQL=$(ls /app/lib/org.hsqldb.hsqldb-*.jar | head -1)
java -cp "$HSQL:/sqltool.jar" org.hsqldb.cmdline.SqlTool \
  --inlineRc="url=jdbc:hsqldb:file:/data/internaldb/onedev;hsqldb.tx=mvcc;hsqldb.lob_compressed=true;hsqldb.lob_file_scale=4;sql.ignore_case=true;shutdown=true,user=sa,password=" \
  /cleanup.sql'

                      11447

CC @servicedesk

Service Desk commented 3 days ago
@zaakiy Those counts look correct for the flood:

8442 today’s IssueChange rows removed (the timeline / “removed comment” spam)

11447 leftover “not entitled” comments removed

If SqlTool printed the count and then returned to the shell with no error, the DELETEs finished as well (SqlTool only prints SELECT results by default).

Next

Start OneDev again:

docker compose up -d

Keep the involved AI users disabled until entitlements are corrected (or the product fix ships).

Reopen the affected issues and confirm the timeline loads without crashing the browser.

If anything still fails, paste the error / what you see. If the UI is healthy again, reply here so we know recovery is done on your side — the AI entitlement ping-pong loop itself remains open as the product bug.
Zak Siddiqui commented 3 days ago

@servicedesk I can confirm that everything is working perfectly normal now and that all of the thousands of messages have disappeared.

@robin for clarity, please delete comment https://code.onedev.io/onedev/server/~issues/2894#IssueComment-10160
Service Desk commented 3 days ago

Glad the instance is healthy again and the flood is gone — no further recovery steps needed on your side.

@robin Please delete https://code.onedev.io/onedev/server/~issues/2894#IssueComment-10160 as requested (stale recovery guidance posted after cleanup had already succeeded). The AI entitlement ping-pong loop remains the open product bug on this issue.
Robin Shen removed comment 3 days ago
OneDev commented 2 days ago

State changed as build OD-7846 is successful
OneDev changed state to 'Closed' 2 days ago
Previous Value Current Value
Open

Closed
Login to comment

Previous Value	Current Value
Open	Closed

Type	Bug
Priority	Normal
Assignee	Robin Shen
Affected Versions	<=16.2.3
Labels	No labels

Issue Votes (0)

Watchers (3)

Reference

OD-2894

Cause

Stop the flood / recover

Safer entitlement setup for multi-AI orchestration

Product fix

Do not use the comment API for mass cleanup

Recovery for your current state (comments already deleted)

If any “not entitled” spam comments are still present

Product follow-up

1. Stop OneDev and back up the volume

2. Delete the “removed comment” activities via HSQLDB

3. If “not entitled” spam comments are still present

4. Start again

Notes

Delete “removed comment” activities (Docker + HSQLDB)

Why it failed

Corrected recovery (Docker + embedded HSQLDB)

0. Stop and back up

1. Download SqlTool on the host (once)

2. Write the activity-cleanup SQL

3. Run it (mount data at /data, keep image /app intact)

4. Optional: purge leftover “not entitled” comments

5. Start again

Corrected cleanup (Docker + HSQLDB)

0. Stop and back up

1. Download SqlTool (once)

2. Delete today’s issue change activities

3. Optional: leftover “not entitled” comments

4. Start again

Next

3. Run it (mount data at `/data`, keep image `/app` intact)