Many approvals by Ai user on PR (OD-2733)

Many approvals by Ai user on PR #2733

pim opened 2 months ago

We have added an AI user to do PR reviews. I have one issue and one question:

Sometimes I see a long list of messages about approving the PR. Sometimes this ends with an error but not always. (See image)
Can an AI user write code-comments or can it only write normal comments? I tried to put it in the prompt but so far it has not made code-comments on a PR.

Prompt that we added to AI user:

You are an expert Senior Software Engineer reviewing a Pull Request. Your primary goal is to ensure code quality, performance, and security.
If you already approved the PR and there are no changes or new comments, stop trying to respond.

### INSTRUCTIONS:
1. FOCUS on logic flaws, security vulnerabilities, performance bottlenecks, and architectural issues.
2. IGNORE minor naming conventions, and auto-generated files (e.g., package-lock.json).
3. BE CONCISE. Do not summarize the PR unless explaining a broader architectural flaw. 
4. BE ACTIONABLE. If you identify an issue, provide a brief explanation of the problem and a specific code suggestion to fix it.
5. TONE: Professional, objective, and constructive.

### OUTPUT FORMAT:
If there are issues, format your response strictly using Markdown with the following sections:
- **Critical Issues:** (Bugs, security risks, crashes)
- **Suggestions:** (Performance improvements, refactoring)
- **Questions:** (Ambiguous logic that needs clarification)

Always include line numbers from the source when showing code.

If there are no issues that require changing, reply ONLY with: "LGTM: No critical issues found." and approve the pull request

pim commented 2 months ago

If I ask it to approve, a response from gemini looks like this:

"candidates": [
    {
      "content": {
        "parts": [
          {
            "text": "LGTM: No critical issues found."
          },
          {
            "functionCall": {
              "name": "approvePullRequest",
              "args": {}
            }
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "finishMessage": "Model generated function call(s)."
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 21081,
    "candidatesTokenCount": 18,
    "totalTokenCount": 21099,
    "cachedContentTokenCount": 17293,
    "promptTokensDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 21081
      }
    ],
    "cacheTokensDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 17293
      }
    ]
  },
  "turnToken": "v1_********************************"
}

It does this many times but I only asked it once.

Robin Shen commented 2 months ago

Which model are you using? As to code comments, it is not supported yet.
pim commented 2 months ago

We're currently using gemini-2.5-pro. Same effect was found on 2.5 flash.
Robin Shen commented 2 months ago

Just tested with 2.5 pro and things are working fine. Can you please create a demo project on this site with a demo PR?
Robin Shen commented 2 months ago

No any demo for now. It is possible that the approval action being called multiple times. Will add some guard condition on this.
pim commented 2 months ago

You don't need a demo project anymore?

Yes it does look like multiple calls very quickly.

I'll also include the prompt that we use (I forgot to add that sorry...)
Robin Shen commented 2 months ago

Yes, no need for it for now.

👍 1
OneDev commented 2 months ago

State changed as build OD-7256 (15.0.0) is successful
OneDev changed state to 'Closed' 2 months ago
Previous Value Current Value
Open

Closed
Login to comment

Previous Value	Current Value
Open	Closed

Type	Bug
Priority	Normal
Assignee	Robin Shen
Affected Versions	14.0.7
Labels	No labels

Issue Votes (0)

Watchers (3)

Reference

OD-2733