Enforce strict Markdown entity reference formatting in skills and tod comment posts #11
Service Desk opened 2 days ago

Summary

Follow-up from issue #10 (and issue #9). Strengthen Markdown entity reference guidance so agents stop emitting wrappers such as pr (path/to/project#123), and add a tod pre-post check for malformed references.

Decisions (from issue #10)

  1. Packaging (1B): Full regex/anti-pattern detail lives only in using-tod. submit-issue-work and submit-pull-request-work keep a short rule plus a pointer to that section.
    • Prefer combining the six regexes into fewer patterns when the combined form is more compact; do not combine if that increases complexity or token cost.
  2. Tests (2A): Commit golden valid/invalid examples under skills/ (or similar), reviewed in the implementation PR.
  3. Enforcement (3B): Add a tod pre-post check that warns or rejects malformed entity references in comments.
  4. Skill scope (4A): Only skills that already have ## Markdown entity references: using-tod, submit-issue-work, submit-pull-request-work.

Implementation

  1. Expand ## Markdown entity references in using-tod with compact valid patterns (case-insensitive), anti-patterns (no wrapping () / [], no : after type), and examples.
  2. Keep the matching sections in submit-issue-work and submit-pull-request-work short; point to using-tod for the full rule set.
  3. Add golden valid/invalid fixtures under skills/.
  4. Implement tod validation on comment-posting paths (warn or reject malformed typed references).

Acceptance

  •  using-tod documents compact patterns and anti-patterns; submit skills stay short with a pointer
  •  Golden examples committed under skills/
  •  tod warns or rejects malformed Markdown entity references when posting comments
  •  Content stays compact for AI token cost
  • Zak Siddiqui commented 2 days ago

    📌 Context & Problem Statement

    When the AI agent (TOD) submits work, it frequently fails to format references to issues, pull requests, or builds correctly.

    For example, TOD recently generated output like:

    pr (path/to/project#123)

    Because TOD wrapped the reference in parentheses, downstream parsers misattribute or fail to render the link properly (e.g., misidentifying a pull request as an issue due to the extra syntax).

    We need a stronger method within all relevant TOD skills under the ## Markdown entity references section to ensure AI models strictly adhere to the required syntax without adding extra brackets, parentheses, or unwanted punctuation.


    🎯 Target Specification

    The ## Markdown entity references section in all relevant TOD skills must explicitly define and enforce the following formatting rules:

    Write <type> <reference>, where type is issue, pr (or pull request), or build. Use #123 in the same project, path/to/project#123 across projects, or PROJ-123 when the project has a key.

    Examples: issue #123, pr path/to/project#123, and build PROJ-123.


    🚫 Explicit Anti-Patterns (To Add to Skill System Prompts)

    Include a negative example table directly in skill prompts so TOD explicitly avoids these formatting errors:

    ❌ Invalid OutputReason✅ Correct Output
    pr (path/to/project#123)Do not wrap reference in parenthesespr path/to/project#123
    issue [#123]Do not wrap reference in bracketsissue #123
    build: PROJ-123Do not add colons after the entity typebuild PROJ-123
    pull request (PROJ-123)Do not wrap key in parenthesespull request PROJ-123

    ✅ Acceptance Criteria

    •  Section ## Markdown entity references in all relevant TOD skills is updated with the exact specification text and the 6 regex definitions.
    •  Prompts explicitly instruct TOD to avoid wrapping references in parentheses (), brackets [], or using colons :.
    •  Test cases are added to verify TOD generates properly formatted references across issues, PRs, and builds.
    •  Written in a compact form so as to minimize AI token usage: The audience is AI, not human.
  • Zak Siddiqui commented 2 days ago

    @robin @servicedesk please hold off while I validate the regex. There are some errors that I need to fix.

  • Service Desk commented 2 days ago

    @zaakiy Understood — holding off on implementation until you've validated and fixed the regex. Ping me when the patterns are ready.

  • Zak Siddiqui commented 2 days ago

    Implementation Specs

    Here is a table built strictly from the single regex.

    I've verified every example against the pattern, including lowercase proj-123 which should NOT match.

    Single regex (replaces all six):

    ^(?i:(?:issue|pr|pull request|build) )?(?:(?:[\w.-]+\/)*[\w.-]*#|\b[A-Z]+\b-)\d+$
    

    Compliance

    Entity TypeReference StyleValid ExamplesMatches the Regex?
    1. IssuePath / Same Projectissue #123
    issue path/to/project#123
    ✅ Yes — Match 1 (0–10), Match 2 (11–36)
    2. IssueProject Keyissue PROJ-123✅ Yes (uppercase key) — Match 3 (37–51)
    ❌ No for lowercase keys (e.g., proj-123)
    3. PRPath / Same Projectpr #123
    pr path/to/project#123
    ✅ Yes — Match 5 (70–77), Match 6 (78–100); also pull request #123Match 7 (101–118)
    4. PRProject Keypr PROJ-123✅ Yes (uppercase key) — Match 9 (142–153); pull request PROJ-123Match 10 (154–175)
    ❌ No for lowercase keys
    5. BuildPath / Same Projectbuild #123
    build path/to/project#123
    ✅ Yes — Match 11 (176–186), Match 12 (187–212)
    6. BuildProject Keybuild PROJ-123✅ Yes (uppercase key) — Match 13 (213–227)
    ❌ No for lowercase keys

    Example-by-example verification

    #ExampleCorresponds toMatch result
    1issue #123Entity Type 1✅ Match 1 (0–10)
    2issue path/to/project#123Entity Type 1✅ Match 2 (11–36)
    3issue PROJ-123Entity Type 2✅ Match 3 (37–51)
    4PROJ-123Entity Type 2 (bare, no prefix)✅ Match 4 (52–60)
    5proj-123Entity Type 2 (lowercase key)❌ No match
    6pr #123Entity Type 3✅ Match 5 (70–77)
    7pr path/to/project#123Entity Type 3✅ Match 6 (78–100)
    8pull request #123Entity Type 3✅ Match 7 (101–118)
    9pr path/to/project#123Entity Type 3✅ Match 8 (119–141)
    10pr PROJ-123Entity Type 4✅ Match 9 (142–153)
    11pull request PROJ-123Entity Type 4✅ Match 10 (154–175)
    12build #123Entity Type 5✅ Match 11 (176–186)
    13build path/to/project#123Entity Type 5✅ Match 12 (187–212)
    14build PROJ-123Entity Type 6✅ Match 13 (213–227)

    Notes

    1. Lowercase project keys are intentionally rejected. The project-key branch \b[A-Z]+\b- is case-sensitive; the (?i:…) case-insensitivity applies only to the entity prefix (issue|pr|pull request|build). So proj-123 fails while PROJ-123 passes — confirmed by match info (13 matches, no match for the lowercase line).
    2. The prefix remains optional. Bare references like PROJ-123 are valid (Match 4); the table examples keep the prefixed forms to stay consistent with the original table.
    3. Behavior change vs. the old six-pattern set: the old pattern matched only the # style, so issue PROJ-123, pr PROJ-123, and build PROJ-123 would not have matched it. The single regex now covers the project-key style for all three entity types.

    Instructions

    @servicedesk all of the above information is for your own context, and for documentation it would be great to document all of these examples within the TOD project. However, with regards to the actual changes that need to be made that AI agents will see, this should be minimal in order to minimize token usage.

  • Service Desk commented 2 days ago

    @zaakiy Thanks — noted the single regex and the direction to keep agent-facing skill text minimal while documenting the full examples in the TOD project.

    @robin Blocked on implementation: tod issue checkout --for-write 11 returned HTTP 403 (no permission to write code in this project). Please take over from here.

  • Zak Siddiqui commented 2 days ago

    @servicedesk please note that if you ever write any regex into a comment in an issue or pull request related to this issue, ensure that the regex is in a code block and not a code span. Placing the regex into a code span will break the rendering in Markdown.

  • Service Desk commented 2 days ago

    Noted — when posting regex related to this issue or related pull requests, I'll put the pattern in a fenced code block rather than an inline code span so Markdown rendering stays intact.

  • Robin Shen commented 1 day ago

    For example, TOD recently generated output like: pr (path/to/project#123)

    @zaakiy What AI model are you using? The working on issue/PR and submit issue/PR skill already has instructions for reference format:

    https://code.onedev.io/onedev/tod/~files/ed511a6044f1f0077d5bf2d69260f035f2729dd6/skills/work-on-issue/SKILL.md?position=source-162.49-166.18-1&mode=blame

    I used Grok 4.5 High and GPT 4.6 Sol Medium and they never take it wrong.

    I am afraid putting exact regex patterns in the skill will not make the situation better since LLM generally prefers examples over formal syntaxes...

1/1
Type
Improvement
Priority
Normal
Assignee
Labels
No labels
Issue Votes (0)
Watchers (3)
Reference
TOD-11
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover