createIssue tool schema does not constrain fields.Type to a valid enum (invalid tool calls from LLM) #3254
Nathan Chappell opened 10 hours ago

Reported by a user via service desk regarding LLM tool/function-call schemas.

Summary

  • The JSON schema generated for the createIssue tool is too permissive / incorrect, which leads to invalid tool calls from models.
  • Specifically, parameters.fields.Type is declared as a plain string ("type": "string") with:
    • no enum restriction on the allowed issue types, and
    • no entry in the schema's required list, and
    • no description documenting valid values.
  • As a result a model emitted an invalid Type and the call failed server-side. Reported model: "gpt-6.1-terra medium with codex".

Observed schema fragment

  • parameters.properties.fields.properties.Type => { "type": "string" } (no enum; not required)

Expected

  • Constrain fields.Type to an enum of valid issue types so the model is guided to valid values and invalid values are avoided at generation time.
  • Ensure the tool/function schema generation (likely the langchain4j integration) preserves enum constraints and required fields instead of degrading enums into free-form strings.
  • Add description text listing the valid values.

Impact

  • Invalid tool calls and errors when creating issues via the agent.
  • Reporter further suspects broader tool-schema problems ("baggage from langchain python"), i.e. other parameters may also be losing enum/required constraints — worth auditing the generated schemas for all tools.

Concrete examples observed while filing this very issue (self-observed)

  • Priority: no enum in schema. First attempt failed with "Invalid choice value 'High'"; valid values are the non-obvious set Minor, Normal, Major, Critical.
  • Affected-Versions: not marked required in the schema, yet submission fails with "Invalid fields: Affected Versions (value must not be empty)". A required field that the schema does not advertise as required.

Next steps / info requested

  • Please share the exact error message and the full tool-call payload (the model's arguments) plus the generated schema for createIssue, so we can reproduce and confirm scope.
  • Audit other tool schemas for similar enum/required degradation (Type, Priority, Labels, Affected-Versions, etc.).
  • Nathan Chappell commented 10 hours ago

    Here's a little funny for you:


    image.png


    I just want to say that this project is bomb, and I'm really looking forward to using it, but we gotta get this llm usage tightened up a bit. In python, langchain is almost always an indication of issues, but I don't know about java. However, if you want to discuss it I can look into the langchain stuff and discuss it with you.

    I'm sorry if I'm insulting anyone's intelligence, like I say this is a great product I look forward to deploying for our team, but some of the LLM calls can definitely be tightened up a bit.

1/1
Type
Bug
Priority
Major
Assignee
Affected Versions
Unspecified
Labels
No labels
Issue Votes (0)
Watchers (2)
Reference
OD-3254
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover