Skip to content

Fix sampling option compatibility metadata for OpenAI reasoning models - #40

Merged
superdav42 merged 9 commits into
WordPress:trunkfrom
henryperkins:fix/gpt5-reasoning-sampling-options
Aug 16, 2026
Merged

superdav42 merged 9 commits into
WordPress:trunkfrom
henryperkins:fix/gpt5-reasoning-sampling-options

Conversation

@henryperkins

@henryperkins henryperkins commented Jul 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The provider previously advertised temperature, top_p, logprobs, and top_logprobs for every text-generation model. That made requirement-based resolution treat sampling-incompatible reasoning models as eligible, with failures deferred to the OpenAI API.

This PR makes the four sampling fields model-aware and keeps them consistent as a group. It also maps the AI Client SDK's native log-probability options into the Responses API, rejects explicit non-none reasoning combined with sampling before a request is sent, adds focused PHPUnit coverage, excludes test assets from both release-package paths, and adds read-only CI across PHP 7.4–8.4.

Compatibility matrix

“Advertise sampling” means the provider metadata advertises all four fields: temperature, top_p, logprobs, and top_logprobs.

Model IDs Documented snapshot(s) Advertise sampling? Basis
gpt-5 gpt-5-2025-08-07 No Reasoning cannot be disabled; documented efforts exclude none.
gpt-5-mini gpt-5-mini-2025-08-07 No Reasoning cannot be disabled.
gpt-5-pro gpt-5-pro-2025-10-06 No Only high reasoning is supported.
gpt-5.1, gpt-5.1-2025-11-13 gpt-5.1-2025-11-13 Yes reasoning.effort: none is the documented default.
gpt-5.2, gpt-5.2-2025-12-11 gpt-5.2-2025-12-11 Yes reasoning.effort: none is the documented default.
gpt-5.4, gpt-5.4-2026-03-05 gpt-5.4-2026-03-05 Yes reasoning.effort: none is the documented default.
gpt-5.4-mini, gpt-5.4-mini-2026-03-17 gpt-5.4-mini-2026-03-17 Yes reasoning.effort: none is the documented default.
gpt-5.4-nano, gpt-5.4-nano-2026-03-17 gpt-5.4-nano-2026-03-17 Yes reasoning.effort: none is the documented default.
gpt-5.2-pro gpt-5.2-pro-2025-12-11 No No none mode is documented.
gpt-5.2-codex gpt-5.2-codex No No none mode is documented.
gpt-5-chat-latest gpt-5-chat-latest Yes Documented as a chat model. The live probe was availability-limited, so documentation remains the contract.
gpt-5.1-chat-latest gpt-5.1-chat-latest No in this Responses provider Still a usable text alias, but an accessible live Responses probe rejected temperature.
gpt-5.2-chat-latest gpt-5.2-chat-latest No in this Responses provider Still catalogued as a usable text alias even though deprecated; an accessible live Responses probe rejected temperature.
codex-mini-latest codex-mini-latest No Reasoning cannot be disabled.
o3 o3-2025-04-16 No Reasoning cannot be disabled.
o4-mini o4-mini-2025-04-16 No Reasoning cannot be disabled.
gpt-4o current: gpt-4o-2024-08-06; documented: gpt-4o-2024-11-20, gpt-4o-2024-08-06, gpt-4o-2024-05-13 Yes Non-reasoning control.
gpt-4.1 gpt-4.1-2025-04-14 Yes Non-reasoning control.

Fine-tuned IDs are classified through their extracted base model while the original fine-tuned ID remains in ModelMetadata; tests cover an eligible ft:gpt-5.2:… and an ineligible ft:gpt-5.2-codex:….

The two versioned chat aliases are deliberately conservative exceptions to their documentation-derived chat classification. Live verification directly established only that temperature is rejected. Because provider metadata exposes the four sampling fields consistently, the grouped advertisement is withdrawn for those aliases; this does not claim that every sampling field was independently live-probed or rejected.

Future GPT-5 minor versions remain conservative until documented and tested. The reasoning-family matcher recognizes only the verified o1, o3, and o4 families; future o-series families are not auto-covered.

Request mapping and validation

  • ModelConfig::getLogprobs() === true maps to include: ["message.output_text.logprobs"].
  • A non-null ModelConfig::getTopLogprobs() maps to top_logprobs.
  • logprobs: false leaves include unset.
  • temperature, top_p, logprobs, and top_logprobs remain allowed with explicit reasoning.effort: none.
  • Any of those fields combined with an explicit non-none reasoning effort raises InvalidArgumentException before the API request.

These names match the official Responses create reference, including message.output_text.logprobs and top_logprobs.

Sanitized live Responses probes

Date: 2026-07-18. Every request used input Reply with OK. and max_output_tokens: 16. No credential, header, request/response identifier, raw response, or account identifier is recorded.

Payload categories:

  • temperature_default_none: temperature: 0.2, relying on the documented default effort.
  • reasoning_none_top_p: reasoning.effort: none, top_p: 0.9.
  • logprobs_top_logprobs: include: ["message.output_text.logprobs"], top_logprobs: 5.
  • chat_temperature: temperature: 0.2 on the named chat alias.
Model Payload category HTTP Result Error code Error type Error parameter
gpt-5.1 temperature_default_none 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.1 reasoning_none_top_p 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.1 logprobs_top_logprobs 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.2 temperature_default_none 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.2 reasoning_none_top_p 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.2 logprobs_top_logprobs 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4 temperature_default_none 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4 reasoning_none_top_p 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4 logprobs_top_logprobs 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4-mini temperature_default_none 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4-mini reasoning_none_top_p 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4-mini logprobs_top_logprobs 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4-nano temperature_default_none 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4-nano reasoning_none_top_p 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.4-nano logprobs_top_logprobs 403 verification limited model_not_found invalid_request_error not recorded
gpt-5-chat-latest chat_temperature 403 verification limited model_not_found invalid_request_error not recorded
gpt-5.1-chat-latest chat_temperature 400 temperature_rejected absent (null) invalid_request_error temperature
gpt-5.2-chat-latest chat_temperature 400 temperature_rejected absent (null) invalid_request_error temperature

temperature_rejected is a normalized report result, not an OpenAI error code. Availability/access/model-not-found responses do not become incompatibility rules. No reasoning_high_temperature probes were sent because none of the five reasoning-model baselines returned a 2xx accessibility confirmation.

Packaging and release scope

  • Composer/git archives exclude tests/ and phpunit.xml.dist via .gitattributes.
  • WordPress.org staging excludes the same assets via .distignore while retaining plugin.php and production source.
  • PR-introduced symbols use @since 1.1.0; the plugin header remains 1.0.3 for the release process.
  • The package keeps PHP >=7.4 and the existing AI Client SDK dependency line.

Verification

Local at a422636d52bc66696528bb1370c63c94eeafc934

  • composer validate --strict --no-check-publish: pass.
  • composer lint: pass; PHPCS 9/9 files, PHPStan 5/5 analyzed paths with no errors.
  • composer test: pass; 35 tests, 178 assertions.
  • git diff --check trunk...HEAD: pass.
  • git status --short: empty.
  • Composer archive: 15 entries, 0 tests/ or phpunit.xml.dist matches.
  • WordPress.org staging: tests/ and phpunit.xml.dist absent; plugin.php and src/Metadata/OpenAiModelMetadataDirectory.php present.

GitHub Actions

Exact-head run: CI run 29640691457

Job Conclusion
Lint, static analysis, and distribution Success
PHPUnit on PHP 7.4 Success
PHPUnit on PHP 8.0 Success
PHPUnit on PHP 8.1 Success
PHPUnit on PHP 8.2 Success
PHPUnit on PHP 8.3 Success
PHPUnit on PHP 8.4 Success

The quality job runs composer lint and both release-package exclusion checks. All Actions use read-only repository permissions and immutable action SHAs.

Fixes #38

claude and others added 3 commits July 18, 2026 00:59
…aware

OpenAI reasoning models (codex-mini-latest, the versioned GPT-5 family,
and the o-series) reject the `temperature`, `top_p`, `logprobs`, and
`top_logprobs` options unless reasoning is disabled via
`reasoning.effort: 'none'`. The metadata previously advertised all of
these options for every text generation model, so a configured sampling
option let incompatible models pass requirement-based model resolution
only to fail at the OpenAI API.

Classify reasoning models via a documented REASONING_MODEL_ID_PATTERN
and stop advertising the sampling options for them - consistently for
all four options, not just `temperature`.

Models whose default reasoning effort is `none` (gpt-5.2 and gpt-5.4,
including dated snapshots) accept the sampling options in that default
mode, so they keep advertising them; otherwise valid default-mode
requests would wrongly become ineligible during model resolution. The
metadata layer cannot express conditional option support, so requests
that explicitly enable reasoning on these models are validated at
request time in OpenAiTextGenerationModel instead. Suffixed variants
(-mini, -pro, -codex) and older GPT-5 versions cannot run without
reasoning and remain excluded.

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DZXsuhii6xotGrDomF6hmD
Models with a default reasoning effort of `none` (gpt-5.2, gpt-5.4)
advertise the sampling options in their metadata because they apply in
the default mode, but the OpenAI API rejects those parameters as soon
as reasoning is enabled. Since a reasoning effort can only be requested
via the `reasoning` custom option, validate the prepared request
parameters and throw an InvalidArgumentException with an explanatory
message when `temperature`, `top_p`, or `top_logprobs` is combined with
a reasoning effort other than `none`, before the API request is sent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DZXsuhii6xotGrDomF6hmD
Add phpunit/phpunit to require-dev with a phpunit.xml.dist
configuration, a tests bootstrap, an autoload-dev mapping, and a
`composer test` script.

The metadata tests cover sampling option support across gpt-5,
gpt-5-mini, gpt-5.1, gpt-5.2 (incl. a dated snapshot), gpt-5.4,
gpt-5-pro, gpt-5.2-pro, gpt-5.2-codex, codex-mini-latest, o3, o4-mini,
and standard GPT controls. The model tests cover the request-time
validation of sampling parameters against explicitly requested
reasoning efforts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DZXsuhii6xotGrDomF6hmD
@henryperkins
henryperkins requested a review from a team July 18, 2026 01:21
@github-actions

github-actions Bot commented Jul 18, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @claude.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: claude.

Co-authored-by: henryperkins <htperkins@git.wordpress.org>
Co-authored-by: slaFFik <slaffik@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@superdav42 superdav42 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me . Thanks!

@superdav42
superdav42 merged commit 1fd7a90 into WordPress:trunk Aug 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gpt-5.5 does not support temperature that is sent when generating ALT text for images

4 participants