Repository navigation
Fix sampling option compatibility metadata for OpenAI reasoning models - #40
superdav42 merged 9 commits into
Conversation
…aware OpenAI reasoning models (codex-mini-latest, the versioned GPT-5 family, and the o-series) reject the `temperature`, `top_p`, `logprobs`, and `top_logprobs` options unless reasoning is disabled via `reasoning.effort: 'none'`. The metadata previously advertised all of these options for every text generation model, so a configured sampling option let incompatible models pass requirement-based model resolution only to fail at the OpenAI API. Classify reasoning models via a documented REASONING_MODEL_ID_PATTERN and stop advertising the sampling options for them - consistently for all four options, not just `temperature`. Models whose default reasoning effort is `none` (gpt-5.2 and gpt-5.4, including dated snapshots) accept the sampling options in that default mode, so they keep advertising them; otherwise valid default-mode requests would wrongly become ineligible during model resolution. The metadata layer cannot express conditional option support, so requests that explicitly enable reasoning on these models are validated at request time in OpenAiTextGenerationModel instead. Suffixed variants (-mini, -pro, -codex) and older GPT-5 versions cannot run without reasoning and remain excluded. Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZXsuhii6xotGrDomF6hmD
Models with a default reasoning effort of `none` (gpt-5.2, gpt-5.4) advertise the sampling options in their metadata because they apply in the default mode, but the OpenAI API rejects those parameters as soon as reasoning is enabled. Since a reasoning effort can only be requested via the `reasoning` custom option, validate the prepared request parameters and throw an InvalidArgumentException with an explanatory message when `temperature`, `top_p`, or `top_logprobs` is combined with a reasoning effort other than `none`, before the API request is sent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZXsuhii6xotGrDomF6hmD
Add phpunit/phpunit to require-dev with a phpunit.xml.dist configuration, a tests bootstrap, an autoload-dev mapping, and a `composer test` script. The metadata tests cover sampling option support across gpt-5, gpt-5-mini, gpt-5.1, gpt-5.2 (incl. a dated snapshot), gpt-5.4, gpt-5-pro, gpt-5.2-pro, gpt-5.2-codex, codex-mini-latest, o3, o4-mini, and standard GPT controls. The model tests cover the request-time validation of sampling parameters against explicitly requested reasoning efforts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZXsuhii6xotGrDomF6hmD
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @claude. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
superdav42
left a comment
There was a problem hiding this comment.
Looks good to me . Thanks!
Summary
The provider previously advertised
temperature,top_p,logprobs, andtop_logprobsfor every text-generation model. That made requirement-based resolution treat sampling-incompatible reasoning models as eligible, with failures deferred to the OpenAI API.This PR makes the four sampling fields model-aware and keeps them consistent as a group. It also maps the AI Client SDK's native log-probability options into the Responses API, rejects explicit non-
nonereasoning combined with sampling before a request is sent, adds focused PHPUnit coverage, excludes test assets from both release-package paths, and adds read-only CI across PHP 7.4–8.4.Compatibility matrix
“Advertise sampling” means the provider metadata advertises all four fields:
temperature,top_p,logprobs, andtop_logprobs.gpt-5gpt-5-2025-08-07none.gpt-5-minigpt-5-mini-2025-08-07gpt-5-progpt-5-pro-2025-10-06highreasoning is supported.gpt-5.1,gpt-5.1-2025-11-13gpt-5.1-2025-11-13reasoning.effort: noneis the documented default.gpt-5.2,gpt-5.2-2025-12-11gpt-5.2-2025-12-11reasoning.effort: noneis the documented default.gpt-5.4,gpt-5.4-2026-03-05gpt-5.4-2026-03-05reasoning.effort: noneis the documented default.gpt-5.4-mini,gpt-5.4-mini-2026-03-17gpt-5.4-mini-2026-03-17reasoning.effort: noneis the documented default.gpt-5.4-nano,gpt-5.4-nano-2026-03-17gpt-5.4-nano-2026-03-17reasoning.effort: noneis the documented default.gpt-5.2-progpt-5.2-pro-2025-12-11nonemode is documented.gpt-5.2-codexgpt-5.2-codexnonemode is documented.gpt-5-chat-latestgpt-5-chat-latestgpt-5.1-chat-latestgpt-5.1-chat-latesttemperature.gpt-5.2-chat-latestgpt-5.2-chat-latesttemperature.codex-mini-latestcodex-mini-latesto3o3-2025-04-16o4-minio4-mini-2025-04-16gpt-4ogpt-4o-2024-08-06; documented:gpt-4o-2024-11-20,gpt-4o-2024-08-06,gpt-4o-2024-05-13gpt-4.1gpt-4.1-2025-04-14Fine-tuned IDs are classified through their extracted base model while the original fine-tuned ID remains in
ModelMetadata; tests cover an eligibleft:gpt-5.2:…and an ineligibleft:gpt-5.2-codex:….The two versioned chat aliases are deliberately conservative exceptions to their documentation-derived chat classification. Live verification directly established only that
temperatureis rejected. Because provider metadata exposes the four sampling fields consistently, the grouped advertisement is withdrawn for those aliases; this does not claim that every sampling field was independently live-probed or rejected.Future GPT-5 minor versions remain conservative until documented and tested. The reasoning-family matcher recognizes only the verified o1, o3, and o4 families; future o-series families are not auto-covered.
Request mapping and validation
ModelConfig::getLogprobs() === truemaps toinclude: ["message.output_text.logprobs"].ModelConfig::getTopLogprobs()maps totop_logprobs.logprobs: falseleavesincludeunset.temperature,top_p,logprobs, andtop_logprobsremain allowed with explicitreasoning.effort: none.nonereasoning effort raisesInvalidArgumentExceptionbefore the API request.These names match the official Responses create reference, including
message.output_text.logprobsandtop_logprobs.Sanitized live Responses probes
Date: 2026-07-18. Every request used input
Reply with OK.andmax_output_tokens: 16. No credential, header, request/response identifier, raw response, or account identifier is recorded.Payload categories:
temperature_default_none:temperature: 0.2, relying on the documented default effort.reasoning_none_top_p:reasoning.effort: none,top_p: 0.9.logprobs_top_logprobs:include: ["message.output_text.logprobs"],top_logprobs: 5.chat_temperature:temperature: 0.2on the named chat alias.gpt-5.1temperature_default_nonemodel_not_foundinvalid_request_errorgpt-5.1reasoning_none_top_pmodel_not_foundinvalid_request_errorgpt-5.1logprobs_top_logprobsmodel_not_foundinvalid_request_errorgpt-5.2temperature_default_nonemodel_not_foundinvalid_request_errorgpt-5.2reasoning_none_top_pmodel_not_foundinvalid_request_errorgpt-5.2logprobs_top_logprobsmodel_not_foundinvalid_request_errorgpt-5.4temperature_default_nonemodel_not_foundinvalid_request_errorgpt-5.4reasoning_none_top_pmodel_not_foundinvalid_request_errorgpt-5.4logprobs_top_logprobsmodel_not_foundinvalid_request_errorgpt-5.4-minitemperature_default_nonemodel_not_foundinvalid_request_errorgpt-5.4-minireasoning_none_top_pmodel_not_foundinvalid_request_errorgpt-5.4-minilogprobs_top_logprobsmodel_not_foundinvalid_request_errorgpt-5.4-nanotemperature_default_nonemodel_not_foundinvalid_request_errorgpt-5.4-nanoreasoning_none_top_pmodel_not_foundinvalid_request_errorgpt-5.4-nanologprobs_top_logprobsmodel_not_foundinvalid_request_errorgpt-5-chat-latestchat_temperaturemodel_not_foundinvalid_request_errorgpt-5.1-chat-latestchat_temperaturetemperature_rejectednull)invalid_request_errortemperaturegpt-5.2-chat-latestchat_temperaturetemperature_rejectednull)invalid_request_errortemperaturetemperature_rejectedis a normalized report result, not an OpenAI error code. Availability/access/model-not-found responses do not become incompatibility rules. Noreasoning_high_temperatureprobes were sent because none of the five reasoning-model baselines returned a 2xx accessibility confirmation.Packaging and release scope
tests/andphpunit.xml.distvia.gitattributes..distignorewhile retainingplugin.phpand production source.@since 1.1.0; the plugin header remains1.0.3for the release process.>=7.4and the existing AI Client SDK dependency line.Verification
Local at
a422636d52bc66696528bb1370c63c94eeafc934composer validate --strict --no-check-publish: pass.composer lint: pass; PHPCS 9/9 files, PHPStan 5/5 analyzed paths with no errors.composer test: pass; 35 tests, 178 assertions.git diff --check trunk...HEAD: pass.git status --short: empty.tests/orphpunit.xml.distmatches.tests/andphpunit.xml.distabsent;plugin.phpandsrc/Metadata/OpenAiModelMetadataDirectory.phppresent.GitHub Actions
Exact-head run: CI run 29640691457
The quality job runs
composer lintand both release-package exclusion checks. All Actions use read-only repository permissions and immutable action SHAs.Fixes #38