Conformance results and limits
The OpenID Foundation conformance suite tests server behaviour against its specifications. A PingFederate installation built from this repository has run six test plans. The results below identify failures and tests that remain outstanding.
release-v5.3.1. Certification requires an authenticated run on the OpenID Foundation’s hosted suite against a reachable server.
conformance/README.md:236-238Results by test plan
A test plan groups automated tests into modules. Each module checks server responses against specification requirements. These results come from PingFederate installations built with conformance/.
| Plan | What a pass shows | Result | When |
|---|---|---|---|
| openid-ssf-transmitter-test-plan | PingFederate, with this repository's Shared Signals servlet, behaves as a transmitter: discovery, stream management and poll delivery, with the receiver authenticating by private_key_jwt. |
19 of 19 passed | 2026-09-25 13.1.3 |
| openid-ssf-transmitter-caep-test-plan | The same transmitter under the CAEP Interoperability Profile, which the repository notes is the plan the Foundation certifies Shared Signals against. | 13 of 13 passed | 2026-09-23 local replica 2026-09-24 public test environment |
| fapi2-security-profile-final-test-plan | Checks FAPI 2.0 Final using private_key_jwt, DPoP sender-constraining, PAR and PKCE. |
50 passed, 0 failed 3 review, 2 warning, 1 skipped, of 56 |
2026-09-24 13.1.3 |
| openid-federation-deployed-entity-test-plan (alpha) | The Entity Configuration, the fetch endpoint including its error cases, pinned anchor keys and the resolve endpoint agree with OpenID Federation 1.0. | 5 warning, 0 failed | 2026-09-25 13.1.3 |
| openid-federation-entity-joined-to-test-federation-op-test-plan (alpha) | A relying party under the suite's own trust anchor registers automatically at the authorisation and PAR endpoints, and thirteen malformed requests are refused. | 20 warning, 0 failed | 2026-09-25 13.1.3 |
| fapi-ciba-id1-test-plan | Decoupled CIBA authentication in poll mode. Three modules cannot pass on this product: see below. | 32 passed, 3 failed | 2026-09-24 13.0.3 and 13.1.3 |
Results requiring qualification
The suite reports failures separately from warnings, manual reviews and skipped tests. The following results need qualification.
request_uri modules. PingFederate answers a reused, expired or foreign
request_uri with an HTTP 400 error page rather than a redirect. The suite captures the response for manual review. A fourth module was REVIEW on 13.0.3 and now passes on 13.1.3.Earlier federation failures identified missing OpenID Connect Discovery metadata and missing checks on requests from registered relying parties. Both defects have been fixed.
conformance/README.md:198-208 docs/federation/conformance.md:32-46FAPI 2.0 enforcement
FAPI 2.0 requires a client assertion’s audience to be the authorisation server’s issuer, expressed as a string. It limits DPoP proof signatures to PS256, ES256 or EdDSA. PingFederate 13.0.3 has no settings for these rules. Version 13.1 adds a server-wide audience setting that still accepts a one-element array; the DPoP algorithm list remains compiled in.
The repository’s FAPI 2.0 filter applies both checks before PingFederate processes the request. It rejects requests that fail and forwards accepted requests unchanged.
The filter applies to named clients. The suite’s Shared Signals client addresses assertions to the token endpoint. Applying the FAPI rules to every client caused 18 of its 19 modules to fail.
# Apply the filter to the two FAPI clients
OIDF_FAPI2_CLIENTS=conformance-fapi2-client1,conformance-fapi2-client2
# Permitted DPoP signature algorithms
ALLOWED_ALGORITHMS =
Set.of("PS256", "ES256", "EdDSA");A UserInfo filter returns x-fapi-interaction-id and rejects access tokens in query strings, as required by FAPI 1.0 Baseline. An OAuth error filter restricts error_description to RFC 6749’s permitted characters. The image build registers all three filters on PingFederate endpoints.
FAPI-CIBA limitation
Three of 35 FAPI-CIBA modules fail because the tested PingFederate versions lack mutual-TLS certificate-bound access tokens. Configuration changes and the repository’s filters do not resolve this limitation.
FAPI-CIBA requires certificate-bound access tokens under RFC 8705 §3, identified by cnf.x5t#S256. The repository’s investigation of PingFederate 13.0.3 and 13.1.3 found no support for this claim or the related metadata. These versions support DPoP sender-constrained tokens.
discovery-end-point-verificationfails on the missing metadata flag.ensure-mtls-holder-of-key-requiredfails on a token issued without a certificate.fapi-ciba-id1, the happy path, fails on its step that presents one client's certificate with another client's access token and expects a refusal the server cannot make.
The tested configurations cannot pass this plan. The other 32 modules passed.
conformance/README.md:225-234Production configuration differences
The test environment uses settings specific to the suite. Production configuration requires the changes listed below.
| Test configuration | Why | In production |
|---|---|---|
Turns off the server-wide client-assertion audience check (Rfc7523bisCompliantAudienceVerification) |
A fresh 13.1 archive accepts only its issuer, and the suite's ordinary clients address their assertions to the token endpoint, as OpenID Connect Core says to. Without the overlay the Shared Signals plan failed 18 of its 19 modules. | Leave the check on. The FAPI clients are held to the stricter rule by the filter either way. |
| Starts federation and attestation with requests blocked | The image carries every module, and two of them refuse to boot without a trust anchor and a trust controller. The rig names PingFederate as both and leaves the anchor keys unset. | Configure the anchor keys, or leave the modules out of the image. |
| Serves a self-signed certificate, and the federation profile disables TLS checking | Communication between local test containers. | Use trusted certificates and enable TLS verification. |
Reproduce the tests
The repository includes a script that builds and starts PingFederate 13.1.3 with the modules and configuration required by the suite.
The build requires Docker, Terraform, Maven and Node, plus Ping DevOps licence credentials. Scripts generate eight key pairs and use Terraform to configure PingFederate through its admin API. They export the configuration and build the image. Licence material is supplied separately.
terraform/ holds the configuration source. The image imports its exported archive at startup. Console changes are lost at the next start because the environment has no persistent volume.
./up.sh
# PingFederate on https://localhost:9031
# discovery /.well-known/openid-configuration
# transmitter /.well-known/ssf-configurationAutomated testing
Licensed jobs, nightly and weekly
A workflow builds PingFederate from the commit and boots it under a licence every night, with the negative boots, and every Sunday runs all six plans - FAPI 2.0, FAPI-CIBA, SSF, CAEP and both OpenID Federation plans - one at a time, each against its committed expected failures. A release needs both green on the tagged commit.
licensed.yml:31-46Licence requirements
The licence is a file on the self-hosted runner, never a repository secret. Without it every licensed job is skipped and says so; output is scrubbed of the licence details before it is shown or kept. Whether the file is there, and the latest run, are checked in GitHub Actions.
licensed.yml:6-20Manual conformance runs
Each plan runs by hand the way the licensed job runs it, against a local suite on the rig's network. The OpenID Provider plan has the suite host a relying party and trust anchor, which PingFederate reaches at host.docker.internal, an alias the suite's web server answers to on the rig's network.
conformance/README.md:297-306Mutation testing, weekly
A weekly mutation-testing workflow checks the federation library and PingFederate integration. The workflow fails when tests detect fewer than 85% of introduced mutations.
mutation.yml:1-10Untested scenarios
OpenID Foundation certification has not been obtained. Both federation plans are alpha. Shared Signals push delivery and CIBA ping mode remain untested because the local suite cannot receive permitted callbacks. The CAEP results combine a local replica run on 2026-09-23 and a public test environment run on 2026-09-24. The relying-party federation plan does not apply to this OpenID Provider implementation.
conformance/README.md:236-242 run-plan.py:7-9Repository test coverage
535 methods at 100% coverage
The build requires 100% line and branch coverage for 535 selected decision methods. Other methods are outside this requirement.
209 of 216 claims linked to tests
Tagged tests cover 209 of 216 conformance-matrix rows in the cited report. The generated dashboard identifies the seven rows without tagged tests.
Coverage scope
The matrix records implementation claims and links them to tests. Its coverage count does not measure full specification conformance.