Skip to content

Tentative testdriver methods #226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Tentative testdriver methods #226

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7cfbf57
Tentative testdriver methods
jcsteh Jun 3, 2025
bbb248e
Add additional risks per review.
jcsteh Jul 3, 2025
eaf2912
Cross-reference documentation on how to mark a test tentative.
jcsteh Jul 23, 2025
58e5694
Clarify that this is about WebDriver endpoints not defined in any spe…
jcsteh Jul 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions rfcs/tentative_testdriver_methods.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# RFC 226: Tentative testdriver methods

## Summary
Allow tentative methods to be added to testdriver.js which use WebDriver endpoints not yet defined in a specification.
This should only be allowed where there is consensus between multiple browser vendors and where there is clear progress towards a specification.

## Background
As part of the [Interop 2025 Accessibility Investigation Area](https://github.com/web-platform-tests/interop-accessibility/issues/148), the intention is to extend browsers and WPT to allow additional accessibility properties to be tested by web platform tests.
However, there are open questions regarding the shape of the API for exposing these properties, what properties should be exposed, how the tests should be written, etc.
While the end goal is to extend the WebDriver specification with the required new endpoints, these open questions need to be answered before this is feasible.
To answer these questions, it would be helpful if browser vendors could collaborate on a tentative but working implementation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For things like the Accessibility Investigation, is there any intention for there to be any, even high-level, explainer of what the proposed endpoints are? I'm mostly concerned about the risk of ending up somewhere where it is hard for anyone else to start passing the tests written using the tentative endpoint.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is WICG/aom#203. However, that's an evolving issue, not a document. I could create an explainer document if that's what we need, though.

Copy link

@cookiecrook cookiecrook Oct 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Goal is to have:

I don't think we need another explainer... We just need to work out the remaining differences and make sure they are clearly documented in the relevant locations.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One open question here is whether we need this RFC at all or whether an explainer is sufficient. See #226 (comment). If an explainer is sufficient, is it sufficient for that explainer to be a GitHub issue (as is the case now) or does it need to be a document (to make it easier to follow)?


## Details
1. There must be publicly documented (e.g. as part of an Interop Investigation Area) consensus between at least two browser vendors and intent to work towards a specification.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the tests for these tentative methods should be marked with tentative. But I think we should treat these tentative methods similarly to how we treat tentative tests for the sake of consistency. Given tentative tests do not require consensus currently, I would remove this requirement from tentative methods.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue that tentative testdriver API pose a greater risk, because removing tests that rely on undocumented API is painful and hard — see #172.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both the tests and the method name will be clearly marked as tentative until in the spec with two implementations.

2. Any tentative methods added to testdriver.js should be prefixed with `tentative_`.
3. Any utility methods which call tentative testdriver methods should also be prefixed with `tentative_`.
Comment on lines +15 to +16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we want there to be a tentative object, or whether we just want to prefix everything like this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if having a tentative object would make it trickier to "graduate" things from tentative once the spec is final (and my thinking is that we want to incentivise finalising tentative things as soon as is feasible). With prefixing, you can basically just search/replace once the spec is final. With an object, things will need to be moved around. That's obviously not a show stopper if an object is preferred, just something I thought worth considering.

Copy link

@cookiecrook cookiecrook Oct 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I agree with the logic to prefix any method names and even properties/testnames until finalized, but not the entire return object. that will be ever-evolving.

4. Any tests directly or indirectly calling tentative testdriver methods must be [marked tentative](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-:%20).

## Alternatives considered
1. Avoid any changes to the WPT repository until the specification is finalised.
This makes it very difficult for browser vendors to collaborate and to learn about problems that are much easier to discover and understand "in practice".
In contrast, if the specification were developed without a working implementation, there is a much higher risk of fundamental design problems in the specification which are much harder to fix later.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps the initial cross-engine prototype of the property bag implementation should include only the properties already exposed by other WebDriver methods, like computedrole and computedlabel.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would make it easier to standardise on a first pass, for sure. However, that first pass would have no practical benefit, since we can already get those properties using the existing methods. The problem I see with this is that we're less likely to spot problems because the implementation is thus entirely "theoretical": we won't spot the kinds of problems we're likely to see in testing new properties, which is the entire point of this work. In contrast, the current prototype supports a few simple ARIA properties and parent/children, which allows us to test the kinds of things we really want to test with this work and thus spot problems we might hit as we move forward.

Copy link

@cookiecrook cookiecrook Oct 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rahimabdi just completed a first pass implementation and may have some thoughts on a minimum list.

The string properties are straightforward, and the reflected IDL enums could also be represented as strings, so the webdriver element refs IDs and accessibility object ref IDs are the main issue. We know these may be different between implementations in places. Do we really want the first merge to be blocked on that, or can we agree on some minimal list of string properties in order to ship and open for parallel test writing while we work out the details on dom-tree-to-ax-tree ref IDs?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be discussed elsewhere, but now that we've ascertained that we don't need to worry about the WebElement-style abstraction for accessibility ids, I think the way in which accessibility ids should be exposed is easy to define: a simple unique string such as a UUID. DOM nodes should be exposed as WebElements just as they are everywhere else. It's just a matter of implementing it. There might be tree structure differences, but that's not relevant to WebDriver exposure; that's a higher level question of which parts of the tree we can standardise (and working around that in tests if need be).

2. Develop the implementation as vendor specific tests with methods in testdriver-vendor.js.
While this does allow a single vendor to iterate on a working implementation, it is still very difficult for multiple browser vendors to collaborate on this.
At the very least, other vendors may wish to contribute tests which exercise the implementation to ensure it fits their needs before agreeing to a final specification.

## Risks
1. Methods could be added which never end up being specified, resulting in cruft and non-standardised functionality.
This is not a significant risk to the web at large because these methods only impact tests and the tests must be marked tentative, preventing them from being considered for Interop scoring, for example.
2. The API could change significantly before it becomes finalized, with many tests depending on the tentative API.
This could mean that migrating the tests to the final API requires significant effort.
3. Because of the lack of specification, other vendors interested in ensuring interoperability with the feature being tested might have to reverse engineer how the tentative API is intended to function.