[WIP] rustdoc testing: Further improve chapters and sections

fmease · fmease · commit d276166be644 · 2025-06-02T19:22:37.000+02:00
diff --git a/src/SUMMARY.md b/src/SUMMARY.md
@@ -101,6 +101,8 @@
 - [Rustdoc internals](./rustdoc-internals.md)
     - [Search](./rustdoc-internals/search.md)
 	- [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md)
+	- [The `rustdoc-gui` test suite](./rustdoc-internals/rustdoc-gui-test-suite.md)
+	- [The `rustdoc-json` test suite](./rustdoc-internals/rustdoc-json-test-suite.md)
 - [Autodiff internals](./autodiff/internals.md)
     - [Installation](./autodiff/installation.md)
     - [How to debug](./autodiff/debugging.md)
diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md
@@ -272,19 +272,10 @@ Some extra reading about `make_test` can be found
 
 ## Dotting i's And Crossing t's
 
-So that's `rustdoc`'s code in a nutshell, but there's more things in the
-compiler that deal with it. Since we have the full `compiletest` suite at hand,
-there's a set of tests in `tests/rustdoc` that make sure the final `HTML` is
-what we expect in various situations. These tests also use a supplementary
-script, `src/etc/htmldocck.py`, that allows it to look through the final `HTML`
-using `XPath` notation to get a precise look at the output. The full
-description of all the commands available to `rustdoc` tests (e.g. [`@has`] and
-[`@matches`]) is in [`htmldocck.py`].
-
-To use multiple crates in a `rustdoc` test, add `//@ aux-build:filename.rs`
-to the top of the test file. `filename.rs` should be placed in an `auxiliary`
-directory relative to the test file with the comment. If you need to build
-docs for the auxiliary file, use `//@ build-aux-docs`.
+<!-- FIXME(fmease):
+* Make this section make sense again
+* Deduplicate with section in ../rustdoc.md#tests
+-->
 
 In addition, there are separate tests for the search index and `rustdoc`'s
 ability to query it. The files in `tests/rustdoc-js` each contain a
@@ -295,10 +286,6 @@ that features results in all tabs can be found in `basic.js`. The basic idea is
 that you match a given `QUERY` with a set of `EXPECTED` results, complete with
 the full item path of each item.
 
-[`@has`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L39
-[`@matches`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L44
-[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
-
 ## Testing Locally
 
 Some features of the generated `HTML` documentation might require local
diff --git a/src/rustdoc-internals/rustdoc-gui-test-suite.md b/src/rustdoc-internals/rustdoc-gui-test-suite.md
@@ -0,0 +1,14 @@
+# The `rustdoc-gui` test suite
+
+> **FIXME**: This section is a stub. Please help us flesh it out!
+
+This page is about the test suite named `rustdoc-gui` used to test the "GUI" of `rustdoc` (i.e., the HTML/JS/CSS as rendered in a browser).
+For other rustdoc-specific test suites, see [Rustdoc test suites].
+
+These use a NodeJS-based tool called [`browser-UI-test`] that uses [puppeteer] to run tests in a headless browser and check rendering and interactivity. For information on how to write this form of test, see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme] as well as [the description of the `.goml` format][goml-script]
+
+[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites
+[`browser-UI-test`]: https://github.com/GuillaumeGomez/browser-UI-test/
+[puppeteer]: https://pptr.dev/
+[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md
+[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md
diff --git a/src/rustdoc-internals/rustdoc-json-test-suite.md b/src/rustdoc-internals/rustdoc-json-test-suite.md
@@ -0,0 +1,3 @@
+# The `rustdoc-json` test suite
+
+> **FIXME**: This section is a stub. It will be populated by [PR #2422](https://github.com/rust-lang/rustc-dev-guide/pull/2422/).
diff --git a/src/rustdoc-internals/rustdoc-test-suite.md b/src/rustdoc-internals/rustdoc-test-suite.md
@@ -1,25 +1,22 @@
 # The `rustdoc` test suite
 
-This page is specifically about the test suite named `rustdoc`.
-For other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests).
+This page is about the test suite named `rustdoc` used to test the HTML output of `rustdoc`.
+For other rustdoc-specific test suites, see [Rustdoc test suites].
 
-The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.
-
-This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath].
+This is achieved by means of [`htmldocck.py`],
+a supplementary checker script invoked by compiletest that leverages [XPath].
 
+[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites
+[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
 [XPath]: https://en.wikipedia.org/wiki/XPath
 
-## Directives
-Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.
+## HtmlDocCk-Specific Directives
 
-In addition to the directives listed here,
-`rustdoc` tests also support most
-[compiletest directives](../tests/directives.html).
+Directives to HtmlDocCk are similar to those given to `compiletest` in that they take the form of `//@` comments.
 
-All `PATH`s in directives are relative to the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`),
-so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid
-having to write a long crate name multiple times.
 To avoid repetition, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument.
+It is conventional to use a `#![crate_name = "foo"]` attribute to avoid
+having to write a long crate name multiple times.
 
 All arguments take the form of quoted strings
 (both single and double quotes are supported),
@@ -35,33 +32,41 @@ In this case, the start of the next line should be `//`, with no `@`.
 
 For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.
 
+<!-- FIXME(fmease): Mention that the  regexes match case-sensitively and in single-line mode? -->
+
 ### `has`
 
 Usage 1: `//@ has PATH`
 Usage 2: `//@ has PATH XPATH PATTERN`
 
 In the first form, `has` checks that a given file exists.
 
-In the second form, `has` is an alias for `matches`,
+In the second form, `has` is the same as `matches`,
 except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
+<!-- FIXME(fmease): It's more important to note *here* that the file under test gets normalized too (PATTERN is in 99% cases already normalized)  -->
 
 ### `matches`
 
 Usage: `//@ matches PATH XPATH PATTERN`
 
-Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.
+Checks that the text of each element selected by `XPATH` in `PATH` matches the Python-flavored regex `PATTERN`.
 
 ### `matchesraw`
 
-Usage: `//@ matchesraw PATH PATTERN`
+Usage: `//@ matchesraw PATH XPATH PATTERN`
 
 Checks that the contents of the file `PATH` matches the regex `PATTERN`.
 
+<!-- FIXME(fmease): This previously didn't mention XPATH, mention it in prose -->
+
 ### `hasraw`
 
-Usage: `//@ hasraw PATH PATTERN`
+Usage: `//@ hasraw PATH XPATH PATTERN`
 
 Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
+<!-- FIXME(fmease): It's more important to note *here* that the file under test gets normalized too (PATTERN is in 99% cases already normalized)  -->
+
+<!-- FIXME(fmease): This previously didn't mention XPATH, mention it in prose -->
 
 ### `count`
 
@@ -79,9 +84,13 @@ determined by the XPath, and compares it to a pre-recorded value
 in a file. The file's name is the test's name with the `.rs` extension
 replaced with `.NAME.html`, where NAME is the snapshot's name.
 
-htmldocck supports the `--bless` option to accept the current subtree
+HtmlDocCk supports the `--bless` option to accept the current subtree
 as expected, saving it to the file determined by the snapshot's name.
-compiletest's `--bless` flag is forwarded to htmldocck.
+compiletest's `--bless` flag is forwarded to HtmlDocCk.
+
+<!-- FIXME(fmease): Also mention that we normalize certain URLS
+both when and checking and when normalizing
+-->
 
 ### `has-dir`
 
@@ -94,19 +103,42 @@ Checks for the existence of directory `PATH`.
 Usage: `//@ files PATH ENTRIES`
 
 Checks that the directory `PATH` contains exactly `ENTRIES`.
-`ENTRIES` is a python list of strings inside a quoted string,
+`ENTRIES` is a Python list of strings inside a quoted string,
 as if it were to be parsed by `eval`.
 (note that the list is actually parsed by `shlex.split`,
-so it cannot contain arbitrary python expressions).
+so it cannot contain arbitrary Python expressions).
 
 Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`
 
 [^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space.  The files themselves are also whitespace-normalized.
 
+## Compiletest Directives
+
+In addition to the directives listed here,
+`rustdoc` tests also support most
+[compiletest directives](../tests/directives.html).
+
+<!-- FIXME(fmease):
+Should definitely also mention `//@ aux-crate` and `//@ proc-macro`
+UNLESS we nuke this paragraph entirely and refer to the compiletest section(s)?
+-->
+To use multiple crates in a `rustdoc` test, add `//@ aux-build:filename.rs`
+to the top of the test file. `filename.rs` should be placed in an `auxiliary`
+directory relative to the test file with the comment.
+
+<!-- FIXME(fmease): We might want to explain why this exists / what this actually means -->
+If you need to build docs for the auxiliary file, use `//@ build-aux-docs`.
+
+<!-- FIXME(fmease): Mention `//@ doc-flags`! -->
+
 ## Limitations
-`htmldocck.py` uses the xpath implementation from the standard library.
+
+HtmlDocCk uses the XPath implementation from the Python standard library.
 This leads to several limitations:
+
 * All `XPATH` arguments must start with `//` due to a flaw in the implementation.
 * Many XPath features (functions, axies, etc.) are not supported.
 * Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags).
 
+<!-- FIXME(fmease): Maybe link to revisions?  -->
+Furthmore, compiletest revisions are not supported.
diff --git a/src/rustdoc.md b/src/rustdoc.md
@@ -67,43 +67,29 @@ does is call the `main()` that's in this crate's `lib.rs`, though.)
 
 ## Code structure
 
-* All paths in this section are relative to `src/librustdoc` in the rust-lang/rust repository.
+All paths in this section are relative to `src/librustdoc/` in the rust-lang/rust repository.
+
 * Most of the HTML printing code is in `html/format.rs` and `html/render/mod.rs`.
-  It's in a bunch of `fmt::Display` implementations and supplementary
-  functions.
-* The types that got `Display` impls above are defined in `clean/mod.rs`, right
-  next to the custom `Clean` trait used to process them out of the rustc HIR.
+  It's in a bunch of functions returning `impl std::fmt::Display`.
+* The data types that get rendered by the functions mentioned above are defined in `clean/types.rs`.
+  The functions responsible for creating them from the `HIR` and the `rustc_middle::ty` IR
+  live in `clean/mod.rs`.
 * The bits specific to using rustdoc as a test harness are in
   `doctest.rs`.
 * The Markdown renderer is loaded up in `html/markdown.rs`, including functions
   for extracting doctests from a given block of Markdown.
 * Frontend CSS and JavaScript are stored in `html/static/`.
+  * Re. JavaScript, type annotations are written using [TypeScript-flavored JSDoc]
+comments and an external `.d.ts` file.
+    This way, the code itself remains plain, valid JavaScript.
+    We only use `tsc` as a linter.
 
-## Tests
+[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html
 
-* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.
-  The format is specified
-  [in the search guide](rustdoc-internals/search.md#testing-the-search-engine).
-* Tests on the "UI" of rustdoc (the terminal output it produces when run) are in
-  `tests/rustdoc-ui`
-* Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser)
-  are in `tests/rustdoc-gui`. These use a [NodeJS tool called
-  browser-UI-test](https://github.com/GuillaumeGomez/browser-UI-test/) that uses
-  puppeteer to run tests in a headless browser and check rendering and
-  interactivity.  For information on how to write this form of test,
-  see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme]
-  as well as [the description of the `.goml` format][goml-script]
-* Tests on the structure of rustdoc HTML output are located in `tests/rustdoc`,
-  where they're handled by the test runner of bootstrap and
-  the supplementary script `src/etc/htmldocck.py`.
-  [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md).
-* Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc]
-  comments and an external d.ts file. The code itself is plain, valid JavaScript; we only
-  use tsc as a linter.
+## Tests
 
-[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html
-[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md
-[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md
+`rustdoc`'s integration tests are split across several test suites.
+See [Rustdoc tests suites](tests/compiletest.md#rustdoc-test-suites) for more details.
 
 ## Constraints
 
diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md
@@ -56,6 +56,9 @@ incremental compilation. The various suites are defined in
 
 The following test suites are available, with links for more information:
 
+[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
+[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
+
 ### Compiler-specific test suites
 
 | Test suite                                | Purpose                                                                                                             |
@@ -71,26 +74,31 @@ The following test suites are available, with links for more information:
 | [`mir-opt`](#mir-opt-tests)               | Check MIR generation and optimizations                                                                              |
 | [`coverage`](#coverage-tests)             | Check coverage instrumentation                                                                                      |
 | [`coverage-run-rustdoc`](#coverage-tests) | `coverage` tests that also run instrumented doctests                                                                |
+| [`crashes`](#crashes-tests)               | Check that the compiler ICEs/panics/crashes on certain inputs to catch accidental fixes                             |
 
 ### General purpose test suite
 
 [`run-make`](#run-make-tests) are general purpose tests using Rust programs.
 
 ### Rustdoc test suites
 
-See [Rustdoc tests](../rustdoc.md#tests) for more details.
-
-| Test suite       | Purpose                                                                  |
-|------------------|--------------------------------------------------------------------------|
-| `rustdoc`        | Check `rustdoc` generated files contain the expected documentation       |
-| `rustdoc-gui`    | Check `rustdoc`'s GUI using a web browser                                |
-| `rustdoc-js`     | Check `rustdoc` search is working as expected                            |
-| `rustdoc-js-std` | Check rustdoc search is working as expected specifically on the std docs |
-| `rustdoc-json`   | Check JSON output of `rustdoc`                                           |
-| `rustdoc-ui`     | Check terminal output of `rustdoc`                                       |
-
-[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
-[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
+| Test suite                           | Purpose                                                                  |
+|--------------------------------------|--------------------------------------------------------------------------|
+| [`rustdoc`][rustdoc-html-tests]      | Check HTML output of `rustdoc`                                           |
+| [`rustdoc-gui`][rustdoc-gui-tests]   | Check `rustdoc`'s GUI using a web browser                                |
+| [`rustdoc-js`][rustdoc-js-tests]     | Check `rustdoc`'s search engine and index                                |
+| [`rustdoc-js-std`][rustdoc-js-tests] | Check `rustdoc`'s search engine and index on the std library docs        |
+| [`rustdoc-json`][rustdoc-json-tests] | Check JSON output of `rustdoc`                                           |
+| `rustdoc-ui`                         | Check terminal output of `rustdoc` ([see also](ui.md))                   |
+
+<!-- FIXME(fmease): Also mention `tests/ui/rustdoc/`! -->
+<!-- | `tests/ui/rustdoc/` | |
+| `tests/run-make/rustdoc-*` | | -->
+
+[rustdoc-html-tests]: ../rustdoc-internals/rustdoc-test-suite.md
+[rustdoc-gui-tests]: ../rustdoc-internals/rustdoc-gui-test-suite.md
+[rustdoc-js-tests]: ../rustdoc-internals/search.md#testing-the-search-engine
+[rustdoc-json-tests]: ../rustdoc-internals/rustdoc-json-test-suite.md
 
 ### Pretty-printer tests
 
diff --git a/src/tests/directives.md b/src/tests/directives.md
@@ -261,14 +261,27 @@ Consider writing the test as a proper incremental test instead.
 
 | Directive   | Explanation                                                  | Supported test suites                    | Possible values           |
 |-------------|--------------------------------------------------------------|------------------------------------------|---------------------------|
-| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags |
+| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json`  | Any valid `rustdoc` flags |
 
 <!--
 **FIXME(rustdoc)**: what does `check-test-line-numbers-match` do?
 Asked in
 <https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/What.20is.20the.20.60check-test-line-numbers-match.60.20directive.3F>.
 -->
 
+#### Test-suite-specific directives
+
+The test suites [`rustdoc`][rustdoc-html-tests],
+[`rustdoc-js`/`rustdoc-js-std`][rustdoc-js-tests] and
+[`rustdoc-json`][rustdoc-json-tests]
+each feature an additional set of directives whose basic syntax resembles the one of
+compiletest directives but which are ultimately read and checked by separate tools.
+For more information, please read their respective chapters as linked above.
+
+[rustdoc-html-tests]: ../rustdoc-internals/rustdoc-test-suite.md
+[rustdoc-js-tests]: ../rustdoc-internals/search.html#testing-the-search-engine
+[rustdoc-json-tests]: ../rustdoc-internals/rustdoc-json-test-suite.md
+
 ### Pretty printing
 
 See [Pretty-printer](compiletest.md#pretty-printer-tests).

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	+# The `rustdoc-json` test suite
	`2`	`+`
	`3`	`+> FIXME: This section is a stub. It will be populated by [PR #2422](https://github.com/rust-lang/rustc-dev-guide/pull/2422/).`