gh-108303: Move all doctest related files and tests to Lib/test/test_doctest/

[sobolevn]

• Issue: Move test files into test subdirectories #108303

[vstinner]

Calling list.pop(0) frequently is inefficient. Maybe the constructor should create a collections.deque() and call self.lines.popleft() instead?

[sobolevn]

It is usually used in cases like sys.stdin = FakeInput(['next', 'print(x)', 'continue']), I don't think it matters much. But, I can do this if you want :)

[sobolevn]

I solved merge conflicts, cc @vstinner

[vstinner]

Since we are moving code, would it be worth it to split test_doctest.py which has 3 374 lines into multiple files? I found the following tests / test cases:

• SampleClass
• sample_func
• SampleNewStyleClass
• test_CLI
• test_debug
• test_DocFileSuite
• test_DocTest
• test_DocTestFinder
• test_DocTestParser
• test_DocTestRunner
• test_DocTestSuite
• test_Example
• test_exception_with_multiple_notes
• test_exception_with_note
• test_lineendings
• test_look_in_unwrapped
• test_no_trailing_whitespace_stripping
• test_pdb_set_trace
• test_pdb_set_trace_nested
• test_run_doctestsuite_multiple_times
• test_syntax_error_subclass_from_stdlib
• test_syntax_error_with_incorrect_expected_note
• test_syntax_error_with_note
• test_testfile
• test_testmod
• test_testsource
• test_trailing_space_in_test
• test_unicode
• test_unittest_reportflags

For example, it would be nice to have 1000 lines or less per file. If you are interested to make such change, I suggest creating a new PR, so we can compare.

About test_doctest2: why is it a separated file? Should it be renamed? The docstring says:

A module to test whether doctest recognizes some 2.2 features, like static and class methods.

I suppose that 2.2 stands for Python 2.2.

Is it a test on the "parser" or "test discovery"?

[sobolevn]

Since we are moving code, would it be worth it to split test_doctest.py which has 3 374 lines into multiple files?

I don't think so, because moving a file is a rather simple operation. It does not spoil git blame. While splitting files can be devastating to its history.

I work with test_doctest quite a lot, I don't think that it has maintability problems.

About test_doctest2: why is it a separated file? Should it be renamed?

I agree that the naming of this file is not optimal. It also clashes with test_doctest2.txt, I don't think that this is intentional.

Looks like that all tests from it are just duplicates. We already test classes, classmethods, staticmethods and everything else in test_doctest. So, it can be simply removed.

It has some details that I want to explicitly port:

• property with a setter
• poluting namespace with other objects

[vstinner]

I cannot see where Lib/test/test_doctest2.py test goes. Did you just remove it?

[sobolevn]

Yes:

I agree that the naming of this file is not optimal. It also clashes with test_doctest2.txt, I don't think that this is intentional.

Looks like that all tests from it are just duplicates. We already test classes, classmethods, staticmethods and everything else in test_doctest. So, it can be simply removed.

[vstinner]

Looks like that all tests from it are just duplicates. We already test classes, classmethods, staticmethods and everything else in test_doctest. So, it can be simply removed.

I'm not convinced that test_doctest2 is useless. It calls doctest.testmod(sys.modules[__name__]) and test_doctest2.py contains 2 non-ASCII docstrings. Example:

"""...
We include some (random) encoded (utf-8) text in the text surrounding
the example.  It should be ignored:

ЉЊЈЁЂ
"""

It runs tests in verbose mode instead of running them in quiet mode:

• test_doctest: doctest.testmod(unicodedata, verbose=False) (quiet)
• test_doctest2: doctest.testmod(sys.modules[__name__]) (verbose)

Even we agree that test_doctest2 is useless, I would prefer to not include this silent removal in the middle of such huge refactoring (move all tests code).

Please add back test_doctest2.py in the new test_doctest/ directory.

[miss-islington-app]

Thanks @sobolevn for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

[bedevere-app]

GH-114254 is a backport of this pull request to the 3.12 branch.

[miss-islington-app]

Sorry, @sobolevn and @vstinner, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 9c93350f582fe6f5fed2cd873869dfe4fbf2dfe8 3.11

[vstinner]

Merged, thanks.

@sobolevn: The automated backport to 3.11 failed, do you want to attempt to backport the change to 3.11 manually?

[sobolevn]

Sure thing!

[brettcannon]

FYI this broke the WASI builds; https://buildbot.python.org/all/#/builders/1046/builds/4033 and https://buildbot.python.org/all/#/builders/1124/builds/623 .

[vstinner]

FYI this broke the WASI builds; https://buildbot.python.org/all/#/builders/1046/builds/4033 and https://buildbot.python.org/all/#/builders/1124/builds/623 .

Fixed by efb81a6 of #114275

[bedevere-app]

GH-114313 is a backport of this pull request to the 3.11 branch.