gh-103092: Convert some _ctypes metatypes to heap types

[aisk]

Make the metatypes in _ctypes into heap types.

• Issue: Isolate Stdlib Extension Modules #103092

[erlend-aasland]

Thanks, this is a good start. First, we should add some tests to Lib/test/test_ctypes/, though, to make sure we're not breaking anything as we proceed:

• test the inheritance hierarchy for _ctypes types
• test immutability for _ctypes types
• test disallow instantiation for relevant _ctypes types

Also, this PR introduces reference leaks; check the traverse/clear/free slots for each new heap type:

$ ./python.exe -m test -R : test_ctypes                                                                                                                                        (clinic/real-warnings *$%)
Using random seed: 4031286472
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.13 Run 1 test sequentially
0:00:00 load avg: 3.13 [1/1] test_ctypes
beginning 9 repetitions
123456789
.........
test_ctypes leaked [494, 494, 494, 494] references, sum=1976
test_ctypes leaked [314, 313, 313, 313] memory blocks, sum=1253
test_ctypes failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test_ctypes

Total duration: 6.4 sec
Total tests: run=432 skipped=29
Total test files: run=1/1 failed=1
Result: FAILURE

[erlend-aasland]

$ ./python.exe -m test -R : test_ctypes                                                                                                                                        (clinic/real-warnings *$%)
Using random seed: 4031286472
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.13 Run 1 test sequentially
0:00:00 load avg: 3.13 [1/1] test_ctypes
beginning 9 repetitions
123456789
.........
test_ctypes leaked [494, 494, 494, 494] references, sum=1976
test_ctypes leaked [314, 313, 313, 313] memory blocks, sum=1253
test_ctypes failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test_ctypes

Total duration: 6.4 sec
Total tests: run=432 skipped=29
Total test files: run=1/1 failed=1
Result: FAILURE

Fixed by 1877f7b; thanks, @aisk!

[erlend-aasland]

I'd really like if we could add tests before proceeding with this; we should do this in a separate PR.

[aisk]

Hi @erlend-aasland, thanks for the review, I'm currently adding the tests.

[aisk]

Sorry, I might not have been clear. Do you mean to add tests in another pull request, and then process this and subsequent pull requests to see if the changes maintain compatibility as before?

[erlend-aasland]

Sorry, I might not have been clear. Do you mean to add tests in another pull request, and then process this and subsequent pull requests to see if the changes maintain compatibility as before?

Yes, I would prefer to add the tests in a separate PR first. Then we can proceed with this.

[aisk]

Got it, working on this.

[neonene]

PyCArrayType_Type and PyCSimpleType_Type can use CDataType_clear/traverse, as they now have the GC protocol. If we care about the overhead of unnecessary visits to StgDict->proto (non-container), a comment would be nice?

[aisk]

I checked the CDataType_clear/traverse, and found that the only difference to current implementation, is that the CDataType version will try to convert self to StgDictObject and clear / visit StgDictObject's own fields.

Since PyCArrayType_Type and PyCSimpleType_Type are not StgDictObject, so I think re-use it is not nessasry, and may leads confusion.

[neonene]

Putting your comments into those functions will be helpful when converting the PyCStgDict type. Please check me (a bit different):

1) PyCStructType_Type
2) UnionType_Type
3) PyCPointerType_Type
4) PyCArrayType_Type
5) PyCSimpleType_Type

They are not StgDictObject, but their dict objects in the tp_dict slot are replaced with StgDictObject by using Py_SETREF(). Previously, (4)(5) did not have Py_TPFLAGS_HAVE_GC, since the stgdict->proto member is a non-container type (e.g. str type: "h","i").

Do you think (1)(2)(3) are StgDictObject? They use CDataType's traverse and clear functions.

[neonene]

Off-topic: StgDictObject itself (self->tp_dict) is handled by PyType_Type.tp_*(self). However, current PyCStgDict_Type does not have Py_TPFLAGS_HAVE_GC, so at least a container proto needs to be visited in CDataType_traverse(), taking account for the cache?/clone? of StgDictObject as well?

[aisk]

Ah, sorry, I made a mistake in my first comment. I've understood your idea, and the codes have been updated. Thank you for pointing it out!

[neonene]

Looks good to me.

[erlend-aasland]

Thanks for the PR, @aisk. Thanks for the reviews @neonene and @encukou.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x has failed when building commit 8e31cdc.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/506/builds/6880) and take a look at the build logs.
4. Check if the failure is related to this commit (8e31cdc) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/506/builds/6880

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 11, done.        
remote: Counting objects:   9% (1/11)        
remote: Counting objects:  18% (2/11)        
remote: Counting objects:  27% (3/11)        
remote: Counting objects:  36% (4/11)        
remote: Counting objects:  45% (5/11)        
remote: Counting objects:  54% (6/11)        
remote: Counting objects:  63% (7/11)        
remote: Counting objects:  72% (8/11)        
remote: Counting objects:  81% (9/11)        
remote: Counting objects:  90% (10/11)        
remote: Counting objects: 100% (11/11)        
remote: Counting objects: 100% (11/11), done.        
remote: Compressing objects:  16% (1/6)        
remote: Compressing objects:  33% (2/6)        
remote: Compressing objects:  50% (3/6)        
remote: Compressing objects:  66% (4/6)        
remote: Compressing objects:  83% (5/6)        
remote: Compressing objects: 100% (6/6)        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 6 (delta 5), reused 0 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '8e31cdc9450b3e644d48954865568e162edad514'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 8e31cdc945 gh-103092: Convert some `_ctypes` metatypes to heap types (GH-113620)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

/tmp/ccvaW277.s: Assembler messages:
/tmp/ccvaW277.s: Fatal error: Objects/genobject.o: No space left on device
make: *** [Makefile:2786: Objects/genobject.o] Error 1

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:2876: clean-retain-profile] Error 1 (ignored)

[encukou]

The buildbot fails with "No space left on device" while building.
(It's the second one to do so today...)