Fix the bundled PhysX BV4 tree build on 32-bit platforms
BVData::mData64 in GuBV4Build.cpp holds either a pointer to a child node
or an encoded primitive index, so it is a size_t, but the marker for an
unused child is PX_INVALID_U64. Where size_t is 32 bits wide the
constructor truncates that marker to 0xffffffff, and every comparison
against it widens the field back to 0x00000000ffffffff, which never
matches. BV4Node::getType() therefore counts four children for every
node, and the ones carrying no data pass for leaves whose primitive has
the index 0x07ffffff.
Two things follow. The cooked tree sends the midphase traversal past the
end of the triangle indices, and the unused children keep the inverted
bounding box setEmpty() leaves behind rather than the empty one
flattenQ() writes for them, which also skews the quantization
coefficients. A scene query against a triangle mesh then reads out of
bounds as soon as the traversal reaches one of those children, which is
the segmentation fault the Android x86 configuration sees.
Which queries crash depends on the mesh, since the inverted box culls
the phantom children unless the query volume is large enough to straddle
it. A 12 triangle cube loses every overlap, sweep and raycast against
it, while a 3200 triangle sphere loses the sweep and the raycast and
passes the overlaps. Contact generation and ordinary character movement
never reach them at all, which is why the mesh tests already running on
that configuration have been passing.
Give the marker the width of the field it marks. Cooking a mesh in a
32-bit build then produces the same bytes as in a 64-bit one, byte for
byte for meshes from 12 to 16200 triangles, with gcc and with clang, and
the 64-bit output does not change.
The bug is upstream and still present in PhysX main. It arrived with
PhysX 5.3.0, which renamed the field from mData to mData64 and widened
the marker without widening the field. PhysX 4.1.2, which we shipped
until the 5.9.0 import, paired the size_t with PX_INVALID_U32 and was
correct on both word sizes, so no released Qt version is affected.
Upstream cannot reach it either: the public build presets are vc16win64,
vc17win64, linux and linux-aarch64, all 64-bit, and Android support was
never merged there in the first place.
Amends 7533abcc02b7f9bd7953203c489ee86d55e8c909
Change-Id: I1d0bd5f56b01185109770edd92008e2d0279dfa6
Reviewed-by: Kristoffer Skau <kristoffer.skau@qt.io>