Fix: Wait for ptp4l/phc2sys death before PF rebind in validation conftest - #1646
Open
awilczyns wants to merge 2 commits into
Open
Fix: Wait for ptp4l/phc2sys death before PF rebind in validation conftest#1646awilczyns wants to merge 2 commits into
awilczyns wants to merge 2 commits into
Conversation
awilczyns
force-pushed
the
fix-ptp-hanging-tests
branch
6 times, most recently
from
July 7, 2026 17:12
26f0ba7 to
02ef58b
Compare
awilczyns
force-pushed
the
fix-ptp-hanging-tests
branch
2 times, most recently
from
July 28, 2026 13:50
8662255 to
e8f49a1
Compare
setup_validation.sh assumed a system-wide MTL install and 2M hugepages, so a runner prepared with .local_install was reported as not ready even though RxTxApp/MtlManager were present. Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>
The @pytest.mark.ptp st20p tests failed EBU LIST compliance with invalid_delta_packet_ts_vs_rtp_ts offsets of ~-116s (vf_only captures were rejected outright as containing no stream at all). Two independent framework defects put the pcap timestamps and the transmitted RTP timestamps on different timescales: 1. ptp_sync started ptp4l on both physical ports. E810/E830 expose a single PHC per adapter (both instances log "selected /dev/ptp1"), so the second daemon did not get its own clock - and when its port was link-down after a previous test rebound it, that instance declared itself grandmaster and free-ran the shared PHC underneath the capture. Sync only the capture interface; it is the only PHC that has to follow the grandmaster. 2. The netsniff hook slept a fixed 50s "for PTP sync" before capturing. MTL's own PTP client can take well over a minute to lock after a PF is bound to DPDK, and until it does MTL stamps RTP from the system clock while the capture PHC already follows the grandmaster. Wait for MTL to report a locked PTP delta (< 1ms) on its stdout instead of guessing. Also carried over from the earlier work on this branch: skip mixed PF/VF interface profiles when both PFs share one IOMMU group (VFIO group is not viable, previously an opaque rc=244 crash), poll for actual ptp4l/phc2sys death before rebinding a PF, and fail fast when no grandmaster is reachable. Verified on mtl-runner-9 (E830, 720p60 VF-to-VF): MTL PTP lock detected in 21s and packet_ts_vs_rtp_ts drops from -116s to 131us (compliant). Signed-off-by: Wilczynski, Andrzej <andrzej.wilczynski@intel.com>
awilczyns
force-pushed
the
fix-ptp-hanging-tests
branch
from
July 28, 2026 15:48
e8f49a1 to
4b4f646
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nightly
run-pytest (e810, ptp)jobs on e810 runners (observed onmtl-runner-2) intermittently hang the host for the rest of the night.
Root cause traced through the host journal:
ptp4lholds the PF's PHCfd, test teardown killed it with a blind
sleep(0.3), and if the kernelhadn't yet released the fd,
nicctl's VF/PF rebind checks (bind_kernel/
disable_vf) timed out and fell back to a PCI remove+rescan. Thatforce-reprobes the PF (
ice_probe) outside the gracefulsriov_numvfs=0teardown path and can hit an
icedriver GPF (RSS flow-profileuse-after-free in
ice_add_prof).Also adds real PTP conformance coverage: the existing
@pytest.mark.ptptest never asserted anything about PTP state, so it passed even with no
reachable grandmaster.
Changes
Fix:_reap_ptp_daemonsnow polls for actual process death(
_wait_daemon_dead) after both SIGTERM and SIGKILL instead ofsleeping a fixed duration, closing the race before callers rebind the
PF.
Test: addstests/validation/tests/single/ptp/ptp_helpers.pyandtests/validation/tests/single/ptp/grandmaster/test_ptp_conformance.py:test_mtl_internal_ptp_converges— asserts MTL's own software PTPclient (
mt_ptp.c's periodicPTP(0): delta avg ...stat line)actually leaves "not connected" and settles within tolerance.
test_mtl_and_ptp4l_agree_on_grandmaster— cross-checks thatagainst
ptp4l(via the existingptp_syncfixture) converging onthe same wire, verifying external-grandmaster interoperability.
that's a topology precondition, not a regression.
Testing
pytest --collect-onlyclean (8 tests collected, no import errors).with a confirmed-reachable grandmaster to verify the new tests assert
(rather than skip).