Azure: check SRIOV on all nics and retry switch_sriov - #4659
Azure: check SRIOV on all nics and retry switch_sriov#4659mcgov (mcgov) wants to merge 3 commits into
Conversation
784f8c6 to
1418d99
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates Azure SR-IOV handling in LISA’s Azure feature layer to make SR-IOV enablement checks accurate for multi-NIC VMs and to improve resiliency of SR-IOV toggling against transient Azure Network RP/ARM errors.
Changes:
- Add retry handling to
switch_sriovto tolerate transientHttpResponseErrorresponses. - Update
is_enabled_sriovto validate accelerated networking state across all VM NICs (not just the primary).
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
1418d99 to
57b3d7c
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:1048
nic_info.enable_accelerated_networkingcan beNone(it’s treated as optional elsewhere in this file), so using&=can raiseTypeError(bool&NoneType) and also reads like a bitwise operation. Use a booleanandcheck (oris True) so the method always returns a realbooland treatsNoneas disabled.
sriov_enabled &= nic_info.enable_accelerated_networking
lisa/sut_orchestrator/azure/features.py:991
- The retry decorator uses a fixed 30s delay with no backoff/jitter. This can significantly extend test runtime (up to ~60s extra plus Azure calls) on transient failures, and it’s inconsistent with other
HttpResponseErrorretries in this module that use short delays with backoff. Consider adding a brief rationale for the 30s delay and/or using a backoff strategy.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
57b3d7c to
c49b9aa
Compare
c49b9aa to
6ba6238
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1048
nic_info.enable_accelerated_networkingfrom the Azure SDK can beNone(Optional[bool]). Usingsriov_enabled &= nic_info.enable_accelerated_networkingcan raise aTypeErrorwhen the value isNone, and&=is a bitwise op that’s less clear than logical AND for booleans. Coerce the SDK value to a boolean (and treatNoneas disabled) while keeping the result strictlybool.
sriov_enabled &= nic_info.enable_accelerated_networking
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:1048
- Major:
nic_info.enable_accelerated_networkingcan beNone(elsewhere this field is compared withis False), andsriov_enabled &= Nonewill raiseTypeError. Coerce the value to a strict bool (treatNoneas disabled) before combining results.
self._resource_group_name, nic_name
)
sriov_enabled &= nic_info.enable_accelerated_networking
lisa/sut_orchestrator/azure/features.py:991
- Minor:
@retry(HttpResponseError, tries=3, delay=30)retries all HttpResponseError cases with a fixed 30s delay, which can make permanent failures slower and is inconsistent with the shorter/backoff-based HttpResponseError retries elsewhere in this file. Consider either (a) documenting why 30s is needed here, or (b) aligning with the backoff pattern used in other Azure RP retries / narrowing the retried conditions to transient cases.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
6ba6238 to
786f228
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1049
sriov_enabled &= nic_info.enable_accelerated_networkingrelies onenable_accelerated_networkingbeing a strictbool. Elsewhere in this file the code checksx.enable_accelerated_networking is False(around line ~809), which suggests this field may also beNone. In that case,&=will raiseTypeError: unsupported operand type(s) for &= 'bool' and 'NoneType'. Consider using logicalandand coercingNonetoFalse.
self._resource_group_name, nic_name
)
sriov_enabled &= nic_info.enable_accelerated_networking
return sriov_enabled
786f228 to
802beb1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1052
is_enabled_sriov()currently accumulates NIC state withsriov_enabled &= (...), which is harder to read than a simple early-return and also forces an Azure NIC GET for every NIC even after one NIC is already known to be disabled. Consider short-circuiting on the first NIC that isn't explicitlyTrueforenable_accelerated_networking.
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= (
accelnet_setting if isinstance(accelnet_setting, bool) else False
)
1527bc9 to
bc1d31b
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:991
- Consider linking the related issue for traceability.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
lisa/sut_orchestrator/azure/features.py:991
- Major: Wrapping the whole operation in
retryresetsstatus_changedon every attempt. If Azure accepts an update but the followinggetraisesHttpResponseError, the retry can observe the NIC already in the target state, leavestatus_changedfalse, and return without_check_sriov_enabledeven whenwait=True; the guest VF state may not have converged. Retry only the failing SDK operation, or preserve and re-run the post-update wait after a retry.
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
| azure_platform: AzurePlatform = self._platform # type: ignore | ||
| network_client = get_network_client(azure_platform) | ||
| sriov_enabled: bool = False | ||
| sriov_enabled: bool = True |
| ) | ||
| sriov_enabled = primary_nic.enable_accelerated_networking | ||
| for nic in vm.network_profile.network_interfaces: | ||
| assert nic.id, "'nic.id' must not be 'None'" |
bc1d31b to
e897db2
Compare
e897db2 to
5d56dfc
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1052
is_enabled_sriov()can be simplified and made more efficient: it currently accumulates with&=and continues querying Azure NIC resources even after one NIC is found with accelerated networking disabled/unknown. Using early-return makes the intent clearer and avoids extranetwork_interfaces.get()calls on multi-NIC VMs. Also, usingassertfornic.idin runtime code can be optimized away withpython -O; raising aLisaExceptionkeeps the validation in all runtimes.
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= (
accelnet_setting if isinstance(accelnet_setting, bool) else False
)
5d56dfc to
5da40eb
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/sut_orchestrator/azure/features.py:1049
is_enabled_sriovaccumulates NIC state using&=onenable_accelerated_networking, which may beNone(Optional[bool]) and will raiseTypeErrorwith bitwise-AND. It’s also harder to read than an early-return/all() pattern. Consider treating anything other thanTrueas disabled and returning early.
nic_info = network_client.network_interfaces.get(
self._resource_group_name, nic_name
)
sriov_enabled &= nic_info.enable_accelerated_networking
return sriov_enabled
lisa/sut_orchestrator/azure/features.py:991
- The new retry uses hard-coded values (
tries=3,delay=30) without explaining why they’re appropriate. Adding a brief rationale inline will make future tuning/debugging easier (and clarifies the added ~60s worst-case delay).
@retry(HttpResponseError, tries=3, delay=30) # type: ignore
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
lisa/sut_orchestrator/azure/features.py:1040
- Minor:
is_enabled_sriovuses bitwise&=and a type guard to combine NIC settings, which is harder to read and doesn’t short-circuit. Sinceenable_accelerated_networkingis effectively a tri-state (bool/None), a directis not Truecheck is clearer and avoids relying on bitwise boolean behavior.
def is_enabled_sriov(self) -> bool:
# check whether sriov is enabled on all nics
azure_platform: AzurePlatform = self._platform # type: ignore
network_client = get_network_client(azure_platform)
sriov_enabled: bool = True
is_enabled_sriov only inspected the primary nic, so a multi nic VM reported accelerated networking as enabled while a secondary nic was still being updated. Check every nic on the VM instead. switch_sriov also failed outright on transient HttpResponseError replies from the network resource provider, so retry it a few times before giving up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5d5f58ad-b9df-4420-ad37-22caee78e925
| accelnet_setting = nic_info.enable_accelerated_networking | ||
| # check if all nics have accelnet enabled | ||
| sriov_enabled &= ( | ||
| accelnet_setting if isinstance(accelnet_setting, bool) else False | ||
| ) |
Part 6 of 9 of a stacked series that reworks the DPDK SRIOV hot plug tests. Stacked on #4658, review only the last commit.
is_enabled_sriovonly inspected the primary nic, so a multi nic VM reported accelerated networking as enabled while a secondary nic was still being updated. It now checks every nic on the VM.switch_sriovfailed outright on transientHttpResponseErrorreplies from the network resource provider, so it is retried a few times before giving up.Key Test Cases:
verify_dpdk_sriov_rescind_failover_send_only|verify_sriov_disable_enable|verify_sriov_add_max_nics
Impacted LISA Features:
Sriov, NetworkInterface
Tested Azure Marketplace Images:
canonical 0001-com-ubuntu-server-jammy 22_04-lts latestredhat rhel 9_5 latest