Skip to content

Azure: check SRIOV on all nics and retry switch_sriov - #4659

Open
mcgov (mcgov) wants to merge 3 commits into
mcgov/stack-5-wget-skip-existsfrom
mcgov/stack-6-azure-sriov
Open

Azure: check SRIOV on all nics and retry switch_sriov#4659
mcgov (mcgov) wants to merge 3 commits into
mcgov/stack-5-wget-skip-existsfrom
mcgov/stack-6-azure-sriov

Conversation

@mcgov

Copy link
Copy Markdown
Collaborator

Part 6 of 9 of a stacked series that reworks the DPDK SRIOV hot plug tests. Stacked on #4658, review only the last commit.

  • is_enabled_sriov only inspected the primary nic, so a multi nic VM reported accelerated networking as enabled while a secondary nic was still being updated. It now checks every nic on the VM.
  • switch_sriov failed outright on transient HttpResponseError replies from the network resource provider, so it is retried a few times before giving up.

Key Test Cases:
verify_dpdk_sriov_rescind_failover_send_only|verify_sriov_disable_enable|verify_sriov_add_max_nics

Impacted LISA Features:
Sriov, NetworkInterface

Tested Azure Marketplace Images:

  • canonical 0001-com-ubuntu-server-jammy 22_04-lts latest
  • redhat rhel 9_5 latest

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Azure SR-IOV handling in LISA’s Azure feature layer to make SR-IOV enablement checks accurate for multi-NIC VMs and to improve resiliency of SR-IOV toggling against transient Azure Network RP/ARM errors.

Changes:

  • Add retry handling to switch_sriov to tolerate transient HttpResponseError responses.
  • Update is_enabled_sriov to validate accelerated networking state across all VM NICs (not just the primary).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lisa/sut_orchestrator/azure/features.py
Comment thread lisa/sut_orchestrator/azure/features.py Outdated
Copilot AI review requested due to automatic review settings August 17, 2026 23:09
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 1418d99 to 57b3d7c Compare August 17, 2026 23:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:1048

  • nic_info.enable_accelerated_networking can be None (it’s treated as optional elsewhere in this file), so using &= can raise TypeError (bool & NoneType) and also reads like a bitwise operation. Use a boolean and check (or is True) so the method always returns a real bool and treats None as disabled.
            sriov_enabled &= nic_info.enable_accelerated_networking

lisa/sut_orchestrator/azure/features.py:991

  • The retry decorator uses a fixed 30s delay with no backoff/jitter. This can significantly extend test runtime (up to ~60s extra plus Azure calls) on transient failures, and it’s inconsistent with other HttpResponseError retries in this module that use short delays with backoff. Consider adding a brief rationale for the 30s delay and/or using a backoff strategy.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

Copilot AI review requested due to automatic review settings August 17, 2026 23:34
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 57b3d7c to c49b9aa Compare August 17, 2026 23:34
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from c49b9aa to 6ba6238 Compare August 17, 2026 23:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1048

  • nic_info.enable_accelerated_networking from the Azure SDK can be None (Optional[bool]). Using sriov_enabled &= nic_info.enable_accelerated_networking can raise a TypeError when the value is None, and &= is a bitwise op that’s less clear than logical AND for booleans. Coerce the SDK value to a boolean (and treat None as disabled) while keeping the result strictly bool.
            sriov_enabled &= nic_info.enable_accelerated_networking

Copilot AI review requested due to automatic review settings August 17, 2026 23:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:1048

  • Major: nic_info.enable_accelerated_networking can be None (elsewhere this field is compared with is False), and sriov_enabled &= None will raise TypeError. Coerce the value to a strict bool (treat None as disabled) before combining results.
                self._resource_group_name, nic_name
            )
            sriov_enabled &= nic_info.enable_accelerated_networking

lisa/sut_orchestrator/azure/features.py:991

  • Minor: @retry(HttpResponseError, tries=3, delay=30) retries all HttpResponseError cases with a fixed 30s delay, which can make permanent failures slower and is inconsistent with the shorter/backoff-based HttpResponseError retries elsewhere in this file. Consider either (a) documenting why 30s is needed here, or (b) aligning with the backoff pattern used in other Azure RP retries / narrowing the retried conditions to transient cases.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

Copilot AI review requested due to automatic review settings August 18, 2026 05:57
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 6ba6238 to 786f228 Compare August 18, 2026 05:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1049

  • sriov_enabled &= nic_info.enable_accelerated_networking relies on enable_accelerated_networking being a strict bool. Elsewhere in this file the code checks x.enable_accelerated_networking is False (around line ~809), which suggests this field may also be None. In that case, &= will raise TypeError: unsupported operand type(s) for &= 'bool' and 'NoneType'. Consider using logical and and coercing None to False.
                self._resource_group_name, nic_name
            )
            sriov_enabled &= nic_info.enable_accelerated_networking
        return sriov_enabled

Copilot AI review requested due to automatic review settings August 18, 2026 06:09
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 786f228 to 802beb1 Compare August 18, 2026 06:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1052

  • is_enabled_sriov() currently accumulates NIC state with sriov_enabled &= (...), which is harder to read than a simple early-return and also forces an Azure NIC GET for every NIC even after one NIC is already known to be disabled. Consider short-circuiting on the first NIC that isn't explicitly True for enable_accelerated_networking.
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= (
                accelnet_setting if isinstance(accelnet_setting, bool) else False
            )

Copilot AI review requested due to automatic review settings August 24, 2026 20:48
@mcgov
mcgov (mcgov) force-pushed the mcgov/stack-6-azure-sriov branch from 1527bc9 to bc1d31b Compare August 24, 2026 20:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:991

  • Consider linking the related issue for traceability.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

lisa/sut_orchestrator/azure/features.py:991

  • Major: Wrapping the whole operation in retry resets status_changed on every attempt. If Azure accepts an update but the following get raises HttpResponseError, the retry can observe the NIC already in the target state, leave status_changed false, and return without _check_sriov_enabled even when wait=True; the guest VF state may not have converged. Retry only the failing SDK operation, or preserve and re-run the post-update wait after a retry.
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

azure_platform: AzurePlatform = self._platform # type: ignore
network_client = get_network_client(azure_platform)
sriov_enabled: bool = False
sriov_enabled: bool = True
)
sriov_enabled = primary_nic.enable_accelerated_networking
for nic in vm.network_profile.network_interfaces:
assert nic.id, "'nic.id' must not be 'None'"
@LiliDeng
LiliDeng force-pushed the mcgov/stack-6-azure-sriov branch from bc1d31b to e897db2 Compare August 25, 2026 12:36
Copilot AI review requested due to automatic review settings August 26, 2026 02:03
@LiliDeng
LiliDeng force-pushed the mcgov/stack-6-azure-sriov branch from e897db2 to 5d56dfc Compare August 26, 2026 02:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1052

  • is_enabled_sriov() can be simplified and made more efficient: it currently accumulates with &= and continues querying Azure NIC resources even after one NIC is found with accelerated networking disabled/unknown. Using early-return makes the intent clearer and avoids extra network_interfaces.get() calls on multi-NIC VMs. Also, using assert for nic.id in runtime code can be optimized away with python -O; raising a LisaException keeps the validation in all runtimes.
            accelnet_setting = nic_info.enable_accelerated_networking
            # check if all nics have accelnet enabled
            sriov_enabled &= (
                accelnet_setting if isinstance(accelnet_setting, bool) else False
            )

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

lisa/sut_orchestrator/azure/features.py:1049

  • is_enabled_sriov accumulates NIC state using &= on enable_accelerated_networking, which may be None (Optional[bool]) and will raise TypeError with bitwise-AND. It’s also harder to read than an early-return/all() pattern. Consider treating anything other than True as disabled and returning early.
            nic_info = network_client.network_interfaces.get(
                self._resource_group_name, nic_name
            )
            sriov_enabled &= nic_info.enable_accelerated_networking
        return sriov_enabled

lisa/sut_orchestrator/azure/features.py:991

  • The new retry uses hard-coded values (tries=3, delay=30) without explaining why they’re appropriate. Adding a brief rationale inline will make future tuning/debugging easier (and clarifies the added ~60s worst-case delay).
    @retry(HttpResponseError, tries=3, delay=30)  # type: ignore

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lisa/sut_orchestrator/azure/features.py:1040

  • Minor: is_enabled_sriov uses bitwise &= and a type guard to combine NIC settings, which is harder to read and doesn’t short-circuit. Since enable_accelerated_networking is effectively a tri-state (bool/None), a direct is not True check is clearer and avoids relying on bitwise boolean behavior.
    def is_enabled_sriov(self) -> bool:
        # check whether sriov is enabled on all nics
        azure_platform: AzurePlatform = self._platform  # type: ignore
        network_client = get_network_client(azure_platform)
        sriov_enabled: bool = True

mcgov (mcgov) and others added 3 commits August 26, 2026 11:51
is_enabled_sriov only inspected the primary nic, so a multi nic VM
reported accelerated networking as enabled while a secondary nic was
still being updated. Check every nic on the VM instead.

switch_sriov also failed outright on transient HttpResponseError
replies from the network resource provider, so retry it a few times
before giving up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d5f58ad-b9df-4420-ad37-22caee78e925

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment on lines +1048 to +1052
accelnet_setting = nic_info.enable_accelerated_networking
# check if all nics have accelnet enabled
sriov_enabled &= (
accelnet_setting if isinstance(accelnet_setting, bool) else False
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants