Skip to content

[Test] Add end2end test to validate tutotial for the upgrade of nvidia software stack. - #7500

Open
gmarciani wants to merge 5 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3160/test-upgrade-nvidia-0722-1
Open

[Test] Add end2end test to validate tutotial for the upgrade of nvidia software stack.#7500
gmarciani wants to merge 5 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3160/test-upgrade-nvidia-0722-1

Conversation

@gmarciani

@gmarciani gmarciani commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Description of changes

Add end2end test to validate tutotial for the upgrade of nvidia software stack.

Also, remove code duplication by making 3 tests use a common function to validate gpu workload.

Tests

ONGOING

test-suites:
  tutorials:
    test_upgrade_nvidia_software.py::test_upgrade_nvidia_software:
      dimensions:
        # The NVIDIA upgrade component targets Amazon Linux 2023 x86_64 only.
        - regions: [ {{ g4dn_2xlarge_CAPACITY_RESERVATION_2_INSTANCES_2_HOURS_NOPG_alinux2023 }} ]
          instances: [ "g4dn.2xlarge" ]
          oss: [ "alinux2023" ]
          schedulers: [ "slurm" ]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@gmarciani gmarciani changed the title Wip/mgiacomo/3160/test upgrade nvidia 0722 1 [Test] Add end2end test to validate tutotial for the upgrade of nvidia software stack. Jul 22, 2026
@gmarciani gmarciani added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x Test labels Jul 22, 2026
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/test-upgrade-nvidia-0722-1 branch from 752dbe4 to 2f5d944 Compare July 22, 2026 22:44
@gmarciani
gmarciani marked this pull request as ready for review July 24, 2026 15:22
@gmarciani
gmarciani requested review from a team as code owners July 24, 2026 15:22
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/test-upgrade-nvidia-0722-1 branch from 2f5d944 to 8683840 Compare July 29, 2026 20:22
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/test-upgrade-nvidia-0722-1 branch from 107b52c to 902faea Compare July 29, 2026 21:01

# NVLSM is versioned independently of the driver and is not pinned in the component: the component
# installs the version bundled in the NVIDIA driver local repository for NVIDIA_DRIVER_VERSION
# (2025.10.14 is the version bundled in the 580.173.02 driver local repository).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why get it from 580.173.02 rather than 595?

# NVLSM is versioned independently of the driver and is not pinned in the component: the component
# installs the version bundled in the NVIDIA driver local repository for NVIDIA_DRIVER_VERSION
# (2025.10.14 is the version bundled in the 580.173.02 driver local repository).
NVLSM_BUNDLED_VERSION = "2025.10.14"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the above comment is correct, then the is_equal_to assertion will fail. Please double check it.

def _wait_for_build_image_complete(image):
"""Wait for the image build to complete and assert it succeeded."""
logging.info("Waiting for build of image %s to complete", image.image_id)
while image.image_status.endswith("_IN_PROGRESS"): # e.g. BUILD_IN_PROGRESS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Block] This while loop doesn't have a timeout, if the AMI build hang, it will stuck forever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-changelog-update Disables the check that enforces changelog updates in PRs Test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants