Support Karpenter core e2e tests

[maxcao13]

What problem are you trying to solve?

The karpenter core library contains a re-usable set of integration/e2e tests that any provider can use to test their implementations core functionalities: https://github.com/kubernetes-sigs/karpenter/tree/main/test#integrating-karpenter-tests-for-cloud-providers

We should follow the steps in the doc to integrate these tests into our project as a means of validation to ensure compatibility.

One caveat we have is that in order to do this, we need to set up a Cluster API environment with a workload and a management cluster and set up the required resources (machine deployments, infrastructure, karpenter itself) in both clusters. I propose that we /re-use/re-work/follow/ the kubemark-cluster-api-provider setup here: https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/blob/e783a75928f5c23336e3a9b581039c80722d675d/hack/tests/Makefile#L179 and build off of this setup into integrating the karpenter e2es. It's not as simple as it may seem, as there are a lot of moving parts in this setup, but ideally we can pin these steps down so that e2e tests can be executed and reproducible in our CI pipeline.

Ideally, we should follow the re-usable step pattern that the core repo has to deploy and test their kwok-provider implementation: https://github.com/kubernetes-sigs/karpenter/tree/main/kwok. In other words, we want some re-usable steps that can be used both in a local smoketesting environment, and can be called by an e2e CI framework (GitHub actions).

This would be something like:

1. Have a hack script/makefile target to install Kubemark CAPI (create 2 kind clusters, one mgmt, one workload).
2. Have a separate script + steps to deploy karpenter into the CAPI environment (eventually we want to probably deploy Karpenter CAPI with our helm chart)
3. Setup required CAPI workloads (MachineDeployments, KubemarkMachineTemplates) that can be scaled with Karepenter CAPI.
4. Follow the rest of the steps in https://github.com/kubernetes-sigs/karpenter/blob/0802160fc5c8efd0a2141a5731a9974beb391697/Makefile#L46

I think we should do this in checkpoints.

• Get CAPI + Kubemark reproducible smoketesting setup working in the project
  • Similar to https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/blob/e783a75928f5c23336e3a9b581039c80722d675d/hack/tests/Makefile#L179
  • If possible, write a github action to exercise this. (Karpenter will come later).
• Extend the setup by installing Karpenter (kubectl apply) into the environment
  • Write a simple e2e test (scale up, scale down machinedeployment) for CAPI Karpenter (not using karpenter core library yet), and integrate it into the github action
• Integrate Karpenter core e2e tests into the CI
  • I anticipate many of the tests not going to work out of the box. That is okay, we can disable most of them and enable only the ones we know should work. This issue should mostly be about getting the CI environment ready, and get good signal on regressions.

How important is this feature to you?

Currently, we don't have a great way for validation of the core karpenter features work in a cluster api environment.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[maxcao13]

/assign