Problem with running the benchmark using my agent

[Tanaz-R]

Hello, since December, I have been working on my project on AIOpsLab, but today I tried to run it 10 times,
"k8s_target_port-misconfig-localization-1"
but after 7 times it will stuck in "Waiting for all pods in namespace 'test-social-network' to be ready... Waiting for pods to be ready...", then I face:

[red]Timeout: Not all pods in namespace 'test-social-network' reached the Ready state within 300 seconds.

I delete ns myself, all pods, even I delete kind cluster and install again, but today what ever I have done I could not fix to run. Even I just put PID to run every problem, but nothing could fix it.

Could you assist me in this regard, please?

Thanks,
Tanaz.

[HacksonClark]

Hmm, seems to me like the social-network app is having trouble getting started. To help debug though I'll need to see some logs from the cluster. Is there a pod that keeps crashing? A pod stuck in pending or ContainerCreating? You can either use kubectl or a tool like k9s to observe this and share it here so I can help you further.

[Tanaz-R]

This one: kubectl logs media-frontend-5cc8b79b86-2mllh -n test-social-network stays in ContainerCreating, the other stays in Init, and all the time were only 2 pods that were not running. I git the social-network app again, now it is running, but what I did, could not fix it yesterday.

[Tanaz-R]

Hi again, the issue starts to repeat again as you can see in the screenshot

The runtime error : raise Exception(f"[red]Timeout: Not all pods in namespace '{namespace}' reached the Ready state within {max_wait} seconds.") Exception: [red]Timeout: Not all pods in namespace 'test-social-network' reached the Ready state within 300 seconds.

`kubectl describe pod media-frontend-5cc8b79b86-lpf44 -n test-social-network
Name: media-frontend-5cc8b79b86-lpf44
Namespace: test-social-network
Priority: 0
Service Account: default
Node: kind-worker/172.18.0.3
Start Time: Wed, 28 Jan 2026 13:38:31 +0100
Labels: app=media-frontend
pod-template-hash=5cc8b79b86
service=media-frontend
Annotations:
Status: Pending
IP: 10.244.1.108
IPs:
IP: 10.244.1.108
Controlled By: ReplicaSet/media-frontend-5cc8b79b86
Init Containers:
alpine-container:
Container ID: containerd://c7f776aef5aae154c39760b0e5326df137add5d51d2157bb903a9358fb10c69d
Image: docker.io/alpine/git:latest
Image ID: docker.io/alpine/git@sha256:e6c6c4fd5fdb742d681fcdc6018cad699ca0a75909f7fe788e626b542a9c0438
Port:
Host Port:
Command:
/bin/sh
Args:
-c
git clone https://github.com/delimitrou/DeathStarBench.git /DeathStarBench && cp -r /DeathStarBench/socialNetwork/media-frontend/lua-scripts/* /lua-scripts/
State: Running
Started: Wed, 28 Jan 2026 13:38:32 +0100
Ready: False
Restart Count: 0
Environment:
Mounts:
/lua-scripts from lua-scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-spnzj (ro)
Containers:
media-frontend:
Container ID:
Image: docker.io/yg397/media-frontend:xenial
Image ID:
Port: 8081/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
fqdn_suffix: .test-social-network.svc.cluster.local
Mounts:
/social-network-microservices/config/jaeger-config.yml from media-frontend-config (rw,path="jaeger-config.yml")
/social-network-microservices/config/service-config.json from media-frontend-config (rw,path="service-config.json")
/usr/local/openresty/nginx/conf/nginx.conf from media-frontend-config (rw,path="nginx.conf")
/usr/local/openresty/nginx/lua-scripts from lua-scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-spnzj (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
media-frontend-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: media-frontend
Optional: false
lua-scripts:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-spnzj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 6m54s default-scheduler Successfully assigned test-social-network/media-frontend-5cc8b79b86-lpf44 to kind-worker
Normal Pulled 6m53s kubelet Container image "docker.io/alpine/git:latest" already present on machine
Normal Created 6m53s kubelet Created container: alpine-container
Normal Started 6m53s kubelet Started container alpine-container
`

and the logs:

kubectl logs media-frontend-5cc8b79b86-lpf44 -n test-social-network Defaulted container "media-frontend" out of: media-frontend, alpine-container (init) Error from server (BadRequest): container "media-frontend" in pod "media-frontend-5cc8b79b86-lpf44" is waiting to start: PodInitializing

[HacksonClark]

Tanaz (@Tanaz-R)

Error from server (BadRequest): container "media-frontend" in pod "media-frontend-5cc8b79b86-lpf44" is waiting to start: PodInitializing

This is certainly the root cause of the issue. The media-frontend pod is not able to start which is what causes the runtime error. I believe I may have had some issues with that container on certain machines. Could you share the specifications of your machine/cluster?

It's very strange that the issue seems to be flaky (it worked before and now is broken again).

[Tanaz-R]

Jackson Clark (@HacksonClark), I agree it is weird. About the certain machine, this is me #117, you can check my machine info there :)

[HacksonClark]

Tanaz (@Tanaz-R) Could you try running the image for the failing pod locally?

[Tanaz-R]

Jackson Clark (@HacksonClark) Hi again, sorry I did not reply your last reply, as the problem was solved I forget about the issue, but again I face it today Error while running problem k8s_target_port-misconfig-localization-1: [red]Timeout: Not all pods in namespace 'test-social-network' reached the Ready state within 300 seconds. but this time I cannot check anything at all as kubectl get pod -n test-social-network No resources found in test-social-network namespace.

[HacksonClark]

Tanaz (@Tanaz-R) could you try inspecting with a tool like k9s? I don't really have any information about what happened so I can't really help you.

[Tanaz-R]

Jackson Clark (@HacksonClark) are you talking about https://github.com/derailed/k9s? I will install, then check. I do not know about k9s, but as there is no pod in the namespace, can it really help? I will understand soon :)

[HacksonClark]

@HacksonClark are you talking about https://github.com/derailed/k9s? I will install, then check. I do not know about k9s, but as there is no pod in the namespace, can it really help? I will understand soon :)

Yes, that is what I'm referring to. I'm hoping we can find what may be causing the test-social-network namespace to not come up.

[Tanaz-R]

I am so confused the whole 7 hours at the office, whatever I did, I got the error, and no resource was in the namespace but just now from the k9s panel, everything is alright, I reported this before that this issue will come, and after many different tries fix again automatticlay there should be a bug in the loading of social-networking app.

[HacksonClark]

Did you typo the namespace or something? It happens.

[Tanaz-R]

No the benchmark failed on each related problem to test-social-network, it said after 300.. no reply from the pod, not all pod in the namesapce are ready this is what I remeber, so I checked each time no resources was found there

[HacksonClark]

Ok. From what I've seen so far it just seems like the agent is failing to solve the problem, that doesn't indicate an error with the benchmark.

[Tanaz-R]

This was the step [19:35:17] Waiting for all pods in namespace 'test-social-network' kubectl.py:93
to be ready...