Broker didn't receive the message from inference service

[uraychang]

Hello team, I was following the instruction here to setup the integration of inference service and drift detector (on my local minikube). If I understand it right, the broker captures the logs from inference service. Then detector as the subscriber consumes the message from broker. So as the first step, I wanna make sure the event is published successfully by the inference service. But, there's no any log in broker ingress pod after I triggered a request to the inference service. Same for the event-display service which is deployed for testing the message consumption. I've check all the status of my components. They looked up and running well. Hope anyone with any idea can provide some thoughts, thanks! Willing to provide more info if needed.

K8s resources:

% k get pods
NAME                                                   READY   STATUS    RESTARTS   AGE
debug-pod                                              1/1     Running   0          30m
hello-display-85f6789687-vttj7                         1/1     Running   0          97m
iris-demo-predictor-00001-deployment-d875664d8-n6dv6   3/3     Running   0          103m

% k get pods -n knative-eventing
NAME                                    READY   STATUS    RESTARTS       AGE
eventing-controller-cc45869cd-lfht8     1/1     Running   3 (112m ago)   131m
eventing-webhook-67fcc6959b-gqnmq       1/1     Running   4 (112m ago)   131m
imc-controller-78848895c-sw88q          1/1     Running   5 (112m ago)   128m
imc-dispatcher-79f95cc86c-p2b6g         1/1     Running   6 (111m ago)   128m
job-sink-7f5d754db-qjm4g                1/1     Running   6 (111m ago)   131m
mt-broker-controller-5f64b86f8c-7kxgk   1/1     Running   0              128m
mt-broker-filter-5b46f6945d-ffm2j       1/1     Running   6 (112m ago)   128m
mt-broker-ingress-58ffc5448-vxmcs       1/1     Running   0              128m

% k get triggers.eventing.knative.dev event-display-trigger
NAME                    BROKER    SUBSCRIBER_URI                                   AGE   READY   REASON
event-display-trigger   default   http://hello-display.default.svc.cluster.local/   58m   True

% k get brokers.eventing.knative.dev default
NAME      URL                                                                        AGE    READY   REASON
default   http://broker-ingress.knative-eventing.svc.cluster.local/default/default   107m   True

Inference yaml:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: iris-demo
spec:
  predictor:
    imagePullSecrets:
        - name: my-dockerhub-secret
    model:
      modelFormat:
        name: sklearn
      storageUri: kit://docker.io/<my-docker-repo>/iris-demo:sk-learn
    logger:
      mode: all
      url: http://broker-ingress.knative-eventing.svc.cluster.local/default/default
     

Broker yaml

apiVersion: eventing.knative.dev/v1
kind: broker
metadata:
 name: default

Event-display yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-display
spec:
  replicas: 1
  selector:
    matchLabels: &labels
      app: hello-display
  template:
    metadata:
      labels: *labels
    spec:
      containers:
        - name: event-display
          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display

---

kind: Service
apiVersion: v1
metadata:
  name: hello-display
spec:
  selector:
    app: hello-display
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Event-display trigger yaml

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: event-display-trigger
spec:
  broker: default
  # filter:
  #   attributes:
  #     type: org.kubeflow.serving.inference.request
  subscriber:
    ref:
      apiVersion: v1
      kind: Service
      name: hello-display

[mg1986jp]

If the inference service is sending CloudEvents to the Broker but the Broker isn't receiving them, there are a few things to check:

1. Make sure the inference service is sending events to the correct Broker URL. You can get it with:

kubectl get broker <broker-name> -o jsonpath='{.status.address.url}'

2. Check if the event is being sent as a proper CloudEvent. The Broker expects either binary or structured content mode with the required CE headers (ce-type, ce-source, ce-id, ce-specversion).

3. Look at the Broker's ingress pod logs:

kubectl logs -l app=kafka-broker-ingress -n knative-eventing

4. If you're using the Kafka Broker, check that the Kafka topic exists and is healthy:

kubectl get kafkatopic -n knative-eventing

5. Network policy / service mesh might be blocking the connection between the inference service and the Broker ingress. Try a quick curl test from the inference service pod to the Broker URL.

What error (if any) does the inference service show when it tries to send the event?