Logging

Meet on 2 Apr 2025

Allocation of Student Guides and Projects
Discussion about future roles - Senior Software Engineers, Subsystem Lead (Interview)
2 students are red flagged
Design Report (30 Apr submission deadline) - Collaborate with Ghoshank Bhaiya
Camera Testing on drone before 5th April
Devised a plan of action for CUASC (Excel Sheet)
How to detect GCP?

April 3-4
3D models to make custom models:

Thoroughly followed this video: https://www.youtube.com/watch?v=2XMkPW_sIGg&list=PL5B692fm6--uQRRDTPsJDp4o0xbzkoyf8&index=14

Used bicubic interpolation to increase the sharpness of the live feed video.
References:

Arducam
Preprocessing
Detecting a rotated image: https://developer.nvidia.com/blog/detecting-rotated-objects-using-the-odtk/

April 6 DeepStream SDK is available to run ONXX models with flexible architectures. https://gitub.com/thatbrguy/Deep-Stream-ONXX is a good reference to follow.
#TODO: Find efficient way of using GUI in Jetson Nano from Laptop

number and alphabets detection: https://github.com/winter2897/Real-time-Auto-License-Plate-Recognition-with-Jetson-Nano/blob/main/doc/plate-ocr.md

April 7 We can use the fisheye code : /opt/nvidia/vpi1/samples/11-fisheye : for preprocessing

april 9 -Kisi bewafa ke zakhmo pe marhamn lagane hum chale magar marhum ki kasam mar hum na mila magar mar hum gaye.-

installed miniconda using video

April 10
remove python 3.8
installed zsh
setup credentials in git
Camera did not start anyway
g_strsplit: assertion 'string != NULL' failed
Apr 11 01:32:48 rakshak-desktop gnome-software[6175]: g_strv_length: assertion 'str_array != NULL' failed
Apr 11 01:32:48 rakshak-desktop gnome-software[6175]: json_object_has_member: assertion 'member_name != NULL' failed
Apr 11 01:32:48 rakshak-desktop gnome-software[6175]: g_strsplit: assertion 'string != NULL' failed
Apr 11 01:32:48 rakshak-desktop gnome-software[6175]: g_strv_length: assertion 'str_array != NULL' failed Apr 11 01:32:13 rakshak-desktop kernel: isp 54680000.isp: FIRST_GET 0x188 Apr 11 01:32:13 rakshak-desktop kernel: isp 54680000.isp: TIMEOUT 10000 Apr 11 01:32:13 rakshak-desktop kernel: isp 54680000.isp: NUM_SLOTS 5 Apr 11 01:32:13 rakshak-desktop kernel: isp 54680000.isp: NUM_HANDLES 1 Apr 11 01:32:13 rakshak-desktop nvargus-daemon[4028]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59) Apr 11 01:32:13 rakshak-desktop nvargus-daemon[4028]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc Apr 11 01:32:15 rakshak-desktop dhclient[5251]: DHCPREQUEST of 10.64.72.144 on wlan0 to 127.0.0.1 port 67 (xid=0x38440ad4) Apr 11 01:32:16 rakshak-desktop unity-panel-ser[5731]: window_menu_model_new: assertion 'BAMF_IS_APPLICATION(app)' failed Apr 11 01:32:16 rakshak-desktop unity-panel-ser[5731]: track_menus: assertion 'IS_WINDOW_MENU(menus)' failed Apr 11 01:32:18 rakshak-desktop zeitgeist-daemon[6059]: [20:02:18.190070 WARNING] zeitgeist-daemon.vala:127: Unable to parse version info! Apr 11 01:32:23 rakshak-desktop nvargus-daemon[4028]: === camera-capture[6049]: Connection closed (7FA0C561D0)=== camera-capture[6049]: There were pending client request9 Apr 11 01:32:23 rakshak-desktop nvargus-daemon[4028]: waitForIdleLocked remaining request 103 Apr 11 01:32:23 rakshak-desktop nvargus-daemon[4028]: waitForIdleLocked remaining request 102 Apr 11 01:32:23 rakshak-desktop nvargus-daemon[4028]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 928) Apr 11 01:32:23 rakshak-desktop nvargus-daemon[4028]: (Argus) Error Timeout: (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169) Apr 11 01:32:31 rakshak-desktop dhclient[5251]: DHCPREQUEST of 10.64.72.144 on wlan0 to 127.0.0.1 port 67 (xid=0x38440ad4) Apr 11 01:32:33 rakshak-desktop kernel: wlan0: deauthenticated from 00:dc:b2:5e:bc:90 (Reason: 4=DISASSOC_DUE_TO_INACTIVITY)

April 17 Reflashed jetson https://developer.nvidia.com/embedded-computing
https://developer.nvidia.com/embedded/jetpack/jetson-platform-services-get-started
Tutorial that can be followed: https://developer.nvidia.com/blog/generate-traffic-insights-using-yolov8-and-nvidia-jetpack-6-0/

May 5 Rushikesh and Param ->We tested new cam, port 0 not working. ->Camera quality bad, might need to change ->tried setting up tensorRT engine but faced type casting issues (ONNX simplifier didnt work)

May 6 Rushikesh and Param

https://chatgpt.com/share/681bbbf7-3f94-8004-b045-edfc9c842279 https://chatgpt.com/share/681bcad2-0e50-8010-b423-f7d1ed25a567

Fixed the typecasting issue...while exporting yolo model into onnx format keep the opset parameter = 20

To run the engine following this link - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/quick-start-guide.html#deploying-a-tensorrt-engine-to-the-python-runtime-api Facing issue in installation of pycuda

May 7 Rushikesh and Param import cv2 import numpy as np import pycuda.driver as cuda import pycuda.autoinit import tensorrt as trt import time

=== Constants ===

INPUT_SHAPE = (1, 3, 640, 640) TARGET_DTYPE = np.float32 CLASS_LABEL = "MyObject"

=== TensorRT Wrapper ===

class ONNXClassifierWrapper: def init(self, file): self.target_dtype = TARGET_DTYPE self.load(file) self.stream = None

def load(self, file):
    with open(file, "rb") as f:
        runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))
        self.engine = runtime.deserialize_cuda_engine(f.read())
        self.context = self.engine.create_execution_context()

def allocate_memory(self, batch):
    self.output = np.empty(self.engine.get_binding_shape(1), dtype=self.target_dtype)
    self.d_input = cuda.mem_alloc(batch.nbytes)
    self.d_output = cuda.mem_alloc(self.output.nbytes)
    self.bindings = [int(self.d_input), int(self.d_output)]
    self.stream = cuda.Stream()

def predict(self, batch):
    if self.stream is None:
        self.allocate_memory(batch)
    cuda.memcpy_htod_async(self.d_input, batch, self.stream)
    self.context.execute_async_v2(bindings=self.bindings, stream_handle=self.stream.handle)
    cuda.memcpy_dtoh_async(self.output, self.d_output, self.stream)
    self.stream.synchronize()
    return self.output

=== Preprocessing ===

def preprocess_frame(frame): resized = cv2.resize(frame, (640, 640)) normalized = resized.astype(np.float32) / 255.0 transposed = normalized.transpose(2, 0, 1) # HWC to CHW batched = np.expand_dims(transposed, axis=0) return np.ascontiguousarray(batched.astype(TARGET_DTYPE)) # Ensure contiguity

=== NMS & Postprocessing (NumPy only) ===

def xywh_to_xyxy(boxes): x1 = boxes[:, 0] - boxes[:, 2] / 2 y1 = boxes[:, 1] - boxes[:, 3] / 2 x2 = boxes[:, 0] + boxes[:, 2] / 2 y2 = boxes[:, 1] + boxes[:, 3] / 2 return np.stack([x1, y1, x2, y2], axis=1)

def compute_iou(box, boxes): x1 = np.maximum(box[0], boxes[:, 0]) y1 = np.maximum(box[1], boxes[:, 1]) x2 = np.minimum(box[2], boxes[:, 2]) y2 = np.minimum(box[3], boxes[:, 3]) inter_area = np.maximum(x2 - x1, 0) * np.maximum(y2 - y1, 0) box_area = (box[2] - box[0]) * (box[3] - box[1]) boxes_area = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1]) union_area = box_area + boxes_area - inter_area return inter_area / (union_area + 1e-6)

def nms_numpy(boxes, scores, iou_thresh=0.45): indices = scores.argsort()[::-1] keep = [] while indices.size > 0: current = indices[0] keep.append(current) if indices.size == 1: break ious = compute_iou(boxes[current], boxes[indices[1:]]) indices = indices[1:][ious < iou_thresh] return keep

def postprocess_output_np(output, conf_thresh=0.25, iou_thresh=0.45): detections = np.array(output).reshape(-1, 6) boxes = detections[:, :4] scores = detections[:, 4] * detections[:, 5] mask = scores > conf_thresh boxes = boxes[mask] scores = scores[mask] if len(boxes) == 0: return [], [] boxes_xyxy = xywh_to_xyxy(boxes) keep_indices = nms_numpy(boxes_xyxy, scores, iou_thresh) return boxes_xyxy[keep_indices], scores[keep_indices]

=== Drawing Boxes ===

def draw_boxes(frame, boxes, scores): for box, score in zip(boxes, scores): x1, y1, x2, y2 = map(int, box) cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2) label = f"{CLASS_LABEL}: {score:.2f}" cv2.putText(frame, label, (x1, y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1) return frame

=== Main Loop ===

def main(): engine_path = "yolo_engine.trt" # Path to your YOLO TensorRT engine trt_model = ONNXClassifierWrapper(engine_path)

cap = cv2.VideoCapture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=30/1 ! nvvidconv flip-method=0 ! video/x-raw, width=640, height=640, format=BGRx ! videoconvert ! appsink", cv2.CAP_GSTREAMER)

if not cap.isOpened():
    print("Failed to open CSI camera.")
    return

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Preprocess the frame
    input_tensor = preprocess_frame(frame)

    # Get the model output
    output = trt_model.predict(input_tensor)

    # Postprocess the output (NMS)
    boxes, scores = postprocess_output_np(output)

    # Draw bounding boxes on the frame
    frame = draw_boxes(frame, boxes, scores)

    # Show the frame
    cv2.imshow("YOLO Real-Time Detection", frame)

    # Exit on pressing 'q'
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

if name == "main": main()

https://chatgpt.com/share/681e897b-f684-8010-af42-aa8cbdfa30fb

import cv2 import numpy as np import pycuda.driver as cuda import pycuda.autoinit import tensorrt as trt import time

=== Constants ===

INPUT_SHAPE = (640, 640) # Model input resolution CLASS_LABEL = "MyObject" # Single class label CONF_THRESH = 0.25 # Confidence threshold IOU_THRESH = 0.45 # NMS IOU threshold ENGINE_PATH = "yolo_engine.trt" # Path to TensorRT engine

=== TensorRT Engine Wrapper ===

class TRTEngineWrapper: def init(self, engine_path): self.logger = trt.Logger(trt.Logger.WARNING) self.engine = self.load_engine(engine_path) self.context = self.engine.create_execution_context()

    # Get binding shapes dynamically
    self.input_shape = self.engine.get_binding_shape(0)
    self.output_shape = self.engine.get_binding_shape(1)
    
    # Allocate memory
    self.inputs, self.outputs, self.bindings, self.stream = self.allocate_buffers()

def load_engine(self, engine_path):
    with open(engine_path, "rb") as f:
        runtime = trt.Runtime(self.logger)
        return runtime.deserialize_cuda_engine(f.read())

def allocate_buffers(self):
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()
    
    for binding in self.engine:
        size = trt.volume(self.engine.get_binding_shape(binding)) * self.engine.max_batch_size
        dtype = trt.nptype(self.engine.get_binding_dtype(binding))
        
        # Allocate host and device buffers
        host_mem = cuda.pagelocked_empty(size, dtype)
        device_mem = cuda.mem_alloc(host_mem.nbytes)
        
        bindings.append(int(device_mem))
        
        if self.engine.binding_is_input(binding):
            inputs.append({'host': host_mem, 'device': device_mem})
        else:
            outputs.append({'host': host_mem, 'device': device_mem})
    
    return inputs, outputs, bindings, stream

def infer(self, input_tensor):
    # Copy input to pagelocked memory
    np.copyto(self.inputs[0]['host'], input_tensor.ravel())
    
    # Transfer data to GPU and run inference
    cuda.memcpy_htod_async(self.inputs[0]['device'], self.inputs[0]['host'], self.stream)
    self.context.execute_async_v2(
        bindings=self.bindings,
        stream_handle=self.stream.handle
    )
    # Copy outputs back
    cuda.memcpy_dtoh_async(self.outputs[0]['host'], self.outputs[0]['device'], self.stream)
    self.stream.synchronize()
    
    return self.outputs[0]['host'].reshape(self.output_shape)

=== Image Processing ===

def preprocess(frame): # Resize and normalize resized = cv2.resize(frame, INPUT_SHAPE) normalized = resized.astype(np.float32) / 255.0 # HWC to CHW and add batch dimension return np.ascontiguousarray(normalized.transpose(2, 0, 1)[np.newaxis])

=== Post-processing ===

def xywh2xyxy(x): # Convert nx4 boxes from [x, y, w, h] to [x1, y1, x2, y2] y = np.copy(x) y[:, 0] = x[:, 0] - x[:, 2] / 2 # x1 y[:, 1] = x[:, 1] - x[:, 3] / 2 # y1 y[:, 2] = x[:, 0] + x[:, 2] / 2 # x2 y[:, 3] = x[:, 1] + x[:, 3] / 2 # y2 return y

def nms(boxes, scores, thresh): # Argsorted indices of scores in descending order order = scores.argsort()[::-1] keep = []

while order.size > 0:
    i = order[0]
    keep.append(i)
    
    # Calculate IoU between current box and remaining boxes
    xx1 = np.maximum(boxes[i, 0], boxes[order[1:], 0])
    yy1 = np.maximum(boxes[i, 1], boxes[order[1:], 1])
    xx2 = np.minimum(boxes[i, 2], boxes[order[1:], 2])
    yy2 = np.minimum(boxes[i, 3], boxes[order[1:], 3])
    
    w = np.maximum(0.0, xx2 - xx1)
    h = np.maximum(0.0, yy2 - yy1)
    intersection = w * h
    
    iou = intersection / ((boxes[i, 2]-boxes[i, 0])*(boxes[i, 3]-boxes[i, 1]) + 
                          (boxes[order[1:], 2]-boxes[order[1:], 0])*(boxes[order[1:], 3]-boxes[order[1:], 1]) - 
                          intersection + 1e-6)
    
    # Keep indices with IoU <= threshold
    inds = np.where(iou <= thresh)[0]
    order = order[inds + 1]

return np.array(keep)

def postprocess(output): # Reshape output: (1, 5, 8400) -> (8400, 5) predictions = output[0].transpose(1, 0)

# Filter by confidence threshold
conf_mask = predictions[:, 4] > CONF_THRESH
boxes_conf = predictions[conf_mask]

if boxes_conf.shape[0] == 0:
    return np.array([]), np.array([])

# Split into boxes and scores
boxes = boxes_conf[:, :4]
scores = boxes_conf[:, 4]

# Convert to xyxy format
boxes_xyxy = xywh2xyxy(boxes)

# Apply NMS
keep_indices = nms(boxes_xyxy, scores, IOU_THRESH)

return boxes_xyxy[keep_indices], scores[keep_indices]

=== Visualization ===

def draw_boxes(frame, boxes, scores): if boxes.size == 0: return frame

# Rescale boxes to original frame size
h, w = frame.shape[:2]
scale = min(INPUT_SHAPE[0]/w, INPUT_SHAPE[1]/h)
pad_x = (INPUT_SHAPE[0] - w * scale) / 2
pad_y = (INPUT_SHAPE[1] - h * scale) / 2

boxes[:, [0, 2]] = (boxes[:, [0, 2]] - pad_x) / scale
boxes[:, [1, 3]] = (boxes[:, [1, 3]] - pad_y) / scale

# Clip coordinates to image dimensions
boxes[:, [0, 2]] = boxes[:, [0, 2]].clip(0, w)
boxes[:, [1, 3]] = boxes[:, [1, 3]].clip(0, h)

for box, score in zip(boxes, scores):
    x1, y1, x2, y2 = map(int, box)
    cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
    label = f"{CLASS_LABEL}: {score:.2f}"
    cv2.putText(frame, label, (x1, y1 - 5), 
               cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 0), 2, 
               cv2.LINE_AA)

return frame

=== Main Processing Loop ===

def main(): # Initialize TensorRT engine trt_engine = TRTEngineWrapper(ENGINE_PATH)

# Initialize camera input
cap = cv2.VideoCapture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=1280, height=720, "
                       "format=NV12, framerate=30/1 ! nvvidconv flip-method=0 ! video/x-raw, "
                       "width=640, height=640, format=BGRx ! videoconvert ! appsink", 
                       cv2.CAP_GSTREAMER)

if not cap.isOpened():
    raise RuntimeError("Failed to open camera input")

try:
    while True:
        # Capture frame
        ret, frame = cap.read()
        if not ret:
            break
        
        # Preprocess
        input_tensor = preprocess(frame)
        
        # Inference
        start = time.perf_counter()
        output = trt_engine.infer(input_tensor)
        inference_time = (time.perf_counter() - start) * 1000
        
        # Postprocess
        boxes, scores = postprocess(output)
        
        # Draw results
        frame = draw_boxes(frame, boxes, scores)
        
        # Display inference time
        cv2.putText(frame, f"Inference: {inference_time:.1f}ms", (10, 30),
                   cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
        
        # Show output
        cv2.imshow("YOLOv8 TensorRT Inference", frame)
        
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
            
finally:
    cap.release()
    cv2.destroyAllWindows()

if name == "main": main()

=== Harshil, Kashish, Param, Rushikesh here ===

https://developer.download.nvidia.com/embedded/L4T/r28_Release_v3.0/Docs/Jetson_TX1_and_TX2_Accelerated_GStreamer_User_Guide.pdf?sUVleY8WwxW7crftAqDJhILK8XW4X7QiDXH5_1EM-VVPYXnaFRHW36WXwlxMx3TE3qOK0GRQWAqzFweWS4Z0oiqKHjxqs-BasmvZOGvP9wNmAKCA6kmkagynGK9odZ__TJuXM70yYE49F8rRkrv6_fR6SQH3YHrrfgUr2G2rmfcjbZgiN1ktTM1-EbQ5veDxaxaatX7wJVg2#page=33&zoom=100,141,96

GUIDE TO SAVE CAMERA OUTPUT: https://docs.nvidia.com/jetson/archives/r35.5.0/DeveloperGuide/SD/CameraDevelopment/CameraSoftwareDevelopmentSolution.html

Helpful Projects

Jetcam: https://github.com/NVIDIA-AI-IOT/jetcam.git
Jetcard: https://github.com/NVIDIA-AI-IOT/jetcard.git
Torch to tensorRT: https://github.com/NVIDIA-AI-IOT/torch2trt.git
Jetracer: https://github.com/NVIDIA-AI-IOT/jetracer.git
Other Projects: https://developer.nvidia.com/embedded/community/jetson-projects?sortBy=jetson_community_projects%2Fsort%2Fdate_added%3Adesc&query=drone

14th May

Camera specifications are: https://robu.in/product/arducam-imx219-wide-angle-camera-module-for-nvidia-jetson-nano-raspberry-pi/?gad_source=1&gclid=Cj0KCQjwzYLABhD4ARIsALySuCT9_FKAW4FWgeO1QNSUnK1EW84CRtIxR_P8L5XH3ZRRQaYWURIfCDYaAkfbEALw_wcB
Focus can be set by rotating the lens.
VNC Viewer- Setup done using official website instruction
Camera Setup done using jetcam library which uses Argus gstreamer
Video recording code done and saved at Desktop/capture/runcam.py

Observation:

1080p image is cropped out/ smaller tha 8MP image
Radial undistortion using cv2 did not work out

Plan of action:

Undistort the image from the camera feed
Scanning or stitching algorithm

15th May

Went for drone testing, could not mount Jetson

16th May

Unable to book flights, drone crashed
Plan discussed, gave GCP statement to freshers

17th May

Pink tint of camera removed: https://jonathantse.medium.com/fix-pink-tint-on-jetson-nano-wide-angle-camera-a8ce5fbd797f
RPI camera not compatible detemined

18th May

Setup mount for camera
Capture code edited
Distortion removal failed
Research on camera: https://blog.arducam.com/focal-length-calculator/
Finalised alternative camera: https://robu.in/product/arducam-8mp-imx219-low-distortion-m12-camera-module-for-jetson-nano/
Searched for Jetson to Pixhawk communication, found nice references and other projects:
https://www.hackster.io/jonmendenhall/jetson-nano-search-and-rescue-ai-uav-9ca547
https://www.hackster.io/salahuddin/nvidia-jetson-nano-with-csi-camera-9c6c59
https://www.hackster.io/SaadTiwana/embedded-diaries-jetson-gstreamer-video-encoding-decoding-585ba7
https://www.hackster.io/pattshibang/running-yolov12-on-jetson-nano-4gb-a-comprehensive-guide-f9042e
https://www.hackster.io/siyuxu/exploring-slam-algorithms-on-myagv-jetson-nano-integration-b7c51d

19th May

Retrained Yolo model using dataset made by freshers for GCP
Dataset validation of freshers
Mavlink setup and Mavproxy connected but dronekit not connected

20th May

Changed dronekit source code to debug connection issue, the problem was that the baudrate was low hence the connection got timed out, changed baud rate and connected wirelessly to build up cache, capturing parameters faster and resolving the timeout error. baud rate: 57600 -> 115200
Bullseye code tested after successful dronekit communication, centering code seemed to work.

Name		Name	Last commit message	Last commit date
All_about_onnx		All_about_onnx
Preprocessing.txt		Preprocessing.txt
Readme.md		Readme.md
arducam-specification.txt		arducam-specification.txt
run_tensorrt.py		run_tensorrt.py

Navigation Menu

Repository navigation

About

Repository files navigation

Logging

=== Constants ===

=== TensorRT Wrapper ===

=== Preprocessing ===

=== NMS & Postprocessing (NumPy only) ===

=== Drawing Boxes ===

=== Main Loop ===

=== Constants ===

=== TensorRT Engine Wrapper ===

=== Image Processing ===

=== Post-processing ===

=== Visualization ===

=== Main Processing Loop ===

=== Harshil, Kashish, Param, Rushikesh here ===

Helpful Projects

14th May

Observation:

15th May

16th May

17th May

18th May

19th May

20th May

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Logging

=== Constants ===

=== TensorRT Wrapper ===

=== Preprocessing ===

=== NMS & Postprocessing (NumPy only) ===

=== Drawing Boxes ===

=== Main Loop ===

=== Constants ===

=== TensorRT Engine Wrapper ===

=== Image Processing ===

=== Post-processing ===

=== Visualization ===

=== Main Processing Loop ===

=== Harshil, Kashish, Param, Rushikesh here ===

Helpful Projects

14th May

Observation:

15th May

16th May

17th May

18th May

19th May

20th May

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages