The lecture Robotic Intelligence: Perception and Scene Understanding combines modern AI-based perception methods with the engineering skills needed to apply them in robotics. It covers the full perception chain, from acquiring and calibrating sensor data to interpreting scenes and integrating perception modules into robotic systems, like mobile walking robots.
Students will first build a foundation in machine learning and computer vision, covering image classification, object detection, tracking, and segmentation in both 2D and 3D. The course then explores advanced deep learning architectures, including Vision Transformers, multimodal perception systems, and end-to-end multimodal machine learning pipelines. Special emphasis is placed on techniques for integrating vision, depth, and other sensing modalities, enabling robust perception in challenging real-world environments.

A key part of the course is practical group work in the Machine Intelligence and Robotics Lab (MaiRo), where students solve hands-on tasks using real sensors and robots, building both technical competence and teamwork skills.

Topics include:

• Fundamentals of machine learning and computer vision for robotics
• Vision Transformers and advanced deep learning architectures
• Object detection, tracking, and segmentation (2D and 3D)
• Instance, semantic, and panoptic segmentation
• Multimodal sensing, fusion, and end-to-end multimodal ML
• World models and their role in robotics
• Sensor modeling, calibration, and perception system integration

By the end of the course, students will be able to design, implement, and integrate intelligent robotic perception systems that enable mobile robots to understand and act within complex, dynamic environments.

Time and place see ILIAS