Description

The ROBOTWIN project (Digital Twin of a Mobile Service Robot for Construction Sites) focuses on enabling people without specialized robotics knowledge to program mobile robots for construction tasks. Using a photorealistic digital twin and natural-language interaction, users will be able to specify workflows intuitively and execute them on physical robots using AI-based methods. The goal is to make robotics more accessible and adaptable to the dynamic and unstructured environments found on construction sites.

A central challenge is reliably translating natural-language instructions into executable, physically feasible robot plans. At the same time, robots must autonomously perform demanding tasks such as painting and grinding while achieving high execution quality. For example, painting requires the robot to account for obstacles such as window frames and electrical outlets while ensuring an even, drip-free application. Another key challenge is transferring methods developed in simulation to physical robots operating on real construction sites, where changing environmental conditions and unexpected obstacles can compromise reliable execution.

ROBOTWIN addresses these challenges through AI-based task and motion planning. Vision-Language-Action (VLA) models will be investigated for interpreting natural-language instructions and translating them into executable robot plans. For the reliable execution of individual construction tasks, the project will explore VLA models, reinforcement learning (RL), and hybrid approaches. The focus is not only on completing tasks successfully but also on achieving high-quality results while respecting task-specific requirements and constraints

Another key aspect is transferring the developed methods from simulation to the physical robot (sim-to-real transfer). The resulting approaches should operate robustly under the dynamic and unstructured conditions of real construction sites. While the project partners are responsible for the digital twin and semantic representation of the environment, KIT focuses on AI-based planning, reliable execution of construction tasks, and bridging the sim-to-real gap.

Ultimately, ROBOTWIN aims to lower the barriers to adopting AI-driven robotics in construction and enable intuitive, flexible programming of mobile robots for demanding on-site tasks.