LLM-Generated XR Content for Manufacturing Safety
In the manufacturing sector, the use of Robots is becoming increasingly important in meeting current production requirements. However, the demand for larger and heavier Robots necessitates the effective safety training of Robot Operators (ROs). eXRercise is designed to prepare ROs for 2 types of potential hazards in the workplace, namely Small-Scale fires and Control software errors, such as malfunctioning robotic arms. At the forefront of Industry 4.0, the vision of eXRercise is to integrate Extended Reality (XR) technology with Machine Learning (ML) to revolutionize training methodologies. Specifically, the project aims to (A) enable trainers to intuitively create – using natural language – ultra-realistic educational VR safety training scenes for ROs and (B) assess the trainee’s performance and the impact of a training scenario on their stress levels to provide suggestions for the adjustment of the training scenes’ difficulty in real time, utilizing an autonomous AI assistant. The mission of eXRercise is to ensure that the personnel of the industrial sector is well trained for hazard scenarios in manufacturing lines. Within eXRercise, we plan to enhance the safety of ROs by modernizing the standard, class-based, training methodologies, and instead offer immersive VR experiences that will expose the RO to real-life situations.
Technology
Throughout the project eXRercise lifecycle, three major technologies will be developed to allow for the effortless deployment of immersive XR training scenes that will facilitate the effective safety training of ROs in different hazardous scenarios.
- Natural Language Processing (NLP) Algorithm
- Complete Validation Tool for Trainees’ Health State
- AI Assistant for Performance Assessment and recommendations
First, a comprehensive Natural Language Processing (NLP) Algorithm will facilitate the implementation of a Language-to-Scene Component. With this advancement, workplace safety trainees can easily deploy XR training scenes by simply describing the scenario specifications in a prompt, which will be automatically transformed into a training scenario.
Another major aspiration of eXRercise is to create a Complete Validation Tool for Trainees’ Health State, measuring -through specialized wearable sensors – their ElectoDermic Activity (EDA) and heart rate in an endeavour to estimate their stress level. This data is expected to be useful in adjusting the scene’s training difficulty.
The final major technology of eXRercise, is the Development of an AI Assistant for Performance Assessment and recommendations. When the wearable measures high stress levels or the trainee underperforms, the AI assistant will autonomously inform the trainee and provide recommendations to the trainer for the training scenes’ re-configuration, to match the trainee’s condition.
The solutions that eXRercise is aspiring to bring forward, are expected to be fully integrated with MASTER’s platform and be utilized for the effective development of educational materials in the project’s next open call. In accordance to the project’s specifications various technologies will be supported like OpenXR, integration with VIROO and others for creating virtual and immersive content and experiences.
