Modeling Human Tasks and Motion in 360-Degree Videos for Real-Time Digital Twin Application

Abstract

This research presents a framework for real-time task detection and digital twin modeling based on human posture estimation from 360° videos. The system integrates markerless human posture estimation with task classification and digital twin visualization. Posture estimation using MediaPipe provided accurate skeletal tracking, while kinematic feature extraction enabled detailed motion analysis. Gaussian Mixture Models effectively segmented task transitions, distinguishing between different phases of ladder use. Gaussian Splatting helped realistic and adaptive visualizations, for a digital twin that accurately represented human-environment interactions. Using these techniques, the framework achieves a non-intrusive and scalable approach to task detection and digital twin modeling. The system captured human movements from 360° videos and classified them into task-specific segments. The results demonstrated that task detection based on posture estimation could improve workplace safety by identifying inefficient or hazardous postures. The digital twin representation can be analyzed for movement patterns and ergonomic risk assessment.