Semantic drawn robot language a method to extract robotic programs from physically drawn symbols

Abstract

This paper builds on existing research that have developed methods and software to instruct industrial robot arms through physical drawings made on workpieces. Previous research presented two primary findings: (1) a calibration and implementation method for a visual system that isolates drawings, creates a digital twin, and allows operators to augment it with fabrication information through a developed software. And (2) a drawing training pipeline, where drawings can be made digitally in a software, labelled, stored in a database, and used to train a Machine Learning pipeline to classify physical drawings. In building upon these findings, this research presents a method to convert physically drawn symbols into robotic fabrication sequences by using computer vision algorithms and kMeans clustering. The method consists of two computational layers, which first utilize computer vision and machine learning algorithms to segment, isolate and classify drawn symbols found in an image. Secondly, computer vision algorithms extract geometric information from the drawn symbol, before kMeans clustering cleans the geometric data enabling to convert it to robotic fabrication sequences. The method is demonstrated through the manufacturing of a wall section using a full-scale robotic setup, consisting of an ABB IRB 6620, equipped with a webcam and 5.6 KW spindle with a 20 mm routing bit.