3D Vision-Based AI For Advanced Robotics

“Mech-Mind is a young and promising company. It has very interesting technologies, works in a booming area and has received investments from top ventur3e capital companies. On the other hand, Siemens has been providing advanced simulation tools for production since the 1980s, and we see Mech-Mind as a strategic partner that we want to grow with”, said Dr. Vincent Guo, Strategy and Portfolio Management, Digital Manufacturing, at Siemens Digital Industries Software.

Mech-Mind used to have a traditional design process starting with the CAD drawing based on customer requirements. As the design of the robot solution is different every time due to different customer needs, the work relies heavily on engineers’ experience to identify potential problems at the design phase. In addition, 3D vision-based robotic applications are relatively new to the industry, therefore experienced engineers are even harder to find. At the design phase, many things should be considered. The engineers must consider the field of view of the camera, the positioning tolerance of various components, the clearance between equipment and parts, the singularity of the robot arms, etc. However, due to the lack of good optimization and verification methods, many things can still go wrong even for an experienced engineer. For example, the mechanical design of a gripper may be improper, which leads to many potential collisions or, the robot base is too high or too low, which limits the robot’s reachability. During the commissioning phase at the shop floor it is not uncommon that the engineers realize these problems and must revise the original design. All those combined would lead to increased cost, late delivery, longer on-site service durations, lower customer satisfaction, and even failure of customer engagements.

Mech-Mind began to work with Process Simulate in the Tecnomatix portfolio of digital manufacturing software from Siemens to upgrade its workflow. Once the CAD model is complete, their engineers build the digital twin for the robot cell and manufacturing process in Process Simulate, including 3D cameras, robot and gripper, parts, and other relevant manufacturing resources. The design engineer can test the camera position to make sure all parts can be properly covered by the camera field of view. The optimal location of the robot base can be calculated automatically by Process Simulate once the target locations are defined. These would allow the engineer to quickly verify the layout of the cell and use it for drafting the proposal or generate a bidding document.

One key innovation Mech-Mind provides to the market is that it allows the robot to automatically figure out which part to grab in a randomly placed pile and then execute the grab. This is enabled by Mech-Mind’s core technology in the 3D vision camera and the artificial intelligence software that processes the 3D images. Its featured deep learning engine is utilized for object classification, segmentation, and defect detection.

With Siemens’ support, Mech-Mind took their 3D vision-based AI for robot control to the next level. Based on the CAD file of the parts to handle, Process Simulate can generate randomly placed parts based on a physics engine and then generate synthetic images of the virtual 3D camera just as if the camera is viewing the real world. Camera pixel resolution, noise, ambient lighting effect, etc., can also be added to the synthetic image generation to mimic the actual camera behavior. The synthetic images can be channeled to Mech-Mind’s image processing and AI training software. In addition, since the synthetic images are automatically annotated the deep learning algorithm can also benefit from a perfect training data set.

Now, users of Mech-Mind’s camera and software enjoy the benefit of a comprehensive digital twin. The engineers can simulate the whole process upfront in Process Simulate to check the process logic and the layout of the robot station. It gives the engineers an opportunity to improve the robot cell layout and gripper design early if they discover that collisions frequently occur during the picking process. With the synthetic images, the AI-guided automatic robot path planning is feasible without collecting data from the physical robot cell. Engineers can test if the AI-guided robotic control scheme can achieve the target cycle time and pick rate required by the customer. Optimization of the AI algorithm and the robot control flow can be done completely within the software environment and immediately validated using Process Simulate.

“Process Simulate is a very powerful tool and by collaborating with Siemens we manage to put our 3D vision AI software, Mech-Vision, together with Process Simulate so that we can design the whole robot cell, which has AI and 3D vision capabilities. And the coolest thing is that we can simulate and validate the process as if we have already built the whole cell on the shop floor. And now we can simulate 3D vision and randomly positioned objects in Process Simulate then run AI image recognition software and grasp and motion planning algorithms to simulate the entire process. A lot of potential errors can be found and solved when you run the project in Process Simulate”, stated Shao Tianlan, Founder and CEO of Mech-Mind Robotics Technologies.

“The value is obvious when using Process Simulate. We have achieved shorter project cycles and made far less mistakes with less costs on the shop floor. And we can use simulated data to improve our AI algorithm, so both design and deployment phases become easier”, continued Mr. SHAO. With the full simulation capacity provided by the digital twin technology, it was estimated that one can save up to 40% of total project time and 20% of total project costs.

“I enjoyed the collaboration very much with Mech-Mind. I believe the future of manufacturing can only be achieved by partnerships. On one end, we have mature technologies from Siemens, such as industrial simulation tools and automation systems. On the other end, we have companies that master emerging technologies like Mech-Mind. We form partnerships to maximize the value that we bring to our customers in the manufacturing sector”, said Alex Greenberg, Director of Robotics 4.0, Digital Manufacturing, at Siemens Digital Industries Software.

For more information: www.plm.automation.siemens.com