Robot Automatically Detects Abnormal Factory Noise and Its Source Direction

1. Introduction

At NTT DATA, we are working on automating factory equipment inspections and anomaly detection using robots. In manufacturing sites, regular inspections are essential for equipment maintenance. As facility scale increases, the number of inspection targets grows, requiring more effort and time. Additionally, inspections that are difficult to conduct in locations such as above-head level, ground level, and tight spaces pose risks like falls or slips. To reduce the burden on such worksites, we are researching and developing an equipment inspection system using robots with active sensing. We are building a system where robots autonomously move within facilities, analyze video and audio data captured from inspection targets, and notify workers when anomalies are detected.

This initiative is the result of a co-creation activity with production engineers from Mitsubishi Chemical Corporation (hereinafter "Mitsubishi Chemical"), who are aiming to reform working practices at manufacturing sites. Based on Mitsubishi Chemical's strengths in factory anomaly detection and prediction knowledge, and NTT DATA's technical capabilities in robotics and sensing, we are advancing the development of robot control technology and practical AI technology.

2. Acoustic Analysis Verification

In this article, we set "motor inspection work" as a use case for equipment within facilities and verified real-time acoustic detection and localization technology (hereinafter "this technology") that can be mounted on robots.

Technology Overview

This technology estimates the presence/absence and direction (azimuth angle, elevation angle) of multiple specific sounds of interest (such as abnormal equipment sounds). To estimate the direction of sound arrival, we use multi-channel microphones, utilizing differences in volume, sound arrival time, and sound reflection patterns based on the positioning of the sound source and microphones. In this case, to improve the accuracy of sound source direction estimation, we estimate the sound generation source considering sound reflections through learning the characteristics of unknown environments. By pre-training the system on noise, we can maintain performance even with non-contact microphones that are prone to noise interference. The process consists of two stages-training and inference. During training, we perform model learning and accuracy verification from audio data. During inference, we input the audio to be analyzed and output the presence/absence of specific sounds and their direction.

For training, this technology requires audio data of the target equipment's mechanical sounds recorded at the actual site or similar facilities. To efficiently collect this audio data, there is a method of separately recording the target equipment sounds, environmental sounds (noise), and environmental characteristics, then combining them to create diverse audio data. While the target equipment sounds and environmental sounds (noise) can be collected with any microphone, the sound for learning environmental characteristics including the microphone must be collected with the same multi-channel microphone used during inference. During inference, the target audio (on-site sounds) needs to be collected with the microphone. Pre-recorded sounds can be analyzed, as well as real-time audio data from the microphone.

Figure: Overview of Verification Flow

Verification Content and Results

In this article, we conducted basic verification in a simulation environment in Mitsubishi Chemical's actual factory to determine whether abnormal motor sounds could be detected. We prepared several environmental sounds (noise) including white noise, footsteps, and door opening/closing sounds. We conducted two types of verification: (1) Can we estimate the presence/absence and direction of normal factory motor sounds? (2) Can we distinguish between normal and abnormal factory motor sounds and estimate their respective presence/absence and direction?

For non-experts, it is difficult to distinguish the difference between the sound of normal and abnormal motors. But with the frequency analysis, we can see that abnormal motor sounds show more low-frequency components in the spectrogram, and also lower Spectral Centroid compared to normal motor sounds.

Figure: Difference Between Normal and Abnormal Motor Sounds

(1) Can we estimate the presence/absence and direction of normal factory motor sounds?

Using this technology, we trained and performed inference on normal motor sounds only. In model accuracy verification, the average error in estimated sound azimuth angle was 19.3 degrees, the F-value for sound presence/absence estimation was 99.8%, and the recall value was 99.8%. We achieved better accuracy than the F-value (94.4%) from a paper that trained on 12 types of sounds.

In the analysis, we obtained results that could identify from which direction the motor sound was coming. In one example, when analyzing a video segment with motor sound at azimuth angle 150 degrees and elevation angle 18 degrees, we found that the motor sound was occurring at azimuth angle 151 degrees and elevation angle 22 degrees.

(2) Can we distinguish between normal and abnormal factory motor sounds and estimate their respective presence/absence and direction?

Using this technology, we trained and performed inference on two types of sounds: normal and abnormal motor sounds. In accuracy verification, the average error in estimated sound azimuth angle was 17.2 degrees, the F-value for sound type estimation was 83.5%, and the recall value was 90.3%. Compared to verification (1), the number of target sounds increased and distinguishing between sounds was more difficult, resulting in a lower F-value, but sufficient accuracy was obtained.

In the analysis, similar to verification (1), we obtained results that could identify from which direction the motor sound was coming. In one example, when analyzing a video segment with abnormal motor sound at azimuth angle 140 degrees and elevation angle 0 degrees, we found that the abnormal motor sound occurred at azimuth angle 137 degrees and elevation angle 4 degrees.

3. Future Development and Expectations

Through this verification, we were able to confirm the basic performance of this technology. By using this technology, we can accurately detect the presence/absence and direction of specific sounds even when noise in non-contact microphones and sounds from multiple devices are mixed. Specifically, in factories where sounds from many devices and noise occur, we can detect even sound differences that would be missed by the human ear. This means we can accurately capture abnormal equipment sounds, detect specific events that could lead to accidents or failures at an early stage, and enable effective instructions for inspections and repairs. Going forward, we will explore applicability in various real-world environments, not limited to factories.

Furthermore, by using robots equipped with this technology, safe and accurate remote inspection work becomes possible. As one example, NTT DATA is conducting demonstration experiments for automatic inspection using a lightweight, four-legged dog-type robot with stable motor performance and advanced sensing functions. By pre-installing markers containing ID and position coordinate information on inspection target equipment, the robot can autonomously estimate target locations and perform inspection work. The robot automatically patrols the verification area and acquires video and audio data. The video and audio data are analyzed by a small PC on the robot's back, and finally, the inspection video and analysis results are transmitted to the PC of the operator who issued the inspection instructions.

Figure: Configuration of Automatic Inspection by Robot

In this article, we introduced a case study on acoustic analysis for equipment inspection. At NTT DATA, we aim to reform work styles at manufacturing sites by replacing part of human work with robots. We continue to leverage advanced AI technology and robotics to reduce burdens on workers and accomplish tasks that humans cannot do.

With "Smart Robotics" as our core technical capability, fusing state-of-the-art robotics and AI, we are expanding active sensing applications to include vibration detection. Moving forward, we will deepen our verification efforts with Mitsubishi Chemical's production engineers and seek collaborative opportunities with operators in the chemical, manufacturing, and infrastructure sectors.