Audiovisual speech recognition is an emerging research topic.Lipreading is the recognition of what someone is saying using visual information,primarily lip movements.In this study,we created a custom dataset for India...Audiovisual speech recognition is an emerging research topic.Lipreading is the recognition of what someone is saying using visual information,primarily lip movements.In this study,we created a custom dataset for Indian English linguistics and categorized it into three main categories:(1)audio recognition,(2)visual feature extraction,and(3)combined audio and visual recognition.Audio features were extracted using the mel-frequency cepstral coefficient,and classification was performed using a one-dimension convolutional neural network.Visual feature extraction uses Dlib and then classifies visual speech using a long short-term memory type of recurrent neural networks.Finally,integration was performed using a deep convolutional network.The audio speech of Indian English was successfully recognized with accuracies of 93.67%and 91.53%,respectively,using testing data from 200 epochs.The training accuracy for visual speech recognition using the Indian English dataset was 77.48%and the test accuracy was 76.19%using 60 epochs.After integration,the accuracies of audiovisual speech recognition using the Indian English dataset for training and testing were 94.67%and 91.75%,respectively.展开更多
In this paper,a new bolt fault diagnosis method is developed to solve the fault diagnosis problem of wind turbine flange bolts using one-dimensional depthwise separable convolutions.The main idea is to use a one-dimen...In this paper,a new bolt fault diagnosis method is developed to solve the fault diagnosis problem of wind turbine flange bolts using one-dimensional depthwise separable convolutions.The main idea is to use a one-dimensional convolutional neural network model to classify and identify the acoustic vibration signals of bolts,which represent different bolt damage states.Through the methods of knock test and modal simulation,it is concluded that the damage state of wind turbine flange bolt is related to the natural frequency distribution of acoustic vibration signal.It is found that the bolt damage state affects the modal shape of the structure,and then affects the natural frequency distribution of the bolt vibration signal.Therefore,the damage state can be obtained by identifying the natural frequency distribution of the bolt acoustic vibration signal.In the present one-dimensional depth-detachable convolutional neural network model,the one-dimensional vector is first convolved into multiple channels,and then each channel is separately learned by depth-detachable convolution,which can effectively improve the feature quality and the effect of data classification.From the perspective of the realization mechanism of convolution operation,the depthwise separable convolution operation has fewer parameters and faster computing speed,making it easier to build lightweight models and deploy them to mobile devices.展开更多
摘要Audiovisual speech recognition is an emerging research topic.Lipreading is the recognition of what someone is saying using visual information,primarily lip movements.In this study,we created a custom dataset for Indian English linguistics and categorized it into three main categories:(1)audio recognition,(2)visual feature extraction,and(3)combined audio and visual recognition.Audio features were extracted using the mel-frequency cepstral coefficient,and classification was performed using a one-dimension convolutional neural network.Visual feature extraction uses Dlib and then classifies visual speech using a long short-term memory type of recurrent neural networks.Finally,integration was performed using a deep convolutional network.The audio speech of Indian English was successfully recognized with accuracies of 93.67%and 91.53%,respectively,using testing data from 200 epochs.The training accuracy for visual speech recognition using the Indian English dataset was 77.48%and the test accuracy was 76.19%using 60 epochs.After integration,the accuracies of audiovisual speech recognition using the Indian English dataset for training and testing were 94.67%and 91.75%,respectively.
基金supported in part by the National Key R&D Program of China(Nos.2021YFE0206100 and 2018YFB1702300)the National Natural Science Foundation of China(No.62073321)+1 种基金the National Defense Basic Scientific Research Program(No.JCKY2019203C029)the Science and Technology Development Fund,Macao SAR(No.0015/2020/AMJ).
摘要In this paper,a new bolt fault diagnosis method is developed to solve the fault diagnosis problem of wind turbine flange bolts using one-dimensional depthwise separable convolutions.The main idea is to use a one-dimensional convolutional neural network model to classify and identify the acoustic vibration signals of bolts,which represent different bolt damage states.Through the methods of knock test and modal simulation,it is concluded that the damage state of wind turbine flange bolt is related to the natural frequency distribution of acoustic vibration signal.It is found that the bolt damage state affects the modal shape of the structure,and then affects the natural frequency distribution of the bolt vibration signal.Therefore,the damage state can be obtained by identifying the natural frequency distribution of the bolt acoustic vibration signal.In the present one-dimensional depth-detachable convolutional neural network model,the one-dimensional vector is first convolved into multiple channels,and then each channel is separately learned by depth-detachable convolution,which can effectively improve the feature quality and the effect of data classification.From the perspective of the realization mechanism of convolution operation,the depthwise separable convolution operation has fewer parameters and faster computing speed,making it easier to build lightweight models and deploy them to mobile devices.