A.A. Lonshchakov, O.V. Darintsev
Comparative analysis of approaches necessary for designing
a robotic module for industrial parts defect detection. Multiphase Systems. 19 (2024) 1. 18–30 (in Russian).
Comparative analysis of approaches necessary for designing a robotic module for industrial parts defect detection
A.A. Lonshchakov∗, O.V. Darintsev∗∗
∗Ufa State Petroleum Technological University, Ufa, Russia
∗∗Mavlyutov Institute of Mechanics UFRC of the RAS, Ufa, Russia
Abstract
Over the past two decades, the most frequently implemented method of oil extraction in the Russian Federation has been a mechanized
method using an electric submersible pump (ESP) unit. Due to the remoteness of the fields from the base of repair enterprises, the cost of
transporting spent but maintainable ESPs significantly exceeds their purchase price, which over time leads to the formation of a significant
amount of decommissioned oil production equipment. To solve this problem, it is proposed to develop a mobile robotic module for sorting,
defecting and storing pump parts, thanks to its use, minor repairs of equipment will be possible directly at production sites. The article deals
with the problems of complex flaw detection of metal and non-metallic parts with the possibility of applying the developed methodology
for a wide range of industrial products, at the same time, the main attention is paid to work with ESP parts. Based on the decomposition of
the problem, the most problematic operations were identified: classification of parts, surface control (identification of defects), dimensional
control. The results of a brief comparative analysis for each of the above subtasks are based on a review of the scientific literature over
the past 30 years, with the largest number of sources reviewed in the last 5 years. As a result optimal methods for solving the task were
derived — a machine learning technique for classifying surface defects, the use of a coordinate measuring machine with a manipulator for
dimensional control. A new approach is also proposed to solve the main problem of machine learning methods (lack of training samples) in
the form of using synthetic photorealistic images for classification with transfer of defect features from semantically close and publicly
available training samples.
Keywordsvision system,
defect detection,
automatic visual inspection (AVI),
surface defect classification,
automatic parts
measurement
Article outline
The issues of ensuring the smooth operation of downhole equipment while reducing operating costs do not lose their relevance at the moment. One of the effective ways to solve this problem is the development of a mobile small-sized robotic module for sorting and defective parts of the ESP. If gross defects and violations of the integrity of parts can be detected by weighing or simple mold control operations using a template, then the identification and classification of surface defects is quite a difficult problem.
The main purpose of this work is to analyze known methods and techniques for controlling the shape and surface of products made of various materials. Classification accuracy and processing speed are recognized as the most significant characteristics for evaluating the effectiveness of the considered methods.
Various approaches and solutions were considered for the task of detecting surface defects, as well as ways to carry out dimensional control of ESP parts. The advantages and disadvantages of the following methods are analyzed: statistical, spectral, complex and machine learning-based. For the selected GAN method, a solution to the problem with the lack of a training sample is shown, which consists in using a synthetic photorealistic sample supplemented with images of defects transferred from the well-known databases DAGM, Kolector SSD 2, Severstal, MVTec AD.
Dimensional control requires increased accuracy from the measuring equipment used, therefore, options for using contact and non-contact measurement methods were considered. The choice was made in favor of two options: laser triangulation and a control and measuring machine in combination with a complete manipulation robot. The final choice can be made only after agreement with the customer.
The main results. It is shown that in order to determine the composition of the developed robotic defect module, it is necessary to first solve the issues of methodological and technological support for the operations performed. Thus, the use of a specialized approach to the identification of surface effects and the modification of training methods made it possible to increase the accuracy of the operation and implement the adjustment of algorithms.
Conclusion. A brief overview of the techniques and approaches used for automated part flaw detection confirms not only the high complexity, but also the great demand for the task for various industries and other fields of application. The considered examples make it possible to exclude doubts about the possibility of practical implementation of an automated module, the effectiveness of which is determined by the correct use of a manipulating robot with peripheral equipment. A plan for further research has been generated, including the construction of a laboratory stand and conducting tests.
References
- Исполнение федерального бюджета и бюджетов системы Российской Федерации за 2022 год. Минфин России.
Execution of the federal budget and the budgets of the system of the Russian Federation for 2022. The Ministry of Finance of Russia (in Russian).
https://minfin.gov.ru/common/upload/library/2023/04/main/ispolnenie_federalnogo_budzheta_predvaritelnye_itogi.pdf
(дата обращения (accessed): 05.10.2023 г.)
- Рудницкий С.В., Зацепин А.Ю., Демин Е.В., Ющенко Т.С. Перспективное скважинное оборудование для добычи сланцевой нефти баженовской
свиты // PROНЕФТЬ. Профессионально о нефти. 2021. Т. 6, № 1. C. 64–75.
Rudnitsky S.V., Zatsepin A.Yu., Demin E.V., Yushchenko T.S. [Promising downhole equipment for the extraction of shale oil of the Bazhenov formation]
Perspektivnoye skvazhinnoye oborudovaniye dlya dobychi slantsevoy nefti bazhenovskoy svity. [Professionally about oil] Professional’no o nefti.
Vol. 6, No. 1. 2021. Pp. 64–75 (in Russian).
DOI: 10.51890/2587-7399-2021-6-1-64-75
- Конторович А.Э., Эдер Л.В., Филимонова И.В., Мишенин М.В. Роль уникальных и крупных месторождений в нефтяной промышленности России:
ретроспектива, современное состояние, прогноз // Энергетическая политика. 2016. № 2. С. 34–43.
Kontorovich A.E., Eder L.V., Filimonova I.V., Mishenin M.V. [The role of unique and large deposits in the Russian oil industry: retrospective, current
state, forecast] Rol’ unikalnykh i krupnykh mestorozhdeniy v neftyanoy promyshlennosti Rossii: retrospektiva, sovremennoye sostoyaniye, prognoz //
EP. No. 2. 2016 (in Russian).
EDN: wxmmyv
- Утверждение стратегического направления в области цифровой трансформации топливно-энергетического комплекса: распоряжение
Правительства РФ от 28 декабря 2021 г. № 3924-р.
[Approval of the strategic direction in the field of digital transformation of the fuel and energy complex: Decree of the Government of theRussian Federation No. 3924-r dated December 28, 2021] Utverzhdeniye strategicheskogo napravleniya v oblasti tsifrovoy transformatsii toplivno-
energeticheskogo kompleksa: rasporyazheniye Pravitel’stva ot 28 decabrya 2021. No. 3924-р (in Russian).
http://publication.pravo.gov.ru/Document/View/0001202112300029
- Гиндуллин И.И, Лонщаков А.А., Салихов Л.С. Структура робототехнического модуля сортировки деталей // Завалишинские чтения: XVII
Международная конференция по электромеханике и робототехнике (СПб., 18–19 апреля 2023 г.). Молодёжная секция: сб. докл.: в 2 ч. Ч. 1.
СПб.: ГУАП, 2023. С. 41–44.
Gindullin I.I., Lonshakov A.A., Salikhov L.S. [The structure of the robotic module for sorting parts]. Struktura robototekhnicheskogo modulya sortirovki
detaley. Zavalishinsky readings: XVII International Conference on Electromechanics and Robotics (St. Petersburg, April 18–19, 2023). Youth section.
Vol. 2, No. 1. St. Petersburg: GUAP. 2023. Pp. 41-44 (in Russian).
- Morbid M., Saber S., Ali R.G. Industrial Parts Change Recognition Model Using Machine Vision, Image Processing in the Framework of Industrial
Information Integration // Journal of Industrial Information Integration. 2022. V. 26. 100277.
DOI: 10.1016/j.jii.2021.100277
- Kanopoulos N., Vasanthavada N., Baker R.L., Design of an image edge detection filter using the Sobel operator // IEEE Journal of Solid-State Circuits.
1988. V. 23(2). Pp. 358–367.
DOI: 10.1109/4.996
- Seif A., Salut M.M., Marsono M.N. A hardware architecture of Prewitt edge detection // 2010 IEEE Conference on Sustainable Utilization and
Development in Engineering and Technology. 2010. Pp. 99-101.
DOI: 10.1109/STUDENT.2010.5686999
- Nguyen C.H., Young C. Comparison of point cloud data and 3D CAD data for on-site dimensional inspection of industrial plant piping systems //
Automation in Construction. 2018. V. 91. Pp. 44–52.
DOI: 10.1016/j.autcon.2018.03.008
- Rabbani T., van den Heuvel F.A., Vosselmann G. Segmentation of Point Clouds Using Smoothness Constraint // ISPRS Commission V Symposium
‘Image Engineering and Vision Metrology’. 2006. Pp. 248–253.
ResearchGate: 228340970
- Schnabel R., Wahl R., Klein R. Efficient Ransac for point-cloud shape detection // Graphics Forum. 2007. V. 26 (2). Pp. 214–226.
DOI: 10.1111/j.1467-8659.2007.01016.x
- Huang R., Gu J., Sun X., Hou Y., Uddin S. A rapid recognition method for electronic components based on the improved YOLO-V3 network // Electronics.
2019. V. 8 (8). 825.
DOI: 10.3390/electronics8080825
- Tian Y., Yang G., Wang Z., Wang H., Li E., Liang Z. Apple detection during different growth stages in orchards using the improved YOLO-V3 model //
Computers and electronics in agriculture. 157. 2019. V. 157. Pp. 417–426.
DOI: 10.1016/j.compag.2019.01.012
- Sinha D., El-Sharkawy M. Thin mobilenet: An enhanced mobilenet architecture // In 2019 IEEE 10th annual ubiquitous computing, electronics &
mobile communication conference (UEMCON). 2019. Pp. 280–285.
DOI: 10.1109/UEMCON47517.2019.8993089
- Andrei-Alexandru T., Henrietta D.E. Low-Cost Defect Detection Using a Deep Convolutional Neural Network // IEEE International Conference on
Automation, Quality and Testing, Robotics (AQTR), Cluj-Napoca, Romania. 2020. Pp. 1–5.
DOI: 10.1109/AQTR49680.2020.9130004
- Muhammad U., Wang W., Chattha S.P., Ali S.August. Pre-trained VGGNet architecture for remote-sensing image scene classification // In 2018 24th
International Conference on Pattern Recognition (ICPR). 2018. Pp. 1622–1627.
DOI: 10.1109/ICPR.2018.8545591
- Deng J., Dong W., Socher R., Li L.J., Kai L., Li Fei-Fei. ImageNet: A large-scale hierarchical image database // 2009 IEEE Conference on Computer Vision
and Pattern Recognition, Miami, FL, USA. 2009. Pp. 248–255.
DOI: 10.1109/CVPR.2009.5206848
- Feng C., Liu M.Y., Kao C.C., Lee T.Y. Deep active learning for civil infrastructure defect detection and classification // In Computing in civil engineering.
2017. Pp. 298–306.
DOI: 10.1061/9780784480823.036
- He K., Zhang X., Ren S., Sun J. Deep Residual Learning for Image Recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), Las Vegas, NV, USA. 2016. Pp. 770–778.
DOI: 10.1109/CVPR.2016.90
- Dawson H.L., Debrule O., John C.M. Impact of dataset size and convolutional neural network architecture on transfer learning for carbonate rock
classification // Computers & Geosciences. 2023. V. 171. 105284.
DOI: 10.1016/j.cageo.2022.105284
- Northcutt C.G., Athalye A., Mueller J. Pervasive label errors in test sets destabilize machine learning benchmarks // In Thirty-fifth Conference on
Neural Information Processing Systems Datasets and Benchmarks Track. 2021.
DOI: 10.48550/arXiv.2103.14749
- Cai L., Xu X., Liew J.H., Foo C.S. Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs // In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. Pp. 10988–10997.
https://openaccess.thecvf.com/content/CVPR2021/html/Cai_Revisiting_Superpixels_for_Active_Learning_in_Semantic_Segmentation_With_Realistic_CVPR_2021_paper.
html
- Roovere P.D, Moonen S., Michiels M., Wyffels F. Dataset of industrial metal objects // arXiv, cs.CV. 2022. 2208.04052.
DOI: 10.48550/arXiv.2208.04052
- Tobin J., Fong R., Ray A., Schneider J., Zaremba W., Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real
world // In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2017. Pp. 23–30.
DOI: 10.1109/IROS.2017.8202133[25]
- Reddi S., Poczos B., Smola A. Doubly Robust Covariate Shift Correction // Proceedings of the AAAI Conference on Artificial Intelligence. 2015. V. 29 (1).
Pp. 2949–2955.
DOI: 10.1609/aaai.v29i1.9576
- Vanherle B., Moonen S., Van Reeth F., Michiels. Analysis of training object detection models with synthetic data // arXiv, cs.CV. 2022. 2211.16066.
DOI: 10.48550/arXiv.2211.16066
- Touvron H., Vedaldi A., Douze M., Jegou H. Fixing the train-test resolution discrepancy // arXiv, cs.CV. 2022. 1906.06423.
DOI: 10.48550/arXiv.1906.06423
- Dosovitski A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., Gelly S., Uszkoreit J. An image is
worth 16x16 words: Transformers for image recognition at scale // arXiv, cs.CV. 2021. 2010.11929.
DOI: 10.48550/arXiv.2010.11929
- Djukic D., Spuzic S. Statistical discriminator of surface defects on hot rolled steel // Proceedings of Image and Vision Computing New Zealand. 2007.
Pp. 158–163.
ResearchGate: 229035389
- Nand G.K., Noopur, Neogi N.G. Defect detection of steel surface using entropy segmentation // 2014 Annual IEEE India Conference (INDICON), Pune,
India. 2014. Pp. 1–6.
DOI: 10.1109/INDICON.2014.7030439
- Neogi N., Mohanta D.K., Dutta P.K. Defect Detection of Steel Surfaces with Global Adaptive Percentile Thresholding of Gradient Image // J. Inst. Eng.
India Ser. 2017. Pp. 557–565.
DOI: 10.1007/s40031-017-0296-2
- Bulnes F.G., Usamentiaga R., Garca D.F., Molleda J. Vision-based sensor for early detection of periodical defects in web materials // Sensors (Basel).
2012. V. 12 (8). Pp. 10788–10809.
DOI: 10.3390/s120810788
- Zhao Y., Yan Y., Song K. Vision-based automatic detection of steel surface defects in the cold rolling process: considering the influence of industrial
liquids and surface textures // Int. J. Adv. Manuf. Technol. 2016. V. 90 (8). Pp. 1665–1678.
DOI: 10.1007/S00170-016-9489-0
- Zhiznyakov A.L., Privezentsev D.G., Zakharov A.A. Using fractal features of digital images for the detection of surface defects // Pattern Recognition
and Image Analysis. 2015. V. 25. Pp. 122–131.
DOI: 10.1134/S105466181501023X
- Yazdchi M., Yazdi M., Mahyari A.G. Steel surface defect detection using texture segmentation based on multifractal dimension // In 2009 International
Conference on Digital Image Processing. 2009. Pp. 346–350.
DOI: 10.1109/ICDIP.2009.68
- Aiger D., Talbot H. The phase only transform for unsupervised surface defect detection // In 2010 IEEE Computer Society Conference on Computer
Vision and Pattern Recognition. 2010. Pp. 295–302.
DOI: 10.1109/CVPR.2010.5540198
- Choi D.C., Jeon Y.J., Kim S.H., Moon S., Yun J.P., Kim S.W. Detection of pinholes in steel slabs using Gabor filter combination and morphological features //
ISIJ International. 2017. V. 57 (6). Pp. 1045–1053.
DOI: 10.2355/isijinternational.ISIJINT-2016-160
- Medina R., Gayubo F., González-Rodrigo L.M., Olmedo D., Gómez-Garcı́a-Bermejo J., Zalama E., Perán J.R. Automated visual classification of frequent
defects in flat steel coils // The International Journal of Advanced Manufacturing Technology. 2011. V. 57. Pp. 1087–1097.
DOI: 10.1007/s00170-011-3352-0
- Randen T., Husoy J.H. Texture segmentation using filters with optimized energy separation // IEEE Transactions on Image Processing. 1999. V. 8 (4).
Pp. 571–582.
DOI: 10.1109/83.753744
- Kumar A., Pang G.K. Defect detection in textured materials using optimized filters // IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics). 2002. V. 32 (5). Pp. 553–570.
DOI: 10.1109/TSMCB.2002.1033176
- Kumar A. Inspection of surface defects using optimal FIR filters // In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing,
2003. Proceedings. 2003. Pp. II–241
DOI: 10.1109/ICASSP.2003.1202339
- Jeon Y.J., Choi D.C., Lee S.J., Yun J.P., Kim S.W. Steel-surface defect detection using a switching-lighting scheme // Applied Optics. 2016. V. 55 (1). Pp. 47–
57.
DOI: 10.1364/AO.55.000047
- Cross G.R., Jain A.K. Markov random field texture models // IEEE Transactions on pattern analysis and machine intelligence. 1983. V. PAMI-5 (1).
Pp. 25–39.
DOI: 10.1109/TPAMI.1983.4767341
- Gayubo F., Gonzalez J.L., de la Fuente E., Miguel F., Perán J.R. On-line machine vision system for detect split defects in sheet-metal forming processes //
In 18th International Conference on Pattern Recognition. 2006. V. 1. Pp. 723–726.
DOI: 10.1109/ICPR.2006.902
- Ke X., Min S., Chaolin Y., Peng Z. Application of hidden Markov tree model to on-line detection of surface defects for steel strips // Journal of
mechanical engineering. 2013. V. 49(22). Pp. 34–40.
https://qikan.cmes.org/jxgcxb/EN/Y2013/V49/I22/34
- Timm F., Barth E. February. Non-parametric texture defect detection using Weibull features // In Image Processing: Machine Vision Applications IV.
2011. Pp. 150–161.
DOI: 10.1117/12.872463
- Liu K., Wang H., Chen H., Qu E., Tian Y., Sun H. Steel surface defect detection using a new Haar–Weibull-variance model in unsupervised manner //
IEEE transactions on instrumentation and measurement. 2017. V. 66 (10). Pp. 2585–2596.
DOI: 10.1109/TIM.2017.2712838
- Song K., Yan Y. Micro surface defect detection method for silicon steel strip based on saliency convex active Contour model // Math. Probl. Eng. 2013.
429094.
DOI: 10.1155/2013/429094
- Yang J., Li X., Xu J., Cao Y., Zhang Y., Wang L., Jiang S. Development of an optical defect inspection algorithm based on an active contour model for
large steel roller surfaces // Applied optics. 2018. V. 57 (10). Pp. 2490–2498.
DOI: 10.1364/AO.57.002490
- Ghorai S., Mukherjee A., Gangadaran M., Dutta P.K. Automatic defect detection on hot-rolled flat steel products // IEEE Transactions on
Instrumentation and Measurement. 2012. V. 62 (3). Pp. 612–621.
DOI: 10.1109/TIM.2012.2218677
- Kang G.W., Liu H.B. Surface defects inspection of cold rolled strips based on neural network // In 2005 international conference on machine learning
and cybernetics. 2005. Pp. 5034–5037.
DOI: 10.1109/ICMLC.2005.1527830
- Weimer D., Scholz-Reiter B., Shpitalni M. Design of deep convolutional neural network architectures for automated feature extraction in industrial
inspection // CIRP annals. 2016. V. 65 (1). Pp. 417–420.
DOI: 10.1016/j.cirp.2016.04.072
- Liu Y., Xu K., Xu J. Periodic surface defect detection in steel plates based on deep learning // Applied Sciences. 2019. V. 9 (15). 3127.
DOI: 10.3390/app9153127
- Li J., Su Z., Geng J., Yin Y. Real-time detection of steel strip surface defects based on improved yolo detection network // IFAC-PapersOnLine. 2018.
V. 51(21). Pp. 76–81.
DOI: 10.1016/j.ifacol.2018.09.412
- Zhao Z., Li B., Dong R., Zhao P. A surface defect detection method based on positive samples // In PRICAI 2018: Trends in Artificial Intelligence: 15th
Pacific Rim International Conference on Artificial Intelligence, Nanjing, China, August 28–31, 2018, Proceedings. 2018. V. 2 (15). Pp. 473–481.
DOI: 10.1007/978-3-319-97310-4_54
- Youkachen S., Ruchanurucks M., Phatrapomnant T., Kaneko H. Defect Segmentation of Hot-rolled Steel Strip Surface by using Convolutional Auto-
Encoder and Conventional Image processing // 10th International Conference of Information and Communication Technology for Embedded Systems
(IC-ICTES). 2019. Pp. 1–5.
DOI: 10.1109/ICTEmSys.2019.8695928
- DAGM dataset. University of Heidelberg. 29th Annual Symposium of the German Association for Pattern Recognition.
https://conferences.mpi-inf.mpg.de/dagm/2007/prizes.html
(дата обращения 05.10.2023 г.)
- Kolector SSD 2 dataset. University of Ljubljana. Visual Cognitive Systems Laboratory.
https://www.vicos.si/resources/kolektorsdd2/
(дата обращения 05.10.2023 г.)
- Severstal dataset. Kaggle.
https://www.kaggle.com/c/severstal-steel-defect-detection
(дата обращения 05.10.2023 г.)
- MVTec AD dataset.
https://www.mvtec.com/company/research/datasets/mvtec-ad
(дата обращения 05.10.2023 г.)
- Sun Y., Lu L., Wu F., Xiao S., Sha J., Zhang L. Error Analysis of a Coordinate Measuring Machine with a 6-DOF Industrial Robot Holding the Probe // In
Actuators. 2023. V. 12 (4). 173.
DOI: 10.3390/act12040173
- Brosed F.J., Santolaria J., Aguilar J.J, Guillom?a D. Laser triangulation sensor and six axes anthropomorphic robot manipulator modelling for the
measurement of complex geometry products // Robotics and Computer-Integrated Manufacturing. 2012. V. 28 (6). Pp. 660–671.
DOI: 10.1016/j.rcim.2012.04.002
- Lemeš S., Šabaravdić M., Zaimović-Uzunović N. Robotic manipulation in dimensional measurement // In 2013 XXIV International Conference on
Information, Communication and Automation Technologies (ICAT). 2013. Pp. 1–7.
DOI: 10.1109/ICAT.2013.6684085