Skip to main content
Springer Nature Link
Log in

A framework for robotic arm pose estimation and movement prediction based on deep and extreme learning models

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Human-robot collaboration has gained a notable prominence in Industry 4.0, as the use of collaborative robots increases efficiency and productivity in the automation process. However, it is necessary to consider the use of mechanisms that increase security in these environments, as the literature reports that risk situations may exist in the context of human-robot collaboration. One of the strategies that can be adopted is the visual recognition of the collaboration environment using machine learning techniques, which can automatically identify what is happening in the scene and what may happen in the future. In this work, we are proposing a new framework that is capable of detecting robotic arm keypoints commonly used in Industry 4.0. In addition to detecting, the proposed framework is able to predict the future movement of these robotic arms, thus providing relevant information that can be considered in the recognition of the human-robot collaboration scenario. The proposed framework has two main modules. The first one contains a convolutional neural network based on self-calibrated convolutions enabling better discriminative feature extraction and the support of extreme learning machine neural networks with different kernels for predicting robotic arm keypoints. The second module is composed of deep recurrent learning models, such as long short-term memory and gated recurrent unit. These models are able to predict future robotic arm keypoints. All experiments were evaluated using the mean squared error metric. Results show that the proposed framework is capable of detecting and predicting with low error, contributing to the mitigation of risks in human-robot collaboration. In addition, it was possible to verify that the use of convolutional neural networks in conjunction with extreme learning machines can offer a lower detection error in a regression task (e.g., keypoint detection), something that, as far as the authors are aware of, is not yet known, nor had been evaluated previously in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Ruchi Goel, Pooja Gupta (2020) Robotics and industry 4.0: Sharp Business and Sustainable Development. A roadmap to industry 4.0: smart production. Springer, pp 157–169

    Google Scholar 

  2. Eloise M, Riccardo M, Zampieri Emanuele GG, Maurizio F, Giulio R (2019) Human-robot collaboration in manufacturing applications: a review. Robotics 8(4):100

    Article  Google Scholar 

  3. Semeraro F, Griffiths A, Cangelosi A (2023) Human-robot collaboration and machine learning: a systematic review of recent research. Robotics Comput-Integr Manufact 79:102432

    Article  Google Scholar 

  4. Arash A, Maria ZA, Serena I, Alin A-S, Kazuhiro K, Oussama K (2018) Progress and prospects of the human-robot collaboration. Auton Robot 42(5):957–975

    Article  Google Scholar 

  5. Bauer A, Wollherr D, Buss M (2008) Human-robot collaboration: a survey. Int J Humanoid Rob 5(01):47–66

    Article  Google Scholar 

  6. Ehsan H-PS, Simon T, Sergey K, Alexandre D (2020) Operations management issues in design and control of hybrid human-robot collaborative manufacturing systems: a survey. Annu Rev Control 49:264–276

    Article  Google Scholar 

  7. Lakomkin E, Zamani MA, Weber C, Magg S, Wermter S (2018) On the robustness of speech emotion recognition for human-robot interaction with deep neural networks. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 854–860. IEEE

  8. Sridharan M, Meadows B (2019) Towards a theory of explanations for human-robot collaboration. KI-Künstliche Intell 33(4):331–342

    Article  Google Scholar 

  9. Fragapane Giuseppe, Hvolby Hans-Henrik, Sgarbossa Fabio, Strandhagen Jan Ola (2020) Autonomous mobile robots in hospital logistics. In: IFIP International Conference on Advances in Production Management Systems, Springer, pp 672–679

  10. Microsoft (2019) Microsoft dynamics 365 manufacturing trends report, 2019. Accessed: 2019-09-09

  11. Reis G, Dantas M, Bezerra D, Nunes G, Dreyer P, Ledebour C, Kelner J, Sadok D, Souza R, Lins S et al (2021) Gripper design for radio base station autonomous maintenance system. Int J Autom Comput 18:1–9

    Article  Google Scholar 

  12. Thors B, Furuskär A, Colombi D, Törnevik C (2017) Time-averaged realistic maximum power levels for the assessment of radio frequency exposure for 5g radio base stations using massive mimo. IEEE Access 5:19711–19719

    Article  Google Scholar 

  13. Vasic M, Billard A(2013) Safety issues in human-robot interactions. In: 2013 IEEE International Conference on Robotics and Automation, pp 197–204. IEEE

  14. Rodrigues IR, Barbosa G, Oliveira Filho A, Cani C, Dantas M, Sadok DH, Kelner J, Souza RS, Marquezini MV, Lins S (2021) Modeling and assessing an intelligent system for safety in human-robot collaboration using deep and machine learning techniques. Multi Tools Appl 81:2213–2239

    Article  Google Scholar 

  15. Jianjing Zhang, Hongyi Liu, Qing Chang, Lihui Wang, Gao Robert X (2020) Recurrent neural network for motion trajectory prediction in human-robot collaborative assembly. CIRP Annals 69(1):9–12

    Article  Google Scholar 

  16. Anvaripour M, Saif M (2019) Collision detection for human-robot interaction in an industrial setting using force myography and a deep learning approach. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp 2149–2154. IEEE

  17. Maceira M, Olivares-Alarcos A, Alenyà G (2020) Recurrent neural networks for inferring intentions in shared tasks for industrial collaborative robots. In: 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp 665–670. IEEE

  18. Miseikis J, Knobelreiter P, Brijacak I, Yahyanejad S, Glette K, Elle OJ, Torresen J (2018) Robot localisation and 3d position estimation using a free-moving camera and cascaded convolutional neural networks. In: 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pages 181–187. IEEE

  19. Zhou F, Chi Z, Zhuang C, Ding H (2019) 3D pose estimation of robot arm with rgb images based on deep learning. In: International Conference on Intelligent Robotics and Applications, pp 541–553. Springer

  20. Lee TE, Tremblay J, To T, Cheng J, Mosier T, Kroemer O, Fox D, Birchfield S (2020) Camera-to-robot pose estimation from a single image. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp 9426–9432. IEEE

  21. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  22. Liu J-J, Hou Q, Cheng M-M, Wang C, Feng J (2020) Improving convolutional networks with self-calibrated convolutions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10096–10105

  23. Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501

    Article  Google Scholar 

  24. Rodrigues IR, da Silva Neto SR, Kelner J, Sadok D, Endo PT (2011) Convolutional extreme learning machines: a systematic review. Informatics 8:33

    Article  Google Scholar 

  25. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  26. Cho K, van Merriënboer B , Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, October. Association for Computational Linguistics, pp 1724–1734

  27. Silva IR, Barbosa GB, Ledebour CC, Oliveira Filho AT, Kelner J, Sadok D, Lins S, Souza R (2020) Assessing deep learning models for human-robot collaboration collision detection in industrial environments. In: Brazilian Conference on Intelligent Systems, Springer, pp 240–255

  28. Robla-Gòmez S, Becerra VM, Lltata JR, Gonzalez-Sarabia E, Torre-Ferrero C, Juan P-O (2017) Working together: a review on safe human-robot collaboration in industrial environments. IEEE Access 5:26754–26773

    Article  Google Scholar 

  29. Lasota PA, Fong T, Shah JA et al (2017) A survey of methods for safe human-robot interaction. Found Trends Robot 5(4):261–349

    Article  Google Scholar 

  30. Deng J, Dong W, Socher R, Li L-J, Li K, F-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp 248–255. IEEE

  31. Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp 6105–6114. PMLR

  32. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

    Article  Google Scholar 

  33. Lipton ZC, Berkowitz J, Elkan C (2015) A critical review of recurrent neural networks for sequence learning. arXiv preprint, arXiv:1506.00019

  34. Abdel-Nasser S, Koustoumpardis Panagiotis N, Nikos A (2020) Human-robot collisions detection for safe human-robot interaction using one multi-input-output neural network. Soft Comput 24(9):6687–6719

    Article  Google Scholar 

  35. Min PK, Jihwan K, Jinhyuk P, Park Frank C (2021) Learning-based real-time detection of robot collisions without joint torque sensors. IEEE Robot Autom Lett 6(1):103–110

    Article  Google Scholar 

  36. Alex K, Ilya S, Hinton Geoffrey E (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  37. Mišeikis J, Brijacak I, Yahyanejad S, Glette K, Elle OJ, Torresen J (2018) Transfer learning for unseen robot detection and joint estimation on a multi-objective convolutional neural network. In: 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR), pp 337–342. IEEE

  38. Heindl C, Zambal S, Scharinger J (2019) Learning to predict robot keypoints using artificially generated images. In: 2019 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), pp 1536–1539. IEEE

  39. Ballas N, Li Y, Pal C, Courville A (2015) Delving deeper into convolutional networks for learning video representations. arXiv preprintarXiv:1511.06432, 2015

  40. Heindl C, Zambal S, Ponitz T, Pichler A, Scharinger J (2019) 3D robot pose estimation from 2D images. arXiv preprint, arXiv:1902.04987

  41. da Silva Neto SR, Tabosa Oliveira T, Teixeira IV, Aguiar de Oliveira SB, Souza Sampaio V, Lynn T, Endo PT (2022) Machine learning and deep learning techniques to support clinical diagnosis of arboviral diseases: A systematic review. PLoS Negl Trop Dis 16(1):e0010061

    Article  Google Scholar 

  42. Jiuxiang G, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J et al (2018) Recent advances in convolutional neural networks. Pattern Recogn 77:354–377

    Article  Google Scholar 

  43. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  44. Cui D, Zhang G, Han W, Lekamalage Chamara Kasun L, Hu K, Huang G-B (2017) Compact feature representation for image classification using elms. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp 1015–1022

  45. Guang-Bin H, Hui WD, Yuan L (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122

    Article  Google Scholar 

  46. filters combination and error model validation (2019) Michel M dos Santos, Abel G da Silva Filho, and Wellington P dos Santos. Deep convolutional extreme learning machines. Neurocomputing 329:359–369

    Google Scholar 

  47. Huang F, Jun L, Tao J, Li L, Tan X, Liu P (2019) Research on optimization methods of elm classification algorithm for hyperspectral remote sensing images. IEEE Access 7:108070–108089

    Article  Google Scholar 

  48. Li D, Qiu X, Zhu Z, Liu Y (2018) Criminal investigation image classification based on spatial cnn features and elm. In: 2018 10th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), Vol 2, pp 294–298. IEEE

  49. Pu H, Zhai J-H, Zhang S-F (2017) A simple and effective method for image classification. In: 2017 International Conference on Machine Learning and Cybernetics (ICMLC), vol 1, pp 230–235. IEEE

  50. Khellal A, Ma H, Fei Q (2018) Convolutional neural network features comparison between back-propagation and extreme learning machine. In: 2018 37th Chinese Control Conference (CCC), pp 9629–9634. IEEE

  51. Lu S, Xia K, Wang S-H (2020) Diagnosis of cerebral microbleed via vgg and extreme learning machine trained by gaussian map bat algorithm. J Ambient Intell Humanized Computi, pp 1–12

  52. Ijjina EP (2017) Human action recognition in rgb-d videos using motion sequence information and deep learning. Pattern Recogn 72:504–516

    Article  Google Scholar 

  53. Zaki Hasan FM, Faisal S, Ajmal M (2019) Viewpoint invariant semantic object and scene categorization with rgb-d sensors. Auton Robot 43(4):1005–1022

    Article  Google Scholar 

  54. Huang Jinghong Yu, Liang Z, Cai Zhaoquan G, Zhenghui CZ, Gao Wei Yu, Qianyun SD (2017) Extreme learning machine with multi-scale local receptive fields for texture classification. Multi Syst Signal Process 28(3):995–1011

    Article  Google Scholar 

  55. Rezaeenour J, Ahmadi M, Jelodar H and Shahrooei R (2022) Systematic review of content analysis algorithms based on deep neural networks. Multimedia Tools and Applications

  56. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536

    Article  MATH  Google Scholar 

  57. Kanagachidambaresan GR, Ruwali A, Debrup B, Prakash KB (2021) Recurrent neural network. Springer International Publishing, Cham, pp 53–61

    Google Scholar 

  58. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  59. Rengasamy D, Jafari M, Rothwell B, Chen X, Figueredo GP (2020) Deep learning with dynamically weighted loss function for sensor-based prognostics and health management. Sensors 20(3):723

    Article  Google Scholar 

  60. Chung J, Gulcehre C, Kyunghyun C and Yoshua B (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on Deep Learning, December 2014

  61. He T, Zhang Z, Zhang H, Zhang Z, Xie J, Li M (2019) Bag of tricks for image classification with convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 558–567

  62. Huang G, Huang G-B, Song S, You K (2015) Trends in extreme learning machines: a review. Neural Netw 61:32–48

    Article  MATH  Google Scholar 

  63. Ribeiro AMNC, do Carmo PRX, Rodrigues IR, Sadok D, Lynn T, Endo PT (2020) Short-term firm-level energy-consumption forecasting for energy-intensive manufacturing: a comparison of machine learning and deep learning models. Algorithms 13(11):274

    Article  MathSciNet  Google Scholar 

  64. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint, arXiv:1406.1078

  65. Universal Robots (2021) Universial robots. https://www.universal-robots.com/. accessed in november

  66. Dutta A, Zisserman A (2019) The via annotation software for images, audio and video. In: Proceedings of the 27th ACM International Conference on Multimedia, pp 2276–2279

  67. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint, arXiv:1602.07360

  68. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  69. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp 4700–4708

  70. Huang G-B, Siew C-K (2005) Extreme learning machine with randomly assigned rbf kernels. Int J Inf Technol 11(1):16–24

    Google Scholar 

  71. Baraha S, Biswal PK (2017) Implementation of activation functions for elm based classifiers. In: 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), pp 1038–1042. IEEE

  72. Zhang C, Benz P, Argaw DM, Lee S, Kim J, Rameau F, Bazin J-C, Kweon IS (2021) Resnet or densenet? Introducing dense shortcuts to resnet. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 3550–3559

  73. Yang S, Yu X, Zhou Y (2020) Lstm and gru neural network performance comparison study: Taking yelp review dataset as an example. In: 2020 International workshop on electronic communication and artificial intelligence (IWECAI), pp 98–101. IEEE

  74. Patel MM, Tanwar S, Gupta R, Kumar N (2020) A deep learning-based cryptocurrency price prediction scheme for financial institutions. J Inform Security Appl 55:102583

    Google Scholar 

Download references

Acknowledgements

This work was financed in part by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), Fundação de Amparo a Ciência e Tecnologia de Pernambuco (FACEPE), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), and Research, Development and Innovation Center, Ericsson Telecommunications Inc., Brazil.

Author information

Authors and Affiliations

  1. Centro de Informática, Universidade Federal de Pernambuco, Recife, Pernambuco, Brazil

    Iago Richard Rodrigues, Marrone Dantas, Assis T. de Oliveira Filho, Gibson Barbosa, Daniel Bezerra, Judith Kelner & Djamel Sadok

  2. Grupo de Pesquisa em Redes e Telecomunicações, Universidade Federal de Pernambuco, Recife, Pernambuco, Brazil

    Iago Richard Rodrigues, Marrone Dantas, Assis T. de Oliveira Filho, Gibson Barbosa, Daniel Bezerra, Judith Kelner & Djamel Sadok

  3. Ericsson Research, Indaiatuba, São Paulo, Brazil

    Ricardo Souza & Maria Valéria Marquezini

  4. Universidade de Pernambuco, Recife, Pernambuco, Brazil

    Patricia Takako Endo

  5. Universidade Católica de Pernambuco, Recife, Pernambuco, Brazil

    Iago Richard Rodrigues & Daniel Bezerra

Authors
  1. Iago Richard Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marrone Dantas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Assis T. de Oliveira Filho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gibson Barbosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel Bezerra
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ricardo Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Maria Valéria Marquezini
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Patricia Takako Endo
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Judith Kelner
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Djamel Sadok
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iago Richard Rodrigues.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rodrigues, I.R., Dantas, M., de Oliveira Filho, A.T. et al. A framework for robotic arm pose estimation and movement prediction based on deep and extreme learning models. J Supercomput 79, 7176–7205 (2023). https://doi.org/10.1007/s11227-022-04936-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04936-z

Keywords

Search

Navigation