Manufacturing Technology 2023, 23(4):399-417 | DOI: 10.21062/mft.2023.057

Devising a Multi-camera Motion Capture and Processing System for Production Plant Monitoring and Operator’s Training in Virtual Reality

Joanna Għbka ORCID...
Faculty of Mechanical Engineering, Wroclaw University of Science and Technology. Wybrzeże Wyspiañskiego 27, 50-370 Wroc³aw, Poland

The paper presents work aimed at building practical applications of virtual reality (VR) in manufacturing environments. It contains studies of the optical properties of cameras and lenses aimed at the selection of an optimal set (camera, adapter, lens) for the realization of recordings and video transmissions in stereoscopic format for VR. In response to the increasing trend in the number of applications of VR systems in the industry, works have been initiated with the purpose of building a system levelling image noise identified thus far as an obstacle to the effective utilization of VR in production systems. It was considered that picture error correction can significantly increase an already big data stream from the recordings. Based on it, a set of parameter values was defined which determined the selection of study equipment. Three research areas were set: the verification of the optical correctness, the study of image defects and their correction and the determination of the maximum optical resolution and the achievable image parameters in various lighting and environmental conditions. An example was presented for the application of a projected system for the monitoring of undesirable events/movement at work stands and key areas of production halls as well as training in the high-risk production zones.

Keywords: Production monitoring, Optical properties measurement, Training Virtual Reality system in manufacturing, Image capture and processing
Grants and funding:

The paper is based on research from project granted by European Union and financed by European Regional Development Fund under the Smart Growth Program. The project is implemented as a part of ‘Szybka Ĥcieżka’ competition organized by The National Centre of Research and Development. POIR.01.01.01-00-1111/17

Received: October 12, 2022; Revised: July 27, 2023; Accepted: August 1, 2023; Prepublished online: August 1, 2023; Published: September 5, 2023  Show citation

ACS AIP APA ASA Harvard Chicago IEEE ISO690 MLA NLM Turabian Vancouver
Għbka J. Devising a Multi-camera Motion Capture and Processing System for Production Plant Monitoring and Operator’s Training in Virtual Reality. Manufacturing Technology. 2023;23(4):399-417. doi: 10.21062/mft.2023.057.
Share...
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. ESWARAN M., BAHUBALENDRUNI M. V. A. R.., (2022). Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: A state of the art review. In: Journal of Manufacturing Systems, Vol. 65 , pp. 260-278. DOI:10.1016/j.jmsy.2022.09.016. Go to original source...
    2. ŻYWICKI K., ZAWADZKI P., GÓRSKI F., Virtual Reality Production Training System in the Scope of Intelligent Factory. in: International Conference on Intelligent Systems in Production Engineering and Maintenance -ISPEM, pp. 450-458. ISSN 1854-6250. Go to original source...
    3. BAKAR M. R. A., RAZALI N. A. M., WOOK M., ISMAIL M. N., SEMBOK T.M.T, (2021), Exploring and Developing an Industrial Automation Acceptance Model in the Manufacturing Sector Towards Adoption of Industry4.0. In: Manufacturing Technology Journal, Vol. 21, No. 4, pp. 434-446. ISSN: 1213-2489. Go to original source...
    4. CHANG M. M. L., NEE A. Y. C., ONG S. K., (2020). Interactive AR-assisted product disassembly sequence planning. In: International Journal of Production Research, Vol. 58, pp. 4916-31. https://doi.org/ 10.1080/00207543.2020.1730462. Go to original source...
    5. SIEW C. Y., ONG S. K., NEE A. Y. C. (2019). A practical augmented reality-assisted maintenance system framework for adaptive user support. In: Robotics and Computer-Integrated Manufacturing, Vol. 59, pp. 115-29. https://doi.org/10.1016/j.rcim.2019.03.010. Go to original source...
    6. HALIM A.Z.A. (2018). Maintenance process in automotive industry. In: Journal of Fundamental and Applied Science. Vol. 10 No.3S, pp. 412-421. ISSN 1112-9867.
    7. BILLINGHURST M, HAKKARAINEN M, WOODWARD C. (2008). Augmented assembly using a mobile phone. In: MUM'08 - : Proceedings of the 7th International Conference on Mobile and Ubiquitous Multimedia, pp. 84-7. https://doi.org/10.1145/1543137.1543153. Go to original source...
    8. EVANS G, MILLER J, IGLESIAS PENA M, MACALLISTER A, WINER E. (2017). Evaluating the Microsoft HoloLens through an augmented reality assembly application. In: Proceedings Volume 10197, Degraded Environments: Sensing, Processing, and Display; 101970V, https://doi.org/10.1117/12.2262626. Go to original source...
    9. ALVES J., MARQUES B., OLIVEIRA M., ARAÚJO T., DIAS P., SANTOS B.S. (2019) Comparing Spatial and Mobile Augmented Reality for Guiding Assembling Procedures with Task Validation. In: Proce. 19th IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). https://doi.org/10.1109/ICARSC.2019.8733642. Go to original source...
    10. MOURTZIS D, SIATRAS V, ANGELOPOULOS J, PANOPOULOS N. (2020). An augmented reality collaborative product design cloud-based platform in the context of learning factory. In: Procedia Manufacturing, Vol. 45:546-51. https://doi.org/10.1016/j. promfg.2020.04.076. Go to original source...
    11. SCHUMANN M, FUCHS C, KOLLATSCH C, KLIMANT P. (2021). Evaluation of augmented reality supported approaches for product design and production processes. In: Procedia CIRP; Vol. 97, pp. 160-5. https://doi.org/10.1016/j.procir.2020.05.219. Go to original source...
    12. ANANTHI K, RAJAVEL R, SABARIKANNAN S, SRISARAN A, SRIDHAR C. (2021). Design and fabrication of IoT based inventory control system. In: 7th International Conference on Advanced Computing and Communication Systems (ICACCS) pp. 1101-4.. https://doi.org/10.1109/ ICACCS51430.2021.9441701. Go to original source...
    13. STOLTZ MH, GIANNIKAS V, MCFARLANE D, STRACHAN J, UM J, SRINIVASAN R. (2017). Augmented reality in warehouse operations: opportunities and barriers. In: IFAC-PapersOnLine, Vol. 50, Issue 1, pp. 12979-12984. https://doi.org/10.1016/j.ifacol.2017.08.1807. Go to original source...
    14. ETONAM A. K., DI GRAVIO G., KULOBA P. W., NJIRI J.G. (2019). Augmented reality (AR) application in manufacturing encompassing quality control and maintenance. In: International Journal of Engineering and Advanced Technology (IJEAT) (Online), Vol.9 No. 1, 197-204. ISSN: 2249-8958. Go to original source...
    15. URBAS U, VRABIˇC R, VUKAˇSINOVI´C N. (2019) Displaying product manufacturing information in augmented reality for inspection. In: Procedia CIRP, Vol.81, pp.832-7. https://doi. org/10.1016/j.procir.2019.03.208. Go to original source...
    16. SEGOVIA D, MENDOZA M, MENDOZA E, GONZ´ALEZ E. (2015). Augmented reality as a tool for production and quality monitoring. In: Procedia Computer Science, Vol. 75, pp.291-300. https://doi.org/10.1016/j.procs.2015.12.250. Go to original source...
    17. KOKKAS A, VOSNIAKOS GC. (2019). An augmented reality approach to factory layout design embedding operation simulation. In: International Journal on Interactive Design and Manufacturing (IJIDeM), Vol. 13, pp 1061-1071. https://doi.org/10.1007/s12008-019-00567-6. Go to original source...
    18. NEE AYC, ONG SK, CHRYSSOLOURIS G, MOURTZIS D. (2012). Augmented reality applications in design and manufacturing. In: CIRP Annals , Vol. 61, No. 2, , pp 657-79. https:// doi.org/10.1016/j.cirp.2012.05.010. Go to original source...
    19. OLWAL A, GUSTAFSSON J, LINDFORS C. (2008). Spatial augmented reality on industrial CNC- machines. Eng Real Virtual Real, In: Proceedings of SPIE - The International Society for Optical Engineering pp. 6804:680409. https://doi.org/ 10.1117/12.760960. Go to original source...
    20. ZHANG J, ONG S. K., NEE A.Y.C. (2010). A multi-regional computation scheme in an AR- assisted in situ CNC simulation environment. In: Computer-Aided Design, Vol. 42, Issue 12, pp. 1167-77. https://doi.org/10.1016/j.cad.2010.06.007. Go to original source...
    21. VAIDYA, S., AMBAD, P. AND BHOSLE, S. (2018), Industry 4.0 - A Glimpse, Procedia Manufacturing. Procedia Manufacturing, Vol. 20, pp. 233-238. https://doi.org/10.1016/j.promfg.2018.02.034 Go to original source...
    22. KRYNKE M. Personnel Management on the Production Line Using the FlexSim Simulation Environment, In: Manufacturing Technology Journal, Vol. 21, 2021, No. 5, pp. 657-667. ISSN: 1213-2489. Go to original source...
    23. KRCMARIK D., PETRU M., MASIN I. (2019). Increasing of Precision Technology of Glass Sorting Based on Very Fast Reconfigurable Image Processing, In: Manufacturing Technology Journal, Vol. 19, No. 3, pp. 431-438. ISSN: 1213-2489. Go to original source...
    24. DOBROCKY D., JOSKA Z., STUDENY Z., POKORNY Z., SVOBODA E., (2020). Quality Evaluation of Carburized Surfaces of Steels Used in Military Technology, in: Manufacturing Technology Journal, Vol. 20, No. 2, pp. 152-161. ISSN: 1213-2489. Go to original source...
    25. SITZMANN V., SERRANO A., PAVEL A., AGRAWALA M., GUTIRREZ D., MASIA B., WETZSTEIN G. Saliency in vr: How do people explore virtual environments? [online2021-09-21].
    26. ROLDÁN J., CRESPO E., MARTÍN-BARRIO A., PEÑA-TAPIA E., BARRIENTOS A. (2019). A training system for Industry 4.0 operatos in complex assemblies based on virtual reality and process mining, In: Robotics and Computer Integrated Manufacturing Vol. 59 305-316. https://doi.org/10.1016/j.rcim.2019.05.004 Go to original source...
    27. GAVISH N. , GUTIÉRREZ T., WEBEL S., RODRÍGUEZ J., PEVERI M., BOCKHOLT U., TECCHIA F. (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks, In: Interactive Learning Environments, Vol.23, No. 6 pp. 778-798. https://doi.org/10.1080/10494820.2013.815221 Go to original source...
    28. GRABOWSKI A., ANKOWSKI J.J. (2015), Virtualreality-based pilot training forunder ground coal miners, Safety Science, Vol. 72:310-314. DOI:10.1016/j.ssci.2014.09.017. Go to original source...
    29. PÉREZ L., DIEZ E., USAMENTIAGA R., GARCÍA D.F. (2019). Industrial robot control and operator training using virtual reality interfaces. Computers in Industry, Vol. 109, No. 6, pp.114-120. DOI:10.1016/j.compind.2019.05.001 Go to original source...
    30. WOLFARTSBERGER J., ZENISEK J., SIEVI C., (2018). Chances and limitations of a virtualreality-supported tool for decision making in industrial engineering. IFAC-PapersOnLine, Vol. 51, No. 11, pp. 637-642. DOI:10.1016/j.ifacol.2018.08.390. Go to original source...
    31. ANDERSON R., GALLUP D., BARRON J.T., KONTKANEN, N. SNAVELY J., HERNANDEZ C. (2016), Jump: virtual reality video. In: Technical Papers, Vol. 35 No. 6, pp. 1-13. ISBN: 978-1-4503-4514-9/16/12 Go to original source...
    32. MULLER W.K., ZIEGLER R., BAUER A. (1995). Virtual reality in surgical arthroscopic training. In: Journal of Image Guided Surgery, Vol. 1, No. 5, pp. 288-294. doi: 10.1002/(SICI)1522-712X(1995)1:5<288::AID-IGS5>3.0.CO;2-6. Go to original source...
    33. MIYAMOTO K.. (1964). Fish eye lens. In: Journal of the Optical Society of America, Vol. 54, No. 8, pp. 1060-1061. DOI:10.1364/JOSA.54.001060. Go to original source...
    34. CUI Z., HENG L., YEO Y.C., GEIGER A., POLLEFEYS M., SATTLET T. (2019) Real-time dense mapping for self-driving vehicles using fisheye cameras. In: International Conference on Robotics and Automation (ICRA) IEEE. https://doi.org/10.48550/arXiv.1809.06132 Go to original source...
    35. FINDEISEN M., MEINEL L., HES M., APITZSCH A., HIRTZ G. (2013). A fast approach for omnidirectional surveillance with multiple virtual perspective views. In: Computer Science Eurocon 2013. IEEE DOI:10.1109/EUROCON.2013.6625187 Go to original source...
    36. LEVOY M., HANRAHAN P. (1996). Light field rendering. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH, pp. 31-42https://doi.org/10.1145/237170.237199 Go to original source...
    37. WILBURN B., JOSHI N., VAISH V., TALVALA E-V., ANTUNEZ E., BARTH A., ADAMS A., HOROWITZ M., LEVOY L. (2005). High performance imaging using large camera arrays. ACM Transactions on Graphics. Vol. 24, No. 3, pp. 765-776. DOI:10.1145/1073204.1073259 Go to original source...
    38. DANSEREAU D.G., SCHUSTER G., FORD J., WETZSTEIN G. A wide-field-ofview monocentric light field camera. CVPR 2017. Go to original source...
    39. TANAKA K., TACHI S. (2005). Tornado: Omnistereo video imaging with rotating optics. In: IEEE transactions on visualization and computer graphics, Vol. 11, No. 6, pp. 614-625. DOI: 10.1109/TVCG.2005.107 Go to original source...
    40. AGGARWAL R., VOHRA A., NAMBOODIRI A.M.( 2016). Panoramic stereo videos with a single camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3755-3763. Go to original source...
    41. MATZEN K., COHEN M.F., EVANS B., KOPF J., SZELISKI R.. (2017). Lowcost 360 stereo photography and video capture. In: ACM Transactions on Graphics (TOG), Vol. 36 No. 4, pp.1-12. https://doi.org/10.1145/3072959.3073645. Go to original source...
    42. ADAM M., JUNG C., ROTH S., BRUNNETT G. (2009). Real-time stereo-image stitching using GPU-based belief propagation. VMV, pp 215-224,
    43. KONRAD R., DANSEREAU D.G., MASOOD A., WETZSTEIN G. (2017) Spinvr: towards live-streaming 3d virtual reality video. In: ACM Transactions on Graphics, Vol. 36, No. 6, pp.1-12, DOI:10.1145/3130800.3130836 Go to original source...
    44. Samsung gear vr. [online 2022].
    45. KOPPAL S., ZITNICK C.L., COHEN M., BING KANG S., RESSLER B., COLBURN A. (2011). A viewer-centric editor for 3d movies. IEEE Computer Graphics and Applications, 31(1):20-35. Go to original source...
    46. GAO Z., HWANG A., ZHAI G., PELI E. (2018). Correcting geometric distortionsin stereoscopic 3d imaging. PlOS ONE, Vol. 13, No. 10:e0205032. .https:// doi.org/10.1371/journal.pone.0205032 Go to original source...
    47. PELEG S., BENEZRA M., PRITCH Y. (2001). Omnistereo: Panoramic stereo imaging. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 3, pp. 279-290. DOI: 10.1109/34.910880 Go to original source...
    48. ROLDÁN J.J., E. PEÑA-TAPIA, A. MARTÍN-BARRIO, M.A. OLIVARES-MÉNDEZ, J. DEL CERRO, BARRIENTOS A., (2017) Multi-robot interfaces and operator situational awareness: study of the impact of immersion and prediction, In: Sensors , Vol. 17 No. 8, pp. 1720. https://doi.org/10.3390/s17081720. Go to original source...
    49. PEDRAM S., P. PEREZ, S. PALMISANO, M. FARRELLY, A systematic approach to evaluate the role of virtual reality as a safety training tool in the context of the mining industry(2016).
    50. FANG H., S. ONG, A. NEE, Interactive robot trajectory planning and simulation using augmented reality, Rob. Comput.-Integr. Manuf. 28 (2) (2012) 227-237. Go to original source...
    51. CECIL J., D. POWELL, VASQUEZ D. (2007), Assembly and manipulation of micro devices - a state of the art survey, In: Robotics and Computer-Integrated Manufacturing, Vol. 23, No. 5, pp. 580-588. DOI:10.1016/j.rcim.2006.05.010 Go to original source...
    52. ZHANG H.. (2017). Head-mounted display-based intuitive virtual reality training system for the mining industry, In: . International Journal of Mining Science and Technology, Vol. 27, No. pp. 717-722. DOI:10.1016/j.ijmst.2017.05.005. Go to original source...
    53. MAVRIKIOS D., KARABATSOU V., FRAGOS D., CHRYSSOLOURIS G.. (2006). A prototype virtual reality-based demonstrator for immersive and interactive simulation of welding processes, Int. J. Comput. Integr. Manuf. Vol. 19, No. 03, pp.294-300. ISSN 1362-3052 Go to original source...
    54. CHRYSSOLOURIS G., D. MAVRIKIOS, D. FRAGOS, V. KARABATSOU, PISTIOLIS K. (2002), A novel virtual experimentation approach to planning and training for manufacturing processes-the virtual machine shop, In International Journal of Computer Integrated Manufacturing, Vol. 15, No. 3, pp. 214-221. DOI:10.1080/09511920110034978 Go to original source...
    55. BALCO P., BAJZÍK P., İKOVIEROVÁ K.. (2022). Virtual and Augmented Reality in Manufacturing Companies in Slovakia, In: Procedia Computer Science Vol. 201 pp. 313-320, https://doi.org/10.1016/j.procs.2022.03.042 Go to original source...
    56. HOUSHIAR H., ELSEBERG J., BORRMANN D., NUCHTER A.. (2015). A study of projections for key point based registration of panoramic terrestrial 3d laser scan. In: Geo-spatial Information Science, Vo;. 18 No. 1, pp.11-31. DOI:10.1080/10095020.2015.1017913 Go to original source...
    57. SWART D. Drawing a spherical panorama. [online].

    This is an open access article distributed under the terms of the Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0), which permits non-comercial use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content