Information technologies and computer sciences

Multimodal Data Representation Model for Complex Description of Observation Objects

Authors

I. A. Dychka
Igor Sikorsky Kyiv Polytechnic Institute
Ye. S. Sulema
Igor Sikorsky Kyiv Polytechnic Institute
DOI https://doi.org/10.31649/1997-9266-2020-148-1-53-60

Keywords

data representation model multimodal data big data

Abstract

The paper is devoted to solving the problem of the complex presentation of multimodal data about an object of observation, the characteristics of which are measured and investigated taking into account the time stamps and the relationship between the data of various modalities. This problem is relevant for many engineering tasks. In particular, the comprehensive presentation of data on a physical object is important in the technology of digital twins, which has recently become more widely used. Based on the fact that a digital twin is determined by both a behavioral model and a visual model, the paper presents a data model based on the concept of a multimodal data element. This model makes possible a comprehensive and consistent presentation of data about the object of observation.

The mathematical model of the complex presentation of multimodal data is based on the concepts of an aggregate and a multi-image defined in an algebraic system of aggregates. The article analyzes the relationship between the complex presentation and processing of multimodal data, which are determined in time, using the apparatus of the algebraic system of aggregates and the consideration of aggregated data as functions of several variables. Thus, the analysis of the data of the muxel model can be performed using logical operations, ordering operations, and relations defined in the algebraic system of aggregates as well as operations and approaches defined in other mathematical concepts.

Since the muxel model presents large amounts of data, it is advisable to optimize the way how data is presented. This can be achieved by data compression, modification of the data structure, or a combination of the first and second approaches. In the first approach, it is advisable to apply muxel model data compression based on the RLE method. In the second approach, it is proposed to apply data structures similar to the Sparse Voxel Octree, which is used to reduce data volume of models in voxel graphics. The practical implementation of the processing of multimodal data presented using the muxel model can be performed by means of ASAMPL programming language.

439 362

How to Cite

[1]
“Multimodal Data Representation Model for Complex Description of Observation Objects”, Вісник ВПІ, no. 1, pp. 53–60, Feb. 2020, doi: 10.31649/1997-9266-2020-148-1-53-60.

Author Biographies

I. A. Dychka, Igor Sikorsky Kyiv Polytechnic Institute

Dr. Sc. (Eng.), Professor, Dean of the Department of Applied Mathematics;

Ye. S. Sulema, Igor Sikorsky Kyiv Polytechnic Institute

Cand. Sc. (Eng.), Associate Professor, Associate Professor of the Chair of Computer Systems Software

References

І. Г. Цмоць, та ін., «Система моніторингу технологічних процесів «розумного підприємства,» Вісник Національного університету «Львівська політехніка» «Інформаційні системи та мережі», вип. 887, c. 10-17, 2018.
M. Grieves, and J. Vickers, “Digital Twin: Mitigating Unpredictable, Undesirable Emergent Behavior in Complex Systems,” in Transdisciplinary Perspectives on Complex Systems. Switzerland: Springer International Publishing, 2018, pp. 85-113. https://dx.doi.org/10.1007/978-3-319-38756-7_4 .
B. A. Talkhestani, N. Jazdi, W. Schlögl, and M. Weyrich, “Consistency check to synchronize the Digital Twin of manufacturing automation based on anchor points,” in Proc. 51st CIRP Conf. on Manufacturing Systems, Stockholm, 2018, pp. 159-164. https://dx.doi.org/10.1016/j.procir.2018.03.166 .
A. Madni, et al., “Leveraging Digital Twin Technology in Model-Based Systems Engineering,” Systems, no. 7 (7), pp. 1-13, 2019. https://dx.doi.org/10.3390/systems7010007 .
L. Hu, et al., “Modeling of Cloud-Based Digital Twins for Smart Manufacturing with MT Connect,” Procedia Manufacturing, vol. 26, pp. 1193-1203, 2018. https://dx.doi.org/10.1016/j.promfg.2018.07.155 .
D. L. Hill, C. Studholme, and D. J. Hawkes, “Voxel similarity measures for automated image registration,” SPIE Proceedings “Visualization in Biomedical Computing,” vol. 2359, no. 205, 1994. https://dx.doi.org/10.1117/12.185180 .
J. Carnero, D. Diaz-Pernil, J. L. Mari, and P. Real, “Doxelo: Towards a Software for Processing and Visualizing Topology Computations in Doxel-based 3D+t Images,” in Proc. 16th International Conf. on Applications of Computer Algebra ACA'10, Vlora, 2010.
Ye. Sulema, I. Dychka, and O. Sulema, “Multimodal Data Representation Models for Virtual, Remote, and Mixed Laboratories Development,” in Lecture Notes in Networks and Systems, vol. 47, Springer Cham, 2018, pp. 559-569.
I. Dychka, and Ye. Sulema, “Logical Operations in Algebraic System of Aggregates for Multimodal Data Representation and Processing,” KPI Science News, vol. 6, pp. 44-52, 2018.
I. Dychka, and Ye. Sulema, “Ordering Operations in Algebraic System of Aggregates for Multi-Image Data Processing,” KPI Science News, vol. 1, pp. 15-23, 2019.
P. Davies, and I. Maconochie, “The Relationship Between Body Temperature, Heart Rate and Respiratory Rate in Children,” Emergency Medicine Journal, vol. 26 (9), 2008. https://dx.doi.org/10.1136/emj.2008.061598 .
В. А. Свідерський, та ін., «Програмне забезпечення технології портландцементу,» Строительные материалы и изделия, № 1, с. 16-17, 2014. [Електронний ресурс]. Режим доступу: http://nbuv.gov.ua/ujrn/smii_2014_1_7. Дата звернення: Січ. 29, 2020.
В. А. Свідерський, та ін., «Програмне забезпечення технології низькотемпературних в’яжучих матеріалів,» Строительные материалы и изделия, № 1-2, с. 22-24, 2017. [Електронний ресурс]. Режим доступу: http://nbuv.gov.ua/ujrn/smii_2017_1-2_11 . Дата звернення: Січ. 29, 2020.
J. Zhu, and A. Wang, “Data Modeling for Big Data,” CA Technology Exchange. Insights from CA Technologies, vol. 3 (2), pp. 77-82, 2012.
G. E. Blelloch, Introduction to Data Compression. Pittsburgh, Pennsylvania, USA: Carnegie Mellon University, 2001.
S. Laine, and T. Karras, “Efficient Sparse Voxel Octrees – Analysis, Extensions, and Implementation,”іn Vidia Technical Report NVR-2010-001, 2010.
K. H. Höhne, A. Pommert, M. Riemer, and U. Tiede, “3D Visualization of tomographic volume data using the generalized voxel model,” The Visual Computer, vol. 6, pp. 28-36, 1990.
CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modelling of Sensory Data from a Mobile Robot, in IEEE DataPort, 2018. [Electronic resource]. Available: https://ieee-dataport.org/open-access/create-multimodal-dataset-unsupervised-learning-and-generative-modeling-sensory-data .
Accessed: Feb. 10, 2020.
S. Brodeur, S. Carrier, and J. Rouat, “CREATE: Multimodal Dataset for Unsupervised Learning, Generative Modelling and Prediction of Sensory Data from a Mobile Robot in Indoor Environments,” 35 p., 2018. [Electronic resource]. Available: https://ieee-dataport.s3.amazonaws.com/docs/2537/specifications.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJOHYI4KJCE6Q7MIQ%2F20200211%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200211T212311Z&X-Amz-SignedHeaders=Host&X-Amz-Expires=86400&X-Amz-Signature=25f1356de0fed1f805bb45d1d144a84f0ede96521c095a25eea8d0d6e1acd360 . Accessed: Feb. 10, 2020.
P. Atzeni, F. Bugiotti, L. Cabibbo, and R. Torlone, “Data modeling in the NoSQL world,” Computer Standards & Interfaces, vol. 67, pp. 1-11, 2020.
Ye. Sulema “ASAMPL: Programming Language for Mulsemedia Data Processing Based on Algebraic System of Aggregates,” in Advances in Intelligent Systems and Computing. Springer, Cham, vol. 725, pp. 431-442, 2018. https://dx.doi.org/10.1007/978-3-319-75175-7_43 .