By: Dijam Panigrahi
We don’t know which of the world’s largest technology companies will drive the best future tools, technologies and resources for manufacturing, healthcare, construction and more. That is why organizations have been working very hard to make sure they are creating changes that will greatly affect humanity. This begins when there have been recent technological advances with artificial intelligence (AI) and immersive mixed reality technologies such as augmented reality (AR) and virtual reality (VR).
Although these technologies differ from each other, they currently work together in advanced three-dimensional (3D) applications and environments, as it benefits companies and their customers.
In virtual reality, a user wears headphones that allow entry into a new world, which can even mimic the real world. Virtual reality allows users to have a visual and audible experience that will duplicate a real environment in a manufacturing environment.
Augmented reality is conceptually similar to virtual reality. However, augmented reality shows digital content in the real world. This allows power manufacturers, utilities or industrial equipment involved in creating new machinery to see the virtual specifications of the design. In turn, they also see how it could work in a real utility or power generation environment.
Certainly, these technologies are promising. The challenge, however, is that they require large doses of data, the ability to process large amounts of data at remarkable speeds, and the ability to scale projects in a technological environment that is rarely allowed in typical office environments.
Immersive mixed reality requires a precise and persistent fusion of both the real and the virtual world. Therefore, it is required to represent complex models and scenes with photorealistic details, represented in the correct physical location with the scale and precise position. To take advantage of AR / VR to design, build or repair components, you need persistent precision and precise nature.
Currently, this is achieved by using discrete GPUs from servers and delivering rendered frames wirelessly to head-mounted displays (HMDs), such as Microsoft HoloLens and Oculus Quest.
One of the main requirements for mixed reality applications is to precisely overlay on an object its model or digital twin. In this way, work instructions for assembly and training can be provided, and possible manufacturing errors can also be detected. This allows the user to also track the object and modify the representation as the work progresses.
Most device tracking systems on the device use tracking based on 2D images and / or markers. This severely limits the accuracy of the 3D overlay, as 2D tracking cannot estimate depth with high accuracy and therefore scale and position. While users can receive what appears to be a good match when viewed from an angle or position, the overlay loses alignment as the user moves in 6DOF.
In addition, object registration is achieved, which is the detection, identification and estimation of scale and orientation of objects. In most cases, this is achieved computationally or through simple computer vision methods with standard training libraries (examples: Google MediaPipe, VisionLib). This can work well for normal, smaller, simpler objects such as hands, faces, cups, tables, chairs, wheels, regular geometry structures, and so on. However, for larger and more complex objects in business use cases, labeled training data (even more so in 3D) is not readily available. As a result, using 2D image-based tracking to align, overlay, and persistently track the object and merge the model represented with it in 3D is extremely difficult, if not impossible. Business-level users are overcoming these hurdles by leveraging 3D environments and artificial intelligence technology in their immersive mixed reality design creation projects.
3D AI based on deep learning allows users to identify 3D objects of arbitrary shape and size in various orientations with high precision in 3D space. This approach is scalable in any arbitrary form and can be used in business use cases that require a complex 3D rendering overlay.