Today the U.S. Patent and Trademark Office has issued an Apple patent that explains how a future iPhone can be used as a 3D controller to play virtual reality games when wearing future virtual reality headsets. The invention also includes an iPhone that is used as a 3D pointer and more.
In its patent history, Apple notes that to allow users to interact with electronic content, it may be desirable for a user to provide information using a separate real-world device, such as an iPhone (or iPad) touch screen. . However, existing systems cannot adequately track the locations of a smartphone or tablet in relation to the content of the CGR environment. This is what the patent granted by Apple solves.
Apple’s patent covers devices, systems, and methods that provide enhanced user interfaces for interacting with electronic content using a variety of electronic devices.
Some implementations include a first device (for example, a head-mounted device (HMD)) that has an image sensor (for example, a camera) and one or more screens, as well as a second device (for example, an iPhone). ) that has display.
The Apple FIG. Figure 2 below illustrates that a user can use an iPhone as an input device in computer-generated reality (CGR) environments while wearing a mixed-reality headset.
The Apple FIG. 3A above illustrates a marker (# 310) showing the second physical device (# 130 an iPhone) as shown in FIG. 2. In some implementations, the user cannot see the physical display of their iPhone because the user is immersed in the virtual scene (No. 205). Consequently, in some implementations, the iPhone displays a marker (# 310) on the iPhone’s physical screen for easy tracking of the iPhone by the head-mounted display (HMD).
In some implementations, the bookmark serves as a reference point for the HMD to accurately track the location and rotation of the iPhone.
In some implementations, the marker (# 310) is an image that contains texture / information that detects the image and determines the position of the image relative to a camera.
Today Apple has updated this patent / invention with 20 new patent claims emphasizing points that were not discussed earlier in order to strengthen the patent of patent trolls and the challenges of competitors.
There are 12 new patent claims that support “One Method”. Here are some examples:
New patent claim # 1: “A method comprising: in a first device comprising a processor, a computer readable storage medium, an image sensor and a first screen: obtaining an image of a physical environment using the image sensor, the physical environment comprising a second device comprising a sensor configured to track a position and orientation of the second device, receiving data corresponding to the position or orientation of the second device followed from the second device determining a relative position and orientation of the second device to the first device based on the received data and generation of a control signal based on the relative position and orientation of the second device.
New patent claim # 2: The method of claim 1, wherein the generated control signal is based on the input of the second device, wherein the first device uses the relative position and orientation of the second device to allow the second device is used as a three-dimensional (3D) controller, a 3D pointer, or a user interface input device. “
New Patent Claim # 7: “The method of claim 1, wherein the sensor is an inertial unit of measurement (IMU).”
To review this patent in general and the new patent claims specifically, see Apple’s continuation patent 20210263584.
Apple’s patent lists a single inventor, Selim BenHimane, senior manager of engineering: computer vision and machine learning. BenHimane leads teams that develop computer vision algorithms and 3D machine learning in the field of augmented reality and virtual reality. BenHimane previously worked at Intel for 4.5 years and at Metaio GmbH for 4.5 years. Apple acquired Metaio in June 2015.