Beyond the hype surrounding the concept of the metaverse, and the many dreamed up future experiences people may have, there remain many obstacles to be tackled.
One such obstacle is the problem of latency.
As David Clark put it in the Pew Research Center paper The Metaverse in 2040: “The fundamental barrier to remote, high interactivity on the Internet is latency, and latency will not improve, because the Internet is moving data today at almost the speed of light, and the speed of light is a constant. If multiple participants in a metaverse are in the same metro area (for example) it may be possible to get the latency down to the point where tight real-time interaction can happen with reasonable quality, but interaction (say) across the country will always have about 100 milliseconds of round-trip delay, and that means (again, for example) that we will never be able to create live music with widely distributed performers. One hundred milliseconds is just too much delay for synchronized music.” (Anderson & Rainie, 2022)
But what if recent developments in AI could help address this issue?
Most people have become familiar with text prediction software, as it has been rolled out across multiple apps such as email (gmail for instance) and instant messaging. Such a system is designed to predict the next word you are about to write to boost your writing speed.
Recently, there have been significant advances in AI, with the emergence of sophisticated chatbots such as ChatGPT, but also research in predictive software for music and image generation.
A few examples include:
- Google’s AudioLM, which is designed to continue an audio sound based on a 3 second prompt: https://google-research.github.io/seanet/audiolm/examples/
- Google’s Deepming “Transframer” project which creates a video based on an image: https://sites.google.com/view/transframer?pli=1
How is this relevant to reducing latency?
Such technologies could be used to anticipate audio or video contents to compensate for the latency. Rather than hearing the “raw” audio or video stream, a user would see an AI generated audio/video feed which compensates for the latency by using the raw data streams as they come in. For the moment, such a solution would probably require heavy computing power. Thus the raw audio/video streams would have to be fed into a data center with the necessary computing power, and transferred to the user on the other end.
Using AI to cut down on latency has already been used, such as for the anticipation of tactile feedback while interacting with virtual objects in simulated environments.
As these technologies improve, it might be possible to use the recipient’s device to generate the predictive audio/video stream on the fly, without having to go through a data center. This technology could also drastically change how we exploit bandwidth. There has also been substantial progress in image enhancement AI tools. Thus if computing power increases sufficiently, data could be sent via the Internet in low or highly compressed quality and enhanced on the fly at the point of viewing. Thus for a band to play together, not only would they use AI tools which compensate for latency, but the actual audio stream would be compressed to limit the bandwidth use, and enhanced on the other end using AI.
Such AI tools would evolve greatly over a short period of time since any such experience would provide training data to further improve the AI, as it could compare the “raw” audio/video to the predicted audio/video it generated. Regarding synchronisation of musicians, such AI tools would probably require musicians to “jam” for a few seconds to calibrate the AI, and allow it to work efficiently, without an adjustment period of a few seconds as the AI tries to make sense of the initial audio/video feed it is getting.
Such tools could also compensate for fluctuations in bandwidth, quality of the data stream and latency in real-time.
Finally, in parallel to these developments, the generalized use of decentralized/distributed computing, mesh networking and multicast standard for broadcast could greatly support the efficiency of such AI tools.
