The Building Blocks of the Metaverse

Extended reality and artificial intelligence will be fundamental building blocks in the Metaverse, with computing power, network capacity and payment systems as key prerequisites. Technology will increase challenges associated with data protection, manipulation and surveillance.

Presence

Embedded in the vision for the Metaverse is the idea of providing people with a sense of actual presence.

A sense of immersive presence could be enhanced both by strengthening the connection between the person and the avatar and by ensuring that the actual surroundings and the experiences in the virtual universes are realistic.

Extended reality and artificial intelligence will be particularly important for people to feel present in the Metaverse through their avatars.

Extended Reality

Extended reality (XR) is a collective term for various types of data-generated environments:

Virtual reality (VR): A fully virtual world that blocks out the physical surroundings. The most immersive experience of VR is achieved through VR headsets that replace reality with virtual environments. Devices with sensor technology, such as haptic suits or gloves, can help make the experience more realistic and immersive. Haptic devices allow people to feel vibrations, movements, resistance and similar mechanical simulations.
Augmented reality (AR): A partially virtual world in which virtual elements are placed on top of the physical surroundings. Unlike VR headsets, AR headsets let in light so that users can see both their physical surroundings and the virtual elements. This technology can also be used through the camera on a smart phone or tablet.
Mixed reality (MR): Combines elements from both VR and AR, for example by using VR headsets to work in a virtual meeting room or office, where you can use several virtual screens, and also see the physical screen and keyboard.
Diminished reality (DR): A data-generated environment where you can reduce and remove physical surroundings and replace them with virtual elements. This technology can, for example, be used to remove the people on a bus and, if you prefer, replace them with cute virtual cats.

XR technology can capture our movements, audio and visual impressions and surroundings. It allows for three-dimensional visual impressions, also referred to as depth perception.²⁴ Unlike 2D, 3D provides an experience of size and depth and constitutes a more visual way of conveying information. VR and AR headsets include projectors, sensors, hologram technology, cameras, speakers and batteries, to name but a few features.

Such headsets can, for example, analyse the surroundings to provide information that the user might find interesting. This information can be displayed in the user’s field of vision. In this way, the physical and virtual elements become interconnected, and reality would as such be extended. Improved audio experiences could, for example, make virtual surroundings more immersive. Research has shown that sound has a crucial effect on zoom fatigue.²⁵ We get more tired from digital meetings because the sound comes from a speaker. In virtual meetings, sound will appear to come from the direction of the person who is talking.

XR technology is becoming more widespread. Snapchat and Pokémon Go filters are examples of existing commercial use of AR. There is also great potential outside of entertainment. The technology can be used for everything from simulation-based training in industry to developing structures, detecting errors or planning and visualisation using virtual 3D models.

In healthcare, the technology has been used to create holograms of organs, which aids diagnostics and surgical planning. Another area that is in development is holographic telemedicine. Virtual doctor’s appointments, for example, could be useful during pandemics in order to reduce the spread of infection.²⁶

Embedded in the vision for the Metaverse is the possibility to experience it fully virtually through VR, but also partially virtually through the usage of other forms of XR technology. Furthermore, the Metaverse could be experienced in 3D, even though games, software and features in the Metaverse would remain in 2D.²⁷

Hardware

To reinforce a sense of presence, hardware – the technological devices that will provide access to the Metaverse – has an important role to play. VR and AR headsets, as well as haptic devices with sensor technology will provide people with the opportunity to perceive, feel and sense in a virtual universe.

Embedded in the vision for the Metaverse is the possibility to experience the Metaverse without VR headsets. Existing digital devices, such as PCs, tablets, smartphones and wearable technology will likely be just as important.²⁸ Our surroundings and devices are becoming smarter and are equipped with sensors and internet connectivity (also referred to as the Internet of Things).²⁹ If the Metaverse becomes a 3D-based internet and reality is extended by virtual elements being placed on top of the physical surroundings, smarter surroundings could make it possible to access the Metaverse without needing specific devices.

In the future, high-tech contacts may replace XR headsets. Perhaps it will even be possible to connect the brain directly to the Metaverse. Currently, these are only hypothetical possibilities. VR and AR headsets will likely provide the most immersive experience of the Metaverse, at least for the foreseeable future.

VR and AR headsets have been available for sale for more than a decade, but the sales figures have been moderate. Apple is rumoured to be developing AR and MR headsets, but no-one knows when these will launch. Meta’s new VR headset, Quest Pro, launched in the autumn of 2022, have received mixed reviews.³⁰ Even though Meta believes that VR headsets will eventually replace PCs, the headsets are still heavy, expensive and the software is underdeveloped.

Mark Zuckerberg himself has stated that bringing together all the technology needed for an immersive experience of the Metaverse in simple spectacle frames is “the hardest technology challenge of our time”.³¹ Billions of dollars are invested in the development. However, whether such glasses will replace or supplement existing digital devices remains to be seen.

Artificial Intelligence

In simple terms, artificial intelligence is advanced data analysis. By interpreting, analysing, and identifying patterns in large volumes of data, artificial intelligence learns to understand and perform new tasks. This could include something like recreating the feeling of touching an object, in a virtual world.

While extended reality will help connect our senses online, artificial intelligence is what will make our sensory perceptions in the Metaverse feel realistic. This technology will be crucial for people to be able to see, listen, feel and participate in virtual experiences using their avatars. This could enhance the experience of presence and immersion in the Metaverse.

Research has shown that the more realistic the avatars seem, the more immersive the virtual surroundings appear.³² Meta’s Quest Pro VR headset includes features that will make avatars more like humans. Quest Pro has an embedded facial expression recognition feature, allowing your avatar to mirror your actual facial expressions. Artificial intelligence enables the sensors to capture, interpret and convey such details.

Artificial intelligence is crucial for several avatar features. Meta announced that “legs are coming” to the Metaverse in October 2022.³³ Until now, avatars have been floating torsos because VR and AR headsets are better at reading and analysing upper body movements than leg movements. Now, however, the systems will be trained to predict leg movements based on how the upper body moves, using artificial intelligence.

The technology will also play an important part in making virtual surroundings and experiences sufficiently detailed. It will be able to process the surroundings in the Metaverse in real time and convey this information to people in an understandable and meaningful way.

Artificial intelligence will also be used to produce content in the Metaverse. The technology can, for example, translate text and language in real time, generate content such as images, video and text, and convey these using advanced algorithms.

Data Collection and Protection

Several challenges arise from the usage of new technology in the Metaverse. The collection and processing of data and the associated data protection challenges could become much more extensive. People’s personal information has become the most important resource in today’s internet. Online surveillance is big business, and the collection and usage of personal data will most likely play a crucial role in the making of the Metaverse.³⁴ The Metaverse is therefore likely to pose numerous challenges to the right to privacy.

Equipment that provides access to the Metaverse, from PCs to smartphones to VR or AR headsets, and gloves with touch sensors, will be able to record, interpret and analyse large volumes of data. This data can be used to improve digital services, but the large amount of data can also make it harder for users and developers to understand what data is collected and what the data can be used for.

Sensors will be able to capture how you move your body, where you are looking, how long you are looking at something and what makes you stressed, happy or tired. The devices can monitor heart rate, breathing and blood pressure.

Research has shown that using a VR headset for just five minutes provides sufficient data to identify a person with an accuracy of 95 %.³⁵ Such data could not only be used for identification, but also for tailoring and personalising virtual experiences and predicting user behaviour in real time. The data can also be used to identify unconscious psychological and physiological conditions that people are not aware of themselves.³⁶

What type of data that can be collected, who has access to the data and how the data can be used are therefore important questions for the Metaverse. The rights of the Metaverse user will also be an important question of concern. Whether users will have the opportunity to request access to self-generated data or be able to request their personal data to be deleted remains to be seen.

Manipulation and Influence

If extensive data collection and processing are permitted in the Metaverse, this could mean that commercial companies will be left with extensive volumes of data relating to intimate aspects of users’ lives. It can also make people vulnerable to manipulation and influence, not to mention surveillance.

Profiling and targeted advertising are currently widespread on social media and represent the core business model of big tech companies.³⁷ In a Metaverse, where companies would have knowledge of intimate details relating to our bodies, actions and lives, profiling and marketing could also become more advanced, and invasive. The companies’ business models could lead to users choosing to share their user data in exchange for “free access” to services.

The Metaverse enables marketing in far more subtle ways than before. Virtual people (also called veeple), for example advanced forms of chatbots, could be placed in the Metaverse and appear to be just as real as the surroundings. Research has shown that people are unable to distinguish between artificial human faces and real faces and that artificial faces appear more trustworthy and reliable than real faces.³⁸ Virtual people could have a variety of functions in the Metaverse, from providing information to users to advertising products on behalf of a company. However, if left unregulated, it can also enable new forms of influence and manipulation.

Meta has already patented technology that will allow for targeted marketing in virtual worlds. The company will place virtual objects in the surroundings.³⁹ You may assume that the surroundings look the same to everyone, but in fact, it could be personalised, customised, and targeted to you specifically. It could become difficult, if not impossible, to distinguish between commercial and non-commercial content in the Metaverse, unless requirements are imposed on advertisers and platforms to regulate, label and manage such content.

In immersive virtual universes, where it is difficult to distinguish real from false, algorithms may have an even greater influence on people, and deepfake technology can become more widespread. This gives commercial companies great power and increases the risk of manipulative content and design.

There are also several unresolved questions linked to the psychological and health-related consequences of virtual worlds and experiences. We do not have a complete picture of how invasive the Metaverse can be. Disinformation, fraud and radicalisation could take place in real time. We know that VR can cause nausea, dizziness and balance disorders, but further research is also required on the psychological and health-related long-term effects of XR technology.⁴⁰

Avatars and Virtual People

A number of legal questions arise in relation to digital representation and the status of virtual people in the Metaverse.

The technology enables our digital twin, whether an avatar or hologram, to share our biometric characteristics. It can have the same voice, eyes, walk, motor skills and facial expressions as the person themselves. Is your avatar you since it connects you and your biometrics? If your avatar is harassed or the victim of a criminal act, was the victim the person, the avatar or both? How would this be regulated? What if it becomes possible to instruct your avatar to perform tasks in the Metaverse even when you are logged off and your avatar is subjected to hate speech or other criminal acts? Did this also happen to your physical person?

The development of virtual people also introduces several new challenges. Who is responsible for virtual people in the Metaverse? Is it the developer of the algorithm behind the virtual person, the company using the technology or the platform that enables such use? Bias in artificial intelligence may occur as a result of machine learning, and this can be difficult to detect. How do you prosecute a virtual person, for example, if they discriminate or express hate speech as a result of machine learning or pre-programmed code? The status of virtual people is also a question of concern. Can people do whatever they want with virtual people simply because they are not real people?

These are complex legal issues of concern for the future Metaverse.

Coherence

One characteristic of the virtual worlds in the Metaverse that will distinguish them from existing online platforms is that they will be connected. This means that people will be able to freely move (their avatars), including their virtual assets and history, between different worlds and services using a single identity (often referred to as data portability).

Interoperability and Data Portability

The Metaverse will likely consist of many connected worlds and services. Unlike operating systems such as Android and iOS, which cannot easily be unified, the vision for the Metaverse is that the different universes and services will be connected in a coherent and transparent manner. Interoperability is an important feature of the Metaverse.

Today’s gaming platforms and virtual worlds are largely closed systems. This means, for example, that they store objects and user data in different formats. They are built and operated using different code and there are not necessarily any efficient systems available to share or convert data across the platforms. They also have their own currencies and economic systems based on the distinctive rules applicable to the platforms. The experiences and opportunities in the platforms are also incompatible, from VR and AR experiences to 2D and 3D games.

An interoperable Metaverse requires agreement on standard formats. Achieving this is easier said than done. It would entail standards for infrastructure, coding, technical devices, user interfaces, data protection, digital currencies, and ownership, to mention some. These standards must also be subject to governance and legislation.

Without interoperability, it would be possible to end up with several prototype metaverses or so-called betaverses. There, users could have different identities in different virtual worlds, but limited opportunities to move themselves, their history and their assets between the worlds.⁴¹

A Virtual Economy

A virtual economy is required for people to be able to buy services and goods in the Metaverse. The economy must work in each virtual world, across different worlds and between the Metaverse and the physical reality. The idea is that people should be able to bring their assets safely and efficiently from one universe to another. One system to track, validate, register, modify and transfer ownership would therefore be critical for ensuring a coherent Metaverse economy.⁴²

A Payment System

Today, the virtual worlds use different economic systems. Gaming platforms like Fortnite and Roblox have developed their own economies, in which money can be exchanged for the in-game currency: Robux in Roblox and V-bucks in Fortnite. The rules are strict, and it can be difficult to give money to other users or to exchange the game currency back to regular money.⁴³ It has also proved difficult to change e.g., Linden dollars from Second Life to Emeralds from Minecraft due to the differences in the underlying system architecture.⁴⁴

Cryptocurrencies are used in the blockchain-based worlds The Sandbox and Decentraland. Virtual assets and property are also sold as non-fungible tokens (NFTs) in these worlds. The NFTs represent ownership of unique, digital files that cannot be copied. These could be unique digital artworks, virtual goods such as the avatar’s trainers or a virtual property.⁴⁵

In order to exchange cryptocurrencies between different blockchain-based virtual worlds, a crypto exchange can be used. Cryptographic wallets can provide a suitable way of storing cryptocurrencies.⁴⁶ Blockchain technology and cryptocurrencies allow for the efficient exchange and transfer of assets across platforms. Cryptocurrencies, NFTs and other cryptoassets can therefore form the basis for the economy in a virtual Metaverse. Proponents of blockchain technology will argue that this technology is critical to realising the Metaverse.

Nevertheless, it is not certain that blockchain technology will become one of the building blocks of the Metaverse. Many people are critical of the technology, and it is too early to say what role it might play.⁴⁷ Cryptocurrencies are characterised by little regulation, major price fluctuations and lots of speculation and have led to new forms of fraud and money laundering.

The development of efficient money and payment systems that work between universes and the physical world will nevertheless be central to several features of the Metaverse. There are many regulatory and policy-related questions associated with a virtual economy – questions relating to ownership, taxation, regulation and supervision will be crucial.

Power and Control

Who gains power and control in the Metaverse will be strongly linked to the economic models and standards that are adopted.

In a centralised Metaverse, it is likely that a small number of players would have great power. Such players could, for example, charge a fee for transactions on the platform as they would be able to determine the rules for the economic system. This is, for example, done by Roblox today.⁴⁸ In such a centralised model, users will not necessarily have the freedom to control or own parts of the digital surroundings.

Users may have a greater degree of control over their own data and assets in a decentralised Metaverse. The blockchain could be used to effectively verify identity and users could have the opportunity to shape and influence the virtual world itself. In such a model, decentralised autonomous organisations (DAOs) – organisations coded as data programs on the blockchain and controlled by the organisation’s members themselves – could rule and make decisions based on what the majority prefers.

It is also possible to envisage a combination of a centralised and decentralised basic structure. Such a Metaverse could, for example, consist of different worlds and services controlled and managed by tech companies, but bound together by cryptocurrencies.

There is great competition to develop the standards that will apply to the Metaverse. The company or companies that succeed with their solutions could assume a dominant position in the future virtual worlds. If the tech companies become the gatekeepers of the payment systems in the Metaverse, this could lead to major market advantages, a concentration of power and prevent innovation and fair competition.⁴⁹ In other words, there is a lot at stake when standards are developed.

Microsoft, Meta and a number of other tech companies recently entered into a collaboration to develop the standards for the Metaverse.⁵⁰ These common standards are supposed to make it easier for developers to build the Metaverse and will apply to 3D graphics, avatars, network functionalities and XR technology. Apple is not part of this network. This could result in an Android-iOS battle for the Metaverse, in the sense that the companies will compete to define the future standards of the Metaverse.⁵¹

Identity

Having a single identity across several virtual platforms and services leads to various challenges. It could blur the line between a private and a public persona and allow for data collection across platforms and services. This could contribute to more advanced profiling and tracking. It may also make people more vulnerable to identity theft and computer attacks. Avatar duplication or other criminal acts against avatars may occur.⁵² Challenges may also arise with regards to identity and age verification, also in a situation in which users are able to choose the appearance and design of their avatars.

This also gives rise to questions relating to users’ freedom of choice. Should you have the right to remain anonymous in the Metaverse? Are you tied to one identity, one gender and one specific design in all universes and to what extent do you have the freedom to choose? As women are subjected to more sexual harassment on gaming platforms than men, it is possible to imagine that fewer people would select the female gender for their avatars. This could lead to less gender diversity and more underrepresented groups in the Metaverse (issues related to discrimination is further discussed in the chapter “Human rights in the Metaverse”).⁵³

Simultaneity

One characteristic of the Metaverse is that an unlimited number of people should be able to interact simultaneously. This may not be too hard to imagine, given that an unlimited number of people can be connected to the internet simultaneously. However, a virtual presence in real time will require far more network connectivity and speed than what is possible with current technology. The Metaverse will exist independently of time and place, and the vision is that anyone should be able to access the Metaverse at any time.

The importance of network capacity, speed and computing power can be illustrated in the Metaverse Sniper Problem.⁵⁴ In theory, if you use a sniper rifle to zoom in on an area far away in the Metaverse, you should be able to view that area in a high resolution to make a precision shot. This is already difficult in today’s computer games. In a Metaverse with thousands of simultaneous users, this would require significant network capacity and computing power.

Computing Power, Bandwidth and Latency

Computing Power

Virtual 3D environments and a large number of concurrent users place completely different demands on computing power and processing than today’s internet. One of the world’s largest chip manufacturers, Intel, believes that computing power needs to increase nearly a thousand times from the current situation in order to support a metaverse.⁵⁵

The need for computing power will also affect consumers and the type of devices and computers they need to access the Metaverse. Many of the current computers and devices for extended reality will likely not have the required computing power to participate in the Metaverse. This creates challenges for developers – can we assume that users will have powerful enough devices to handle the need for computing power locally or do we need to focus on cloud solutions for the majority of processing? While a local solution would increase the price of user devices, a cloud solution would entail incredibly high demands in terms of internet access and latency to avoid lag.⁵⁶

Another possible solution is so-called edge computing. Here, the storage and processing of data will take place closer to the consumer using a decentralised solution. In this way, it becomes possible to save both time and computing power as data is transmitted over a shorter distance rather than to a centralised server in the cloud.

Bandwidth

The internet connection will be important in the Metaverse as it determines how much data can be transferred per second. Services that transfer text, sound and images require significantly more bandwidth than simple services like e-mail.

Norway has excellent broadband compared to other countries. According to the Norwegian Communications Authority, 90 % of all Norwegian households have access to broadband with download speeds of at least 100 Mbit per second, while 86,5 % have access to at least 1,000 Mbit/s via fibre and cable TV networks.⁵⁷ This is not the case globally. Nearly half of the global population does not have access to the internet.⁵⁸

Latency

Latency describes the time it takes to transmit data from one point to another. In practice, the term is used to describe lag, for example how long it takes from clicking the pause button in a YouTube video until the video actually stops. Lag rarely plays a big part in asynchronous services and many other services. It usually makes little difference if clicking pause results in a few milliseconds or even a whole second of lag.

However, in games and other services in which your actions need to take place in real time, there is a very low tolerance for lag.⁵⁹ For example, you might find that the tackle you make in a football game occurs after your opponent has passed you, because the lag is too long. This contributes to lowering the quality of the experience.

In video calls, it will also be frustrating if audio and video are out of sync or facial expressions and gestures do not occur at the same time as the voice is heard. This means that social interactions become more unnatural and robotic, which is the very opposite of the vision for the Metaverse.

Further challenges may arise when using technology for virtual or extended reality. For example, if you turn your head or use handheld controllers to react to something, latency must be low enough to ensure that the surroundings do not “lag behind” these movements.

The lowest possible latency will not only ensure a better experience for the user but will likely also be important to developers for commercial reasons. Studies have shown that an increase or decrease in latency of 10 milliseconds can result in an increase or decrease in weekly gaming time of six per cent.⁶⁰ If the Metaverse aims to keep users engaged and active, latency will be one of the main challenges that need to be solved.

Distribution and Access

The Metaverse will require significant computing power, speed and network capacity. This gives rise to challenges with regards to who the infrastructure will be developed by and who will pay for it.

Various partnerships are taking shape – Meta has, for example, partnered with the telecommunications operator Telefonica. Together, they run an innovation hub in Madrid where they explore new possibilities for the Metaverse and network connectivity.⁶¹ In the UK, there is already an emerging discussion about who will pay for the development of high-speed infrastructure: telecommunication operators, authorities, service providers (like Meta) or the users themselves.⁶²

In Norway, the right to open access to the internet is enshrined in law. The net neutrality principle means that the internet must be an open and non-discriminatory platform for all forms of communication.⁶³ At some point in the future when we connect to the Metaverse, it will be easy to think that digital services are completely detached from our physical location. However, since network connectivity will be essential, the physical location will have a significant impact on the experiences we can access. In other words, the Metaverse may enhance digital inequality within countries, but also globally.

Carbon Footprint

A Metaverse with unlimited capacity will likely require large amounts of energy.

Data will be stored in cloud solutions run from data centres somewhere around the world. These are critical to current internet infrastructure and enable internet features such as streaming, social media and cloud storage.

A single data centre can use as much power as a small town and most data centres today are not based on renewable energy sources.⁶⁴ Researchers estimate that global data centres will use seven times as much electricity as Norway does today by 2030.⁶⁵

Renewable data centres, on the other hand, can contribute to reducing the carbon footprint of the Metaverse. Several of the largest players seeking to build the Metaverse, have also made promises to reduce emissions. Microsoft intends to become carbon-negative by 2030 and Google aims for its data centres to be carbon-free by the same year.⁶⁶ However, the credibility of these climate goals have been questioned.⁶⁷

Not only could the Metaverse be highly energy-intensive, but the development of the technology, from machine learning for artificial intelligence to the development and production of hardware, could also lead to substantial emissions.⁶⁸

On the other hand, the Metaverse could have climate-positive effects through virtual goods replacing physical goods and virtual interactions reducing the need to travel.⁶⁹ An office based in the Metaverse could be more energy-efficient than operating physical office buildings. However, whether these effects would compensate for the substantial energy consumption required for the development and operation of a Metaverse remains highly uncertain.

Fotnoter:

¹Høvding, Gunnar (2020)
²Sparrow, Mark (2020)
³Brun, Henrik et al. (2020)
⁴Ball, Matthew (2022)
⁵Ball, Matthew (2022)
⁶Norwegian Board of Technology (2015)
⁷Robertson, Adi (2022)
⁸Zuckerberg, Mark (2021)
⁹Dionisio, J. D. N. et al., (2013)
¹⁰Ghaffary, Shirin (2022)
¹¹Norwegian Board of Technology (2016)
¹²Miller, M.R et al. (2020)
¹³Rosenberg, Louis (2022)
¹⁴Norwegian Board of Technology (2016)
¹⁵Nightingale, Sophie J. and Hany Farid (2021)
¹⁶Murphy, Hannah (2022)
¹⁷Davis, Nicola (2016)
¹⁸Betaverses are used by the Copenhagen Institute for Future Studies in a scenario that deals with the emergence of multiple virtual universes at the expense of one connected Metaverse – see Copenhagen Institute for Future Studies (2022).
¹⁹Ball, Matthew (2022, p.124)
²⁰Østbye, Peder (2022)
²¹Chen, Youngwei and Hua Cheng (2022)
²²Norwegian Board of Technology (2022a)
²³Norwegian Board of Technology (2022b)
²⁴O’Reilly, Tim (2021)
²⁵Norwegian Media Authority (2022b)
²⁶EPRS (2022)
²⁷Metaverse Standards Forum (2022)
²⁸Heath, Alex (2022b)
²⁹EPRS (2022)
³⁰Thorpe, Lucy (2022)
³¹Macaulay, Thomas (2022)
³²Gartenberg, Chaim (2021)
³³Ball, Matthew (2022)
³⁴Norwegian Communications Authority (2021)
³⁵World Economic Forum (2020)
³⁶Ball, Matthew (2022)
³⁷Ball, Matthew (2022)
³⁸Dobrynin, Vlad (2022)
³⁹Lewis, Leo (2022)
⁴⁰Norwegian Communications Authority (2022)
⁴¹Haga, Atle (2022)
⁴²Andrae, Anders S. G. (2022); Statistics Norway (2022)
⁴³Paresh, Dave (2020)
⁴⁴Calma, Justine (2022)
⁴⁵Ekin, Anette (2019)
⁴⁶Morini Bianzino, Nicola (2022)