In today’s tool-driven world, technology is at the forefront of human ambition, interconnectivity, and understanding. As a keen humanist, with a mind for product and an eye for design, Alessandro's ultimate aim is to deliver a fresh user experience in the journalism space.
As addictive and stimulating technology proliferates across society, we are losing our most ancient and coveted ability. Join us as we explore the loss of our ability to do nothing and how stand-up comedians have become the unlikely torch bearer of an inconvenient digital truth.
Have you ever tried sitting in a room and doing nothing? And when I mean nothing, I mean absolutelynothing. Chances are you won’t last very long and that’s mainly because the human brain has a ferocious appetite for information stimuli. It’s why meditation is so hard and yet advocated by so many. Fundamentally, we aren’t very good at quieting our brain and the past decade of technological advancement has been anything but helpful.
Setting the scene: Human-Computer Interaction
According to the basic fundamentals of human computer interaction (HCI), there are three mainways or modalities by which we interact with computers:
Visual (Poses, graphics, text, UI, screens, and animations)
Auditory(Music, tones, sound effects, voice)
Physical(Hardware, buttons, haptics, real objects)
Regardless of what computer type you are using — whether it’s a smart phone or laptop — physical inputs and audio/visual outputs dominate HCI. Indeed, these forms of interaction and feedback are the very foundation of how humans have developed computers to function alongside them.
Now take into account another fundamental theme of HCI development because with every successful iteration of technology, there exists a main defining principle: Mainly, people who use technology want to capture their ideas more quickly and more accurately. Keep this in mind for later.
Whether it’s 1839s’ Joseph Jacquard who used programmable weaving looms to create a portrait of himself using 24,000 punched cards or WWII military agencies that invested in the development of the first ‘monitor’ to allow radar operators to plot aircraft movement, the development and evolution of technology is largely predicted by this theme of speed and accuracy.
From the fax machine making information vulnerable to loss and theft to the internet making malware easy for susceptible users to download, malicious actors have always found a way to exploit our naivety to new technology. What dangers should users, businesses, and governments expect from immersive technology?
You’re sitting in a virtual meeting room. Although the marble walls and mahogany table encompassing the space appear vectored and block-like, you feel oddly at ease. As you look around the room, everything feels intuitively wrapped around your eyes. You’re surprised to find how fluid your hands feel as you gesticulate to a nearby avatar. Hovering between the two of you is a larger than life three dimensional model of your proposed project.
Snap back to the reality of your boring home-office. You’re actually on Zoom. Your computer monitor is bright but the glare from the nearby window hurts your eyes. The video-chat interface is cluttered with tiny webcams talking over one another. You’re connected to the internet but you feel disconnected from your team and although you may not see it now, you are living on the verge of a paradigm shift.
The immersive paradigm shift is a moment in time where the line between what we perceive is ‘real’ and what is not will blur indefinitely. This is a world where cameras are programmed to defy reality, bodies swing and walk into nothing, and eyes become sentient portals to a collective imagination.
If you haven’t guessed it by now, of course I’m talking about the trifecta of incoming immersive technology, or rather the much anticipated mass market emergence of augmented reality (AR), virtual reality (VR), and mixed reality (MR). While all three somewhat differ from one another, they share one important aspect: that is, the representation of a new dimension to human computer interaction (HCI).
That’s not to say this is our first rodeo. Over the past 25 years we’ve seen technology bring forth dramatic changes to the economic and social fabric of our society. From the internet powering our knowledge economy to mobile computing transforming how we communicate, these significant evolutions are judged not just by their technical sophistication but by their intrinsic ability to transform our lives.
Thanks to advances in computer vision — particularly in object sensing, gesture identification, and tracking — sensor fusion and artificial intelligence has furthered our interaction with computers as well as the machines understanding of the real world.
Moreover, advances in 3D rendering, optics — such as projections, and holograms — and display technologies have made it possible to deliver highly realistic virtual reality experiences. As a result, immersive technologies can now allow us to interact with ourselves and machines in a completely different manner as we will no longer be confined to a 2D screen.
Asking the right questions
As scary as that may sound, governments and businesses need to be preparing for the various modalities that will be introduced by immersive tech across their products and processes. This moment in time is no different to the shift from fax to email or the introduction of the smartphone. Moving to VR and AR will simply be the next natural step in staying relevant and competitive.
So if immersive technologies are poised to profoundly change the way we work, live, learn, and play, what ramifications should we come to expect? As speech, spatial, and, biometric data are fed into artificial intelligence, new questions will emerge over the extent of our virtual privacy and security. As technology becomes more comfortable and intuitive, we are at risk of going under the illusion of control.
Throughout the history of computing, every significant shift in modality has brought with it new and potentially destabilising threats. If we fail to ask the right questions, the problems we will experience adjusting to this new technology may be greater than those posed by the internet and mobile computing combined. Let’s explore some examples:
Privacy & security concerns (AR)
It’s no secret that augmented reality technologies, which overlay virtual content on users’ perceptions of the physical world, are now a commercial reality. Recent years saw the success of AR powered camera filters such as Instagram stories, with more immersive AR technologies such as head-mounted displays and automotive AR windshields now being shipped or in development.
With over 3.4 billion AR capable devices expected to hit the market by 2022, augmented reality is predicted to make the earliest splash amongst consumers. We should expect wearables that will allow us to navigate in the real world through Google Maps and camera applications that will scan the relevant objects surrounding you in a grocery store. Therefore, anticipating and addressing the security, privacy, and safety issues behind AR is becoming increasingly paramount.
Buggy or malicious AR apps on an individual user’s device are at risk of:
Recording privacy-sensitive information from the user’s surroundings | Productivity tools
Leak sensitive sensor data (e.g., images of faces or sensitive documents) to untrusted apps | Instagram & Snapchat
Disrupt the user’s view of the world (e.g., by occluding oncoming vehicles or pedestrians in the road) | Google Maps
Multi-user AR systems can experience:
Vandalismsuch as with this incidence with augmented reality art in Snapchat
Privacy risks that bystanders may face due to non-consensual recording by the devices of nearby AR users
For the most part, AR security research focuses exclusively on individual apps and use cases; mainly because many problems we have already experienced with internet and mobile computing are expected to crossover to the new AR medium.
For instance, when the app store was first launched, many iPhone apps were nefariously designed to siphon and package individual mobile data in the background. Security analysts expect similar issues to arise with AR; only this time it won’t just be our location data they’re after but our more sensitive biometric data. More on that later.
Virtual reality is the use of computer technology to create a simulated virtual environment. As visual creatures, humanity has been dreaming of creating virtual environments since the inception of VR developmental research in the early 60s. At first, commercial uses were mainly in video games and advanced training simulations (NASA) but as the technology advanced, so did our potential for using it.
Since the 2012 launch of the Oculus Rift, digital tools for VR have slowly begun to emerge. From Facebook’s all-in approach with the virtually collaborative social-media-esque Horizon and Valve’s newly released and highly praised virtual zombie game Half-Life: Alyx, there are plenty of examples today to show off the prowess of current-state VR. Indeed, with so many development and hardware companies competing for market share, it may feel like virtual reality has finally arrived.
For instance, items bearing a brand, recognisably shaped cars, dangerous weapons, and iconic places, have appeared in video games for years. A great example is Rockstar’s Grand Theft Auto series which has had numerous IP battles after satirically recreating the cities and environments of Miami and Los Angeles.
Once we reach a form of ‘near reality’ within a game environment (one that is higher fidelity than the current 2D experience), we should expect intellectual property issues in virtual reality to sky rocket. For instance, a printed image of a painting from Google Images is much less of an IP issue than perhaps a virtual high-quality model of the same painting within a future virtual space.
Couple this with the fact that the depth-sensors in our phones are increasingly more capable of scanning real-life objects and modelling it real-time, means that in the future, anyone will be able create a virtual model of anything, and place it virtually anywhere.
Intellectual property predictions to expect:
IP protection of places, and buildings is a growing trend with EU lawmakers continuing to debate whether built structures, which are open to the public, should have rights attached to them. This is known as the so-called “freedom of panorama”.
IP protection of experiences such as touching or smelling a particular store, airline or hotel chain is possible with haptic virtual technologies. Although it is difficult to justify protecting an aesthetic today (only Apple has managed with its store layout), this may be more relevant in the future with VR.
Featuring a branded item or a well-known person is currently seen as a potential intellectual property infringement. How will this change if it is the player who is inserting self-scanned models rather than the game developer? Who is going to be liable?
This last point is most interesting because it relates to whether the platform or developer is liable even though it may be the user who is placing IP-protected models into the virtual environment. This concept is a similar crossover to the early days of peer to peer technologies with Napster and Limewire when users uploaded IP protected MP3 and video files to shared servers.
In the future, VR should expect similar IP problems that we get today. Faster computers and smarter artificial intelligence programs will allow users and developers to upload virtual objects at an unprecedented ease. Add on to this the idea that virtual reality will someday be as realistic as real-life and we’ll have an interesting problem on our hands.
Processing concerns with biometric data (VR & MR)
Unlike virtual reality which immerses the end user in a completely digital environment, or augmented reality which layers digital content on top of a physical environment, mixed reality (MR) occupies a sweet spot between the two by blending digital and real world settings into one.
When it comes to mixed reality, biometric and environmental data is an essential yet consequential by-product of sensory technology. This is mainly because developers need access to data to tweak specific functionalities and perfect the comfort and usability behind an immersive tool.
Thus, as immersive tools enter our homes, we are at risk of digitalising and exposing our most personal of information. The potential by-product of these applications siphoning biometric data is fundamentally tied to our security and privacy. Nobody at first knew how much user data the mobile phone was collecting through our apps. Why shouldn’t we expect the same with immersive devices?
The data collected will someday include:
Hand & face geometry
Electrical muscle activity
Eye movement detection
Hand pose recognition
Unconscious limb movement tracking
At its core, there is nothing more sensitive and unique than an individual’s biometric data. For instance, heart rate, skin response, and eye movement within a controlled virtual environment can be collected to potentially analyse an individual’s reaction to a virtual advertisement. Thus, a feeling that is meant to be reserved for your own inner-self can someday be downloaded and scrutinised by external corporate entities.
Additionally, it’s important to mention that unauthorised collection of biometric data is prohibited under article 4(14) of the GDPR. However, despite this, questions on the potential consequences of this data being mis-collected or misused remains highly relevant. Advertising will be the first to enter this space but expect greater consequences with the continued advent of the surveillance nation state.
Every major modality shift in technology has brought with it new threats and dangers. From the fax machine making information vulnerable to loss and theft to the internet making malware easy for susceptible users to download, malicious actors will always find a way to exploit our naivety and ignorance.
As users and consumers of digital technology, we need to be aware of the privacy risks involved when hooking ourselves up to sensor-laden devices. Virtual reality can be really useful and fun but remember to make sure you’re biometric data isn’t getting funneled to a third party.
In the business world, immersive technology will force many companies to rethink their internal and external processes — due diligence and hiring the right people will be important. So is taking the necessary steps towards protecting your IP or making sure your virtual products can’t be hacked or ‘vandalised’.
Lastly, governments and public institutions need to prove to the public that they can preempt the various threats immersive tech will bring to business, social well-being, and user privacy. So far, legislators and tech companies have been playing a game of cat and mouse. As we move forward, a firm hand and some much needed transparency will be key.
The future of technology will no doubt be impressive. Someday we will look up to the skies to access our information instead of down to our phones. Yet, the warning here is that comfort is never bliss. Where there is comfort, there is an opportunity for naivety and exploitation. As we gear up for the immersive paradigm shift, please remember to stay informed.
The greatest weapon in a sound designer’s arsenal is the mere fact that we listen first and react second. Join us as we briefly explore the neurological, anthropological, and digital histories behind how we interpret sound and why not everything you hear should sound like the truth.
Spooked by the sound
Remember that time when you were alone in that quiet house for the first time and heard a creepy sound? Maybe it was a windy day and the floor creaked and the window bellowed. That sound you heard, was clearly the logical result of wind pushing into a creaky wooden structure, yet the auditory impact is interpreted by the hypothalamus (a small but very important part of your brain that regulates fight or flight) as a threat.
Your thoughts quickly flow into scenarios: is it a ghost? Or perhaps a robber? For the first five seconds these possibilities are all you might consider. They dominate your imagination and thought processing. Until the rational side of your brain — granted some time passed without other similarly scary sounds — convinces you that the sound is nothing to be afraid of. But part of you still believes that, during those first five seconds, you actually saw, or at least heard, a spooky ghost making that sound.
Why Conditioning Matters
If you are unlucky enough to believe you have witnessed paranormal activity, you can consider yourself conditioned. In humans, conditioning is part of a behavioural repertoire of intelligent survival mechanisms supported by our neurobiological system.These underlying mechanisms promote adaptation to changing ecologies and efficient navigation of natural dangers. In this case, you have been conditioned to be aware of a sound attached to a particular danger.
Conditioning is a big reason why our brains don’t like to be surprised. Otherwise known as the Survival Optimization System (SOS), our response to most danger usually begins with a sound. This is because, as far as the human experience goes, you hear way faster than you see and at over 300,000 kilometers per second, sound gets into the ear so fast that it modifies all other input and sets the stage for it.
We hear first and listen second because in this Darwinian struggle we call life, it’s considerably faster and more effective for our brains to react to the possibility of a threat then to wait for its validity. Thus, the bi-product of a ghostly trauma, is a deep mechanistic rewiring of our neurobiological system to that specific occurence of sound. So for at least the near future, any sounds you hear alone in a quiet house will trigger your brain’s survival mechanisms and neurobiological nervous system to react fearfully to the potential presence of a spooky ghost.
From alarms to drums and push notifications
Yet, conditioning doesn’t only happen with things that scare us. As we said before, conditioning is a natural process the brain undergoes when faced with repetitive sensory information. It is a software-like response that codes a defence mechanism into our subconscious reactions.
In psychology, sound conditioning is defined as:
A process in which a stimulus that was previously neutral, comes to evoke a particular response, by being repeatedly paired with another stimulus that normally evokes the response.
A classic example of a sound conditioning experiment is the Pavlov Experiment which sought to establish if salivation in dogs could be caused with the pairing of a bell sound stimulus. Everytime Pavlov rang the bell, he would feed the dogs. After doing this repeatedly, the pairing of food and bell eventually established the dog’s conditioned response of salivation to the sound of the bell. After repeatedly doing this pairing, Pavlov removed the food and when ringing the bell the dog would salivate.
What the Pavlov experiment demonstrated is that most intelligent animals, including humans, given sensory repetition, are capable of experiencing a conditioned response to a conditioned stimulus. It’s a big reason why we people listen for cars before crossing the road, why particular songs make us remember the past, and in a more humorous sense, why children run after the ice cream truck.