[WATCH] Bring the crime scene into the court

“Take the scene of the crime home with you...” – this will be the new evolution in the Maltese justice system, as a budding new technology inside the University of Malta’s Faculty of Media and Knowledge Sciences gets ready to enter the mainstream.

Inside the MaKS Immersion Labs, criminologist Prof. Saviour Formosa is showing me a bird’s eye view of Floriana, that I can see through a headset for me to hover above by simply taking a few steps inside this laboratory.

In another contained space, known as the SIntegraM C.A.V.E, where overhead projectors beam 3D scanned models of places such as Victoria’s Citadel or Għar Dalam, researchers at the Immersion Lab can walk through these places without taking a single step, turning corners, peeking behind boulders, or suddenly floating above the space they are in.

Prof. Formosa, familiar to the general public as the compiler of annual crime statistics and the author of various studies on Maltese crime patterns, hopes the work being done at the University of Malta can bring ‘eXtended Reality’ (XR) – a term encompassing augmented reality, virtual reality and mixed reality – into a myriad of spatial and planning applications, from planning to tourism, restoration to healthcare, as well as the criminal justice system.

A case in point, he mentions, is using scanned areas of Malta’s geography to recreate or rebuild, or even being able to reimagine, what land and surfaces looked like before they were demolished or suffered a collapse. Examples abound – a recent example is the collapse of the Bighi peninsula boundary wall beneath what is now Heritage Malta’s headquarters, or even the Azure Window in Gozo.

“We can scan entire zones and feed it to our computers, where all the photographs of a particular area can be used to generate into a detailed 3-D mesh,” he explains, as he hands me a joystick and a virtual reality headset. I am suddenly plunged into the underground tunnels of Valletta, where I can roam from one room to the other and into the great vaulted chambers beneath the capital city, fully immersed into a different reality.

Working with the EU-funded SIntegraM equipment within the University’s Faculty of Media and Knowledge Sciences, the research team – which comprises researchers Tram Nguyen and Fabrizio Calì – has scanned underground structures, old buildings, cliff faces, and a good portion of Malta’s surface... in the latter case, allowing users to have an entire village reproduced before their eyes in their headset in the form of a bird’s eye view. Other responsibilities involve assisting other students and departments with varied XR research projects, as well as a facial reconstruction project with the 3D scanned skull of Maria Adeodata Pisani, a beatified nun.

But Prof. Formosa believes the technology has great potential to be applied within criminal investigations and the courts by capturing the crime scenes in 3D, after a crime and also allowing for comparisons of these scenes with archived 3D scans from prior to the criminal event. Particularly useful in sites severely damaged by explosions or similar destructive crimes.

“We can use this technology to take the actual scene of the crime inside a courtroom... literally, the jurors, defence, prosecution and the judge or magistrate can ‘walk’ around the scene of the crime, and see it from all possible angles,” Prof. Formosa says. He adds that before the admission of guilt by the Degiorgio brothers in the assassination of journalist Daphne Caruana Galizia, the scanned car bomb scene was to be part of the evidence being shown inside the very courtroom trying the Degiorgios.

Prof. Formosa says the technology is especially necessary to be able to create comparative imagery of the scene of the crime, before it gets disturbed by the various personnel that inspects it. “We are even able to analyse the projectional direction of bullets from blood splatter traces.” The spatial dynamics of a scene may also be better understood in a 3D virtual experience rather than from photographs or other traditional forensic documentation methods.

Scanning is done through a combination of photogrammetry – taking multiple photos and then stitching them together – and LIDAR, which uses lasers to plot thousands of points which are then used to capture a scene in 3D. That data is then processed on high-end computers and converted into 3D scenes before it is published to AR technologies such as Magic Leap or Virtual Reality headsets such as the Oculus Rift or Meta Quest.

XR technologies are also set to play a fundamental role in the metaverse, where the next evolution of the Internet will converge real, digital, and virtual worlds into new realities, accessed with VR headsets or a pair of AR smart glasses.

“Essentially, what requires a screen today could be redundant in a couple of decades,” says Ph.D researcher Fabrizio Calì. “Today there are contact lenses being developed that can act as screens... you are receiving a call, answering an email... you won’t need access to a computer because everything will be popping up before your eyes. The virtual and real will become interchangeable based on whatever you need to be doing.”

Prof. Formosa hopes to take the technology further within the next months – he thinks medical technology can be a net beneficiary of this type of scanning that allows surgeons unprecedented access to the human body, or even allow the surgeon in one part of the world to use XR, with a robot in another part of the world to carry out the surgery. Another use could be tourism and heritage, where people can see the Azure Window in its past glory, to relive once more the majesty of the structure before its fateful collapse. And then there are disabled users who are unable to access certain areas or structures, but who can, through XR, be able to access these areas virtually. “Its usage could be manifold,” Prof. Formosa says.