Written by Xavier Oliva Galván, Business Development Manager at LANACCESS and Associate Professor at the Pompeu Fabra University
The market for video surveillance systems is very dynamic. New solutions are constantly appearing to facilitate the operation of video surveillance systems, which are increasingly powerful, increasingly attractive, and promise to make the lives of security managers easier, by improving the performance of their work. Expectations are growing, based on the latest advances in artificial intelligence, both at the level of hardware and of algorithms based on new neural networks, which promise to identify people even when they are wearing sunglasses or a mask to protect themselves from the latest virus, detect faces covered by a ski mask, detect weapons, recognise suspicious behaviour or carry out powerful and instantaneous searches on video recordings.
In this article, I could go on to explain how a wonderful recording search tool based on artificial intelligence will quickly locate the video recording I need to deal with the latest security incident in record time. But no, I’ll save that for another time, and in this article, I’ll focus on something more prosaic, the availability of the recordings, or, expressed another way, the robustness of the solutions that can provide me with better availability. Because my system can be spectacular, but what if the wonderful search tool does not find the recording because it is simply not available? And not because it is older than the allowed recording period, and has been automatically deleted to comply with the data protection law, but because some irrecoverable loss has occurred in the recording system. Or because the wonderful tool has stopped working.
Bad luck? Maybe, but could it have been avoided? Someone said, and I agree, that luck must be sought.
In highly dynamic and competitive markets, solutions that are immature or based on unreliable technologies are often offered. When I select a new video surveillance system, my selection criteria must of course include the functionalities that meet my needs as a system user, but also other features that are perhaps more difficult to evaluate, but no less important: I should never give up on having a robust solution, one that I can trust, that gives me guarantees that I will have the recordings when I need them and that I will not have maintenance problems.
And how do I assess whether a solution is robust? Different factors influence the robustness of a solution, of which I would highlight cybersecurity, reliability, and correct maintenance, which affects availability.
Cybersecurity has often not been considered when assessing the robustness of a solution, but today it has become truly relevant. Over the past few years, we have witnessed several waves of malware and it is known that several of these have affected security systems and companies in the security sector. We all remember how several ransomware attacks, such as WannaCry in 2017 or Ryuk more recently in 2019, demanding a ransom payment to recover the information encrypted by the virus, have been echoed in the media.
There have also been multiple instances of video recorder and security camera infections creating slave botnets, which have been used in distributed denial of service (DDoS) attacks targeting other systems, for spamming or for bitcoin mining. For example, in 2016 there was a DDoS attack against the company DYN, which offers dynamic DNS services, from a botnet made up of virus-infected cameras and video recorders, or in 2014 when infected Hikvision video recorders were used to generate bitcoins. In early 2020 it became known that Zyxel’s network storage devices (NAS) and LILIN video recorders with one vulnerability were exploited by several botnets.
On the other hand, today’s security systems are mostly IP-based, so it is necessary to realise that the IP network has become a key security element. Besides, some of the components of video surveillance systems, such as IP cameras, are often located outside the premises or in public access areas, which means that their point of connection to the network can be a dangerous gateway to the security system and the network in general.
All of this entails risks that we must analyse, evaluate, and mitigate, as well as monitor and have a response ready to be implemented as soon as they become a reality. Some of the risks are obvious, such as people being attacked or premises, goods and chattels that we are protecting being stolen or damaged while the system may be inoperative. We are going to analyse in a little more detail two other associated risks, the loss of recordings, which is very much linked to the robustness of the solution, and reputation.
Malware can affect the availability of recordings in different ways. It can be entirely intentional, i.e. someone is directing an attack on the video surveillance system to render it inoperative while committing a criminal act. A criminal may attempt to sabotage the power supply to the facility to disable the security systems, which as a countermeasure may dispose of a UPS (Uninterruptible Power Supply) based on batteries. In the same way, cameras, video recorders, video management tools, or the entire system may also be rendered inoperable by a computer attack, and we should try to dispose of countermeasures to avoid it.
Another way of affecting the system is indirect, i.e. a generic attack affects the video recordings or the video surveillance system without it actually being the target. For example, a generic ransomware attack affecting video recorders can block access to the recordings. Although we will hardly pay a ransom to recover them, and this is the target of the attacker, we will not be able to access existing recordings and, furthermore, video recorders will cease to be operational until the infection is removed. On the other hand, if our infected video recorder is being exploited by a botnet to carry out a DDoS attack or for bitcoin mining, its performance will be affected and it will probably not record video correctly.
Reputational risks may arise as a result of the collapse of the security system itself, depending on how notorious the incident is. The publication of a newspaper article explaining that “The Acme Museum has closed its doors over the past weekend after a computer attack that produced a crash of the video surveillance system” can be very damaging.
Reputational risks can also arise if our systems are infected to create a botnet, being used in a denial of service attack or to introduce a worm that infects another company’s computer systems, which can blame us for the damage caused by negligence in failing to properly protect our own systems. This risk is significantly reduced if the video surveillance IP network is effectively isolated from other networks, but this is not always possible.
To protect myself, I can take various measures to reduce the attack surface as much as possible. It is especially important to protect the key elements:
Some of the terms that are used to talk about reliability can be initially a little complicated to understand and sometimes confusing, but once understood they are very useful. Among others, we talk about lifetime (this is the easiest term to understand, but the most difficult to find in the technical specifications of a product) and also about MTBF and AFR (this term will be explained later). MTBF, which stands for Mean Time Between Failures, can be sometimes misleading and it is often confused with lifetime, because it is also expressed in time units (usually in hours).
I sometimes ask my students at the university to search on the Internet for the MTBF parameter of, for example, a hard disk, and they find values in the order of 1,000,000 to 1,500,000 hours. If we transform them into years, we are talking about values of… between 114 and 171 years!
When I ask them how long they expect their computer’s hard drive to last if it were running all day, they answer a few years, in the order of 3 to 5, at most. When I ask them if they believe that the MTBF values provided by the hard disk manufacturers are true, or if they are trying to fool us, some do not know what to say, others reply convinced that they are, that it is a shame that the manufacturers try to fool us this way. Are they really trying to fool us?
No, the problem is that we are confusing MTBF with lifetime, and these are two different concepts. I admit it, I have misled the students myself by the way I have asked the question, but it is a mistake that is easy to fall into, and doing so can be the best way to learn.
What a hard drive manufacturer with an MTBF of 1,500,000 hours and a lifetime of 5 years is telling us is that, as long as we change the drives every 5 years, we can expect a 0.6% annual failure probability.
What happens if we want to extend the life of a hard drive and we do not replace it when the lifetime is already over? We can do it, but then we cannot expect to observe such a low failure rate. MTBF, by definition, is only valid during the lifetime of the equipment.
In terms of reliability, electronic devices are said to have a certain level of “infant mortality”, and after a certain time of use, breakdowns occur due to wear and tear. Both MTBF and AFR are valid values during the lifetime of the device, i.e. after the “infant mortality” (which should occur during factory testing of the device) and before wear-out failures (when the device should be replaced).
MTBF is a measure of reliability, as is AFR (Annualized Failure Rate), which is a concept that can be more easily understood. AFR has an inverse relationship with MTBF, it can be calculated as 8766/MTBF. We all understand it better when we are told that we are going to have a 3% annual failure rate for an installed plant of 100 units, than when we are told that the units have an MTBF of 292,200 hours. Therefore, if the parameter available in the datasheet is the MTBF, we should better transform it to AFR to get a clear idea of the number of expected failures. Obviously, we will prefer equipment that has an MTBF of 300,000 hours, i.e. with approximately 3% of annual failures, to equipment that has an MTBF of 30,000 hours, i.e. approximately 30% of annual failures! Pay attention to maintenance costs!
In short, when I, as a security manager, consider how long I can keep the current video surveillance system before renewing it, I am wondering about the lifetime. However, when I want to know how many failures I will have to deal with per year, the answer lies behind the MTBF or AFR of the equipment
The reliability of the equipment depends on several factors. In the end, if the equipment’s MTBF parameter is available, the higher it is, the better, or in the case of AFR, the lower it is, the more reliable. Sometimes these values are not available. In these cases some of these tips may be useful:
To know the probability of having the video recordings, i.e. their availability, it is necessary, but not sufficient, to know the reliability parameters, since availability depends not only on reliability but also on the interconnection of the elements and the time it takes to repair them when they break down, i.e. on maintenance service.
Let us analyse the availability of the video recorder independently from the rest of the elements. If I have very reliable equipment, but when it breaks down it takes weeks or months to repair, the availability may be low. If we know the Mean Time to Repair (MTTR), the availability can be calculated as
Availability may also be reduced by the duration and frequency of incidents due to computer attacks, but there is no parameter that allows us to introduce this effect in the availability calculation.
The average time for repairing breakdowns depends on:
Therefore, if I want to be able to find the recordings when I need them, not only do I have to choose the equipment well, but also check that the manufacturer provides adequate monitoring tools and evaluate whether they have a good and close technical service. Besides, I will have to buy parts or even a whole set of replacement equipment if the size of the equipment plant justifies it, as well as hire a good maintenance service that will quickly resolve any problem that may arise.
When I want to retrieve a video recording to deal with a security incident, I may be unpleasantly surprised that it is not available. To avoid this type of situation, I need the availability of my video recorders to be high.
When choosing a video surveillance solution, I must ensure that it is robust, that its key elements, such as the video recorders, are reliable, that the system is well protected against computer attacks, and also well maintained.
Protection against computer attacks depends on the characteristics of the video recorder itself, the network elements, and the interconnection with other networks.
The attack surface, i.e. the accessibility to equipment and services by potential attackers, should be kept to a minimum. Some video recorders dispose of internal switches with special security measures, which provide increased protection of the cameras and the rest of the elements of the system.
General-purpose operating systems, such as Windows or Linux, are the target of most computer attacks. The video surveillance system can be indirectly affected by general attacks or also directly by intentional attacks, which seek to bring down the security system to commit a criminal act without being discovered. In addition to technical measures, adequate training and awareness of system operators at the cybersecurity level is essential.
We have seen that reliability parameters such as MTBF and AFR are very useful to know if an electronic device is reliable. We have also seen that we should not trust questions from teachers, who may be trying to fool us more than the manufacturers of hard disks, although we do have to be aware that hard disks have a relatively short lifetime.
If we don’t know the reliability parameters, embedded equipment, with reduced power consumption, simple architecture, and few mechanical elements, tends to be more reliable.
Finally, we have also highlighted the importance of plant monitoring tools, spare parts, and maintenance services to achieve high availability of video recordings.
It is clear that if I want a robust solution, which I certainly want for my peace of mind, I cannot leave it to good or bad luck, but I must choose my video surveillance solution with the most professional criteria possible.