lanaccess@lanaccess.es
+34 93 298 86 50

Where is the video recording?

Lanaccess > Articles > Where is the video recording?
  • Posted by: Lanaccess
  • Category: Articles

Written by Xavier Oliva Galván, Business Development Manager at LANACCESS and Associate Professor at the Pompeu Fabra University

The market for video surveillance systems is very dynamic. New solutions are constantly appearing to facilitate the operation of video surveillance systems, which are increasingly powerful, increasingly attractive, and promise to make the lives of security managers easier, by improving the performance of their work. Expectations are growing, based on the latest advances in artificial intelligence, both at the level of hardware and of algorithms based on new neural networks, which promise to identify people even when they are wearing sunglasses or a mask to protect themselves from the latest virus, detect faces covered by a ski mask, detect weapons, recognise suspicious behaviour or carry out powerful and instantaneous searches on video recordings.

In this article, I could go on to explain how a wonderful recording search tool based on artificial intelligence will quickly locate the video recording I need to deal with the latest security incident in record time. But no, I’ll save that for another time, and in this article, I’ll focus on something more prosaic, the availability of the recordings, or, expressed another way, the robustness of the solutions that can provide me with better availability. Because my system can be spectacular, but what if the wonderful search tool does not find the recording because it is simply not available? And not because it is older than the allowed recording period, and has been automatically deleted to comply with the data protection law, but because some irrecoverable loss has occurred in the recording system. Or because the wonderful tool has stopped working.

Bad luck? Maybe, but could it have been avoided? Someone said, and I agree, that luck must be sought.

In highly dynamic and competitive markets, solutions that are immature or based on unreliable technologies are often offered. When I select a new video surveillance system, my selection criteria must of course include the functionalities that meet my needs as a system user, but also other features that are perhaps more difficult to evaluate, but no less important: I should never give up on having a robust solution, one that I can trust, that gives me guarantees that I will have the recordings when I need them and that I will not have maintenance problems.

And how do I assess whether a solution is robust? Different factors influence the robustness of a solution, of which I would highlight cybersecurity, reliability, and correct maintenance, which affects availability.

Cybersecurity incidents in video surveillance systems

Cybersecurity has often not been considered when assessing the robustness of a solution, but today it has become truly relevant. Over the past few years, we have witnessed several waves of malware and it is known that several of these have affected security systems and companies in the security sector. We all remember how several ransomware attacks, such as WannaCry in 2017 or Ryuk more recently in 2019, demanding a ransom payment to recover the information encrypted by the virus, have been echoed in the media.

There have also been multiple instances of video recorder and security camera infections creating slave botnets, which have been used in distributed denial of service (DDoS) attacks targeting other systems, for spamming or for bitcoin mining. For example, in 2016 there was a DDoS attack against the company DYN, which offers dynamic DNS services, from a botnet made up of virus-infected cameras and video recorders, or in 2014 when infected Hikvision video recorders were used to generate bitcoins. In early 2020 it became known that Zyxel’s network storage devices (NAS) and LILIN video recorders with one vulnerability were exploited by several botnets.

grabación de vídeo

The network is a security element

On the other hand, today’s security systems are mostly IP-based, so it is necessary to realise that the IP network has become a key security element. Besides, some of the components of video surveillance systems, such as IP cameras, are often located outside the premises or in public access areas, which means that their point of connection to the network can be a dangerous gateway to the security system and the network in general.

All of this entails risks that we must analyse, evaluate, and mitigate, as well as monitor and have a response ready to be implemented as soon as they become a reality. Some of the risks are obvious, such as people being attacked or premises, goods and chattels that we are protecting being stolen or damaged while the system may be inoperative. We are going to analyse in a little more detail two other associated risks, the loss of recordings, which is very much linked to the robustness of the solution, and reputation.

grabación de vídeo

Risks of loss of recordings

Malware can affect the availability of recordings in different ways. It can be entirely intentional, i.e. someone is directing an attack on the video surveillance system to render it inoperative while committing a criminal act. A criminal may attempt to sabotage the power supply to the facility to disable the security systems, which as a countermeasure may dispose of a UPS (Uninterruptible Power Supply) based on batteries. In the same way, cameras, video recorders, video management tools, or the entire system may also be rendered inoperable by a computer attack, and we should try to dispose of countermeasures to avoid it.

Another way of affecting the system is indirect, i.e. a generic attack affects the video recordings or the video surveillance system without it actually being the target. For example, a generic ransomware attack affecting video recorders can block access to the recordings. Although we will hardly pay a ransom to recover them, and this is the target of the attacker, we will not be able to access existing recordings and, furthermore, video recorders will cease to be operational until the infection is removed. On the other hand, if our infected video recorder is being exploited by a botnet to carry out a DDoS attack or for bitcoin mining, its performance will be affected and it will probably not record video correctly.

Reputational risks

Reputational risks may arise as a result of the collapse of the security system itself, depending on how notorious the incident is. The publication of a newspaper article explaining that “The Acme Museum has closed its doors over the past weekend after a computer attack that produced a crash of the video surveillance system” can be very damaging.

Reputational risks can also arise if our systems are infected to create a botnet, being used in a denial of service attack or to introduce a worm that infects another company’s computer systems, which can blame us for the damage caused by negligence in failing to properly protect our own systems. This risk is significantly reduced if the video surveillance IP network is effectively isolated from other networks, but this is not always possible.

How can I protect myself against computer attacks?

To protect myself, I can take various measures to reduce the attack surface as much as possible. It is especially important to protect the key elements:

  • Protect the access to the network. This protects the entire video surveillance system. Several measures can be taken, one of the most obvious being to segment networks: isolate the video surveillance network from the general-purpose network and the Internet. This measure has an undesirable side effect, which is that the operating system of the operator stations and servers will not be automatically updated. Therefore, we should schedule manual security patches updates at regular intervals. Another important measure is to take care to protect the network connections that are in public access areas. This can be achieved by using recording systems that include a switch that provides security measures, such as the “PoE+ CCTV switch” in Lanaccess’ video recorders, or by using procedures available in standard manageable switches, such as MAC address filtering and 802.1X authentication, which are, however, quite complex to implement. Many manufacturers of video surveillance systems do not yet consider the network a security element and do not dispose of integrated switches. Moreover, most of those that do offer them are just generic switches installed inside the recorder chassis, without providing any special feature in terms of security.
  • Protect especially the key elements of the system. The first step is to identify what the key elements of the system are. Operator stations are usually replaceable with some ease, can be reinstalled or replaced with little impact on the system, so they are not key elements. They are computers running general-purpose operating systems, often Windows. It is advisable to install an antivirus and the corresponding security patches. Servers can be more critical, since it is more time-consuming to replace them and their failure in some cases can leave all the operators or part of the system inoperative, but they are usually still not key elements since they are also recoverable. The measures to be taken are the same as for the operator stations. On the other side, cameras and especially recording systems are key elements. If they fail, the evidence of a possible incident is not stored, or we can even lose previously-stored evidence, so we must pay special attention to select them. Not all cameras and video recorders offer the same levels of security. In the case of cameras, it is advisable to avoid brands that are suspected of being less mature or untested. In the case of video recorders, there are some with advanced security measures, capable of protecting the cameras from the network and even isolating them from each other. Besides, it is possible to avoid using Windows-type operating systems or generic distributions of Linux or Android in video recorders by opting for some robust alternative that gives us guarantees at the level of cybersecurity, working with another type of secured embedded operating system. This way, we will avoid being affected, indirectly, by the majority of attacks, which are massively directed against the most popular operating systems.
  • Training and awareness of all people who interact with the video surveillance system. Very often, computer security incidents are initiated with the carelessness of a person, who introduces malware or provides access credentials. It is fundamental and the responsibility of the company to adequately train and sensitise the people who will interact with the security system to reduce this type of risk. Sometimes the attempts to fool someone to get the necessary information are very evident and easy to discover, but let’s not forget that there are criminals who are true professionals when it comes to gaining the trust of anyone, which is why we talk about “social engineering”.
grabación de vídeo

Reliability

Some of the terms that are used to talk about reliability can be initially a little complicated to understand and sometimes confusing, but once understood they are very useful. Among others, we talk about lifetime (this is the easiest term to understand, but the most difficult to find in the technical specifications of a product) and also about MTBF and AFR (this term will be explained later). MTBF, which stands for Mean Time Between Failures, can be sometimes misleading and it is often confused with lifetime, because it is also expressed in time units (usually in hours).

I sometimes ask my students at the university to search on the Internet for the MTBF parameter of, for example, a hard disk, and they find values in the order of 1,000,000 to 1,500,000 hours. If we transform them into years, we are talking about values of… between 114 and 171 years!

When I ask them how long they expect their computer’s hard drive to last if it were running all day, they answer a few years, in the order of 3 to 5, at most. When I ask them if they believe that the MTBF values provided by the hard disk manufacturers are true, or if they are trying to fool us, some do not know what to say, others reply convinced that they are, that it is a shame that the manufacturers try to fool us this way. Are they really trying to fool us?

No, the problem is that we are confusing MTBF with lifetime, and these are two different concepts. I admit it, I have misled the students myself by the way I have asked the question, but it is a mistake that is easy to fall into, and doing so can be the best way to learn.

What a hard drive manufacturer with an MTBF of 1,500,000 hours and a lifetime of 5 years is telling us is that, as long as we change the drives every 5 years, we can expect a 0.6% annual failure probability.

What happens if we want to extend the life of a hard drive and we do not replace it when the lifetime is already over? We can do it, but then we cannot expect to observe such a low failure rate. MTBF, by definition, is only valid during the lifetime of the equipment.

Probability of failure

In terms of reliability, electronic devices are said to have a certain level of “infant mortality”, and after a certain time of use, breakdowns occur due to wear and tear. Both MTBF and AFR are valid values during the lifetime of the device, i.e. after the “infant mortality” (which should occur during factory testing of the device) and before wear-out failures (when the device should be replaced).

video recording

MTBF is a measure of reliability, as is AFR (Annualized Failure Rate), which is a concept that can be more easily understood. AFR has an inverse relationship with MTBF, it can be calculated as 8766/MTBF. We all understand it better when we are told that we are going to have a 3% annual failure rate for an installed plant of 100 units, than when we are told that the units have an MTBF of 292,200 hours. Therefore, if the parameter available in the datasheet is the MTBF, we should better transform it to AFR to get a clear idea of the number of expected failures. Obviously, we will prefer equipment that has an MTBF of 300,000 hours, i.e. with approximately 3% of annual failures, to equipment that has an MTBF of 30,000 hours, i.e. approximately 30% of annual failures! Pay attention to maintenance costs!

In short, when I, as a security manager, consider how long I can keep the current video surveillance system before renewing it, I am wondering about the lifetime. However, when I want to know how many failures I will have to deal with per year, the answer lies behind the MTBF or AFR of the equipment

How can I select equipment to last and have a low failure rate?

The reliability of the equipment depends on several factors. In the end, if the equipment’s MTBF parameter is available, the higher it is, the better, or in the case of AFR, the lower it is, the more reliable. Sometimes these values are not available. In these cases some of these tips may be useful:

  • Embedded systems are often more reliable than PC-based equipment. Usually, an embedded system has been designed for the function it performs, heats up less, and has fewer elements (both hardware and software) that are useless but also possible causes of failure.
  • Equipment with simple internal architecture tends to be more reliable. A device with multiple interconnected modules has more possible points of failure, the wiring and connectors are also sources of failures. The more elements, the more possible causes of failure.
  • Low consumption helps both reliability and lifetime. Electronic equipment that heats little breaks down less and lasts longer.
  • The fewer mechanical elements in the equipment, the better. Mechanical elements are a common source of failure and, due to wear and tear, their lifetime is shorter. An equipment that heats up little can even do without fans, which are one of the most common mechanical elements. Equipment that needs them will stop working soon after the fans break down. Traditional hard disks also have internal mechanical elements, the read/write heads. If their use can be avoided by using flash memory, especially for the operating system, or solid-state hard disks, if the application allows it because it does not require large storage capacities, we will improve reliability.
  • Another key element in the reliability of the equipment is the power supply, which is often one of the first causes of failure, so it has to be chosen with care.
  • A typical measure to increase availability is redundancy, for example, of the power supply. Indeed, redundancy of a not so reliable element increases availability, but we have to be aware that we pay a price in a higher number of failures. If there are twice as many power supply modules, the number of failures can consequently double, so depending on the case, a good power supply module may be preferable to two less reliable modules.

What is the probability that the recordings will be available?

To know the probability of having the video recordings, i.e. their availability, it is necessary, but not sufficient, to know the reliability parameters, since availability depends not only on reliability but also on the interconnection of the elements and the time it takes to repair them when they break down, i.e. on maintenance service.

Let us analyse the availability of the video recorder independently from the rest of the elements. If I have very reliable equipment, but when it breaks down it takes weeks or months to repair, the availability may be low. If we know the Mean Time to Repair (MTTR), the availability can be calculated as

Availability= MTBF/(MBTF+MTTR)

Availability may also be reduced by the duration and frequency of incidents due to computer attacks, but there is no parameter that allows us to introduce this effect in the availability calculation.

The average time for repairing breakdowns depends on:

  • The fault detection time. It is essential to know as soon as possible that a fault has occurred. Powerful plant monitoring tools are necessary.
  • The reaction time of the maintenance staff.
  • The availability of spare parts in the vicinity of the faulty equipment.
  • The proper training of the maintenance staff and the effectiveness of the manufacturer’s technical support service if you have to go to it.
  • The manufacturer’s repair service if the equipment has to be sent in for repair.

Therefore, if I want to be able to find the recordings when I need them, not only do I have to choose the equipment well, but also check that the manufacturer provides adequate monitoring tools and evaluate whether they have a good and close technical service. Besides, I will have to buy parts or even a whole set of replacement equipment if the size of the equipment plant justifies it, as well as hire a good maintenance service that will quickly resolve any problem that may arise.

grabación de vídeo

I want a robust solution

When I want to retrieve a video recording to deal with a security incident, I may be unpleasantly surprised that it is not available. To avoid this type of situation, I need the availability of my video recorders to be high.

When choosing a video surveillance solution, I must ensure that it is robust, that its key elements, such as the video recorders, are reliable, that the system is well protected against computer attacks, and also well maintained.

Protection against computer attacks depends on the characteristics of the video recorder itself, the network elements, and the interconnection with other networks.

The attack surface, i.e. the accessibility to equipment and services by potential attackers, should be kept to a minimum. Some video recorders dispose of internal switches with special security measures, which provide increased protection of the cameras and the rest of the elements of the system.

General-purpose operating systems, such as Windows or Linux, are the target of most computer attacks. The video surveillance system can be indirectly affected by general attacks or also directly by intentional attacks, which seek to bring down the security system to commit a criminal act without being discovered. In addition to technical measures, adequate training and awareness of system operators at the cybersecurity level is essential.

We have seen that reliability parameters such as MTBF and AFR are very useful to know if an electronic device is reliable. We have also seen that we should not trust questions from teachers, who may be trying to fool us more than the manufacturers of hard disks, although we do have to be aware that hard disks have a relatively short lifetime.

If we don’t know the reliability parameters, embedded equipment, with reduced power consumption, simple architecture, and few mechanical elements, tends to be more reliable.

Finally, we have also highlighted the importance of plant monitoring tools, spare parts, and maintenance services to achieve high availability of video recordings.

It is clear that if I want a robust solution, which I certainly want for my peace of mind, I cannot leave it to good or bad luck, but I must choose my video surveillance solution with the most professional criteria possible.

Xavier Oliva Galván
Business Development Manager at LANACCESS Associate Professor at the Pompeu Fabra University