Research on foreign body detection in transmission lines based on a multi-UAV cooperative system and YOLOv7

The unique plateau geographical features and variable weather of Yunnan, China make transmission lines in this region more susceptible to coverage and damage by various foreign bodies compared to flat areas. The mountainous terrain also presents great challenges for inspecting and removing such objects. In order to improve the efficiency and detection accuracy of foreign body inspection of transmission lines, we propose a multi-UAV collaborative system specifically designed for the geographical characteristics of Yunnan's transmission lines in this paper. Additionally, the image data of foreign bodies was augmented, and the YOLOv7 target detection model, which offers a more balanced trade-off between precision and speed, was adopted to improve the accuracy and speed of foreign body detection


Introduction
The province of Yunnan in China is characterized by its high-altitude plateau, abundant forests, and variable climate.Due to the harsh climate and environment, transmission lines and power towers in Yunnan are more prone to foreign body coverage.These foreign bodies include lightweight kites, balloons, agricultural plastic sheeting, plastic bags, and bird nests, and can cause power outages and affect the normal operation of the power system on a large scale.Therefore, regular inspection and cleaning of foreign bodies on transmission lines is crucial.
In the past, the detection method of foreign bodies in transmission lines in Yunnan, China, was mainly based on manual detection.As mentioned above, transmission lines in Yunnan are often located in complex geographical environments, where the natural environment is harsh, traffic is inconvenient, and 4G/5G signals are poor [1].The traditional way of relying on simple manual detection and paper media recording is inefficient, heavy work intensity, poor reliability, complex detection environment and harsh climate will also cause danger to patrol personnel.The UAV (unmanned aerial vehicles) is a kind of aircraft controlled by radio remote control equipment and an application program.As an auxiliary tool, the research on its application in different fields is increasing.It is widely used in remote sensing [2,3], topographic survey [4 -6], intelligent agriculture [7,8], power system inspection and monitoring [9 -11].With the maturity and popularization of UAV technology, using UAV for foreign object inspections on power transmission lines has become an important means.This method is easy to operate, high ef-ficiency, and the inspection field is adjustable and has been widely used.At the same time, combined with the rapid development of image detection technology in recent years, it is possible to use UAV to detect foreign bodies in transmission lines automatically, so as to no longer rely on manual judgment, which greatly improves the inspection effect of foreign bodies in transmission lines.However, compared with plain areas, the high mountains, and deep valleys of Yunnan have caused many difficulties for UAV inspections.For example, the inspection of transmission lines is complex, and there is a large difference in altitude.This makes the efficiency of using UAVs to inspect power transmission lines in Yunnan low.At the same time, the foreign body detection results of transmission lines need to be fed back at all times (rather than after the event).Therefore, the foreign body detection algorithm with fast detection speed and high detection accuracy is necessary.The current detection algorithms still have many limitations in terms of speed and accuracy.
To overcome the above limitations and in combination with the characteristics of Yunnan's power transmission lines, this paper proposes a multi-UAV cooperative foreign body detection method for transmission lines based on the latest target detection model YOLOv7.Our main contribution is in the following three parts.
1. Establish a cooperative system of multi-UAV that can plan a path based on the detection task and design the path according to the global planning of the detection line and flight path, to address the low efficiency of foreign object detection on transmission lines caused by the mountainous and deep valley terrain in Yunnan.
2. YOLOv7 target detection model with more balanced speed and accuracy is used.So that the transmission line foreign body detection is timelier and more accurate.The efficiency of identification is improved and leakage identification is reduced.
3. Aiming at the problem of fewer data sets of foreign body images in transmission lines, an image augmentation method for multi-UAV foreign body image was proposed to improve the detection ability of YOLOv7 for foreign bodies.

Related work 1.Research on the application of UAV in transmission line inspection
Most of the transmission lines are located in sparsely populated outdoor areas with inconvenient transportation.This special working environment causes difficulties in detection.Therefore, there is a very early detection method using helicopters instead of manual walking.However, it cannot be widely adopted because of the high cost of using helicopters.In addition, when flying helicopters cannot pass close to the line, some small objects cannot be easily detected.The UAV combines satellite positioning and controllable programming technology, showing the characteristics of stability, economy, portability, weather resistance, and durable cruise.In transmission, line inspection has been a lot of research and application.Li [12] studied the development process of UAVpowered patrol and implemented intelligent rapid mission planning through programming.Toth et al. [13] studied and summarized the application of UAVs in the transmission line inspection of British Columbia Transmission Corporation.Zormpas et al. [14] combined basic image processing methods and assembled specific UAVs for the job to fully specify the job according to the consideration of the purpose of transmission line inspection.He et al. [15] studied safety inspection rules and automatic detailed inspection methods for transmission towers of multi-rotor UAVs.A theoretical model for transmission line detection of multi-rotor UAVs is established.The UAV waypoints are connected in order to automatically generate the inspection path and realize automatic detailed inspection.The results show that the position error of UAV automatic detailed inspection is less than 10 cm.The height error is between 1.26 and 1.76 meters.

Research on foreign body detection algorithm in transmission lines
In order to realize automatic foreign body detection in transmission lines, it is necessary to use object detection algorithm to detect foreign body from image data collected by camera carried by UAV.Object detection algorithms include classical algorithms (such as SIFT and ORB feature matching algorithms.) and deep learning algorithms.Compared with deep learning, the detection accuracy of the classical algorithm for edge-blurred images and complex background images is lower.Therefore, deep learning has become a mainstream object detection algorithm in recent years.
Guo et al. [16] train the Faster R-CNN model to detect falling objects, kites, balloons, and other foreign objects in the transmission lines.And experimental results show that compared with the traditional target detection method, fast R-CNN can realize the detection of foreign bodies in transmission lines in complex scenes compared with the classical detection algorithm.Yao and Zhu [17] proposes an unsupervised foreign object detection algorithm based on GMM (Gaussian Mixture Model) and kmeans.But the accuracy of the algorithm depends on the number of samples.Li et al. [18] proposes a method to improve the YOLOv3 thus it can be easily deployed to embedded platforms without losing performance.Song et al. [19] research a high-voltage line foreign object intrusion detection model based on Yolov4.But there is no comparison with other Yolo models.Wang et al. [20] improve the original YOLOv5 object detection model and propose a novel fusion detection model using multi-scale appearance and relationship features.

A multi-UAV cooperative system
We propose a multi-UAV cooperative system to address the issue of low efficiency in foreign object detection for power transmission lines due to the unique terrain and landforms in Yunnan [21].This system carried out path planning through the platform and assigned reasonable tasks to the UAVs in the working group.Meanwhile, the platform has an open design and can carry any number of UAVs.The system is mainly composed of three parts: (1) A multi-UAV cooperative system; (2) Path planning module; and (3) Task allocation module.The system is responsible for planning the inspection paths and tasks of multi-UAV for transmission lines, and sending image data back to YOLOv7 for foreign body detection, and feeding back the results to staff.The system interface is shown in Fig. 1.

Fig. 1. A multi-UAV cooperative system interface. Where A, B and C are power towers
The system is designed to establish a multi-UAV cooperative inspection system that is built on an open interface platform.DJI UAVs are not only cheap, excellent performance.More importantly, DJI also provides a stable SDK-DJI Mobile SDK, which is convenient for secondary development.Users can use it to perform various functions, such as automatic flight, control the camera and universal joint, receive real-time video and monitor the status of other components, and so on.The multi-UAV system allows for the seamless integration of any number of UAVs.
The path planning module is designed for the global planning of transmission lines and flight paths of UAVs, which can obtain the position of each UAV visually through map markers.Path planning is realized through the waypoint task, that is, arranging a set of ordered geographic coordinate points of the power tower that the UAVs need to reach in turn.The selection of points can be done manually on the map module, after which the system completes the path planning.
The task allocation module distributes the path obtained from the path planning to the UAVs in the working group, enabling them to the UAVs can complete the route inspection and detection.The working relationship between UAVs can be divided into independent line inspection and single-line cooperative inspection.Independent line inspection means that the task allocation module assigns a path to each UAV in the working group to avoid the duplication of paths.The coordinated inspection of a single line refers to the joint inspection of a line by UAVs.Due to visual angle issues, a single UAV may be obstructed by steel frames or other objects, resulting in missed foreign object detection.

Method for detection
As the most typical representative of the one-stage target detection algorithm, the YOLO series algorithm is based on deep neural network for object detection and positioning, which runs fast and can be used in real-time systems.The accuracy and speed of detection are very balanced in real detection applications.The YOLOv7 [22], the latest version of the YOLO series, outperforms all known target detectors in both speed and accuracy in the range of 5 FPS to 160 FPS.In addition, the performance of YOLOv7 in speed and accuracy is also better than YOLOR, YOLOX, Scaled-YOLOv4, YOLOv5, DETR and other target detectors.Therefore, this paper uses YOLOv7 as the detection model.
This paper focuses on the detection of foreign bodies in transmission lines.The detection targets mainly include the detection of bird nests on power towers, foreign bodies on wires, and the detection of foreign bodies cover on insulators.In the following sections, we will introduce our detection methods.

Detection of bird nests on power tower
The power tower is relatively high, so ordinary foreign objects are unlikely to be blown onto the towers.However, some birds prefer to nest on the towers, which can affect the electrical equipment.Therefore, the main object of foreign bodies detection on the power tower are bird nests.
Difficulty: Bird nests on power towers are usually built on the supporting structure, which is mainly located near the inside of the tower.Therefore, bird nests are susceptible to the influence of the external tower structure, and the whole structure of the nests cannot be clearly photographed by the UAV.
Methods: Aiming at the problem of bird nests image occlusion and insufficient data.We adopted a multi-UAV cooperative inspection to increase the field of view of inspection, and added the method of bird nests image not on the tower to the YOLO training dataset to strengthen the detection ability of neural network for such objects.And the image augmentation operation, the main method for bird nests is to give the original data artificial occlusion.To enhance the ability of the network to resist occlusion.Data augmentation are explained in detail in Sect 4.

Detection of foreign bodies on wires
Foreign objects on the wire are generally due to their lightweight, which causes them to be blown onto the facility by wind, such balloons, kites, plastic bags, and so on.
Difficulty: Because foreign objects are relatively lightweight, they can appear in many different forms depending on the perspective of the UAV.Under the same Angle of view, they also change different shapes, especially kites and plastic bags.This makes it more difficult for neural networks to detect this type.
Methods: We used artificially generated foreign bodies in multiple forms to expand the dataset.Additionally, we included image data of these foreign bodies in nonpower line backgrounds to increase variety in the dataset.Data augmentation is also explained in detail in Sect 4.

Detection of foreign body cover on insulators
Insulators are very important components in transmission lines.They are easy to be covered and wrapped with long and light objects, such as plastic bags and long fabrics.
Difficulty: Foreign bodies on the insulator are a significant safety hazard for the insulator, but they are not very common.Therefore, there will be insufficient data on foreign bodies coverage of insulators.At the same time, the shape of the insulator is long and short, and its appearance varies from different angles of view.
Methods: In view of the shortage of foreign bodies data on insulators, we simulated the real situation by manually covering or winding objects on insulators to increase the amount of data, and used data augmentation techniques to further expand the dataset.

Data augmentation
As mentioned above, although foreign bodies on power transmission lines have an important impact on power equipment, it is difficult to collect data on these foreign bodies, results in an inadequate amount of data.At the same time, in each foreign body data, there is still a problem that some categories of data are large and some categories of data are small.Given the network and limited prior knowledge, introducing more new data for training means that the trained network is more robust.A common problem in object detection is that the training dataset is too small, which leads to poor network detection capability after training.Data augmentation is an effective method to increase the quantity and diversity of limited data.It extracts more useful information from limited data and generates the value of more data.Therefore, when the training sample size is limited, data expan-sion and augmentation are very necessary.The data augmentation method used in this paper mainly includes five operations (as shown in Fig. 2): 1. refusion of foreign object and background; 2. partial occlusion and erasure; 3. Affine transformation; 4. Brightness and contrast transformation.Each operation simulates the imaging environment of a UAV inspecting a power transmission line.

Fig. 2. Example of the data augmentation process
The image data of certain foreign bodies in transmission lines is limited, and it is not feasible to reproduce these scenarios on power lines in reality.Therefore, we use a re-fusion method to combine the foreign bodies with the background and augment the available data.The method of refusion refers to the segmentation of foreign bodies from images of foreign bodies in different background environments and then refusion with the environment of transmission lines.
Partial occlusion and erasure: Foreign bodies in transmission lines will be blocked by branches or power facilities due to the environment, making capturing all foreign bodies when using UAVs impossible.we artificially added partially occluded and erased image data.
Affine transformation: Affine transformation was used to change the size, direction, and position of foreign bodies in the original image, and the transmission line inspection scene shot by UAV was simulated from different angles.

Data construction
The datasets used in the experiment consist of six classes: the insulator, covered insulator, balloon, kite, nest, and plastic trash.The first two classes are the detection of uncovered and covered insulators, and the last four classes are the classes of foreign bodies on transmission lines.It should be noted that all six classes of original images were either obtained from the Internet or captured by UAVs.Besides, we randomly divide the whole images into a training set and validation set at an approximate ratio of 3/4 and 1/4.A test set with 200 images was also collected, which were captured in the real situation for the six classes.The detailed composition of the datasets is shown in Tab. 1.

Experimental configuration
All of our experiments are based on the PyTorch deep learning framework, deployed on the Ubuntu operating system.In the process of training models, we use a computer with an NVIDIA GeForce RTX 3090 GPU, whose memory is 24 GB.And in the process of testing models, the GPU is NVIDIA GeForce RTX 3080 Ti, whose memory is 12 GB.

Evaluation criteria
The four commonly used criteria for evaluating the performance of different object detection models are Precision (P), Recall (R), mean Average Precision (mAP), and F 1 Score (F 1 ).The formula of P, R, and F 1 are defined as follows: P = T P \(T P + F P ), R = T P \(T P + F P ), and where F P (false positive) represent the number of false samples which are incorrectly detected to be positive samples, F N (false negative) the same way to define, and T P (true positive) represents the number of correct detected ones.T P + F P is the total number of detected objects, and T P + F N is the total number of actual objects.F 1 is the harmonic mean of P and R.

Implementation details
We aim to detect foreign bodies on transmission line equipment quickly and accurately, to reduce the damage of foreign bodies to transmission lines and improve the detection effect.In view of the good real-time performance and accuracy of the YOLO model, we choose to use the YOLO model as the target detection model, and use four variants of the YOLO model for training to compare the detection accuracy and speed.These models are YOLOv5-s, YOLOv5-m, YOLOv7 and YOLOv7-X.
All models are trained based on the weights before training, and we use the same dataset built before to train and validate all models.YOLOv7 training parameters of the specific Settings include Epoch: 300, Batch size: 16, IoU threshold: 0.2, Initial learning rate: 0.01, and Momentum: 0.937.In the experiment, the detection performance of a series of YOLO models is compared on the same validation dataset to choose a detection model that can not only accurately identify foreign bodies but can also be applied to real-life situations that require quick detection.The precision trend of four models in the first 150 epochs is depicted in Fig. 3, and the final detection results are detailed in Tab. 2. As we can see, although the speed of YOLOv5s is faster, YOLOv7 has the best performance among P, R, mAP, and F 1 parameters.At the same time, compared to the processing power of the camera carried by the UAV, the speed of 3.6 milliseconds is sufficient for our detection system.Considering the balance between detection accuracy and speed, we have selected YOLOv7 as the model for foreign body detection in transmission lines in this study.The detection results of the above five models in the testing datasets are shown in Fig. 4.

Results in realistic transmission line
To illustrate the performance of our trained YOLOv7 model, we evaluated it with both test and real datasets, which are composed of images taken by UAVs and contain a large number of objects of all these classes.The testing performance is shown in Tab. 3. As we can see in Tab. 3, Fig. 5 and 6.Our method performs very well on every class in both the test set and the real dataset.We also use real data sets to measure the speed of our entire system.The results show that our final system can detect each image at a speed of 3.6 milliseconds, which meets the accuracy and speed of foreign body detection in transmission lines.

Conclusion
This paper proposes a multi-UAV cooperative system for foreign body detection in transmission lines based on the latest object detection model, YOLOv7.In order to meet the needs of the application, combined with a multi-UAV cooperative system, we adopted the YOLOv7 target detection model with a more balanced detection speed and accuracy.So that the detection of foreign bodies in the transmission line is timelier and more accurate.At the same time, aiming at the problem of insufficient training samples, a series of foreign body image augmentation methods for transmission lines for UAVs are proposed to improve the foreign body detection ability of YOLOv7.However, there are still some incorrect detection results because of unexpected shapes and wrong logical judgment of the bounding box.In the future, in order to further improve the detection accuracy and speed, we will improve the detection network structure and modify the algorithm logic of foreign body detection.

Fig. 4 .
Fig. 4. Compare and test results (the detection performance of 4 models)

Fig. 5 .
Fig. 5. Compare and test results (detection effect on the test dataset)

Fig. 6 .
Fig. 6.In terms of real data, the detection effect of YOLOv7 indifferent classes