1.

Introduction

A new era of surveillance has been ushered in due to advances in camera, data storage, and communications technology coupled with concerns for public safety, police-public relations, and cost effective law enforcement. According to IHS,¹ there were 245 million professionally installed video surveillance cameras active and operational globally in 2014. While the majority of these cameras were analog, over 20% were estimated to have been network cameras and around 2% were HD CCTV cameras. The number of cameras installed in the field is expected to increase by over 10% per year through 2019. Traditional police static video surveillance systems typically consist of networks of linked cameras mounted at fixed locations in public spaces such as transportation terminals, walkways, parks, and government buildings. There is also a great increase of law enforcement cameras on mobile platforms, such as dash cams and body cams. A 2013 Bureau of Justice Statistics release indicated that 68% of the 12,000 local police departments used in-car cameras.² A survey of large city and county law enforcement agencies on body-worn camera technology indicated that 95% planned to deploy body cameras.³ These public, as well as private, surveillance systems have been a great aid in identifying and capturing suspects, as well as revealing behaviors between the public and law enforcement.

This vast amount of video data being collected poses a challenge to the agencies that store the data. Records created and kept in the course of government business must be disclosed under right-to-know laws unless there is an exception that prevents disclosure. The Freedom of Information Act (FOIA), 5 U.S.C. 552, is a federal law that establishes a presumption that federal governmental information is available for public review. Under FOIA, federal agencies are required to issue a determination on a disclosure within 20 working days of receipt of the request or appeal, which can be extended by 10 days in circumstances such as the request is for a significant volume of records or requires collection of records from different offices or consultation with another agency. State law enforcement agencies operate under similar disclosure guidelines.

Video recordings are considered public records for the purpose of right-to-know laws, and privacy is one exception to reject a request for disclosure. Rather than an outright rejection of a complete video record of an event, it is becoming common practice to redact portions of the video record.

Redaction protects the privacy and identity of victims, innocent bystanders, minors, and undercover police officers. Redaction involves obscuring identifying information within individual video frames. Identifying information often includes but is not limited to faces, license plates, identifying clothing, tattoos, house numbers, and computer screens. Obscuring the information typically involves blanking out or blurring the information within a region. The region could be a tight crop around a person’s face, for example. Alternatively, redaction could involve blanking out the entire frame except portions that are not considered private. An example of redaction is shown in Fig. 1, where blurring is used to obfuscate the body. Figure 2 shows the high level process of releasing a redacted video.

Fig. 1

Example of redaction with blurring.

Fig. 2

Example of a typical video redaction system.

The present paper reviews the video redaction problems and challenges, rigorously explores detection and tracking methods to enable redaction, and introduces a detection and tracking metric specifically relevant to redaction. The remainder of the introduction reviews current redaction practices. Section 2 reviews the two main components of a redaction system: object detection and object tracking. Section 3 presents a metric for evaluating the redaction system. Section 4 examines various types of obfuscations, and Sec. 5 discusses open problems in the redaction space.

2.

Components of a Redaction System

A typical redaction system relies on two key components: object detection and tracking. Object detection is required for automatic localization of relevant objects in a scene or video. Such automated localization prevents requiring manual tagging of the objects of interest. A tracking module then uses the tagged object information to estimate object positions in the subsequent frames. The performance of the detection and tracking modules control the efficiency of the redaction system—higher accuracy requires less manual review and/or validation of results. We review common detection and tracking techniques along with relevant datasets.

2.1.

Object Detection

In the field of computer vision, object detection encompasses detecting the presence of and localizing objects of interest within an image. Object detection can assist the redaction problem by finding all of a certain category of object, for example faces, in a given image. The output of an object detection algorithm is typically a rectangular bounding box that encloses each object or a pixel-level segmentation of each object from its surroundings.

A variety of methods exist for object recognition and localization on still images (e.g., single video frames). A brief overview of some recent state-of-the-art techniques relevant to redaction is presented in the sections below. In Sec. 2.2, the ability to leverage temporal information to extend the results from object detection across a video sequence (i.e., series of video frames) is covered.

2.1.1.

Sliding window approach

When considering the output of an object detection algorithm as a bounding box around the object(s) of interest, one intuitive method that has been applied is a sliding window approach. Here, a detection template is swept across the image, and at each location the response of an operation such as an inner product with the template or a more complex image classification is computed. The resulting detections (desired bounding box locations) are then selected as the template locations (center and template size) that meet a predetermined response threshold. For example, the Viola–Jones sliding-window method¹⁰ was considered the state-of-the-art in face detection for some period of time. Unfortunately, sliding window-based approaches are computationally expensive as the number of windows can be very large to detect objects of different scale and sizes. As such, sliding window approaches are less common for video-based redaction applications.

2.1.2.

Region proposals

Region proposals are candidate object subregions (windows) in an image that are computed based on low-level image features. A variety of studies^11–17 have suggested different region proposal generation methods using techniques such as hierarchical grouping, superpixel features, and graph cuts to score candidate region proposals. In most cases, region-proposal-based methods tend to oversample the image. Producing more candidate regions than actual objects reduces the likelihood of missing an object of interest (i.e., trades off more false alarms for fewer missed detections). Improved localization performance is then often achieved by pruning the raw set of candidate regions using some form of image-based classification step.

Object detection using the region proposals is generally based on a classifier. The classifier [e.g., neural networks, support vector machine (SVM), $k$-nearest neighbor, etc.] is applied to the features [e.g., CNN, Harris corners, SIFT, histogram of oriented gradient (HOG), etc.] extracted for each of the candidate region proposals to obtain a confidence score or probability for each candidate region. In alternate approaches, the model directly regresses a refined set of bounding box parameters (e.g., location, width, and height) for the final set of object detections.

2.1.3.

Recent methods

Deep learning has achieved state-of-the-art results in a variety of different classification tasks in computer vision.¹⁸ In 2014, Region Convolutional Neural Network (RCNN)¹⁹ first applied the Convolutional Neural Network (CNN) architecture to the task of object localization. Since then, a variety of other deep learning-based methodologies for addressing the object detection and localization problem have been published, including fast-RCNN²⁰ and faster-RCNN.²¹

More recently, the you only look once (YOLO)²² architecture was shown to achieve computational throughput speeds compatible with real-time video processing while also producing object localization results comparable with the prior methods. YOLO formulates object detection as a regression problem with the objective to predict bounding box coordinates and class confidence in an image by applying a single-pass CNN architecture. For redaction applications, the class labels and bounding box coordinates can be used to determine which image subregions should or should not be obfuscated.

2.1.4.

Pixel-based methods

For redaction purposes, the output of an object detection might need to be inferred at a scale finer than a bounding box. For instance, consider the scenario in which there is PII for a bystander near a suspect’s face in an image. Here, choosing to not redact a bounding box region around the suspect’s face might be insufficient, allowing unwanted PII to be visible. An example illustrating the deficiency of using just a bounding box region around the suspect is shown in Fig. 3.

Fig. 3

Example of application of pixel-level segmentation on a sample image from AFLW²³ dataset.

As an alternative to a bounding box approach, it is possible to assign a class label to each pixel in an image. Although similar learning-based algorithmic techniques as described above can be used, the output is a semantically segmented image. Recent approaches^24–27 based on fully convolutional neural network (FCN) methods^24,28 take in arbitrary size images and output region level classification for simultaneous detection and classification. Chen et al.²⁴ used conditional random fields to fine tune the fully convolutional output. Wu et al.²⁹ did extensive experiments to find optimum size and number of layers, then used bootstrapping and dropout for highly accurate segmentation.

For redaction applications, pixel-based methods for object localization can provide some advantages in terms of better differentiating foreground areas of interest (i.e., nonredacted regions) and surrounding background content that is to be redacted. However, any pixel-level missed detections in such a scenario translate into the potential for under-redaction of personal information. Thus, appropriate design choices must still be made to ensure acceptable overall redaction performance. Here, both over- and under-redactions must be appropriately considered. Appropriate performance measures for redaction will be discussed in more detail in Sec. 3.

2.2.

Object Tracking

Similar to object detection in images, object tracking in videos is an important technology for automated redaction. Given the initial position and size of an object, a tracking algorithm should estimate the state of the object in subsequent video frames. By maintaining a “lock” on the object of interest (person, face, license plate, etc.), the tracking algorithm helps to maintain object localization despite potential errors being committed by the object detector running on each video frame. An example of this is shown in Fig. 4.

Fig. 4

Illustration of object detection compared with object tracking for a video stream. In the case of occlusions (center), tracking techniques are able to project the location of the object using temporal information. Example video from PEVid dataset.³⁰

Fundamentally, tracking an object involves extracting features of that object when first detected and then attempting to find similar (matching) features in subsequent frames. The major challenges associated with object tracking include illumination variation, changes in object scale, occlusions (partial or complete), changes in object pose or perspective relative to the camera, and motion blur. To be successful, tracking methods need to be robust to these types of noise factors, and tracking performance depends heavily on the features used.³¹

2.2.1.

Motion-model-based approaches

Temporal differencing algorithms can detect objects in motion in the scene; alternatively, background subtraction, which requires the estimation of the stationary scene background, followed by subtraction of the estimated background from the current frame, can detect foreground objects (which include objects in motion). The output of either approach is a binary mask with the same pixel dimensions as the input video that has values equal to 0 where no motion/foreground objects are detected and values equal to 1 at pixel locations where motion/foreground objects are detected. This detection mask is usually postprocessed via morphological operations that discard detected objects with size and orientation outside predetermined ranges determined by the geometry of the image-capture scenario. Once candidate foreground objects have been detected, methods such as particle filtering are typically applied to leverage temporal information in linking objects in the current image frame with observations of these objects from prior frames.

2.2.2.

Appearance-based tracking

Appearance-based trackers^32–34 rely on hand-crafted or machine-learned features of each object’s visual appearance to isolate and match objects. Simple examples of this type of approach would include using fixed template matching (e.g., using two-dimensional correlation) or color histogram matching (e.g., using the mean-shift algorithm) to follow objects from frame-to-frame.

Appearance-based methods tend to be susceptible to large appearance changes due to varying illumination, heavy shadows, dramatic changes in perspective, etc. To address these issues, some approaches make use of adaptive color features³⁵ or online learned dictionaries of appearance models (such as in the track-learn-detect paradigm^36,37) for objects that are being tracked. A key challenge with these types of methodologies then becomes the difficulty in tuning the online adaptation parameters. Here, the appearance models must be updated fast enough to accommodate changes in object appearance within the scene. However, tracker failures can increase if the models become overly responsive—incorporating too many extraneous appearance characteristics due to noise or surrounding background image content.

2.2.3.

Track-by-detection

Detection of objects in individual frames can be extended to enable tracking and trajectory estimation. Recent success in object detection has led to development of tracking by detection^38–40 that uses CNN to track objects by detecting them in real time. Such approaches, however, have limitations in handling complex and long occlusions, where the fundamental object detection will struggle. A recent work on object detection in videos⁴¹ used a temporal convolution network to predict multiple object locations and track the objects through a video.

Since the image-based detector must be applied at each video frame, a key challenge for track-by-detection methods tends to be the computational overhead required. However, since the YOLO approach leverages a single-shot network, it has been shown to achieve state-of-the-art object detection and localization in images at video frame rate speeds. Thus, YOLO is a natural candidate for tracking-by-detection-based approaches to following objects of interest in videos.

2.2.4.

Recurrent networks for object detection

An object detector used for tracking performs frame-by-frame detection but fails to incorporate the temporal features present in the video. To overcome this, a recurrent neural network can be applied to exploit the history of object location.⁴⁰ The recurrent units in the form of long short term memory (LSTM)⁴² cells use features from an object detector to learn temporal dependencies. The loss function for training the LSTM minimizes the error between the predicted and ground truth bounding box coordinates. Although the utilization of temporal information by tracking across multiple frames can increase robustness, it significantly deteriorates the ability to perform tracking in real time.

In another recent method,⁴³ an online and offline tracker is proposed for multiobject tracking. Appearance (based on CNN features), shape, and motion of objects are used to compute the distance between current tracklets and obtained detections into an affinity matrix. The affinity is used as a measure to associate the current tracklets with the detections obtained in a frame using the Kuhn–Munkres algorithm. The offline tracker uses $K$-dense neighbors to associate tracklets and current detection.

2.3.

Datasets of Interest

2.3.1.

Image datasets

The most common objects redacted for privacy protection are faces, persons (i.e., full human body), house numbers, vehicle license plates, visible computer screens (which may be displaying PII), and skin regions or markings (e.g., tattoos). There are a number of published object detection datasets that are relevant to redaction. Although not tailored specifically to redaction, many of these datasets contain objects of interest for redaction. In addition, these data sets typically provide annotation of individual object locations and class labels, so they can easily support performance evaluation of redaction methods.

The widely used PASCAL visual object classes (VOC) dataset^44,45 has detailed semantic boundaries of 20 unique objects taken from consumer photos from the Flickr website. Among the object categories, the most relevant to redaction are the person and TV/monitor classes. However, the car, bus, bicycle, and motorbike classes could also be of interest depending on the redaction application. The complete dataset consists of 11,530 training and validation images. The “person” category is present in over 4000 images and the “tv/monitor” category in roughly 500 images.

Alternative datasets include an increased number of annotated categories and images. For example, MSCOCO objects⁴⁶ have 80 categories with 66,808 images having person and 1069 images having tv/monitor categories as pixel-level segmentations. The ImageNet objects⁴⁷ have 200 categories of common objects annotated with bounding boxes in over 450,000 images. The KITTI dataset⁴⁸ consists of 7481 training and 7518 test images collected from an autonomous driving setting and consists of annotated cars and pedestrians. Figure 5 shows sample images from the KITTI⁴⁸ and PASCAL datasets with annotated detection boxes and segmented pixels, respectively.

Fig. 5

Sample images from Penn-Fudan⁴⁹ and LFW⁵⁰ datasets with annotated detection boxes and segmented pixels.

For exploring algorithms that specifically target the redaction of human faces, the annotated facial landmarks in the wild²³ (AFLW) is a popular dataset of $\sim 25,000$ annotated faces from real world images. This dataset includes variations of camera viewpoints, human poses, lighting conditions, and occlusions, all of which make face detection challenging. To make face detection more robust, Vu et al.⁵¹ introduced a head detection dataset that contains 369,846 human heads annotated from movie frames, including difficult situations such as strong occlusions and low lighting conditions.

Face recognition datasets are also useful in redaction systems to test the efficacy of different obfuscation techniques. The LFW Face dataset⁵² consists of 13,233 images of 5749 people collected from the internet. Similarly, the AT&T database⁵³ of faces contains 10 images each of 40 individuals. A more recent FaceScrub dataset⁵⁴ consists of 100,000 images of 530 celebrities.

2.3.2.

Video datasets

Beyond still images, there are video datasets relevant to algorithm development and performance evaluation of redaction methods. One example is PEViD,³⁰ which was designed specifically with privacy and redaction-related issues in mind. The dataset consists of 21 clips of 16 s sampled at 25 fps at $1920\times 1080$ resolution of surveillance videos in indoor and outdoor settings. The dataset has four different activities: walking, stealing, dropping, and fighting and has ground truth annotations for human body, face, body parts, and personal belongings such as bags. Another video dataset that can be useful to redaction is VIRAT,⁵⁵ which annotates vehicles and pedestrians on roadways.

In addition to detecting faces and heads, detection of the entire human body is very common in redaction. The Caltech Pedestrian⁵⁶ dataset is perhaps the most popular for detecting persons in images. It consists of $\sim 10\mathrm{h}$ of $640\times 480$ 30 Hz video taken from a vehicle driving through regular traffic in an urban environment. About 250,000 frames (in 137 approximately minute long segments) with a total of 350,000 bounding boxes and 2300 unique pedestrians are annotated.

Object tracking in videos is a very popular task among researchers; hence, there are numerous datasets available to evaluate and benchmark tracking algorithms. The most common and recent ones are Object Tracking Benchmark (OTB),⁵⁷ YouTube Object dataset,⁵⁸ and ImageNet object detection from video.⁵⁹ In most datasets relevant to redaction, the target objects are humans or cars of small size in a surveillance-like setting with less motion in the background.

In current publicly available datasets, collections of annotated images are much more prevalent than video datasets. However, a common practice is to train on (large) image datasets and then apply the resulting models to video frames for evaluation (track by detection). Until the redaction-relevant, publically available video datasets increase substantially in size, this approach of training algorithms on available large image datasets will likely remain a key component of many redaction solutions.

3.

Evaluation Metrics

The PASCAL VOC challenges⁴⁵ have been instrumental in setting forth standardized test procedures that enable fair comparison for benchmarking classification, object detection, and pixel segmentation algorithms. The questions “Is there a car in the image?,” “Where are the cars located in the image?,” and “Which pixels are devoted to cars?” are examples of classification, detection, and pixel segmentation problems. Scores are computed for each class and reported along with the average across all 21 classes. The PASCAL VOC challenges⁴⁵ use area under precision-recall curves. This is estimated at 11 equally spaced precision values from $0:0.1:1$ to ensure that only methods with high recall across all precision values rank well.

Taking the detection task as an example, each object has a ground truth detection box. An automated method attempts to find each object and return a bounding box around the object. If the detected box precisely overlays the ground truth box, we have detected the object. But what do we do if the detected box is shifted up and to the left by a few pixels? How about shifted down and to the right by half the width of the object? PASCAL VOC utilizes the intersection over union (IOU) metric, whereby a bounding box is said to detect an object if IOU is $>0.5$. IOU is defined as

For a given application, FN and FP pixel detections can have different levels of importance. For instance, it is likely that redaction applications cannot tolerate many FN detections as personally identifying information (PII) may be exposed. Similarly, if a face is correctly obfuscated in all but a handful of frames, the video cannot be considered properly redacted. Likewise, the amount of tolerable FP can be dependent on the application. In some applications, it can be acceptable to blur a region slightly beyond a face. The acceptable amount of blurring beyond the face can depend on factors such as not wanting to, or conversely not having a concern for, obscuring neighboring information. Increasing FP tends to decrease FN. Taken to a limit, the entire frame can be obscured leading to zero FN, but very high FP. Once again, the acceptable levels of FN and FP will be highly dependent on the redaction application.

To enable optimization for a given problem, we first define normalized errors

These error measures can be extended by considering that certain pixels in a detection area can be more critical than others. For instance, pixels in the periocular region can be more useful in identifying a person than points farther out in a bounding box. Also, pixels in the bounding box but not directly on the object of interest can have a low level of importance. One approach to addressing critical pixels is with a saliency weighting. Let $g{t}_i$ be the saliency weight of a pixel $x_i\in {\mathrm{GT}}_{\mathrm{BB}}$. The saliency weighted $\overline{\mathrm{FN}}$ becomes the sum of the saliency weights in the missed pixels normalized by the sum of the saliency weights in the ground truth

Saliency can also be used to avoid over redaction or redacting objects that need to be viewed, such as a weapon. The saliency weights $h_i$ would be for pixels in the image frame $H$ but not in the ground truth bounding box $x_i\in (H-{\mathrm{GT}}_{\mathrm{BB}})$. Saliency weighted false positive can be written as

This paper introduces the general concept of saliency for redaction; however, due to its application dependence, it is outside the scope of the paper to exercise it for various applications. Instead, we focus on unweighted $\overline{\mathrm{FN}}$ and $\overline{\mathrm{FP}}$ as per Eqs. (2) and (3).

To maintain similarity with the IOU metric that is prevalent in the field, we invert $\overline{\mathrm{FN}}$ and $\overline{\mathrm{FP}}$ errors to convert each into an accuracy, and then combine them into a single metric

Different values of $\alpha$ can be tailored to the specific applications. For example, in redaction, minimizing $\overline{\mathrm{FN}}$ is generally more important than minimizing $\overline{\mathrm{FP}}$; thus, $\alpha >0.5$.

To demonstrate the applicability of ${\mathrm{ACC}}_R$, Figs. 6 and 7 contain a few illustrative examples. The dotted green and blue dashed regions are $s$ and ${\mathrm{Det}}_{\mathrm{BB}}$, respectively. The IOU row uses Eq. (1), and the ${\mathrm{ACC}}_R$, $\alpha =0.5$ and ${\mathrm{ACC}}_R$, $\alpha =0.75$ rows use Eq. (6). With regards to Fig. 6, on the left, the ${\mathrm{Det}}_{\mathrm{BB}}$ of case 1 fills the entire image; as we step to the right, it occupies less and less until we get to case 4 where it occupies the area identical to ${\mathrm{GT}}_{\mathrm{BB}}$. As we continue moving to right, in case 7, ${\mathrm{Det}}_{\mathrm{BB}}$ occupies zero area. The $\overline{\mathrm{FN}}$ and $\overline{\mathrm{FP}}$ rows show the false negative and positive errors, respectively, due to the mismatch between ${\mathrm{Det}}_{\mathrm{BB}}$ and ${\mathrm{GT}}_{\mathrm{BB}}$. The ${\mathrm{ACC}}_R$, $\alpha =0.75$ row shows our recommended usage of Eq. (6) where, as desired for many redaction applications, false positives (on the left) are not penalized as much as false negatives (on the right).

Fig. 6

Performance of IOU and ${\mathrm{ACC}}_R$. The ${\mathrm{Det}}_{\mathrm{BB}}$ goes from maximum $\overline{\mathrm{FP}}$ to maximum $\overline{\mathrm{FN}}$ in seven equal increments from left to right. Case 4 represents ${\mathrm{Det}}_{\mathrm{BB}}$ that matches ${\mathrm{GT}}_{\mathrm{BB}}$ exactly. The dotted green region represents ${\mathrm{GT}}_{\mathrm{BB}}$, and the blue dashed region represents ${\mathrm{Det}}_{\mathrm{BB}}$.

Fig. 7

Performance of IOU and ${\mathrm{ACC}}_R$ on various test cases. The dotted green region represents ${\mathrm{GT}}_{\mathrm{BB}}$, and the dashed blue region represents ${\mathrm{Det}}_{\mathrm{BB}}$.

With regards to Fig. 7, case 8 is a typical example, whereby the ${\mathrm{Det}}_{\mathrm{BB}}$ is a mismatch to ${\mathrm{GT}}_{\mathrm{BB}}$ and, in this case, is smaller than ${\mathrm{GT}}_{\mathrm{BB}}$. In case 8, by setting $\alpha =0.75$, the $\overline{\mathrm{FN}}$ error is penalized more heavily, and the $\overline{\mathrm{FP}}$ error is penalized less. Since $\overline{\mathrm{FP}}=0$, the ${\mathrm{ACC}}_R$, $\alpha =0.75$ score is lower. In case 9, ${\mathrm{Det}}_{\mathrm{BB}}\subset {\mathrm{GT}}_{\mathrm{BB}}$ by the same amount as case 10 where ${\mathrm{GT}}_{\mathrm{BB}}\subset {\mathrm{Det}}_{\mathrm{BB}}$. As such, IOU and ${\mathrm{ACC}}_R$, $\alpha =0.5$ treat case 9 and case 10 the same. For obfuscation purposes, it is preferable to fully enclose the ${\mathrm{GT}}_{\mathrm{BB}}$. By weighting $\overline{\mathrm{FN}}>\overline{\mathrm{FP}}$, with $\alpha =0.75$ in Eq. (6), the ${\mathrm{ACC}}_R$, $\alpha =0.75$ row of Fig. 7 shows the benefits of the introduced ${\mathrm{ACC}}_R$ metric. Similarly, by comparing case 11 with case 12, the ${\mathrm{ACC}}_R$ rows of Fig. 7 correctly report low values when ${\mathrm{Det}}_{\mathrm{BB}}$ is much smaller or larger than ${\mathrm{GT}}_{\mathrm{BB}}$. Unlike IOU, which reports the same value for case 11 and case 12, ${\mathrm{ACC}}_R$, $\alpha =0.75$ clearly distinguishes the penalty for false negatives, which is how one might anticipate a redaction metric to behave.

To examine ${\mathrm{ACC}}_R$ further, Figs. 8 and 9 compare a sweep similar to Fig. 6. Figure 8 shows that the $\overline{\mathrm{FN}}$ is zero until ${\mathrm{Det}}_{\mathrm{BB}}$ becomes a subset of ${\mathrm{GT}}_{\mathrm{BB}}$. Similarly, $\overline{\mathrm{FP}}$ is zero when ${\mathrm{Det}}_{\mathrm{BB}}\subset {\mathrm{GT}}_{\mathrm{BB}}$. By comparing IOU to ${\mathrm{ACC}}_R$, $\alpha =0.75$, we can see that $\overline{\mathrm{FN}}$ is more important than $\overline{\mathrm{FP}}$. Figure 9 examines the behavior of $\alpha$ in Eq. (6). When $\alpha =0$, only the $\overline{\mathrm{FP}}$ term in Eq. (6) is used, and it only penalizes ${\mathrm{ACC}}_R$ when ${\mathrm{GT}}_{\mathrm{BB}}\subset {\mathrm{Det}}_{\mathrm{BB}}$. Similarly, when $\alpha =1$, only the $\overline{\mathrm{FN}}$ term in Eq. (6) is used, and it only penalizes ${\mathrm{ACC}}_R$ when ${\mathrm{Det}}_{\mathrm{BB}}\subset {\mathrm{GT}}_{\mathrm{BB}}$. When $\alpha =0.5$, ${\mathrm{ACC}}_R$ offers behavior similar to IOU and does not penalize false negatives appropriately for redaction applications. The solid blue line in Fig. 8 demonstrates our recommended $\alpha =0.75$, appropriately favoring false positives over false negatives.

Fig. 8

Comparison of IOU, $\overline{\mathrm{FN}}$, $\overline{\mathrm{FP}}$, and ${\mathrm{ACC}}_R$. ${\mathrm{GT}}_{\mathrm{BB}}$ is fixed at 50% of input image volume, and ${\mathrm{Det}}_{\mathrm{BB}}$ goes from maximum $\overline{\mathrm{FP}}$ to maximum $\overline{\mathrm{FN}}$ from left to right.

Fig. 9

Analysis of ${\mathrm{ACC}}_R$. ${\mathrm{GT}}_{\mathrm{BB}}$ is fixed at 50% of input image volume, and ${\mathrm{Det}}_{\mathrm{BB}}$ goes from maximum $\overline{\mathrm{FP}}$ to maximum $\overline{\mathrm{FN}}$ from left to right.

The above usage of Eq. (6) assumes that there is a single ${\mathrm{Det}}_{\mathrm{BB}}$ and ${\mathrm{GT}}_{\mathrm{BB}}$ per image. When multiple bounding boxes exist, all detection and ground truth boxes are merged into single, possibly fragmented, masks before applying Eq. (6). This ensures that all ${\mathrm{GT}}_{\mathrm{BB}}$ regions are fully enclosed by one or more ${\mathrm{Det}}_{\mathrm{BB}}$. Using $\alpha =0.75$ continues to penalize false negatives more than false positives.

Figure 10 compares the change in ${\mathrm{ACC}}_R$ with varying bounding box detections. We observe that, as the detected bounding box covers more ground truth area, i.e., the FN decreases, ${\mathrm{ACC}}_R$ becomes higher. As more area from the ground truth is missed, the ${\mathrm{ACC}}_R$ score is penalized. This is an important property of the redaction accuracy.

Fig. 10

Comparison of IOU and ${\mathrm{ACC}}_R$ with varying values of $\alpha$. Solid blue line is detected bounding box and dashed green line is ground truth bounding box. Example image from PEVid dataset.³⁰

3.1.

Comparison of Methods

We evaluate the proposed metric on four different object categories that are relevant to redaction—faces, human heads, persons, and tv/monitors. For faces, we use the AFLW²³ dataset. The faster-RCNN technique uses 50% of images for training and the remaining 50% for testing. To compare recent deep learning methods with a classical object detector, we used the HOG feature combined with a linear classifier, an image pyramid, and sliding window detection scheme for the face and person categories. The mean redaction accuracy (${\mathrm{mACC}}_R$) is defined as the average score over all test images to compare the performance over a dataset.

As reported in Table 1, the faster-RCNN method achieves lower FN but higher FP compared with the HOG-feature-based method. This is typically a desirable property in a redaction system where failing to redact parts of an object may reveal sensitive information. Analogously, faster-RCNN is advantaged for higher alpha values where FN results in a higher penalty than FP. Conversely, applications that are required to penalize FP more than FN would benefit from lower $\alpha$ values and the HOG-based method.

Since for a redaction application, missing side views or occluded faces can also reveal the identity of persons, we run experiments for comparison of face and head detectors. We train a head detector to compare the robustness in detecting faces due to occlusions and view angles. The YOLO model was trained on images from the Head Annotations⁵¹ dataset. Testing was done on the FDDB face detection dataset⁶⁰ with 5171 faces in 2845 images, and the results are reported in Table 2. The testing was done on a dataset with ground truth only for faces (and not heads), and the qualitative improvement do not directly translate to the metrics. Therefore, we show examples in Fig. 11 comparing face and head detectors. The mAP scores are low due to cross dataset testing. The YOLO framework has strong spatial constraints imposed by limiting two boxes per grid cell; hence, it fails to detect small objects that appear in groups. Moreover, since the learning is done to predict bounding boxes from data, it struggles to generalize to objects in new or unusual aspect ratios or configurations.

Table 1

Performance comparison for face detection using classical HOG features with a linear classifier and the more recent faster-RCNN method. mFN¯ and mFP‾ are mean normalized errors, mACCR is mean accuracy for redaction, and mAP is mean average precision.

Method	HOG + Lin. classifier	Faster-RCNN
$\mathrm{m}\overline{\mathrm{FN}}$	0.322	0.058
$\mathrm{m}\overline{\mathrm{FP}}$	0.089	0.338
mAP	0.364	0.672
${\mathrm{mACC}}_R(\alpha =0.1)$	0.887	0.689
${\mathrm{mACC}}_R(\alpha =0.3)$	0.840	0.745
${\mathrm{mACC}}_R(\alpha =0.5)$	0.793	0.801
${\mathrm{mACC}}_R(\alpha =0.7)$	0.747	0.857
${\mathrm{mACC}}_R(\alpha =0.9)$	0.700	0.913

Table 2

Performance comparison of YOLO model trained on face and head datasets for testing on a face dataset. mFN‾ and mFP‾ are mean normalized errors, mACCR is mean accuracy for redaction, and mAP is mean average precision.

Method/training data	YOLO/face	YOLO/head
$\mathrm{m}\overline{\mathrm{FN}}$	0.687	0.566
$\mathrm{m}\overline{\mathrm{FP}}$	0.138	0.172
mAP	0.090	0.142
${\mathrm{mACC}}_R(\alpha =0.1)$	0.367	0.787
${\mathrm{mACC}}_R(\alpha =0.3)$	0.477	0.709
${\mathrm{mACC}}_R(\alpha =0.5)$	0.587	0.630
${\mathrm{mACC}}_R(\alpha =0.7)$	0.696	0.551
${\mathrm{mACC}}_R(\alpha =0.9)$	0.806	0.472

Table 3

Performance comparison for person detection using classical HOG features with a linear classifier and more recent faster-RCNN and YOLO methods. mFN‾ and mFP‾ are mean normalized errors, mACCR is mean accuracy for redaction, and mAP is mean average precision.

Method	HOG + Lin. classifier	Faster-RCNN	YOLO
$\mathrm{m}\overline{\mathrm{FN}}$	0.415	0.119	0.189
$\mathrm{m}\overline{\mathrm{FP}}$	0.372	0.220	0.202
mAP	0.194	0.758	0.654
${\mathrm{mACC}}_R(\alpha =0.1)$	0.057	0.776	0.735
${\mathrm{mACC}}_R(\alpha =0.3)$	0.048	0.796	0.752
${\mathrm{mACC}}_R(\alpha =0.5)$	0.039	0.816	0.769
${\mathrm{mACC}}_R(\alpha =0.7)$	0.031	0.836	0.786
${\mathrm{mACC}}_R(\alpha =0.9)$	0.022	0.856	0.803

Fig. 11

Examples to qualitatively compare a face and head detector. (a) Original images, (b) YOLO face, and (c) head detector outputs, where the bounding boxes are outputs of the YOLO models. Example images from the AFLW dataset.²³

For the “person” and “tv/monitor” object categories, we used the standard train/test splits from the PASCAl-VOC 2007⁴⁵ dataset. The results are reported in Tables 3 and 4. The faster-RCNN achieved the lowest false negatives and hence the best mAP scores among the three methods. Among different techniques, the selection is based on the method that is best suited to detect an object category. Similarly, the selection of $\alpha$ values depends on the desired performance in terms of FN versus FP.

4.

Types of Obfuscation

Once all objects with private information are detected, the information needs to be obfuscated in a manner that protects the privacy. These obfuscations can be simple methods such as blanking or masking objects such as faces with shapes in individual video frames. Other common obfuscations are blurring, pixelation, or interpolation with the surroundings. More complex methods include geometric distortion and scrambling that allows decryption with a key. We discuss common obfuscation methods below. Several examples are shown in Fig. 12.

Fig. 12

Illustration of various obfuscation techniques. (a) Original image, (b) full frame blurred, (c) bounding box of face blurred, (d) bounding box of face blanked out, and (e) pixelated bounding box of face.

First, consider various approaches that can be taken for bounding the region to be obscured using blurring as an example obscuration method. At a coarse level, an entire image frame that contains any sensitive information can be blurred. This may be useful when the video is relatively long compared with a small number of frames that need to be obscured or it is determined that blurred information is sufficient for the viewer. For instance, in a courtroom showing an auto accident, the overall movement of the vehicles may be adequately observed in a video that is blurred to a degree that obscures the identify of persons and license plates in the video. Blurring the entire frame is a simple method for protecting information and can ensure a high level of protection, but important context of the scene may get lost.

The sensitive region can be defined as the detected bounding box around the subject of interest. The tolerable “looseness” of the bounding box balances the trade-off between false positives (which can obscure context) and false negatives (which potentially reveal PII). A looser bounding box increases FP and decreases FN, and vice versa. This trade-off should be selectable according to the given application requirements. If the general shape of the sensitive information is known, the detection box can be used to place an alternative mask in that region. For example, ellipses of different aspect ratios are sometimes used for face and body redaction. As indicated in Sec. 2.1.4, object detection might need to be inferred at a scale finer than a bounding box.

Obscuration methods must be understood in the context of their ability to suitably mask the sensitive information and the parameters used within the method. Fully blanking out or masking pixels in the sensitive region is the most secure method, but this can significantly affect certain contextual information, such as movement and actions in the region. More typical is blurring using a Gaussian blur kernel, which brings in the issue of selecting Gaussian parameters that provide adequate obfuscation. Pixelation (mosaicing) is another common obfuscation method. The region to be obfuscated is partitioned into a square grid, and the average color of the pixels contained within each square is computed and used for all pixels within the square. Increasing the size of the squares increases the level of obfuscation. Figure 13 shows example images with varying degrees of blurring and pixelation. Blurring was performed using the Gaussian blurring function in OpenCV.⁶³ The standard deviation of the Gaussian kernel was varied to achieve multiple degrees of blurring. The degree of pixelation was controlled by changing the size of the squares used in averaging.

Fig. 13

Varying degrees for obfuscation on example images from (top to bottom) AT&T,⁶⁴ AFLW,²³ VOC,⁴⁵ and MediaLab LPR⁶⁵ datasets. All images are cropped to dimensions $92\times 112$. Standard deviation of Gaussian kernel is varied as $5\times 5$, $15\times 15$, $25\times 25$, and $45\times 45$. Pixelation window size is varied as $2\times 2$, $4\times 4$, $8\times 8$, and $16\times 16$.

Interpolation⁶⁶ with the background can be useful in applications that require the blurred image to be free from redaction artifacts. Studies such as Ref. 67 have also studied skin-color-based face detection.

In some applications, there is a requirement to retrieve the original object after redaction. This can allow release of the video where authorized parties possess a key that enables decryption. A system to retrieve the original data with proper authentication is presented by Cheung et al.⁶⁸ They use a rate-distortion optimized data-hiding scheme using an RSA key that allows access only to authenticated individuals. Similarly, Ref. 69 presented a retrievable object obscuring method.

Similar to the visual content, audio is also an integral part of surveillance videos. Detecting the audio segment to redact can either be based on the object detection in the parallel video stream or can be an independent search for audio clips. The audio segment could be replaced with a beep, muted, or modulated such that the original sound is protected.

5.

Open Problems

We discuss several open problems and challenges associated with video redaction systems.

Although state-of-the-art computer vision is increasingly robust in detecting certain objects, such as faces, bodies, and license plates, the sensitive PII can take on many diverse forms that will confound attempts to fully automate the process (e.g., skin, tattoos, house numbers written in script, logos, store front signs, street signs, and graffiti). Skin occurs in many tones, and color-based segmentation is not robust for sensitive applications. While character recognition may be robust for conventional documents, recognition in the outdoors is a different problem. The video may have been captured in very suboptimal conditions, such as poor lighting and geometric perspective. In any given application, particular objects may need to be obfuscated while other instances of that object class must be clearly visible (blur face 1 but not face 2).

The public concern over privacy coupled with the need for low cost ever vigilant security will drive privacy protection into smart cameras, so certain material is never stored or transmitted, except possibly with special encryption. While some complex custom obscurations may not be possible, mainline tasks such as face obscuration could be performed by computing on the edge. The performance of such redaction systems would depend on the accuracy of the face detection and obfuscation methods. Moreover, the processing time becomes a critical requirement since the amount of data is ever increasing.

Law enforcement applications cannot release any sensitive data. If a single frame in a video is missed by a redaction system, it could reveal the identity, for example, of a witness and put them in danger. That one missed frame can defeat the value of redacting thousands of other frames in the video sequence. This sensitivity necessitates a manual review of the redacted output. Efficient review methods can greatly reduce labor costs.

6.

Conclusion

With the rising popularity of surveillance, body, car, and cell phone recording devices, imagery is increasingly being used for public purposes such as law enforcement, criminal courts, and news services. Often, the personal identity of people, their cars, businesses, or homes are identifiable in these recordings. Video redaction or obfuscation of personal information in videos for privacy protection is becoming very important. Object detection and tracking are two key components of a redaction system. The current advances in the field of deep learning achieve state-of-the-art performances in object detection and tracking. However, the current evaluation metrics do not consider redaction-specific constraints. The presented redaction metric is promising for evaluating redaction systems. We compare classical methods with recent deep learning-based methods on redaction-specific object categories. While designing a video redaction system, the most desired property is having a fewer number of frames that require a manual review. This depends on factors such as threshold on the accuracy, the object of interest, detection and tracking method, and desired performance in protecting the object (FP versus FN). More challenges such as processing time, raw video retrieval, and manual review remain active research areas.