Full Speed Ahead for Full Motion Video

Written by Cheryl Gerber

GIF 2009 Volume: 7 Issue: 5 (October)

 

While full motion video (FMV) analysts must search through terabytes of video and image data coming from many disparate sensors and systems, ongoing developments to improve FMV analysis are now moving full speed ahead, sparking efforts that have fused, automated and standardized the technology. The result is more comprehensive coverage, leading to increased situational awareness and actionable intelligence.

The anchor for broad FMV analysis improvement is a standardsbased architecture in which to assimilate and unify all the different sensors, systems and tools, thus increasing interoperability and information sharing.

Ongoing efforts to achieve a standard architecture reached a milestone this year, when the National Geospatial-Intelligence Agency advanced the National System for Geospatial Intelligence Objective Video Architecture (NOVA) from pilot to operational status. NOVA is a standards-based, services-oriented architecture (SOA) that assimilates and unifies disparate GEOINT sensors and systems.

In addition, during Empire Challenge 2009 this summer, NGA demonstrated NOVA on the same network with SensorWeb, an SAIC architectural framework, for the first time. Empire Challenge is an annual, live interoperability demonstration of joint and coalition ISR capabilities, executed by Joint Forces Command and sponsored by the under secretary of defense for intelligence.

NOVA just took its first operational step. “We still have questions about how this will perform on an operational network in wartime. Dealing with differences in how networks perform is a big challenge,” said Commander Joe Smith, NGA military deputy for the Sensor Assimilation Division of the Acquisition Directorate. “So we are now moving equipment from R&D to implement it in labs that are part of the operational environment.

“The NOVA proof of concept started because of the rapid increase of full motion video collected by the Department of Defense in a piecemeal fashion from Operation Enduring Freedom—Predator and Global Hawk, for example. This year we looked at how we would implement TiVo-like functions in NOVA to give advanced functionality to DoD and other federal, state and local agencies,” said Smith.

NOVA is the emerging standard architectural platform on which to achieve interoperability and the sharing of FMV analysis information. “NOVA specifies how to engineer a system for full motion video and motion imagery in order to use NGA data. The goal of NOVA is to ensure interoperability without specifying any point solution. If any of the military and intelligence services implement the standards correctly and understand the engineering principles in NOVA, then we have a high reliability that those systems will be interoperable and able to share data relatively seamlessly,” he said.

The Predator is an armed or unarmed UAV used by the Air Force and CIA primarily for reconnaissance, while Global Hawk is used by the Air Force for surveillance. TiVo-like digital video recorders (DVR) contain a search engine and save designated video on a hard drive. Users can then fast-forward, rewind, pause, remote-program, search video files with set parameters and many other features.

“We want to handle video as streaming media—like YouTube— instead of breaking it into 17 small files that are each six minutes long and then asking the user to pull it all together. If you tag video with information tags, then you can find it and combine it better,” Smith said.

METADATA TAGS

To tag video files, FMV analysis software has increasingly incorporated metadata, such as Extensible Markup Language (XML), the widely used standard recommended by the World Wide Web Consortium. XML contains both the data and the description of the data. XML organizes and tags data so it may be transmitted and interpreted between applications and organizations and shared on the Web. The Open Geospatial Consortium (OGC) has approved standards using XML.

SAIC’s SensorWeb is one of many systems that have incorporated XML. SensorWeb is an SOA based on standards approved by the OGC and NGA’s Motion Imagery Standards Board (MISB). It provides near real-time access to disparate sensors and systems with sensor-tosensor and sensor-to-user interoperability.

“The MISB standards allowed us to support NOVA with much less development effort than it would have been otherwise. We were able to share data quickly and easily,” said Charlie Gates, SAIC SensorWeb lead in operations and intelligence. “SensorWeb is an R&D project for the Department of Defense to enhance interoperability between DoD, coalition partners and government agencies such as the Department of Homeland Security.”

As a means to that end, SAIC’s Exploitation Technologies Division is working to build out the standards. “We’re working with the MISB to convert video and metadata from different sensors into a standard profile that can then be used by a suite of tools. We’re finally starting to get traction. We are now moving that up with SensorWeb’s integration of sensors,” said Alan Schaar, director of engineering for SAIC’s Exploitation Technologies Division.

All of the major FMV analysis platforms and tools incorporate standards, and are upgrading to provide DVR features and fuse disparate GEOINT video data. These include the Full Motion Video Asset Management Engine (FAME) from Harris, Motion Video Exploitation (MVE) from Intergraph, and adLib or Mobile Data and Retrieval from EchoStorm, as well as video content analysis products like VideoAnalyst from IntuVision. All of these illustrate the move to incorporate standards, provide digital video recorder features and fuse disparate GEOINT video data to achieve more comprehensive situational awareness.

“Most systems today were stovepiped, with a lot of different unconventional warfare flaring up in different locations. We wanted to be able to fuse data, but we couldn’t unless we had a standard platform to fuse it on,” said Gates.

WEB ENABLEMENT

As a result, SensorWeb architecture is based on Sensor Web Enablement (SWE), a set of Web services approved by the OGC that enables sensor discovery and report dissemination using common, open source data exchange standards. SensorWeb also maintains support for the Common Alerting Protocol, used by local civilian authorities and first responders.

“The goal of the OGC’s Sensor Web Enablement group is to provide standard technology with common ways to describe discovery and tasking sensors and to subscribe to alerts or retrieve observations,” said Mike Botts, chair of the SWE working group and a member of the OGC.

Using SWE standards can resolve the problem of disparate incompatible systems. “There is a wide range of disparate sensors, satellites and UAVs. Some are simple and some are complex, but each has a different means of tasking the sensors or getting the data back from the sensors. Different types of sensors have communities that have formed around them, and they have their own way of dealing with data and applications,” said Botts.

One of the most salient emerging standards to raise the bar on FMV analysis is the Sensor Model Language (Sensor ML), an XMLbased language describing sensor systems and the processes surrounding observations that allows on-demand, real-time geo-location of video with analysis software. XML provides the standard Web encoding of information.

SensorML was approved by the OGC, Geospatial Intelligence Standards Working Group (GWG) and DoD Information Technology Standard and Profile Registry. “Sensor ML was under development with NASA funding, then it was brought into the OGC, where it served as a catalyst for SWE activities,” Botts said.

FMV analysis is significantly enhanced by using standard Web interfaces, such as three other OGC-approved sensor standards used by SAIC’s SensorWeb that are now under consideration by the GWG. They are Sensor Observation Service, which allows users to retrieve observations through a standard Web interface; Sensor Planning Service, which allows users to task a sensor through a standard Web interface; and Sensor Alert Service, which allows providers and users to publish and subscribe to alerts with a standard Web interface.

DATA FUSION

FMV analysis is also the latest enhancement to established enterprise geospatial solutions, such as Intergraph’s open and extensible geospatial solution, Motion Video Exploitation (MVE), which natively accesses multiple sources of intelligence, including video, in their original formats.

The MVE solution fuses multiple motion video data streams with other intelligence data in one high-powered analytical environment, where analysts can place clipmarks, annotations and generate a report as a static GeoTiff or NITF image for broad dissemination.

Intergraph also relies on Open Geospatial Consortium (OGC) Web services standards for retrieval and dissemination of geospatial information, including KLV (key, length and value) metadata, a dataencoding protocol that complies with the International Standards Organization. KLV was designed to make metadata easier to exchange between different media and metadata standards. The use of open standards allows the Intergraph technology to blend into existing architectures and connect to a vast amount of intelligence data that resides in other systems.

“The video feeds arrive with KLV metadata that contains the time and geographic location of the video stream. The software uses the KLV metadata to georegister FMV in a Multi-INT environment so the user can look at the same location at the same time from different sensors and camera angles, and be instantly oriented. This is vital to today’s analysts that need to make decisions very rapidly, but need to deal with continually increasing amounts of video data. This is a unique capability that only Intergraph provides,” said Leah Wood, Intergraph defense and intelligence industry manager. “Combining video data with other geospatial intelligence data, Web services and open source data enables intelligence analysts to make more informed decisions for actionable intelligence.”

The MVE early adopter program will yield valuable feedback for Intergraph’s video geo-registration techniques and allow early users to participate in the ground breaking development of this leap in technology. This program allows Intergraph to ultimately release MVE as a COTS solution that addresses very sensitive and complex requirements.

“We wrote technically advanced software for this. In order to project any raster to the Earth, you have to have a quick and efficient way to calculate where the Earth’s surface is for each pixel,” said Elaine Woodling, product manager of Intergraph’s GeoMedia Motion Video Analyst Professional, the flagship product of Intergraph’s MVE application suite.

The multi-faceted MVE solution also provides a video warehouse for the management and exploitation of FMV on an enterprisewide basis. Complete with an automatic data ingest capability, the video warehouse provides the ability to store and query clipmarks, then use the query results to control the DVR with TiVo-like features.

The five major MVE applications include Motion Video Analysis, which provides clipmark and single frame extraction capabilities for near real-time decision-making, and Motion Video Analysis Professional, which provides graphic annotation of FMV and related geographic data, including raster maps, vector data, and satellite imagery. Intelligence analysts use this version to generate video clips from previously placed clipmarks or mosaic images of the clips and place annotations on videos and images to produce image products such as GeoTIFF and NITF images.

Multi-INT Exploitation provides an advanced environment for the analysis of all types of geospatial intelligence at central command, including raster maps, vector data, motion video, satellite imagery, signal intelligence, terrain data and 3D visualization. The environment allows the analyst to fuse motion video with other geospatial intelligence data to form a comprehensive operational picture from which to make tactical and strategic decisions.

The fourth application is Video Exploitation, an Adobe-based viewer for use on ruggedized laptops to review video clips in the field using enhancement and stabilization techniques and the ability to add additional field annotations for later review. The fifth is Geospatial Content Management, which facilitates the ingest and management of video formats that provide geospatial analysis and other services that require the collection, storage and retrieval of FMV for historical situational awareness.
MVE complies with MISB standards and utilizes EchoStorm’s adLib product as the interface to a varied number of sensor formats. A standards-based SOA, adLib also complies with MISB and the OGC’s Web Map Service and can capture video from a broad variety of sources and supply them to a variety of devices.

BROADCAST PERSPECTIVE

Harris approached FMV analysis from a broadcast technology perspective when it introduced its FAME architecture for intelligence networks in 2007. “The FAME system architecture comprises the same hardware and software tools used in Fox, CNN and the BBC as it collects, manages, fuses and distributes rich media content and associated data,” said Lucius Stone, director of government solutions for Harris Broadcast Communications Division.

FAME integrates video analytics, coding, video/audio/chat and imagery capture, processing storage, graphics and dynamic distribution on one platform, employing MISB standards for FMV and metadata. Stone views FAME as a solution to the same operational splintering that others have observed. “There are a lot of people who collected video on one system, imagery on another system and signal intelligence on yet another. They wound up with silos of systems. Our mission with FAME is to fuse a lot of different kinds of formats and content into one common operating picture,” said Stone.

“FAME performs tasking, collection, processing, exploitation and dissemination of content, bringing it to the analysts, who send it to the decision-makers, who then send it back to the warfighter, who is often only one mile away from where the data was collected,” he said.

To broaden its reach, Harris signed an agreement this year with Lockheed Martin to develop advanced technologies for the management and analysis of video intelligence. The companies plan to create new video analysis solutions for defense and intelligence customers.

Lockheed Martin will contribute expertise derived from its Audacity video analytics engine, which tags and sorts huge libraries of digital video footage into a hierarchal, searchable catalogue. The system sorts through video streams in near real-time, automatically alerting users as new critical video becomes available. Rule-based alerts allow users to view video ranked in priority order.

Audacity users can place icons, reference grids and other markings into video stills to highlight key events. Videos can be uploaded automatically to a common operating picture, blog or wiki to share clips with colleagues. Audacity plugs into the Distributed Common Ground System Integration Backbone (DIB) with a two-way interface for defense and intelligence users to share global video feeds over an established DIB network. Future Audacity upgrades will address low bandwidth communications on the front lines, where video feeds can overwhelm tactical networks.

AUTOMATED TOOLS

FMV intelligence analysts also have been using automated tools from IntuVision—Panoptes (“all seeing” in Latin) and VideoAnalyst. Deployed at a classified DoD site, Panoptes watches video streams and auto-detects events, such as parameter intrusions or trip wire violations as they occur in real-time, at which point the system sends out alarms. VideoAnalyst is a post-forensic tool that searches through archives of video looking for pre-designated events such as scene changes. The product automatically ignores massive amounts of irrelevant data, while focusing on segments of interest.

“One of the newest features in full motion video analysis software that users like is the ability to query a detected event to reveal patterns within a parameter, such as intrusions,” said Sadiye Guler, chief executive officer of IntuVision. “The time element is also important, such as that of a vehicle waiting repeatedly at certain times.

“Up until now, analysts did not have the automated tools to search many hours of video sources quickly they way they had been able to search text and documents. Now you can search video for faces, for example,” she added.

Guler noted that the software has been enriched in the past year with the merging of multiple functions. And now, increasingly, FMV analysis software will incorporate computer vision technologies—the science and technology of machines that see—used for industrial robots, autonomous vehicles, event detection and visual surveillance.

Yet to achieve the ultimate FMV analysis improvement, which translates into saving lives, the multiple disparate systems of today will need to merge into the standards-based architecture of tomorrow.

“When the different systems and sensors speak the same language and present their findings in a unified manner, it will bring a whole new level of usability to the picture. There’s unlimited potential to prevent disasters from occurring,” she said. ♦

Back to Top