Machine Vision Group
Introduction Architecture and Interfaces Current Implementation Demo Download Contact
Introduction

Distributed smart cameras and sensors have been an active area of research in recent years. Most of the research has focused on either the machine vision algorithms or on a specific implementation. There has been less activity on building generic frameworks which allow different algorithms, sensors and distribution methods to be used.

Scallop framework aims to provide the user with simple and configurable interfaces for accessing sensors and communicating with other nodes. Various sensor and network types can be used, enabling node heterogeneity. Uniform event based interfaces are provided for both sensors and networks, with bindings for Microsoft .NET Framework (.NET) languages and we have plans for C++ support. Node deployment and runtime configuration is simplified through the use of XML configuration files.

The framework has been designed and implemented with the following guidelines in mind:

  • Allow researchers to focus on machine vision algorithms by providing access to sensors and communication networks.
  • Allow new sensor and network types to be implemented easily by defining an interface that implementations must conform to.
  • Keep the interfaces simple and uniform over different types of sensors and networks.
  • Separate sensors and communication from machine vision algorithms.
  • Allow customisations by releasing/open sourcing the code.

The .NET Framework was chosen as the implementation platform due to it's increasingly widespread use, availability of tools and built-in support for networking, databases and other needed resources. User code is expected to run on PC workstations (for ease of development), a platform for which the .NET Framework is well suited for. This environment lends itself well to rapid prototyping and development.

As of version R2009a, MATLAB includes an interface to the .NET Framework. In consequence of this, it is now possible to use the Scallop framework even in MATLAB environment. The source code package includes an example of using MATLAB to access frames from an Axis IP camera and doing some image processing.

Example Usage Scenarios

The following are some simple examples of scenarios where the framework could be employed.

Multicamera surveillance systems

A system consisting of multiple smart camera based surveillance nodes and a smaller set of monitoring nodes. The surveillance nodes receive image frames from their sensors, extract features from the images and track detected objects. They then inform each other and the monitoring nodes of detected and tracked objects. As an object leaves the node’s field of view, it communicates with other nodes over the network, passes them object data and hands over the responsibility for tracking the object if possible. The monitoring nodes can present this information to the operator of the system. No central point of control is in control of overall tracking, the nodes distribute the responsibility among themselves.

Example scenario

Nodes with multiple sensors or networks

Using the framework, nodes can use several sensors. One possible scenario is a set of nodes, each tasked to monitor a room. A node can use two or more different cameras or other sensors, and communicate with other nodes or a central monitoring node based on the information it collects from the sensors.

Example scenario

Sensorless nodes

Not all nodes need to have a sensor attached. They can be bare processing nodes, only processing raw data sent to them by sensor nodes. Another possible scenario is a database node that can save and be queried about object features. User interfaces can also be implemented in this way.

Example scenario

Smart environments

Smart environments, where sensors interact and assist people in their activities, are also a good example of where the framework could be used. The networking interface can be used to connect the sensors, displays and user interaction devices together, while the sensors can take advantage of the sensor interface.

Architecture and Interfaces

System Architecture

The system architecture is shown below. It consists of interfaces for sensor and network access, and modules for specific types implementing that interface. The commands and events used to interact with the resources are the same for each type, allowing the same code to be used while changing the resource type. Only the runtime configuration needs to be changed.

  • Separate interfaces for sensors and network.

  • Sensor and network modules are configured using XML files.

Example XML file

  • XML schemas are provided for easy editing and validation of the configuration files.

Network Interface

  • Contains method definition for interacting with the network layer.
  • Asynchronous message delivery using XML documents.
  • Schemas for validation of outgoing and filtering incoming messages.

  • Designed for mesh-based peer systems but can also be used for multicasting or structured P2P networks.

Sensor Interface

  • Provides methods for configuring and starting and stopping.
  • Sensor access through primitives.
  • Data is passed using event callbacks.
Current Implementation

Network

We have implemented a network module utilising the Microsoft P2P Framework using Windows Communication Foundation (WCF) PeerChannel, the managed implementation in .NET 3.5. The module creates an internal communication endpoint for messaging with other nodes and presents the user with the Scallop interface for communication with peers.

In the PeerChannel implementation, the nodes are connected by a P2P mesh, allowing ad-hoc network formation and node dispersal. The mesh is formed using the Peer Name Resolution Protocol (PNRP), which in turn utilises the Simple Service Discovery Protocol (SSDP) to find and connect to neighbouring peers.

Communication between nodes takes place over a many-to-many broadcast channel, constructed using the point-to-point connections of the underlying mesh. Messages are propagated by flooding them to neighbouring nodes. The nature of the peer mesh also allows message propagation to be restricted by limiting the number of hops a message can traverse. Several channels can be overlaid on the mesh and messages can be encrypted with a password or certificate using Transport Level Security (TLS).

Our network implementation consists of code to expose the PeerChannel functionality through a Scallop network interface. The network only provides a best effort service, messages are not guaranteed to reach all nodes. In addition, the mesh nature of the network leads to a transmission delay, which might not be acceptable in some user scenarios.

Sensor

We have implemented sensor source modules for Axis IP cameras and video files. The Axis cameras are accessed through the Axis Application Programming Interface (Axis API). The camera unit provides a Hypertext Transfer Protocol (HTTP) Motion JPEG stream, that is parsed into individual frames that get passed on to the user. The camera modules are fully configurable. They can also automatically recover from network outages which can be frequent in a wireless environment.

Demo

A demonstration system was implemented to test the suitability of the framework for a distributed multi-camera application. The system is composed of a set of processing nodes running on PC workstations, and Axis 210A/213 IP-cameras acting as sensors. These are accessed through the Scallop sensor interface. Each camera is configured to produce frames of 320x240 pixels at a rate of 15 fps. These are passed to the user code as bitmaps, and an object detector is then used to extract human shapes from them. Each object is given a unique ID and tracked between successive frames to produce individual trajectories.

The node communication takes place over a hybrid wired and wireless network. Every time a node detects an old or new object, it informs the other nodes through the network interface. Object data and features from the detector are sent to the peer nodes using XML, with an encoded thumbnail image of the detected object. This provides a greater amount of test traffic on the network than simply broadcasting the first and last time an object is seen.

Videos

  • Processing node in action

    Adobe Flash Player video
  • Visualization UI that collects all the messages sent by the processing nodes

    Adobe Flash Player video
Download

License

Licensed under the MIT License, Scallop is free and open source software.

Code

Publications

Contact