PROTOS Genome Test Suite c10-archive


Archive formats are used to serialise a set of files and directories into a single byte stream, usually applying a form of compression in the process. The archive files can then be stored or transmitted on various media conveniently and economically, and later extracted. The use of archiving formats is ubiquitous in transmitting files over email and in distribution of software, among other areas.

The present set of archive formats were chosen as the subject protocols for vulnerability assessment through structure inference directed fuzzing and test suite creation.

A list of frequently observed archiving formats was drawn up. Test material was prepared and tests were carried out against a sample set of existing anti-virus programs. Results were gathered and reported.

Most of the implementations available for evaluation failed to perform in a robust manner under test. Some failures had information security implications, and should be considered as vulnerabilities.

In order to achieve a robustness baseline for archival products, this test material should be adopted for their evaluation and development. Anti-virus and other security products employing archive formats should be considered the most important subjects in this respect.


This test suite is a byproduct of the [Genome] project, hereby referred as GENOME. The test suite contains a set of fuzzed archive files in different formats, some of which may cause and some are known to cause problems for example in common decompression and anti-virus tools.

This test suite covers a limited set of information security and robustness related implementation errors for subsets of the chosen protocols. The subject protocols, along with their scrutinised subsets, are illustrated in the Analysis section below.


The purpose of this test suite is to evaluate implementation level security and robustness of programs handling archive files of different formats. Archive formats were considered a viable topic for a test suite due to the following factors:

The field of archive formats was analysed with the methods of OUSPG's MATINE project. The focus of this analysis was on the different formats and their specifications, their different technical and organisational uses and prior security issues affecting them. The analysis methods lay weight on issues regarding the history of code and specifications (inheritance, re-use), historical data on the usage and prevalence of different implementations and expert opinion.

The analysis highlighted anti-virus software as representative, or topical, subject for this test suite. Motivation for producing test material targeted at ensuring robustness of anti-virus tools include:

It was noted that anti-virus tools parse many different kinds of data, and this test material, being limited to archive formats only, can only serve as a decent first aid for related vulnerability assessment. A proper security evaluation of anti-virus software would involve scrutinising a much greater set of file formats.

In this test-suite, the focus was set on the certain archive formats, namely ACE, ARJ, BZ2, CAB, GZ, LHA, RAR, TAR, ZIP and ZOO. This set encompasses the most commonly used archive formats.

The specifications for the archive file formats are in some cases available. However, since there are many versions and variants of many of the formats, and there are in many cases no formal easily processable specifications of the contents, basing testing on this knowledge would require too much human time. On the other hand, purely random changes can be applied to sample files, or purely random data can be used, to blindly test the behaviour of programs. This approach generally requires too much computer time. The GENOME approach does not require manual modeling of the tested protocol/file format, unlike the PROTOS Classic[2] approach of test suite development.

Most of the files in the test suite have been built using an intelligently automated combination of the approaches stated above. A set of valid files is first collected. A program is then used to analyse the structure of these files, yielding a rough model of the underlying file format. This model is then used to generate similar files, which often have modifications that would be extremely unlikely to appear, were one to use purely random methods. Because most of the testing and processing involved in building a test set is automatic, we were able to test a fairly large set of file formats.

The test suite can be used as robustness testing material for programs that process corresponding file formats. Usually programs should simply report that the files are invalid and resume operation in a controlled manner. For example program termination, altered behaviour and infinite loops indicate unintentional and in many cases exploitable errors.


Subject Survey

Freely available and evaluation versions of some common UNIX-based anti-virus products were selected as test subjects, and the common archive formats processed by the tools were selected for testing.

No sample list of implementations is presented herein. A large number of vendors include anti-virus or archive products in their product portfolios. A list of vendors with anti-virus products or archive products may include at least Alwil, Apple Computer, Avira, Cisco Systems, Comodo, Computer Associates, F-Secure, FRISK Software, Grisoft, Hewlett-Packard, IBM, ?McAfee, ?MicroWorld, Microsoft, Norman, Norton, Novatix, Panda Software, Pkware, Proland Software, RARLAB, Red Hat, Softwin, Sophos, Sun Microsoystems, Symantec, Trend Micro, Winzip, and many others.

The following image gives a faint approximation on the extent to which different archive formats can be used in computer systems. It represents a scenario in which two network peers, commonly a client and a server, communicate over a communication network. Potential archive format implementations involved are highlighted. Illustration on the scope of archive implementations.


  1. Network payload compression, implemented in hardware or software. Although compression per se was not targeted in this test suite, some compression might have been encapsulated in archive formats in this context. Note that software payload compression includes the compression used in many cryptographic message formats and the gzip content encoding in the prevalent MIME protocol.
  2. Network content filtering (spam, phishing and other undesired content) and virus scanning may need to handle archived content.
  3. Network caches, proxies and load balancing devices may parse archived payloads.
  4. Network firewalls (especially stateful/application level firewalls), intrusion detection/prevention systems may need to handle archived content.
  5. Client-side (or personal) firewalls, intrusion detection/prevention systems, content filtering, anti-virus, anti-malware, anti-spyware and anti-rootkit software may need to handle archived content.
  6. Different kind of client and server software handles archived content for various purposes. This includes the handling of archived configuration or customisation files (e.g. skins) and media files as some formats include data compression. Note that many programs include add-on plugins or modules that also may employ archive formats.
  7. APIs of operating systems and various libraries enable or involve the handling of archived content. Many environments also include indexing services that study filesystem content at regular intervals, and GUI functions designed for the handling of archives. Many programming languages handle archives containing library files and software packages. Many software packet installation management systems handle archived content.
  8. Connected embedded devices, most notably backup drives, may involve hardware or software archival functions.
  9. Connected palmtop and mobile appliances, which are often embedded devices, may require archival for communications or other functions. Note that the client and server systems depicted in this image may also be such devices.

Prior public vulnerabilities related to archive formats have been evident in most of the implementation categories listed above.

Injection and instrumentation methods

The injection vector survey, or delivery vector survey, analyses the different methods of delivering the test-cases to the implementations under test (IUT). Often, there are several methods of injection and one test-suite cannot cover them all, or might miss some vectors not available in all implementations.

Most anti-virus software focus on inspecting files that reside in a file system. As this test suite is focused on testing anti-virus software, it uses file system as the method of injection. Because all of the tested anti-virus tools and decompression tools could be run from command line, the injection could be handled by simple shell scripts. These scripts fed the test cases to the test subject one by one while monitoring their execution. The used injection scripts are not bundled with the test suite as they are very case-specific and easily reproducible for most subjects.

With instrumentation on the target platform we are able to monitor for undesired behaviour of the subject implementation. Typically this manifests as exceptions or signals such as 'access violation' or 'segmentation fault'. For most of the testing we used isolated Linux installations of the x86/IA32 architecture. In addition, sporadic testing was carried out with Mac OS X and Windows operating systems.

Strace and a kernel patch to report all fatal signals were used to monitor the operation of programs when the fuzzed files were processed. The value of eip register at the time of the fatal signal was used to rule some terminations as probably manifestations of the same error. The used instrumentation is not bundled with the test material as it is freely available via other sources for various platforms.

Production of Test Material

Computer programs usually process input. Often some of the input comes from a file. By using specially crafted files, it is often possible to expose un-handled and potentially exploitable errors in programs.

The test suite consists of modified archive files of corresponding file formats. Some of the files were generated using fairly simple content fuzzing techniques, and some were generated using a model-assisted approach. In both cases the files were generated using a set of sample files.

For each file format, set of valid files were first collected. The contents of the files are for the most part text documents and files of other common document formats. Freely available archival tools with different parameters were then used to create them. The collected files were processed with structure inference tools developed in GENOME to yield simple models of the content. The models were then used to generate similar data.

Fuzzed testing material was generated by applying probabilistic changes to the generated data. The fuzzing thus mostly involved selecting a good set of initial training material, and then finding reasonable parameters to produce suitably fuzzed data. The generated files are usually tested with a program as they are generated, and files causing interesting errors are collected.

This test suite contains both files that are known to cause problems in at least one program, and files that may or may not cause problems in some programs. In many cases files in this test suite expose severe un-handled errors, many of which have direct security implications and should be considered as vulnerabilities.

The structure inference and fuzzer tools used in the production of the test suite were provided by the GENOME project. The described automatic model assisted approach is new to our knowledge, and it has been very effective in producing various test input.

Test Suite Package

The tests are divided into separate test-material packages for each file formats. Each test-material package consists of a certain amount of test-cases, as specified in the table below. Number of test cases by archive format Archive format # cases

ace     91518
arj     255343
bz2     321818
cab     130823
gz      227311
lha     176631
rar     198865
tar     40549
zip     189833
zoo     163595
total   1632691

Package Information

The package is distributed as a cd-rom image containing:

The license allows free use and redistribution of the test material package. If you modify the material, please consider renaming the package.

In most Linux systems the iso image file can be used directly without burning it to a cd-rom, by issuing the following command:

$ mount -o loop testsuites.iso /cdrom

The cd-rom contains the test suites bundled by file format in tar.bz2 archives.

The archives can be decompressed with any decompression software supporting BZIP2 archives and having no limit for the number of files in one archive. In UNIX systems this can be done by issuing the following commands:

$ bunzip2 < suite.tar.bz2 | tar -xvf -

One suitable tool for Windows environment is ICEOWS, available at no cost. Note that each x.tar.bz2 package is first decompressed to a x.tar file, which is then similarly decompressed into a directory x containing the files. Note that OUSPG neither endorses any decompressor in particular, nor guarantee that they will not have issues with the test suite.

The decompression will take anything from a few minutes to several hours, depending on the computer. After decompression, the complete test suite contains 5.22GB of data, which on a typical Windows system occupies a bit over 10GB of physical space. Note that when using Windows, the test suite directories can be removed faster from the command line.

Testing with the test suite is carried out by feeding the files in the test suite to the desired subjects. Often this process can be automated to some degree, for example by scripts or batch processing. While testing, the test subject should be monitored for any unorderly behaviour, such as crashes, hangs or the overt consumption of system resources. This document does not cover the details of instrumentation, and we leave it up to users of the test material to come up with techniques to monitor whether test subjects handle the test cases in a satisfactory manner.

We recommend some additional guidelines for testing, although these are not imposed by the test material licence. These guidelines can be found from the Test suite releases in Theory and Practice document.


Use of latest release (highest number) is recommended. Older releases are provided for completeness and reproduction.

Release 1

Results from the Test Runs

Test-runs were conducted against the chosen set of sample implementations. The test material consisted of the fit test cases selected during the production of the test suite.

Test Result Definitions


In this test suite, the failed status is granted if any of the following criteria are met and a single test case can be identified to be responsible of it: a process or a child process crashes with fatal signal.


If no single test case can be identified but similar effects are observed, the status is inconclusive.


Otherwise, the status is passed.

Each failed test case represents at minimum a denial of service type chance of exploiting the found vulnerability. In most cases, they represent memory corruption, stack corruption or other fatal error conditions. Some of these may lead to exposure to typical exploits, allowing running of arbitrary code or modification of the target system (eg. buffer overflows).

Test Results by Archive Format

A limited subset of the test material was used in test runs against some anti-virus products. Tables below represent the observations from feeding the test-material against the chosen subject software. Product names of the actual subjects are omitted to protect the innocent.

These tables illustrate how different archive formats were handled in the test runs. A test group is marked failed if any single case or combination of cases cause the subject to fail. The results therefore represent a lower bound on implementation problems uncovered in tested software using the test material.

Result summary by archive format

Subject ace arj bz2 cab gz lha rar tar zip zoo
1 x x x x - x - - x x
2 - x n/a x - x x - - n/a
3 - x x x - x x - - -
4 - x - - - x x - x -
5 n/a n/a n/a - - n/a n/a - - n/a


Following table shows total number of failing cases found per format.

Subject ace arj bz2 cab gz lha rar tar zip zoo
1 283 8 7 2 - 44 - - 94 31
2 - 11 - 3552 - 10406 39 - - -
3 - 40 8 1 - 5 38 - - -
4 - 11 - - - 6 1603 - 2 -
5 - - - - - - - - - -

Following table shows number of unique bugs found per format. Value of EIP at the moment of crash was used to determine whether bug is unique or not. Unique bugs by archive format Subject

Subject ace arj bz2 cab gz lha rar tar zip zoo
1 3 2 1 1 - 3 - - 3 1
2 - 5 - 12 - 2 1 - - -
3 - 5 2 1 - 3 2 - - -
4 - 1 - - - 1 1 - 1 -
5 - - - - - - - - - -


Parser implementations are intricate pieces of code that are prone to implementation level faults, and archive file format parsers are no exception in this manner. Almost all of the tested tools seemed to be easy to crash using our relatively simple automated techniques. Some of the observed failures had information security implications, and should be considered as vulnerabilities. This is alarming considering the tested products were advertised as security products.


We are grateful to NISCC and CERT-FI for their help and advice during the vulnerability process.

Prior Public Vulnerabilities

At the outset of of this test suite, past implementation security issues regarding archive formats were investigated. This work included tracking archive format implementations, products, vulnerabilities, among other data. This data was gathered with the methods developed in project MATINE and visualised with Graphingwiki [3]. An example graph of CVE entries related to the RAR archive format is included below.

Graph of RAR-related CVE entries rar.png

Note that the above graph is in no way related to vulnerabilities possibly uncovered using this test material, it's just an automatically generated graph from CVE data.

Prior vulnerabilities, as reported in the CVE database, regarding the archive formats in this test suite, include but are not limited to the following:

The Vulnerability Process

During the prerelease phase all verified vulnerabilities were reported to the respective vendors through this test material. The vulnerability reports were tracked by CERT-FI and NISCC in the role of independent coordinators and advisors. An attempt was made to seek a channel to distribute the test material to vendors whose products we were not able to obtain for testing. Advisories and Vendor Statements

Vendor statements or security advisories issued in order to address the vulnerabilities uncovered by this test suite are collected. Advisories that we are aware of are listed here-in: