Our Product Customer Solutions About Us Contact Us

Features Benchmarks Applications FAQ

Benchmarks

The functionality and performance of WIDAS have been tested in two major benchmarks supervised by the Swiss Federal Statistical Office (SFSO)

These benchmarks were made several years ago. It must be noted, however, that no recent benchmarks indicate any substantial acceleration of DBMS access times that are not simply accreditable to improved hardware or disk cache management.

1999: WIDAS/3 Defeats ORACLE-8 and SAS in a Comprehensive Data Warehousing Benchmark

A select group of international software companies and organisations was recently invited by the Swiss Federal  Statistical Office (SFSO) to participate in a five day official benchmark. The competitors were required to load, organise, aggregate and extract data from 2.2 billion historical records relating to Swiss population, trade and real-estate rental statistics.

Scope of the Benchmark and Goals of the SFSO

The key goals of the benchmark were:

• To provide the SFSO with vital information about the functionality, usability and performance of software that will be required to build and maintain a Data Warehouse containing large volumes of detailed statistical data
• To verify the aggregation of detailed data into multidimensional data structuresTo ascertain the efficiency of retrievals from very large, detailed as well as aggregated sets of data

 

Five days time was allotted to each participant to complete the following tasks:

1. Install DBMS software and load central metadata. The latter consisted of 30 tables containing code descriptions of variables and the regionalizations of the Swiss territory
2. Set-up a relational database to hold individual detail records (Micro Database). Load raw data from ASCII flatfiles into the Micro DB. This data had a volume of 105 Gigabytes and was taken from the domains of population statistics, trade statistics and real-estate rental statistics
3. Perform ad-hoc queries on microdata. Execute data retrievals against the relational database. Present results in tabular form
4. Aggregate microdata (individual detail records) into various multidimensional datacubes. Load results (aggregated data) into a multidimensional database
5. Perform ad-hoc queries on aggregated macrodata. Retrieve data from the multidimensional database. Combine data from different statistical domains. Present results in tabular form

Benchmark Environment

The benchmark was run at the Compaq European Test Centre in Sophia-Antipolis on the French Riviera, in June 1999. The platform consisted of a Compaq ES40 system with 4 Alpha CPUs, 8 Gigabyte main memory and 1 Terabyte of RAID 5 disks. The operating system was Tru64-UNIX.

The SAS benchmark was run in September 1999 at the same location and under similar conditions.

Participants

ORACLE Corp. using ORACLE-8.1.5. for microdata handling, PL-SQL for microdata queries and ORACLE Express for the multidimensional macrodata

SAS Institute using its SAS Statistical Analysis System

MSI AG using WIDAS/3

SAS was able to run its tests 3 months after the other benchmark participants.

NCR / TERADATA and a European statistical office using SYBASE withdrew from the competition.

Results

The aggregation of detail data required in Step 3 of the benchmark involved the generation of a multidimensional database from a relational microdata database. It proved to be the most demanding and difficult task to accomplish. In fact WIDAS/3 was able to solve this key problem within the allotted time. ORACLE and SAS were not able to go beyond the first level of data aggregation and Consequently could not deliver results for most of the queries against aggregated data. At actual testing time, only 3 processors were available to ORACLE and WIDAS, whereas all 4 processors were operational for SAS.

Diskspace Occupancy in Gigabytes

	ORACLE	SAS	WIDAS
Relational Micro-Database (including microdata indices)	264	136	120
Multidimensional Macro Database	128 (a)	46(a)	30

(a) Estimated value; aggregation not completed

Task Processing Times in Minutes

	ORACLE	SAS	WIDAS
Load Metadata (Nomenclature)	180	45	30
Load Microdata into Relational DB	>785(b)	1380	440
Aggregate Microdata and Load Resulting icrodata into Multidimensional DB	>600(c)	>1580(c)	108

(b) Without generation of indices
(c) Covering only first level of aggregation

Microdata Query Processing Times in Seconds

	ORACLE	SAS	WIDAS
Sum for Queries 1 to 6	613(d)	370(e)	64
Sum for Queries 7 to 10	not done	332(e)	70
(Queries 8-10 were multiuser queries)

(d) For ORACLE, the queries were prepared and optimised by PL-SQL programming specialists. On the WIDAS side, all queries were executed ad-hoc
(e) For SAS, a server with 4 CPUs was made available. The ORACLE and WIDAS bechmarks were run on a 3-processor machine.

Macrodata Query Processing Times in Seconds 

	ORACLE	SAS	WIDAS
Sum for Queries 1 to 2	not done	86	2

Only few comparisons are available for the macrodata queries on the multidimensional database, since only WIDAS/3 was able to proceed beyond the first level of data aggregation. On this specific task, a benchmark was carried out in November 1997 between ORACLE-7.3 and WIDAS/3 on a Sun multiprocessor with 64 CPUs.

On that occasion, the WIDAS/3 software with its multidimensional data management capability turned out to be 40 -1000 times faster that the purely relational ORACLE-7.3 DBMS. ORACLE-Express was not available at that time on the Sun platform.

Please read the 1997 benchmark report below for further details.

Conclusions

WIDAS had much shorter set-up, load, aggregate and query times than either ORACLE-8 or SAS.

WIDAS occupied significantly less disk space and made a much more economical use of computer resources than its competitors.

WIDAS alone was able to perform all the benchmark tasks required by the SFSO. Our software also proved to be much easier to use than its competitors.

WIDAS proved that it had the necessary functionality to handle the entire range of complex data management problems that were set in the benchmark. (See CODAM Benchmark Summary Report of the SFSO, July 1999)

WIDAS excelled in performance because it makes full use of parallel processing architectures and advanced multithreading techniques.

The features that contributed most to the success of WIDAS were undoubtedly the strong integration of its data management, data analysis and reporting components, coupled with the lean software engineering principles that are fundamental to the WIDAS philosophy.

 

1997: Data Mart Benchmark against ORACLE

This benchmark, run for the Swiss Federal Statistical Office in November 1997 on a SunSparc E10000 machine with 64 processors and 48 GB of RAM against ORACLE, proved that WIDAS is between 41 and 1384 times faster than ORACLE in retrieving data from a multidimensional database. The results of the benchmark, which was supervised by the Swiss Federal Statistical Office, were as follows:

	ORACLE 7.3	WIDAS/3
Time to install and set-up the test database*	> 3 days	3 hours
Disk space occupied by test database	14.8 GB	2.5 GB
Benchmark queries that were successfully run	7 out of 12	12 out of 12
Response times per query	21 - 110 minutes	1 - 85 seconds

* Using Swiss trade statistics for 1989 as a baseline, participants in the benchmark were requested to generate additional data for a number n of successive years. The benchmark was run for n=5 years. However WIDAS was also successfully run for n=50 years.