Performance Baseline History
GIS user performance expectations have changed dramatically over the past 10 years. This change in user productivity is enabled primarily by faster platform performance and lower hardware costs.
Figure 9.1 identifies the hardware desktop platforms selected by GIS users as their performance baseline since the ARC/INFO 8.2 release in May 2000. Vendor published SPEC benchmarks are used to identify relative performance of each platform core release. Published SPEC rate integer baseline throughput values divided by number of platform core provides estimate of per core performance.
The vendor published SPECrate_int95 baseline results were used to identify relative performance of server platforms from 1993 through 1996.
In February 1997, ESRI released ARC/INFO 7.1.1. The first Windows ARC/INFO software was supported on the Intel Pentium Pro 200 MHz platform. The new Windows workstation environment supported GIS users with performance almost twice what was available the previous year on UNIX platforms and at a fraction of the cost.
In April 1998, ESRI released ARC/INFO 7.2.1. Intel was selling the Pentium II 300 MHz platform, which was 50 percent faster than the Pentium Pro 200. GIS users quickly moved to the new platform environment to improve productivity.
ARC/INFO application performance improved throughout this same period as the code was optimized with new ARC/INFO 7 incremental releases. A script developed with ARC/INFO 7.1.1 would run faster using the ARC/INFO 7.2.1 release, both running on the same platform. The change in the ARC/INFO performance baseline was not a software-driven requirement but a change in user performance expectations brought about by the faster and less expensive workstation and server technology.
In July 1999, ESRI provided the first release of ArcInfo 8 (ArcGIS Desktop software). The Intel Pentium III 500 MHz platform was the popular ArcInfo Workstation candidate. The Pentium III 500 MHz platform was more than 2.5 times faster than the Pentium Pro 200.
In May 2000, ESRI released ArcInfo 8.0.2. The Pentium III 733 MHz platform was selected as the performance baseline supporting the summer 2000 deployments. The Pentium III 733 MHz platform is more than 4.7 times faster than the Pentium Pro 200.
The vendor published SPECrate_int2000 baseline results were used to identify relative performance of server platforms from 2001 through 2006.
Modest performance improvements continued into 2001. Rapid performance enhancements were delayed because of vendor development of the next-generation processor technology. The Pentium III 900 MHz platform was selected as the performance baseline supporting the ArcInfo 8.1 release.
In June 2002, the Intel Xeon MP 1500 MHz server provided a significant performance gain. Intel had some performance problems with the 1500 MHz chips, and deployment was delayed until the fall. The Intel Xeon MP 1500 MHz platform was selected as the performance baseline for the ArcInfo 8.2 release.
In June 2003, the Intel Xeon MP 2000 MHz platform was the current technology Windows server. Performance issues for the Intel Xeon MP were resolved, and the 2000 MHz processors supported the current server platforms. This server provided an impressive gain over the previous year, matching performance of the Intel Xeon 2400 MHz workstation (almost 11 times faster than the Pentium Pro 200). Intel was releasing the Intel Xeon 3060 MHz workstation platforms before midyear and was planning to release a 2800 MHz server version in the fall. The Intel Xeon 2400 MHz platform was selected as the 2003 performance baseline.
In June 2004, the Intel Xeon MP 3000 MHz platform was the current technology Windows server. This server provided an impressive gain over the previous year, matching performance of the Intel Xeon 3200 MHz workstation. Intel was releasing the Intel Xeon 3600 MHz workstation platforms before midyear. The Intel Xeon 3200 MHz platform was selected as the 2004 performance baseline (more than 19 times faster than the Pentium Pro 200). Estimated SPECrate_int2000 baseline = 18/core.
In 2005, hardware performance improved very slightly because of CPU technology thermal limits. The Intel Xeon MP 3000 MHz platform continued to be the server of choice through summer 2005. This was the first time in more than 10 years we did not see a change in the GIS user platform performance baseline.
Hardware vendors started making some progress in 2006 and future performance gains looked promising. Dual-core processor technology was introduced doubling the capacity of commodity server platforms and reducing hardware cost. The Intel Xeon 4 CPU (2 dual core) 3773 MHz servers matched the performance of earlier Intel Xeon 2 CPU 3800 MHz servers and supported twice the compute capacity. The 3800 MHz servers (dual-core 3773 MHz servers) were selected as the 2006 GIS user performance baseline (SPECrate_int2000 baseline = 22/core).
SPEC benchmark translation
A new SPEC2006 release was published in 2006 providing the platform baseline metrics for ESRI system architecture design sizing models starting with the 2007 baseline (there is normally a 6–12 month overlap in testing and published results once SPEC introduces the new benchmarks). Test results for over 45 vendor platforms were published on the SPEC2000 and the SPEC2006 benchmark sites, and these test results were used to calculate a transfer function between the two benchmark sets for capacity planning purposes. The ratios of the published benchmark results (SPECrate_2000/SPECrate_2006) were plotted on a graph to identify a mean value for the transfer function. Figure 9-6 provides the results of this analysis.
A published SPECrate_int2000 benchmark divided by 2.1 will provide an estimate for the equivalent SPECrate_int2006 benchmark value. A published SPECrate_int2006 benchmark multiplied by 2.1 will provide a translated estimate for the equivalent SPECrate_int2000 benchmark value. It is interesting to notice that the older platform technology has better benchmark values on the SPEC2006 benchmarks than the newer technology (the conversion ratio for the new Intel Xeon 51xx series platforms is over 2.2 while the conversion factor for the older Intel Pentium D platforms are under 2.0).
The vendor published SPECrate_int2000 baseline results were used to identify relative performance of server platforms from 2007 through 2010.
Advanced Micro Devices (AMD) is a chip development company that competes with Intel, and their platforms grew in popularity as customers experienced growing concerns about the high power consumption and heat generated by the high MHz Intel servers. AMD was able to provide a lower MHz platform solution that matched the performance of the higher MHz Intel processor platforms. Hewlett-Packard favored the cooler temperature and lower power consumption of the new AMD platform and promoted the AMD platforms as its server technology choice when it introduced the new HP Blade servers. Dell decided to start selling AMD server technology as an alternative to Intel. By midyear, Intel released a new Intel Xeon 3000 MHz commodity server with dual-core sockets, 1333 MHz front side bus, and 4 MB L2 cache that performed 40 percent faster than the Intel Xeon 3800s.
The dual core processor technology challenged the continued use of the acronym CPU as the abbreviation for central processing units, and vendors introduced a new set of names for identifying the core processing units used to enable compute intensive platform performance and capacity. A new naming convention was adopted for this document in 2007 replacing the use of CPU by core, and introducing the use of socket to refer to the processor (socket was the term used in ESRI software pricing models). Thus the new Intel Xeon 3000 MHz platform with dual core processors was referenced as the Intel Xeon 4 core (2 socket) 3000 MHz platform.
The Intel Xeon 4 core (2 socket) 3000 MHz platform with the 4 MB L2 cache was selected as the 2007 performance baseline. This platform was introduced by hardware vendors in mid 2006. Since that time vendors have released different configurations of this same chip technology, ranging from 1600 MHz to 3000 MHz dual core socket and 1600 MHz – 2666 MHz quad core socket configurations.
There was a broad mix of hardware configurations provided by vendors in CY2007, some designed for lower cost and throughput and others for performance, making it challenging to identify the specific hardware configuration that would best meet customer performance needs. Vendors started to include the processor model number with each platform to more clearly identify the specific chip configuration. The 2007 baseline was more specifically identified as the Xeon 5160 4 core (2 socket) 3000 MHz platform (SPECrate_int2006 baseline = 14/core).
Platform performance continued to improve in CY2008, and quad-core platforms were growing in popularity. ESRI software pricing was based on number of processor core, so the term for processor was changed to chip. The Xeon 5260 4 core (2 chip) 3333 MHz platform was selected as the CY2008 performance baseline (SPECrate_int2006 baseline = 17.5/core).
Intel introduced a new chip design in CY2009 that was 70 percent faster than the previous year. Dual core chip technology was no longer an option, replaced by the new Quad core chip technology for workstations and servers. The Xeon X5570 8 core (2 chip) 2933 MHz platform was selected as the CY2009 performance baseline (SPECrate_int2006 baseline = 30/core).
Intel introduced a new chip design in CY2010 that was 15-20 percent faster than the previous year. Virtual server technology is being adopted as the framework for many IT data centers. Private and public Cloud hosting is becoming more popular. Hardware vendors are building higher capacity platforms focusing on the virtual server and cloud computing markets. Intel introduced new 6 core per chip 5600 series platforms and a range of higher capacity 7500 series platforms with 6 and 8 core per chip configurations. Intel released a Xeon X7550 32 core (4 chip) 2000 MHz platform that was about 40 percent slower per core than the X5677 platform and about 2.5 times the capacity. SGI released an Intel Xeon X7560 512 core (64 chip) 2266 MHz platform with per core performance the same as the Xeon X7550 and about 40 times the capacity of the X5677 platform. The Xeon X5677 8 core (2 chip) 3467 MHz platform was selected as the CY2010 performance baseline (SPECrate_int2006 baseline = 35/core).