Server Software Performance
From Wiki.GIS.com
Spring 2012 GIS Server Software Performance 31st Edition (Under Construction)
This section shares lessons learned about deploying effective ArcGIS Server configurations that satisfy operational performance and scalability needs. Server parameters must be configured properly to take full advantage of server resources and satisfy system capacity needs.
Proper configuration and tuning can improve display performance and leverage available server resources. Guidelines are provided for establishing proper service configuration settings for optimum server deployment.
Contents |
[edit] ArcGIS Server Terminology and Tuning
Figure 4-1 defines the instances, processes and threads within the ArcGIS Server configuration. Instances, processes, and threads identify what software will be deployed to leverage the available hardware processing resources.
An ArcGIS Server install instance refers to a complete ArcGIS Server software installation. A single site can have multiple ArcGIS Server install instances managed by different server object managers.
An ArcGIS Server service instance refers to the individual service threads deployed to support a specific ArcGIS Server service configuration. Service instances and threads are used within the software interchangeably. A service instance is a SOC process thread deployed by the SOM available to execute a service request. In general terms, instances or threads represent a single service request and how the request will be serviced through the software stack.
The term “thread” is also used at the hardware level to represent an on-chip location to park a service instance close to the core for processing (a multi-threaded core can reduce the time required for context switching when multiple concurrent execution threads are sharing a single core processor; objective is to better utilize available core processing cycles).
The term “Process” refers to the package of program executables contained within each deployed SOC (each deployed SOC contains one copy of the ArcObjects program executables). A single multi-threaded SOC process can host multiple service instances.
[edit] Selecting high isolation or low isolation
For example, a SOM deploying 12 service instances using high isolation would be launching 12 separate SOC executables each providing one instance thread. A SOM deploying the same 12 service instances using low isolation (4 instances per SOC process) would be launching 3 separate SOC processes, each with four (4) service instance threads. The low-isolation SOC configuration requires less host machine memory, but if one service instance (thread) fails, the SOC process will fail along with its remaining instances. When a high-isolation service instance fails, the SOC executable failure is isolated to loss of a single service instance.
[edit] Selecting a pooled or nonpooled service model
Figure 4-3 identifies SOC process pooling settings. The Map Service Properties pooling tab is used to define whether the SOC process is pooled or non-pooled.
The pooled service model is the best selection for most service configurations. The current-state information (extent, layer visibility, etc.) is maintained by the Web application or with the client browser. The deployed service instances are shared with inbound users, released after each service transaction for another client assignment. The pooled service model scales much better than the non-pooled model because of this shared object pool.
The way the non-pooled service executes is similar to an ArcGIS Desktop session on a Windows Terminal Server. Both use the same ArcObject executable functions; both are restricted to one user able to take advantage of the service instance. ArcGIS Desktop on Windows Terminal Server would perform best for most implementations. The ArcGIS Server deployment (40-50 concurrent user sessions on a 4 core server platform) would be a less expensive implementation due to the difference in software licensing costs.
The non-pooled service model should only be used when an application’s function requires it. A nonpooled SOC is assigned to a single-user session and holds its reference to the client application for the duration of the user session. In this case, the current state information (extent, layer visibility, etc.) can be maintained by the SOC. Non-pooled SOC will not be supported beyond the ArcGIS 10 release (editing functions will be supported by pooled feature services in ArcGIS 10.1).
[edit] Configuring SOM service instances
Figure 3-30 also shows the user interface for configuring the SOM service instances. The service instance parameters are identified on the pooling tab within ArcGIS Server Map Service Properties. Use the pooled service whenever possible. For optimum user performance and system capacity, a single pooled service can handle many concurrent clients, while access to a single non-pooled service is limited to a single user session. The minimum number of instances would normally be “1” to avoid startup delays if requested by a single user. If the service is seldom used, this setting could be “0.” The maximum number of instances establishes the maximum system capacity the SOM can expose to this service configuration. It’s worth repeating: configuring service instances beyond the maximum platform capacity will not increase system throughput, but it may very well reduce user display performance.
[edit] Map Service configuration
For a popular service, the maximum instances number should be large enough to take advantage of the full site capacity. Full site capacity could support 3-5 service instances per core. Two 4-core container machines could handle up to 32 concurrent service instances—the maximum recommended instance setting for a popular map service.
The blue line in Figure 4-4 shows the maximum host platform utilization and system throughput in displays per minute (DPM) for a series of ArcGIS Server service configuration instance settings responding to random Web service requests. The bars show host platform service time (colored tier) and service queue times (processing queue times result from random arrival of service requests). The host platform has 4 core; the four core processors are shared resources used to execute the deployed service instances.
The first bar represents a service configuration with two (2) service instances. Peak service load is limited to 36 percent host platform utilization with peak throughput of 224 DPM and display response time just over 0.5 seconds, well below the maximum host platform throughput capacity. The seventh bar represents a service configuration with fourteen (14) service instances. Host platform is over 90 percent utilization with a display response time of around 1.5 seconds. Average display response time continues to increase as additional service instances are deployed with minimum increase in throughput. Peak throughput is normally reached at about 3 to 5 service instances per host platform core. Increasing the number of service instances will only increase the average display response time with minimal throughput gain.
There are some challenges if you limit host capacity settings to 16 instances. If several services are competing for peak service loads at the same time, the SOM may need to stop and start SOC processes within host capacity setting limits in response to the changing concurrent peak service requests. Extra SOC startup processing overhead will increase host platform processing loads and reduce peak throughput levels – this may not provide optimum throughput response. Ideally it would be good to avoid SOC startups while trying to service peak throughput loads.
In this example, the host capacity could be set at a level based on a reasonable peak display response time rather than peak throughput levels. Display response time with twenty (20) service instances is around 2 seconds. We can expect display response time to increase linearly above this point, so 40 instances would render 4 second display response times with limited throughput gain. If 4 second response time is acceptable, we could set host capacity at 40 instances and provide the SOM more flexibility to address changing service request levels during peak throughput loads.
The best solution is to provide enough hardware to avoid these high utilization load profiles. There is no simple configuration prescription for optimizing host platform throughput and client response times as the server approaches maximum utilization levels.
[edit] Batch process service configuration
A batch process can be any service that runs for a long time without user input. Batch processes include geoprocessing, map cache generation, system backup, map production script, replication services, or any other calculation that may take longer than 60 seconds to complete without user input. These type of services should be initiated from the client application as a work request, and delivered to a queue for sequential processing by a batch process service.
For handling a heavy batch service, the maximum instances should be a small number to protect site resources. A single, heavy batch service can consume a server core for the length of the service process time. Four concurrent batch requests on a 4-core server can consume all available host processing resources.
The blue line in Figure 4-5 shows the maximum host platform utilization and system throughput in displays per minute (DPM) for a series of ArcGIS Server batch process service configuration instance settings. The bars show host platform service time (colored tier) and service wait times (wait times are due to shared use of the available core processor). The host platform has 4 core; the four core processors are shared resources used to execute the deployed service instances.
The first bar represents a batch process configuration with one (1) service instances. Peak service load is limited to 22 percent host platform utilization with peak throughput of 137 DPM, well below the maximum host platform throughput capacity (display service time is normally not important for batch process loads – more attention is given to how long the total batch job will run). The fifth bar represents a service configuration with five (5) service instances. Host platform reaches 100 percent utilization with minimum increase in batch run time. Display response time (including total batch service run time) will increase linearly once server is operating at 100 percent utilization. A service configuration with ten (10) service instances would take twice as long to complete each batch job. Peak throughput is normally reached at N+1 service instances (host platform core + 1). Increasing the number of service instances will only increase batch processing times – it is better to queue up processes and complete jobs sequentially that try to run them all at the same time.
Batch process workflow can be configured in the Capacity Planning Tool by setting the workflow minimum think time = 0. Batch workflows can be used on the CPT Design tab to reserve system resources for batch processing during heavy loads. You will need to use the RESET ADJUST function to calculate productivity loads. Batch process functionality (RESET ADJUST function) is not included on the CPT Calculator tab.
[edit] Configuring host platform capacity
Implementing the right software configuration can improve user display performance and increase system throughput capacity. The right number of service instances to assign (to enable maximum throughput) will vary slightly, based on the type of service (reference heavy batch process and popular map service examples above). If all the service execution is supported on the host container platform, a single instance per core may be an optimum capacity setting (heavy batch process). For most map services, however, the execution is distributed across the Web Server, container machine, and database server, which shares the processing load across multiple platforms. Additional delays can occur due to random instruction arrival times (queuing theory, in chapter 9), which can account for over 50 percent of the total processing time when approaching peak throughput loads. The test results in figure 3-35 show the optimum capacity setting at 3-5 service instances per core. In both the tool and testing, the performance fundamentals presented in chapter 9 have been applied to find the right instance capacity. ESRI capacity recommendations are based on the results of performance tests and our basic understanding of performance fundamentals, backed up by customer operational experience.
Figure 4-6 shows the user interface for establishing the host capacity settings. The host capacity setting restricts the number of maximum SOC instances each SOM can deploy on the assigned host container machines. The recommended capacity setting—or the maximum number of instances the parent SOM should deploy on each of the host container machines—is 3-5 service instances per core. A standard commodity Windows server has a total of 4 processor cores, so the server will approach peak throughput at around 16 concurrent instances. A good rule of thumb is to set host capacity at 2 to 3 times the number of instances required to reach full capacity (optimum settings will depend on your specific service distribution).
A standard, high-available, 3-tier configuration for ArcGIS Server could have 2 Web servers (each with a SOM), 3 host container machines, and a database server. If each of the host container machines was a standard 4-core Windows server, the optimum capacity would be 32 service instances on each host machine. In a high-available configuration, the SOMs are not cluster-aware, so separate configurations would be allocated for the shared environment, with a separate capacity setting needed for each parent SOM.
It is very easy to inadvertently allow too many SOC instances. With the same ArcGIS Server configuration identified above, full capacity of the host machine tier (3 servers, each with 4 cores) would approach maximum throughput with 48 active service instances (16 on each host machine). If all the host capacity settings were 32 (not necessarily the right answer with both Web servers SOM operational), the fully operational configuration would allow a total of 192 concurrent service instances (96 per SOM, each SOM would deploy 32 on each host machine for a total of 192).
There are two potential performance problems when deploying too many service instances per host machine. When server loads approach peak throughput loads, an optimum capacity configuration would minimize response time by completing work in process (first in, first out) before assigning more requests (concurrent arrivals would not be assigned for processing until the current work is completed). When you deploy too many service instances (more than the number of core processors can service at the same time), the available CPUs must share their time to process all the assigned service requests simultaneously—and they do this in a very fair way. All assigned service requests are completed at about same time—all must wait until all processing is complete for them to be done. Too high a capacity setting would result in increased user response times during peak system loads due to a high number of deployed service instances executed by the limited number of processor cores. Service response time increases linearly when peak capacity loads are reached with minimum increase in server throughput levels.
[edit] Service configuration and optimum capacity
Figure 4-7 includes a handful of terms and configuration strategies – there is no simple recipe for getting it right. It all depends on your user environment – how popular will your services be and how will people access your site. The overall goal is to configuration the server for the maximum number of requests with the minimum amount of processing overhead.
ArcGIS Server is made up of two types of software components, Server Object Manager (SOM) and Server Object Containers (SOC). The SOM manages the SOCs based on instructions provided with each service configuration. Services are what you publish on ArcGIS Server. Web applications consume these services to produce the client display. The SOC process threads are executed by the hardware core processors.
Each published service will have a service configuration file. The service configuration file will contain the parameters you identify when publishing each service configuration. How you configure your services will determine how they will be deployed by the SOM. Service configuration files are created using the ArcGIS Manager or ArcCatalog administrator tools.
There are some key things to keep in mind when configuring server. The SOM will deploy SOC instances within the bounds established by the Service Configuration MIN/MAX instance settings and the assigned Host Capacity settings. The minimum service instances for all services will be deployed during SOM startup – this establishes the minimum number of instances that will be distributed equally across the assigned host platforms. During peak concurrent loads, the SOM can increase the number of SOC instances up to the maximum service levels required to support concurrent inbound service requests. The maximum number of total instances deployed by the SOM cannot exceed the identified host platform capacity. For multiple SOM sharing a common host platform, each SOM will be able to deploy to the maximum assigned host capacity.
Keep in mind, there is extra platform processing overhead required every time the SOM has to start a new SOC process. Ideally, you would like to deploy just the right amount of service instances so there is one available for immediate SOM assignment for each client request. During peak server loads, you want to have just the right number of maximum service instances identified to fully utilize available host platform compute resources staying well within available platform memory limitations. During maximum peak loads, you want the host capacity setting to limit concurrent processing loads to allow optimum service throughput. For high performance services (processing time less than 1 second) you may want to configure host capacity at two or three times the number of instances required for peak system throughput.
During installation, a server object container agent is installed on each host machine. This provides the ArcObjects code needed to support SOC deployment. During startup, the SOM deploys the minimum number of service instances for each server configuration, distributing the instances evenly across the assigned host container machines. Service instances are deployed in SOC processes.
During operations, if concurrent requests for a specific service exceed the number of deployed service instances, the SOM will increase the number of deployed service instances to handle peak request rates up to the maximum value allowed for in the service configuration. If the inbound concurrent requests for that service exceed the maximum deployed instances you’ve set in the service configuration, requests will wait in the service queue until an existing service instance is available for service assignment. If necessary, service instances for less popular service configurations will be reduced (closed) to make room for the popular service. Non-active services can be reduced down to the minimum instances specified in their service configuration file.
Deployment algorithms within the SOM provide even distribution of service instances across the assigned host platforms. The deployment algorithms along with the service queue work to balance the ArcGIS Server processing load across the host container machines. The SOM should be used as the final load balance solution for the host container machine tier.
Understanding what each of these performance parameters do and how they should be configured to satisfy your specific service needs is important for optimum utilization of your deployed services. Getting this right will take some thinking and some careful planning. Once you deploy your services, you will need to monitor to see if your settings are working. Modifications can be applied to optimize your specific service loads. Remember, the goal is to configure the system to minimize processing overhead (excessive SOC process startup during peak service loads is overhead you would like to avoid). Also, a sufficient number of service instances must be deployed to consume available host platform compute resources and enable the host platform to service peak throughput levels.
[edit] Selecting the right physical memory
The platform configuration must include sufficient physical memory to accommodate all active process executions.
Sufficient number of instances must be deployed to take advantage of processor core capacity. If the ArcSOC process memory requirements exceed available physical memory, performance will start to degrade and at some point start to fail. Setting host capacity can limit the number of SOC instances deployed to make sure memory is sufficient to support peak deployed instance memory requirements.
[edit] Building the data cache
The cached map service would consist of a pyramid of pre-processed data imagery or vector data, starting at a single map resolution at the highest layer and breaking each image into four images at twice the map scale for each additional layer included in the pyramid.
The chart above shows the estimated processing hours starting with one tile at the top layer and building the required number of layers to reach the largest map scale. Three map generation times are plotted - 0.5 seconds for a simple map display, 1 second for a medium map display, and 5 seconds for a heavy map display. Charts shows about 100 hours to generate 9 layers with average map generation time of 5 seconds. It would take over 2000 hours to generate just two more layers (11 layers at 5 sec per map tile). This chart is generated simply by multiplying the time to generate one map tile by the total number of tiles required to complete the map cache pyramid.
The recommended procedure for estimating map cache generation times is to select a same area dataset that represents a small percentage of the total caching area. Build a map cache of the sample area and test the output symbology, labeling, and performance with your primary map client. Use the cache time for the small sample area to estimate processing time for the remaining map cache.
The ArcGIS 9.3.1 release includes partial data cache and cache on demand options for building and maintaining map cache making this technology adaptive to a broad range of operational scenarios.
Partial Data Cache. A partial data cache can be defined for high priority areas within your display environment. The partial data cache can be specified by both area and level of resolution, providing optimum flexibility for pre-processing custom areas and levels of map cache.
Cache on Demand. The initial map query can be generated from a dynamic data source, and the resulting tiles can be added to the map cache. The next request for these tiles would come from the map cache, significantly improving display performance for popular map display environments.
Figure 4-11 shows a view of the ArcGIS Server Map Service Properties Caching tab.
When using a local image file data source, each map cache instance will consume a platform core. The optimum service configuration would specify 5 instances to make sure the server is operating at 100 percent capacity. The windows performance task monitor can be used to identify the machine is operating at 100 percent utilization. If your map cache include a DBMS data source, you may be able to include one or two additional service instances to reach 100 percent utilization. It is important to configure sufficient instances to take full advantage of your hardware platform processing resources.
It is a best practice to execute longer cache jobs in sections. It is good to plan cache areas that can be completed within an 8 or 10 hour period, providing an opportunity to complete the jobs in a stable platform environment. This will also provide an opportunity to periodically reboot system between each job section to maintain a reliable platform environment.
Figure 4-12 provides an example of taking advantage of the hardware, as described above.
A variety of caching scenarios are being evaluated to expand the feasibility of using pre-processed data cache to improve performance and scalability of future GIS operations. Experience shows pre-processing map data can make a difference for our customers. ArcGIS Server is leading the way toward more productive and higher quality Enterprise level GIS operations.
[edit] Selecting the right technology: A case study
Selecting the right software technology can make a big difference in performance and scalability, and cost of the production system.
Our customer had a requirement to design a Web application solution that would be used to collect national property location and census information during a three month national citizen declaration period. Citizens would report to regional government centers, use a local desktop computer to locate their home residence on a map display generated from a national imagery and geospatial features repository. Citizen would place a point on the map identifying their residence, and then fill out a reference table identifying their census information. The citizen input would be consolidated at a centralized national data center and shared with all regional government centers throughout the declaration process.
Figure 4-14 provides an overview of the user locations and central data center architecture. The initial system design was developed using an earlier ArcGIS Server Web application development framework (ADF) map editor, hosting a centralized ArcGIS Server dynamic Web application with browser clients located at 60 regional national sites. Following contract award, the customer reviewed available technology options to finalize the system design.
The ArcGIS Server dynamic Web ADF application was the solution provided in the initial design proposal two years earlier. The current ArcGIS Server technology included improvements in Web application performance and user experience that would be evaluated in the design review.
ArcGIS Server provides a data cache option where reference map layers could be pre-processed and stored in a map cache pyramid file data source. Pre-processing the reference layers would significantly reduce server processing loads during production operations. A single point declaration layer contained all features that would be edited and exchanged during the citizen declaration period, all remaining reference layers could be cached. Changes would be displayed at all remote site locations with each client display refresh.
There were three different Web application options that could leverage a reference data cache.
1) ArcGIS Server Web ADF application with a centralized map cache service.
2) ArcGIS Server REST service, leveraging a Rich Internet Application (RIA) Flash or Silverlight Map Editor application with a centralized reference map cache service.
3) ArcGIS Mobile application, leveraging a Mobile client application with a local reference map cache data source. Point changes to the declaration layer would be exchanged using a centralized ArcGIS Server mobile synchronization service.
The Esri Capacity Planning Calculator was used to evaluate the architecture for the four different workflow technology patterns identified above. Peak system loads were estimated at 2400 concurrent users with standard Web productivity of 6 displays per minute. System design results are provided in the following paragraphs.
[edit] Dynamic Web ADF Application
An ArcGIS Server 10 ADF MXD light 100% Dynamic standard Esri workflow was used to generate hardware requirements and traffic loads required to represent the dynamic web application solution.
Peak central data center traffic loads were estimated to reach 480 Mbps, well beyond the bandwidth available with the current data center Wide Area Network (WAN) service connection. Larger regional office sites (50 concurrent users) would require 24 Mbps bandwidth and smaller regional office sites (10 concurrent users) 6 Mbps WAN connections to support the projected peak citizen declaration traffic loads. Major infrastructure bandwidth increases would be needed to handle projected traffic flow requirements.
The central hardware solution was supported by Intel Xeon X5677 8 core (2 chip) 3467 MHz Windows 64-bit Servers each with 24 GB memory (data server required 60 GB memory). Total of 4 servers were required for the Web Application Server tier, 7 servers for the Container Machine tier, and one SDE geodatabase server.
[edit] Cached Web ADF Application
A custom ArcGIS Server ADF MXD light 10% Dynamic mashup with Cached Reference layer basemap service was used to support the cached web ADF workflow analysis.
Peak central data center traffic load estimates dropped to 252 Mbps with larger regional office sites requiring 12 Mbps WAN connections. Smaller offices would be configured with 3 Mbps WAN connections. The Web tier remained at 4 servers to support the centrally hosted ADF Map Viewer Web applications. The ArcGIS Server Container Machine tier reduced to 1 server and the DBMS load was reduced to 6.2 percent utilization. The Cached data source provided a significant server cost reduction from the initial proposal.
A sample data set was used to evaluate basemap caching timelines, and the complete country reference map cache could be generated within one week of processing time. Pre-caching the base reference layers would be well worth the effort, since there would be no need to update or change the reference cache during the peak citizen declaration period (data would be static).
[edit] Cached Rich Internet Client Application (REST API)
A custom ArcGIS Server 10 REST MXD light 10% Dynamic mashup with Cached Reference layer basemap service was used to support the cached web REST workflow analysis.
Peak central data center traffic load estimates were the same as the previous ADF cached workflow. The Web load was significantly reduced since the Map Viewer application was supported by the RIA browser clients. Web and server object containers were supported on two (2) servers in a two tier architecture. The database load remained the same. The lighter REST server architecture reduced hosting environment by 4 servers.
[edit] ArcGIS Mobile Application
The fourth design option was to use the ArcGIS Mobile application with a local reference cache data source. A demo of the ArcGIS Mobile client was provided on a Windows desktop platform to demonstrate feasibility of supporting the required editing functions with this client technology. The ArcGIS Mobile client technology operates very well on a standard Windows display environment and performed all the functions needed to support the citizen declaration requirements.
The ArcGIS Mobile standard Esri workflow synchronization service was used to support the design analysis. This workflow was generated by the CPT Calculator using a SOAP MXD Light service with a feature output (display features streamed to the client application). A 95 percent data cache setting was used to represent traffic for point feature exchanges (only point changes would be exchanged between the client and server displays). Cached reference layers would be distributed to each regional site in advance, and access would be provided by a file share to the ArcGIS Mobile clients running on the local workstations. The ArcGIS Mobile client would synchronize point changes to the dynamic citizen declaration layer over the government WAN. The peak concurrent SOAP service load would be reduced to 600 concurrent users, representing 25 percent of the total client displays (point changes are made only during edit transactions).
User display performance would be very fast, supported by the local reference map cache and the point layer in the ArcGIS Mobile application cache.
Peak central data center traffic loads were reduced to 15 Mbps, well within the T3 (45 Mbps) bandwidth available with the current data center Wide Area Network (WAN) service connection. Large regional office sites peak traffic was 1.3 Mbps, which would function well within 3 Mbps WAN connections. Small regional offices site peak traffic was 0.25 Mbps supported well by available T-1 connections. The existing infrastructure would be able to support peak WAN traffic loads with guaranteed service to each of the remote desktop locations (ArcGIS Mobile client would continue to function as a standalone system if WAN communication were lost, and edits would be sent to the central server when communication was restored). The central hardware requirements were reduced to 2 composite Web/container machine servers and the data server load was minimal (less than 1 percent utilization)
It was very clear that the cached client application provided significant cost and performance benefits over the centralized Web application dynamic solution included in the initial proposal.
[edit] Software performance summary
Experience suggests we can do a better job configuring ArcGIS Server deployments. Proper server configurations can reduce implementation risk and save customer time and money. Projects can be delivered within project cost, time, and satisfy performance budgets.
The next section will take a closer look GIS Data Administration.
[edit] Previous Editions
Software Performance 30th Edition
Software Performance 29th Edition
Software Performance 28th Edition
Software Performance 27th Edition
Page Footer
Specific license terms for this content
System Design Strategies 26th edition - An Esri ® Technical Reference Document • 2009 (final PDF release)
del.icio.us
facebook
reddit
twitter
