Server Software Performance

Spring 2019 GIS Server Software Performance 44th Edition

ArcGIS Server ia a component of the ArcGIS Enterprise software license. ArcGIS Enterprise provides a variety of ArcGIS Server capabilities, supported by a fully integrated services architecture including portal content management, ArcGIS Server processing capabilities, and a variety of optimized data source configurations. This chapter will focus on the ArcGIS Server component configuration strategies for optimum system performance and throughput capacity.

ArcGIS Server performance and capacity depends on a proper service configuration. Once you have a basic understanding of software performance, you are ready to better understand how to properly configure services for optimum server utilization.

The previous chapter on software performance introduced the concepts required to recognize the primary software components and data structure that contributes to system performance. This chapter looks more closely at the specific decisions map service authors and system administrations must make in properly configuring services for optimum capacity and throughput.

ArcGIS Enterprise platform selection and licensing assumes you understand how to select the proper service configurations for optimum throughput and performance. Selecting the proper hardware and licensing establishes the potential for optimum system performance; proper service configuration is essential to realize this potential. This chapter provides you with the guidelines and best practices you need to achieve optimum performance from your ArcGIS Server system deployment.

After this chapter, you will be ready to explore how network communications, platform architecture, and server platform selection complete the system design components required for implementation success.

ArcGIS Enterprise software architecture
Figure 4.1 shows the ArcGIS Enterprise architecture components.
 * ArcGIS Server provides the layers, services, and horsepower required to power your Web GIS. ArcGIS Server includes capabilities for traditional Standard and Advanced capabilities with additional licensing for a variety of integrated ArcGIS Server roles.
 * Portal provides the web interface and infrastructure content that supports a user’s interaction and overall experience with your Web GIS.
 * ArcGIS Data Store provides the Esri managed data repository that stores the spatial content that has been shared to Portal. Configurations include the Relational, Tile Cache, and Spatiotemporal Data Stores.  ArcGIS Enterprise also integrates with traditional data sources (Enterprise Geodatabase, tabular business systems, etc).
 * Web Adaptor is an Esri built software load balancer that appropriately directs network traffic and serves as a reverse proxy for Web GIS access.

"Note: Further discussion on ArcGIS Enterprise architecture is provided in Chapter 9: GIS product architecture." 

ArcGIS Server site configurations
The standard ArcGIS Server site supports multiple machines deployed within a single site configuration, with each machine supporting a common set of published service configurations. ArcGIS 10.5 retains a legacy option for multiple clusters within a single site, with each cluster of machines able to support a different set of published service configurations. Multiple-cluster site configurations include an internal cluster aware load balancing service.

ArcGIS Server component functions
Figure 4.2 provides an overview of the key web components used to define the ArcGIS Server architecture.

ArcGIS Server key component functions.
 * Web gateway: Web server software installed with the web adaptor.
 * Web adaptor: Provides site-aware load distribution, reverse proxy, and failover.
 * GIS server: Provides ArcGIS software for service computing.
 * ConfigStore: File share that stores site service configuration parameters.
 * SvrDirectories: File share that stores site directories.
 * Data source: Common data source for site services.

"Warning: If you lose the ConfigStore or data source, you lose the site. ConfigStore backup and recovery was introduced with the 10.2 release. "

ArcGIS Server site capabilities can be modified for improved performance and scalability. 

Read-only mode (introduced with the ArcGIS 10.4 release)
Figure 4.3 shows a view of the ArcGIS Server read-only architecture. ArcGIS Server read-only mode improves site stability by reducing dependency on connection to the ConfigStore network share.
 * ConfigStore cached on each local GIS Server machine.
 * Service publishing and changes are not supported.
 * Allows read operations in degraded mode.

Operates in degraded mode with lost connection to file share
 * Geoprocessing services will not work.
 * Consumption of cached services will not work unless they are highly available.
 * Exporting a map or image service using a URL is not possible.

Optimum configuration for static read-only publishing 
 * Improved display performance.
 * Improved system stability.

Read-only mode selection
The ArcGIS Server Manager is used to enable the read-only site mode configuration. Figure 4.4 shows the read-only mode selection. ArcGIS Server Site Administrator can enable the read-only mode on the ArcGIS Server Manager site settings tab.

Beginning at 10.4, ArcGIS Server provides a read-only mode that allows you to control changes to the site. Enabling this mode prohibits the publishing of new services and blocks most administrative operations. Your existing services function as they previously did. For example, you can still edit data in feature services when the site is in read-only mode. 

ArcGIS Server Terminology
Instances, processes, and threads determine what software will be deployed to leverage the available hardware processing resources. Figure 4.5 shows the instances, processes and threads within the ArcGIS Server configuration.

ArcGIS Server Site
An ArcGIS Server site refers to one or more GIS Servers with a common configstore.
 * ArcGIS Server licensing requirements are established by the highest service instance capability deployed within the site.
 * Licensing is based on customer number and computer datacenter location
 * Single ArcGIS Server site can be configured to support multiple machine clusters.
 * All machines within a cluster have the same service configurations.

Server process
A server process refers to a software server executable.
 * ArcGIS GIS Server (SOC.exe) process
 * Web server process
 * Web Adaptor process

Process threads
The term “Process” refers to the package of program executables contained within each deployed SOC (each deployed SOC contains one copy of the ArcGIS program executables). Some server processes are multi-threaded (low isolation), and allow multiple service requests to be processed at the same time using a single copy of process executables.
 * ArcSOC.exe processes can be configured to support multiple service threads.

The term “thread” is used at the hardware level to represent an on-chip location to park a service instance close to the core for processing (a hyper-threaded core can reduce the time required for context switching when multiple concurrent execution threads are sharing a single core processor; objective is to better utilize available core processing cycles).

Service instance
Service instance is a process thread made available for servicing a map request. The service configuration identifies the minimum and maximum number of service instances that will be deployed by ArcGIS Server to satisfy inbound web service requests.
 * Minimum number of specified service instances for each service will be deployed during server startup.
 * Additional service instances will be deployed based on the number of concurrent service request arrivals, and can expand the number of service instances up to the maximum specified for each service instance configuration.

Service instance configuration
Each published service has a defined service instance configuration which represents how that service will be deployed through the software stack. 
 * Each GIS Server machine within the site will deploy the same service configurations.
 * ArcGIS Server Site is defined by a common (shared) Configuration Store (ConfigStore). The ConfigStore is a file that contains the service configurations for the ArcGIS Server site.

Batch process service instance configuration (geoprocessing services)
The brown line in Figure 4.6 shows the maximum host platform utilization and system throughput in displays per minute (DPM) for a series of ArcGIS Server batch process service configuration instance settings. The bars show host platform service time (colored tier) and service wait times (white wait times are due to shared use of the available core processor). The host platform has 4 core; the four core processors are shared resources used to execute the deployed service instances.

The first bar represents a batch process configuration with one (1) service instances. Peak service load is roughly 25 percent host platform utilization with peak throughput of 190 DPM, well below the maximum host platform throughput capacity (display service time is normally not important for batch process loads – more attention is given to how long the total batch job will run).

The fifth bar represents a service configuration with five (5) service instances. Host platform approaches 100 percent utilization with minimum increase in batch run time. Display response time (including total batch service run time) will increase linearly once server is operating over 99 percent utilization. A service configuration with ten (10) service instances would take twice as long to complete each batch job. Peak throughput is normally reached at N+1 service instances (host platform core + 1). Increasing the number of service instances will only increase batch processing times – it is better to queue up processes and complete jobs sequentially than try to run them all at the same time.

Batch process minimum and maximum instance settings are different than map services. Batch processes consume a server processor core.
 * Process queue provides consistent processing load.
 * Minimum number of service instances should be assigned to optimize processing performance.

Service configuration min/max instance settings impact server performance.
 * More service instances provide more throughput when available instances are less than available processor cores.
 * More service instances increase batch runtime when available instances are more than available processor cores.

"Best Practice: Set map service configuration minimum and maximum instance settings to optimize site performance. If you need to work on the site cluster during batch geoprocessing, you may want to configure less than N+1 instances to allow for an administrative processing reserve."

Optimum service instance configuration:
 * Minimum instance = Zero (0) for long running batch processes, One (1) for lighter network services.
 * ArcSOC process start-up time is often short compared to overall batch load.
 * Reserves memory and capacity space for other active map services.
 * Service instances can grow up to the max instance setting during peak loads.
 * Maximum instance = Provide one more batch instance than available server machine cores (N+1 instances where N = number of server cores).
 * Provides optimum throughput during peak service loads.
 * Minimizes batch runtime performance during peak loads.
 * Limiting maximum number of service instances can be used to limit platform core processor utilization.
 * Single service instance will use less than one core processor during maximum service loads.

Many ArcGIS services perform as batch processes.
 * Most heavy geoprocessing services
 * Batch address locator services
 * Map caching services
 * Map printing services
 * Heavy network routing services
 * Imagery processing services

A batch process can be any service that runs for a long time without user input. Batch processes include geoprocessing, map cache generation, system backup, map production script, replication services, or any other calculation that may take longer than 60 seconds to complete without user input. This type of service should be initiated from the client application as a work request, and delivered to a queue for sequential processing by a batch process service.

"Best Practice: Batch processing can be the most efficient way to handle heavy processing loads. "

"Warning: For handling a heavy batch service, the maximum instances should be a small number to protect site resources. A single, heavy batch service can consume a server core for the length of the service process time. Four concurrent batch requests on a 4-core server can consume all available host processing resources. "

Batch process workflow can be configured in the Capacity Planning Tool by setting the workflow minimum think time = 0. Batch workflows can be used on the CPT Design tab to reserve system resources for batch processing during heavy loads. You will need to use the RESET ADJUST function to calculate batch service productivity for the maximum server load. 

CPT Calculator and Design tabs used to demonstrate the optimum number of batch process service instances.
 * CPT ArcGIS Server batch process instance configuration

Web service instance configuration (concurrent users)
The brown line in Figure 4.7 shows the maximum host platform utilization and system throughput in displays per minute (DPM) for a series of ArcGIS Server service configuration instance settings responding to random Web service requests. The bars show GIS Server machine service time (colored tier) and service queue times (white processing queue times due to random arrival of service requests). The server machine has 4-core; the four core processors are shared resources used to execute the deployed service instances.

The first bar represents a service configuration with two (2) service instances. Peak service load is limited to less than 38 percent host platform utilization with peak throughput of 287 DPM and display response time just over 0.40 seconds, well below the maximum host platform throughput capacity. The eighth bar represents a service configuration with sixteen (16) service instances. Host platform is over 92 percent utilization with a display response time over 1.35 seconds. Average display response time continues to increase as additional service instances are deployed with minimum increase in throughput. Peak throughput is normally reached at about 3 to 5 service instances per host platform core (12 to 20 service instances on a 4-core host platform). Increasing the number of service instances beyond this level will only increase the average display response time with minimal throughput gain.

Random arrival distribution reduces peak throughput per service instance.
 * Peaks and valleys in the arrival distribution place varying demands on server processing loads.
 * Sufficient number of instances must be available for service assignment to achieve peak throughput values.

Service configuration min/max instance settings impact server performance.
 * During light server loads, more service instances can increase server peak throughput.
 * During heavy server loads, more service instances have minimum impact on peak throughput.
 * More service instances increase display response time.

For a popular service, the maximum service instances should be large enough to take advantage of the full site capacity. Full site capacity may require 3-5 service instances per core. A site with 4-core GIS Server machines would require a maximum service instance setting of 16—the recommended instance setting for a popular map service.

"Best Practice: Set map service configuration minimum and maximum instance settings to optimize site performance. "

Optimum Web mapping service instance configuration.
 * Minimum instance = One, for most popular map service configurations.
 * Provides rapid service response for first user access.
 * Reserves memory and capacity space for other active map services.
 * Service instances can grow up to the maximum instance setting during peak loads.
 * Maximum instance = Three to five instances per server machine core.
 * Provides optimum throughput during peak service loads.
 * Optimizes display response time performance during peak loads.


 * CPT Design map service instance configuration

CPT Design tab can be used to demonstrate the optimum number of Map service instances.

Map Service instance configuration strategies
Minimum and maximum service instances are identified when publishing a map service. It is important to identify the proper instance configuration for each map service deployment. Proper service instance configurations depend on the expected peak service demands and the server machine processor core configuration. 

Pooled service model (service instance settings)
Figure 4.8 identifies ArcGIS Server ArcSOC process pooling options. Process pooling settings are configured when publishing the service.

Pooled ArcSOC processes handle service requests for multiple concurrent client sessions.
 * Pooled services can be supported by low or high isolation ArcSOC processes.
 * Pooled services provide optimum server utilization and service throughput.

The pooled service model supports all service configurations. The current-state information (extent, layer visibility, etc.) is maintained by the Web or client browser application. The deployed service instances are shared with inbound users and released after each service transaction for another client assignment.

"Best Practice: Proper min/max service instance settings are critical to optimizing service performance and maximum service throughput. " 

ArcGIS Server pooling settings (service instance min/max settings)
Figure 4.9 ArcGIS Service Editor pooled service instance settings. The service instance parameters are identified under Service Editor Pooling.
 * The minimum number of instances would normally be “1” to avoid SOC startup delays when the service is requested by a single user. If the service is seldom used, this setting could be “0”, in which case there would be a SOC startup delay with the first service request.
 * The maximum number of instances setting establishes the maximum system capacity the GIS Server can expose to this service configuration. It’s worth repeating: configuring service instances beyond the maximum platform capacity will not increase system throughput, but it may very well reduce user display performance and system stability.

Minimum and maximum service instance setting:
 * Establishes parameters for GIS Server service instance management.
 * - Minimum service instances are deployed during GIS Server startup and after idle instance timeout.
 * - Maximum service instances establish limit for GIS Server service instance deployment.

Timeouts (reference Figure 4.9):
 * Service instance (use)
 * Maximum time a client can use a service. Establishes timeout for hung service instances.


 * Service queue (wait)
 * Maximum time a client will wait to get a service. Establishes maximum service queue time.


 * Service request (idle)
 * Maximum time an idle instance can be kept running. Reduces number of deployed service instances to minimum setting when request for this service has not been received for this amount of time.



ArcGIS Server process configuration
Figure 4.10 shows the available ArcGIS Server SOC process configurations. There are two types of SOC executable configurations, high isolation and low isolation. Process isolation settings are configured when publishing the service.

The isolation setting determines how the server manages the ArcSOC processes.
 * Each SOC process represents a unique published map service.
 * High isolation deploys single-threaded ArcSOC processes (one service instance per process).
 * Low isolation deploys multi-threaded ArcSOC processes (up to 256 service instances per process).
 * Each thread (service instance) follows a pointer within the ArcSOC process tracking execution of the assigned service request (all requests share the same copy of the executables).


 * High isolation provides more stable operations.
 * Each process contains only one service instance.
 * If the service instance fails, only one process fails.


 * Low isolation provides more efficient instance capacity adjustments.
 * Single process start-up deploys multiple service instances.
 * Several service instances share the same process memory.

For example, a GIS Server deploying 12 service instances using high isolation would be launching 12 separate ArcSOC processes each providing one service instance thread. A GIS Server deploying the same 12 service instances using low isolation (4 instances per ArcSOC process) would launch 3 separate ArcSOC processes, each with four (4) service instance threads.

The low-isolation SOC configuration requires less host machine memory; but if one service instance (thread) fails, the SOC process will fail along with its remaining instances. When a high-isolation service instance fails, the SOC executable failure is isolated to loss of a single service instance.

"Best Practice: High isolation processes provide the most stable solution for standard mapping services. " 

ArcGIS Service Editor processes settings
Figure 4.11 shows the ArcGIS Service Editor processes settings. During the map publishing process, the author will select the SOC process isolation, identify the maximum number of instances per ArcSOC process (low isolation only), identify recycle start times, and set the keep alive connection cycles.

Select SOC isolation.
 * High isolation deploys single-threaded ArcSOC processes (one service instance per ArcSOC process)
 * Low isolation enables multi-threaded ArcSOC processes (multiple service instances per ArcSOC process)

Set maximum instances per ArcSOC process.
 * Low isolation only, can support up to 256 instances per ArcSOC process.
 * ArcSOC instances are managed by the GIS Server, based on service configuration settings.

"Warning: Low isolation GIS Server configurations with several instances per SOC can result in unstable server performance "

Recycle time schedules GIS Server service instance recycle operations.
 * Cycles through and sequentially stops and restarts each ArcSOC instance.
 * Schedule during off-peak hours (batch process load sequentially restarting each instance).

Keep alive connections. 
 * Checks and repairs data connections for idle instances. Set inspection schedule if you identify problems maintaining stable data connections.

ArcGIS Server cacheControlMaxAge property
ArcGIS Server map service responses include an entity tag (ETag) and Cache-Control header. The ETag header value is a unique identifier of the response. The Cache-Control header has a max-age value that provides information to the browser regarding the maximum time period for which it can reuse a response from the browser's cache. This value is controlled by the cacheControlMaxAge property.

Add the cacheControlMaxAge property to the ArcGIS Server service properties in order to take advantage of the browser side caching. Specify the value (in seconds) for the property to optimize performance.

For map services with frequently changing datasets or symbology, a value such as 5 minutes (300 seconds) is recommended. In applications with animations or time-aware data, consider increasing the value to give the application a smoother animation experience. Refer to the How the cacheControlMaxAge property is used help document for details on how to add the cacheControlMaxAge property. 

Work with server logs
Monitoring server logs helps you identify errors, troubleshoot problems, and address issues with your site. You'll use ArcGIS Server Manager to view, query, configure, and delete portal logs. Manager aggregates the log messages into a table you can read and manipulate.

When you install or upgrade ArcGIS Server, the server's log level is set to Verbose. This means that very detailed messages are logged during the installation or upgrade process. After the installation or upgrade process completes, the server's log level is set to Warning by default.

Often the "Debug" level is left on or forgotten after troubleshooting. Collecting excessive server log files can cause a decrease in server performance.

"Best Practice: Make sure after troubleshooting the log level is set to Warning in a production environment." 

Cached map service
ArcGIS Server provides map cache services for use as a data source for Web applications and ArcGIS Desktop or custom desktop clients developed with the ArcGIS Engine or ArcGIS Runtime SDKs. ArcGIS Server can also stream pre-processed map cache images to ArcGIS Engine and ArcGIS Desktop with 3D Analysis client for 3D visualization. In both cases, once downloaded, the cached images are stored at the client for high performance display.

ArcGIS supports on-the-fly translation of 2D map cache to 3D images with both map cache and cached Imagery base maps. For optimum 3D performance, tiles can be delivered from a 3D Globe cache.

Figure 4.12 provides an overview of the cached map service image structure. ArcGIS Server includes automated geoprocessing services to build and maintain (pre-process) optimized map cache pyramids.

"Best Practice: Cached tiles enable a highly scalable static map service. "

The cached map service would consist of a pyramid of pre-processed data imagery or vector data, starting at a single map resolution at the highest layer and breaking each image into four images at twice the map scale for each additional layer included in the pyramid.

Tiles are pre-rendered at fixed scales.
 * Use common ArcGIS Online, Google, and ArcGIS Online world map scales for optimum data integration and mash-up performance.

Rapid display of static base maps.
 * Use the map cache to serve static basemap layers for the best display performance.

Richer symbols and more information available with high-capacity publishing capability. 
 * Create high-quality basemaps. High-quality map cache tiles display as fast as simple tiles.

Map service caching configuration
Figure 4.13 shows a view of the ArcGIS Service Editor Caching settings. A Portland dataset is selected and the display shows the suggested tiling scheme. A tool is included that calculates an estimate of the cache size. You can choose to publish a dynamic or cached service. You can have the cache creation start automatically once you complete the configuration, or save the configuration and start the caching job manually at a later time.

Draw this map service:
 * Dynamically from the data (do not cache the source data)
 * Using tiles from a cache (use a cached tile service)

Caching settings
 * Suggesting tile scheme is provided as default. You can modify default setting to identify minimum and maximum map scale levels that you want cached.
 * Estimated cache size is calculated based on the scale selection

Identify caching method
 * Build cache automatically (CachingTools geoprocessing service is used to generate cached tiles when publishing the service)
 * Build cache manually (Manage Map Services Cache tools can be used to configure your own caching instances to generate the map cache)

"Warning: Map cache service instances will consume CPU resources.  Do not use the ArcGIS Server production site for manually building your cache "

"Best Practice: Use preconfigured CachingTool instances when generating map cache. Batch processing loads for maximum CachingTool service instance configuration must be included in your capacity planning analysis to ensure adequate server resources during peak loads. "

Advanced settings
Figure 4.14 shows three options available for map service caching.
 * Generate full cache of available data set. (This will generate a map cache based on your caching specifications for the entire dataset based on your minimum and maximum scale selections)

"Warning: Map cache processing instances will consume CachingTools service instances while generating the cache.  If all published CachingTools instances are busy, caching job will wait in queue until service instances are available. "


 * Generate partial cache of available data set. (This allows you to select an area of interest for caching. This can be defined by a polygon layer or buffer that shows the areas on the map you wish to cache.)

"Best Practice: Partial cache can reduce overall caching time. For most implementations, service requests access a relatively small area of the total dataset. "


 * Generate cache on the fly. (This will generate a map cache for an area including the display extent with the first service request and add the tiles to your map cache. Subsequent service requests for the same extent will use the cached map tiles).  Published map service instances are used to generate the on-demand cached tiles while servicing the map request.

"Best Practice: Cache on demand should only be used for generating cache on sparsely populated areas (map request for that area of the map would be a rare event). "

Additional Advanced caching settings
The Service Editor Caching Advanced Settings tab includes an additional Advanced Cache Settings tab (Figure 4.14) that is important to review. This tab includes three very important performance settings:


 * Storage Format setting identifies how the cached tiles will be stored. The COMPACT setting stores the cached tiles in bundles for reduced storage volume and improved file migration performance.  A single bundle can store up to 16,384 (128 x 128) tiles.
 * Allow clients to cache tiles locally.  This setting is a very important performance consideration for remote clients with limited bandwidth.  An initial map display can require over 10 times the traffic to download cached tiles over the network.  Once delivered to the client, the tiles can be stored in browser cache and local tiles can be used for subsequent displays.  If you do not allow clients to cache tiles locally, all tiles will be downloaded from the server for each display contributing to network saturation and poor display performance.

"Best Practice: Map cache clients should periodically clear their local browser cache to ensure use of the latest tile updates available on the server. "


 * Allow clients to export cache tiles. This setting allows clients to export tiles to their local machine for low bandwidth or mobile operations (use when not connected to the server).  Exporting tiles to the local machine prior to working in the field can significantly improve display performance over low bandwidth connections.

Generating the map cache
ArcGIS Server creates cache tiles using a geoprocessing service named CachingTools. This service is configured for you in the System folder when you create the ArcGIS Server site. The number of instances you allow for the CachingTools service determines how much power your machine can dedicate toward caching jobs.

Estimated map cache generation time
Client access to the cached data would deliver tiles that correspond to the requested map scale. Tiles would be blended together as a single reference layer by the client application. Total pre-processing time would depend on the total number of tiles in the map cache and the average map service time for rendering each map tile image. Figure 4.15 can be used to get a rough estimate of the expected map cache generate time.

Map cache generation time is a function of the number of tiles and average tile generation time.
 * Chart shows one tile at the top layer (multiply result by number of tiles in top layer)
 * The number of tiles increases by a factor of four with each additional layer.
 * Tile render time can vary from less than one second to several seconds, depending on the average tile map complexity.
 * Processing hours based on single CachingTools service instance (divide result by total concurrent service instances used when generating the cache).

Rendering time increases exponentially with each tile layer. With average 0.5 tile render time, we expect the following generation times:
 * One (1) hour for seven layers
 * Five (5) hours for eight layers

Estimating map cache generation time.
 * Build a small area to test the output symbology, labeling, and performance criteria for your primary map client.
 * Execute cache jobs in sections to manage production time.

"Warning: The caching process can be time consuming. ''" 

Cache processing profile
Figure 4.16 provides an example of taking advantage of the hardware, as described above. ArcGIS Server will use the CachingTools maximum available service instances to process the map cache as quickly as possible with the available hardware.

Cache processing timelines can be improved by increasing the number of concurrent processor cores utilized in the caching process. Recommended CachingTools service instance configuration is N+1, where N=number of physical server core processors. If you want to view caching status and manage cache jobs while executing the cache, maximum number of instances = physical server process core leaves some processing resources for managing caching services.

Testing was performed on identical 4-core server platforms.
 * Single service instance (thread) on a single 4-core server took 500 hours.
 * Five (5) service instances (threads) on a 4-core server took 125 hours (four times faster).
 * 10 service instances on two 4-core servers took 65 hours (eight times faster).

"'Best Practice: Take advantage of available hardware resources" 

If you have the service instances configured properly, you should be able to take full advantage of the available hardware processing capabilities. CPU usage should peak at 100 percent when running with N+1 service instances with hyperthreading disabled.

"Warning: ArcGIS Server GIS Servers are cluster-aware, and common service instance configurations apply for each GIS server machine within a site cluster. If you establish a maximum service configuration of 5 instances, then 5 service instances will be deployed for each GIS Server machine within the Site. "

This video on [ArcGIS 10.2.1 Map Caching in ArcGIS Server] provides a demo showing the new workflows and features of map caching since the ArcGIS 10.2.1 for Server release. <br style="clear: both" />

Manage Services caching tools


Manage services caching instances are configured during site installation to enable background caching services.
 * CachingControllers: Establish number of concurrent caching jobs per server machine supported by the CachingTools instances.
 * CachingTools: Establish number of caching service instances per server machine available for caching job processing.
 * You should always run the CachingTools service and the CachingControllers service in the same cluster.

Service Editor service configurations apply to every GIS Server machine within the assigned cluster.

"Best Practice: Managed Services caching tools should be configured in a separate ArcGIS Server Site cluster from production Web services."

Production servers within the GIS Server Site can be reassigned to the Map Caching cluster to expand caching capacity during off-peak hours. <br style="clear: both" />

System Caching Pooling configurations
A dedicated CachingControllers instance is required for each caching job. Pooling and Processes settings must be configured for the CachingControllers service. CachingTools will be configured to optimize utilization of available platform hardware resources during peak cache processing loads.

Figure 4.18 shows the CachingTools processes configuration. An identical tab will be configured for the CachingControllers processes configuration.

CachingControllers Pooling settings

 * Min service instances = 0. There is no need to have instances running if there are no assigned caching jobs.  CachingController instance will be running to manage each active caching job.
 * Max service instance setting identifies the max number of concurrent caching jobs. If all available instances are busy, the job will be held in the process queue until a CachingControllers instance is available.
 * Timeouts should match the CachingTools settings below.

CachingTools Pooling settings
Figure 4.18 shows the Pooling settings for the CachingTools service. The recommended service instance setting for caching services is N+1 (number of available core on the server machine plus one). Maximum setting of 3 service instances would be appropriate for 2-core GIS Server machines.

The CachingControllers manage caching jobs to take advantage of all available CachingTool service instances. For a single caching job, the CachingController will assign multiple caching processes to leverage all available CachingTool service instances (up to the total number of bundles in the caching configuration). Multiple concurrent jobs will share available CachingTool service instances. Assignment of multiple concurrent caching jobs (multiple CachingControllers) will optimize utilization of available server processing resources.

Timeout settings should be appropriate to the caching service.
 * Maximum time a client can use a service = 360000 seconds (provide sufficient time to complete the caching job). CachingController process will be dedicated for the complete caching job.  CachingTools instances will be reassigned after completing each bundled cache assignment.
 * Maximum time a client can wait to use a service = The caching service is the client within this configuration, and the caching service wait time is hard coded to around 30 days (use default setting).
 * Maximum time an idle instance can be kept running = 180 seconds (this can be set for a short time, allowing instances to be shut down if not in use).

"Best Practice: Think carefully what works best for your Site's caching jobs and select appropriate service instance configurations and timeout settings to optimize utilization of your server environment." <br style="clear: both" />

System Caching Processes configurations
Figure 4.19 shows the CachingController processes configuration. An identical tab will be configured for the CachingTools processes configuration.

Map cache generation is an intense batch process which can often run for hours and days at a time often generating hundreds of thousands of cached map tiles. It is important to support this type of service with the most stable process configuration. For this reason, both the CachingController and CachingTools SOC processes should be configured in high isolation mode.

Recycling the processes can ensure stable instances during the map cache build, promoting optimum performance during cache generation. Caching jobs are processed in bundles of 16,384 tiles. CachingTools instances can be recycled between processing bundles once during each recycle process. CachingControllers and CachingTools processes configurations should be scheduled for a recycle every 24 hours to promote optimum site stability and performance. CachingController instances will recycle between caching jobs. CachingTools instances can recycle during job processing between bundle caching assignments. <br style="clear: both" />

GIS Server machine memory configuration
The platform configuration must include sufficient physical memory to accommodate all active process executions. Systems that do not have sufficient memory will experience performance degradation during peak load conditions (processing overhead increases due to operating system swapping executables between disk and memory). If memory is not sufficient to support concurrent server processing requirements, system will start to experience random process failures (process executables dropped from memory during execution). Figure 4.20 shows memory performance considerations for an ArcGIS Server host machine deployment.

"Best Practice: Sufficient platform physical memory is critical to successful stable operations. More memory can improve performance."

Active software processes must be in physical memory for successful execution. If sufficient memory is not available:
 * Inactive processes will be swapped to memory cache to make room for active processes.
 * Active processes will crash if swapped from memory during execution.

Too many instances per server can exhaust memory.
 * Increased paging when not enough memory.
 * Slower processing due to shared compute resources.

Optimum total active service instance assignments can vary based on service popularity. Limiting maximum of 10 instances per core is reasonable for most map service scenarios.
 * Provides sufficient overhead for server instance management.
 * Avoids excessive starting and stopping ArcSOC processes during peak throughput loads.

Too few instances per server:
 * Can limit utilization of host hardware.
 * Minimum of three instances per core is reasonable.

"Best Practice: Provide sufficient memory to support optimum performance."

ArcGIS Server memory recommendations
ArcGIS platform memory recommendations
 * Minimum of 4 GB memory per core recommended.
 * Additional 1 GB memory per host platform core recommended for virtual server host platforms.
 * More memory may be required when using large file data sources (imagery).
 * More memory may improve throughput performance.

The operating system will use extra memory for data cache, which for some workflows can improve throughput performance. <br style="clear: both" />

Managing host platform service instance capacity
There are a handful of GIS Server terms and configuration variables – there is no simple recipe for getting it right. It all depends on your user environment – how popular will your services be and how will people access your site. The overall goal is to configure GIS Server to handle the maximum number of requests with the minimum amount of processing overhead. A summary of the service configuration and optimum capacity discussion for ArcGIS 10.0 for Server, along with a nice summary diagram, is available in Chapter 4: Software Performance 30th Edition.

ArcGIS Server manages the internal ArcSOC service instance deployment based on instructions provided with each service configuration. Services are what you publish on the GIS Server. Web applications consume these services to produce the client display. The service requests are sent to ArcSOC process threads (service instances) that are executed by the hardware core processors.

Each published service will have a service description file. The service description file will contain the parameters you identify when publishing each service description. How you configure your services will determine how they will be deployed by the GIS Server. Service description files are created from ArcGIS Desktop (ArcMap) during the map publishing process.

There are some key things to keep in mind when publishing services. The GIS Server will deploy SOC service instances within the bounds established by the service description MIN/MAX instance settings. The minimum service instances for all services will be deployed during GIS Server startup – this establishes the minimum number of instances that will be deployed within each GIS Server machine. During peak concurrent loads, the GIS Server can increase the number of SOC instances up to the maximum service description levels to support concurrent inbound service requests.

Keep in mind, there is extra platform processing overhead required every time the GIS Server has to start a new SOC process. Ideally, you would like to deploy just the right amount of service instances so there is one available for immediate GIS Server assignment for each client request. During peak server loads, you want to have just the right number of maximum service instances identified to fully utilize available host platform compute resources staying well within available platform memory limitations. During maximum peak loads, you want to limit concurrent processing loads to allow optimum service throughput. For high performance services (processing time less than 1 second) you may want to allow the total of several active services to be two or three times the number of instances required for peak system throughput.

During installation, GIS Server is installed on each host machine. This provides the ArcGIS code needed to support SOC deployment. During startup, the GIS Server deploys the minimum number of service instances for each service description within each GIS Server machine. Service instances are deployed in SOC.exe processes.

During operations, if concurrent requests for a specific service exceed the number of deployed service instances, the GIS Server will increase the number of deployed service instances to handle peak request rates up to the maximum value allowed for in the service description. If the inbound concurrent requests for that service exceed the maximum deployed instances you’ve set in the service description, requests will wait in the service queue until an existing service instance is available for service assignment. Non-active services can be reduced down to the minimum instances specified in their service description file based on their service instance timeout settings (max idle time).

Deployment algorithms within the GIS Server provide even distribution of service instances across the assigned host platforms. The deployment algorithms along with the service queue work to balance the GIS Server processing load across machines assigned to the same GIS Server Site. The GIS Server will be used as the final load balance solution for the machines within the GIS Server site.

Understanding what each of these performance parameters do and how they should be configured to satisfy your specific service needs is important for optimum utilization of your host servers. Getting this right will take some thinking and some careful planning. Once you deploy your services, you will need to monitor the service logs to see if your settings are working. Modifications can be applied to optimize your specific service loads. A sufficient number of service instances must be deployed to consume available host platform compute resources and enable the host platform to service peak throughput loads. Sufficient memory must be available to handle the maximum number of active service instances or the system can become unstable. Remember, the goal is to configure the system to minimize processing overhead during peak throughput periods (excessive SOC process start-up loads during peak service periods is overhead you would like to avoid).

Additional ArcGIS platform memory configuration guidelines are provided in the appendix on Windows Memory Management. <br style="clear: both" />

Selecting the right technology: A case study
Selecting the right software technology can make a big difference in performance and scalability, and cost of the production system. The following case study shares an experience with a real customer implementation which clearly represents the value of selecting the right software technology.

Greek citizen case study background
The customer had a requirement to design a web application solution that would be used to collect national property location and census information during a three-month national citizen declaration period. The Figure 4.21 shows the area of Greece involved in the census.

Citizens would report to local regional government centers and use a local desktop computer to locate their home residence on a map display generated from a national imagery and geospatial feature repository. The citizen would place a point on the map identifying their residence, and then fill out a reference table identifying their census information.

The citizen input would be consolidated at a centralized national data center and shared with all regional government centers throughout the declaration process. <br style="clear: both" />

User requirements for web mapping solution
Figure 4.22 provides an overview of the national architecture. The initial system design was developed using ArcGIS Server feature service with an imagery tile cache with browser clients located at 60 regional national sites. Following contract award, the customer reviewed available technology options to finalize the system design.

Peak web service use requirements <br style="clear: both" />
 * 2400 concurrent client edit sessions.
 * 75 percent map query to find home location
 * 25 percent simple edits (select point and complete attribute table)
 * 60 remote user locations, one central national data center.
 * Large site: 50 concurrent clients
 * Small site: 10 concurrent clients

Web mapping services architecture patterns.
Figure 4.23 shows the ArcGIS Server architecture patterns that were considered for the Greek citizen declaration solution.


 * Initial hardware proposal

The following workflow was used to generate system loads for the initial hardware proposal.
 * Web REST map services with central dynamic enterprise geodatabase data layers and imagery tile cache.
 * CPT Workflow: AGS REST 2D V Lite 100%Dyn 13x7 Feature +$$
 * System implementation design review (after grant approval) 

After some time, the European Union approved the Greek Citizen Declaration grant based on the initial hardware proposal. The Greek cadastral team traveled to Esri to review available technology options for final implementation.

The following web mapping services architecture patterns were reviewed to identify optimum deployment scenario.
 * Web REST map feature service with central dynamic SDE data layers and Imagery cache
 * CPT workflow: AGS REST 2D V Lite 100%Dyn 13x7 Feature +$$


 * Web map service with point feature layer + central map cache
 * CPT Workflow: AGS REST 2D V Lite 10%Dyn 13x7 Feature +$$


 * Web mobile application + local map cache
 * CPT Workflow: AGS SOAP 2D V Lite 5%Dyn 13x7 Feature

"Best Practice: Significant technology improvements have become available since the initial proposal. It is always good to update the final solution architecture based on current technology before final implementation. "

Web map feature service with central dynamic SDE data layers and imagery tile cache
CPT Calculator analysis shows platform solution when supporting business needs using 100 percent dynamic web map feature service and remote client browser technology.

Hardware solution:
 * 4 Xeon Gold 5122 4 core (1 chip) 3600 MHz servers
 * ArcGIS Server licensing for up to 8 cores

Total estimated technology cost
 * Hardware cost = $38,567
 * Software licensing = $24,000/year

Peak network traffic estimates
 * 36 Mbps for large sites, recommend 90 Mbps bandwidth
 * 7.2 Mbps for small sites, recommend 18 Mbps bandwidth
 * Estimated network cost of $8,177/month.

Web map single point feature service layer with central map cache
CPT Calculator Analysis shows a Web map single point feature service layer with central map cache.

ArcGIS Server provides a data cache option where reference map layers could be pre-processed and stored in a map cache pyramid file data source. Pre-processing the reference layers would significantly reduce server processing loads during production operations. A single point declaration layer contained all features that would be edited and exchanged during the citizen declaration period; all remaining reference layers could be cached. Changes would be displayed at all remote site locations with each client display refresh.

Hardware solution:
 * 2 Xeon Gold 5122 4 core (1 chip) 3600 MHz servers
 * ArcGIS Server licensing for up to 4 cores

Total estimated technology cost
 * Hardware cost = $20,117
 * Software licensing = $12,000/year

Peak network traffic estimates
 * 5.9 Mbps for large sites, recommend 12 Mbps bandwidth
 * 1.2 Mbps for small sites, recommend 3 Mbps bandwidth
 * Estimated network cost of $1,901/month.

Web mobile application with edit feature synchronization + local map cache
CPT Calculator analysis. The third design option was to use the ArcGIS Mobile application with a local reference cache data source. A demo of the ArcGIS Mobile client was provided on a Windows desktop platform to demonstrate feasibility of supporting the required editing functions with this client technology. The ArcGIS Mobile client technology operates very well on a standard Windows display environment and performed all the functions needed to support the citizen declaration requirements.

Hardware solution:
 * 1 Xeon Gold 5122 4-core (1 chip) 3600 MHz servers
 * ArcGIS Server licensing for up to 4 cores

Total estimated technology cost:
 * Hardware cost = $9,225
 * Software licensing = $12,000/year

Peak network traffic estimates: <br style="clear: both" />
 * 1.7 Mbps for large sites, recommend 3 Mbps bandwidth
 * 0.33 Mbps for small sites, recommend 1.5 Mbps bandwidth
 * Estimated network cost of $563/month

Caching advantage summary
It was very clear that the cached client application provided significant cost and performance benefits over the centralized Web application dynamic solution included in the initial proposal. Pre-processing of map reference layers as an optimized map cache pyramid can significantly improve display performance. Use of an intelligent desktop client that can access reference layers from a local map cache can minimize network traffic and improve display performance even more. Selecting the right technology can make a big difference in total system cost and user productivity. Figure 4.24 highlights the advantage of selecting the right technical solution. <br style="clear: both" />

"'Best Practice: Selecting the right technology solution can make a big difference in price and user performance."

Business analysis identifies clear advantages in the ArcGIS Mobile solution.
 * Over $27,000 savings in hardware cost alone.
 * Significant additional savings on software licensing and reduced infrastructure network bandwidth costs.

Server Software performance summary
Experience suggests we can do a better job configuring ArcGIS Server deployments. Proper server configurations can reduce implementation risk and save customer time and money. Projects can be delivered within project cost, time, and satisfy performance budgets. <br style="clear: both" />

=CPT Capacity Planning videos=

The next section will take a closer look GIS Data Administration.

Previous Editions
Server Software Performance 43rd Edition

Page Footer Specific license terms for this content System Design Strategies 26th edition - An Esri ® Technical Reference Document • 2009 (final PDF release)