System Design Process
Spring 2014 System Design Process 34th Edition
The purpose of this document is to share a system design methodology that promotes successful deployment of geographic information system (GIS) technology. This system design methodology includes guidelines for identifying business requirements, making appropriate software selection, using properly configured data sources, and providing sufficient hardware to meet user productivity needs. This document focuses on system performance and scalability - building a GIS that will perform during peak operational loads.
Much more can be said about business requirements analysis (GIS User Needs) and available software functionality - this is not the focus of this documentation. Dr Roger Tomlinson (Father of GIS) provides an excellent book that shares a proven framework for comprehensive GIS planning called Thinking about GIS. Understanding the information products you want out of the GIS and identifying the software candidates you might use to produce these information products is a prerequisite for completing your system architecture design.
What Is System Architecture Design?
The system architecture design process aligns identified business requirements (user needs) derived from business strategy, goals, and drivers (business processes) with recommended business information systems infrastructure technology (network and platform) recommendations. The computer display transaction is the work unit used to translate business requirements to associated server and network loads. Display service times (software processing) generates platform loads and display traffic generates network loads. Peak throughput loads (business needs) are used to generate platform capacity and network bandwidth requirements.
System design starts with identifying business needs. This includes identifying user locations and required information products, identifying required data resources, and developing appropriate software applications to do the work. Business needs are represented by project workflows that identify the traffic and processing associated with each display transaction.
System architecture design translates business needs to identified IT requirements. Hardware requirements are generated based on peak software processing loads. Network connectivity requirements are generated based on peak data flow. Capacity Planning tools are provided to automate the design analysis.
Warning: Trying to build a GIS without completing a proper system architecture design can lead to system deployment failure.
Why Is System Architecture Design Important?
A distributed computer environment must be designed properly to support user performance (productivity) requirements. The weakest "link" in the system will limit performance. The system architecture design process identifies specifications for a balanced hardware solution. Investment in hardware and network components based on a balanced system load model provides the highest possible system performance at the lowest overall cost as represented by the chain in Figure 1.2.
Building a high performance GIS requires more than getting the hardware right. User workflows must be designed to optimize client productivity (simple maps) and efficiently manage heavier geoprocessing loads (service request queue). The geodatabase design and data source selection should be optimized to address system performance and scalability requirements. The selected production platform components (servers, workstations, storage) must have the capacity to handle peak user workflow processing loads within an acceptable service response time. The system architecture design must address performance needs and bandwidth constraints over distributed communication networks—technology and solution architecture must be selected to conserve shared infrastructure resources. System architecture design can provide a solid foundation for building a productive operational environment.
Workflow complexity determines processing and data flow loads that must be handled by the computing infrastructure. The computing architecture must be selected with the appropriate capacity to service the required business loads. Workflow complexity is a measure of the amount of processing loads and network traffic required to refresh the user display. Complexity is imposed on the computing architecture by the following design attributes:
- Database design and data format: DBMS, geodatabase, and ArcSDE
- User workflow software design: Application development
The computing infrastructure must provide sufficient capacity to handle peak operational loads.
- Server platform processor core and deployment architecture must handle peak processing loads.
- Network bandwidth and remote site connectivity must be adequate to avoid traffic contention.
- Storage access performance and capacity must be adequate to provide required data access.
Server performance, network capacity, and efficient storage strategies can improve user productivity and reduce system cost.
The system architecture design process can be used to identify specifications for a balanced hardware solution. Investment in hardware and network components based on a balanced system load model provides the highest possible system performance at the lowest overall cost.
Why we do planning
The primary reasons for planning include identifying business needs, defining project requirements, and reducing implementation risk. In practical terms, we need a plan if we hope to get something done. The plan provides a foundation for successful implementation. Figure 1.3 identifies some of the primary planning objectives.
Business requirements may include a variety of user workflows and Web services that would improve business operations – these workflows and services must be identified early during the planning process. The workflow and service definitions establish a foundation for the rest of the planning.
The peak user workflow and service processing requirements must be identified and translated to appropriate system design loads – server processing times and network traffic. These processing loads provide a foundation for generating appropriate hardware specifications (performance and capacity requirements) and network infrastructure needs.
A project must be defined to procure and implement the required technology. The project will include a budget and schedule along with a variety of project performance milestones. Project approval is required before any major design and/or implementation efforts begin. Project approval is based on identified or perceived benefits that will accrue to the business resulting from the proposed level of time (schedule) and financial (budget) investment.
A good plan can be used to reduce system implementation risk. Delivering a system that will satisfy identified business requirements is fundamental to project success. Understanding key system performance parameters, identifying incremental system performance targets, and establishing a system performance validation plan can chart a solid framework for managing implementation risk.
Planning is an incremental process driven by rapidly changing technology. The most effective plans are designed to adapt to changing business needs and evolving technology opportunities. The established change control process must be agile and effective to ensure project tasks are managed to deliver within established schedule and budget constraints.
What questions are we trying to answer?
There are many questions about system architecture design that we answer on a daily basis. The System Design Strategies documentation deployed on this wiki site is developed and maintained to answer most of these questions, and provide a reference to help customers better understand their design needs.
Here is a short list of some of the more common questions:
- How many users can I support with my existing hardware?
- What hardware do I need to purchase?
- How many servers (cores) do I need?
- What are the software licensing requirements?
- What workflow loads should I use for my existing applications?
- What are my current workflow service times?
- What is the capacity of my current system?
This documentation, along with the companion Capacity Planning Tools, was developed so you can answer these questions about your GIS, and many more. We have been supporting GIS customers for over 20 years, and much of what we have learned is shared so you can benefit and build a successful GIS.
What demands does GIS place on the computing infrastructure?
GIS applications place heavy demands on server processing and network bandwidth resources. For most standard mapping operations, these demands push the limits of platform processor (CPU) and network bandwidth (I/O) capacity.
Several key infrastructure components work together to service business processing loads.
- Network I/O: Network traffic demands on available bandwidth
- Processing: Platform processor core processing demands
- Memory: Platform physical memory utilization
- Graphics: Video graphics card loads
- Disk I/O: Storage disk access performance
Different workflows place different loads on the system. Relative processing demands for the four use patterns above show how these loads are distributed differently, based on the workflow.
- Data query and analysis is network I/O intensive.
- Analysis and processing is compute (processing) intensive.
- Fly-through and 3D animation can be graphics intensive.
- Data loading and conversion can be disk I/O intensive.
GIS operations are typically network and/or processing intensive—these are the subsystems that require the closest attention. GIS applications place heavy demands on server processing and network bandwidth resources. For most standard mapping operations, these demands push the limits of platform processor and network bandwidth capacity.
Additional critical design components include the following:
- Memory is critical if you do not have enough; memory requirements can normally be satisfied by following standard configuration guidelines. Platform memory guidelines are provided in Windows Memory Management appendix.
- Some GIS desktop applications are graphics intensive (3D animation); video graphics are less critical for GIS than for the video gaming industry, so there is plenty of graphics technology available at reasonable cost to meet GIS needs.
- Disk I/O can be a concern if you are moving large volumes of data in or out of storage, such as with imagery or map cache generation (normally a background administrative process).
Any one of these components, if configured incorrectly, can become a bottleneck and limit system performance.
System architecture design is about building a balanced solution that satisfies peak performance needs at the lowest possible cost. Each of these critical subsystems should be considered in establishing the right system design.
Cost of a change
Your GIS plan should include performance validation early in the design process. ArcGIS for Desktop authors should have performance targets they verify when publishing a new map service. GIS programmers should have performance targets they build to when developing a new application. Performance should be a consideration for data administrators, network administrators, and selected storage subsystems. Getting the design right from the start can save money and improve overall business operations.
What is the System Design Process?
The system design process includes a GIS needs assessment and a system architecture design. The system architecture design is based on user workflow requirements identified in the GIS needs assessment.
Traditional system design process
GIS Needs Assessment: The GIS needs assessment includes a review of user workflow requirements and identifies where GIS technology can improve the quality and productivity of the business process flow. This assessment identifies GIS application and data requirements and an implementation strategy for supporting GIS user needs. The user organization must be actively involved throughout the user needs assessment. A GIS solutions architect familiar with current GIS technology patterns and customer business practices can help facilitate this planning effort.
System Architecture Design: The system architecture design is based on user requirements identified by the GIS needs assessment. Customers must have a clear understanding of their GIS application and data requirements before they are ready to complete their system architecture design. An Enterprise system design consultant familiar with GIS software patterns and system design analysis is typically used to complete the final system design and analysis. Technology purchase decisions should be delayed until after completing the system architecture design analysis.
The system design begins with a technology exchange. The technology exchange provides a foundation for client support throughout the design process. Client participation is a key ingredient in the design process. The design process includes a review of the existing computer environment, GIS user requirements, and system design alternatives. System design capacity planning tools can be used to translate projected peak user workflow requirements to specific platform specifications. An integrated implementation strategy can then be developed to support GIS deployment milestones.
Warning: Problems can occur when software technology selection is made based on a GIS needs assessment without considering system architecture design implications.
Integrated Business Needs Assessment
Traditionally, the user needs assessment and the system architecture design were two separate efforts. There are some key advantages in completing these efforts together. Figure 1.6 shows the project time line relationship between the traditional user’s needs assessment and the system architecture design process.
- The GIS software technology selection process should include a discussion of architecture options and available system deployment alternatives.
- The existing hardware environment, peak user workflow operations, and user location information required for a system design should be identified during the user needs workflow interviews.
- Software technology selection should consider hardware configuration options, required platform selection, peak system loads for each configuration option, and overall system costs. User productivity may be a primary consideration in making the right software technology choice.
- And finally the system implementation schedule must consider hardware and application delivery milestones.
A primary goal for developing capacity planning tools is to automate the system architecture design analysis in such a way that GIS solution architects can use the tools to complete a user needs assessment and a system architecture design within an integrated business needs assessment.
Building a GIS: Implementation strategy
- User needs establish a foundation for completing the design.
- User location and peak business loads establish a foundation for system architecture design.
- Infrastructure upgrade requirements must be identified to identify deployment costs.
- Network communication capacity is an important consideration for GIS deployments.
- Hardware and software procurement requirements must be identified.
- Software development requirements and data acquisition needs must be identified.
Best Practice: Business decisions for project funding and procurement authorization are often required for project effort to proceed beyond this phase.
- System procurement authorization, based on the design budget and deployment timeline.
- Data acquisition and database design efforts begin.
- Procurement authorization for application design and development.
- Prototype testing plans completed and scheduled to validate product delivery within design performance targets.
- Initial deployment and operational testing.
- Final system delivery, user training, and workflow migration complete.
- System maintenance operations.
Best Practice: Deployment process is repeated incrementally on a periodic schedule to leverage technology change.
The Capacity Planning Tools (CPT) were developed as a framework to promote successful GIS system design and implementation. CPT functions contribute throughout the implementation cycle. The CPT tasks include a review of business needs, establish performance targets, identify user locations, review network suitability, select product architecture, and complete the system architecture design analysis. Additional tools are provided to validate system performance during the design, construction, and implementation phases. CPT models can be easily updated to reflect changes in business requirements and review alternative system deployment strategies, providing an adaptive model for addressing a variety of incremental planning activities.
Best Practice: Build and maintain a simple system performance model that links user requirements with system design.
System design strategies overview
System design topics will be discussed throughout the SDSwiki documentation.
- Software technology: Chapter 2
- Software performance: Chapters 3 and 4
- Data administration: Chapter 5
- Network communications: Chapter 6
- Platform architecture: Chapter 7
- Platform performance: Chapter 8
- Information security: Chapter 9
- Performance Management: Chapter 10
- System Implementation: Chapter 11
All of the SDSwiki content will be brought together in a system architecture design case study for the fictional City of Rome. The City of Rome will bring all the chapters together in a realistic enterprise design.
- City of Rome case study: Chapter 12
System architecture design terminology
New technology introduces new terminology, and the language evolves with each new idea. Often a word is assigned to express a particular concept or explain a particular role a component plays in the overall design—and often common words are assigned and used to describe new concepts. It is important to take the time to understand how the word is being used within the concept that is being discussed.
The words used in the System Design Strategies wiki documentation share what is being assigned by software development, IT, and the business operations groups the system design is bringing together. Take time to listen to the words, and the context in which they are being used. These words provide a framework for discussions on system architecture design.
The [Acronyms and Glossary] describes many of the key words used within the system design strategies documentation.
System architecture design process
The enterprise GIS system design process aligns identified business requirements (user needs) derived from business strategy, goals, and drivers (business processes) with recommended business information systems infrastructure technology (network and platform) recommendations.
- User needs assessment (results of a GIS user needs assessment provides inputs for the system architecture design analysis)
- Workflow loads analysis (translate user needs to project workflows with baseline traffic and processing transaction loads based on estimated workflow complexity)
- Technical architecture strategy (identify user locations, network connectivity, and data center server locations)
- User requirements analysis (translate peak user workflow loads to peak throughput transaction loads)
- Network suitability analysis (translate peak site/network throughput loads to peak site/network traffic and compare with available network bandwidth)
- Platform architecture selection (Identify data center platform tier configuration and identify platform selection for each tier)
- Software configuration (Identify platform assignment for each workflow software component peak transaction processing load)
- Enterprise design solution (combine all peak workflow software component processing loads on the assigned platform tier, translate baseline processing load to selected platform processing load, and generate number of nodes required for each platform tier with estimate of capacity utilization)
The Esri Capacity Planning Tool (CPT) was developed to simplify business requirements data collection and automate system design analysis tasks associated with each phase of the development cycle. This section will introduce the available capacity planning tools and their function in completing a system architecture design. Hyperlinks are provided to associated CPT demos located in the Capacity Planning Tool appendix.
The CPT can be used to identify your system design requirements and model performance and scalability of your enterprise design. The CPT provides a framework for integrating business, data, applications, and technical architecture needs required to design, deploy, and manage successful enterprise GIS operations.
Pre design efforts
Figure 1.10 shows how you can prepare for your system architecture design. Business needs must be understood before you are ready to complete the system architecture design. Business requirements analysis includes a review of the enterprise vision, the existing business architecture, and the user workflow requirements. Each of these areas must be explored in some detail before you begin the design.
Figure 1.11 identifies the CPT tabs for documenting findings from your pre-design efforts.
Review Business Needs
Business needs (user requirements) establish a foundation for building a proper system design. Business requirements are represented by user workflows. The first step in completing a system architecture design is to select the appropriate project workflows. GIS Software Technology chapter provides an overview of the primary GIS workflow patterns along with best practices for making a proper technology decision. Software Performance chapter identifies how you can generate project workflows with appropriate transaction loads to represent your business requirements.
The Project Workflows section on the CPT Workflow tab is located at the top of the workflow list. Project Workflows are selected from the Standard Workflows included on the CPT Workflow tab or custom workflows generated from the CPT Calculator or Test tabs - identifying your project workflows is the process referred to as your user workflow loads analysis. Once your software technology (project workflow) selection is complete, your selected workflow performance targets will be available at the top of the CPT workflow selection list for use in your system architecture design.
System Design Process
- Technical architecture strategy—High-level overview showing user site locations, network bandwidth connections, and central data center locations. User location information is collected during the user needs analysis.
- Workflow loads analysis—CPT Requirements analysis section is configured to represent the site locations, user workflows, peak loads, and network bandwidth for the enterprise design solution.
- Network suitability analysis—CPT Design completes the network suitability analysis and identifies any communication bottlenecks. Network bandwidth upgrades are identified to complete the network suitability analysis.
- Platform architecture selection—CPT Design Platform tier is configured to represent the design solution. Identify platform tier nicknames, select platforms, and identify platform rollover settings.
- Software configuration—CPT Design Software Configuration module is used to assign workflow software to supporting platform tier (software install) and make workflow data source selection.
- Enterprise design solution—Once configured, the CPT Design tab completes the system architecture design analysis and provides the platform solution.
Figure 1.13 shows the steps to complete the system design. System design process includes the user requirements analysis, network suitability analysis, platform architecture selection, software configuration (installation), and the Enterprise design solution. The CPT was designed to complete the analysis. Once the CPT is properly configured and business requirements are defined, the CPT will complete the system architecture design analysis and display the hardware solution.
Once you identify your business workflows, you are ready to complete a user system loads analysis. The CPT Design tab includes a Requirements Analysis module where you can identify user locations and peak throughput loads.
Once you configure the CPT Design tab to reflect your user requirements and identify the network connections, the CPT completes a network suitability analysis. Network suitability analysis is completed by the CPT Design tab. The CPT analysis evaluates network bandwidth and latency to ensure adequate capacity to accommodate peak traffic flow loads. You can then increase network bandwidth as required to accommodate peak traffic loads.
Platform selection is a critical decision in any design process. The selected platform directly contributes to user display performance, platform capacity, and software licensing cost. A faster processor core improves user performance and reduces license cost. Higher capacity servers reduce the total number of server machines required to satisfy business requirements.
Warning: Platform selection directly impacts overall system cost.
Note: CPT Design software configuration will be discussed in more detail in Lesson 7: GIS product architecture.
Once you make your platform selections and install workflow software components (software configuration), the CPT will complete the platform sizing and show the required server configuration (number of platforms and peak utilization levels for each platform tier).
The Design tab provides a snapshot view of your user requirements and associated platform solution as a single integrated information product.
Note: Chapter 12 provides a case study for the City of Rome, demonstrating how to use the CPT to complete an enterprise system architecture design.
The CPT Hardware tab is used to list SPEC performance benchmark values for use in completing the system architecture design.
Monitor performance compliance
- Measuring display rendering time when authoring a service is the first opportunity to validate performance.
- Measuring deployed service rendering time is another opportunity to validate performance.
Best Practice: Monitoring progress in meeting performance milestones can reduce deployment risk and ensure project delivery success.
When performance issues are identified early in deployment, proper adjustments can be made before impacting production workflow productivity. Simpler map displays, simpler data models, and cached data layers are proven ways to reduce workflow complexity. Warnings identified by the map publishing analysis tool identify how to reduce display complexity.
Best Practice: Identifying and resolving performance issues before they become production level performance problems will promote deployment success.
Measuring performance during the system build and deployment reduces implementation risk. System design is only the beginning of the process; then you need to build a system that performs within the system architecture design performance targets. If measured performance loads exceed planned budgets, adjustments can be made to workflow complexity during the build process to deliver services within initial performance targets.
Platform capacity calculations
Platform capacity for a standard map display can be calculated for any selected platform. Platform capacity varies based on selected platform throughput performance and published map display complexity.
- 2013 Baseline System Specifications
- Arc13 4-core platform performance SPEC baseline = 200 (SPECint_rate2006 Baseline)
- ArcGIS 10.2 for Server REST medium complexity workflow
- Baseline 4-core medium complexity throughput capacity = 70,000 TPH
- Platform SPECint_rate2006 baseline = 400, medium complexity, throughput = 140,000 TPH
- Platform SPECint_rate2006 baseline = 100, medium complexity, throughput = 35,000 TPH
- Platform SPECint_rate2006 baseline = 200, light complexity, throughput = 140,000 TPH
- Platform SPECint_rate2006 baseline = 200, heavy complexity, throughput = 47,000 TPH
This section ends by introducing a simple tool used for proper hardware platform selection. The Platform Capacity Calculator identifies the single server throughput capacity of the selection platform configuration.
The Platform Capacity Calculator provides a peak throughput range for selected workflows, showing medium complexity output in blue and light complexity output in red. This provides a simple tool to help customers select the appropriate platform configuration.
CPT Calculator tab provides a much more accurate platform capacity planning assessment for a single workflow configuration, taking into account software performance variables and hardware configuration strategies.
CPT Design tab provides a much more adaptive enterprise capacity planning assessment integrating system loads for multiple workflows supporting multiple network local and remote sites and a variety of platform configuration strategies.
Proper GIS planning is the most important investment any organization can make in building a GIS. Understanding your GIS needs, selecting the right technology at the right time, and establishing documented implementation milestones to measure your progress can ensure your success. This document is focused on sharing how to build and maintain successful GIS operations. The Capacity Planning Tools provide a framework for collecting what you know about your business needs and your system environment. The Capacity Planning Tool models connect what you understand about GIS user requirements with the network and platform loads your IT support teams can measure in the data center.
The next Chapter will review the most common GIS technology patterns and share best practices in making the right technology selection.
System Design Process 33rd Edition
System Design Process 32nd Edition
System Design Process 31st Edition
System Design Process 30th Edition
System Design Process 29th Edition
System Design Process 28th Edition
System Design Process 27th Edition