In this module, you be developing methods to evaluate your GIS system through heuristic examinations, focus groups, card sorting, and tabletop exercises. Additionally, you will develop specific heuristic examination methodology based on methods that best suits your geospatial design.

You will also be developing a cost-benefit analysis that outlines the specific monetary costs of developing, implementing, and maintaining your geospatial system. This will require researching the costs of your system as well as understanding the monetary benefits that your proposed geospatial design will provide over the current system in use.

Finally, you will develop a Gantt chart to visualize the timeline for development, implementation, evaluation, and maintenance. Usually, a GIS design will require a minimum of 6 months, but depending on the complexity of your system, it may take much more.

Overall, evaluation, cost/benefit, and maintenance are very important components of your project since the stakeholders will be most interested in knowing how much they are requested to fund as well as how long it will take for the final project to be beta tested.

A project design incorporates evaluation frequently, leading to iterative updates and changes to the design. Throughout the process, you have implemented the needs assessment and the cognitive walkthrough for evaluation. This week, you are incorporating the last type of evaluation: usability testing and heuristic evaluations. Therefore, you can see that evaluation is iterative and continues throughout the entire design process.

Figure 1: Sequence of steps for designing and evaluating a geospatial design.

Credit: Brandi Gaertner © Penn State is licensed under CC BY-NC-SA 4.0

What is Evaluation?

Evaluation is typically categorized into two broad areas: formative evaluation and summative evaluation. Formative evaluations focus on developing and refining designs. Summative evaluations compare an implemented system to an alternative system with the goal of measuring differences in performance or user satisfaction between the two systems. Quite often, formative evaluations happen in the early/middle stages of a design exercise and summative evaluations take place toward the end when a system has been implemented.

Common methods used in both types of evaluation include:

• Heuristic examinations - measure user responses to the system based on a set of common system design criteria
• Surveys - use open or closed-ended questions to identify system needs or areas for improvement
• Focus groups - involve group discussion of design options, user experiences, or other topics to inform or critique a design
• Interviews - make use of one-on-one questioning with users or customers to explore design options or gather feedback on tools
• Card-sorting - is an activity in which users organize system tools/functions using paper cards to suggest interface organization
• Expert evaluation - has system design and usability experts critique designs, prototypes, or final systems
• Field & Tabletop Exercises - put the system through a "test run" using realistic data, scenarios, and tasks
• Cost-Benefit Analysis - uses metrics to measure the costs of developing/using a system versus the benefits associated with its products

Formal vs. Informal Evaluation

A distinction used quite often is to characterize evaluation efforts as formal or informal depending on the degree to which the evaluation activity makes use of rigorous methods to ensure unbiased participants, sound methodology, and careful analysis of results. An informal evaluation might make use of a few of your coworkers to look over a prototype design, while a formal evaluation could involve a dozen real end-users who complete a realistic exercise using the new GISystem and complete a post-activity interview and survey to gather structured and unstructured feedback.

Usability testing evaluates the users' ability to learn and operate the geospatial design, limit user error, appeal to user aesthetics, and adhere to accessibility requirements. The Ansyah, 2023 article outlines several usability testing methods, which I will briefly describe below. However, you will also be required to read the article to develop a more detailed background on how to implement the different usability testing methods.

Evaluation Criteria

When considering the criteria to include in a usability study, you may want to consider the different elements of a geospatial design (see Schulz, 2021 from module 2):

1. Basemap Quality
2. Cartography
3. UX/UI
4. Mobile Design Conventions
5. Usability
6. Location Based Services
7. User Tasks
8. Functionality
9. Navigability
10. Accessibility

GOMS (Goals, Operators, Methods, and Selection rules)

A population approach for evaluating human-computer interaction (HCI), and evaluating the ability of a user to complete a task is the GOMS usability testing method. This is very similar to a cognitive walkthrough, but with the finalized version of a design instead of the prototype. The results of the GOMS testing will provide information on the completion rate of each task and/or the time to completion for each task, therefore providing valuable information on the navigability and usability of the design.

The example below is extracted from Ansyah, 2023 and shows a GOMS usability testing sheet, with the “Goals” and “methods”, which describes the tasks the users will complete as well as the steps the user needs to take to complete that task.

The GOMS usability testing approach.

Click for a text description.

GOMS Task Scenario

No	Goals	Action	Methods
Task 1. Find a cafe
1.1	Search for a location	Tap	Tap search column
1.2	Search for a location	Typing	Type keyword
1.3	Search for a location	Tap	Tap search button
1.4	Read search results	Scroll	Scroll screen to view search results
1.5	Confirm the specified location	Tap	Tap the specified location
Task 2. Share a location of Gubeng Station with a friend
2.1	Confirm the specified location	Tap	Tap the specified location
2.2	Share to the specified app	Tap	Tap share button
2.3	Share to the specified app	Swipe	Swipe to find the app to share the location
2.4	Confirm the app to share the location	Tap	Tap the app button
Task 3. Add Stop to the current route
3.1	Find the Add Stop menu	Tap	Tap additional options menu button
3.2	Find the Add Stop menu	Tap	Tap menu add stop
3.3	Find the Add Stop menu	Tap	Tap the searched location in the search results
3.4	Search the specified location	Tap	Tap search column
3.5	Search the specified location	Typing	Type keyword
3.6	Search the specified location	Tap	Tap search button

Credit: Ansyah, A. S. S., Masruri, M. Z., & Rochimah, S. (2023, October). Usability Testing of User Experience and User Interface Design on Mobile Map Applications: A Comparative Study of User Perception and Interaction. In 2023 14th International Conference on IEEE

System Usability Study

This type of usability testing focuses on “effectiveness and efficiency” and relies on a Likert scale (answers ranging from 1 to 5, with 1 indicating strongly disagree and 5 indicating strongly agree). The total sum of scores from the Likert scale answers can indicate if users thought the geospatial design was effective and efficient.

You can design a system usability test that includes 10 more questions regarding the specific details of your design including the navigability, aesthetics, accessibility, user error, and/or additional elements that would help with understanding your users' evaluation.

A/B Testing

A/B testing is a “split testing” method that is useful in comparing two different systems, and requires deploying two different versions of the system to your users and gathering feedback about the usability of both systems, to determine which is most efficient. 

One method for A/B usability testing is to provide two images (one for each of the versions) and asking the users specific questions about the aesthetic appeal, navigability, accessibility, and operability of both versions.

A/B usability testing can also be on a likert scale from 1 to 5, with 1 as strongly disagree and 5 as strongly agree

References:

A cost benefit analysis should show the total monetary costs of each component of the design, as well as the monetary benefits the design would provide over either not having the design and/or in comparison to another design that currently exists.

The goal of your geospatial project proposal is to “pitch” it to a stakeholder to request approval to move forward on it development and, hopefully, request and gain funding for the project. The stakeholders will be very interested to know how much the design will cost them. By providing specific details and explanation on the costs of the system, you’re also proving your understanding and knowledge of the system, therefore “proving” to the stakeholder that you are capability of managing the project and the design.

Therefore, the cost benefit analysis is a very important component of your project. Read Babinski, 2012 for an example on how to construct a cost benefit table.

References:

Gantt charts are an easy method to visualize to your users and stakeholders the timeline for development, implementation, maintenance, and evaluation of your system. An effective Gantt chart should include all of the different steps of creating your design through beta testing the finalized result.

An effective Gantt chart should show all the different steps for designing the geospatial design. The project proposal you’ve outlined in this term project is the “planning” stage of your GIS design. The next stage(s) are developing the design and evaluating the final version (which is an iterative process). Consider steps that a GIS analyst would implement including setting up the workstation, buying the appropriate licenses/computer hardware (if necessary), accessing and analyzing the data, conducting a needs assessment/cognitive walkthrough, developing the geospatial software (whether that includes from a low/no code option or from an open source may require more time/steps), performing the usability testing, beta testing, and maintenance. You may have many more steps for your project, considering the entire life cycle of developing a Geospatial Design.

There are several different methods for developing a Gantt chart.

Gantt Chart in Excel

From the website above (click “Gantt Chart in Excel”) for creating a Gantt chart in Excel, you can expect to create a Gantt chart as displayed below. However, your Gantt chart should have more specific details (e.g. instead of four tasks, you should have at least 10 – depending on the complexity of your project). You may also consider incorporating specific “quarters” for your development.

A very brief example Gantt chart for visualizing the four stages of geospatial design.

Credit: Brandi Gaertner © Penn State is licensed under CC BY-NC-SA 4.0

Gantt Chart in R programming Language

Using the detailed step-by-step explanation above, you can produce a much more customizable Gantt chart with the same data in the R programming language using ggplot and tidyverse. (Link Above: click “Gantt chart in R Programming Language”).

Below is an example of the Gantt chart you can create, which has different “stages” of design, specific dates, different tasks, and an additional detail of completed versus not yet done.

A detailed example of a Gantt chart that can be created in R programming language that offers customization for dates, tasks, and completion.

Credit: Community Contribution Gantt Chart by Joyce Robbins is licensed under GNU General Public License. Accessed Dec. 2, 2024.

Charts in Python

Depending on your fluency with python, you can create a Gantt chart using MatPlotLib (link above: Click “Gantt Charts in Python”), which offers similar customization as R including dates, project tasks, and additional details of completed/Not Yet Completed.

The example below shows different tasks broken down by “team”; however, you can easily break it down by “Completed” and “Not Yet Completed”. You can add additional details if you are comfortable with Python and/or interested in providing detail on your chart.

A detailed example of a Gantt chart produced using MatPlotLib in Python. Screen capture by Brandi Gaertner

Credit: Kosourova Elena “How to Make a Gantt Chart in Python with Matplotlib.” datacamp. November 16, 2022.

The lesson 7 readings focus on usability testing and the cost benefit analysis. Both are very important components of your analysis, since they provide information on the usability of design, as well as valuable information to stakeholders on the costs of the system.

Once you have created a GISystem, it is time to evaluate the effectiveness of the system and how well it meets user needs. You have several evaluation methods depending on your GISystem Design, your users, and the needs requirements. It is important to frequently evaluate your system as an iterative approach to GISystem maintenance. 

A critical decision point occurs after a conceptual design has been created, and before detailed design is initiated. Here, supervisor or client approval are required for a commitment in money, staff, effort, and potential business disruption. All organizations will make such commitments when expecting payoff to justify the effort. Such justification begins with assigning a value to the benefits and the costs of acquiring the capability. This often leads to what has become known as a benefit–cost analysis or economic analysis.

The Future Work section summarizes principal technical and non-technical features of the product. Unresolved issues should be highlighted at this time. Future work is outlined in an anticipated work schedule with milestones for the next phase of the project. Here is where you ask for approvals or authorizations for project continuation.

This week, and the remaining modules, will focus on geospatial technology “trends” or current new and developing technologies. One of the most discussed current geospatial field is GeoAI, which of course encompasses many different tools, methods, models, datasets, and more. However, one well known GeoAI method is Deep Learning, which at the most basic, uses 1000s of images to “train” a model to automate the detection of an object. Deep learning has been used in a variety of disciplines and continues to be implemented in new and innovative ways.

Deep learning has been used to detect graffiti on building facades using street view images, extracted from OpenStreetMap, which can then be used to automatically classify and identify buildings in need of repair. For example, in the screenshot extracted from Novack et al, 2020 below, graffiti has automatically been converted to a ranked discrete shapefile.

Example of deep learning used for automating detecting of graffiti (Figure 7).

Novack, T., Vorbeck, L., Lorei, H., & Zipf, A. (2020). Towards detecting building facades with graffiti artwork based on street view images. ISPRS International Journal of Geo-Information, 9(2), 98.

Although many examples exist, deep learning has also been used to automate detection of solar panel installations, which are automatically digitized and converted to discrete file systems for sharing and analysis. 

Figure of deep learning used to automate detection of solar voltaic installations.

Kausika, B. B., Nijmeijer, D., Reimerink, I., Brouwer, P., & Liem, V. (2021). GeoAI for detection of solar photovoltaic installations in the Netherlands. Energy and AI, 6, 100111.

References: