Lesson 7: Adding Search and Query Capability

Lesson 7: Adding Search and Query Capability jed124

Overview

Overview jed124

An important element of many geospatial applications is the ability to search for features that meet certain criteria. The nature of those criteria might be spatial (e.g., land parcels adjacent to a particular street) or non-spatial (e.g., land parcels owned by a particular person).

In Lesson 7, we’ll see how Query objects can be used in an ArcGIS Maps SDK for JS app to answer both spatial and non-spatial questions. Before that, however, we’ll look at the SDK’s Search widget, which can be used to find values in a layer’s attribute table or to find places when configured to work with a locator service.

Objectives

At the successful completion of this lesson, you should be able to:

  • configure the Search widget using sources (layers and/or locators);
  • understand the concept of a promise (in a JS context);
  • set a layer's definition expression;
  • define and execute queries.

Questions?

If you have any questions now or at any point during this week, please feel free to post them to the Lesson 7 Discussion Forum. (That forum can be accessed at any time by clicking on the Discussions tab.)

Checklist

Checklist jed124

Lesson 7 is one week in length. (See the Calendar in Canvas for specific due dates.) To finish this lesson, you must complete the activities listed below. You may find it useful to print this page out first so that you can follow along with the directions.

Steps to Completing Lesson 7
StepActivityAccess/Directions
1Work through Lesson 7.Lesson 7
2

You will choose a scenario from the list in the poll linked from the assignment page requiring the development of a map based on Esri's Maps SDK for JS.
 

  • Post the map meeting your scenario's requirements to your e-portfolio.
  • Below the map, include some reflection on what you learned from the lesson, any concepts that you found to be challenging, and what resources you used to complete the assignment (minimum 300 words).
Follow the directions throughout the lesson and in the scenario chosen from the poll.
3Post your initial ideas for the final project app in the Lesson 7 discussion forum.Read the directions on the Lesson 7 Assignment page and post your ideas in the Lesson 7 Discussion Forum.
4Take Quiz 7 after you read the online content.Click on "Lesson 7 Quiz" to begin the quiz.

7.1 The Search Widget

7.1 The Search Widget jed124

A common feature of many mapping apps is the ability to search for a place by entering text in a search box. This feature can be added to an Esri app through the Search widget. This widget can be configured to find locations using a Locator (geocoding) service or features in a FeatureLayer. Let’s take a look at various implementations of this widget.

7.1.1 Basic Implementation

7.1.1 Basic Implementation ksc17

A basic implementation of this widget is really quite simple, as demonstrated by Esri’s Search Widget (3D) sample. After adding the Search module to the import() array, all that is necessary to implement the widget is to create a new object of the Search class, set its view property to the app’s view, then add the widget to the view’s UI in the desired position. The widget will match the user’s text against Esri’s World Geocode service by default.

Some properties of the widget that you might consider customizing include:

  • allPlaceholder – text users see in the search box prompting them on what to enter
  • minSuggestCharacters – number of characters that must be entered before queries are made against the widget’s source
  • popupEnabled – Boolean specifying whether a popup can be opened over a search result
  • popupOpenOnSelect – Boolean specifying whether a popup should open automatically over the selected result
  • popupTemplate – object that defines what the user sees in the popup

7.1.2 Using a Custom Locator

7.1.2 Using a Custom Locator ksc17

Esri’s World Geocoding service is suitable for many apps, particularly those built at a global or continental scale. However, for apps built for relatively small areas of interest, it may be possible to utilize a locator built using data from an authoritative source, such as a city or county. The advantage to using a locator from such a source is that it is much more likely to be kept up to date (e.g., to contain new subdivisions).

See the Pen Search Widget Locator Demo by Jim Detwiler (@jimdetwiler) on CodePen.

The example above was built using a locator developed specifically for the City of St. Paul, Minnesota. The first step in creating an app like this is discovering the desired locator. Many locators can be found in ArcGIS Online by searching for locator in the Tools category.  (1277 Madison St is an address in St. Paul if you're looking to try the locator.) If this sample does not work for you, it may be that the City has taken its geocoder / locator service offline probably due to costs as geocoding is one of the more expensive services to maintain due to the number of credits used.

Once you've identified the ArcGIS Server REST endpoint of your desired locator, incorporating it into your app involves modifying the Search widget’s sources property. In my example, I included only the St. Paul locator, but as the property name implies, you can wire the widget up to multiple locators (e.g., you might have locators available for adjacent jurisdictions). The widget is designed such that Esri's World Geocoding service is included as a source by default.  I've disabled that behavior by setting the includeDefaultSources property to false.  We’ll see a multi-source example later in the lesson.

As explained in the SDK documentation, the Search widget can have its sources property set to a LocatorSearchSource, LayerSearchSource, or some combination of the two. Looking at the LocatorSearchSource documentation, you’ll notice that a LocatorSearchSource object has some of the same properties mentioned on the previous page (e.g., popupEnabled). While perhaps a bit confusing, this duplication allows for having different settings for these properties on a source-by-source basis. For example, you might have a reason to disable popups for one of your widget’s sources, but enable them for another.

Getting back to the St. Paul example, the sources property is set using a single LocatorSearchSource object. Unsurprisingly, the most important property to set for this object is its url, which should be set using the REST endpoint of the locator service.

From there, because the widget provides just a single text box, you’ll want to look for matches against the locator field that is labeled as the Single Line Address Field in the REST Services Directory. In the case of the St. Paul service, that field is called SingleLine.

Finally, the placeholder property is used to provide a prompt or hint to the user on what should be entered in the search box.

In addition to its locator, there are several other LocatorSearchSource properties that you might want to set. Here are a few of them:

  • autoNavigate – do you want to zoom to the selected location?
  • countryCode – used to restrict searches to a single country (e.g., "US")
  • resultSymbol – used to customize the symbol used to show the selected location
  • zoomScale – used to control the scale that the app will zoom to after finding a match

7.1.3 Searching in a FeatureLayer

7.1.3 Searching in a FeatureLayer ksc17

As mentioned earlier, the Search widget can also be configured to find features in a FeatureLayer. The app below allows for finding cities and counties in Jen & Barry's world.

See the Pen Search Widget FeatureLayer Demo by Jim Detwiler (@jimdetwiler) on CodePen.

The key difference as compared to the previous example is that a LayerSearchSource (actually two) is used instead of a LocatorSearchSource. The SDK shows that a LayerSearchSource has many properties in common with a LocatorSearchSource, though it obviously has others that are unique to it.

Here’s a quick rundown of what’s happening in this app:

  • The two FeatureLayers are created using their portalItem ids as we’ve done earlier in the course.
  • An appropriate PopupTemplate is defined for each FeatureLayer.
  • The searchFields property is set to the field that should be searched for the user’s text.
  • The exactMatch property is set to false, allowing for partial matches.
  • The outFields property is set to the array of fields used by the app. In this case, only the NAME field is needed in both layers.
  • When more than one source is specified for the widget, a drop-down arrow appears on the left side of the search box. The drop-down list contains a choice for each of the sources. Setting a LayerSearchSource’s name property specifies how the source will appear in the drop-down list. (In this case, "Cities" and "Counties" is assigned to the name property.)
  • The allPlaceholder property (which is assigned to the Search object) controls the hint that appears in the search box when it has multiple sources.
  • The placeholder property (set at the source level) controls the hint that appears if that individual source is selected from the drop-down list discussed above.
  • The zoomScale property is set for the cities layer source. Unlike lines and polygons, points have no logical extent envelope to zoom to. Leaving this property unset on a point layer is likely to cause the app to zoom in closer than you would like. Leaving the property unset on the county layer makes sense since zooming to the polygon extent works nicely.

7.2 Querying

7.2 Querying jed124

One of the most common operations performed in GIS is the query. The Maps SDK for JS makes it possible for developers to query layers in a number of different ways. For one, you can set a layer’s definition expression so that it displays just a subset of its data. You can also query a layer to get a count of features meeting the criteria, to get the IDs of the features, or to get the features (attributes & geometries) themselves.

7.2.1 Definition Expressions

7.2.1 Definition Expressions ksc17

Perhaps the simplest form of query you can perform using Esri’s SDK is defining a layer’s definitionExpression. This is a property supported by a few different layer sub-classes, including FeatureLayer and ImageryLayer. If you’ve worked with desktop GIS, you’ve probably encountered this sort of feature. The idea is that you can restrict a layer’s underlying data to a subset meeting certain criteria. Using a definition expression is common when working with datasets that cover multiple political subdivisions (e.g., states in the U.S.).

The definitionExpression property can be set to any valid SQL where clause. Here is an example that filters out roughly half of the Jen & Barry's cities by selecting only the features whose UNIVERSITY value is 1.  Note that the names of the cities meeting the criterion are outputted to the console.

See the Pen Definition Expression Demo by Jim Detwiler (@jimdetwiler) on CodePen.

7.2.2 Getting Feature Counts

7.2.2 Getting Feature Counts ksc17

As mentioned, Esri’s SDK provides methods for getting a count of features meeting some selection criteria, getting the IDs of those features, or getting the features themselves. Regardless of which kind of response you require, the first step in the process is creating an object of the Query class. Perhaps the most-used property on the Query class is where, which takes the same sort of SQL where clause that we saw earlier when discussing the definitionExpression property.

There are many other Query properties, some of which we’ll discuss momentarily. For now, let’s look at this example that reports the number of counties in Jen & Barry’s world that meet the criterion of NO_FARMS87 > 150.

See the Pen queryFeatureCount() Demo by Jim Detwiler (@jimdetwiler) on CodePen. 

Note that after creating a FeatureLayer of the counties, a Query object is created on lines 29-31. The object’s where property is set to a variable that was defined near the top of the script on line 8. The Query object is then used on line 34 as the argument to the queryFeatureCount() method (a method of the FeatureLayer class). Line 35 contains the alert() statement that produces the message you saw when you first opened this page. What we skipped over at the end of line 34 is some code that handles what’s returned by the queryFeatureCount() method: a promise. You’ve probably seen references to promises while poking around the SDK. Well, now we’re finally going to discuss what a promise is.

7.2.3 Promises

7.2.3 Promises ksc17

The folks who run the Mozilla Developer Network define a promise object as follows:

The Promise object represents the eventual completion (or failure) of an asynchronous operation and its resulting value.

The basic idea behind promises in JavaScript (and this goes well beyond geospatial apps) is that a lot of the operations that scripts perform take some time to complete. Rather than grinding the user experience to a halt while waiting for one of these operations, the browser gets to work on it using part of the device’s resources, but continues on executing the code that comes next. The notion of a promise came about as a way to simplify the coding of applications that contain asynchronous operations.

A promise can be in one of three states: resolved, rejected, or pending. As a developer working with a promise, you can write code to be executed when the promise resolves successfully and when it is rejected (fails to finish successfully).

Returning to the example from the previous page, running a query against a layer on some server somewhere takes some time. So Esri wrote the queryFeatureCount() method to return a promise. As is typical, working with this promise typically involves calling on its then() method. The then() method requires that you specify a callback function to be executed when the promise has been resolved. In the example, I inserted an anonymous function, though it is also acceptable to plug in the name of a function that’s been defined elsewhere. This can be a good idea when the function is relatively long.

Referring back to the definition of a promise object, when a promise is resolved successfully, it returns a value. In the case of queryFeatureCount(), as explained on its page in the SDK documentation, the returned value is the number of features meeting the query criteria. Something that is a bit tricky getting used to in working with promises is that the return value is passed along to the callback function specified in the then() method call. When defining the function, you need to create a variable to hold that passed value. In my example, I called this variable count; in the documentation example, it’s called numFeatures. The important thing is that you know the data type being returned -- the documentation conveys this to you in this case with the return type being listed as Promise<Number> -- and write your code to work with it properly.

As mentioned, you can also write code to be run in the event that the promise is rejected (fails). This error handling function should come immediately after the success handling function. The example below again uses queryFeatureCount(), this time with a misspelling of the field name in the query. Note the second anonymous function embedded within the then() method, which logs an error to the browser console when queryFeatureCount() fails.

See the Pen Promise Rejected Demo by Jim Detwiler (@jimdetwiler) on CodePen.

Now that you know how to handle methods that return a promise, you should be aware that there are certain classes in the SDK (MapView, SceneView, and all of the Layer sub-classes) that also return a promise when you create an instance of the class.  So when you create a MapView, for example, you can write code that defines what you want to happen when that view is ready.  It might help to conceptualize this second type of promise as class-based, as opposed to the method-based promises discussed above.

An important change in working with class-based promises occurred with the release of version 4.7 of the SDK.  Prior to that release, developers would use then() to specify what to do with the object once it is ready to be used.  In other words, then() was used with both types of promises.  Beginning with version 4.7, class-based promises are instead handled using when().  The reasoning behind this change, having to do with compatibility with native JavaScript promises, is detailed in this Esri blog post.  

Returning to the queryFeatureCount() example, handling an object as a promise was actually an important part of the coding.  The FeatureLayer referenced by the counties variable takes a moment to load, so the counties.when on line 34 essentially tells the browser to wait to execute the queryFeatureCount() method until that layer has finished loading.

To help illustrate the change in class-based promise handling, below is the same app written for version 4.6, in which then() is used instead of when().  A lesson to learn here is that the version of the SDK employed by an app matters.  

See the Pen Untitled by Jim Detwiler (@jimdetwiler) on CodePen.

7.2.4 Selecting Features

7.2.4 Selecting Features ksc17

Simply getting a count of the features meeting certain criteria is sometimes sufficient, but it’s often necessary to work with the features themselves. To do that, you use the queryFeatures() method. As with queryFeatureCount(), queryFeatures() returns a promise. The difference is that the queryFeatures() promise resolves to a FeatureSet object rather than a Number.

See the Pen queryFeatures() Demo by Jim Detwiler (@jimdetwiler) on CodePen.


The example above uses the same Query where clause as the previous ones and displays the counties meeting the criterion as graphics. Note the following important points:

  • A new empty GraphicsLayer object is created to hold the counties meeting the query criterion.
  • The Query has its returnGeometry property set to true. This is necessary to be able to map the results.
  • The queryFeatures() method is executed after the counties layer has finished loading (line 43).
  • The FeatureSet that gets returned by the queryFeatures() method is passed along to a function called displayResults().
  • The displayResults() function is defined such that the FeatureSet returned by queryFeatures() is stored in a variable called results.
  • Getting at the counties in the FeatureSet is done by reading its features property, which returns an array of Graphic objects.
  • The array of Graphic objects is immediately passed to a JavaScript array method called map. JavaScript developers use the map() method to do something to each item in an array and return the result as a new array. In this case, the anonymous function plugged into the map() method changes the symbol of each graphic to a yellow SimpleFillSymbol.
  • The new array of Graphic objects is added to the GraphicsLayer (in the resultsLayer variable).

So far, where and returnGeometry are the only Query properties we’ve looked at. However, there are several others that are important in certain contexts. Here is a brief description of just a few of these properties:

  • num – the maximum number of features to return; often used together with the start property to implement paging of results
  • orderByFields – an array of fields to use for sorting the results
  • outFields – an array of fields to include in the results; limiting this array to only what you need can improve performance

There are actually a few more Query properties that I think are worth discussing, but I left them out of this list because they’re considered in greater depth in the next section on spatial queries.

7.2.5 Spatial Queries

7.2.5 Spatial Queries ksc17

If you’re an ArcGIS Pro user, the sort of query we’ve dealt with so far has been analogous to the kind you’d build using the Select By Attributes dialog. Now let’s look at how you’d go about performing a query like one you’d build using the Select By Location dialog.

To implement a spatial query, the main properties of concern are geometry and spatialRelationship. The geometry should be set to some point, line or polygon that you’d like to check against the features in the layer you’re applying the query to. The spatialRelationship should be set to a string defining the sort of check to run, such as "intersects", "overlaps", "touches", etc.

For example, let’s say you were writing an app in which the user was given the ability to draw a shape on the map and you want to identify the land parcels that intersect that shape. The basic steps for carrying out this sort of operation would be to:

  1. Get a reference to the land parcel layer.
  2. Create a Query, setting its geometry to the user-defined shape and its spatialRelationship to "intersects".
  3. Invoke the queryFeatures() method on the parcel layer.
  4. Write code that handles the FeatureSet returned by queryFeatures().

Have a look at the example below, which essentially carries out the Jen & Barry's site selection queries (minus the ones having to do with the recreation areas and highways).

See the Pen Spatial query demo by Jim Detwiler (@jimdetwiler) on CodePen.


Again, here is a list of the main points to take away from this script:

  • A series of variables holding the selection criteria are defined at the top of the script, making it easier to modify those values if desired.
  • Those values are incorporated into a couple of where clause variables using string concatenation.
  • After the cities and counties layers have loaded, the county-level query is executed and the resulting FeatureSet is passed along to a function called findGoodCities().
  • The findGoodCities() function is defined to store that FeatureSet in a variable called goodCounties.
  • In findGoodCities(), a Query is created to be used against the cities layer.
  • The expression goodCounties.features returns the array of Graphics that met the county-level criteria.
  • The JavaScript forEach method is used to loop through the array of county Graphics. On each pass through the loop, the geometry property of the cities Query is set to the geometry of the county in that iteration of the loop.
  • The cities Query is then applied to the cities layer, with the FeatureSet from that query being passed to the callback function called displayResults().
  • The displayResults() function is similar to how it looked in the previous example, creating a yellow graphic for the cities identified by the query. One difference is that embedded within the map() function is a console.log statement that prints the name of the city currently being processed. The expression graphic.attributes.NAME returns the value from the NAME field for the current city. Other field values can be retrieved in this way also, provided they are included in the outFields list when defining the Query. (Be sure to open your browser’s developer console to see the results listed.)
  • Note that cityQuery was used here to evaluate both attribute and spatial criteria at the same time.
  • We’ll see how to write content like the cities list to a more user-friendly place in the next lesson when we look at user interface development.

Assignment

Assignment jed124

For this week's assignment, please select from one of the scenarios below.  Regardless of the scenario you choose, I'd like you to follow these guidelines:

  • Use the JavaScript prompt() method to ask the user for a subset of data to map (e.g., a state).  We'll see better methods for getting user input and return to your selected scenario to "do it right" in the next lesson.
  • Configure your app so that appropriate information is displayed through pop-up windows.
  • Use ColorBrewer to obtain logical color values.
  • Look at the service in the ArcGIS Online Map Viewer or ArcGIS Pro to get a sense of its data fields and values.

Before reading over all of the scenarios, go to the sign-up page in Canvas to see which of them are still available. Here are the scenarios:

  1. 2020 U.S. Presidential election
    Using this U.S. Presidential election feature service, prompt the user for a state, then display a county-level map of the voting in that state. Use ClassBreaksRenderer to display the margin of victory for either candidate. Set up class breaks like 0-10%, 10-20% and >20% with smaller-margin counties drawn in a light shade of blue/red and the larger-margin counties in a dark shade. The best way is to set the renderer's valueExpression property to an Arcade expression that calculates the value for mapping.  Note: If you'd prefer to map a different election, such as 2024 (I couldn't find a service with county-level data for that election), that's possible.  Just run your idea by me first.
  2. World cities
    Using the World cities feature service, prompt the user for a continent, then display the cities in that continent symbolized by population (varying either size or color). This scenario is a bit trickier than the others in that there is no continent identifier in the Cities layer. To identify the correct cities, you can query this Continents feature service to get the polygon geometry of the selected continent, then use a spatial query to find the appropriate features in the Cities layer.
  3. U.S. National Parklands
    Using the U.S. National Park lands feature service, prompt the user for a National Park Service region, then display the parklands within that region. Use different symbols for the common park types (National Monument, National Historic Site, National Park, National Historical Park, National Memorial) and depict all other types in a class called Other.
  4. U.S. County Demographics
    Using this Popular Demographics of the United States feature service, prompt the user for a state and display the counties in that state based on the proportion of its population comprised of a generation of your choice (Greatest Generation, Baby Boomer, Gen X, Millennial, or Gen Z). Don't worry that the service is listed as deprecated; the updated version is in beta and requires a subscription.  Make sure you base your app on the county layer, not the web map.  Similar to the election scenario, you can calculate the population percentage using valueExpression and an Arcade expression in either ClassBreaksRenderer or VisualVariable.
  5. Alternative fuel stations
    Using the alternative fuel stations feature service, prompt the user for a state, then display the alternative fuel stations in that state. Use PictureMarkerSymbol to show the fuel types as appropriate icons. I've put together a set of free icons.
  6. 2016 EU Referendum in the UK
    Using this 2016 EU Referendum feature service, prompt the user for a region (e.g., Northern Ireland, North East, North West), then display the voting districts in that region based on their remain/leave vote. The remain/leave vote percentages add to 100, so you can safely base your ClassBreaksRenderer on one of the two percentage fields.  As with the presidential election scenario above, set up vote margin classes like 0-10%, 10-20%, >20% and use shades of one color to depict the remain districts and shades of another color to depict the leave districts.  Note: this service will not display properly in a SceneView using SDK version 4.8 or higher.  So use a MapView or version 4.7 of the SDK.
  7. Wildfires
    Using this Historic Wildfire Perimeter Service, prompt the user for a year (between 2000-2018), then display the fire perimeters from that year.  Render the layer by fire size in acres using either ClassBreaksRenderer or VisualVariable with an appropriate color ramp.
  8. Hurricanes
    Using this Hurricane feature service from Living Atlas, prompt the user for a year, then display hurricanes from that year.  The service has a default symbology, but I'd like you to override that using dotted lines (in a 2D Renderer because it's not available in 3D) rather than solid and a light-to-dark color ramp to indicate the hurricane category (using the Wind_Speed field).  You'll find the ClassBreaksRenderer class to be helpful. 

Note: ArcGIS Server-hosted feature services are configured by default to return a maximum of 1000 features.  This limitation comes into play for some of the scenarios above and is clearly not ideal.  The publisher of the service has the ability to override this setting, but app developers like yourselves do not.  One solution for developers is to query the service recursively, such that the full set of features is retrieved in 1000-feature chunks.  If you found that you were able to complete the assignment fairly easily, you might consider researching and attempting such an approach.  But we will not be expecting you to work out a solution to this problem if it affects your app.

Final Project Initial Concept

Review the Final Project requirements. Think about what topic you'd like to use as the basis for the final project app and post your ideas in the Lesson 7 discussion forum. This does not need to be a complete proposal, but I want to know you've started thinking about it. What would you like to do or display? Have you identified any data to use? Is there any specific interactivity you'd like to include or explore? 

This is only a concept, so your idea can evolve or completely change over the next few weeks. You won't be evaluated based on what you propose now, and it's also OK if you don't know some of these details yet. At a minimum, I want to know you are thinking about a topic. I'm sure you're ideas about interactivity will evolve over the next few weeks, too. If you are unsure about something, share that too. 

Please also read through other students' responses and share any helpful thoughts you have about their concepts. 

Deliverables

This project is one week in length. Please refer to the Canvas course Calendar for the due date.

  1. Remember to sign up for a scenario before you start working on it.
  2. Edit your e-Portfolio so that it includes a link to your map, and post a link to your e-Portfolio through the Assignment 7 page.  (80 of 100 points)
  3. Beneath the map or on your e-Portfolio page, provide a short description of your app and how you approached solving this project. Reflect on what you learned from the lesson, what you found challenging, and include what resources you used to get your code right.  (20 of 100 points)
  4. Post your initial ideas for the final project app in the Lesson 7 discussion forum. (Ungraded, but -5 if missing)
  5. Complete the Lesson 7 quiz.

Summary and Final Tasks

Summary and Final Tasks jed124

In Lesson 7, you learned how to add place-finding capability to an app through the use of locator services and the Search widget. You also saw how that widget can be used to search for values in layer attribute tables. Finally, you looked at the use of the Query class in defining queries (spatial, non-spatial, or both at once) and how Query objects can be used as inputs to methods that return feature counts, feature IDs, or the features themselves. Importantly, you learned how promises play a prominent role in handling the results of asynchronous operations both in geospatial and more generic JavaScript applications.

One last note on the topic of queries: in looking at the SDK or the source code of other apps, you may come across a class called QueryTask. This class was used in earlier versions of Esri's API to execute queries defined in Query objects. While queries can still be executed using a QueryTask, you should note that the QueryTask class supports only layers derived from ArcGIS Server services (i.e., it does not support ArcGIS Online layers).

One aspect of this lesson that you may have found dissatisfying was that the queries were all hard-coded into the apps. There was no way for users to supply their own query parameters, as we're all used to seeing in more sophisticated apps. The good news is that Lesson 8 is all about UI development. With the knowledge you'll gain from that lesson, combined with what you learned here in Lesson 7, you should be well on your way to developing your own killer apps.