Overviews in LP360 (Scaling and Pyramiding)

LP360 has the ability to manage very large data sets. Unlike some other LIDAR processing applications, LAS files in LP360 are visible as LIDAR content regardless of the "zoom" level. In this article, I will explain how we accomplish this and what it means in terms of interacting with LP360.

When you are zoomed out on an image, you cannot see the same level of detail that you can see when you are viewing the image at one-to-one. This is because the screen on which you are viewing will not support the image resolution. For example, suppose you are viewing data in a window on your computer that is 100 by 100 pixels and that your image is 1,000 by 1,000 pixels. If you perfectly fit the image to your display window, each screen pixel covers 10 x 10 image pixels. To accommodate this difference, display software will amalgamate each 10 x 10 group of image pixels into a display pixel by a process such as averaging so that the image can be displayed. However, here's the rub; to display the zoomed out image in the 100 x 100 window, all 1,000 x 1,000 image pixels have to be fetched from disk. This means (in our example) that 1 million pixels (1,000 × 1,000) have to be read to feed a display of only 10,000 pixels (100 × 100). This is a ratio of 100 to 1, a very inefficient (and slow) process.

To address the above issue, most image display systems make use of pre-computed "zoom levels" (called image pyramids or reduced resolution data sets, RRDS). This pre-computation might generate a sub-sampled set of images that are 500 x 500 (the "first" overview), 250 x 250 (the second overview), 125 x 125 and so forth. The display logic of the software then selects the best match from the overviews for the current zoom level and display window size. In our example above, the system would select the 125 x 125 overview and sub-sample this to the 100 x 100 screen. This then reduces the data read requirement from the original 1 million pixels to 15,625 pixels (125 × 125). If it takes 1 microsecond to read each pixel (your system is certainly faster than this!) then the read time is reduced from 1 second (the time to read 1 million pixels) to 16 milliseconds! Thus you can see that this image pyramiding is vital to a high performance system.

We do a similar process with LP360 to enable fast displays. These are the "qvr" files that you see in your data directory beside the standard LAS files. However, generating a reduced resolution LIDAR file is much more complex than generating an image pyramid. To generate an image pyramid, all that is required is some sort of averaging (for those of you interested in the technical details, this is usually a Lagrangian or Gaussian subsampler). However, the data in a LAS file is actually a sequence of database records (one per point). There are many fields associated with each point and these records are generally not amenable to an averaging process. Consider "class" for example. How would one average a "ground" point and a "Medium Vegetation" point? This is a complex consideration but we solve this for you in LP360.

Hopefully that was helpful background information for you. Now, the question of interest to you is how all of this impacts your use of the software! In the following examples, I will use screen shots from the 64 bit standalone version of LP360. However, the 32 bit ArcGIS® version functions the same when it comes to LAS overviews. You may have noticed the Point Drawn readout in LP360 (Figure 1). This readout indicates the number of points being currently drawn and, more importantly, what percentage this represents of all points that would be drawn if all points that would fit the current display and filter settings were displayed. In the example of Figure 1, only 1.56% of the possible points are being displayed (in image parlance, this would be similar to displaying the second overview).

It is important to note that all aspects (pun intended) of the display in LP360 are governed by the point display density. For example, if you render contours, we will (behind the scenes) render the current resolution of LIDAR points as a Triangulated Irregular Network (TIN) and create contours based on this TIN. This is why you can see the general shape of contours change as you zoom the display.

You can force the display to 100% using the pyramid icon (red and white) of Figure 1. This will result in a full density display if 100% of the points can fit in your display buffer (discussed in a moment) or you may get an overflow indicator. An overflow is indicated by the text "OVF" in the point count display ( Figure 2). Note that in this example case, the maximum number of points that can be rendered in the display is 2.5 million.

You can modify the maximum number of points that can be rendered when forcing 100% in the Performance tab (Figure 3). The Performance tab is accessed from the Active LAS Layer Properties tool on the main LP360 toolbar. Be very careful when changing the performance properties! They are setting the amount of virtual memory available for the various functions in LP360. Memory is a very limited resource on 32 bit operating systems so we recommend great care in changing these on that platform. Unfortunately, ArcGIS® Desktop had not yet been moved to 64 bit architecture so you are fairly limited when it comes to memory management. As you are no doubt aware, each license of LP360 includes the 64 bit standalone version so, if you have a lot of RAM in your machine and are running a 64 bit Windows Operating system, you can really get some large point displays! There is more information about this in this month's "Tool Tips."

Note that Interactive classification (LP360 Standard edition) requires that the points be at 100% density. If this were not the requirement, you would be in the situation of reclassifying points that were not visible in the display! The pencil tool on the left of the Classify toolbar not only enables interactive classify mode but also forces 100% display density (note that the maximum number of points to allow for interactive classification is controlled by the MEM parameter). If you are zoomed out too far for the current data set, the pencil tool will remain depressed but the edit tools will not enable and the density will read "OVF". If this occurs, zoom in a bit. The most common place I run in to this issue is when I want to reclassify large areas back to "Unclassified." However, this type of large area reclassification can be easily accomplished by using the Basic task in the Point Cloud Task functions.

Note that in some areas we are not as smart as we should be (and this will be corrected in a future update to LP360). For example, if you are using classification flags (this will be the subject of this column in the next issue of LP360 News), we use the point count of the classes on which the flag is set rather than the count of flagged points. This can be an issue if you have derived Model Key Points (LP360 Advanced) and have indicated the MKPs via the MKP flag as opposed to moving these points to a different class. If you are filtering the display to show only points flagged as MKP, you may need to force 100% to see all of these points. A word of caution on forcing 100% displays. ArcGIS® can be a bit finicky when it runs low on available physical memory. Forcing 100% display when the current density readout is very low (say 10%) can consume a lot of memory, slowing Arc to a crawl.

The pyramid strategy of LP360 is extremely powerful. For example, I routinely load an 800 square mile project of 1 meter post spacing LIDAR data in a single session and have a full view of the project. I can also draw a single profile line across the hundreds of flight lines that comprise this project (when colored by Source ID, a common check for flight line edge matching). Hopefully the discussion of this feature has helped you understand the nuances and compromises of using overviews. Remember, if you need to work with very large data sets, use your copy of LP360 x64.