Catalog v0.8 Release Notes

April 2019 - Catalog v0.8 (beta)

Highlights

  • New: Ability to include projected datasets in the PW portal
    • Includes major backend changes to search and filter interface and download interface
    • Includes major front-end and back-end changes to data preview interface
    • Adds python configuration on web server for on-the-fly projected graphics generation
    • See full release notes for more details
  • New: Download interface features
    • Adds stride subset options and additional download formats
    • Adds alerts of download size while filling out the form and alerts users to large downloads before download begins
    • Suggestions are now provided for help with data subsetting and alternative access methods
    • See full release notes for details
  • New: Preview interface features-Adds land mask overlay, adds new base map with gray background.
  • Improved: Updated all code to python 3.6
  • Improved: Updated all code to https/ssl
  • New: Datasets Added:
    • METOP ASCAT winds, 4 ERDDAP datasets as 1 catalog entry
    • OSTIA SST, 1 ERDDAP dataset as 1 catalog entry
    • NSIDC Sea Ice Concentration CDR, NRT CDR – 8 ERDDAP datasets as 4 catalog entries
  • New: Requirement for projected dataset integration. ERDDAP dataset metadata will require an attribute proj_crs_code which distinguishes projected datasets from standard datasets.

Details on Key Features

New Feature details: Ability to include projected datasets

This release includes broad changes to all python scripts, templates and javascript to accommodate projected datasets which do not use latitude and longitude. There are broad changes across the code base to move from lat/lon references to use ycoord/xcoord instead.

Map Preview Interface Functionality

This release includes major changes to how the map preview interface functions and major revisions to server side code for map graphics generation. Moved from using ERDDAP generated graphics to generating on the fly graphics in python. NetCDF data is pulled from ERDDAP to generate the preview images. This replaces the previous code that used GDAL to reproject rasters from ERDDAP. Working with the data directly instead of an image has many benefits including higher resolution graphics for the preview display, customization of preview images and providing the framework for future data visualization plans.

The interface was previously limited based on the time it took GDAL to reproject the rasters. Now, in the same amount of time python generates projected images that are higher resolution. Customization of the preview display graphics. Because we are generating the images ourselves we can set transparent values, use any color pallete whether or not it is in the host ERDDAP, etc. These changes also set the stage for a “make a plot” interface that will function similarly by pulling data from ERDDAP for on the fly web plotting.

Legends are now generated in python, not by ERDDAP, and we can customize the color palettes more freely. Each parameter has a color palette and range specified in it's corresponding catalog configuration file. The default color bar ranges are optimized for the polar regions.

Map Preview Interface Display Changes

  • Projected datasets offered map previews only in corresponding map views. Current display options are arctic, antarctic and global, with potential to add Alaska.
  • The layout was improved in several ways. The page layout is improved for large wide screens where previously the map looked like it was short/cut off. The map area is now set to 8/12 width on large screens, with a square map and white background.
  • Arctic and Antarctic maps now have the same default zoom levels.
  • Legends moved to below the map.
  • Single Download button in the menu.
  • Map mouse hover displays coordinates in lat/lon
  • Data previews now have a land mask available (and applied by default).

New: Download Interface Features

Stride

Stride inputs were added for latitude, longitude and time. Help text references how to use these to reduce request size.

New formats

This release adds download formats that are options in ERDDAP but were not integrated with earlier versions of PolarWatch. Previously we had only netcdf download as an option. ERDDAP griddap output formats were reviewed and two additional download format options are included in this release.

  • .mat - This behaves very similarly to the netcdf output and will be useful to those who work primarily in Matlab
  • .png - Providing images could be a “can of worms” and we have set aside time in the spring to implement a larger solution. The feature in this release provides an unprojected, non-transparent png image of a predetermined output size.

Other ERDDAP formats were considered but not included. A new link is provided on the form directly to the dataset ERDDAP download page if users are interested in these or any of the other many formats availalble through ERDDAP for data and metadata queries.

  • .tif - The output was not as expected, the ERDDAP response was a low-resolution image file in grayscale. PolarWatch users have requested a geotiff option that has more of the features of a netcdf format download along with image customization options. We plan to include a geotif output as an option in an upcoming release.
  • .kml - The output of the ERDDAP KML is not what PolarWatch would like to provide with a kml download option. We will plan to add a KML option in the future that is a single polygon showing the spatial extent of the query.
Auto-estimate download size

This feature increases the user-friendliness of downloading data through ERDDAP. Many users have complained that they don’t get an error message until after clicking download and that the error messages could be more descriptive/helpful. This is an initial attempt to address that user feedback. With this release, download size is dynamically estimated and presented to the user each time an input is changed on the form. The download size estimate is displayed next to the form and highlighted with warning or error colors if the size meets certain thresholds.

  • If the size estimate is above 2GB (the ERDDAP limit), the download is prevented and help text is displayed.
  • If the request size is over 30MB but less than 2GB a warning message is displayed upon clicking download, alerting the user that the download may take a while depending on their connection speed.

In either case, help text is displayed to highlight the subsetting features of the form and to make the user aware of potential alternative access methods like the underlying files or THREDDS. The alternative access method information is pulled from ERDDAP sourceURL. If the source is unknown or not available this help text is not displayed.

Additional Minor Features

New: Page layouts include dataset info on preview and download pages

Fix: Fixed bug with display of NOAA VIIRS card on search and refine interface where duplicate entry was being created and did not filter out.

Improved: Changes to the catalog entry config file structure: Catalog entry config files been cleaned up a little to remove fields no longer in use including duplicate color bar fields.