The NTLIS Imagery Catalogue offers improved file storage and metadata management. The design and functionality features are described in the attached document.
There is a requirement to better capture the metadata relating to satellite and ortho-rectified imagery assets held by the NTG, in particular by DPI and NRETA. This will make it easier for the data custodians and end users to find and use the assets, and will reduce duplication of storage and procurement across the agencies.
The capture process will require the development of a management application and database into which custodians can input the metadata information. The scope of this document is to describe the management application and database, and requirements of custodians for management of the data.
Currently the Department of NRETA has a Microsoft Access database that contains a catalogue of remotely sensed data aquired by the Department. The new system is being designed in the Oracle / web environment, complying with the NT government spatial dissemination database paradigm.
Along with the development of the metadata management application and database is the establishment of online storage for the imagery itself, allowing for easy extraction of data once the desired image asset has been identified. Imagery extraction allows users to download an image to their machine if the various online linkage methods are not sufficient.
A metadata harvester has also been developed, able to extract georeferencing details such as extent and coordinate system and create a metadata record including all online linkage fields, minimising the amount work required of the custodian.
The fields of metadata able to be captured in the Imagery Catalogue have been determined by NRETA and DPI image custodians and are listed below. Completion of these fields will ensure each image asset is able to be described in the ISO 19115/19139 metadata standard. For further information regarding NT spatial metadata go to: http://www.ntlis.nt.gov.au/forum/ntlis_documents/ntlis-metadata-tool/
Table 1 (below) is an outline of the fields and expected data input for the main database table. Fields in bold are able to be automatically harvested from the imagery itself, as described in the next section.
|UFI||Unique Feature Identification: auto generated|
|START DATE||Date of capture for the data or earliest date for collective dataset (mosaic)|
|END DATE||End date for collective dataset ie mosaic, time series dataset|
|CUSTODIAN||Custodian name, sourced from a list|
|SENSOR||Landsat, SPOT, MODIS, etc|
|SENSOR TYPE||Panchromatic, Multispectral, Hyperspectral|
|BANDS||Specific to sensor or what was purchased ie 1-7 or Infra-red|
|RESOLUTION||Pixel size represented in meters, e.g: 0.25|
|PROCESS LEVEL||Raw, projected, rectified, mosaic, calibrated|
|PROCESS LEVEL COMMENTS||Brief comments regarding process steps ie calibrated to yr2000 data|
|FILE FORMAT||.ers, .img, .dat, .jp2, .bil, .udf|
|FILE NAME||Name of the file where it is stored|
|STORAGE LOCATION||Where cd/dvd physically located i.e. Berrimah|
|STORAGE NO||CD, DVD or Tape reference number|
|LAYER NAME||Link to image data file, used for online linkages. Usually same as the file name|
|WMS SERVER URL||Link to WMS GetCapabilities document|
|ECWP SERVER URL||Link to image via ECWP protocol|
|DATUM||AGD66, GDA94, AGD84|
|PROJECTION||MGA50-54, Lat Long|
|USAGE RESTRICTIONS||Coded i.e. 1=Unrestricted, 2=NTG restricted, 3=In-confidence, 4=Protected, 5= Highly protected|
|PROJECT NAME||Deptartment project name acquiring remotely sensed data|
|USAGE RESTRICTIONS TEXT||Describe copyright limitations|
|PROVIDER||Company the image was sourced from, e.g: GeoImage, SKM, ACRES|
|PREVIEW||Link to a quicklook image, used as a thumbnail for users search through images|
|PURCHASER NAME||Nominated officer purchasing data|
|DATE GOOGLE FUSED||Date the image was fused into the NT Visualiser world|
|DATE GOOGLE RETIRED||Date the image was removed from the NT Visualiser world|
Table 1: Imagery Catalogue Fields
Automatic Harvesting of Metadata
NTLIS has developed a procedure whereby custodians are able to copy imagery into a specific network location and have it harvested and inserted into the Imagery Catalogue, assuming the file meets certain minimum criteria. The bolded rows in Table 1 are able to be automatically harvested. This process automates the extraction of spatial properties and online linkage information, as well as copies the file itself to a secure, accessible location. At this time the automatic harvesting only works if the imagery is georeferenced and stored as a JPEG 2000 file.
The information not able to harvested can be filled in after the image has been inserted by the Harvester.
Figure 1 (below) describes the structure of the Imagery Catalogue and its associated components. As illustrated, the metadata capture in the Management Application is stored as a regular Oracle Spatial table on the NRDB dissemination database, allowing a range of clients including MapInfo, ArcGIS, FME and custom web services to query and display the information.
Importantly, the Management Application of the Imagery Catalogue is only used by custodians, in the future there will be a Discovery Application developed allowing internal and external users to search the catalogue and optionally extract imagery data.
Figure 1: Technical Architecture
As per the architectural diagram, the data will be stored and made available in the NRDB dissemination database so that 3rd party discovery applications can easily be built. In addition, NTLIS will develop an 'official' Discovery Application, as is the case with the current NTLIS Aerial Photography Index. The discovery application will allow users to search on any field stored in the metadata, as well as its spatial extent.
Database search criteria will include:
- Drawing a boundary on a map; e.g. NRETA maps integration
- Path / row searches – Landsat, SPOT
- 1:250K Map Sheets
- Date or date range
- Spatial datasets i.e. Bioregion, pastoral or parcel name/number, district
- A combination of all or some of the above
Images that have been made available on an online data storage point will be able to be extracted in a range of formats and coordinate systems for use in their 3rd party program, similar to the Spatial Data Download function in ILISMaps (NT Government only). In addition, users will be able to connect to online imagery using the WMS and ECWP protocols, removing the need to store the image locally.
A number of free plug-ins supporting output formats to assist clients in utilising the images for a variety of purposes or software they utilise are available.
Imagery Catalogue Migration and Management
Content from the old MS Access Satellite Imagery Database will be migrated to the new online system. The remainder of the database entry is the responsibility of the respective custodians, for the data they are responsible for at the divisional level; i.e. Biodiversity, Bushfires, Resource Management, Land & Water etc. Custodians need to check the data transferred from the Access database and fill in all the appropriate fields relevant to the individual dataset.
Custodians are to have edit access to only datasets that they are responsible for, all other files will be view/read only.
Custodians are the persons best placed to update and maintain the database, due to their knowledge of the data, requests for the data by clients, and physical location of the data.
- Custodians will be responsible for;
- Data correctness within the database to ensure integrity
- Loading required data files onto the server
- Generate metadata for all data sets available in the database. This will in time allow the database to have web based delivery similar to DPI Aerial Photography Index or through NRETA Maps. Besides this requirement Metadata needs to be up to standard for the purposes of correct data management procedures.
- Generating JPEG2000 (lossless) files with the most appropriate bands and stretch applied to generate quicklook images for use by clients. Unless this can be automatically generated.
Procedure for Updating Database and Acquiring Imagery Data
Custodians will be responsible for updating the database with any new data acquired, filling required fields in the database, generating quicklook and metadata files and linking these files and the original data files to the database as described previously. Before purchasing new data it is advisable to adhere to the following steps:
- Ascertain the appropriate satellite sensor to meet the requirements of the project/s.
- Check the database for similar data types previously purchased for the area of interest and possibly adjacent areas. This could help determine what type of data, time of year, and who else may benefit from a new purchase.
- Check with other projects or personnel in regards to the purchase and whether mutual benefits from a shared purchase can be achieved. This may require either an email system to all custodial managers regarding the proposed purchase and gain feedback or a centralised forum by which to communicate.
- These steps will hopefully generate a greater understanding of what is to be purchased (sensor, process level, date, area etc) by various groups or projects and possibly maximising benefits for other projects through sharing of data and costs, and knowledge of other projects in operation.
After purchasing new data or creating an improved version of the purchased data (eg rectified, calibrated, mosaic) the following steps need to be adhered to maintain and update the database:
Metadata generated for the data – linked to the database.
- A copy of the image data file to the storage facility – linked to the database.
- A copy of the quicklook generated and placed on the server – linked to the database.
- Geometry for the image data generated for search capabilities linked to the database. This maybe the same as a previous purchase covering the same area, hence the same geometry can be used.
- All other fields need to be filled in for the new data, a proportion of which can be accessed from the metadata.
- Check that all links to image data, quicklooks and metadata are active and correct.