It has been six years since the National Geo-spatial Information (NGI) undertook a project to restructure and integrate all its data into a single data repository. This article provides an inside look at the processes, technology and data content at the NGI since the inception of the datamodel project.
National geo-spatial Information (NGI) provides many geospatial products to the geospatial industry, the most important product being the aerial photography which is the base product from which all mapping products are produced. Products derived directly or indirectly from the photography include aerial triangulation data, digital elevation model (DEM) data, orthorectified imagery, topographical vector data and a variety of mapping products at different scales. In order for the NGI to meet the demands of the dynamic changing environment due to the occurrences of aggressive social and economic activities, products have to be produced through seamless processes throughout the enterprise with optimal access to the data repository, hence the development of the Integrated Topographical Information System (ITIS) at the NGI. The increase in the temporal acquisition of digital aerial photography has led to an increase in storage demands which is further compounded by an increase in the number of product revision cycles requiring more storage. processes, technology and data content and future opportunities will be addressed in this article. A typical daily life cycle at the NGI is illustrated in Fig. 1.
Fig. 1: Typical daily life cycle at NGI.
The heart of all the processes and product activity is the data centre which is the repository for all metadata, imagery and products from which the ITIS portal and geospatial portal will consume and derive information for presentation. The datacentre consists mainly of an oracle spatial cartridge database, tape robot archive system and storage area network consisting of 240 TB of disc space. The SAN is made up of a collection of Enterprise Virtual Array (EVA) storage systems provided by Hewlett Packard (HP) and is managed by the StorNext Storage Manager application. The archive system is supported by a tape system that stores two copies of each data element stored in the SAN. The databases are also allocated SAN storage and are backed up onto a tape robot system using the Legato application software.
The establishment and management of the data centre is a very expensive exercise. EVA enclosures and servers have to be purchased and maintained, discs and tapes need purchasing to meet the increasing data acquisition and product production cycles. Software licensing and maintenance is required, uninterrupted power supply (UPS) and alternate power source generators are required, air conditioners are required to keep all the electronic equipment in the datacentre cool and finally a team of IT specialists needs to be contracted in order to manage the system. Additional costs come later with the replacement of servers and EVA enclosures due to advances in technology and the purchasing of additional storage management licences, which is charged proportionally to the amount of storage that needs to be managed on the SAN and tape robot system which is ever increasing with revision cycles of data.
Fig. 2: A geographical name search on 50 K maps.
Processes and technology used at the NGI
National Survey Network
The national network consists of active and passive beacons. Strategic passive beacons are identified for maintenance annually. The maintenance is performed by the NGI beacon building team. The active network is monitored daily and data continuously streamed to the control centre at the NGI. GNSS data is packaged into products and is available online.
Image acquisition
The NGI has acquired its own digital mapping camera (DMC) for digital aerial photography acquisition. National imagery acquisition is undertaken within a three year cycle. A partnership is established with a service provider, through bid process, to acquire two thirds of the country utilising the NGI DMC and the other one third is acquired from the industry via another bid. All photography is acquired strictly according to the Digital Aerial Imagery Standard and flown according to mission/flight planning prepared by the NGI.
Fig. 3: An extended list derived from results in Fig. 2.
Mission planning
Flight planning is prepared by the NGI and is accepted after consultation with the service provider before the image acquisition cycle is started. The NGI will scrutinise the supplied imagery for conformance with the flight plan details. The software utilised for the flight planning is Leica Mission Pro from the Hexagon group.
Image management and serving
A formal naming convention has been set up for all image products that contain the job number, strip number and photo number within the image identifier. The naming convention forms part of the validation process during database population. Ancillary data files also have formal naming conventions and the data within is specifically formatted/structured so that it can be consumed seamlessly into process activities down the line without any formatting directly from the SAN and without any local disc copying thereof. On acquisition of a job the images are immediately archived into the system and information related thereto is ingested automatically into the ITIS database. Validation is carried out on ingestion and any image data not conforming to the flight plan index is logged for inspection and correction.The ingestion is carried out via the Terrashare image server client application supplied by the Intergraph/Hexagon group.
On successful ingestion the image client creates a logical structure corresponding to the physical storage structure from which images and stereo photogrammetric 3D image models can be viewed by navigating the implied flight plan created by the ancillary support flight data. Stereo capability is very useful for the topographical vector data capture component in carrying out photo interpretations during their capture process. The imagery is served throughout the enterprise immediately after ingestion via the Terrashare Client application. The ingestion process also updates the central database with metadata so that the availability of the images can be derived by the portal applications for presentation and quoting thereof by the sales team.
Fig. 4: A coordinate search on the aerial photography library.
Aerial triangulation
The aerial triangulation process is seamlessly performed directly from the SAN without copying any images to local drive or reformatting thereof. The ancillary flight data support files enable the consumption of the data directly into the aerial triangulation software for immediate processing. Aerial triangulation is performed using Image Station Aerial Triangulation (ISAT) and PATB software applications. The output is stored on the SAN in specific formats so that it can be seamlessly consumed into the DEM and orthorectifications application.
Ortho rectification
All the components for othorectification, e.g. DEMS, calibrated images and final adjusted image orientation parameter files, are stored within the SAN as outputs from previous processes. The rectification process can seamlessly consume these components directly into the orthorectification application software. Areas consisting of hundreds of images can be consumed for processing during a single session. Rectification is performed on a single job area at a time and the final output is 100 mosaic images that correspond to the 1:10 000 map reference index. The orthorectified images are stored on the SAN and ingested through the Terrashare image client for serving to the enterprise and population and validating of metadata into the database for publishing in the ITIS portal application. The most important component for orthorectification is DEM, which is performed after the aerial triangulation process using the Usmart and Image Station Automatic Elevation applications supplied by SmartTech and Intergraph/Hexagon respectively. The orthorectification process is performed using the Image station Orthopro Application supplied by Intergraph/Hexagon.
Fig. 5: A coordinate search on the aerial photography library; listing specifies job photos.
Data management
The topographical vector data is stored in a 10g Oracle Spatial Cartridge database and is a fully versioned implementation. The version management implementation is “just in time”, i.e. rows in the database are locked at the time of data manipulation and remain locked to particular users for exclusive use until it is returned/committed to the database, unlike conflict management which allows for multiple changes on a single row and then later resolving the conflicts for final committal. All modified, deleted and inserted data is recorded on the database as belonging to a revision set and these changes to the database will only become visible in the database as soon as the supervisor has accepted the revision set for committal, i.e. the authorised owner of the revision set will be able to observe all the changes in the revision set while the online live database user will only have access to the current state of the database and not the revision set.
Versions of all edited data are stored within the database as a temporal entity, hence the database state is continuously changing and it consists of both current and temporal data. The more edits you have on the database the larger the storage demands so we need to make sure that the changes that are occurring on the database are for temporal updates and not correcting bad digitising or else we will have both versions of edited and temporal data stored. The NGI has implemented topological data structuring rules that are checked against the data by a supervisor before final committal to the ITIS database in order to minimise the edits due to bad digitising. The management of versioned data is performed via Oracle Workspace Manger from Oracle and GeoMedia Transaction Manager from Intergraph/Hexagon.
Fig. 6: Default landing page with base datasets present via OGC services and Google
map backdrop.
Topographical vector data capture
Topographical vector data is primarily performed using NGI orthorectified imagery and ancillary data sourced by the NGI as an overlay for aiding the capture process. Data is interpreted and mono-digitised from the imagery into a revision set before final committal to the database. Geomedia Professional and GeoMedia Transaction Manager is utilised for performing this procedure. Since the inception of the data model project, all data is captured according to the new classification system. The new data is now available from the NGI sales team.
Fig. 7: WMS implementation of the clipped 1:50 000 maps.
Cartographic map production
The cartographic products are by-products of the topographical vector data and ancillary mapping data. Medium and small scale, 1:50 000, 1:250 000, 1:500 000, 1:1 000 000 and provincial maps are produced using the Gothic Developer Digital Cartographic mapping system from 1Spatial solution and the large scale, 1:10 000 orthophoto, maps are produced using the Esri ArcGIS product. Images and vector data are consumed seamlessly into this environment from the SAN and Oracle spatial database. Feature Manipulation Engine (FME) is used as a data translator for ingesting the topographical vector data from the oracle spatial database into the Gothic spatial database for map productions. All mapping products are archived in the SAN according to formalised file structures and naming conventions. The map products are also ingested into the terrashare image server environment for enterprise viewing, metadata ingestions and validation for presentation through the portal application.
Fig. 8: WMS of orthorectified imagery and image photo centres of the photography library.
Portal application processes
Portal processes are performed by the sales team for searching for client product requests, creating quotes thereof and stock management. All imagery and map products that have not been ingested through the Terrashare client application will not be available to the web portal application. The portal application can derive information about products via geographical name searches, coordinate searches (DMS and DD) and map reference index searches. The derived data can be viewed and added to a shopping cart facility for final quoting. The data is derived from the ITIS database. Figs. 2 and 5 illustrate the various types of searches that can be performed via the ITIS portal web application. Fig. 2 is the initial search for maps containing the geographical name “Newlands” and Fig. 3 is the extended list of the desired map selected, the extended list will list the latest copy that is available on the shelf and all back copies of the map including the date, edition, map ellipsoid, geo-referenced ellipsoid and the processing method. The desired map may be previewed or selected for purchasing and added to the shopping cart for invoicing. Figs. 4 and 5 illustrate a coordinate search, DMS, on the aerial photography library. The list of related jobs are served up and presented in Fig. 4, and Fig. 5 displays the specific photos that relate to the coordinate entered and job derived for selection and adding to the shopping cart.
Fig. 9: WMS query result invoked by selecting an image photo centre.
Geospatial portal (Current prototype created)
The Geospatial Portal is in its infancy. A prototype implementation has been concluded and is being tested for comment and improvements before final implementation, illustrated in
Figs. 6 to 12.
Opportunities and strategies to consider
The NGI has a large variety of geospatial information content which includes the following.
Topographical vector data: cultural, hydrography, hypsography, LULC, physiographic, transportation and geodetic.
DEM data
Aerial photography, current and historical
Orthorectified imagery
Geographical names
Flight plan indexes
Landcover thematic raster
Various small scale map series
Medium scale map series
Orthophoto maps – large scale
Fig. 10: Google terrain map implementation within the portal.
It is important that the geospatial community has uninterrupted access to the data. The data has been accumulated at huge cost to the tax payer. Access to the data can be perceived as being restrictive. Data is via pre-packed product DVDs or hard drives that are sent to the sales team for downloading onto external hard drive. A trend is to request all the data, national coverage, even though only a portion of the country is required. This could be as a result of the client not wanting to incur repeat costs related for courier services or business entrepreneurs wanting to setup their own portal facilities which may incur further costs to the tax payer in acquiring access to that data. A typical order scenario may be a request for all the orthorectified imagery of the country from five clients at the same time, this results in the downloading/copying of 5 x 4 TB = 20 TB of data from the SAN over the LAN to the external hard drive at a single instant. On average 36 TB of data is transferred to external hard drives monthly by the NGI. Increased network traffic has an adverse effect on the daily production activities within the enterprise. NGI data is copied and scattered throughout the country, silos of the same data.
The NGI foots the bill for the entire storage requirement as mentioned in the introduction, servers, tape robot system, storage tapes, storage discs, EVA enclosures, software maintenance, hardware maintenance, storage licences for the SAN and tape system which is charged per megabyte and is ever expanding, UPS, air conditioners, backup power generators and the IT specialists that are responsible for management and sound daily operation of the systems. These are the challenges the NGI faces on a daily basis.
Data is very well organised and structured at the NGI, it can be easily consumed and presented through a geospatial portal. The internal access and processing of the data within the LAN is good. Web software applications are available, pilot implementation completed, but internet band width is restricting publishing on the World Wide Web. OGC services have been available for a while and cannot be deployed due to the restrictive nature of the WAN (wide area network) and the band width that is available to the NGI. As a result data is duplicated throughout the country and on private geospatial portals. The cost of storing and managing the data is increasing annually and a new strategy should be considered.
Fig. 11: Landcover data presented directly from the SAN into the portal environment using a Catalog service.
The NGI could consider going it alone or look at other solutions such as cloud technology. Cloud technology is an opportunity and not a threat and should be considered for deployment. The creeping nature of the cost of maintaining and equipping an internal datacentre must be weighed up against a cloud solution. A cloud can be experienced as a virtual datacentre and all the hardware and operational elements become implicit to the user and matters related to data can become the main focus of the enterprise, i.e. create the data and allow the IT experts to optimise the physical environment. Technological trends and infrastructure/data security challenges are changing rapidly and a service level agreement with a supplier of a cloud facility may ensure that the datacentre is always secure and equipped with the latest IT technology.
Government could consider procurement of a cloud service for all government departments participating in the geospatial community, together is better. There will inevitably be
policy challenges.
Fig. 12: Zoom in on the landcover data crawled from the catalog service.
Acknowledgement
This paper was presented at AfricaGEO 2014 and is republished here with permission.
References
[1] Software application: Intergraph, viewed 10 April 2014, www.intergraph.com.
[2] Software application: ESRI, viewed 10 April 2014, www.esri.com.
[3] GeoSpatial Portal: INSPIRE, viewed 10 April 2014, www.inspire.ec.europa.eu.
[4] Geospatial and location standards: OGC (Open Geospatial Consortium), viewed 10 April 2014, www.opengeospatial.org.
Contact Raoul Duesimi, DRDLR, Tel 021 658-4372, rduesimi@gmail.com
The post Processes and data content at the NGI appeared first on EE Publishers.