Metadata extraction

METAMOD supports extracting metadata from netCDF files. The metadata extraction is triggered in three different ways.

Which method is used for triggering uploads are decided by the EXTERNAL_REPOSITORY configuration variables.

If EXTERNAL_REPOSTORY is set to false the two first methods can be used. In this case METAMOD is also responsible for moving the data files to the data repository in addition to extracting the metadata.

If EXTERNAL_REPOSITORY is set to true the second method is used. In this case METAMOD will not move the data files to the data repository. The data repository is then some other system responsibility.

Upload via web interface

When a user uploads a new data file to the system using the web interface the following steps are executed.

Upload via FTP

Upload via FTP is very similar to upload via web except for the first steps that triggers the metadata extraction.

Instead of using the queue system the FTP upload is triggered by the script upload/scripts/ftp_monitor.pl. This script supports several different modes but the most relevant here is the daemon mode where it is running like a CRON-like job.

When running as a CRON-like job the script will wake up at regular intervals and start looking for files in the FTP_UPLOAD_DIRECTORY.

When looking for files in the ftp upload directory it will not process all files that it finds. What files to process is specificed in the ftp_events file in the WEBRUN_DIRECTORY.

When a file is found in the FTP directory the metadata extraction continous in the same way for web uploads using the Metamod::UploadHelper.

Metadata extraction from external repository

Metadata extraction when data is stored in a external repository works differently than the two other methods.

In this case the process is as follows: