METAMOD supports extracting metadata from netCDF files. The metadata extraction is triggered in three different ways.
Which method is used for triggering uploads are decided by the
EXTERNAL_REPOSITORY configuration variables.
EXTERNAL_REPOSTORY is set to
false the two first methods can be used.
In this case METAMOD is also responsible for moving the data files to the data repository in addition to extracting the metadata.
EXTERNAL_REPOSITORY is set to
true the second method is used.
In this case METAMOD will not move the data files to the data repository.
The data repository is then some other system responsibility.
When a user uploads a new data file to the system using the web interface the following steps are executed.
The web application also inserts an "Upload" job into the queue to tell the backend system that a new file is available.
At this point the web application sends a reply back to the client.
For the Upload job,
Metamod::Queue::Job::Upload is used to perform the job.
The reason for having two modules for doing this work is that Worker module is specific the queue system used,
while the Job module is independent of the actual queue system.
This makes it easier to switch from one queue system to another at a later date.
Metamod::Queue::Job::Uploadhas started to execute the job it will leave the heavy lifting to
Metamod::UploadHelper. UploadHelper will do the following:
MetNo::DigestNcto extract the metadata from the netCDF files. This processing in done twice. Once for all the uploaded at the same time to extract the level 1 metadata for the dataset. Then DigestNc is used again for each file to extract the level 2 metadata for that file.
Metamod::Dataset for reading and writing metadata so all data is stored both in the XML metadata repository and the index database.
In case of any error in the processing of the file the UploadHelper will generate a HTML file with a description of all the errors.
Upload via FTP is very similar to upload via web except for the first steps that triggers the metadata extraction.
Instead of using the queue system the FTP upload is triggered by the script upload/scripts/ftp_monitor.pl. This script supports several different modes but the most relevant here is the daemon mode where it is running like a CRON-like job.
When running as a CRON-like job the script will wake up at regular intervals and start looking for files in the
When looking for files in the ftp upload directory it will not process all files that it finds.
What files to process is specificed in the
ftp_events file in the
When a file is found in the FTP directory the metadata extraction continous in the same way for web uploads using the
Metadata extraction when data is stored in a external repository works differently than the two other methods.
In this case the process is as follows: