Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
public:lta_howto [2016-12-23 15:30] – [HTTP download] wiel | public:lta_howto [2017-10-16 11:40] – [How to find data in the archive] Reinoud Bokhorst | ||
---|---|---|---|
Line 3: | Line 3: | ||
This is a short manual on how to search for and retrieve data from the LOFAR Long Term Archive. | This is a short manual on how to search for and retrieve data from the LOFAR Long Term Archive. | ||
- | To access the LTA, go to: [[http://lofar.target.rug.nl|lofar.target.rug.nl]] | + | To access the LTA, go to: [[https://lta.lofar.eu|lta.lofar.eu]] |
For background information and in case of problems, please refer to the [[public: | For background information and in case of problems, please refer to the [[public: | ||
Line 17: | Line 17: | ||
The LTA catalogue can be searched using the anonymous account (AWWORLD). This account gives direct access to the project "All public data", and search queries will return results of the entire catalogue because metadata are public for all LTA content. | The LTA catalogue can be searched using the anonymous account (AWWORLD). This account gives direct access to the project "All public data", and search queries will return results of the entire catalogue because metadata are public for all LTA content. | ||
- | Staging and subsequent downloading of public data always requires an account (if you have it, this is your MoM account) with LTA user access. If you do not have an account yet, you can register with " | + | Staging and subsequent downloading of public data always requires an account (if you have it, this is your MoM account) with LTA user access. If you do not have an account yet, you can register with " |
=== Project data === | === Project data === | ||
- | To directly access project-related data in the LTA you need to have an account in [[https:// | + | To directly access project-related data in the LTA you need to have an account in [[https:// |
- | - This automatically happens if you were a member of the original project proposal in Northstar/ | + | |
- | - Otherwise Science Support can add you to the project for which you need access upon request. | + | |
- | If you were not originally a member | + | If you were not included among the co-authors |
====== How to find data in the archive ====== | ====== How to find data in the archive ====== | ||
Line 36: | Line 34: | ||
- | {{:public:lta_navigation.png|}} | + | {{:public:lta_menu.png|}} |
A " | A " | ||
Line 42: | Line 40: | ||
Searching the catalogue is easiest by selecting one of the otions in the " | Searching the catalogue is easiest by selecting one of the otions in the " | ||
- | Another option is to choose "Show Latest", | + | Another option is to choose "Show Latest", |
=== Finding Data === | === Finding Data === | ||
- | {{:public:lta_observations.png|}} | + | {{:public:lta_observations_2017.png?900|}} |
Depending on the search parameters, e.g., which data products were requested (observation, | Depending on the search parameters, e.g., which data products were requested (observation, | ||
Line 68: | Line 66: | ||
==== Unspecified Data/ | ==== Unspecified Data/ | ||
- | Some data has had problems somewhere in the automation and control part of the LOFAR software during observation or processing. Sometimes a few subbands might be affected, sometimes an entire observation. Science | + | Some data has had problems somewhere in the automation and control part of the LOFAR software during observation or processing. Sometimes a few subbands might be affected, sometimes an entire observation. Science |
If an Observation is missing, or is missing subbands, please check if it ended up under Unspecified. | If an Observation is missing, or is missing subbands, please check if it ended up under Unspecified. | ||
Line 100: | Line 98: | ||
- The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or LOFAR operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed. | - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or LOFAR operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed. | ||
- We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again. | - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again. | ||
- | - We are continuously trying to improve the reliability and speed of the available services. Please contact Science Support if you have any problems or suggestions for improvement. | + | - We are continuously trying to improve the reliability and speed of the available services. Please contact Science |
- | - The data centres the LTA uses also have maintenance or small outages sometimes. Science Support can advice you if this is the case and when it is planned to end, if you are having trouble accessing data. In general this will not be at the same dates as the LOFAR stop days. | + | - The data centres the LTA uses also have maintenance or small outages sometimes. Science |
===== Download data ===== | ===== Download data ===== | ||
Line 126: | Line 124: | ||
Set access authorizations of the .wgetrc file to user only so that the credentials are not exposed to anybody else, e.g.: | Set access authorizations of the .wgetrc file to user only so that the credentials are not exposed to anybody else, e.g.: | ||
- | chmod 600 .wgetrc | + | chmod 600 .wgetrc |
- | As an example, you can use timeout switches and run ~10 processes in parallel to improve download speed: | + | |
- | cat html.txt | xargs -n 1 -P 10 wget --no-verbose --timeout=60 --read-timeout=60 | + | |
- | | + | |
There is no easy way to have wget rename the files as part of the command directly. It does not accept the -O flag inside a file it gets with -i. You can either rename files afterward, e.g. using the following command: | There is no easy way to have wget rename the files as part of the command directly. It does not accept the -O flag inside a file it gets with -i. You can either rename files afterward, e.g. using the following command: | ||
find . -name " | find . -name " | ||
Line 221: | Line 216: | ||
==== SRM download ==== | ==== SRM download ==== | ||
- | If you open the file '' | + | If you open the file '' |
An example command line would be: | An example command line would be: | ||
srmcp -server_mode=passive -copyjobfile=srm.txt | srmcp -server_mode=passive -copyjobfile=srm.txt |