public:lta_howto

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
public:lta_howto [2016-11-30 13:12] – [User Access] Luciano Cerrigonepublic:lta_howto [2018-02-16 09:40] – [User Access] more detailed description on how to access public data wiel
Line 3: Line 3:
 This is a short manual on how to search for and retrieve data from the LOFAR Long Term Archive. This is a short manual on how to search for and retrieve data from the LOFAR Long Term Archive.
  
-To access the LTA, go to: [[http://lofar.target.rug.nl|lofar.target.rug.nl]]+To access the LTA, go to: [[https://lta.lofar.eu|lta.lofar.eu]]
  
 For background information and in case of problems, please refer to the [[public:lta_faq|Frequently Asked Questions]] page. For background information and in case of problems, please refer to the [[public:lta_faq|Frequently Asked Questions]] page.
Line 17: Line 17:
 The LTA catalogue can be searched using the anonymous account (AWWORLD). This account gives direct access to the project "All public data", and search queries will return results of the entire catalogue because metadata are public for all LTA content. The LTA catalogue can be searched using the anonymous account (AWWORLD). This account gives direct access to the project "All public data", and search queries will return results of the entire catalogue because metadata are public for all LTA content.
  
-Staging and subsequent downloading of public data always requires an account (if you have it, this is your MoM account) with LTA user access. If you do not have an account yet, you can register with "[[https://lofar.astron.nl/useradministration/public/setUpUserAccount.do|Create account]]". LTA access privileges will need to be granted by Science Support. Please, __once you have created your account, contact Science Support asking for LTA privileges__.+Staging and subsequent downloading of public data always requires an account (if you have it, this is your MoM account) with LTA user access. If you do not have an account yet, you can register with "[[https://lofar.astron.nl/useradministration/public/setUpUserAccount.do|Create account]]". LTA access privileges will need to be granted by Science Operations and Support. Please, __once you have created your account, contact Science Operations and Support asking for LTA privileges__.  
 + 
 +Step-by-step guide to access public data:  
 +  * log in to https://lta.lofar.eu/  
 +  * click SELECT PROJECT in the top menu, then select 'All Public data' from the list of project groupings 
 +  * click SEARCH in the top menu 
 +  * click on "here" (more advanced search option) 
 +  * choose “All Observations and Pipelines” 
 +  * find the relevant observation by either:  
 +    * choosing the project code from the dropdown list (if you know which project your observation of interest was taken under) 
 +    * specify the 'observation id' of the observation(s) and/or pipeline(s) that you are interested in 
 +  * click the 'Search' button at the bottom 
 +  * from the screen that follows, you should be able to stage the dataproducts
  
 === Project data === === Project data ===
  
-To directly access project-related data in the LTA you need to have an account in [[https://lofar.astron.nl/mom3|MoM]] that is enabled for the archive. +To directly access project-related data in the LTA you need to have an account in [[https://lofar.astron.nl/mom3|MoM]] that is enabled for the archive. This automatically happens if you were a member of the original project proposal in Northstar/MoM.
-  - This automatically happens if you were a member of the original project proposal in Northstar/MoM. +
-  - Otherwise Science Support can add you to the project for which you need access upon request.+
  
-If you were not originally a member of the project in MoM, you can request to be added by Science Support. When you send such a request, you must add the project's PI in cc. After Science Support adds you to the project, you might get an email asking you to set a new password in [[https://webportal.astron.nl/pwm/private/Login|ASTRON Web Applications Password Self Service]].  Please note that this will set a new password not just for the LTA //but for MoM (LOFAR/WSRT) and Northstar as well//.+If you were not included among the co-authors of the project but are still a collaborator, you can request to be added by Science Operations and Support. When you send such a request, you must add the project's PI in cc. After Science Operations and Support adds you to the project, you might get an email asking you to set a new password in [[https://webportal.astron.nl/pwm/private/Login|ASTRON Web Applications Password Self Service]].  Please note that this will set a new password not just for the LTA //but for MoM (LOFAR/WSRT) and Northstar as well//.
  
 ====== How to find data in the archive ====== ====== How to find data in the archive ======
Line 36: Line 46:
  
  
-{{:public:lta_navigation.png|}}+{{:public:lta_menu.png|}}
  
 A "project" can be selected by registered users that have access to a specific project. Subsequent searches will then be done within that project. A "project" can be selected by registered users that have access to a specific project. Subsequent searches will then be done within that project.
Line 42: Line 52:
 Searching the catalogue is easiest by selecting one of the otions in the "Search" menu. Depending on the knowledge of the data you are looking for (if any), searches can be performed on e.g., target ("Simple Search"), date and/or observation-ID and more advanced parameters per data type ("Advanced Search"). Searching the catalogue is easiest by selecting one of the otions in the "Search" menu. Depending on the knowledge of the data you are looking for (if any), searches can be performed on e.g., target ("Simple Search"), date and/or observation-ID and more advanced parameters per data type ("Advanced Search").
  
-Another option is to choose "Show Latest", which will show the latest additions in the LTA. This is particularely useful for project related searches.+Another option is to choose "Show Latest", which will show the latest additions in the LTA. This is particularly useful for project related searches.
  
  
 === Finding Data === === Finding Data ===
  
-{{:public:lta_observations.png|}}+{{:public:lta_observations_2017.png?900|}}
  
 Depending on the search parameters, e.g., which data products were requested (observation, pipeline), lists of observations and/or pipelines will be returned (see observations example above). From this point there are several options: Depending on the search parameters, e.g., which data products were requested (observation, pipeline), lists of observations and/or pipelines will be returned (see observations example above). From this point there are several options:
Line 68: Line 78:
 ==== Unspecified Data/Process ==== ==== Unspecified Data/Process ====
  
-Some data has had problems somewhere in the automation and control part of the LOFAR software during observation or processing. Sometimes a few subbands might be affected, sometimes an entire observation. Science support will check the data, (re)run things manually or fix things if needed and then archive the data. This does mean that the automation and control sometimes loses track of the files and the archiving process has no information beyond the Observation ID and filename itself. In such cases a few subbands or an entire observation might end up under "Unspecified Process". We do attempt to fix things at a later date, but that is not always feasible. If the files were archived, the data itself is usable. What is missing is the information that the LTA needs to properly label and query the data.+Some data has had problems somewhere in the automation and control part of the LOFAR software during observation or processing. Sometimes a few subbands might be affected, sometimes an entire observation. Science Operations and Support will check the data, (re)run things manually or fix things if needed and then archive the data. This does mean that the automation and control sometimes loses track of the files and the archiving process has no information beyond the Observation ID and filename itself. In such cases a few subbands or an entire observation might end up under "Unspecified Process". We do attempt to fix things at a later date, but that is not always feasible. If the files were archived, the data itself is usable. What is missing is the information that the LTA needs to properly label and query the data.
  
 If an Observation is missing, or is missing subbands, please check if it ended up under Unspecified. If an Observation is missing, or is missing subbands, please check if it ended up under Unspecified.
Line 76: Line 86:
 Once you have a list of dataproducts, observations or pipelines, you can use the check boxes to select which files you want to download. The first check box can be used to select or deselect all files or observations on a page. Once you have a list of dataproducts, observations or pipelines, you can use the check boxes to select which files you want to download. The first check box can be used to select or deselect all files or observations on a page.
  
-{{:public:lta_howto2.png|}}+{{:public:lta_staging_1.png|}}
  
 The LOFAR Archive stores data on magnetic tape. This means that it cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging'. The LOFAR Archive stores data on magnetic tape. This means that it cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging'.
Line 100: Line 110:
   - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or LOFAR operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed.    - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or LOFAR operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed. 
   - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again.   - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again.
-  - We are continuously trying to improve the reliability and speed of the available services. Please contact Science Support if you have any problems or suggestions for improvement. +  - We are continuously trying to improve the reliability and speed of the available services. Please contact Science Operations and Support if you have any problems or suggestions for improvement. 
-  - The data centres the LTA uses also have maintenance or small outages sometimes. Science Support can advice you if this is the case and when it is planned to end, if you are having trouble accessing data. In general this will not be at the same dates as the LOFAR stop days.+  - The data centres the LTA uses also have maintenance or small outages sometimes. Science Operations and Support can advice you if this is the case and when it is planned to end, if you are having trouble accessing data. In general this will not be at the same dates as the LOFAR stop days.
  
 ===== Download data ===== ===== Download data =====
Line 218: Line 228:
 ==== SRM download ==== ==== SRM download ====
  
-If you open the file ''srm.txt'' this file contains a list of srm locations which you would feed to ''srmcp''. SRM is a GRID specific protocol that is currently supported for data at the SARA and Jülich locations. It is faster, especially if you have significantly more than 1 Gbit/s bandwidth. It requires a valid [[GRID certificate]] and installation of the [[GRID srm software installation|GRID srm software]]. NB There is an [[srmclientinstallation|alternative installation that does not require root privileges]]. Contact Science Support if you think you might need a GRID account but it can not be provided by your own institute.+If you open the file ''srm.txt'' this file contains a list of srm locations which you would feed to ''srmcp''. SRM is a GRID specific protocol that is currently supported for data at the SARA and Jülich locations. It is faster, especially if you have significantly more than 1 Gbit/s bandwidth. It requires a valid [[GRID certificate]] and installation of the [[GRID srm software installation|GRID srm software]]. NB There is an [[srmclientinstallation|alternative installation that does not require root privileges]]. Contact Science Operations and Support if you think you might need a GRID account but it can not be provided by your own institute.
 An example command line would be: An example command line would be:
   srmcp -server_mode=passive -copyjobfile=srm.txt   srmcp -server_mode=passive -copyjobfile=srm.txt
  • Last modified: 2020-11-04 15:36
  • by Bernard Asabere