public:lta_howto

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Next revisionBoth sides next revision
public:lta_howto [2018-07-11 08:32] – Reinoud Bokhorstpublic:lta_howto [2020-10-13 15:45] – [Staging data (Prepare for download)] Sander ter Veen
Line 11: Line 11:
  
 ^ Release ^ Description ^ ^ Release ^ Description ^
-| July 2018 | 1) All data is now searchable, not only released data or data of projects you're a member of. Note that for downloading data (staging) the proprietary restrictions still apply. \\ 2) All projects can now be selected by all users, not only by project members. This provides the means to filter data based on project.\\ 3) The user interface has been changed to improve user friendliness. Key points are a more intuitive menu structure, improved project selection/deselection and restyling. \\ 4) Implemented cone search algorithm based on the Haversine formula for angular distance calculation. The calculated angular distance to the reference coordinates is now displayed in the search results. \\ 5) Search keys added for "time resolution", "Frequency resolution" and "nr. of subbands per subband group" \\ 6) Search keys removed: "sky model database", strategy name" |+| July 2018 | 1) All data is now searchable, not only released data or data of projects you're a member of. Note that for downloading data (staging) the proprietary restrictions still apply. \\ 2) All projects can now be selected by all users, not only by project members. This provides the means to filter data based on project.\\ 3) Cone search algorithm implemented based on the Haversine formula for angular distance calculation. The calculated angular distance to the reference coordinates is now displayed in the search results. \\ 4) Search keys: added "time resolution", "Frequency resolution" and "nr. of subbands per subband group", removed "sky model database" and strategy name" |
  
-====== Public vs Proprietary data ====== 
  
 ====== User Access ====== ====== User Access ======
Line 19: Line 18:
 === I forgot my password === === I forgot my password ===
  
-Please visit https://webportal.astron.nl/pwm/public/ForgottenPassword to request a new one.+Please visit [[https://webportal.astron.nl/pwm/public/ForgottenPassword|https://webportal.astron.nl/pwm/public/ForgottenPassword]] to request a new one. 
 +\\ \\ 
  
 === Searching / retrieving data === === Searching / retrieving data ===
Line 25: Line 25:
 The LTA catalogue can be searched directly without needing any account. Access to all projects and search queries will return results of the entire catalogue because metadata are public for all LTA content. The LTA catalogue can be searched directly without needing any account. Access to all projects and search queries will return results of the entire catalogue because metadata are public for all LTA content.
  
-Staging and subsequent downloading of __**public**__ data always requires an account with LTA user access privileges. This automatically happens if you were a member of the original project proposal in Northstar/MoM. If you do not have an account yet, you can register with "[[https://lofar.astron.nl/useradministration/public/setUpUserAccount.do|Create account]]". LTA access privileges will need to be granted by Science Operations and Support. Please, __once you have created your account, submit a support request to the [[https://support.astron.nl/rohelpdesk|Radio Observatory helpdesk]] asking for LTA privileges__.+Staging and subsequent downloading of __**public**__  data always requires an account with LTA user access privileges. This automatically happens if you were a member of the original project proposal in Northstar/MoM. If you do not have an account yet, you can register with "[[https://lofar.astron.nl/useradministration/public/setUpUserAccount.do|Create account]]". LTA access privileges will need to be granted by Science Operations and Support. Please, __once you have created your account, submit a support request to the [[https://support.astron.nl/rohelpdesk|Radio Observatory helpdesk]] asking for LTA privileges stating clearly your username__.
  
-To stage and retrieve project-related data in the LTA which are __**proprietary**__ you need to have an account in [[https://lofar.astron.nl/mom3|MoM]] that is enabled for the archive and coupled with the projects of interest. To this aim you can request Science Operations & Support to be added to the list of co-authors of the project. When you send such a request, you must add the project's PI in cc. After Science Operations and Support adds you to the project, you might get an email asking you to set a new password in [[https://webportal.astron.nl/pwm/private/Login|ASTRON Web Applications Password Self Service]].  Please note that this will set a new password not just for the LTA //but for MoM (LOFAR/WSRT) and Northstar as well//.+To stage and retrieve project-related data in the LTA which are __**proprietary**__  you need to have an account in [[https://lofar.astron.nl/mom3|MoM]] that is enabled for the archive and coupled with the projects of interest. To this aim you can request Science Operations & Support to be added to the list of co-authors of the project. When you send such a request, you must add the project's PI in cc. After Science Operations and Support adds you to the project, you might get an email asking you to set a new password in [[https://webportal.astron.nl/pwm/private/Login|ASTRON Web Applications Password Self Service]]. Please note that this will set a new password not just for the LTA //but for MoM (LOFAR/WSRT) and Northstar as well//. 
 + 
 +Please read the [[https://old.astron.nl/radio-observatory/observing-proposals/lofar-data-policy/lofar-data-policy|LOFAR Data Policy]] for more information about //proprietary// vs //public// data. 
 +\\ \\ 
  
 === Step-by-step guide to search and retrieve data === === Step-by-step guide to search and retrieve data ===
  
 __Basic search__: __Basic search__:
-  * log in to https://lta.lofar.eu/ + 
 +   * log in to [[https://lta.lofar.eu/|https://lta.lofar.eu/]]
   * click SEARCH DATA in the top menu   * click SEARCH DATA in the top menu
   * specify the data product types of interest and a target name or coordinated   * specify the data product types of interest and a target name or coordinated
Line 39: Line 43:
  
 __Advanced search__: __Advanced search__:
-  * log in to https://lta.lofar.eu/ + 
 +  * log in to [[https://lta.lofar.eu/|https://lta.lofar.eu/]]
   * click SEARCH DATA in the top menu   * click SEARCH DATA in the top menu
   * click on the side Advanced Search drop-down list   * click on the side Advanced Search drop-down list
Line 48: Line 53:
  
 __Project search (to restrict all data searches to that project only)__: __Project search (to restrict all data searches to that project only)__:
-  * log in to https://lta.lofar.eu/ + 
 +  * log in to [[https://lta.lofar.eu/|https://lta.lofar.eu/]]
   * click BROWSE PROJECTS in the top menu   * click BROWSE PROJECTS in the top menu
   * at this level membership can be checked, with the first column showing if you are a member of the project or for finding public projects. Available options are:   * at this level membership can be checked, with the first column showing if you are a member of the project or for finding public projects. Available options are:
-    * click on the project name to view the project details +      * click on the project name to view the project details 
-    * use the 'search' button to select the project and go to the search page +      * use the 'search' button to select the project and go to the search page 
-    * use the 'show data' button to select the project and to show all data in it+      * use the 'show data' button to select the project and to show all data in it
   * from the screen that follows, you should be able to either search / select / stage the data products   * from the screen that follows, you should be able to either search / select / stage the data products
 +
 +
 ====== How to find data in the archive ====== ====== How to find data in the archive ======
  
Line 104: Line 112:
 Once you have a list of dataproducts, observations or pipelines, you can use the check boxes to select which files you want to download. The first check box can be used to select or deselect all files or observations on a page. Once you have a list of dataproducts, observations or pipelines, you can use the check boxes to select which files you want to download. The first check box can be used to select or deselect all files or observations on a page.
  
-{{:public:lta_staging_1.png?900|}}+{{:public:lta_staging_1.png?900}}
  
 The LOFAR Archive stores data on magnetic tape. This means that it cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging'. The LOFAR Archive stores data on magnetic tape. This means that it cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging'.
Line 110: Line 118:
 When you have made your selection of files, click on //stage//. This shows you the following message. It means that a request has been sent to the LTA staging service to start retrieving the requested files from the tape and make them available on disk. You will get a confirmation e-mail, to acknowledge that your staging request was received and the process was queued. When the files are staged, you will get a notification email informing you that your data are ready for retrieval. When you have made your selection of files, click on //stage//. This shows you the following message. It means that a request has been sent to the LTA staging service to start retrieving the requested files from the tape and make them available on disk. You will get a confirmation e-mail, to acknowledge that your staging request was received and the process was queued. When the files are staged, you will get a notification email informing you that your data are ready for retrieval.
  
-{{:public:lta_howto_3.png?900|}}+{{:public:lta_howto_3.png?900}}
  
 The e-mail that you get when the staging on disk is complete gives you a list of files and has several attachments. Amongst them are two files ''html.txt'' and ''srm.txt'': The e-mail that you get when the staging on disk is complete gives you a list of files and has several attachments. Amongst them are two files ''html.txt'' and ''srm.txt'':
  
-{{:public:lta_howto4.png|}}+{{:public:lta_howto4.png}}
  
-There are two different ways to download your files with these attachments: [[#HTTP Download|http]] and [[#SRM Download|srm]]+There are two different ways to download your files with these attachments: [[#http_download|http]] and [[#srm_download|srm]]
  
-We also attach plain lists of the files/SURLs that were scheduled for staging (in the confirmation mail), those that were successfully staged, and (if any) those that could not be staged (in the success / partial success notifications). +We also attach plain lists of the files/SURLs that were scheduled for staging (in the confirmation mail), those that were successfully staged, and (if any) those that could not be staged (in the success / partial success notifications).
  
 +=== Please take note of the following ===
  
-=== Please take note of the following ==== +  - Unless you have an extremely fast connection (10 Gbit/s or more), **it is in general advisable to stage no more than 5 TB at a time**  (see also point 4). At maximum efficiency a 1 Gbit/s connection will already take 12 hours to retrieve 5 TB of data, in practice it will often take quite a bit more.
- +
-  - Unless you have an extremely fast connection (10 Gbit/s or more), **it is in general advisable to stage no more than 5 TB at a time** (see also point 4). At maximum efficiency a 1 Gbit/s connection will already take 12 hours to retrieve 5 TB of data, in practice it will often take quite a bit more.+
   - On a 1 Gbit/s connection as a general rule of thumb, you should be able to retrieve data at about 100-500 GB/hour, especially if you try to retrieve 4-8 files concurrently. If you see speeds much lower than this, you might have some kind of network problem and should in general contact your IT staff.   - On a 1 Gbit/s connection as a general rule of thumb, you should be able to retrieve data at about 100-500 GB/hour, especially if you try to retrieve 4-8 files concurrently. If you see speeds much lower than this, you might have some kind of network problem and should in general contact your IT staff.
   - Staging the data from tape to disk might take quite a bit of time. In the large data centres that the LTA uses, the tape drives are shared with all users and requests are queued. This is not just users of LOFAR but large data other projects like the LHC. This might mean that it takes anywhere from a few hours to a day or more to stage a copy of your data from tape to disk.   - Staging the data from tape to disk might take quite a bit of time. In the large data centres that the LTA uses, the tape drives are shared with all users and requests are queued. This is not just users of LOFAR but large data other projects like the LHC. This might mean that it takes anywhere from a few hours to a day or more to stage a copy of your data from tape to disk.
-  - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or LOFAR operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed. +  - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/or LOFAR operations is transferring large amounts of data, the system might temporarily run low on disk space. You might then get a message that your request was only partially successful. In general the request will still finish 1-2 days later and we do monitor if requests don't get stuck and restart if needed.
   - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again.   - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again.
   - We are continuously trying to improve the reliability and speed of the available services. Please contact Science Operations and Support if you have any problems or suggestions for improvement.   - We are continuously trying to improve the reliability and speed of the available services. Please contact Science Operations and Support if you have any problems or suggestions for improvement.
   - The data centres the LTA uses also have maintenance or small outages sometimes. Science Operations and Support can advice you if this is the case and when it is planned to end, if you are having trouble accessing data. In general this will not be at the same dates as the LOFAR stop days.   - The data centres the LTA uses also have maintenance or small outages sometimes. Science Operations and Support can advice you if this is the case and when it is planned to end, if you are having trouble accessing data. In general this will not be at the same dates as the LOFAR stop days.
 +
 +==== Staging Transient Buffer Board (TBB) data ====
 +
 +TBB data needs to be staged by hand. Please send a request at support.astron.nl/rohelpdesk to stage the data for you, specifying the filenames to be staged. To download the data, please follow the instruction under Download Data for proper authentication. Data will then be available for download using
 +<file>
 +
 +wget --no-check-certificate https://lofar-download.grid.surfsara.nl/lofigrid/SRMFifoGet.py?surl=<filename> .
 +
 +</file>
 +
 +You will need a valid LTA account to access this data.
 +
  
 ===== Download data ===== ===== Download data =====
  • Last modified: 2020-11-04 15:36
  • by Bernard Asabere