Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
public:lta_howto [2020-05-20 08:04] – [Release Notes] Reinoud Bokhorst | public:lta_howto [2020-11-04 15:18] – Bernard Asabere | ||
---|---|---|---|
Line 5: | Line 5: | ||
To access the LTA, go to: [[https:// | To access the LTA, go to: [[https:// | ||
- | For background information and in case of problems, please refer to the [[public: | + | For background information and in case of problems, please refer to the [[:public: |
====== Release Notes ====== | ====== Release Notes ====== | ||
- | + | ^Release^Description| | |
- | ^ Release ^ Description | + | |July 2018|1) All data is now searchable, not only released data or data of projects you're a member of. Note that for downloading data (staging) the proprietary restrictions still apply. \\ 2) All projects can now be selected by all users, not only by project members. This provides the means to filter data based on project. \\ 3) Cone search algorithm implemented based on the Haversine formula for angular distance calculation. The calculated angular distance to the reference coordinates is now displayed in the search results. \\ 4) Search keys: added "time resolution", |
- | | July 2018 | 1) All data is now searchable, not only released data or data of projects you're a member of. Note that for downloading data (staging) the proprietary restrictions still apply. \\ 2) All projects can now be selected by all users, not only by project members. This provides the means to filter data based on project.\\ | + | |
- | + | ||
- | ====== Public vs Proprietary data ====== | + | |
====== User Access ====== | ====== User Access ====== | ||
Line 25: | Line 22: | ||
The LTA catalogue can be searched directly without needing any account. Access to all projects and search queries will return results of the entire catalogue because metadata are public for all LTA content. | The LTA catalogue can be searched directly without needing any account. Access to all projects and search queries will return results of the entire catalogue because metadata are public for all LTA content. | ||
- | Staging and subsequent downloading of __**public**__ | + | Staging and subsequent downloading of __**public**__ |
+ | |||
+ | To stage and retrieve project-related data in the LTA which are __**proprietary**__ | ||
- | To stage and retrieve project-related data in the LTA which are __**proprietary**__ | + | Please read the [[https://old.astron.nl/radio-observatory/observing-proposals/lofar-data-policy/lofar-data-policy|LOFAR Data Policy]] for more information about //proprietary// vs //public// data. |
=== Step-by-step guide to search and retrieve data === | === Step-by-step guide to search and retrieve data === | ||
Line 57: | Line 56: | ||
* use the 'show data' button to select the project and to show all data in it | * use the 'show data' button to select the project and to show all data in it | ||
* from the screen that follows, you should be able to either search / select / stage the data products | * from the screen that follows, you should be able to either search / select / stage the data products | ||
- | |||
====== How to find data in the archive ====== | ====== How to find data in the archive ====== | ||
Line 67: | Line 65: | ||
The LTA menu, as shown below, gives access to the main functionality. | The LTA menu, as shown below, gives access to the main functionality. | ||
+ | {{: | ||
- | {{: | + | A search in the LTA catalogue can be initiated by clicking on the SEARCH DATA button on the menu. At this point a default basic search is setup, where users can select the data product type of interest and perform a cone search. An advanced search mode, with more advanced parameters per data type, can also be selected by clicking on the drop menu on the left side. |
- | + | ||
- | A search in the LTA catalogue can be initiated by clicking on the SEARCH DATA button on the menu. At this point a default basic search is setup, where users can select the data product type of interest and perform a cone search. An advanced search mode, with more advanced parameters per data type, can also be selected by clicking on the drop menu on the left side. | + | |
A " | A " | ||
- | | + | |
- | * Use the ' | + | |
- | * Use the 'show data' button to select the project and to show all data in it. | + | * Use the ' |
+ | * Use the 'show data' button to select the project and to show all data in it. | ||
=== Finding Data === | === Finding Data === | ||
- | {{: | + | {{: |
Depending on the search parameters, e.g., which data products were requested (observation, | Depending on the search parameters, e.g., which data products were requested (observation, | ||
Line 91: | Line 89: | ||
To see whether observations or pipelines have data products in the LTA, look for the " | To see whether observations or pipelines have data products in the LTA, look for the " | ||
- | Once you have a list of dataproducts on your screen, the " | + | Once you have a list of dataproducts on your screen, the " |
- | You can also hover with your mouse over the checkbox and get more information, | + | |
- | + | ||
- | There is a separate page with **[[lta_tricks|more detailed information]]** and advanced tricks to help you find and download your data. | + | |
- | You will find examples of how to **browse the archive with your own python script**. | + | |
+ | There is a separate page with **[[: | ||
==== Unspecified Data/ | ==== Unspecified Data/ | ||
- | Some data has had problems somewhere in the automation and control part of the LOFAR software during observation or processing. Sometimes a few subbands might be affected, sometimes an entire observation. Science Operations | + | Some data has had problems somewhere in the automation and control part of the LOFAR software during observation or processing. Sometimes a few subbands might be affected, sometimes an entire observation. Science |
If an Observation is missing, or is missing subbands, please check if it ended up under Unspecified. | If an Observation is missing, or is missing subbands, please check if it ended up under Unspecified. | ||
Line 108: | Line 103: | ||
Once you have a list of dataproducts, | Once you have a list of dataproducts, | ||
- | {{: | + | {{: |
The LOFAR Archive stores data on magnetic tape. This means that it cannot be downloaded right away, but has to be copied from tape to disk first. This process is called ' | The LOFAR Archive stores data on magnetic tape. This means that it cannot be downloaded right away, but has to be copied from tape to disk first. This process is called ' | ||
Line 114: | Line 109: | ||
When you have made your selection of files, click on //stage//. This shows you the following message. It means that a request has been sent to the LTA staging service to start retrieving the requested files from the tape and make them available on disk. You will get a confirmation e-mail, to acknowledge that your staging request was received and the process was queued. When the files are staged, you will get a notification email informing you that your data are ready for retrieval. | When you have made your selection of files, click on //stage//. This shows you the following message. It means that a request has been sent to the LTA staging service to start retrieving the requested files from the tape and make them available on disk. You will get a confirmation e-mail, to acknowledge that your staging request was received and the process was queued. When the files are staged, you will get a notification email informing you that your data are ready for retrieval. | ||
- | {{: | + | {{: |
- | The e-mail that you get when the staging on disk is complete gives you a list of files and has several attachments. Amongst them are two files '' | + | The e-mail that you get when the staging on disk is complete gives you a list of files and has several attachments. Amongst them are two files '' |
- | {{: | + | {{: |
- | There are two different ways to download your files with these attachments: | + | There are two different ways to download your files with these attachments: |
- | We also attach plain lists of the files/SURLs that were scheduled for staging (in the confirmation mail), those that were successfully staged, and (if any) those that could not be staged (in the success / partial success notifications). | + | We also attach plain lists of the files/SURLs that were scheduled for staging (in the confirmation mail), those that were successfully staged, and (if any) those that could not be staged (in the success / partial success notifications). |
+ | === Please take note of the following === | ||
- | === Please take note of the following ==== | + | |
- | + | ||
- | | + | |
- On a 1 Gbit/s connection as a general rule of thumb, you should be able to retrieve data at about 100-500 GB/hour, especially if you try to retrieve 4-8 files concurrently. If you see speeds much lower than this, you might have some kind of network problem and should in general contact your IT staff. | - On a 1 Gbit/s connection as a general rule of thumb, you should be able to retrieve data at about 100-500 GB/hour, especially if you try to retrieve 4-8 files concurrently. If you see speeds much lower than this, you might have some kind of network problem and should in general contact your IT staff. | ||
- Staging the data from tape to disk might take quite a bit of time. In the large data centres that the LTA uses, the tape drives are shared with all users and requests are queued. This is not just users of LOFAR but large data other projects like the LHC. This might mean that it takes anywhere from a few hours to a day or more to stage a copy of your data from tape to disk. | - Staging the data from tape to disk might take quite a bit of time. In the large data centres that the LTA uses, the tape drives are shared with all users and requests are queued. This is not just users of LOFAR but large data other projects like the LHC. This might mean that it takes anywhere from a few hours to a day or more to stage a copy of your data from tape to disk. | ||
- | - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/ | + | - The amount of space available for staging data is limited although quite large. This space is however shared between all LOFAR LTA users. This includes LTA operations for buffering data from CEP to the LTA before it gets moved to tape. If many users are staging data at the same time, and/ |
- We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again. | - We strive to keep a copy of data that was staged on disk for 1-2 weeks so you have some time to download it. After that it might get removed to make space for more recent requests. The copy of the data on tape is only read and will still be available if you need to access the data again at a later stage but you might need to stage a copy to disk again. | ||
- | - We are continuously trying to improve the reliability and speed of the available services. Please contact | + | - We are continuously trying to improve the reliability and speed of the available services. Please contact |
- | - The data centres the LTA uses also have maintenance or small outages sometimes. | + | - The data centres the LTA uses also have maintenance or small outages sometimes. |
+ | |||
+ | ==== Staging Transient Buffer Board (TBB) data ==== | ||
+ | |||
+ | TBB data needs to be staged by hand. Please send a request at [[https:// | ||
+ | < | ||
+ | |||
+ | wget --no-check-certificate https:// | ||
+ | |||
+ | The filename should start or be prepended with srm:// | ||
+ | |||
+ | </ | ||
+ | |||
+ | You will need a valid LTA account to access this data. If the filename is very short, you can view (e.g. cat) it to view errors that have occured. | ||
===== Download data ===== | ===== Download data ===== | ||
- | You can download your requested data with the files from your e-mail notification. | + | You can download your requested data with the files from your e-mail notification. There are different possibilities and tools to do this. If you're unsure, which one to use, please refer to the according [[:public: |
- | There are different possibilities and tools to do this. If you're unsure, which one to use, please refer to the according [[public: | + | |
==== HTTP download ==== | ==== HTTP download ==== | ||
- | If you open '' | + | If you open '' |
For wget you can use the following command line: | For wget you can use the following command line: | ||
- | | + | < |
- | This will download the files in '' | + | |
+ | wget -i html.txt | ||
+ | |||
+ | </ | ||
+ | |||
+ | This will download the files in '' | ||
Preferrably, | Preferrably, | ||
- | | + | |
+ | < | ||
+ | wget -ci html.txt | ||
+ | |||
+ | </ | ||
Do not set the username and password on the wget command line because this allows other users on the system to view them in the process list. Instead you should create a file ~/.wgetrc with two lines according to the following example: | Do not set the username and password on the wget command line because this allows other users on the system to view them in the process list. Instead you should create a file ~/.wgetrc with two lines according to the following example: | ||
- | | + | |
- | password=secret | + | < |
+ | user=lofaruser | ||
+ | password=secret | ||
+ | |||
+ | </ | ||
Note: This is only an example, you have to edit the file and enter your own personal user name and password! | Note: This is only an example, you have to edit the file and enter your own personal user name and password! | ||
- | + | ||
Set access authorizations of the .wgetrc file to user only so that the credentials are not exposed to anybody else, e.g.: | Set access authorizations of the .wgetrc file to user only so that the credentials are not exposed to anybody else, e.g.: | ||
- | | + | |
+ | < | ||
+ | chmod 600 .wgetrc | ||
+ | |||
+ | </ | ||
There is no easy way to have wget rename the files as part of the command directly. It does not accept the -O flag inside a file it gets with -i. You can either rename files afterward, e.g. using the following command: | There is no easy way to have wget rename the files as part of the command directly. It does not accept the -O flag inside a file it gets with -i. You can either rename files afterward, e.g. using the following command: | ||
- | | + | |
+ | < | ||
+ | find . -name " | ||
+ | |||
+ | </ | ||
or add the -O option to each line in html.txt but then feed each line to wget separately like this: cat '' | or add the -O option to each line in html.txt but then feed each line to wget separately like this: cat '' | ||
The following Python script will take care of renaming and untarring the downloaded files: | The following Python script will take care of renaming and untarring the downloaded files: | ||
- | |||
< | < | ||
+ | |||
#M.C. Toribio | #M.C. Toribio | ||
# | # | ||
Line 192: | Line 221: | ||
print outname+' | print outname+' | ||
+ | |||
</ | </ | ||
Line 200: | Line 230: | ||
import sys | import sys | ||
import glob | import glob | ||
- | |||
# AUTHOR: J.B.R. OONK (ASTRON/ | # AUTHOR: J.B.R. OONK (ASTRON/ | ||
# - changes LTA retrieval filename to standard filename | # - changes LTA retrieval filename to standard filename | ||
- | # - run in the directory where LTA files are located | + | # - run in the directory where LTA files are located |
# FILE DIRECTORY | # FILE DIRECTORY | ||
Line 243: | Line 271: | ||
print ' | print ' | ||
+ | |||
</ | </ | ||
Note that wget does not overwrite existing files. If you use the continue option (' | Note that wget does not overwrite existing files. If you use the continue option (' | ||
- | There are some small example links if you browse to [[https:// | + | There are some small example links if you browse to [[https:// |
==== SRM download ==== | ==== SRM download ==== | ||
- | If you open the file '' | + | If you open the file '' |
- | An example command line would be: | + | < |
- | srmcp -server_mode=passive -copyjobfile=srm.txt | + | |
+ | srmcp -server_mode=passive -copyjobfile=srm.txt | ||
+ | |||
+ | </ | ||
to retrieve all requested files contained in srm.txt or e.g. | to retrieve all requested files contained in srm.txt or e.g. | ||
- | srmcp -server_mode=passive srm:// | ||
- | to retrieve a single file. You need '' | ||
- | If you do experience insufficient transfer speeds with srmcp, you may want to look into using srmcp with a [[public:srmclientinstallation# | + | < |
+ | srmcp -server_mode=passive srm://lofar-srm.juelich.de: | ||
+ | </ | ||
+ | to retrieve a single file. You need '' | ||
+ | |||
+ | If you do experience insufficient transfer speeds with srmcp, you may want to look into using srmcp with a [[: | ||
===== Troubleshooting ===== | ===== Troubleshooting ===== | ||
- | * There is a [[public: | + | * There is a [[:public: |
+ | |||
+ | \\ | ||
+ |