public:documents:raw_olap_data_formats

This is an old revision of the document!


OLAP produces several data formats, which are intended to be replaced by their final format, such as HDF5. The formats below are not officially supported and subject to change without notice.

Beamformed Data

Beamformed data can be recorded as either complex voltages (yielding X and Y polarisations) or one or more stokes. In either case, a sequence of blocks will be stored, each of which consists of a header and data. The header is defined as:

struct header {
  uint32_t sequence_number; /* big endian */
  char padding[508];
};

in which sequence_number starts at 0, and is increased by 1 for every block. Missing sequence numbers implies missing data. The padding can have any value and is to be ignored.

Complex Voltages

Each (pencil) beam produces two files: one containing the X polarisation, and one containing the Y polarisation. The names of these files adhere to the following scheme:

Lxxxxx_Byyy_S0-bf.rawX polarisations of beam yyy of observation xxxxx
Lxxxxx_Byyy_S1-bf.rawY polarisations of beam yyy of observation xxxxx

Each file is a sequence of blocks of the following structure:

struct block {
  struct header header;
 
  /* big endian */
  fcomplex voltages[SAMPLES|2][SUBBANDS][CHANNELS];
}

Below is a list of the constants used, and which key they represent in the parset file as produced by OLAP:

Stokes

Each (pencil) beam produces one or four files: one containing the Stokes I (power) values, and optionally three files for Stokes Q, U, and V, respectively. The names of these files adhere to the following scheme:

Lxxxxx_Byyy_S0-bf.rawStokes I of beam yyy of observation xxxxx
Lxxxxx_Byyy_S1-bf.rawStokes Q of beam yyy of observation xxxxx
Lxxxxx_Byyy_S2-bf.rawStokes U of beam yyy of observation xxxxx
Lxxxxx_Byyy_S3-bf.rawStokes V of beam yyy of observation xxxxx

Currently (release 2010-09-20), the Stokes U and V are multiplied by a factor of 1/2, but that will be changed in a subsequent release.

Each file is a sequence of blocks of the following structure:

struct block {
  struct header header;
 
  /* big endian */
  float stokes[SAMPLES|2][SUBBANDS][CHANNELS];
}

Below is a list of the constants used, and which key they represent in the parset file as produced by OLAP:

SAMPLES The number of time samples in a block OLAP.CNProc.integrationSteps
SUBBANDSThe number of subbands (beamlets) specified len(Oberservation.subbandList)
CHANNELSThe number of channels per subband Observation.channelsPerSubband
The number of files per beam len(OLAP.Stokes.which)

A 'float' is a 32-bit IEEE floating point number. An 'fcomplex' is a complex number defined as

struct fcomplex {
  float real;
  float imag;
};
  • Last modified: 2010-10-21 13:30
  • by Jan David Mol