Differences between revisions 35 and 73 (spanning 38 versions)
⇤ ← Revision 35 as of 2014-04-23 12:49:42 →
Size: 5211
Editor: gra-lx17
Comment:
← Revision 73 as of 2014-11-07 09:05:40 → ⇥
Size: 10511
Editor: NikaBolometer
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
##acl hermelo,CarstenKramer,NicolasBillot,SamuelLeclerc:read,write,delete,revert,admin Default
Line 5: Line 3:
The following diagram shows the NIKA data flow from the IRAM 30m telescope to the observers' accounts.

{{attachment:NIKApipeline.png | Neel Pipeline| width=700}}
Line 7: Line 9:
'''[[ http://www.iram.es/IRAMES/mainWiki/NIKA/Main | Go back to the NIKA Main Wiki ]]''' '''Go to the [[http://www.iram.es/IRAMES/mainWiki/NIKA/Main|NIKA]] Main Wiki.'''
Line 11: Line 13:
Line 13: Line 16:
For a detailed explanation of the data products for the 1st NIKA pool please click [[attachment:NikaDataProductsV1.pdf | here (outdated) ]].
Line 17: Line 18:
The version v0 of the NIKA first open pool products has been made by the NIKA
team by the end of the pool (Feb 28,2014) and it was distributed by IRAM.
The products are described [[attachment:NIKA_DataProducts_v0.pdf | here ]].

For each project, there are fits files containing the maps of each scan and
combined scans of each object. The data processing has been oriented towards
point-source extraction or diffuse emission depending on the aim of your
project.

There are also logbook files and illustrative pdf files. On the main directory
you will find:

 * '''logfile_all_scans.csv''': is a comma separated file ready for your favorite spreadsheet handler (libreoffice or excel). It gives the summary information of all the scans that could be used including opacity measurements, sky noise in Jy (per band of frequency from 1mHz to 10 Hz), camera noise NEFD (in mJy.s^1/2) and finally a flux in mJy of a point-source at the center of the map (fixed central position, fixed FWHM, Gaussian fit) with the error at both wavelengths.

Then for each object, there is a directory that contains:

 * the '''fits''' files.

 * '''flux_source.csv''': a comma separated file giving the flux as measured in the scan maps and the combined scans maps.

 * '''Figures''': many pdf files were generated out of the fits files to get a quick look at them. Units in Jy.


To retrieve the data, please type:

{{{
scp XXX-YY@enter.iram.es:~/observationData/nika/v0.tar.gz /path/in/your/computer
}}}

where '''XXX-YY''' is your project number. If you get an error message, enter the following command:

{{{
ssh-keygen -R enter.iram.es
}}}

and then try to retrieve the data again.

'''CAVEAT:'''
At this stage, the data offline products are not of scientific quality. The
selection of scans was unfortunately rendered difficult because of an
unexpected problem with the pointing data. We hope to solve that problem with
post-processing steps that we did not have time to implement. Instead, scans
were manually selected with some mistakes still present. The photometry is
probably correct at the 20% level at 1mm and 10% level at 2mm for scans of
good quality.
Don't shoot. We will prepare the next version in the coming weeks. Contact
your NIKA friend of project to give us your feedback.

''The NIKA team''
Line 20: Line 71:
This version V1 of the NIKA first open pool products has been made by the NIKA
team by the end of May 2014. It is distributed by IRAM.
It follows the quick release done at the end of the pool (v0).
The products are described [[attachment:NIKA_DataProducts_v1.pdf | here ]]

For each project, there are fits files containing the maps of each scan and
combined scans of each object. The data processing has been oriented towards
point-source extraction or diffuse emission depending on the aim of your
project. The main processing has improved by including more scans and dealing
with internal problems more effectively.

There are also logbook files and illustrative pdf files. On the main directory
you will find:

 * logfile_all_scans.csv : is a comma separated file ready for your favorite
spreadsheet handler (libreoffice or excel) It gives the summary information of
all the scans that could be used including opacity measurements, sky noise in
Jy (per band of frequency from 1mHz to 10 Hz), camera noise NEFD (in
mJy.s^1/2) and finally a flux in mJy of a point-source at the center of the
map (fixed central position, fixed FWHM, Gaussian fit) with the error at both
wavelengths. For pointing sources the position is allowed to vary.

Then for each object, there is a directory that contains:

 * The fits files (!RaDec2000 projection for target sources, !AzEl projection for pointing sources and Planets)

 * flux_source.csv : a comma separated file giving the flux as measured in the
scan maps and the combined scans maps.

 * Figures: many pdf files were generated out of the fits files to get a quick
look at them. Units in Jy/beam unless otherwise stated.

The calibration was done assuming a Gaussian main beam of 12.5 and 18.5
arcseconds (FWHM). The primary calibrator is Uranus with fluxes of 36.4 and
15.3 Jy.

For Lissajous scans (!V1Liss directories), the pipeline introduces a filtering
effect for point sources of about 0.70 and 0.56 (at 1 and 2mm). This effect
has been corrected in the .csv files but not in the maps. Thus fluxes in Jy in
the .csv files should be correct.

For you to assess the filtering effect, we have rerun the pipeline with a much
less agressive decorrelation that should not affect the fluxes. For that just
look at !V1cLiss directories made for calibration sources only.

For OTF maps (directories V1otf): the main beam to full beam correction is by
1.56+-0.10 at 1mm and 1.35+-0.10 at 2mm. It has not been applied to the maps.

At this stage, the offline products can be used for a first scientific
analysis as most of the scans have been included. The point-source photometry
is probably correct at the 15% level at 1mm and 10% level at 2mm.

To retrieve the data, please type:

{{{
scp XXX-YY@enter.iram.es:~/observationData/nika/v1.tar.gz /path/in/your/computer
}}}

where '''XXX-YY''' is your project number. If you get an error message, enter the following command:

{{{
ssh-keygen -R enter.iram.es
}}}

and then try to retrieve the data again.


''The NIKA team''


=== Data products v2 ===

We hope to deliver a final V2 version of the first NIKA open pool data products in the Summer including our best strategy of systematic removals.
Your feedback is essential to improve the quality of the v1 data and help the NIKA team to develop the final version.

''The NIKA team''
Line 28: Line 155:
   $ ssh -X nikaw-13@mrt-lx1    $ ssh -X nikas-14@mrt-lx1
Line 34: Line 161:
There are two types of observation: the “science” scans and the “calibration” scans.
There is a specific routine to analyze each type of scan.

=== Calibration scans ===

All calibration scans are reduced by the same routine:

'''IDL> rta_reduce, day, scan_num'''

'''day is in the format YYYYMMDD, e.g. 20140218, scan_num is a simple integer.'''

Informations or actions to be taken by the observer appear on the plots and explicitely in IDL terminal window:


==== cont_pointing_cross ====

{{attachment:cross.png||width=400}}

 * The plots show 1D profiles of the azimuth or elevation scans at both wavelenght for the reference detector and a gaussian fit of the source along the timeline. The displayed offsets refer to the reference detectors and are not meant to be passed to PAKO.
 * The maps show the source, a cross at the location of the reference pixel and gaussian fit of the source
 * In the terminal window, the observer should follow the instruction (e.g.): B 2mm (MAP) (for PAKO) SET POINTING 3.0 0.7
 * the pointing reference is the 2mm channel.
 * If the sky conditions were correct, instructions at 1 and 2mm should be in good agreement.
 * if a glitch corrupts the maps but the 1d profiles are ok, the user can follow the instruction B 2mm, : (TOI) (for PAKO) SET POINTING 3.0 0.7
 * In good conditions, all instructions agree to better than 1 arcsec.
 * If the pointing is far off, the default reference pixels may not see the source and the timelines may look bad. In the terminal window, you may read the ID of the closest kids to the source and pass them to rta_reduce as keywords e.g. '''rta_reduce, day, scan_num, numdet1=5, numdet2=494'''


==== cont_pointing_liss ====		 There are two types of observation: the “science” scans and the “calibration” scans. All scans should be reduced with the same routine (except the "focus_liss" sequence as of now, see below):

'''IDL> nk_rta, scan'''

'''scan is a string the format YYYYMMDDsN, e.g. '20141112s3'. Mind the simple quote, IDL does not accept double quotes when the string starts by a number.'''

If this is a calibration scan (pointing, focus), informations or actions to be taken by the observer appear explicitely in IDL terminal window:


==== cross ====

{{attachment:cross_1.png||width=400}}

 * The above window shows up (after 3 sets of monitoring plots that show examples of timelines and power spectra)
 * The first column show the azimuth and elevation timelines and the samples that were flagged out based on their instantaneous speed.
 * The second column plots show 1D profiles of the azimuth or elevation scans at both wavelengths for the reference detector and a gaussian fit of the source along these profiles. The displayed offsets refer to the reference detectors and are not meant to be passed to PAKO.
 * The maps show the source, as mapped by all the detectors, a large cross at the location of the reference pixel, a gaussian fit of the source in red and the map center in green for reference.
 * In the terminal window, the observer should follow the instruction (e.g.): B 2mm (MAP) (for PAKO) SET POINTING xx.x yy.y
 * Notes:
 **the pointing reference is the 2mm channel.
 ** If the sky conditions are correct, instructions at 1 and 2mm should be in good agreement.
 ** if a glitch corrupts the maps but the 1d profiles are ok, the user can follow the instruction B 2mm, : (TOI) (for PAKO) SET POINTING xx.x yy.y
 ** In good conditions, all instructions agree to better about 1 arcsec.
 ** If the pointing is far off, the default reference pixels may not see the source and the timelines may look bad. In the terminal window, you may read the ID of the closest kids to the source and pass them to nk_rta as keywords e.g. '''nk_rta, scan, ref_det_1mm=X, ref_det_2mm=Y.


==== pointliss ====
Line 70: Line 195:
==== cont_focus_track ====

{{attachment:focus.png||width=400}}

 * you may change the reference kids as before e.g. '''rta_reduce, day, scan_num, numdet1=5, numdet2=494'''		 ==== focusp ====

{{attachment:focus_1.png||width=400}}
Line 85: Line 209:
==== cont_focus_liss ==== ==== focusliss ====
Line 90: Line 214:
 * As soon as the data of a scan of this sequence are on Sami's disk, you may run '''rta_reduce, day, scan_num'''
 * When at least 3 of the 5 scans are done, you may call '''focus_liss, day, [scan_num1, scan_num2, scan_num3]''' to have a first guess of the optimal focus		  * As soon as the data of a scan of this sequence are on Sami's disk, you may run '''nk_rta, scan'''
 * When at least 3 of the 5 scans are done, you may call '''focus_liss, day, [scan1, scan2, scan3]''' to have a first guess of the optimal focus
Line 95: Line 219:
==== cont_beammap ====

{{attachment:otf_geometry.png||width=400}}

 * Instead of rta_reduce, please call '''otf_geometry, day, scan_num'''
 * The top plots compare the current derived beam maps to the reference ones
 * The bottom plots display the distance between the current beam parameters and the reference ones in terms of centroids positions, FWHM and absolute calibration.


=== Science scans ===
Line 109: Line 221:
 * rta_reduce, day, scan_num  * nk_rta, scan
Line 112: Line 224:
=== Diffuse emission === ==== Diffuse emission ====
Line 117: Line 229:
----

== Neel pipeline ==

The following diagram shows the NIKA data flow from the IRAM 30m telescope to the observers' accounts.

{{attachment:NIKApipeline.png | Neel Pipeline| height=600}}

NIKA Data Reduction

The following diagram shows the NIKA data flow from the IRAM 30m telescope to the observers' accounts.

Neel Pipeline

Contents

  1. NIKA Data Reduction
    1. Offline data reduction
      1. Data products v0
      2. Data products v1
      3. Data products v2
    2. Online data reduction

Go to the NIKA Main Wiki.

Offline data reduction

Data products v0

The version v0 of the NIKA first open pool products has been made by the NIKA team by the end of the pool (Feb 28,2014) and it was distributed by IRAM. The products are described here.

For each project, there are fits files containing the maps of each scan and combined scans of each object. The data processing has been oriented towards point-source extraction or diffuse emission depending on the aim of your project.

There are also logbook files and illustrative pdf files. On the main directory you will find:

  • logfile_all_scans.csv: is a comma separated file ready for your favorite spreadsheet handler (libreoffice or excel). It gives the summary information of all the scans that could be used including opacity measurements, sky noise in Jy (per band of frequency from 1mHz to 10 Hz), camera noise NEFD (in mJy.s^1/2) and finally a flux in mJy of a point-source at the center of the map (fixed central position, fixed FWHM, Gaussian fit) with the error at both wavelengths.

Then for each object, there is a directory that contains:

  • the fits files.

  • flux_source.csv: a comma separated file giving the flux as measured in the scan maps and the combined scans maps.

  • Figures: many pdf files were generated out of the fits files to get a quick look at them. Units in Jy.

To retrieve the data, please type: 

scp  XXX-YY@enter.iram.es:~/observationData/nika/v0.tar.gz  /path/in/your/computer

where XXX-YY is your project number. If you get an error message, enter the following command: 

ssh-keygen -R enter.iram.es

and then try to retrieve the data again.

CAVEAT: At this stage, the data offline products are not of scientific quality. The selection of scans was unfortunately rendered difficult because of an unexpected problem with the pointing data. We hope to solve that problem with post-processing steps that we did not have time to implement. Instead, scans were manually selected with some mistakes still present. The photometry is probably correct at the 20% level at 1mm and 10% level at 2mm for scans of good quality. Don't shoot. We will prepare the next version in the coming weeks. Contact your NIKA friend of project to give us your feedback.

The NIKA team

Data products v1

This version V1 of the NIKA first open pool products has been made by the NIKA team by the end of May 2014. It is distributed by IRAM. It follows the quick release done at the end of the pool (v0). The products are described here

For each project, there are fits files containing the maps of each scan and combined scans of each object. The data processing has been oriented towards point-source extraction or diffuse emission depending on the aim of your project. The main processing has improved by including more scans and dealing with internal problems more effectively.

There are also logbook files and illustrative pdf files. On the main directory you will find:

  • logfile_all_scans.csv : is a comma separated file ready for your favorite

spreadsheet handler (libreoffice or excel) It gives the summary information of all the scans that could be used including opacity measurements, sky noise in Jy (per band of frequency from 1mHz to 10 Hz), camera noise NEFD (in mJy.s^1/2) and finally a flux in mJy of a point-source at the center of the map (fixed central position, fixed FWHM, Gaussian fit) with the error at both wavelengths. For pointing sources the position is allowed to vary.

Then for each object, there is a directory that contains:

  • The fits files (RaDec2000 projection for target sources, AzEl projection for pointing sources and Planets)

  • flux_source.csv : a comma separated file giving the flux as measured in the

scan maps and the combined scans maps.

  • Figures: many pdf files were generated out of the fits files to get a quick

look at them. Units in Jy/beam unless otherwise stated.

The calibration was done assuming a Gaussian main beam of 12.5 and 18.5 arcseconds (FWHM). The primary calibrator is Uranus with fluxes of 36.4 and 15.3 Jy.

For Lissajous scans (V1Liss directories), the pipeline introduces a filtering effect for point sources of about 0.70 and 0.56 (at 1 and 2mm). This effect has been corrected in the .csv files but not in the maps. Thus fluxes in Jy in the .csv files should be correct.

For you to assess the filtering effect, we have rerun the pipeline with a much less agressive decorrelation that should not affect the fluxes. For that just look at V1cLiss directories made for calibration sources only.

For OTF maps (directories V1otf): the main beam to full beam correction is by 1.56+-0.10 at 1mm and 1.35+-0.10 at 2mm. It has not been applied to the maps.

At this stage, the offline products can be used for a first scientific analysis as most of the scans have been included. The point-source photometry is probably correct at the 15% level at 1mm and 10% level at 2mm.

To retrieve the data, please type: 

scp  XXX-YY@enter.iram.es:~/observationData/nika/v1.tar.gz  /path/in/your/computer

where XXX-YY is your project number. If you get an error message, enter the following command: 

ssh-keygen -R enter.iram.es

and then try to retrieve the data again.

The NIKA team

Data products v2

We hope to deliver a final V2 version of the first NIKA open pool data products in the Summer including our best strategy of systematic removals. Your feedback is essential to improve the quality of the v1 data and help the NIKA team to develop the final version.

The NIKA team

Online data reduction

To start the IDL session for the real time analysis, open a terminal and type: 

   $ ssh -X nikas-14@mrt-lx1
   $ ssh_sami
   > idl

Shortly after a scan is done, the NIKA scientific data, the xml information generated by PAKO (and the AntennaIMBfits shortly after) are written on SAMI and can be processed. There are two types of observation: the “science” scans and the “calibration” scans. All scans should be reduced with the same routine (except the "focus_liss" sequence as of now, see below):

IDL> nk_rta, scan

scan is a string the format YYYYMMDDsN, e.g. '20141112s3'. Mind the simple quote, IDL does not accept double quotes when the string starts by a number.

If this is a calibration scan (pointing, focus), informations or actions to be taken by the observer appear explicitely in IDL terminal window:

cross

cross_1.png

  • The above window shows up (after 3 sets of monitoring plots that show examples of timelines and power spectra)
  • The first column show the azimuth and elevation timelines and the samples that were flagged out based on their instantaneous speed.
  • The second column plots show 1D profiles of the azimuth or elevation scans at both wavelengths for the reference detector and a gaussian fit of the source along these profiles. The displayed offsets refer to the reference detectors and are not meant to be passed to PAKO.
  • The maps show the source, as mapped by all the detectors, a large cross at the location of the reference pixel, a gaussian fit of the source in red and the map center in green for reference.
  • In the terminal window, the observer should follow the instruction (e.g.): B 2mm (MAP) (for PAKO) SET POINTING xx.x yy.y
  • Notes:
  • *the pointing reference is the 2mm channel.
  • * If the sky conditions are correct, instructions at 1 and 2mm should be in good agreement.
  • * if a glitch corrupts the maps but the 1d profiles are ok, the user can follow the instruction B 2mm, : (TOI) (for PAKO) SET POINTING xx.x yy.y
  • * In good conditions, all instructions agree to better about 1 arcsec.

  • * If the pointing is far off, the default reference pixels may not see the source and the timelines may look bad. In the terminal window, you may read the ID of the closest kids to the source and pass them to nk_rta as keywords e.g. nk_rta, scan, ref_det_1mm=X, ref_det_2mm=Y.

pointliss

pointing_liss.png

  • The routine displays one map for each band, an estimation of the flux where the source is fit (red), an estimation of the flux at the center of the map (green). It also provides a quick estimate of the NEFD and the map RMS for 10arcsec equivalent pixels and the values of Tau derived by NIKA.
  • The observer should follow instructions given in the terminal, e.g.: B 2mm, : (MAP) (for PAKO) SET POINTING -4.7, 1.4

focusp

[ATTACH]

  • By default, optimize the focus at 1mm

cont_skydip

skydip_results.png

  • No action is requested from the observer other than checking that the displayed histograms look reasonable. These results will be used by the NIKA team later on offline.

focusliss

pointing_liss.png fits.png

  • As soon as the data of a scan of this sequence are on Sami's disk, you may run nk_rta, scan

  • When at least 3 of the 5 scans are done, you may call focus_liss, day, [scan1, scan2, scan3] to have a first guess of the optimal focus

  • When the 5 scans are done, call focus_liss with the complete list of scans and follow instructions in the terminal window and by default, optimize the focus at 1mm

Point source maps

  • nk_rta, scan

  • If you want to coadd several scans, run combine_scans, scan_list. Note however that to allow to combine scans from different days, scan_list is here defined slightly differently, it is an array of strings of the format YYYYMMDDsScan_num, e.g. '20140127s290'

Diffuse emission

  • rta_reduce, day, scan_num, /diffuse
  • to combine several scans, use combine_scans in the same way as for point source maps

Continuum/NIKA/DataReduction (last edited 2014-11-07 09:05:40 by NikaBolometer)