Size: 312
Comment:
|
Size: 12631
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 7: | Line 7: |
'''[[ http://www.iram.es/IRAMES/mainWiki/NIKA/Main | Go back to the NIKA Main Wiki ]]''' | Go back to the '''[[ http://www.iram.es/IRAMES/mainWiki/NIKA/Main | NIKA ]]''' main wiki. |
Line 11: | Line 11: |
== Task list == | == Goals of the run == |
Line 13: | Line 13: |
* IH: Create the organization wiki | 1. Check that gaps in NCS data streams do not occur 1. Debug the IMBFITS formating so that they are at last produced without errors, and finally allow real time processing. 1. Test for the first time operations with only IRAM Granada personnel at the Telescope and NIKA Grenoble team using remote control. == Organization Plan == The dominant components of the goals 1 and 2 do not necessitate sensitive KID, so no cool down of the instrument is necessary. However the full instrumental setup for observations must be operational; this includes the electronic chain and real scans of the telescope commanded via !PaKo. Remark: We could test the new network cabling installed in the cable spiral room for the future control of the NIKA2 compressors, keeping the electronics at its storage location and making it run to control the detectors acquisition via the network from there. But we have decided to lift it up in the cabin hence mimicking real observations as much as possible with a warm cryostat, since the problems we had may be related to the network setup we used. We will debug the IMBFITS step by step. People involved in the NIKA data & IMBFITS making & checking (Albrecht, Juan MP, Xavier, Alain, Robert, Samuel, and possibly Walter and Nicolas P) communicate via this wiki page to establish very clearly the main problems and actions to do to solve them (see tables below), and possibly use the data we currently have to start debugging, so that we are as ready as possible for May 7. Remark: We need confirmation that the warm KIDs with the observation setup planed for this dry run will produce usable (in math. sense) values, otherwise this might cause somewhere problems. After this dry run, we will make the status, and determine a date for another short test run (e.g. begin of June ?). Hopefully for this one only the problems that necessitate valid KID data remain (e.g. KID flags). In that case this short run would be cold. == Dates == Telescope time booked for the NIKA Dry run test: * Wednesday May 7, from 11:00 to 15:00. * Monday May 12, from 12:00 to 16:00. They appear in the [[http://www.iram.fr/IRAMFR/PV/sche/14/s19v1.html|official telescope schedule]] under the project name is T21-13. == Table of IMBFITS problems to solve == The NIKA IMBFITS have had problems since their first creation (2^nd^ test run, Oct 2010). The latest exhaustive list of problems dates from the run 6 (June 2013) processing and has been [[http://www.iram.fr/wiki/nika2/index.php/RZdataProc|tabulated on a dedicated wiki page]]; most of them are still valid. For reading convenience we gather here only the most important problems that need to be solved with the May 7 dry run. Remark: Note that problems 4, 5 & 6 are most probably related. || ''' Problem number (origin) ''' || ''' Description of the problem ''' || ''' Status ''' || ''' How to identify & diagnose the problem ''' || ''' How to investigate and possibly solve the problem '''|| || 1 (NCS) || Some scans miss part(s) of the antenna traces. || Pandemic during pool 1 (run 8, Feb 2014), but marginal after mrt-lx1 kernel update by GP; to be verified as goal #1 of this dry run. || Monitor time vs sample number of antenna traces. || Monitor network & mrt-lx1 activity.|| || 2 (IMBFITS & NIKA) || KID parameter coded in IMBFITS mismatch the data (e.g. numbers of KIDs) - fatal error. || Partially solved: AS's IMBFITS making script should now read the NIKA Kidpar, but we miss a description of the Kidpar by the NIKA team. || Compare at least the number of KIDs in the IMBFITS and of the hardware, e.g. with any FITS viewer. || Need better communication between NIKA team (FXD ?) & IRAM (AS, SL, RZ).|| || 3 (IMBFITS or NCS ?) || Mis-synchronization of subscans in the IMBFITS tables: some subscans are incomplete and partially appear in the following subscan table - fatal error. || This problem appeared first time in IMBFITS of run 6; not solved; it is due to a synchronization problem in NCS and/or IMBFITS creation. || Check subscan length in IMBFITS, e.g. with any FITS viewer. || Check the time used as subscan start and end, delay of NCS messages ...|| || 4 (NIKA) || NIKA FITS do not correspond to the time interval of observation. || Permanent problem since the first try in Oct 2010; not solved. || At least the last subscan missing. Visible with any FITS viewer. || Creation of the NIKA FITS (in IDL ?) not OK. || || 5 (NIKA) || NIKA tables in IMBFITS empty. || Permanent problem since the first try in Oct 2010; not solved. || Visible with any FITS viewer. ||IMBFITS created offline do not show this problem, i.e. NIKA FITS were not available when makeIMBFITS was called in auto mode. || || 6 (NIKA) || IMBFITS are created many minutes after the observation, e.g. after 2 further pointings ! || Permanent problem since the first try in Oct 2010; not solved. || Compare the end time of observations and of creation of IMBFITS; the last takes seconds. || #5 & 6 point to a significant delay in creation of the NIKA FITS. || || 7 (NIKA) || KIDs flags & types wrong (good/bad KID mis-identified). || Solved after run 8 on NIKA side. Need to be double checked. || Create IMBFITS with updated/correct flags and match KID characteristics with flag & type signification. || Need detailed documentation on the KID params.|| Anyone interested in the file sizes of the antenna streams can take a look at the continously updated plot created by PM: https://mrtweb.iram.es/plotAmd/ == Step by step guide for the Dry Run == === Hardware preparation === * Installation of the NIKA electronics at the receiver cabin (done). * Switch on the electronics (done). * Installation of the computers at the control room? === Data acquisition === * '''NIKA team''' * Start the software (Tue 6th before the system check) * '''IRAM computer group''' * The nikaw-13 computer account will be used for the observations. * The NIKA team must inform us about the software that we need to run. * Installation of AB's Elvin and NCS monitor software? === Data reduction === In order to mimic as much as possible a typical NIKA session the '''online pipeline''' must be running. The instructions to start the online pipeline and to reduce the data can be found [[ http://www.iram.es/IRAMES/mainWiki/Continuum/NIKA/DataReduction#Online_data_reduction | here ]]. === Observing session === The observations will be done using the account created for the 1st NIKA pool: {{{ $ ssh -X nikaw-13@mrt-lx1 (ask I. Hermelo for the password) }}} The operator has to set the project name to nikaw-13. To open the !PaKo version for NIKA type: {{{ $ gopako $ pakodisplay $ pakoNIKA }}} Once !PaKo is open, you can run the following script: {{{ PAKO> @ observe_DryRun }}} The script simulates a typical NIKA session of ~2 hours of duration: {{{ PAKO> SET PROJECT t21 PAKO> SOURCE Uranus PAKO> @ cont_pointing PAKO> SET POINTING 5 5 PAKO> @ cont_focusZ -1.8 PAKO> SET FOCUS -1.5 PAKO> SOURCE 3C454 /cat * PAKO> @ cont_onthefly 10 20 0 -45 PAKO> @ cont_onthefly 10 20 0 +45 PAKO> @ cont_lissajous 2 PAKO> @ cont_onthefly 10 20 45 -45 PAKO> @ cont_onthefly 10 20 45 +45 PAKO> @ cont_lissajous 4 PAKO> @ cont_onthefly 10 20 90 -45 PAKO> @ cont_onthefly 10 20 90 +45 PAKO> @ cont_lissajous 6 PAKO> @ cont_onthefly 10 20 135 -45 PAKO> @ cont_onthefly 10 20 135 +45 PAKO> @ cont_lissajous 8 }}} The following figure shows the sky at the beginning of the first slot of the NIKA Dry Run. {{ attachment:SkyNikaDryRun.png | SkyNikaDryRun | width=1000 }} == Debriefing of the run == === Wed 7th May === 1. NIKA electronics was plugged few days beforehand. Data were obtained a day before the run without problems 1. Usual setups at the start took 2 hours to work (ssh keys are changed, data place on computer, acquisition computer X33, imbfits generator with nika_filing) 1. Telescope got stuck for a computer reason for half an hour 1. Nothing significant in data delays. Some singular errors : a few holes in the mjd from the antenna. Typically 5-6 errors during the 4 hours. 1. Standard series of scans logged on Tapas ('''Issue:''' Tapas does not list the on-the-fly scans) 1. After that trial period, the last 1h30 went smoothly 1. Running acquisition software at 60 Hz (instead of 29) without pointing errors (although the nika files are full of 0). Daemon on mrt-lx1 to acquire antenna pointing is running smoothly. 1. Plan for Monday second dry run is to do the whole thing again to check against accumulating errors if any and have another statistics on holes in the antenna data. 1. ''Who are the contacts at Neel ? Do we miss documentation on how to set-up the software by IRAM staff ? <CK>'' It is not foreseen that NIKA software can be started by IRAM staff. For NIKA2, we'll try to make an automatic start system for IRAM staff to operate. For NIKA, remote operations by Neel team are required. ''I'd suggest to start writing instructions for IRAM staff already for NIKA, as it allows to gradually increase knowledge, awareness, and also some responsibility. It also allows to eventually create a good useful NIKA2 documentation. <CK>'' === Mon 12th May === 1. The run was prepared and tuning was done at the beginning. Having zeros in the data was corrected by resetting the NIKEL boxes at scan 130. 1. Antenna got stuck. Fixed after reboot of vac1 and go_kill. One hour lost. 1. Really useful data acquired at scan 138. See Tapas 1. Data acquisition set at 79.5 Hz to stress the new Mac computer! Errors appear only in the ethernet input buffer if interactive use of Mac (by changing the window size with the mouse for example scan 147). 1. No antenna datastream error after scan 139! 1. First glance at IMBFITS: They are created with readable data (no apparent corruption, real values in the tables). But data is missing at the end of the Lissajous scans (0.1 to 7 seconds missing). There's only 1 (track) focus and 1 (cross) pointing; they are readable but we miss 1/3rd of the 1st subscan. 3 apparent problems: noise increase with resonance frequency (explained), delay between subscans too short, instabilities in one data parameter; more explanations on these problems will be given in the coming days, after a deeper analysis. === Some conclusions from the IMBFITS analysis (RZ, SL) === * TAPAS still shows only the Lissajous. It's about time to correct this problem and display the other type of scans (zig-zag OTF, (track) focus, (cross) pointing, etc.)! * Available as imbFITS are just 3 maps, 1 pointing and 1 focus, 0 skydips, all the rest is Lissajous; i.e. the current data allow statistically relevant investigation only for Lissajous. However, there is only 1 (!!) imbFITS which does not show errors => still problems in the transfer of NIKA data into IMBFITS. * Scans missing: 20140507s151 => corrupted because NIKA frequency changed; 20140512s124 => aborted because of a system restart, 20140512s130 to 20140512s135 => Reseting NIKEL boxes, and then telescope got stuck. * Only 3 Zig-Zag OTF maps on May 7, none on May 12. Scans 144 and 145 have empty NIKA data tables, scan 142 has empty NIKA data table for the 10 last subscans, why ? * The focus scan (20140512s168) has 47% of NIKA data missing in the first subscan, all Lissajous show missing data at the beginning and/or the end of the subscan; the number of missing records vary from a few to thousands, i.e. from a fraction of a second to a minute, why ? '''Answer from FXD: At about 29 Hz, a 10 second subscan should have about 300 samples which is the case for most subscans except the first and the last ones. For the first subscan, it was decided at the beginning of the open pool to have a final tuning at the beginning of the 1st subscan. That explains that we loose 3 seconds of the first subscan. Nothing we can do or want to do about that. For the last subscan, we trig the subscan valid data on the subscan status = 5 (ie subscan done). But this does not exist for some reason on the last subscan, hence we throw the last subscan away... This could be investigated on your side ?''' * The pointing scan (20140512s167) is the only file readable by MOPSIC without error messages ! |
NIKA dry run organization
Contents
Go back to the NIKA main wiki.
Goals of the run
- Check that gaps in NCS data streams do not occur
- Debug the IMBFITS formating so that they are at last produced without errors, and finally allow real time processing.
- Test for the first time operations with only IRAM Granada personnel at the Telescope and NIKA Grenoble team using remote control.
Organization Plan
The dominant components of the goals 1 and 2 do not necessitate sensitive KID, so no cool down of the instrument is necessary. However the full instrumental setup for observations must be operational; this includes the electronic chain and real scans of the telescope commanded via PaKo.
- Remark: We could test the new network cabling installed in the cable spiral room for the future control of the NIKA2 compressors, keeping the electronics at its storage location and making it run to control the detectors acquisition via the network from there. But we have decided to lift it up in the cabin hence mimicking real observations as much as possible with a warm cryostat, since the problems we had may be related to the network setup we used.
We will debug the IMBFITS step by step. People involved in the NIKA data & IMBFITS making & checking (Albrecht, Juan MP, Xavier, Alain, Robert, Samuel, and possibly Walter and Nicolas P) communicate via this wiki page to establish very clearly the main problems and actions to do to solve them (see tables below), and possibly use the data we currently have to start debugging, so that we are as ready as possible for May 7.
- Remark: We need confirmation that the warm KIDs with the observation setup planed for this dry run will produce usable (in math. sense) values, otherwise this might cause somewhere problems.
After this dry run, we will make the status, and determine a date for another short test run (e.g. begin of June ?). Hopefully for this one only the problems that necessitate valid KID data remain (e.g. KID flags). In that case this short run would be cold.
Dates
Telescope time booked for the NIKA Dry run test:
- Wednesday May 7, from 11:00 to 15:00.
- Monday May 12, from 12:00 to 16:00.
They appear in the official telescope schedule under the project name is T21-13.
Table of IMBFITS problems to solve
The NIKA IMBFITS have had problems since their first creation (2nd test run, Oct 2010). The latest exhaustive list of problems dates from the run 6 (June 2013) processing and has been tabulated on a dedicated wiki page; most of them are still valid. For reading convenience we gather here only the most important problems that need to be solved with the May 7 dry run.
Remark: Note that problems 4, 5 & 6 are most probably related.
Problem number (origin) |
Description of the problem |
Status |
How to identify & diagnose the problem |
How to investigate and possibly solve the problem |
1 (NCS) |
Some scans miss part(s) of the antenna traces. |
Pandemic during pool 1 (run 8, Feb 2014), but marginal after mrt-lx1 kernel update by GP; to be verified as goal #1 of this dry run. |
Monitor time vs sample number of antenna traces. |
Monitor network & mrt-lx1 activity. |
2 (IMBFITS & NIKA) |
KID parameter coded in IMBFITS mismatch the data (e.g. numbers of KIDs) - fatal error. |
Partially solved: AS's IMBFITS making script should now read the NIKA Kidpar, but we miss a description of the Kidpar by the NIKA team. |
Compare at least the number of KIDs in the IMBFITS and of the hardware, e.g. with any FITS viewer. |
Need better communication between NIKA team (FXD ?) & IRAM (AS, SL, RZ). |
3 (IMBFITS or NCS ?) |
Mis-synchronization of subscans in the IMBFITS tables: some subscans are incomplete and partially appear in the following subscan table - fatal error. |
This problem appeared first time in IMBFITS of run 6; not solved; it is due to a synchronization problem in NCS and/or IMBFITS creation. |
Check subscan length in IMBFITS, e.g. with any FITS viewer. |
Check the time used as subscan start and end, delay of NCS messages ... |
4 (NIKA) |
NIKA FITS do not correspond to the time interval of observation. |
Permanent problem since the first try in Oct 2010; not solved. |
At least the last subscan missing. Visible with any FITS viewer. |
Creation of the NIKA FITS (in IDL ?) not OK. |
5 (NIKA) |
NIKA tables in IMBFITS empty. |
Permanent problem since the first try in Oct 2010; not solved. |
Visible with any FITS viewer. |
IMBFITS created offline do not show this problem, i.e. NIKA FITS were not available when makeIMBFITS was called in auto mode. |
6 (NIKA) |
IMBFITS are created many minutes after the observation, e.g. after 2 further pointings ! |
Permanent problem since the first try in Oct 2010; not solved. |
Compare the end time of observations and of creation of IMBFITS; the last takes seconds. |
#5 & 6 point to a significant delay in creation of the NIKA FITS. |
7 (NIKA) |
KIDs flags & types wrong (good/bad KID mis-identified). |
Solved after run 8 on NIKA side. Need to be double checked. |
Create IMBFITS with updated/correct flags and match KID characteristics with flag & type signification. |
Need detailed documentation on the KID params. |
Anyone interested in the file sizes of the antenna streams can take a look at the continously updated plot created by PM: https://mrtweb.iram.es/plotAmd/
Step by step guide for the Dry Run
Hardware preparation
- Installation of the NIKA electronics at the receiver cabin (done).
- Switch on the electronics (done).
- Installation of the computers at the control room?
Data acquisition
NIKA team
- Start the software (Tue 6th before the system check)
IRAM computer group
- The nikaw-13 computer account will be used for the observations.
- The NIKA team must inform us about the software that we need to run.
- Installation of AB's Elvin and NCS monitor software?
Data reduction
In order to mimic as much as possible a typical NIKA session the online pipeline must be running. The instructions to start the online pipeline and to reduce the data can be found here.
Observing session
The observations will be done using the account created for the 1st NIKA pool:
$ ssh -X nikaw-13@mrt-lx1 (ask I. Hermelo for the password)
The operator has to set the project name to nikaw-13. To open the PaKo version for NIKA type:
$ gopako $ pakodisplay $ pakoNIKA
Once PaKo is open, you can run the following script:
PAKO> @ observe_DryRun
The script simulates a typical NIKA session of ~2 hours of duration:
PAKO> SET PROJECT t21 PAKO> SOURCE Uranus PAKO> @ cont_pointing PAKO> SET POINTING 5 5 PAKO> @ cont_focusZ -1.8 PAKO> SET FOCUS -1.5 PAKO> SOURCE 3C454 /cat * PAKO> @ cont_onthefly 10 20 0 -45 PAKO> @ cont_onthefly 10 20 0 +45 PAKO> @ cont_lissajous 2 PAKO> @ cont_onthefly 10 20 45 -45 PAKO> @ cont_onthefly 10 20 45 +45 PAKO> @ cont_lissajous 4 PAKO> @ cont_onthefly 10 20 90 -45 PAKO> @ cont_onthefly 10 20 90 +45 PAKO> @ cont_lissajous 6 PAKO> @ cont_onthefly 10 20 135 -45 PAKO> @ cont_onthefly 10 20 135 +45 PAKO> @ cont_lissajous 8
The following figure shows the sky at the beginning of the first slot of the NIKA Dry Run.
Debriefing of the run
Wed 7th May
- NIKA electronics was plugged few days beforehand. Data were obtained a day before the run without problems
- Usual setups at the start took 2 hours to work (ssh keys are changed, data place on computer, acquisition computer X33, imbfits generator with nika_filing)
- Telescope got stuck for a computer reason for half an hour
- Nothing significant in data delays. Some singular errors : a few holes in the mjd from the antenna. Typically 5-6 errors during the 4 hours.
Standard series of scans logged on Tapas (Issue: Tapas does not list the on-the-fly scans)
- After that trial period, the last 1h30 went smoothly
- Running acquisition software at 60 Hz (instead of 29) without pointing errors (although the nika files are full of 0). Daemon on mrt-lx1 to acquire antenna pointing is running smoothly.
- Plan for Monday second dry run is to do the whole thing again to check against accumulating errors if any and have another statistics on holes in the antenna data.
Who are the contacts at Neel ? Do we miss documentation on how to set-up the software by IRAM staff ? <CK> It is not foreseen that NIKA software can be started by IRAM staff. For NIKA2, we'll try to make an automatic start system for IRAM staff to operate. For NIKA, remote operations by Neel team are required. I'd suggest to start writing instructions for IRAM staff already for NIKA, as it allows to gradually increase knowledge, awareness, and also some responsibility. It also allows to eventually create a good useful NIKA2 documentation. <CK>
Mon 12th May
- The run was prepared and tuning was done at the beginning. Having zeros in the data was corrected by resetting the NIKEL boxes at scan 130.
- Antenna got stuck. Fixed after reboot of vac1 and go_kill. One hour lost.
- Really useful data acquired at scan 138. See Tapas
- Data acquisition set at 79.5 Hz to stress the new Mac computer! Errors appear only in the ethernet input buffer if interactive use of Mac (by changing the window size with the mouse for example scan 147).
- No antenna datastream error after scan 139!
- First glance at IMBFITS: They are created with readable data (no apparent corruption, real values in the tables). But data is missing at the end of the Lissajous scans (0.1 to 7 seconds missing). There's only 1 (track) focus and 1 (cross) pointing; they are readable but we miss 1/3rd of the 1st subscan. 3 apparent problems: noise increase with resonance frequency (explained), delay between subscans too short, instabilities in one data parameter; more explanations on these problems will be given in the coming days, after a deeper analysis.
Some conclusions from the IMBFITS analysis (RZ, SL)
- TAPAS still shows only the Lissajous. It's about time to correct this problem and display the other type of scans (zig-zag OTF, (track) focus, (cross) pointing, etc.)!
Available as imbFITS are just 3 maps, 1 pointing and 1 focus, 0 skydips, all the rest is Lissajous; i.e. the current data allow statistically relevant investigation only for Lissajous. However, there is only 1 (!!) imbFITS which does not show errors => still problems in the transfer of NIKA data into IMBFITS.
Scans missing: 20140507s151 => corrupted because NIKA frequency changed; 20140512s124 => aborted because of a system restart, 20140512s130 to 20140512s135 => Reseting NIKEL boxes, and then telescope got stuck.
- Only 3 Zig-Zag OTF maps on May 7, none on May 12. Scans 144 and 145 have empty NIKA data tables, scan 142 has empty NIKA data table for the 10 last subscans, why ?
The focus scan (20140512s168) has 47% of NIKA data missing in the first subscan, all Lissajous show missing data at the beginning and/or the end of the subscan; the number of missing records vary from a few to thousands, i.e. from a fraction of a second to a minute, why ? Answer from FXD: At about 29 Hz, a 10 second subscan should have about 300 samples which is the case for most subscans except the first and the last ones. For the first subscan, it was decided at the beginning of the open pool to have a final tuning at the beginning of the 1st subscan. That explains that we loose 3 seconds of the first subscan. Nothing we can do or want to do about that. For the last subscan, we trig the subscan valid data on the subscan status = 5 (ie subscan done). But this does not exist for some reason on the last subscan, hence we throw the last subscan away... This could be investigated on your side ?
- The pointing scan (20140512s167) is the only file readable by MOPSIC without error messages !