SA wiregrid analysis

Preparations

Requirements of python libraries

There are several python libraries aside from the libraries installed by env-shell.sh. Please intall them by

pip3 install -r requirements.txt

Do NOT forget to use pip3 instead of pip!

Requirements for SA pipeline library

loadbolo_pipeline.py needs SA pipeline libraries(simons_array_offline_software). Please make a symbolic link to your SA pipeline library as ./library/simons_array_offline_software.

1. $ mkdir library
2. $ ln -s library/simons_array_offline_software

Requirements for databases

To make a DB by using wiregrid, you need to prepare two DBs:

1. Kyohei's planet DB: data/ykyohei/mapping/pb2a_mapping_postvX.db
   • Please copy from tostada:/home/ykyohei/mydatabase.
2. An old pb2a_focalplane DB: data/pb2a-20210205/pb2a_mapping.db
   • Please copy or make a link to tostada:/data/pb2/ChileData/databases.

Setting

Environmental setting:

. env-shell.sh

Versions

output_ver2

• version description: first version having theta_det (DB + each bolometer demod fit/TODs)
• input data: raw data
• data : 2021/02/05 Run22300609
• output_ver2/db/all.db
  • CREATE TABLE wiregrid(id INTEGER PRIMARY KEY,boloname STRING,lmfit_x0 REAL,lmfit_y0 REAL,lmfit_a REAL,lmfit_b REAL,lmfit_alpha REAL,lmfit_chisqr REAL,x0 REAL,y0 REAL,theta0 REAL,a REAL,b REAL,r REAL,alpha REAL,x0_err REAL,y0_err REAL,theta0_err REAL,a_err REAL,b_err REAL,r_err REAL,alpha_err REAL,chisqr REAL,wireangle0 REAL,wireangle0_err REAL,theta_wire0 REAL,theta_wire0_err REAL);
  • Maybe this DB has bug or wrong data.

output_ver3

• version description: first version having theta_det (DB + each bolometer demod fit)
• input data: output_ver2 each bolometer TOD
• output_ver3/db/all.db
  • theta_det : detector angle (NO time-constant correction)
  • theta_det_err : detector angle error (but it is NOT correct now.)

output_ver4

• version description: theta_det has time-constant correction (only DB)
• input data: output_ver3 DB
• output_ver4/db/all_pandas_correct_label.db
  • theta_det : detector angle (Wt time-constant correction)
    • If there is no stimulator data, theta_det is not corrected. (2710/4324 bolos are corrected.)
  • theta_det_err : detector angle error (but it is NOT correct now.)
  • Stimulator data used for the correction is also included.

output_ver5

• version description: DB with each wire angle demod data [(x,y) in demode complex plane]
  • reproduce from demod fit in ver3

output_ver6

• version description: add wire_angle=180 deg. data
  • change the time period of 180 deg. data (select only first 20 sec. This is similar length of the other data.)
  • reproduce from g3 data

output_ver7, plot_ver7

• version description: change the libg3py3.py & loadbolo.py
  • Use libg3py3_v2.py and loadbolo_v2.py (Old name: libg3py3_test.py, loadbolo_test.py)
  • Change to import loadbolo_v2.py instead of loadbolo.py in OneAngleData.py
  • Modify reference offset in HWP angle calculation (using getOffsets() func. in libg3py3_v2.py
  • Add implementation of stimulator template DB to retrieve measured stimulator temperature in grid_rotation_analysis.py
    • Use averaged temperature measured by Jupiter
  • output_ver7 is for bsub jobs. / plot_ver7 is for test.

output_ver8, plot_ver8

• version description: restore the change of the libg3py3.py & loadbolo.py
  • Use libg3py3.py and loadbolo.py
  • Change to import loadbolo.py instead of loadbolo_test.py in OneAngleData.py
  • Main change from ver6: Add implementation of stimulator template DB to retrieve measured stimulator temperature in grid_rotation_analysis.py
    • Use averaged temperature measured by Jupiter
  • demod wire_angle=180 deg data, but do not use it in fitDemodResult.py.
  • reproduce from TOD pickle file in ver6, which (output_ver6/pkl///A.._..deg.pkl were copied to output_ver8/pkl///

output_ver9, plot_ver9

• version description: Use libg3py3_v2.py & loadbolo_v2.py
  • Use libg3py3.py and loadbolo.py
  • Change to import loadbolo.py --> loadbolo_v2.py in OneAngleData.py
    • HWP encoder 0 point should be corrected.
  • demod wire_angle=180 deg data, but do not use it in fitDemodResult.py.

ver9.2 (tag, branch)

The last version using libg3py3.py

ver10

• version description: Use sa_pipeline_software to retrieve TOD & HWP encoder angle data

From this version, sa pipeline is used to read TOD. To use simons_array_offline_software at kekcc, there are several modifications.

1. Make a sa_config.py file for myself (in library/mychange)
2. Comment-out from toast import qarray as qa
   • Importing toast has an issue at kekcc.
3. Comment-out from .sa_cuts import ... to avoid toast importing
4. Add paths on PYTHONPATH in env-shell.sh

Other modification on simons_array_offline_software:

1. To use time clip operation on the TOD, I modified sa_pipline_filters.py/OperatorClipBeginEnd()
2. To fix bug in sa_pipline_filters.py/OperatorClipBeginEnd(), add items on the list that would go with bolometer (L195).

• Update database to 20211004 (database/pb2a-20211004) after fitDemod

  • Amplitude calibration of ADC counts: NOT updated
  • Tau correction:
    • updated in merge.py: pb2a-20211004/pb2a_stim.db
      • It has duplicate bolometers. (Use DISTINCT to retrieve sqlite3 database.)
      • To avoid the duplicated bolos, add new selection run_subid=='[1, 4, 7, 10, 13, 16, 19, 22]'
  • pb2a_mapping (offset_det_x,y):
    • NOTE: No updating in merge.py because newer database than p22a-20210205 has calibrated pol_angles but it needs design values in pol_angle.
  • Kyohei's pb2a_focalplane DB :
    • updated in compare_DB_for_labelcorrection.py: ykyohei/mapping/pb2a_focalplane_postv4.db
      • (The HWM hash is not changed) 6f306f8261c2be68bc167e2375ddefdec1b247a2

• Add errors, tau correction in wiregrid DB

  • Add systematics from tau_err in merge.py
    • theta_det_err : (Same as before) Statistical error of theta_det calibration
    • thete_det_err_tau : Error from tauerr in tau calibration
    • thete_det_err_total : Root of squared sum of the above errors
  • Add tau correction on theta_det in merge.py
    • theta_det_taucorr : - 2 * tau * hwp_speed * 2pi
    • theta_det : theta_det original + theta_det_taucorr

• Fill 0 in NULL in tau, tauerr in wiregrid in merge.py

• Update compare_DB_for_labelcorrection.py for ver10

  • Wiregrid DB is merged with an old pb2a_focalplane DB which has original detector labels and design pol_angle.
  • It is compared with Kyohei's DB (postv4) to get corrected pixel labels.
  • Drop NaN bolometers in readout_name or theta_det (wiregrid data) for label corrected DB
  • Add a column of 'isCorrectLabel'
    • True if mislabel is corrected or not mislabel.
  • Fix bugs in bolo_name or pixel_number

• Update check_absolute_nocorr.py for ver10

  • Remove old check_absolute_nocorr.py & create from check_absolute.py
  • No plots with det_offset_x/y or mislabel because the DB before label correction does not have them
  • Outliers cut: >45deg --> >15deg.

• Update check_absolute.py for ver10

  • Outliers cut: >45deg --> >15deg.

• Add modifyDB.py

  • Make a pb2a_wiregrid_ver10.db from all_pandas_correct_label.db.
  • Modify & check the DB
  • theta_det
    • Demod[TOD * exp(-i(4HWP_angle-2(theta_det+tauhwp_speed2.*pi)))

  • of detectors in pb2a_wiregrid_ver10.db: 2499 bolos

ver10.2

Just change quality cut in modifyDB.py

• Change final wiregrid DB: pb2a_wiregrid.db
• Modify modifyDB.py
  • Remove tau>0 cut

  • of detectors in pb2a_wiregrid_ver10.2.db: 2833 bolos

• Modify check_absolute.py and check_absolute_nocorr.py
  • remove stimulator quality cut: tau>0.

run scripts

• (./plot.sh: make plot of TODs)

• ./test_pipeline.sh

  • From demod to fit circle
  • run each stage of analysis by using loadbolo_pipeline.py

• ./analysis_misc.sh

  • For test or check run
  • From demod to fit circle

• python3 run_batch.py

  • Run demod & fit for many detectors by using batch job

• ./analysis_DB.sh

  • Run DB modification or check after the fit circle

Scripts for each analysis steps

• (plot.py: simple TOD plotter)

• grid_rotation_analysis.py

  • get TOD & demod each wire angle data
  • Using scripts:
    • OneAngleData.py

• fitDemodResult.py

  • fit circle data of demod datas
  • Using scripts:
    • minuitfit.py
    • LMfit.py (not used now)
    • plotFitResult.py

DB modification

Summary of DB modification steps

1. mergeDB.py
2. compare_DB_for_labelcorrection.py
3. check_absolute.py
4. modifyDB.py

merge & modify DB

• mergeDB.py:
  • merge output of run_batch.sh (fitDemodResult.py) & modify SQL database
    • Tau (timing constant) calibration
    • Add hardware map DB
  • input : output_verX/db/"wafer name"/*.db
  • output:
    • output_verX/db/all.db : sqlite DB without modification
    • output_verX/db/all_mod.db : sqlite DB with modification
    • output_verX/db/all_pandas.pkl : pandas DB with modification
    • output_verX/db/all_pandas.db : sqlite DB converted from pandas DB with modification

Compare DB

• compare_db.py

  • merge two sqlite DBs with 'readout_name' column & compare them
  • This script is used in the following scripts:
    • compare_DB_for_labelcorrection.py
    • compare_and_make_DB_for_labelcorrection.py
  • input :
    • original hardware map : data/pb2a-20210205/pb2a_mapping.db
    • kyohei's corrected hardware map: data/ykyohei/mapping/pb2a_mapping_postv2.db
    • my wiregrid DB : output_verX/db/all_pandas.db
  • output :
    • aho.png : 2D plot between varname1 v.s. varname2
    • aho.csv : converted from pandas of bolometers with varname!=varname2

• compare_DB_for_labelcorrection.py

  • compare wiregrid DB and Kyohei's DB to find mislabel bolometers.
  • make a new sqlite DB with corrected pol_angle,pixel_type,bolo_type on found mislabeled bolometers by wiregrid DB
    • base DB : my wire grid DB
  • add Kyohei's det_offset_x/y datas
  • input :
    • my wiregrid DB : output_verX/db/all_pandas.db
    • kyohei's corrected hardware map: data/ykyohei/mapping/pb2a_mapping_postv2.db or a new pb2a_focalplane DB in pb2a_mapping.db
  • output:
    • output_verX/db/all_pandas_correct_label.db/pkl

• compare_and_make_DB_for_labelcorrection.py (Old)

  • compare wiregrid DB and kyohei's DB to find mislabel bolometers.
  • make a new sqlite DB for SA official DB with corrected pol_angle,pixel_type,pixel_handedness,bolo_type on found mislabeled bolometers by wiregrid DB
    • base DB : original hardware map DB
  • If tau is Nan or 0 (stimulator data is not good), theta_det is set to nan.
  • input :
    • original hardware map : data/pb2a-20210205/pb2a_mapping.db
    • kyohei's corrected hardware map: data/ykyohei/mapping/pb2a_mapping_postv2.db
    • my wiregrid DB : output_verX/db/all_pandas.db
  • output:
    • output_verX/db/pb2a_mapping_corrected_label_v2.db

Create wiregrid DB for public from all_pandas_correct_label.db

• modifyDB.py
  • Make a new sqlite DB for SA official DB with corrected pol_angle,pixel_type,pixel_handedness,bolo_type by wiregrid DB
  • Modifications:
    • Keep the following columns:
      • bolo_name
      • pixel_name
      • wafer_number
      • band
      • pixel_name
      • pixel_number
      • pixel_type
      • pixel_handedness
      • bolo_type
      • mislabel
      • pol_angle
      • theta_det
      • theta_det_err
    • theta_det is shifted by 90 deg. (0~pi) to match the definition as in pol_angle.
    • Drop bolometers:
      • if tau is Nan or 0 (stimulator data is not good)
      • if pol_angle is Nan
      • if theta_det_err is >=0.5deg
    • If bolo_name is Nan and labels are corrected, bolo_name is corrected.
      • There were corrected bolo_name labels. The band name in bolo_name was wrong. (534 bolos in ver10)
    • Rename theta_det_err_total --> theta_det_err

Plotting scripts

Check TOD

• checkspeed.py
  • print average frequency [Hz] of HWP
  • input : output_verX/pkl/PB20.13.13/PB20.13.13_Comb01Ch01/*.pkl
    • TOD for only one bolometer
  • output: No (only printing)

Check DB

• printDB.py :

  • Print contents of pandas pickle file
  • input : output_verX/db/all_pandas.pkl : pandas DB with modification
  • output: No (only printing)

• checkDB.py

  • Make plots of pandas pickle file
  • input : output_verX/db/all_pandas.pkl : pandas DB with modification
  • output: output_verX/check_db/*.pdf : figures

Check diff between wiregrid measured angle and design detector angle

• check_absolute.py

  • make angle plots of DB after wiregrid corrected labels
    • 2D plot: theta_det_angle (wiregrid measured angle) v.s. pol_angle (design value) for good data (tau!=nan, theta_det_err<0.5deg, pol_angle!=nan)
    • 2D plot: theta_det_angle (wiregrid measured angle) v.s. pol_angle (design value) for correct labels
    • 1D plot: diff. between measured angle(theta_det_angle) and design angle(pol_angle)
  • make focal plane plot
  • input : output_verX/db/all_pandas_correct_label.db
  • output: output_verX/check_absolute/*.png

• check_absolute_labelcorrecteddb.py

  • make angle plots of DB with wiregrid corrected labels
  • input :
    • output_verX/db/all_pandas.db # No correction data
    • output_verX/db/pb2a_mapping_corrected_label_v2.db.db # Corrected data
  • output: out_check_absolute/check_absolute_labelcorrectedDB.png

• check_absolute_nocorr.py

  • make angle plots of DB before wiregrid label correction
  • input :
    • output_verX/db/all_pandas.db
  • output: output_verX/check_absolute_nocorr/*.png

Check diff between 2 DBs for the same readout_name

• diff_db.py
  • make diff or same histograms for each columns for the same readout_name
  • input : 2 DBs (any DB is OK if it has the readout_name column.)
  • output: output_verX/db/diff_db.png, diff_db2.png, diff_db.csv (different bolometer info)

Others

• makehist.py

  • make histogram of wiregrid measured angels for each bolometer groups
  • input : output_verX/db/all_pandas.pkl
  • output: output_verX/summary/*.png

• check_jobs.py

  • check if all the jobs run by bsub is finished successfully.
    • check if error log is empty.
    • check if 'error' word exists in the output log of fitDemodResult.py.
    • check if 'error' word exists in the output log of grid_rotation_analysis_.py.
  • output information : 0. # of all failed checks

    1. of failed checks except the error of "Function minimum is not valid." (Fmin error)

       • This number should be 0.
       • "Function..." error is caused by failure of fit. It can be occurred by bad data and can be ignored.

    2. of memory limit error (TERM_MEMLIMIT)

  • results :
    • output_ver5 (only fitDemodResult.py) 0. # of all failed checks = 88

      1. of failed checks except Fmin error = 1

         • bsub log error in ./output_ver5/txt/PB20.13.12/bsub/bsub_PB20.13.12_Comb15Ch26.out
         • memory limit error (TERM_MEMLIMIT)
         • It has been rerun successfully.

      2. of memory limit error = 1

    • output_ver6 (all analysis from g3 files)

      Summary for check_job for output_ver6

      bsub error log (5461) : ./output_ver6/txt/PB20.13.12/bsub/bsub_PB20.13.12_Comb11Ch17.err... bsub log (5461) : ./output_ver6/txt/PB20.13.12/bsub/bsub_PB20.13.12_Comb09Ch22.out... fitDemodResult.py log (5461) : ./output_ver6/txt/PB20.13.12/fit/PB20.13.12_Comb03Ch20.out... grid_rotation_analysis.py log (5461) : ./output_ver6/txt/PB20.13.12/gridana/PB20.13.12_Comb03Ch20.out... There are errors in some jobs for output_ver6

      of all failed checks = 203

      of failed checks except the error of "Function minimum is not valid." = 0

      of memory limit error = 0

      #################################################
    • output_ver8 (analysis from TOD obtained in output_ver6)

      Summary for check_job for output_ver8

      bsub error log (5461) : ./output_ver8/txt/PB20.13.12/bsub/bsub_PB20.13.12_Comb11Ch17.err... bsub log (5461) : ./output_ver8/txt/PB20.13.12/bsub/bsub_PB20.13.12_Comb09Ch22.out... fitDemodResult.py log (5461) : ./output_ver8/txt/PB20.13.12/fit/PB20.13.12_Comb03Ch20.out... grid_rotation_analysis.py log (5461) : ./output_ver8/txt/PB20.13.12/gridana/PB20.13.12_Comb03Ch20.out... There are errors in some jobs for output_ver8

      of all failed checks = 186

      of failed checks except the error of "Function minimum is not valid." = 0

      of "Function minimum is not valid." error = 186

      of memory limit error = 0

      #################################################

• hwpss/...

  • Here is scripts for label correction and modification of HWPSS DB measured by Masaaki
  • How to use is wrote in ./analysis_DB.sh