VLA Holography Tutorial

Important External Information

• xarray Official Documentation (docs).

• Dask Official Documentation (docs).

• zarr Official Documentation (docs)

import os

try:
    import astrohack

    print('AstroHACK version', astrohack.__version__, 'already installed.')
except ImportError as e:
    print(e)
    print('Installing AstroHACK')

    os.system("python -m pip install --upgrade astrohack")

    import astrohack
    print('astrohack version', astrohack.__version__, ' installed.')

Download Tutorial Data

import astrohack
import graphviper

graphviper.utils.data.download('ea25_cal_small_after_fixed.split.ms', folder='data')

Holography Data File API

As part of the astroHACK API a set of functions to allow users to easily open on disk holography files has been provided. Each function takes an astroHACK holography file name as an argument and returns an object related to the given file type, ie. holog, image, panel, point. Each object allows the user to access data via dictionary keys with values consisting of the relevant holography dataset. Each object also provides a summary() helper function to list available keys for each file. An example call for each file type is show below and the API documentation for all data-io functions can be found here.

from astrohack.dio import open_holog
from astrohack.dio import open_image
from astrohack.dio import open_panel
from astrohack.dio import open_pointing

holog_data = open_holog(file='./data/ea25_cal_small_spw1_4_60_ea04_after.holog.zarr')
image_data = open_image(file='./data/ea25_cal_small_spw1_4_60_ea04_after.image.zarr')
panel_data = open_panel(file='./data/ea25_cal_small_spw1_4_60_ea04_after.panel.zarr')
pointing_data = open_pointing(file='./data/ea25_cal_small_spw1_4_60_ea04_after.point.zarr')

Setup Dask Local Cluster

The local Dask client handles scheduling and worker managment for the parallelization. The user has the option of choosing the number of cores and memory allocations for each worker howerver, we recommend a minimum of 8Gb per core with standard settings.

A significant amount of information related to the client and scheduling can be found using the Dask Dashboard. This is a built in dashboard native to Dask and allows the user to monitor the workers during processing. This is especially useful for profilling. For those that are interested in working soley within Jupyterlab a dashboard extension is availabe for Jupyterlab.

from graphviper.dask.client import local_client
client = local_client(cores=2, memory_limit='8GB')

[2024-02-14 12:52:40,103]     INFO  graphviper:  Checking parameter values for client.local_client
[2024-02-14 12:52:40,104]     INFO      logger:  /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10//site-packages/
[2024-02-14 12:52:40,104]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:52:40,113]  WARNING  graphviper:  It is recommended that the local cache directory be set using the dask_local_dir parameter.
[2024-02-14 12:52:41,487]     INFO  graphviper:  Created client <MenrvaClient: 'tcp://127.0.0.1:33735' processes=2 threads=2, memory=14.90 GiB>

Extract Holog

The extraction and restructuring of the holography data is done using the extract_holog function. This function is similar in function to the UVHOL task in AIPS. The holography data that is extracted can be set using the compound dictionary holog_obs_description: mapping, scan, and antenna id. A detailed description of the structure of the holog_obs_description dictionary can be found in the documentation here. The extract_holog can automatically generate the holog_obs_description by inspecting the pointing table.

Inline information on the input parameters can also be gotten using help(extract_holog) in the cell.

from astrohack.extract_pointing import extract_pointing
from astrohack.extract_holog import extract_holog

extract_pointing(
      ms_name='data/ea25_cal_small_after_fixed.split.ms',
      point_name='data/ea25_cal_small_after_fixed.split.point.zarr',
      parallel=True,
      overwrite=True
)

holog_mds = extract_holog(
    ms_name='data/ea25_cal_small_after_fixed.split.ms',
    point_name='data/ea25_cal_small_after_fixed.split.point.zarr',
    data_column='CORRECTED_DATA',
    parallel=True,
    overwrite=True
)

[2024-02-14 12:52:41,493]     INFO  graphviper:  Checking parameter values for extract_pointing.extract_pointing
[2024-02-14 12:52:41,493]     INFO   astrohack:  /export/home/ajax/jhoskins/Development/astrohack/
[2024-02-14 12:52:41,494]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:52:41,495]     INFO   astrohack:  Failed to find module in PARAMETER_CONFIG_PATH ... attempting to check common directories ...
[2024-02-14 12:52:41,495]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/Development/astrohack/ for configuration file, please wait ...
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/ANTENNA: 8 columns, 3 rows
[2024-02-14 12:52:47,678]     INFO  graphviper:  Finished processing
[2024-02-14 12:52:47,712]     INFO  graphviper:  Checking parameter values for extract_holog.extract_holog
[2024-02-14 12:52:47,713]     INFO   astrohack:  /export/home/ajax/jhoskins/Development/astrohack/
[2024-02-14 12:52:47,713]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:52:47,714]     INFO   astrohack:  Failed to find module in PARAMETER_CONFIG_PATH ... attempting to check common directories ...
[2024-02-14 12:52:47,714]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/Development/astrohack/ for configuration file, please wait ...
[2024-02-14 12:52:47,760]     INFO  graphviper:  Creating output file name: data/ea25_cal_small_after_fixed.split.holog.zarr
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/DATA_DESCRIPTION: 3 columns, 2 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/ANTENNA: 8 columns, 3 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms: 23 columns, 36580 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/STATE: 7 columns, 44 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/SPECTRAL_WINDOW: 20 columns, 2 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/POLARIZATION: 4 columns, 1 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/OBSERVATION: 9 columns, 1 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/HISTORY: 9 columns, 157 rows
[2024-02-14 12:52:47,804]     INFO  graphviper:  Processing ddi: 0, scans: [8 ... 57]
[2024-02-14 12:52:47,805]     INFO  graphviper:  Processing ddi: 1, scans: [8 ... 57]
Successful readonly open of default-locked table data/ea25_cal_small_after_fixed.split.ms/ANTENNA: 8 columns, 3 rows
Successful readonly open of default-locked table data/ea25_cal_small_after_fixed.split.ms/OBSERVATION: 9 columns, 1 rows
[2024-02-14 12:53:01,368]     INFO    worker_0:  Writing holog file to data/ea25_cal_small_after_fixed.split.holog.zarr
[2024-02-14 12:53:01,542]     INFO    worker_0:  Writing holog file to data/ea25_cal_small_after_fixed.split.holog.zarr
Successful readonly open of default-locked table data/ea25_cal_small_after_fixed.split.ms/ANTENNA: 8 columns, 3 rows
Successful readonly open of default-locked table data/ea25_cal_small_after_fixed.split.ms/OBSERVATION: 9 columns, 1 rows
[2024-02-14 12:53:01,648]     INFO    worker_1:  Writing holog file to data/ea25_cal_small_after_fixed.split.holog.zarr
[2024-02-14 12:53:01,663]     INFO    worker_0:  Finished extracting holography chunk for ddi: 1 holog_map_key: map_0
[2024-02-14 12:53:01,854]     INFO    worker_1:  Writing holog file to data/ea25_cal_small_after_fixed.split.holog.zarr
[2024-02-14 12:53:02,079]     INFO    worker_1:  Finished extracting holography chunk for ddi: 0 holog_map_key: map_0
[2024-02-14 12:53:02,094]     INFO  graphviper:  Finished processing
[2024-02-14 12:53:02,136]  WARNING  graphviper:  Cell size not consistent: [0.000719484970141879, 0.000719484970141879, 0.0006386556122807017, 0.0006386556122807017]
[2024-02-14 12:53:02,137]  WARNING  graphviper:  Calculating suggested cell size ...
[2024-02-14 12:53:02,138]  WARNING  graphviper:  [0.00071948 0.00071948 0.00063866 0.00063866]
[2024-02-14 12:53:02,141]     INFO  graphviper:  The suggested cell size is calculated to be: 0.0006790702912112905
[2024-02-14 12:53:02,144]  WARNING  graphviper:  Number of pixels not consistent: [729, 729, 961, 961]
[2024-02-14 12:53:02,147]  WARNING  graphviper:  Calculating suggested number of pixels ...
[2024-02-14 12:53:02,148]  WARNING  graphviper:  [729 729 961 961]
[2024-02-14 12:53:02,150]     INFO  graphviper:  The suggested number of pixels is calculated to be: 845 (grid: 29 x 29)

Two files are created by extract_holog: The extracted pointing information in the form of <point_name>.point.zarr and the extracted holography data as <point_name>.holog.zarr. In addition, a holography data object is returned. This is the same holography data object returned by the hologrphy data API above. The holog_mds object is a python dict containing the extracted holography data found in .holog.zarr but with extended functionality such as providing a summary of the run infomation in table form. Below for each DDI we can see the available scan and antenna information.

point_name.point.zarr: The pointing zarr file contains position and pointing information extracted from the pointing table of the input measurement set. In addition, the antenna and mapping scan information is listed for each antenna. The pointing object is structured as a simple dictionary with key:value sets with the key being the antenna id and the value being the pointing dataset.

point_mds =
{
   ant_0: point_ds,
            ⋮
   ant_n: point_ds
}

holog_name.holog.zarr: The holog zarr file contains ungridded data extracted from the pointing and main tables in the measurement set. The holog file includes the directional, visibility and weight information recorded on a shared time axis; the sampling is done because the native sample rates between the pointing and main tables are not the same. In addition, the meta data such as sampled parallactic data (beginning, middle and end of scan) and l(m) extent is recorded in the file attributes. The holog file structure is a compound dictionary keyed according to ddi -> map -> ant with values consisting of the holog dataset.

holog_mds =
{
   ddi_0:{
          map_0:{
                 ant_0: holog_ds,
                          ⋮
                 ant_n: holog_ds
                },
              ⋮
          map_p: …
         },
       ⋮
   ddi_m: …
}

An example of the holog dataset object is show below.

holog_mds['ddi_0']['map_0']['ant_ea25']

<xarray.Dataset>
Dimensions:                    (time: 8914, lm: 2, chan: 64, pol: 4)
Coordinates:
  * chan                       (chan) float64 1.41e+10 1.411e+10 ... 1.423e+10
  * pol                        (pol) <U2 'RR' 'RL' 'LR' 'LL'
  * time                       (time) float64 5.17e+09 5.17e+09 ... 5.17e+09
Dimensions without coordinates: lm
Data variables:
    DIRECTIONAL_COSINES        (time, lm) float64 dask.array<chunksize=(8914, 2), meta=np.ndarray>
    IDEAL_DIRECTIONAL_COSINES  (time, lm) float64 dask.array<chunksize=(8914, 2), meta=np.ndarray>
    VIS                        (time, chan, pol) complex64 dask.array<chunksize=(2229, 16, 2), meta=np.ndarray>
    WEIGHT                     (time, chan, pol) float64 dask.array<chunksize=(2229, 16, 2), meta=np.ndarray>
Attributes:
    antenna_name:             ea25
    ddi:                      0
    grid_params:              {'cell_size': 0.000719484970141879, 'n_pix': 729}
    holog_map_key:            map_0
    l_max:                    0.009675024871632517
    l_min:                    -0.009674930411087166
    m_max:                    0.009698666161358812
    m_min:                    -0.00975229251000802
    parallactic_samples:      [5.308157433326323, 5.357028871639436, 5.502977...
    telescope_name:           EVLA
    time_smoothing_interval:  1.0

xarray.Dataset

• Dimensions:
  • time: 8914
  • lm: 2
  • chan: 64
  • pol: 4
• Coordinates: (3)
  • chan
    (chan)
    float64
    1.41e+10 1.411e+10 ... 1.423e+10

    array([1.4104e+10, 1.4106e+10, 1.4108e+10, 1.4110e+10, 1.4112e+10, 1.4114e+10,
           1.4116e+10, 1.4118e+10, 1.4120e+10, 1.4122e+10, 1.4124e+10, 1.4126e+10,
           1.4128e+10, 1.4130e+10, 1.4132e+10, 1.4134e+10, 1.4136e+10, 1.4138e+10,
           1.4140e+10, 1.4142e+10, 1.4144e+10, 1.4146e+10, 1.4148e+10, 1.4150e+10,
           1.4152e+10, 1.4154e+10, 1.4156e+10, 1.4158e+10, 1.4160e+10, 1.4162e+10,
           1.4164e+10, 1.4166e+10, 1.4168e+10, 1.4170e+10, 1.4172e+10, 1.4174e+10,
           1.4176e+10, 1.4178e+10, 1.4180e+10, 1.4182e+10, 1.4184e+10, 1.4186e+10,
           1.4188e+10, 1.4190e+10, 1.4192e+10, 1.4194e+10, 1.4196e+10, 1.4198e+10,
           1.4200e+10, 1.4202e+10, 1.4204e+10, 1.4206e+10, 1.4208e+10, 1.4210e+10,
           1.4212e+10, 1.4214e+10, 1.4216e+10, 1.4218e+10, 1.4220e+10, 1.4222e+10,
           1.4224e+10, 1.4226e+10, 1.4228e+10, 1.4230e+10])

  • pol
    (pol)
    <U2
    'RR' 'RL' 'LR' 'LL'

    array(['RR', 'RL', 'LR', 'LL'], dtype='<U2')

  • time
    (time)
    float64
    5.17e+09 5.17e+09 ... 5.17e+09

    array([5.170354e+09, 5.170354e+09, 5.170354e+09, ..., 5.170365e+09,
           5.170365e+09, 5.170365e+09])

• Data variables: (4)
  • DIRECTIONAL_COSINES
    (time, lm)
    float64
    dask.array<chunksize=(8914, 2), meta=np.ndarray>

    Array Chunk Bytes 139.28 kiB 139.28 kiB Shape (8914, 2) (8914, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • IDEAL_DIRECTIONAL_COSINES
    (time, lm)
    float64
    dask.array<chunksize=(8914, 2), meta=np.ndarray>

    Array Chunk Bytes 139.28 kiB 139.28 kiB Shape (8914, 2) (8914, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • VIS
    (time, chan, pol)
    complex64
    dask.array<chunksize=(2229, 16, 2), meta=np.ndarray>

    Array Chunk Bytes 17.41 MiB 557.25 kiB Shape (8914, 64, 4) (2229, 16, 2) Dask graph 32 chunks in 2 graph layers Data type complex64 numpy.ndarray

  • WEIGHT
    (time, chan, pol)
    float64
    dask.array<chunksize=(2229, 16, 2), meta=np.ndarray>

    Array Chunk Bytes 17.41 MiB 557.25 kiB Shape (8914, 64, 4) (2229, 16, 2) Dask graph 32 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • chan
    PandasIndex

    PandasIndex(Index([14104000000.0, 14106000000.0, 14108000000.0, 14110000000.0,
           14112000000.0, 14114000000.0, 14116000000.0, 14118000000.0,
           14120000000.0, 14122000000.0, 14124000000.0, 14126000000.0,
           14128000000.0, 14130000000.0, 14132000000.0, 14134000000.0,
           14136000000.0, 14138000000.0, 14140000000.0, 14142000000.0,
           14144000000.0, 14146000000.0, 14148000000.0, 14150000000.0,
           14152000000.0, 14154000000.0, 14156000000.0, 14158000000.0,
           14160000000.0, 14162000000.0, 14164000000.0, 14166000000.0,
           14168000000.0, 14170000000.0, 14172000000.0, 14174000000.0,
           14176000000.0, 14178000000.0, 14180000000.0, 14182000000.0,
           14184000000.0, 14186000000.0, 14188000000.0, 14190000000.0,
           14192000000.0, 14194000000.0, 14196000000.0, 14198000000.0,
           14200000000.0, 14202000000.0, 14204000000.0, 14206000000.0,
           14208000000.0, 14210000000.0, 14212000000.0, 14214000000.0,
           14216000000.0, 14218000000.0, 14220000000.0, 14222000000.0,
           14224000000.0, 14226000000.0, 14228000000.0, 14230000000.0],
          dtype='float64', name='chan'))

  • pol
    PandasIndex

    PandasIndex(Index(['RR', 'RL', 'LR', 'LL'], dtype='object', name='pol'))

  • time
    PandasIndex

    PandasIndex(Index([5170354120.000001, 5170354121.000001, 5170354122.000001,
           5170354123.000001, 5170354124.000001, 5170354125.000001,
           5170354126.000001, 5170354127.000001, 5170354128.000001,
           5170354129.000001,
           ...
           5170364896.000001, 5170364897.000001, 5170364900.000001,
           5170364901.000001, 5170364902.000001, 5170364903.000001,
           5170364904.000001, 5170364905.000001, 5170364906.000001,
           5170364907.000001],
          dtype='float64', name='time', length=8914))

• Attributes: (11)

  antenna_name :

  ea25

  ddi :

  0

  grid_params :

  {'cell_size': 0.000719484970141879, 'n_pix': 729}

  holog_map_key :

  map_0

  l_max :

  0.009675024871632517

  l_min :

  -0.009674930411087166

  m_max :

  0.009698666161358812

  m_min :

  -0.00975229251000802

  parallactic_samples :

  [5.308157433326323, 5.357028871639436, 5.502977933572945]

  telescope_name :

  EVLA

  time_smoothing_interval :

  1.0

In this case, there is only one selection in the holography file as seen in the summary. Using the available keys we can see an overview of the Dask dataset structure. In addition, the numpy arrays for the data are accessed by calling values on a given dataset variable. For instance accessing the data for the DIRECTIONAL_COSINES below would be simply

>> holog_mds['ddi_0']['map_0']['ant_ea25'].DIRECTIONAL_COSINES.values
>> array([[-0.00433549, -0.0027946 ],
       [-0.00870191, -0.00682571],
       [-0.00965634, -0.00908509],
       ...,
       [ 0.00966373,  0.00957556],
       [ 0.00966267,  0.00957601],
       [ 0.00965895,  0.00956941]])

>> holog_mds['ddi_0']['map_0']['ant_ea25'].DIRECTIONAL_COSINES.values.shape
>> (9145, 2)

where the dimension are given in the mds output for each data variable (in this case (time, lm)). A more in-depth overview of how to interact with Dask dataset can be found here.

A summary of the available key values can be obtained using the summary convenience function.

holog_mds.summary()

####################################################################################################
###                                         Summary for:                                         ###
###                       data/ea25_cal_small_after_fixed.split.holog.zarr                       ###
####################################################################################################

Full documentation for AstrohackHologFile objects' API at:
https://astrohack.readthedocs.io/en/stable/_api/autoapi/astrohack/mds/index.html#astrohack.mds.AstrohackHologFile

Input Parameters:
+---------------------------+--------------------------------------------------+
| Parameter                 | Value                                            |
+---------------------------+--------------------------------------------------+
| ms_name                   | data/ea25_cal_small_after_fixed.split.ms         |
| point_name                | data/ea25_cal_small_after_fixed.split.point.zarr |
| holog_name                | data/ea25_cal_small_after_fixed.split.holog.zarr |
| holog_obs_dict            | None                                             |
| ddi                       | all                                              |
| baseline_average_distance | all                                              |
| baseline_average_nearest  | all                                              |
| data_column               | CORRECTED_DATA                                   |
| time_smoothing_interval   | None                                             |
| parallel                  | True                                             |
| overwrite                 | True                                             |
| version                   | 0.4.5                                            |
| origin                    | extract_holog                                    |
+---------------------------+--------------------------------------------------+

Contents:
+-------+-------+--------------------------+
| DDI   | Map   | Antenna                  |
+-------+-------+--------------------------+
| ddi_0 | map_0 | ['ant_ea25', 'ant_ea06'] |
| ddi_1 | map_0 | ['ant_ea25', 'ant_ea06'] |
+-------+-------+--------------------------+

Available methods:
+----------------------+-----------------------------------------------------------------------+
| Methods              | Description                                                           |
+----------------------+-----------------------------------------------------------------------+
| summary              | Prints summary of the AstrohackHologFile object, with available data, |
|                      | attributes and available methods                                      |
| select               | Select data on the basis of ddi, scan, ant. This is a convenience     |
|                      | function.                                                             |
| plot_diagnostics     | Plot diagnostic calibration plots from the holography data file.      |
| plot_lm_sky_coverage | Plot directional cosine coverage.                                     |
+----------------------+-----------------------------------------------------------------------+

Estimated Memory Requirements

A new functionality, currently being refined, is a function to estimate the amount of memory per core max that would be required to process a given file. The estimation is given as the suggested memory per core need to not spilling over into swap memory. If the user has already computed the holog_obs_dict, it can be added as a parameter to speed up the estitmate as this is a serial function currently.

In the resulting table the following definitions are important:

• Available memory: The available memory on the system currently, ie. the total not currently in use.

• Total memory: The total system memory

• Suggested memory per core: Memory allocation per core estimated to not spill ove rinto swap memory.

  Reference: https://psutil.readthedocs.io/en/latest/#psutil.virtual_memory

from astrohack.extract_holog import model_memory_usage
from astrohack.extract_holog import generate_holog_obs_dict

holog_obs_dict = generate_holog_obs_dict(
    ms_name="data/ea25_cal_small_after_fixed.split.ms",
    point_name="data/ea25_cal_small_after_fixed.split.point.zarr",
)

model_memory_usage(
    ms_name="data/ea25_cal_small_after_fixed.split.ms",
    holog_obs_dict=holog_obs_dict
)

Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms/ANTENNA: 8 columns, 3 rows
Successful readonly open of usernoread-locked table data/ea25_cal_small_after_fixed.split.ms: 23 columns, 36580 rows
[2024-02-14 12:53:02,344]     INFO  graphviper:  Writing distance matrix to /export/home/ajax/jhoskins/Development/astrohack/docs/.baseline_distance_matrix.csv ...

/export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator ElasticNet from version 1.3.2 when using version 1.4.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(

                                      System Info                                       
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ N-cores ┃ Available memory (MB) ┃ Total memory (MB) ┃ Suggested memory per core (MB) ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│       4 │ 3017                  │ 15881             │                          11911 │
└─────────┴───────────────────────┴───────────────────┴────────────────────────────────┘
    Available memory: represents the system memory available without going into swap    

11911

Holog

The holog function processes the holography data and produces a holog image file on disk with the suffix, .image.zarr. This function is a direct replacement for the task HOLOG in AIPS. It is required that the user provide the grid_size and cell_size when processing holography data. The grid_size defines the number of l x m points used to when doing the gridding. The cell_size defines the value in arseconds of each grid spacing. More in-depth parameter information can be found in readthedocs here.

Inline information on the input paramters can also be gotten using help(holog) in the cell.

import numpy as np

from astrohack import holog

cell_size = np.array([-0.0006442, 0.0006442]) # arcseconds
grid_size = np.array([31, 31])                # pixels

image_mds = holog(
    holog_name='data/ea25_cal_small_after_fixed.split.holog.zarr',
    overwrite=True,
    phase_fit=True,
    apply_mask=True,
    to_stokes=True,
    parallel=True
)

[2024-02-14 12:53:09,721]     INFO  graphviper:  Checking parameter values for holog.holog
[2024-02-14 12:53:09,722]     INFO   astrohack:  /export/home/ajax/jhoskins/Development/astrohack/
[2024-02-14 12:53:09,723]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:53:09,725]     INFO   astrohack:  Failed to find module in PARAMETER_CONFIG_PATH ... attempting to check common directories ...
[2024-02-14 12:53:09,726]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/Development/astrohack/ for configuration file, please wait ...
[2024-02-14 12:53:09,772]     INFO  graphviper:  Creating output file name: data/ea25_cal_small_after_fixed.split.image.zarr
[2024-02-14 12:53:09,774]     INFO  graphviper:  Cell size: [-0.00067907  0.00067907], Grid size [29 29]

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:68: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_chan = ant_data_dict[ddi][map0].dims["chan"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:69: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_pol = ant_data_dict[ddi][map0].dims["pol"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:68: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_chan = ant_data_dict[ddi][map0].dims["chan"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:69: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_pol = ant_data_dict[ddi][map0].dims["pol"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:119: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  time_centroid_index = ant_data_dict[ddi][holog_map].dims["time"] // 2
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:119: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  time_centroid_index = ant_data_dict[ddi][holog_map].dims["time"] // 2

[2024-02-14 12:53:10,180]     INFO    worker_1:  Calculating aperture pattern ...
[2024-02-14 12:53:10,180]     INFO    worker_1:  Calculating aperture illumination pattern ...
[2024-02-14 12:53:10,181]     INFO    worker_0:  Calculating aperture pattern ...
[2024-02-14 12:53:10,181]     INFO    worker_0:  Calculating aperture illumination pattern ...
[2024-02-14 12:53:10,502]     INFO    worker_0:  Applying phase correction
[2024-02-14 12:53:10,554]     INFO    worker_1:  Applying phase correction

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:68: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_chan = ant_data_dict[ddi][map0].dims["chan"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:69: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_pol = ant_data_dict[ddi][map0].dims["pol"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:68: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_chan = ant_data_dict[ddi][map0].dims["chan"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:69: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  n_pol = ant_data_dict[ddi][map0].dims["pol"]
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:119: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  time_centroid_index = ant_data_dict[ddi][holog_map].dims["time"] // 2

[2024-02-14 12:53:15,146]     INFO    worker_0:  Calculating aperture pattern ...
[2024-02-14 12:53:15,146]     INFO    worker_0:  Calculating aperture illumination pattern ...

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_holog.py:119: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  time_centroid_index = ant_data_dict[ddi][holog_map].dims["time"] // 2

[2024-02-14 12:53:15,411]     INFO    worker_1:  Calculating aperture pattern ...
[2024-02-14 12:53:15,411]     INFO    worker_1:  Calculating aperture illumination pattern ...
[2024-02-14 12:53:15,462]     INFO    worker_0:  Applying phase correction
[2024-02-14 12:53:15,609]     INFO    worker_1:  Applying phase correction
[2024-02-14 12:53:15,873]     INFO  graphviper:  Finished processing

image_name.image.zarr: The image zarr file contains gridded image data the beam, extracted aperture and the amplitude and phase components. It also contains all of the relevant coordinate information. The image file structure is a compound dictionary keys according to ant -> ddi with the dictionary values consisting of the image dataset.

image_mds =
{
   ant_0:{
          ddi_0: image_ds,
                 ⋮
          ddi_m: image_ds
         },
       ⋮
   ant_n: …
}

An example of the image dataset object is show below.

image_mds['ant_ea25']['ddi_0']

<xarray.Dataset>
Dimensions:          (time: 1, chan: 1, pol: 4, u_prime: 552, v_prime: 552,
                      u: 627, v: 627, l: 29, m: 29, ddi: 1)
Coordinates:
  * chan             (chan) float64 1.417e+10
  * ddi              (ddi) <U5 'ddi_0'
  * l                (l) float64 0.009507 0.008828 ... -0.008828 -0.009507
  * m                (m) float64 -0.009507 -0.008828 ... 0.008828 0.009507
  * pol              (pol) <U1 'I' 'Q' 'U' 'V'
  * u                (u) float64 735.1 732.8 730.4 ... -730.4 -732.8 -735.1
  * u_prime          (u_prime) float64 648.2 645.9 643.5 ... -643.5 -645.9
  * v                (v) float64 -735.1 -732.8 -730.4 ... 730.4 732.8 735.1
  * v_prime          (v_prime) float64 -648.2 -645.9 -643.5 ... 643.5 645.9
Dimensions without coordinates: time
Data variables:
    AMPLITUDE        (time, chan, pol, u_prime, v_prime) float64 dask.array<chunksize=(1, 1, 1, 276, 276), meta=np.ndarray>
    APERTURE         (time, chan, pol, u, v) complex128 dask.array<chunksize=(1, 1, 1, 157, 314), meta=np.ndarray>
    BEAM             (time, chan, pol, l, m) complex128 dask.array<chunksize=(1, 1, 4, 29, 29), meta=np.ndarray>
    CORRECTED_PHASE  (time, chan, pol, u_prime, v_prime) float64 dask.array<chunksize=(1, 1, 1, 276, 276), meta=np.ndarray>
Attributes:
    ant_id:               ant_ea25
    ant_name:             ea25
    aperture_resolution:  [1.413428251773375, 1.413428251773375]
    ddi:                  ddi_0
    telescope_name:       EVLA
    time_centroid:        5170359434.000001

xarray.Dataset

• Dimensions:
  • time: 1
  • chan: 1
  • pol: 4
  • u_prime: 552
  • v_prime: 552
  • u: 627
  • v: 627
  • l: 29
  • m: 29
  • ddi: 1
• Coordinates: (9)
  • chan
    (chan)
    float64
    1.417e+10

    array([1.4167e+10])

  • ddi
    (ddi)
    <U5
    'ddi_0'

    array(['ddi_0'], dtype='<U5')

  • l
    (l)
    float64
    0.009507 0.008828 ... -0.009507

    array([ 0.009507,  0.008828,  0.008149,  0.00747 ,  0.006791,  0.006112,
            0.005433,  0.004753,  0.004074,  0.003395,  0.002716,  0.002037,
            0.001358,  0.000679, -0.      , -0.000679, -0.001358, -0.002037,
           -0.002716, -0.003395, -0.004074, -0.004753, -0.005433, -0.006112,
           -0.006791, -0.00747 , -0.008149, -0.008828, -0.009507])

  • m
    (m)
    float64
    -0.009507 -0.008828 ... 0.009507

    array([-0.009507, -0.008828, -0.008149, -0.00747 , -0.006791, -0.006112,
           -0.005433, -0.004753, -0.004074, -0.003395, -0.002716, -0.002037,
           -0.001358, -0.000679,  0.      ,  0.000679,  0.001358,  0.002037,
            0.002716,  0.003395,  0.004074,  0.004753,  0.005433,  0.006112,
            0.006791,  0.00747 ,  0.008149,  0.008828,  0.009507])

  • pol
    (pol)
    <U1
    'I' 'Q' 'U' 'V'

    array(['I', 'Q', 'U', 'V'], dtype='<U1')

  • u
    (u)
    float64
    735.1 732.8 730.4 ... -732.8 -735.1

    array([ 735.126479,  732.777832,  730.429185, ..., -730.429185, -732.777832,
           -735.126479])

  • u_prime
    (u_prime)
    float64
    648.2 645.9 643.5 ... -643.5 -645.9

    array([ 648.226543,  645.877896,  643.52925 , ..., -641.180603, -643.52925 ,
           -645.877896])

  • v
    (v)
    float64
    -735.1 -732.8 ... 732.8 735.1

    array([-735.126479, -732.777832, -730.429185, ...,  730.429185,  732.777832,
            735.126479])

  • v_prime
    (v_prime)
    float64
    -648.2 -645.9 ... 643.5 645.9

    array([-648.226543, -645.877896, -643.52925 , ...,  641.180603,  643.52925 ,
            645.877896])

• Data variables: (4)
  • AMPLITUDE
    (time, chan, pol, u_prime, v_prime)
    float64
    dask.array<chunksize=(1, 1, 1, 276, 276), meta=np.ndarray>

    Array Chunk Bytes 9.30 MiB 595.12 kiB Shape (1, 1, 4, 552, 552) (1, 1, 1, 276, 276) Dask graph 16 chunks in 2 graph layers Data type float64 numpy.ndarray

  • APERTURE
    (time, chan, pol, u, v)
    complex128
    dask.array<chunksize=(1, 1, 1, 157, 314), meta=np.ndarray>

    Array Chunk Bytes 23.99 MiB 770.28 kiB Shape (1, 1, 4, 627, 627) (1, 1, 1, 157, 314) Dask graph 32 chunks in 2 graph layers Data type complex128 numpy.ndarray

  • BEAM
    (time, chan, pol, l, m)
    complex128
    dask.array<chunksize=(1, 1, 4, 29, 29), meta=np.ndarray>

    Array Chunk Bytes 52.56 kiB 52.56 kiB Shape (1, 1, 4, 29, 29) (1, 1, 4, 29, 29) Dask graph 1 chunks in 2 graph layers Data type complex128 numpy.ndarray

  • CORRECTED_PHASE
    (time, chan, pol, u_prime, v_prime)
    float64
    dask.array<chunksize=(1, 1, 1, 276, 276), meta=np.ndarray>

    Array Chunk Bytes 9.30 MiB 595.12 kiB Shape (1, 1, 4, 552, 552) (1, 1, 1, 276, 276) Dask graph 16 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (9)
  • chan
    PandasIndex

    PandasIndex(Index([14167000000.0], dtype='float64', name='chan'))

  • ddi
    PandasIndex

    PandasIndex(Index(['ddi_0'], dtype='object', name='ddi'))

  • l
    PandasIndex

    PandasIndex(Index([  0.009506984076958066,   0.008827913785746776,   0.008148843494535485,
             0.007469773203324195,   0.006790702912112905,  0.0061116326209016145,
             0.005432562329690324,   0.004753492038479033,   0.004074421747267742,
            0.0033953514560564526,   0.002716281164845162,   0.002037210873633871,
             0.001358140582422581,  0.0006790702912112905,                   -0.0,
           -0.0006790702912112905,  -0.001358140582422581,  -0.002037210873633871,
            -0.002716281164845162, -0.0033953514560564526,  -0.004074421747267742,
            -0.004753492038479033,  -0.005432562329690324, -0.0061116326209016145,
            -0.006790702912112905,  -0.007469773203324195,  -0.008148843494535485,
            -0.008827913785746776,  -0.009506984076958066],
          dtype='float64', name='l'))

  • m
    PandasIndex

    PandasIndex(Index([ -0.009506984076958066,  -0.008827913785746776,  -0.008148843494535485,
            -0.007469773203324195,  -0.006790702912112905, -0.0061116326209016145,
            -0.005432562329690324,  -0.004753492038479033,  -0.004074421747267742,
           -0.0033953514560564526,  -0.002716281164845162,  -0.002037210873633871,
            -0.001358140582422581, -0.0006790702912112905,                    0.0,
            0.0006790702912112905,   0.001358140582422581,   0.002037210873633871,
             0.002716281164845162,  0.0033953514560564526,   0.004074421747267742,
             0.004753492038479033,   0.005432562329690324,  0.0061116326209016145,
             0.006790702912112905,   0.007469773203324195,   0.008148843494535485,
             0.008827913785746776,   0.009506984076958066],
          dtype='float64', name='m'))

  • pol
    PandasIndex

    PandasIndex(Index(['I', 'Q', 'U', 'V'], dtype='object', name='pol'))

  • u
    PandasIndex

    PandasIndex(Index([ 735.1264785029531,  732.7778316067775,  730.4291847106019,
            728.0805378144264,  725.7318909182508,  723.3832440220752,
            721.0345971258996,  718.6859502297241,  716.3373033335486,
            713.9886564373729,
           ...
           -713.9886564373729, -716.3373033335486, -718.6859502297241,
           -721.0345971258996, -723.3832440220752, -725.7318909182508,
           -728.0805378144264, -730.4291847106019, -732.7778316067775,
           -735.1264785029531],
          dtype='float64', name='u', length=627))

  • u_prime
    PandasIndex

    PandasIndex(Index([  648.226543344457,  645.8778964482815,  643.5292495521059,
            641.1806026559303,  638.8319557597547,  636.4833088635792,
            634.1346619674036,   631.786015071228,  629.4373681750525,
            627.0887212788768,
           ...
           -624.7400743827013, -627.0887212788768, -629.4373681750525,
            -631.786015071228, -634.1346619674036, -636.4833088635792,
           -638.8319557597547, -641.1806026559303, -643.5292495521059,
           -645.8778964482815],
          dtype='float64', name='u_prime', length=552))

  • v
    PandasIndex

    PandasIndex(Index([-735.1264785029531, -732.7778316067775, -730.4291847106019,
           -728.0805378144264, -725.7318909182508, -723.3832440220752,
           -721.0345971258996, -718.6859502297241, -716.3373033335486,
           -713.9886564373729,
           ...
            713.9886564373729,  716.3373033335486,  718.6859502297241,
            721.0345971258996,  723.3832440220752,  725.7318909182508,
            728.0805378144264,  730.4291847106019,  732.7778316067775,
            735.1264785029531],
          dtype='float64', name='v', length=627))

  • v_prime
    PandasIndex

    PandasIndex(Index([ -648.226543344457, -645.8778964482815, -643.5292495521059,
           -641.1806026559303, -638.8319557597547, -636.4833088635792,
           -634.1346619674036,  -631.786015071228, -629.4373681750525,
           -627.0887212788768,
           ...
            624.7400743827013,  627.0887212788768,  629.4373681750525,
             631.786015071228,  634.1346619674036,  636.4833088635792,
            638.8319557597547,  641.1806026559303,  643.5292495521059,
            645.8778964482815],
          dtype='float64', name='v_prime', length=552))

• Attributes: (6)

  ant_id :

  ant_ea25

  ant_name :

  ea25

  aperture_resolution :

  [1.413428251773375, 1.413428251773375]

  ddi :

  ddi_0

  telescope_name :

  EVLA

  time_centroid :

  5170359434.000001

A summary of the available key values can be obtained using the summary convenience function.

image_mds.summary()

####################################################################################################
###                                         Summary for:                                         ###
###                       data/ea25_cal_small_after_fixed.split.image.zarr                       ###
####################################################################################################

Full documentation for AstrohackImageFile objects' API at:
https://astrohack.readthedocs.io/en/stable/_api/autoapi/astrohack/mds/index.html#astrohack.mds.AstrohackImageFile

Input Parameters:
+-------------------------+--------------------------------------------------+
| Parameter               | Value                                            |
+-------------------------+--------------------------------------------------+
| holog_name              | data/ea25_cal_small_after_fixed.split.holog.zarr |
| grid_size               | None                                             |
| cell_size               | None                                             |
| image_name              | data/ea25_cal_small_after_fixed.split.image.zarr |
| padding_factor          | 50                                               |
| grid_interpolation_mode | linear                                           |
| chan_average            | True                                             |
| chan_tolerance_factor   | 0.005                                            |
| scan_average            | True                                             |
| ant                     | all                                              |
| ddi                     | all                                              |
| to_stokes               | True                                             |
| apply_mask              | True                                             |
| phase_fit               | True                                             |
| overwrite               | True                                             |
| parallel                | True                                             |
| version                 | 0.4.5                                            |
| origin                  | holog                                            |
+-------------------------+--------------------------------------------------+

Contents:
+----------+--------------------+
| Antenna  | DDI                |
+----------+--------------------+
| ant_ea25 | ['ddi_0', 'ddi_1'] |
| ant_ea06 | ['ddi_0', 'ddi_1'] |
+----------+--------------------+

Available methods:
+----------------+------------------------------------------------------------------------------+
| Methods        | Description                                                                  |
+----------------+------------------------------------------------------------------------------+
| summary        | Prints summary of the AstrohackImageFile object, with available data,        |
|                | attributes and available methods                                             |
| select         | Select data on the basis of ddi, scan, ant. This is a convenience function.  |
| export_to_fits | Export contents of an AstrohackImageFile object to several FITS files in the |
|                | destination folder                                                           |
| plot_beams     | Beam plots from the data in an AstrohackImageFIle object.                    |
| plot_apertures | Aperture amplitude and phase plots from the data in an AstrohackImageFIle    |
|                | object.                                                                      |
+----------------+------------------------------------------------------------------------------+

Each of the holography output files is a compound dictionary with respect to the run parameters and contains a xarray Dataset, this means that the holography files have access to all native xarray functionality. The user can use their favorite plotting package to visualize the data or use xarray’s internal functions to do simple filtering and plotting.

image_mds['ant_ea25']['ddi_0'].CORRECTED_PHASE.isel(chan=0, pol=0).plot()

<matplotlib.collections.QuadMesh at 0x7f18f7d3e620>

Panel

The panel function takes the place of and expands the PANEL AIPS function to processes the image information and derives adjustements to the dish panels. This produces a file on disk of format .panel.zarr containing information on corrections, residuals and screw adjustments. As an added bonus the panel function has a helper function to convert aips data to astrohack format and process it using the aips_holog_to_astrohack function. For a full description of the operation and arguments of the panel function see docs.

from astrohack.panel import panel

panel_model = 'rigid'

panel_mds = panel(
    image_name='data/ea25_cal_small_after_fixed.split.image.zarr',
    panel_model=panel_model,
    panel_margins=0.2,
    clip_type='relative',
    clip_level=0.2,
    parallel=True,
    overwrite=True
)

[2024-02-14 12:53:36,343]     INFO  graphviper:  Checking parameter values for panel.panel
[2024-02-14 12:53:36,345]     INFO   astrohack:  /export/home/ajax/jhoskins/Development/astrohack/
[2024-02-14 12:53:36,346]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:53:36,347]     INFO   astrohack:  Failed to find module in PARAMETER_CONFIG_PATH ... attempting to check common directories ...
[2024-02-14 12:53:36,347]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/Development/astrohack/ for configuration file, please wait ...
[2024-02-14 12:53:36,395]     INFO  graphviper:  Creating output file name: data/ea25_cal_small_after_fixed.split.panel.zarr
[2024-02-14 12:53:37,189]     INFO    worker_0:  processing ant_ea25 ddi_0

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:77: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  if inputxds.dims['chan'] != 1:
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:93: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  self.npoint = np.sqrt(inputxds.dims['l'] ** 2 + inputxds.dims['m'] ** 2)

[2024-02-14 12:53:37,529]     INFO    worker_1:  processing ant_ea25 ddi_1

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:77: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  if inputxds.dims['chan'] != 1:
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:93: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  self.npoint = np.sqrt(inputxds.dims['l'] ** 2 + inputxds.dims['m'] ** 2)

[2024-02-14 12:53:40,159]     INFO    worker_0:  processing ant_ea06 ddi_0

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:77: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  if inputxds.dims['chan'] != 1:
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:93: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  self.npoint = np.sqrt(inputxds.dims['l'] ** 2 + inputxds.dims['m'] ** 2)

[2024-02-14 12:53:41,502]     INFO    worker_1:  processing ant_ea06 ddi_1

/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:77: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  if inputxds.dims['chan'] != 1:
/export/home/ajax/jhoskins/Development/astrohack/src/astrohack/_utils/_panel_classes/antenna_surface.py:93: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  self.npoint = np.sqrt(inputxds.dims['l'] ** 2 + inputxds.dims['m'] ** 2)

[2024-02-14 12:53:43,122]  WARNING    worker_1:  Fit failed with the rigid model and a simple mean has been used instead for the following panels: ['ea06', 'ddi_1']
[2024-02-14 12:53:43,122]  WARNING    worker_1:  ['6-10']
[2024-02-14 12:53:43,762]     INFO  graphviper:  Finished processing

panel_name.panel.zarr: The panel zarr file contains process information regarding the per panel screw corrections as well as residuals, masks and phase corrections used to produce them. The panel file structure is a compound dictionary keyed according to ant -> ddi with the value consisting of the panel dataset.

panel_mds =
{
   ant_0:{
          ddi_0: panel_ds,
                 ⋮
          ddi_m: panel_ds
         },
       ⋮
   ant_n: …

An example of the panel dataset object is show below.

panel_mds['ant_ea25']['ddi_0']

<xarray.Dataset>
Dimensions:             (u: 552, v: 552, labels: 172, pars: 3, screws: 4)
Coordinates:
  * labels              (labels) object '1-1' '1-2' '1-3' ... '6-39' '6-40'
  * pars                (pars) int64 0 1 2
  * screws              (screws) <U2 'il' 'ir' 'ol' 'or'
  * u                   (u) float64 13.72 13.67 13.62 ... -13.57 -13.62 -13.67
  * v                   (v) float64 -13.72 -13.67 -13.62 ... 13.57 13.62 13.67
Data variables:
    AMPLITUDE           (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
    CORRECTIONS         (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
    DEVIATION           (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
    MASK                (u, v) bool dask.array<chunksize=(276, 552), meta=np.ndarray>
    PANEL_DISTRIBUTION  (u, v) int32 dask.array<chunksize=(276, 276), meta=np.ndarray>
    PANEL_PARAMETERS    (labels, pars) float64 dask.array<chunksize=(172, 3), meta=np.ndarray>
    PANEL_SCREWS        (labels, screws) float64 dask.array<chunksize=(172, 4), meta=np.ndarray>
    PHASE               (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
    PHASE_CORRECTIONS   (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
    PHASE_RESIDUALS     (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
    RESIDUALS           (u, v) float64 dask.array<chunksize=(138, 276), meta=np.ndarray>
Attributes: (12/16)
    amp_unit:             V
    ant_name:             ea25
    aperture_resolution:  [1.413428251773375, 1.413428251773375]
    clip:                 0.7180989426650104
    ddi:                  ddi_0
    fitted:               True
    ...                   ...
    panel_margin:         0.2
    panel_model:          rigid
    solved:               True
    telescope_name:       VLA
    theoretical_gain:     100.1817827185667
    wavelength:           0.021161322651231735

xarray.Dataset

• Dimensions:
  • u: 552
  • v: 552
  • labels: 172
  • pars: 3
  • screws: 4
• Coordinates: (5)
  • labels
    (labels)
    object
    '1-1' '1-2' '1-3' ... '6-39' '6-40'

    array(['1-1', '1-2', '1-3', '1-4', '1-5', '1-6', '1-7', '1-8', '1-9', '1-10',
           '1-11', '1-12', '2-1', '2-2', '2-3', '2-4', '2-5', '2-6', '2-7', '2-8',
           '2-9', '2-10', '2-11', '2-12', '2-13', '2-14', '2-15', '2-16', '3-1',
           '3-2', '3-3', '3-4', '3-5', '3-6', '3-7', '3-8', '3-9', '3-10', '3-11',
           '3-12', '3-13', '3-14', '3-15', '3-16', '3-17', '3-18', '3-19', '3-20',
           '3-21', '3-22', '3-23', '3-24', '4-1', '4-2', '4-3', '4-4', '4-5',
           '4-6', '4-7', '4-8', '4-9', '4-10', '4-11', '4-12', '4-13', '4-14',
           '4-15', '4-16', '4-17', '4-18', '4-19', '4-20', '4-21', '4-22', '4-23',
           '4-24', '4-25', '4-26', '4-27', '4-28', '4-29', '4-30', '4-31', '4-32',
           '4-33', '4-34', '4-35', '4-36', '4-37', '4-38', '4-39', '4-40', '5-1',
           '5-2', '5-3', '5-4', '5-5', '5-6', '5-7', '5-8', '5-9', '5-10', '5-11',
           '5-12', '5-13', '5-14', '5-15', '5-16', '5-17', '5-18', '5-19', '5-20',
           '5-21', '5-22', '5-23', '5-24', '5-25', '5-26', '5-27', '5-28', '5-29',
           '5-30', '5-31', '5-32', '5-33', '5-34', '5-35', '5-36', '5-37', '5-38',
           '5-39', '5-40', '6-1', '6-2', '6-3', '6-4', '6-5', '6-6', '6-7', '6-8',
           '6-9', '6-10', '6-11', '6-12', '6-13', '6-14', '6-15', '6-16', '6-17',
           '6-18', '6-19', '6-20', '6-21', '6-22', '6-23', '6-24', '6-25', '6-26',
           '6-27', '6-28', '6-29', '6-30', '6-31', '6-32', '6-33', '6-34', '6-35',
           '6-36', '6-37', '6-38', '6-39', '6-40'], dtype=object)

  • pars
    (pars)
    int64
    0 1 2

    array([0, 1, 2])

  • screws
    (screws)
    <U2
    'il' 'ir' 'ol' 'or'

    array(['il', 'ir', 'ol', 'or'], dtype='<U2')

  • u
    (u)
    float64
    13.72 13.67 13.62 ... -13.62 -13.67

    array([ 13.717331,  13.667631,  13.61793 , ..., -13.56823 , -13.61793 ,
           -13.667631])

  • v
    (v)
    float64
    -13.72 -13.67 ... 13.62 13.67

    array([-13.717331, -13.667631, -13.61793 , ...,  13.56823 ,  13.61793 ,
            13.667631])

• Data variables: (11)
  • AMPLITUDE
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

  • CORRECTIONS
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

  • DEVIATION
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

  • MASK
    (u, v)
    bool
    dask.array<chunksize=(276, 552), meta=np.ndarray>

    Array Chunk Bytes 297.56 kiB 148.78 kiB Shape (552, 552) (276, 552) Dask graph 2 chunks in 2 graph layers Data type bool numpy.ndarray

  • PANEL_DISTRIBUTION
    (u, v)
    int32
    dask.array<chunksize=(276, 276), meta=np.ndarray>

    Array Chunk Bytes 1.16 MiB 297.56 kiB Shape (552, 552) (276, 276) Dask graph 4 chunks in 2 graph layers Data type int32 numpy.ndarray

  • PANEL_PARAMETERS
    (labels, pars)
    float64
    dask.array<chunksize=(172, 3), meta=np.ndarray>

    Array Chunk Bytes 4.03 kiB 4.03 kiB Shape (172, 3) (172, 3) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • PANEL_SCREWS
    (labels, screws)
    float64
    dask.array<chunksize=(172, 4), meta=np.ndarray>

    Array Chunk Bytes 5.38 kiB 5.38 kiB Shape (172, 4) (172, 4) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • PHASE
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

  • PHASE_CORRECTIONS
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

  • PHASE_RESIDUALS
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

  • RESIDUALS
    (u, v)
    float64
    dask.array<chunksize=(138, 276), meta=np.ndarray>

    Array Chunk Bytes 2.32 MiB 297.56 kiB Shape (552, 552) (138, 276) Dask graph 8 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (5)
  • labels
    PandasIndex

    PandasIndex(Index(['1-1', '1-2', '1-3', '1-4', '1-5', '1-6', '1-7', '1-8', '1-9', '1-10',
           ...
           '6-31', '6-32', '6-33', '6-34', '6-35', '6-36', '6-37', '6-38', '6-39',
           '6-40'],
          dtype='object', name='labels', length=172))

  • pars
    PandasIndex

    PandasIndex(Index([0, 1, 2], dtype='int64', name='pars'))

  • screws
    PandasIndex

    PandasIndex(Index(['il', 'ir', 'ol', 'or'], dtype='object', name='screws'))

  • u
    PandasIndex

    PandasIndex(Index([ 13.717331034804708,  13.667630560040925,  13.617930085277138,
            13.568229610513352,  13.518529135749567,  13.468828660985782,
            13.419128186221998,  13.369427711458211,  13.319727236694428,
            13.270026761930641,
           ...
           -13.220326287166856, -13.270026761930641, -13.319727236694428,
           -13.369427711458211, -13.419128186221998, -13.468828660985782,
           -13.518529135749567, -13.568229610513352, -13.617930085277138,
           -13.667630560040925],
          dtype='float64', name='u', length=552))

  • v
    PandasIndex

    PandasIndex(Index([-13.717331034804708, -13.667630560040925, -13.617930085277138,
           -13.568229610513352, -13.518529135749567, -13.468828660985782,
           -13.419128186221998, -13.369427711458211, -13.319727236694428,
           -13.270026761930641,
           ...
            13.220326287166856,  13.270026761930641,  13.319727236694428,
            13.369427711458211,  13.419128186221998,  13.468828660985782,
            13.518529135749567,  13.568229610513352,  13.617930085277138,
            13.667630560040925],
          dtype='float64', name='v', length=552))

• Attributes: (16)

  amp_unit :

  V

  ant_name :

  ea25

  aperture_resolution :

  [1.413428251773375, 1.413428251773375]

  clip :

  0.7180989426650104

  ddi :

  ddi_0

  fitted :

  True

  input_gain :

  100.077799701843

  input_rms :

  0.0004255220113628886

  output_gain :

  100.16090012500686

  output_rms :

  0.0001854182409937069

  panel_margin :

  0.2

  panel_model :

  rigid

  solved :

  True

  telescope_name :

  VLA

  theoretical_gain :

  100.1817827185667

  wavelength :

  0.021161322651231735

A summary of the available key values can be obtained using the summary convenience function.

panel_mds.summary()

####################################################################################################
###                                         Summary for:                                         ###
###                       data/ea25_cal_small_after_fixed.split.panel.zarr                       ###
####################################################################################################

Full documentation for AstrohackPanelFile objects' API at:
https://astrohack.readthedocs.io/en/stable/_api/autoapi/astrohack/mds/index.html#astrohack.mds.AstrohackPanelFile

Input Parameters:
+--------------------+--------------------------------------------------+
| Parameter          | Value                                            |
+--------------------+--------------------------------------------------+
| image_name         | data/ea25_cal_small_after_fixed.split.image.zarr |
| panel_name         | data/ea25_cal_small_after_fixed.split.panel.zarr |
| clip_type          | relative                                         |
| clip_level         | 0.2                                              |
| panel_model        | rigid                                            |
| panel_margins      | 0.2                                              |
| polarization_state | I                                                |
| ant                | all                                              |
| ddi                | all                                              |
| parallel           | True                                             |
| overwrite          | True                                             |
| version            | 0.4.5                                            |
| origin             | panel                                            |
+--------------------+--------------------------------------------------+

Contents:
+----------+--------------------+
| Antenna  | DDI                |
+----------+--------------------+
| ant_ea25 | ['ddi_0', 'ddi_1'] |
| ant_ea06 | ['ddi_0', 'ddi_1'] |
+----------+--------------------+

Available methods:
+----------------+-----------------------------------------------------------------------+
| Methods        | Description                                                           |
+----------------+-----------------------------------------------------------------------+
| summary        | Prints summary of the AstrohackPanelFile object, with available data, |
|                | attributes and available methods                                      |
| get_antenna    | Retrieve an AntennaSurface object for interaction                     |
| export_screws  | Export screw adjustments to text files and optionally plots.          |
| export_to_fits | Export contents of an Astrohack MDS file to several FITS files in the |
|                | destination folder                                                    |
| plot_antennas  | Create diagnostic plots of antenna surfaces from panel data file.     |
+----------------+-----------------------------------------------------------------------+

Additional Functions

The panel_mds object provides two helper functions for the user to export or investigate the results of the panel function. - export_screws(): This method exports the screw and panel adjustements from the panel output file. - plot_antennas(): This method plots one of three diagnostics plots from the panel output file data. The plots types are: deviation, phase and ancillary.

Examples usage for each helper functions are given below and more detailed documentation can be found in the visualization tutorial.

export_folder = 'exports'

panel_mds.export_screws(
    destination=export_folder,
    ant='ea25',
    ddi=0,
    unit='mm',
    threshold=0.5, # Threshold in mm for significant adjustments
    display=True
)

[2024-02-14 12:53:43,938]     INFO  graphviper:  Checking parameter values for mds.AstrohackPanelFile.export_screws
[2024-02-14 12:53:43,940]     INFO   astrohack:  /export/home/ajax/jhoskins/Development/astrohack/
[2024-02-14 12:53:43,941]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:53:43,946]     INFO   astrohack:  Failed to find module in PARAMETER_CONFIG_PATH ... attempting to check common directories ...
[2024-02-14 12:53:43,948]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/Development/astrohack/ for configuration file, please wait ...
[2024-02-14 12:53:44,035]  WARNING  graphviper:  Destination folder already exists, results may be overwritten

with open(export_folder+'/panel_screws_ant_ea25_ddi_0.txt', 'r') as file:
    for _ in range(30):
        print(file.readline()[:-1])

# Screw adjustments for VLA ea25 antenna
# Adjustments are in mm

# Lower means away from subreflector
# Raise means toward the subreflector
# LOWER the panel if the number is POSITIVE
# RAISE the panel if the number is NEGATIVE


Panel           il         ir         ol         or
1-1           -0.15       0.21       0.15       0.83
1-2           -0.25      -0.36       0.35       0.13
1-3            0.41       0.39       0.01      -0.03
1-4           -0.43      -0.75      -0.09      -0.68
1-5           -0.39      -0.24      -0.40      -0.13
1-6           -0.74      -0.67       0.01       0.13
1-7           -1.04      -1.05       0.18       0.16
1-8           -1.45      -1.20       0.15       0.62
1-9            0.37       0.12      -0.14      -0.61
1-10           0.32       0.71      -0.84      -0.11
1-11           0.95       1.16      -0.37       0.01
1-12           0.63       0.11       0.70      -0.27
2-1            0.57       0.74       0.24       0.49
2-2           -0.26      -0.35       0.03      -0.10
2-3            0.29       0.20      -0.43      -0.56
2-4           -0.08      -0.08      -0.18      -0.18
2-5           -0.49       0.10      -0.45       0.45
2-6            0.27      -0.31       0.89       0.01
2-7           -0.80      -0.50      -0.40       0.06
2-8            0.22       0.11      -0.24      -0.42

panel_mds.plot_antennas(
    destination=export_folder,
    ant='ea25',
    ddi=0,
    plot_type='deviation',
    plot_screws=False,
    dpi=300,
    parallel=False,
    display=True
)

[2024-02-14 12:53:47,289]     INFO  graphviper:  Checking parameter values for mds.AstrohackPanelFile.plot_antennas
[2024-02-14 12:53:47,292]     INFO   astrohack:  /export/home/ajax/jhoskins/Development/astrohack/
[2024-02-14 12:53:47,293]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/.conda/envs/viper-3.10/lib/python3.10/site-packages/graphviper/config/ for configuration file, please wait ...
[2024-02-14 12:53:47,294]     INFO   astrohack:  Failed to find module in PARAMETER_CONFIG_PATH ... attempting to check common directories ...
[2024-02-14 12:53:47,298]     INFO  graphviper:  Searching /export/home/ajax/jhoskins/Development/astrohack/ for configuration file, please wait ...
[2024-02-14 12:53:47,365]  WARNING  graphviper:  Destination folder already exists, results may be overwritten