Exploring Spelunker tools and features ======================== -------------- **Authors:** Derod Deal (dealderod@ufl.edu), Néstor Espinoza (nespinoza@stsci.edu) \| **Last update:** January 22, 2024 **Program version:** ver. 1.1.9 The JWST telescope carries four different instruments: NIRCam, NIRSpec, MIRI and FGS/NIRISS — the latter containing the Fine Guidance Sensor (FGS). FGS Spelunker is a package designed to conveniently analyze guidestar data. In this notebook, we cover the following main functions of this package. 1. `Getting started <#getting-started>`__ - Installation - Using ``Spelunker`` 2. `Downloading data <#downloading-data>`__ 3. Spatial fitting guide stars - `Gaussian fitting <#gaussian-fitting>`__ - `Quick fitting <#quick-fitting>`__ 4. `Plotting parameters <#plotting-parameters>`__ 5. Periodograms - `Creating a periodogram <#periodograms>`__ 6. `Mnemonics <#mnemonics>`__ 7. `Animations <#animations>`__ 8. `Getting tables <#getting-tables>`__ Getting started --------------- To get started with FGS Spelunker, first call ``spelunker.load`` into a variable while setting a given Program ID. .. code:: python import spelunker spk = spelunker.load(pid=1534) Calling load without the pid parameter ``spelunker.load()`` will initialize ``spelunker`` without downloading any of the files. This is useful if you already have guidestar FITS available locally. Downloading data ---------------- To load Spelunker with a given Program ID for JWST, simply call ``spelunker.load`` with the Program ID ``pid`` as a parameter. This will create a directory called ``spelunker_results``, which is where the FITS files from a selected Program ID and other data will be downloaded and saved. You can define your own directory by using ``dir=``. The Program IDs that can be loaded are limited to programs without an exclusive access period or are available publicly (https://jwst-docs.stsci.edu/accessing-jwst-data/jwst-data-retrieval/data-access-policy#DataAccessPolicy-Exclusiveaccessperiod). .. code:: python import spelunker spk = spelunker.load(dir='/Users/ddeal/JWST-Treasure-Chest/', pid='1534') .. parsed-literal:: Current working directory for spelunker: /Users/ddeal/JWST-Treasure-Chest/spelunker_outputs Connecting with astroquery... INFO: Found cached file ./mastDownload/JWST/jw01534002002_04101_00001_guider2/jw01534002002_gs-fg_2022338021919_cal.fits with expected size 10428480. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534001004_03101_00001_guider1/jw01534001004_gs-fg_2022340010755_cal.fits with expected size 8766720. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534002004_03101_00001_guider2/jw01534002004_gs-fg_2022338025056_cal.fits with expected size 8769600. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534001002_03101_00001_guider1/jw01534001002_gs-fg_2022340003651_cal.fits with expected size 8772480. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534002003_03101_00001_guider2/jw01534002003_gs-fg_2022338023521_cal.fits with expected size 8772480. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534001003_03101_00001_guider1/jw01534001003_gs-fg_2022340005224_cal.fits with expected size 8772480. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534002001_05101_00002_guider2/jw01534002001_gs-fg_2022338014704_cal.fits with expected size 10941120. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534002001_05101_00002_guider2/jw01534002001_gs-fg_2022338015941_cal.fits with expected size 7830720. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534001001_03101_00002_guider1/jw01534001001_gs-fg_2022340000825_cal.fits with expected size 9388800. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534001001_03101_00002_guider1/jw01534001001_gs-fg_2022340002102_cal.fits with expected size 7827840. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534004004_03101_00001_guider2/jw01534004004_gs-fg_2023123213436_cal.fits with expected size 8769600. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534004003_03101_00001_guider2/jw01534004003_gs-fg_2023123211905_cal.fits with expected size 8766720. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534004002_03101_00001_guider2/jw01534004002_gs-fg_2023123210335_cal.fits with expected size 8766720. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534004001_03101_00002_guider2/jw01534004001_gs-fg_2023123203053_cal.fits with expected size 12974400. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534004001_03101_00002_guider2/jw01534004001_gs-fg_2023123204330_cal.fits with expected size 7827840. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534003001_03101_00002_guider1/jw01534003001_gs-fg_2023125174543_cal.fits with expected size 9809280. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534003001_03101_00002_guider1/jw01534003001_gs-fg_2023125175812_cal.fits with expected size 7793280. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534003002_02101_00001_guider1/jw01534003002_gs-fg_2023125181351_cal.fits with expected size 8337600. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534003003_02101_00001_guider1/jw01534003003_gs-fg_2023125182911_cal.fits with expected size 8337600. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534003004_02101_00001_guider1/jw01534003004_gs-fg_2023125185519_cal.fits with expected size 8337600. [astroquery.query] To download the data after initialization, use ``spk.download()`` with given proposal id with the optional parameters observation number ``obs_num`` and visit number ``visit``. You can also set the calibration level ``calib_level``. This information are required to use ``astroquery.mast`` to search and download the necessary files. The download function will download the selected files in the given directory and create a 2D array of the guidestar data as well as an array of time and a flux timeseries. The same parameters work with ``spelunker.load``. .. code:: python spk2 = spelunker.load(pid=1534, obs_num='2', visit='1', calib_level=2) spk2.download(1534, obs_num='2', visit='2', calib_level=2) # This overwrites the object data in spk2 with data from the input parameters .. parsed-literal:: Current working directory for spelunker: /Users/ddeal/JWST-Treasure-Chest/spelunker_outputs Connecting with astroquery... .. parsed-literal:: 2023-08-02 21:11:34,101 - stpipe - INFO - Found cached file ./mastDownload/JWST/jw01534002001_05101_00002_guider2/jw01534002001_gs-fg_2022338014704_cal.fits with expected size 10941120. 2023-08-02 21:11:34,195 - stpipe - INFO - Found cached file ./mastDownload/JWST/jw01534002001_05101_00002_guider2/jw01534002001_gs-fg_2022338015941_cal.fits with expected size 7830720. .. parsed-literal:: INFO: Found cached file ./mastDownload/JWST/jw01534002001_05101_00002_guider2/jw01534002001_gs-fg_2022338014704_cal.fits with expected size 10941120. [astroquery.query] INFO: Found cached file ./mastDownload/JWST/jw01534002001_05101_00002_guider2/jw01534002001_gs-fg_2022338015941_cal.fits with expected size 7830720. [astroquery.query] Connecting with astroquery... .. parsed-literal:: 2023-08-02 21:11:41,186 - stpipe - INFO - Found cached file ./mastDownload/JWST/jw01534002002_04101_00001_guider2/jw01534002002_gs-fg_2022338021919_cal.fits with expected size 10428480. .. parsed-literal:: INFO: Found cached file ./mastDownload/JWST/jw01534002002_04101_00001_guider2/jw01534002002_gs-fg_2022338021919_cal.fits with expected size 10428480. [astroquery.query] After we downloaded our data, we can access preprocessed spatial, time, and flux arrays for all FITS files images under the specified Program ID. Use the attributes ``spk.fg_array``, ``spk.fg_time``, and ``spk.fg_flux`` to access the arrays. .. code:: python spk2.fg_array.shape, spk2.fg_time.shape, spk2.fg_flux.shape .. parsed-literal:: ((10240, 8, 8), (10240,), (10240,)) Previously downloaded FITS files in a given directory will not be re-downloaded. If there are multiple files downloaded for the given parameter, ``spk.download`` will automatically stitch the data from the files into an array based on the date and time for each file, along with the time and flux arrays. FGS Spelunker can also handle guidestar FITS already stored locally by using: .. code:: python spk2 = spelunker.load() spk2.readfile(pid=1534) Any files under the initialized directory and specified Program ID, observation number, and visit number will be loaded into ``spk2``. ``spelunker.load.readfile`` now has access to the same attributes as ``spelunker.load.download``. So, using ``spk.object_properties`` and ``spk.fg_table`` will work. Spatial fitting guide stars --------------------------- After downloading the data, we can perform spatial fitting gaussians to each frame in a guidestar timeseries. This uses parallel processing through ``ray`` to speed up the process. We can also perform quick fits to speed through a given timeseries, though this method is a lot less accurate in the fitting. Gaussian fitting ~~~~~~~~~~~~~~~~ The downloaded data comes as a spatial timeseries of a selected guidestar. To measure the centriods and PSF width of each frame, we need to apply fitting. We will use Gaussian spatial fitting to measure x and y pixel coordinates, x and y standard deviations, theta, and the offset. To perform spatial gaussian fitting, use ``gauss2d_fit`` with guidestar arrays (the timeseries needs to be in an 8 by 8 array, which should be the same for all guidestar fine guidence products). .. code:: python spk.gauss2d_fit() # ncpus sets the number of cpu cores your computer has. Defaults to 4 cores. .. code:: python # We are going to limit the amount of frames that we input into gauss2d_fit and other methods # since the gauss2d_fit can take a few houts for very large arrays. spk.fg_array = spk.fg_array[0:10000] spk.fg_flux = spk.fg_flux[0:10000] spk.fg_time = spk.fg_time[0:10000] .. code:: python table_gauss_fit = spk.gauss2d_fit(ncpus=6) .. parsed-literal:: 2023-08-02 21:12:50,384 INFO worker.py:1636 -- Started a local Ray instance. The ``gauss2d_fit`` function outputs an astropy table, which can bee accessed with the ``spk.gaussfit_results`` attribute. If ``gauss2d_fit`` fails to fit a frame, it will return ``nan`` for that frame. .. code:: python spk.gaussfit_results .. raw:: html
Table length=10000
amplitudex_meany_meanx_stddevy_stddevthetaoffset
float64float64float64float64float64float64float64
280706.154657657963.17742943562499972.74653028381352060.63509760703873010.614009020575321-1.91035951306502283023.106318279726
280706.154657657963.17742943562499972.74653028381352060.63509760703873010.614009020575321-1.91035951306502283023.106318279726
280963.55405048133.1776044623331862.74835974624525470.63064545439651040.6193386849707871-2.0579729027468763149.3240730860866
280963.55405048133.1776044623331862.74835974624525470.63064545439651040.6193386849707871-2.0579729027468763149.3240730860866
282706.52503123613.17648618370687162.7498178715159130.63342731998220010.6145497343103167-1.95041910925019433053.0948632606123
282706.52503123613.17648618370687162.7498178715159130.63342731998220010.6145497343103167-1.95041910925019433053.0948632606123
277126.336302669843.17488276017285642.74774958743966740.61897978990402090.6340116557887706-3.484499592581963105.682301707251
277126.336302669843.17488276017285642.74774958743966740.61897978990402090.6340116557887706-3.484499592581963105.682301707251
280742.33449827863.17190307379999232.7566363376512710.61540401930754330.6363143600933248-3.5706448233072173017.796074602062
280742.33449827863.17190307379999232.7566363376512710.61540401930754330.6363143600933248-3.5706448233072173017.796074602062
.....................
288936.65879971443.15148489959746142.8164213379227280.60784140781271580.6255153338398373-0.7241022199442983159.747623016102
288936.65879971443.15148489959746142.8164213379227280.60784140781271580.6255153338398373-0.7241022199442983159.747623016102
287608.52048828263.1480812095191212.80975749133361540.60922683786757550.6288855374510539-0.63644189044221643098.4078599410695
287608.52048828263.1480812095191212.80975749133361540.60922683786757550.6288855374510539-0.63644189044221643098.4078599410695
286304.07276267293.14716231186941762.81020832089688130.60853555211725780.6298236704220975-0.55916152973308633183.299010073181
286304.07276267293.14716231186941762.81020832089688130.60853555211725780.6298236704220975-0.55916152973308633183.299010073181
284871.64866898213.14994650780066142.80721672756537060.61119152360922850.6277931861719188-0.70472530498261133261.2487765038327
284871.64866898213.14994650780066142.80721672756537060.61119152360922850.6277931861719188-0.70472530498261133261.2487765038327
288107.097027307433.149404345356172.8079165522166670.60815053482865080.6295003348022744-0.60306506505780553197.4098077599647
288107.097027307433.149404345356172.8079165522166670.60815053482865080.6295003348022744-0.60306506505780553197.4098077599647
---------- Quick fitting ~~~~~~~~~~~~~ There are some situations where you need to quickly obtain rough statistics of changes in guidestar products overtime. Quick fitting fits the x and y pixel locations and standard deviations as an astropy table using centroid and variance calculations. To perform quick fitting, run ``quick_fit`` with an appropriate array. .. code:: python table_quick_fit = spk.quick_fit() .. code:: python spk.quickfit_results .. raw:: html
Table length=10000
amplitudex_meany_meanx_stddevy_stddevthetaoffset
float32float64float64float64float64int64int64
254451.563.2403148508618452.80339422974957581.744621754142441.815822823818850300
254451.563.2403148508618452.80339422974957581.744621754142441.815822823818850300
255055.253.32060047780173842.84345743035654631.85432577855573971.829339484667176400
255055.253.32060047780173842.84345743035654631.85432577855573971.829339484667176400
256947.423.35058451627363762.9256908584508491.80772926679694221.894347125504328300
256947.423.35058451627363762.9256908584508491.80772926679694221.894347125504328300
251888.123.30393893016007262.8862332312709871.8546779260188131.843317890559891500
251888.123.30393893016007262.8862332312709871.8546779260188131.843317890559891500
257109.623.28351647739718062.7743180826775341.8371070637094731.764773262302626400
257109.623.28351647739718062.7743180826775341.8371070637094731.764773262302626400
.....................
273886.843.3072480705704332.94595811378880961.86385429661336421.824857328223436800
273886.843.3072480705704332.94595811378880961.86385429661336421.824857328223436800
272548.83.3032740249933822.8885581474901681.82828363670852071.758076055683799300
272548.83.3032740249933822.8885581474901681.82828363670852071.758076055683799300
271490.13.2288204479723623.0559122192827161.81890496136441881.875506651337819100
271490.13.2288204479723623.0559122192827161.81890496136441881.875506651337819100
269606.93.3282214867590652.9637169597236311.87062236593869541.858665437469233500
269606.93.3282214867590652.9637169597236311.87062236593869541.858665437469233500
272629.93.3046554310949872.96157024048635261.8732619967099391.928847958172767800
272629.93.3046554310949872.96157024048635261.8732619967099391.928847958172767800
Plotting parameters ------------------- We can plot a timeseries of a given parameter or flux from guidestars. The method ``timeseries_binned_plot`` will generate a matplotlib axes object of a given timeseries. .. code:: python import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize = (12,4), dpi=200) ax = spk.timeseries_binned_plot() .. image:: fgs-spelunker_quickstart_files/fgs-spelunker_quickstart_35_0.png :width: 800pt Within guidestar data, changes in the PSF can impact the observed flux of the star. Certain events might see changes in all fitted parameters. In this case, subplots of each parameter will provide more information to the user about the event, giving them the change of guidestar position, brightness, and FWHM overtime. .. code:: python ax = spk.timeseries_list_plot() .. image:: fgs-spelunker_quickstart_files/fgs-spelunker_quickstart_37_0.png :width: 800pt Periodograms ------------ FGS Spelunker comes with various tools to visualize and explore guidestar data. Periodograms are useful for guidestar products to detect periodicities not only within flux timeseries, but also within centroids, FWHM, theta, and offset. From a selected fitting method, we can use the table output to apply Lomb-Scargle periodograms to our parameters. ``periodogram`` ~~~~~~~~~~~~~~~ To obtain the power and frequencies of Lomb-Scargle periodograms for each fitted parameter, use ``periodogram``. The periodograms for each given parameter from a fit can be conveniently plotted in a single figure with the same method. .. code:: python ax = spk.periodogram() .. image:: fgs-spelunker_quickstart_files/fgs-spelunker_quickstart_41_0.png :width: 800pt To get the frequency and power for each fitted parameter, use ``spk.pgram_{parameter}``. Available parameters: - ``spk.pgram_amplitude`` - ``spk.pgram_x_mean`` - ``spk.pgram_y_mean`` - ``spk.pgram_x_stddev`` - ``spk.pgram_y_stddev`` - ``spk.pgram_theta`` - ``spk.pgram_offset`` .. code:: python freq = spk.pgram_x_mean[0] # periodogram frequency power = spk.pgram_x_mean[1] # periodogram power freq[0], power[0] .. parsed-literal:: (0.0003127661546504965, 0.005397779092056495) Mnemonics --------- When observing the timeseries of the guidestar, there might be technical events from the JWST that causes changes in obtained data. For example, high gain antenna or filter changes in NIRCAM can cause noticeable changes in flux or other guidestar properties. We can overlay these events onto fitted parameters using ``mnemonics`` and ``mnemonics_plot``. You will need a MAST API token to use ``mnemonics``, as well as the ``jwstuser`` package: - https://auth.mast.stsci.edu/docs/ (MAST API TOKEN) - https://github.com/spacetelescope/jwstuser/tree/main (jwstuser) Current supported mnemonics: - *SA_ZHGAUPST* (high-gain antenna), - *INIS_FWMTRCURR* (NIRISS Filter Wheel Motor Current). There are thousands of different mnemonics to explore on https://mast.stsci.edu/viz/api/v0.1/info/mnemonics. Use `spk.mnemonics` to try the mnemonics you are interested in comparing with any JWST data, not just guidestars from the FGS. .. code:: python spk2 = spelunker.load('/Users/ddeal/JWST-Treasure-Chest/', pid=1534) .. code:: python spk2.mast_api_token = 'enter_mast_token_id_here' # input mast_api token here! fig, ax = plt.subplots(figsize=(12,4),dpi=200) ax = spk2.mnemonics_local('GUIDESTAR') # plots when the JWST tracks onto a new guidestars as a vertical line ax = spk2.mnemonics('SA_ZHGAUPST', 60067.84, 60067.9) # plots the start and end of high gain antenna movement ax.plot(spk2.fg_time, spk2.fg_flux) plt.legend(loc=3) plt.xlim(60067.84, 60067.9) .. image:: fgs-spelunker_quickstart_files/fgs-spelunker_quickstart_47_1.png :width: 800pt If you do have a MAST API token, you will have access to any program under that token. If you do not have access to a MAST API token, you can only download and use publicly available Program IDs. However, with the readfile function, you can use fine guidance files you already have downloaded locally, and with the current version, with no drawbacks. Animations ---------- Spatial data of guidestar imaging can bring essential information about how the point spread function changes overtime. Animations of the spatial timeseries are convenient and helpful methods to analyze guidestar data. To get a side by side comparison of the evolution of a spatial timeseries and a parameter, use ``flux_spatial_timelapse_animation``. You may have to install ``ffmpeg`` on your computer to get ``mp4`` formats. .. code:: python plt.plot(spk2.fg_flux[2600:3100]) .. parsed-literal:: [] .. image:: fgs-spelunker_quickstart_files/fgs-spelunker_quickstart_50_1.png .. code:: python spk2.flux_spatial_timelapse_animation(start=2600,stop=3100,) # to save an animation with a filename, use *filename=*. Defaults to movie.gif .. parsed-literal:: 2023-08-02 21:19:50,803 INFO worker.py:1636 -- Started a local Ray instance. .. image:: fgs-spelunker_quickstart_files/movie.gif :width: 800pt Getting tables -------------- After downloading a selected proposal id with ``download``, we can easily output metadata about each downloaded file, including extracted data from the filename including ``visit_group``, ``parallel_sequence_id``, and ``exposure_number``. The guide star used in each file is also included, as well as filter magnitudes and other stellar properties. .. code:: python spk.fg_table # We can simply call this attribute after using spk.download() to obtain our table! We can obtain a neat DataFrame of each tracked guidestar, which gives us information such as the intergation start times and galactic coordinates. .. code:: python spk.object_properties .. raw:: html
guidestar_catalog_id gaiadr1ID gaiadr2ID int_start int_stop ra dec Jmag Hmag
0 S1HP079555 4658077781377287680 4658077781376437888 59917.066396 59917.074354 80.837584 -69.541124 13.659 12.898
1 S1HP080554 4658077991763987712 4658077991799023616 59917.089163 59917.096759 80.806837 -69.530972 15.001 14.282
2 S1HP078573 4657983910572904320 4657983910572904320 59917.112547 59917.118705 80.807043 -69.553474 13.839 13.078
3 S1HP079590 4657986831103727872 4657986835382982016 59918.999015 59919.005848 80.510790 -69.545479 15.410 14.839
4 S1HP079769 4657986831078120832 4657986835433225728 59919.019436 59919.025598 80.518235 -69.543415 15.231 14.341
5 S1HP078292 4657986796681532672 4657986801073794432 59919.041018 59919.047165 80.519564 -69.558464 12.804 11.883
6 S1HP077850 4657986762384054144 4657986766713867264 60067.871344 60067.877490 80.573531 -69.562862 12.957 12.227
7 S1HP197501 4657986865463528832 4657986869793061376 60067.882117 60067.888264 80.571447 -69.551750 13.063 12.168
8 S1HP773376 4658078124973829632 60069.733171 60069.740086 80.794522 -69.504084 13.426 12.654
9 S1HP081366 4658078056254368128 4658078056254368128 60069.753592 60069.759620 80.758291 -69.524143 12.765 11.899
10 S1HP082164 4658077953064455552 4658077957439332608 60069.764246 60069.770278 80.865554 -69.514107 12.753 11.871