{ "cells": [ { "cell_type": "markdown", "id": "9fdd3e66", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "# SWESARR Tutorial" ] }, { "cell_type": "markdown", "id": "56f3be28", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "![NASA](http://www.nasa.gov/sites/all/themes/custom/nasatwo/images/nasa-logo.svg)\n", "\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "4188e5b9-a513-4b1a-8bc8-9d27ba9eee0b", "metadata": { "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "source": [ "
\n", "Objectives: \n", " This is a 30-minute tutorial where we will ...\n", "
    \n", "
  1. Introduce SWESARR
    2. \n", "
  2. Briefly introduce active and passive microwave remote sensing
    3. \n", "
  3. Learn how to access, filter, and visualize SWESARR data
    4. \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "63bda652-ba87-4029-8b90-62fbb517a6e7", "metadata": {}, "source": [ "## Quick References\n", "\n", "
    \n", "
  1. SWESARR SAR Data Pre-release FTP Server \n", "
  2. SWESARR Radiometer Data, SnowEx20, v1\n", "
  3. NSIDC Datasets\n", "
  4. SWESARR Blogspot\n", "
  5. A Version of This Repo That Doesn't Require Access to Amazon Servers \n", "
" ] }, { "cell_type": "markdown", "id": "ece33f6d-f565-4bcc-b056-79874e84f45c", "metadata": {}, "source": [ "## What is SWESARR?\n", "\n", "

\n", "SWESARR gives us insights on the different ways active and passive signals are influenced by snow over large areas." ] }, { "cell_type": "markdown", "id": "2ce50fa9-17de-4718-b463-086d4a1b3173", "metadata": {}, "source": [ "## Active and Passive? Microwave Remote Sensing?" ] }, { "cell_type": "markdown", "id": "1fc2036c-5b3c-457b-a9a8-b991892ac54b", "metadata": {}, "source": [ "\n", "### Passive Systems\n", "\n", "* All materials can naturally emit electromagnetic waves\n", "* What is the cause?\n", "\n", "

\n", "\n", "
\n", "

\n", "\n", "* Material above zero Kelvin will display some vibration or movement of particles\n", "* These moving, charged particles will induce electromagnetic waves\n", "* If we're careful, we can measure these waves with a radio wave measuring tool, or \"radiometer\"\n", "
\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "* Radiometers see emissions from many sources, but they're usually very weak\n", "* It's important to design a radiometer that (1) minimizes side lobes and (2) allows for averaging over the main beam\n", "* For this reason, radiometers often have low spatial resolution\n", "\n", "**✏️ Radiometers allow us to study earth materials through incoherent averaging of naturally emitted signals**\n", "\n", "### Active Systems\n", "* While radiometers generally measure natural electromagnetic waves, radars measure man-made electromagnetic waves\n", "* Transmit your own wave, and listen for the returns\n", "* The return of this signal is dependent on the surface and volume characteristics of the material it contacts\n", "\n", "
\n", "\n", "
\n", "\n", "**✏️ Synthetic aperture radar allows for high spatial resolution through processing of coherent signals**" ] }, { "cell_type": "code", "execution_count": null, "id": "bb939cd3-32f5-47ab-9bd2-e0d08256ea26", "metadata": {}, "outputs": [], "source": [ "%%HTML\n", "" ] }, { "cell_type": "markdown", "id": "54df5103-646f-4ab8-831c-585181100275", "metadata": {}, "source": [ "## SWESARR Sensors\n", "
\n", "

SWESARR Frequencies, Polarization, and Bandwidth Specification

\n", "
\n", "\n", "| Center-Frequency (GHz) | Band | Sensor | Bandwidth (MHz) | Polarization |\n", "| ---------------------- | ---------- | ------------ | --------------- | ------------ |\n", "| 9.65 | X | SAR | 200 | VH and VV |\n", "| 10.65 | X | Radiometer | 200 | H |\n", "| 13.6 | Ku | SAR | 200 | VH and VV |\n", "| 17.25 | Ku | SAR | 200 | VH and VV |\n", "| 18.7 | K | Radiometer | 200 | H |\n", "| 36.5 | Ka | Radiometer | 1,000 | H |\n", "\n", "
\n", "
\n", "
\n", " \n", "
\n" ] }, { "cell_type": "markdown", "id": "6974a849-5b7b-4deb-b699-926ad4a8d184", "metadata": {}, "source": [ "## SWESARR Coverage\n", "\n", "\n", "\n", "* Below: radiometer coverage for all passes made between February 10 to February 12, 2020\n", "* SWESARR flights cover many snowpit locations over the Grand Mesa area as shown by the dots in blue\n", "\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "df320f07-895c-45be-9859-706516fd9cea", "metadata": {}, "source": [ "## Reading SWESARR Data\n", "\n", "- SWESARR's SAR data is organized with a common file naming convention for finding the time, location, and type of data\n", "- [Lets look at the prerelease data on its homepage](https://glihtdata.gsfc.nasa.gov/files/radar/SWESARR/prerelease/)\n", "
\n", "\n", "
\n", "\n", "***\n", "

" ] }, { "cell_type": "markdown", "id": "c5d5a065-efbf-4735-b514-b259f8ea2822", "metadata": {}, "source": [ "
\n", "

\n", "Accessing Data: SAR\n", "

\n", "
\n", "\n", "### SAR Data Example" ] }, { "cell_type": "code", "execution_count": null, "id": "7bf5bc1a-e9eb-4813-a4aa-5205a8762ebe", "metadata": {}, "outputs": [], "source": [ "# Import several libraries. \n", "# comments to the right could be useful for local installation on Windows.\n", "\n", "from shapely import speedups # https://www.lfd.uci.edu/~gohlke/pythonlibs/\n", "speedups.disable() # <-- handle a potential error in cartopy\n", "\n", "import requests # !conda install -c anaconda requests \n", "\n", "# raster manipulation libraries\n", "import rasterio # https://www.lfd.uci.edu/~gohlke/pythonlibs/\n", "from osgeo import gdal # https://www.lfd.uci.edu/~gohlke/pythonlibs/\n", "import cartopy.crs as ccrs # https://www.lfd.uci.edu/~gohlke/pythonlibs/\n", "import rioxarray as rxr # !conda install -c conda-forge rioxarray\n", "import xarray as xr # !conda install -c conda-forge xarray dask netCDF4 bottleneck\n", "\n", "# plotting tools\n", "from matplotlib import pyplot # !conda install matplotlib\n", "import datashader as ds # https://www.lfd.uci.edu/~gohlke/pythonlibs/\n", "import hvplot.xarray # !conda install hvplot\n", "\n", "# append the subfolders of the current working directory to pythons path\n", "\n", "import os\n", "import sys\n", "\n", "swesarr_subdirs = [\"data\", \"util\"]\n", "tmp = [sys.path.append(os.getcwd() + \"/\" + sd) for sd in swesarr_subdirs]\n", "del tmp # suppress Jupyter notebook output, delete variable\n", "\n", "from helper import gdal_corners, join_files, join_sar_radiom" ] }, { "cell_type": "markdown", "id": "9f066718-3181-4645-92dd-7d55d612cc03", "metadata": {}, "source": [ "#### Select your data" ] }, { "cell_type": "code", "execution_count": null, "id": "f36402f2-a63f-4d3d-b874-da53ef328c59", "metadata": {}, "outputs": [], "source": [ "# select files to download\n", "\n", "# SWESARR data website\n", "source_repo = 'https://glihtdata.gsfc.nasa.gov/files/radar/SWESARR/prerelease/'\n", "\n", "# Example flight line\n", "flight_line = 'GRMCT2_31801_20007_016_200211_225_XX_01/'\n", "\n", "# SAR files within this folder\n", "data_files = [\n", " 'GRMCT2_31801_20007_016_200211_09225VV_XX_01.tif',\n", " 'GRMCT2_31801_20007_016_200211_09225VH_XX_01.tif',\n", " 'GRMCT2_31801_20007_016_200211_13225VV_XX_01.tif',\n", " 'GRMCT2_31801_20007_016_200211_13225VH_XX_01.tif',\n", " 'GRMCT2_31801_20007_016_200211_17225VV_XX_01.tif',\n", " 'GRMCT2_31801_20007_016_200211_17225VH_XX_01.tif'\n", "]\n", "\n", "# store the location of the SAR tiles as they're located on the SWESARR data server\n", "remote_tiles = [source_repo + flight_line + d for d in data_files]\n", "\n", "# create local output data directory\n", "output_dir = os.getcwd() + '/data/'\n", "try:\n", " os.makedirs(output_dir)\n", "except FileExistsError:\n", " print('output directory prepared!')\n", "\n", "# store individual TIF files locally on our computer / server\n", "output_paths = [output_dir + d for d in data_files]" ] }, { "cell_type": "markdown", "id": "324d06be-dece-4f4f-bd0e-e207f0028142", "metadata": {}, "source": [ "#### Download SAR data and place into data folder" ] }, { "cell_type": "code", "execution_count": null, "id": "d4c6cd77-0841-4e66-8199-f6e463a856de", "metadata": {}, "outputs": [], "source": [ "## for each file selected, store the data locally \n", "##\n", "## only run this block if you want to store data on the current \n", "## server/hard drive this notebook is located.\n", "##\n", "################################################################\n", "\n", "for remote_tile, output_path in zip(remote_tiles, output_paths):\n", " \n", " # download data\n", " r = requests.get(remote_tile)\n", "\n", " # Store data (~= 65 MB/file)\n", " if r.status_code == 200:\n", " with open(output_path, 'wb') as f:\n", " f.write(r.content)" ] }, { "cell_type": "markdown", "id": "782a372e-471e-48e3-af7e-688f0873c3a1", "metadata": {}, "source": [ "#### Merge SAR datasets into single xarray file" ] }, { "cell_type": "code", "execution_count": null, "id": "2833b2ee-9460-46ab-8f3b-884e89a91a85", "metadata": {}, "outputs": [], "source": [ "da = join_files(output_paths)\n", "da" ] }, { "cell_type": "markdown", "id": "1fbf46f0-6a7c-4b3e-a18e-edfd7e3f7f02", "metadata": {}, "source": [ "#### Plot data with hvplot" ] }, { "cell_type": "code", "execution_count": null, "id": "6e61bac1-6721-453c-89fb-cafa72ba25c2", "metadata": {}, "outputs": [], "source": [ "# Set clim directly:\n", "clim=(-20,20)\n", "cmap='gray'\n", "crs = ccrs.UTM(zone='12') #12n\n", "tiles='OSM'\n", "tiles='EsriImagery'\n", "\n", "da.hvplot.image(x='x',y='y',groupby='band',cmap=cmap,clim=clim,rasterize=True,\n", " xlabel='Longitude',ylabel='Latitude',\n", " frame_height=500, frame_width=500,\n", " xformatter='%.1f',yformatter='%.1f', crs=crs, tiles=tiles, alpha=0.8)" ] }, { "cell_type": "markdown", "id": "f45328d8-7d96-4cf3-bf0c-be2033d835d0", "metadata": {}, "source": [ "**🎉 Congratulations! You now know how to download and display a SWESARR SAR dataset !**" ] }, { "cell_type": "markdown", "id": "04764272-38b1-437a-b194-1bfea9ff535c", "metadata": {}, "source": [ "### Radiometer Data Example\n", "* SWESARR's radiometer data is publicly available at NSIDC\n", " * Will be available on [SnowExSQL if the fans demand it!](https://github.com/SnowEx/snowexsql/search?q=swesarr)\n", "* [Radiometer Data v1 Available Here](https://nsidc.org/data/SNEX20_SWESARR_TB/versions/1)" ] }, { "cell_type": "code", "execution_count": null, "id": "493a7452-cc73-4224-9237-ed038d7fa5da", "metadata": {}, "outputs": [], "source": [ "import pandas as pd # !conda install pandas\n", "import numpy as np # !conda install numpy\n", "import xarray as xr # !conda install -c anaconda xarray \n", "\n", "import hvplot # !conda install hvplot\n", "import hvplot.pandas\n", "import holoviews as hv # !conda install -c conda-forge holoviews \n", "from holoviews.operation.datashader import datashade\n", "from geopy.distance import distance #!conda install -c conda-forge geopy " ] }, { "cell_type": "markdown", "id": "3d9430bc-3eec-440c-81a0-b6940e94d09c", "metadata": {}, "source": [ "#### Downloading SWESARR Radiometer Data with `wget`\n", "\n", "* If you are running this on the SnowEx Hackweek server, `wget` should be configured.\n", "* If you are using this tutorial on your local machine, you'll need `wget`.\n", " * Linux Users\n", " - You should be fine. This is likely baked into your operating systems. Congratulations! You chose correctly.\n", " * Apple Users\n", " - The author of this textbox has never used a Mac. There are many command-line suggestions online. `sudo brew install wget`, `sudo port install wget`, etc. Try searching online!\n", " * Windows Users\n", " - [Check out this tutorial, page 2](https://blogs.nasa.gov/swesarr/wp-content/uploads/sites/305/2020/10/how_to_download_SWESARR_radar_data.pdf) You'll need to download binaries for `wget`, and you should really make it an environment variable!\n", " \n", "Be sure to be diligent before installing anything to your computer.\n", " \n", "Regardless, fill in your NASA Earthdata Login credentials and follow along!" ] }, { "cell_type": "code", "execution_count": null, "id": "0f29d818-7abc-4ebe-a14a-f067e23b1486", "metadata": {}, "outputs": [], "source": [ "# !wget --user=USERNAME_HERE --password=PASSWORD_HERE --quiet https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX20_SWESARR_TB.001/2020.02.12/SNEX20_SWESARR_TB_GRMCT2_13801_20007_000_200211_XKKa225H_v01.csv -O {output_dir}/SNEX20_SWESARR_TB_GRMCT2_13801_20007_000_200211_XKKa225H_v01.csv" ] }, { "cell_type": "code", "execution_count": null, "id": "4a2dc074-19cb-42bd-9c3f-fed14442429e", "metadata": {}, "outputs": [], "source": [ "# !wget --quiet https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX20_SWESARR_TB.001/2020.02.12/SNEX20_SWESARR_TB_GRMCT2_13801_20007_000_200211_XKKa225H_v01.csv -O {output_dir}/SNEX20_SWESARR_TB_GRMCT2_13801_20007_000_200211_XKKa225H_v01.csv" ] }, { "cell_type": "markdown", "id": "5a5be5aa-e652-4bea-8bb4-8831209a2b55", "metadata": {}, "source": [ "#### Select an example radiometer data file" ] }, { "cell_type": "code", "execution_count": null, "id": "3ba1a0b6-b1f5-4f60-ab99-57444f08b9c4", "metadata": {}, "outputs": [], "source": [ "# use the file we downloaded with wget above\n", "excel_path = f'{output_dir}/SNEX20_SWESARR_TB_GRMCT2_13801_20007_000_200211_XKKa225H_v01.csv'\n", "\n", "# read data\n", "radiom = pd.read_csv(excel_path)" ] }, { "cell_type": "markdown", "id": "c7157153-9328-463f-ba4e-5d1c00584b49", "metadata": {}, "source": [ "#### Lets examine the radiometer data files content" ] }, { "cell_type": "code", "execution_count": null, "id": "bda67928-c05f-4648-a11b-8403f7c78eb8", "metadata": {}, "outputs": [], "source": [ "radiom.head()\n", "#radiom.hvplot.table(width=1100) # sortable table in jupyterlab" ] }, { "cell_type": "markdown", "id": "0297d88d-a757-4141-ad60-7118b35d4c85", "metadata": {}, "source": [ "#### Plot radiometer data with hvplot" ] }, { "cell_type": "code", "execution_count": null, "id": "750dcc7d-6a05-4036-8241-546b691a3dae", "metadata": {}, "outputs": [], "source": [ "# create several series from pandas dataframe\n", "\n", "lon_ser = pd.Series( radiom['Longitude (deg)'].to_list() * (3) )\n", "lat_ser = pd.Series( radiom['Latitude (deg)'].to_list() * (3) )\n", "\n", "tb_ser = pd.Series(\n", " radiom['TB X (K)'].to_list() + radiom['TB K (K)'].to_list() + \n", " radiom['TB Ka (K)'].to_list(), name=\"Tb\"\n", " )\n", "\n", "# get series length, create IDs for plotting\n", "sl = len(radiom['TB X (K)'])\n", "id_ser = pd.Series(\n", " ['X-band']*sl + ['K-band']*sl + ['Ka-band']*sl, name=\"ID\"\n", " )\n", "\n", "frame = {'Longitude (deg)' : lon_ser, 'Latitude (deg)' : lat_ser,\n", " 'TB' : tb_ser, 'ID' : id_ser}\n", "radiom_p = pd.DataFrame(frame)\n", "\n", "del sl, lon_ser, lat_ser, tb_ser, id_ser, frame\n", "\n", "radiom_p.hvplot.points('Longitude (deg)', 'Latitude (deg)', groupby='ID', geo=True, color='TB', alpha=1,\n", " tiles='ESRI', height=400, width=500)" ] }, { "cell_type": "markdown", "id": "9a7c80c3-3c7e-4f31-997d-22b71853af54", "metadata": {}, "source": [ "**🎉 Congratulations! You now know how to download and display a SWESARR radiometer dataset !**" ] }, { "cell_type": "markdown", "id": "b3d0d1df-05f0-4b61-bd4b-560b1c826037", "metadata": {}, "source": [ "## SAR and Radiometer Together" ] }, { "cell_type": "markdown", "id": "21fbbd01-a1ae-4fcd-ad8d-6127010ab703", "metadata": {}, "source": [ "* The novelty of SWESARR lies in its colocated SAR and radiometer systems\n", "* Lets try filtering the SAR dataset and plotting both datasets together\n", "* For this session, I've made the code a function. We can look at it together by opening `.util/helper.py`" ] }, { "cell_type": "code", "execution_count": null, "id": "b0855fa7-762b-40ab-afab-84d76e2070a7", "metadata": {}, "outputs": [], "source": [ "data_p, data_ser = join_sar_radiom(da, radiom)\n", "\n", "data_p.hvplot.points('Longitude (deg)', 'Latitude (deg)', groupby='ID', geo=True, color='Measurements', alpha=1,\n", " tiles='ESRI', height=400, width=500)" ] }, { "cell_type": "markdown", "id": "275c601d-079b-40ef-b172-9bc6f1147b1a", "metadata": {}, "source": [ "## Exercise" ] }, { "cell_type": "markdown", "id": "9ad1a774-9994-4705-b9e0-fbfd7342b518", "metadata": {}, "source": [ "
\n", "Exercise: \n", "
    \n", "
  1. Plot a time-series visualization of the filtered SAR channels from the output of the \n", " join_sar_radiom() function\n", "
    2. \n", "
  2. Plot a time-series visualization of the radiometer channels from the output of the \n", " join_sar_radiom() function \n", "
    3. \n", "
  3. Hint: the data series variable ( data_ser ) is a pandas data series. \n", " Use some of the methods shown above to read and plot the data!\n", "
    4. \n", "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "ea283a7a-28fc-42fc-ab64-f6c9aab2da6b", "metadata": {}, "outputs": [], "source": [ "\n", "### Your Code Here #############################################################################################################\n", "#\n", "# Two of Many Options:\n", "# 1.) Go the matplotlib route\n", "# a.) Further reading below:\n", "# https://matplotlib.org/stable/tutorials/introductory/pyplot.html\n", "#\n", "# 2.) Try using hvplot tools if you like\n", "# a.) Further reading below:\n", "# https://hvplot.holoviz.org/user_guide/Plotting.html\n", "#\n", "# Remember, if you don't use a library all of the time, you'll end up -ing it. Go crazy!\n", "#\n", "################################################################################################################################\n", "\n", "# configure some inline parameters to make things pretty / readable if you'd like to go with matplotlib\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.rcParams[\"figure.figsize\"] = (16, 9) # (w, h)" ] }, { "cell_type": "markdown", "id": "8276b584-0564-4054-b2ce-f4a8c385e2da", "metadata": {}, "source": [ "


\n", "## [Warnings](#Table-of-Contents)\n", "
\n", "Interpreting Data: After the 2019 and 2020 measurement periods for SWESARR, an internal timing error was found in the flight data which affects the spatial precision of the measurements. While we are working to correct this geospatial error, please consider this offset before drawing conclusions from SWESARR data if you are using a dataset prior to this correction. The SWESARR website will announce the update of the geospatially corrected dataset.\n", "
\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.5" } }, "nbformat": 4, "nbformat_minor": 5 }