<![CDATA[SWIRLL - Severe Weather Institute - Radar & Lightning laboratory]]>https://swirll.km4yhi.com/https://swirll.km4yhi.com/favicon.pngSWIRLL - Severe Weather Institute - Radar & Lightning laboratoryhttps://swirll.km4yhi.com/Ghost 3.27Mon, 21 Dec 2020 19:39:35 GMT60<![CDATA[Hurricane Laura Deployment]]>An example IOP post.

Laura made landfall at Cameron, LA as a category 4 hurricane on Thursday, August 27th, 2020 at ≈ 12:10 A.M. CDT / 0510z. Catastrophic damage occurred along the coast near Cameron, as well as inland toward Lake Charles and beyond. The UAH MIPS and MAX measurement

]]>
https://swirll.km4yhi.com/iop-laura/5f49b63075bdd4000157d0d3Sat, 29 Aug 2020 02:07:43 GMTAn example IOP post.

Laura made landfall at Cameron, LA as a category 4 hurricane on Thursday, August 27th, 2020 at ≈ 12:10 A.M. CDT / 0510z. Catastrophic damage occurred along the coast near Cameron, as well as inland toward Lake Charles and beyond. The UAH MIPS and MAX measurement systems were deployed just north of Lake Charles, LA throughout the event and observed gusts of 104 mph and 87 mph, respectively.

Below are pictures and videos taken by Nick Perlaky at various times during the event.


Setup and Science Goals


Pre-Arrival


Eye Wall

Video from Lake Charles, LA During the Eye Wall of Hurricane Laura

The eye wall of Laura was not as intense as expected at the UAH MIPS location. We observed a maximum wind gust of 104 mphtime?. Sustained winds varied between 50 and 75 mph during the eye wall. Most gusts peaked around 90 mph. These measurements were taken at ≈3m AGL, as our 10m telescoping mast would not have survived the higher wind gusts.

Small branches and pieces of tin roofing blew about the MIPS location during and after the eywall.

KLCH WSR-88D Failure

Just before 1:00 A.M. CDT, the MIPS team observed several bright blue flashes on the southern horizon. These were thought to be the cause of lost data from KLCH, though later imagery showed complete destruction of the radome and dish.

Here are the final 30 scans from KLCH on 2020-08-27:


Post-Event

Many wave-like features were observed in the hours after storm passage. (Perhaps Tim can speak to this?)

Some medium-voltage power transmission poles were snapped at their midpoint or their base

Area Damage

]]>
<![CDATA[Technical Information]]>How is this site set up?

During development, all components of this site will be hosted on my personal web server.

Each individual component of the system (Ghost, RShiny, QuestDB, Apache) is containerized with Docker, allowing easy generation of new services. When the time comes to transfer the site to

]]>
https://swirll.km4yhi.com/tech-info/5f34454d3111b6000113f1f0Wed, 12 Aug 2020 19:49:52 GMTHow is this site set up?

During development, all components of this site will be hosted on my personal web server.

Each individual component of the system (Ghost, RShiny, QuestDB, Apache) is containerized with Docker, allowing easy generation of new services. When the time comes to transfer the site to an NSSTC/UAH system, data can be easily exported to the new containers.

My server employs HAProxy as a reverse-proxy and load-balancer (currently there is only one node). I use Letsencrypt as my HTTPS certificate authority.

It should be noted that creating posts with the titles "API" or "Data" will render them inaccessible, as HAProxy will prevent their defualt URIs from ever getting to Ghost.


Recommendations for Code

As you write code which uses links to the RESTful API and other resources in this system, I would recommend using #definitions or some sort of variables to define the domain name (swirll.km4yhi.com).

When services are transferred to NSSTC/UAH systems, the domain name will become something like "nsstc.uah.edu/swirll" or "swirll.edu" (swirll.edu is my personal preference, but budgetary concerns may prohibit this change).


Notes for NSSTC/UAH IT

This system will likely need to become operational very rapidly in order to support operations in the Fall 2020 semester. Here are some of my thoughts about transitioning away from my server.

AWS

If IT feels comfortable running these Docker containers on Vortex (or other), I have no qualms with that method.

My personal recommendation—simply to ease the setup process for IT—would be to host this entire system in AWS, as there will be fairly high storage requirements. Distancing the system from Vortex would also eliminate any potential network security issues related to remote uploads.

Containers

Here is a list of the necessary containers for IT's reference:

  • Ghost (content management system at the root domain) - 1 port
  • Apache/PHP (RESTful API and file server... this can potentially be hosted directly on the Vortex web server... not sure if this is Apache or Nginx) - 1 TCP port
  • RShiny (live data views) - 2 TCP ports... must account for websockets in the proxy
  • QuestDB (time series database) - likely only 1 TCP port
  • I may potentially run an RStudio cloud instance - 1 TCP port with websockets
  • I run this on a personal server, but would like to have one or two instances on UAH hardware if possible: an AWIPS/CAVE cloud instance - also 1 TCP port with websockets

I am working to create custom containers for each of these services. These containers will be available on my server, but I can push them to Docker Hub or Github if needed.

Each container will need at least one directory mounted on the host, some more. I am still working out how data will be imported into new containers if one fails. This decision will likely be dependent upon how IT would like to handle storage and backups.

Network

Note: I am not implying that the container ports should be exposed to the Internet. Only Vortex itself should be able to access these ports (i.e. only accept traffic on these ports from the localhost). Traffic to the Internet will be secured over HTTPS on the standard port 443.

At this time, I am only running HTTP and websocket traffic within this system, so I do not presently see a need to run a TCP-level proxy (i.e. HAProxy or Nginx, though I assume one of these solutions is already employed on Vortex or some other load-balancing machine).

I will provide my rewrite rules from my HAProxy configuration. Several rewrites are needed to handle path proxying (i.e. /api/* or /data/* directing to a different server than /insert-document-name). Note: path proxying relies on the condition that we will run under the "nsstc.uah.edu/swirll" domain. If we acquire "swirll.edu" or similar, simpler proxying methods may be employed by using subdomains (data.swirll.edu, api.swirll.edu, etc.).


Other Notes

At some point, we will need to create a method of uploading data to this system from platforms in the field... in real time. We would also like to host several cameras around the region which will also upload imagery in real time.

My personal preference is to use the REST API and simple Token-based authentication. We could simply write files without execution permissions to avoid accidental malware injection. I may be missing other security concerns with this method, but it's my first thought.


Contact

I can be reached at nick.perlaky@uah.edu anytime. I am fine to meet remotely on any platform if needed.

]]>
<![CDATA[Development Information]]>General information about this platform:
  • Posts will hold data from each IOP.
  • Pages will hold information about instruments.

More information will be added over the next few days. Here is a permalink to this post: https://swirll.km4yhi.com/dev-info/.


URLs:

/ghost points to the administration panel

/api will point

]]>
https://swirll.km4yhi.com/dev-info/5f2dace5e877e70001f883f6Fri, 07 Aug 2020 19:35:31 GMTGeneral information about this platform:
  • Posts will hold data from each IOP.
  • Pages will hold information about instruments.

More information will be added over the next few days. Here is a permalink to this post: https://swirll.km4yhi.com/dev-info/.


URLs:

/ghost points to the administration panel

/api will point to the data retrieval system (QuestDB for the Berm & surface stations, PHP scripts for NetCDF file and image retrieval). Users will generally not interact with this system directly unless they need scripted/programmatic access (authenticated).

/data will point to the data view and retrieval interface. This dashboard will be written in R, making use of the RShinyDashboard platform. Python can also be used to generate imagery.


Using the Administration Interface

This website primarily makes use of the Ghost CMS (content management system). Imagine WordPress, but with less clutter.

Posts and Pages are created and published using the admin interface at /ghost. The web editor makes use of Markdown syntax (tutorial), allowing rapid and painless development. The editor also allows direct HTML entry, so custom elements may be developed (e.g. the sounding "slideshow" view).

*Note: the site is currently using a predeveloped theme. We can change things like font and layout at our leisure, or select an entirely different theme.


Using the Database

I chose QuestDB to support time-series datasets with minimal effort. This database is accessible at the following endpoints:

  • /api/imp -> import data into the DB
  • /api/exec -> run queries and administrative commands (returns JSON)
  • /api/exp -> export results as csv (queries can be run directly here)

This database will hold data from the KUAH ASOS and any other platform which can store its data in a text-based/csv format (csv, JSON, tsv, etc.). Examples include roaming surface stations, radiosondes, and ceilometers.

Large datasets such as RADAR imagery are not appropriate for this database, as they add uncessary load to servers and are generally stored in a custom binary format. Similar platforms—some of which store data in a NetCDF format—will be accessible through my custom REST API (see below).


Using the PHP API

I am developing a RESTful API for large file and dataset retrieval. This API will rely on PHP scripts to search for files. Queries will require the following variables: platform type/id, desired measurements, start time, end time, and authorization token.

The API will be accessible at the /api/data endpoint.

A preliminary example query to this API:

In Plain English:

0.1 degree tilt reflectivity from MAX, starting on March 3rd, 2020 at 2100z and ending on March 4th, 2020 at 1000z. Authorization token is 1234567890

RESTful API, using curl:

curl -G "https://swirll.edu/api/data" -H "Authorization: Basic 1234567890" --data-urlencode "platform=max&product=0.1degtilt&start=2020-03-03T21:00:00&end=2020-03-04T10:00:00"

This query will return appropriate file types for each platform. Times will be matched to the closest available time within the reqested period; if no data is available within the requested time period, an error will be returned in JSON-formatted plaintext.

Requests containing incorrect platform/product identifiers will also return an error in JSON.


Automated Post Generation

The Ghost CMS has a RESTful API for programmatic administration of posts and pages. My intention with this API is to automatically create posts for each IOP in real time.

In other words, as fielded platforms collect and upload data to Tornado(?), scripts will create "quicklook" imagery (including soundings) which will be automatically pushed to this site. Live views will also be available in the /data dashboard.


Contact

If you need assistance with the interface, send me an email at any time: nick.perlaky@uah.edu.

]]>