Development Information

General information about this platform:

Posts will hold data from each IOP.
Pages will hold information about instruments.

More information will be added over the next few days. Here is a permalink to this post: https://swirll.km4yhi.com/dev-info/.

URLs:

/ghost points to the administration panel

/api will point to the data retrieval system (QuestDB for the Berm & surface stations, PHP scripts for NetCDF file and image retrieval). Users will generally not interact with this system directly unless they need scripted/programmatic access (authenticated).

/data will point to the data view and retrieval interface. This dashboard will be written in R, making use of the RShinyDashboard platform. Python can also be used to generate imagery.

Using the Administration Interface

This website primarily makes use of the Ghost CMS (content management system). Imagine WordPress, but with less clutter.

Posts and Pages are created and published using the admin interface at /ghost. The web editor makes use of Markdown syntax (tutorial), allowing rapid and painless development. The editor also allows direct HTML entry, so custom elements may be developed (e.g. the sounding "slideshow" view).

*Note: the site is currently using a predeveloped theme. We can change things like font and layout at our leisure, or select an entirely different theme.

Using the Database

I chose QuestDB to support time-series datasets with minimal effort. This database is accessible at the following endpoints:

/api/imp -> import data into the DB
/api/exec -> run queries and administrative commands (returns JSON)
/api/exp -> export results as csv (queries can be run directly here)

This database will hold data from the KUAH ASOS and any other platform which can store its data in a text-based/csv format (csv, JSON, tsv, etc.). Examples include roaming surface stations, radiosondes, and ceilometers.

Large datasets such as RADAR imagery are not appropriate for this database, as they add uncessary load to servers and are generally stored in a custom binary format. Similar platforms—some of which store data in a NetCDF format—will be accessible through my custom REST API (see below).

Using the PHP API

I am developing a RESTful API for large file and dataset retrieval. This API will rely on PHP scripts to search for files. Queries will require the following variables: platform type/id, desired measurements, start time, end time, and authorization token.

The API will be accessible at the /api/data endpoint.

A preliminary example query to this API:

In Plain English:

0.1 degree tilt reflectivity from MAX, starting on March 3rd, 2020 at 2100z and ending on March 4th, 2020 at 1000z. Authorization token is 1234567890

RESTful API, using curl:

curl -G "https://swirll.edu/api/data" -H "Authorization: Basic 1234567890" --data-urlencode "platform=max&product=0.1degtilt&start=2020-03-03T21:00:00&end=2020-03-04T10:00:00"

This query will return appropriate file types for each platform. Times will be matched to the closest available time within the reqested period; if no data is available within the requested time period, an error will be returned in JSON-formatted plaintext.

Requests containing incorrect platform/product identifiers will also return an error in JSON.

Automated Post Generation

The Ghost CMS has a RESTful API for programmatic administration of posts and pages. My intention with this API is to automatically create posts for each IOP in real time.

In other words, as fielded platforms collect and upload data to Tornado(?), scripts will create "quicklook" imagery (including soundings) which will be automatically pushed to this site. Live views will also be available in the /data dashboard.

Contact

If you need assistance with the interface, send me an email at any time: nick.perlaky@uah.edu.