{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Small overview of HoloViz capability of data exploration" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook is intended to present a small overview of HoloViz and the capability for data exploration, with interactive plots (show difference between matplotlib and bokeh). Many parts are based on or copied from the official [HoloViz Tutorial](https://holoviz.org/tutorial/index.html) (highly recommended for a more extensive overview of the possibilities of HoloViz).\n", "\n", "Note: In June 2019 the project name changed from [PyViz](https://pyviz.org/) to [HoloViz](https://holoviz.org/). The reason for this is explained in this [blog post](http://blog.pyviz.org/pyviz-holoviz.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## HoloViz Packages used for this notebook\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exploring Pandas Dataframes\n", "\n", "If your data is in a Pandas dataframe, it's natural to explore it using the ``.plot()`` method (based on Matplotlib). Let's have a look at some automatic weather station data from Langenferner:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "url = 'https://cluster.klima.uni-bremen.de/~oggm/tutorials/aws_data_Langenferner_UTC+2.csv'\n", "df = pd.read_csv(url, index_col=0, parse_dates=True)\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just calling ``.plot()`` won't give anything meaningful, because of the different magnitudes of the parameters:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.plot();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course we can have a look at one variable only:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.TEMP.plot();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This creates a static plot using matplotlib. With this approach we also can make some further explorations, like calculating the monthly mean temperature:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dfm = df.resample('m').mean()\n", "dfm.TEMP.plot();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see the course of the parameter but we can not tell what was the exact temperature at January and we also cannot zoom in." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exploring Data with hvPlot and Bokeh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we are using ```hvplot``` instead we can create interactive plots with the same plotting API:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "you might need to install first [hvplot](https://hvplot.holoviz.org/getting_started/index.html) via e.g. conda install -c pyviz hvplot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import hvplot.pandas\n", "\n", "df.TEMP.hvplot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now you have an interactive plot using bokeh with zooming option and hover with additional information (get the exact values and timestamps), also possible for all variables but again not very meaningful:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot = df.hvplot()\n", "plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But at least you can use your mouse to hover over each variable and explore their values. Furthermore, by clicking on the legend the colors can be switched on/off. Still, different magnitudes make it hard to see all parameters at once.\n", "\n", "Here the interactive features are provided by the [Bokeh](http://bokeh.pydata.org) JavaScript-based plotting library. But what's actually returned by this call is a overlay of something called a [HoloViews](http://holoviews.org) object, here specifically a HoloViews [Curve](http://holoviews.org/reference/elements/bokeh/Curve.html). HoloViews objects *display* as a Bokeh plot, but they are actually much richer objects that make it easy to capture your understanding as you explore the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(plot)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This object can be converted to a `HoloMap` object (using the HoloViews Package and declare bokeh to use for plotting) to create a widget that can be used to select the variables from." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import holoviews as hv\n", "hv.extension('bokeh')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "holo_plot = hv.HoloMap(plot)\n", "print(holo_plot)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "holo_plot.opts(width=700, height=500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But first have a look at the HoloViews Objects." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## HoloViews Objects" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a simple HoloViews Object:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "xs = np.arange(-10, 10.5, 0.5)\n", "ys = 100 - xs**2\n", "df_xy = pd.DataFrame(dict(x=xs, y=ys))\n", "\n", "simple_curve = hv.Curve(df_xy, 'x', 'y')\n", "print(simple_curve)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "``:Curve [x] (y)`` is HoloViews's shorthand for saying that the data in ``df_xy`` is a set of samples from a continuous function ``y`` of one independent variable ``x``, and ``simple_curve`` simply pairs your dataframe ``df_xy`` with this semantic declaration.\n", "\n", "Once we've captured this crucial bit of metadata, HoloViews now knows enough about this object to represent it graphically, as it will do by default in a Jupyter notebook:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "simple_curve" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This Bokeh plot is much more convenient to examine than a column of numbers, because it conveys the entire set of data in a compact, easily appreciated, interactively explorable format. HoloViews knew that a continuous curve like this is the right representation for what would otherwise be just a table of numbers, because we explicitly declared the element type as ``hv.Curve``. Crucially, ``simple_curve`` itself is not a plot, it's just a simple wrapper around your data that happens to have a convenient graphical representation. The full dataframe will always be available as ``simple_curve.data``, for any numerical computations you would like to do:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "simple_curve.data.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, with HoloViews you don't have to select between plotting your data and working with it numerically. Any HoloViews object will let you do *both* conveniently; you can simply choose whatever representation is the most appropriate way to approach the task you are doing. This approach is very different from a traditional plotting program, where the objects you create (e.g. a Matplotlib figure or a native Bokeh plot) are a dead end from an analysis perspective, useful only for plotting. \n", "### HoloViews Elements\n", "\n", "Holoview objects merge the visualization with the data. For an Holoview object you have to classify what the data is showing. A Holoview object could be initialised in several ways: \n", "\n", "```\n", "hv.Element(data, kdims=None, vdims=None, **kwargs)\n", "```\n", "\n", "This standard signature consists of the same five types of information:\n", "\n", "- **``Element``**: any of the dozens of element types shown in the [reference gallery](http://holoviews.org/reference/index.html).\n", "- **``data``**: your data in one of a number of formats described below, such as tabular dataframes or multidimensional gridded Xarray or Numpy arrays.\n", "- **``kdims``**: \"key dimension(s)\", also called independent variables or index dimensions in other contexts---the values for which your data was measured.\n", "- **``vdims``**: \"value dimension(s)\", also called dependent variables or measurements---what was measured or recorded for each value of the key dimensions. \n", "- **``kwargs``**: optional keyword arguments specific to that ``Element`` type (rarely needed).\n", "\n", "Elements could be for example ```Curve```, ``` Scatter```, ```Area``` and also different ways of declaring the key dimension(s) and value dimension(s) are shown below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(hv.Curve(df_xy, kdims=('x','x_label'), vdims=('y','y_label')) + \n", " hv.Scatter((xs,ys)).redim.label(x='x_label', y='ylabel') + \n", " hv.Area({'x':xs,'y':ys}))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The example also shows two ways of labeling the variables, one is directly by the initialisation with tuples ```('x','x_label')``` and ```('y','y_label')``` and a other option is to use ```.redim.label()```.\n", "\n", "The example above also shows the simple syntax to create a layout of different Holoview Objects by using `+`. With `*` you can simply overlay the objects in one plot:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from holoviews import opts\n", "\n", "(hv.Curve(df_xy, 'x', 'y') * \n", " hv.VLine(5).opts(color='black') * \n", " hv.HLine(75).opts(color='red'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With ```.opts()``` you can change some characteristics of the Holoview Objects and you can use the `[tab]` key completion to see, what options are available or you can use the ```hv.help()``` function to get more information about some `Elements`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# hv.help(hv.Curve)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So now we can use some Holoview object for the data exploration for the glacier data. We create a `Layout` with some subplots for the different parameters. With ```opts.defaults()``` we can change some default properties of the different HoloView Elements, here we activate the ```hover``` tool for all ```Curve``` elements. Try to zoom into one plot!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "opts.defaults(opts.Curve(tools=['hover']))\n", "\n", "(hv.Curve(df, 'index', 'TEMP') + \n", " hv.Curve(df,'index','RH') + \n", " hv.Curve(df,'index','SWIN').opts(color='darkorange') * hv.Curve(df,'index','SWOUT').opts(color='red') +\n", " hv.Curve(df,'index','LWIN').opts(color='darkorange') * hv.Curve(df,'index','LWOUT').opts(color='red') +\n", " hv.Curve(df,'index','WINDSPEED') + \n", " hv.Curve(df,'index','WINDDIR')).cols(3).opts(opts.Curve(width=300, height=200))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So here we created a ```Curve``` Element for some Parameters and put them together in subplots by using `+` and overlay some in one subplot with `*`. With ```.opts()``` I define the color of some parameters and set the ```width``` and ```height``` propertie for the used ```Curve``` Elements and with ```.cols()``` I define the number of columns. \n", "\n", "Now we can zoom in and use a hover for data exploration and because all Holoview Objects using the same dataframe and the same key variable the x-axes of all plots are linked. So when you zoom in in one plot all the other plots are zoomed in as well.\n", "\n", "### HoloView Dataset and HoloMap Objects\n", "\n", "A HoloViews `Dataset` is similar to a HoloViews Element, without having any specific metadata that lets it visualize itself. A Dataset is useful for specifying a set of `Dimension`s that apply to the data, which will later be inherited by any visualizable elements that you create from the Dataset.\n", "\n", "A HoloViews Dimension is the same concept as a **dependent** or **independent** variable in mathematics. In HoloViews such variables are called value dimensions and key dimensions (respectively). So lets take again our glacier pandas DataFrame and create a HoloView `Dataset`. Beforehand we define some new columns for the date. Then we create our HoloView DataFrame with the key variables (independent) ``month``, ``year`` and ``day_hour``. The remaining columns will automatically be inferred to be value (dependent) dimensions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df['month'] = df.index.month\n", "df['year'] = df.index.year\n", "df['day_hour'] = df.index.day + df.index.hour/24\n", "df['timestamp'] = df.index.strftime('%d.%m.%Y %H:%M')\n", "\n", "df_month = hv.Dataset(df, ['month', 'year', 'day_hour'])\n", "df_month = df_month.redim.label(day_hour='day of month')\n", "df_month" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Out of this Dataset we now can create a `Holomap` with ``.to``. The ``.to`` method of a Dataset takes up to four main arguments:\n", "\n", "1. The element you want to convert to\n", "2. The key dimensions (i.e., independent variables) to display\n", "3. The dependent variables to display, if any\n", "4. The dimensions to group by, if nothing given the remaining key variables are used" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "slider = df_month.to(hv.Curve, ['day_hour'], ['TEMP', 'RH', 'SWIN', 'timestamp'])\n", "slider = slider.opts(width=600, height=400, tools=['hover'])\n", "print(slider)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now created a ``HoloMap`` with to grouped variables ``[month, year]``, one key variable ``[day_hour]`` and five dependent variables ``(TEMP, RH, SWIN, WINDSPEED, timestamp)``. Now look at the visualisation (some months/year pairs are missing and cannot be visualized):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "slider" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that a widget was created where we can choose the ``'month'`` and the ``'year'`` (the two grouped variables). The plot is showing the `'day_hour'` (key) variable against the first dependent variable ``'TEMP'``. The other dependent variables are not shown but their values are displayed in the hover.\n", "\n", "For a better comparison we also can look at grouped variables at once when we use ``.overlay()``:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "overlay = df_month.to(hv.Curve, ['day_hour'], ['TEMP', 'RH', 'SWIN','WINDSPEED','timestamp']).overlay()\n", "overlay = overlay.opts(width=800, height=500, tools=['hover'])\n", "overlay.opts(opts.NdOverlay(legend_muted=True, legend_position='left'))\n", "print(overlay)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we are creating an ``NdOverlay`` Object which is similar to a ``HoloMap``, but has a different visualisation:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "overlay" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here now no widget is created, instead there is a interactive legend where we can turn the color *on* by clicking in the legend on it. So we can compare the months with each other (for example the same month in different years).\n", "\n", "It is also easy to look at some mean values, for example looking at mean diurnal values for each month and year you can use ```.aggregate```, which combine the values after the given function:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df['hour'] = df.index.hour\n", "df_mean = hv.Dataset(df, ['month', 'year', 'hour']).aggregate(function=np.mean)\n", "df_mean = df_mean.redim.label(hour='hour of the day')\n", "print(df_mean)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```.aggregate()``` uses the key variables and looks where all of them are the same. It uses the provided function (in the case above ```np.mean```) to calculate new values. So in the above case we calculate mean daily cycles for each month and year. The calculated ``Dataset`` then can be displayed as we have seen it above." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "slider = df_mean.to(hv.Curve, ['hour'], ['TEMP', 'RH', 'SWIN']).opts(width=600, height=400, tools=['hover'])\n", "print(slider)\n", "slider" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "overlay = df_mean.to(hv.Curve, ['hour'], ['TEMP', 'RH', 'SWIN']).opts(width=600, height=400, tools=['hover']).overlay()\n", "overlay.opts( opts.NdOverlay(legend_muted=True, legend_position='left'))\n", "print(overlay)\n", "overlay" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using GeoView for displaying geographical data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a small example for using geoview I want to show how to display a shapefiles of glaciers in an interactive plot." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import geoviews as gv\n", "import geopandas as gpd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "you might have to install [geoviews](https://geoviews.org/index.html#installation) for that!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tile sources\n", "\n", "Tile sources are very convenient ways to provide geographic context for a plot and they will be familiar from the popular mapping services like Google Maps and Openstreetmap. The ``WMTS`` element provides an easy way to include such a tile source in your visualization simply by passing it a valid URL template. GeoViews provides a number of useful tile sources in the ``gv.tile_sources`` module:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import geoviews.tile_sources as gts\n", "\n", "layout = gv.Layout([ts.relabel(name) for name, ts in gts.tile_sources.items()])\n", "layout.opts('WMTS', xaxis=None, yaxis=None, width=225, height=225).cols(4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To read the shape file geopandas could be used:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from oggm import utils\n", "europe_glacier = gpd.read_file(utils.get_rgi_region_file('11', version='61'))\n", "hintereisferner = europe_glacier[europe_glacier.Name == 'Hintereisferner'].geometry.iloc[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then create a GeoViews Object with a GeoViews Element Shape, display it and put a ``gv.tile_sources`` in the background. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# hv.help(gv.Shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(gv.Shape(hintereisferner).opts(fill_color=None) *\n", " gts.tile_sources['EsriImagery']).opts(width=800, height=500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The GeoViews Object and Element is similar to HoloViews Objects and Elements for geographical data. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(gv.Shape(hintereisferner))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And so similar a visualisation is stored for each GeoView Element, which can be used like an HoloView Object. So as a last example you also can plot all European glaciers in one interactive plot by using an Polygons Element of GeoViews:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(gv.Polygons(europe_glacier.geometry) * \n", " gts.tile_sources['StamenTerrain']).opts(width=800, height=500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So this only was a very small look at the capability of HoloViz for data exploration and visualisation. There are much more you can do with HoloViz, but I think it is a package you should have a look at, because with only a few lines of code you can create an interactive plot which allow you to have an quick but also deep look at your data. I really recommend to visit the official [HoloViz Tutorial](https://holoviz.org/tutorial/index.html) and start using HoloViz :)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What's next?\n", "\n", "- return to the [OGGM documentation](https://docs.oggm.org)\n", "- back to the [table of contents](../welcome.ipynb)" ] } ], "metadata": { "hide_input": false, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.9" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }