jlstevens

2019-11-20 (Last Updated: 2024-04-11)

Download project

Run notebook

Open app(s)

IEX Trading#

IEX Stocks#

In the previous notebook we saw how all the trades on the IEX stock exchange could be interactively visualized over the course of a whole day (Monday 21st of October 2019). Using datashader, all the trades are rasterized interactively to reveal the density of trades via a colormap.

When viewing a million trades at once for a whole day, it is extremely difficult to identify individual trades using a global view. In order to identify particular trades, it is necessary to zoom into a time window small enough that individual trades can be distinguished at which point the trade metadata can be inspected using the Bokeh hover tool.

What the global visualization helps reveal is the overall pattern of trades. In this notebook we will focus on interactively revealing the trading patterns for individual stocks by partitioning on a set of stock symbols selected with a widget.

Loading the data#

First we will load the data as before, converting the integer timestamps into the correctly offset datetimes before counting the total number of events:

import warnings
warnings.simplefilter('ignore')

import datetime
import pandas as pd
df = pd.read_csv('./data/IEX_2019-10-21.csv')
df.timestamp = df.timestamp.astype('datetime64[ns]')
df.timestamp -= datetime.timedelta(hours=4)
print('Dataframe loaded containing %d events' % len(df))

Dataframe loaded containing 1222412 events

Next we will identify the top ten most traded stocks on this day and compute how much of the trading volume (i.e summed over the `size column) that they account for:

symbol_volumes = df.groupby('symbol')['size'].sum()
top_symbol_volumes = symbol_volumes.sort_values()[-10:]
top_symbols = list(top_symbol_volumes.index)
top_symbol_info = (', '.join(top_symbols),
                   top_symbol_volumes.sum() /symbol_volumes.sum()  * 100)
'The top ten symbols are %s and account for %.2f%% of trading volume' % top_symbol_info

'The top ten symbols are PINS, CHK, SNAP, NIO, MDR, TEVA, HPE, BAC, GE, INFY and account for 10.24% of trading volume'

The following dictionary below show the names of the companies that each of these ten symbols correspond to:

symbol_info = {
    "PInterest":'PINS',
    'Chesapeake Energy Corporation': 'CHK',
    "Snap Inc": 'SNAP',
    "NIO Inc": 'NIO',
    "McDermott International": 'MDR',
    "Teva Pharmaceutical Industries": 'TEVA',
    "Hewlett Packard Enterprise":'HPE',
    "Bank of America": 'BAC',
    "GE": 'General Electric',
    "Infosys":'INFY',
    }

Before we can visualize each of these stocks individually, we will need to import the necessary libraries:

import holoviews as hv
from bokeh.models import HoverTool
import datashader as ds

from holoviews.operation.datashader import spikes_aggregate
hv.config.image_rtol = 10e-3 # Suppresses datetime issue at high zoom level
hv.extension('bokeh')

Visualizing trade volume by stock#

As in the previous notebook, we will create a Spikes element containing our entire dataset. Instead of immediately rasterizing it, we will be selecting individual stocks from it and rasterizing those components individually.

Note: If you display the spikes object at this time, it will probably freeze or crash your browser tab!

spikes = hv.Spikes(df, ['timestamp'], ['symbol', 'size', 'price'])

Visualizing two stocks at once#

In order to understand to build a fairly complex, interactive visualization, it is useful to build a simple prototype to identify the necessary concepts and decide whether it will satisfy our goals. In this section, we will prototype a fixed view that will let us directly compare the trading patterns for the top two stocks (PINS and CHK).

We start by defining some options called raster_opts used to customize the rasterized output of the visualize_symbol_raster function. We will use the responsive=True option to make our rasters fill the screen:

raster_opts = hv.opts.Image(min_height=400, responsive=True,
                            colorbar=True, cmap='blues', xrotation=90,
                            default_tools=['xwheel_zoom', 'xpan', 'xbox_zoom'])

def visualize_symbol_raster(symbol, offset):
    selection = spikes.select(symbol=symbol)
    return spikes_aggregate(selection, offset=offset,
                            aggregator=ds.sum('size'),
                            spike_length=1).opts(raster_opts)

In the visualize_symbol_raster function, the .select method on our spikes object is used to select only the spikes that match the symbol specified in the argument. This function also take an integer offset argument that offsets the rasterized Image vertically by one unit (the spikes are unit length as specified with spike_length=1).

One other difference from the previous notebook is that now a datashader aggregator over the 'size' column is used in order to visualize the trade volume as opposed to the trade count.

Now we can use this function with our two chosen stock symbols (PINS and CHK) to create an overlay. Lastly, we want to use the y-axis to label these stocks so we use a custom yticks option and set the ylabel.

overlay = visualize_symbol_raster('PINS', 0) * visualize_symbol_raster('CHK', 1)
overlay.opts(yticks=[(0.5, 'PINS'), (1.5, 'CHK')], ylabel='Stock Symbol')

We now want to generalize this example in the following ways:

  1. We wish to choose from any of the top ten stocks with a widget.

  2. We want to reveal the stock metadata with the Bokeh hover tool in the same way as the previous notebook.

The next section will demonstrate one way this can be done.

Visualizing the top stocks interactively#

Our prototype is generalized in three steps:

  1. First the hover behavior is reintroduced per symbol.

  2. Next the process of overlaying the visualizations for the different symbols is generalized.

  3. Finally panel is used to add an interactive widget to select from the top ten stocks.

Adding interactive hover for metadata#

To enable the desired hover behavior, we shall create a custom Bokeh hover tool that formats the ‘Symbol’, ‘Price’ and ‘Timestamp’ columns of the dataframe nicely. In addition, a simple RangeX stream is declared:

hover = HoverTool(tooltips=[
    ('Symbol', '@symbol'),
    ('Size', '@size'),
    ('Price', '@price'),
    ('Timestamp', '@timestamp{%F %H:%M %Ss %3Nms}')],
                  formatters={'timestamp': 'datetime'})

range_stream = hv.streams.RangeX()

Note that in this instance, the source of the RangeX stream is not defined upon construction: we will be setting it dynamically later. The xrange_filter function, however, is the same as the previous notebook (with the number of allowed hover spikes lowered to 200):

def xrange_filter(spikes, x_range):
    low, high = (None, None) if x_range is None else x_range
    ranged = spikes[pd.to_datetime(low):pd.to_datetime(high)]
    return (ranged if len(ranged) < 200 else ranged.iloc[:0]).opts(spike_length=1, alpha=0)

The next function, visualize_symbol builds on the approach used in visualize_symbol_raster above by overlaying each raster with the appropriate x-range filtered (invisible) Spikes in order to enable hovering.

This is done using the same approach as the previous notebook where we use the apply method on the spikes to apply the xrange_filter function. Note that as the symbol argument changes, the Spikes object returned by the select method also changes. This is why we need to set the source on our stream dynamically.

In addition, to keep everything consistent, we want to use our single range_stream everywhere, including in the DynamicMap returned by spikes_aggregate. This is done by passing range_stream explicitly in the streams argument. This approach of using a single RangeX and setting the source ensures that you can zoom in and then select a different set of stocks to be displayed without resetting the zoom level.

Lastly, we need to pass expand=False to prevent datashader from filling the whole y-range (with NaN colors where there is no data) for each raster generated.

def visualize_symbol(symbol, offset):
    selection = spikes.select(symbol=symbol)
    range_stream.source = selection
    rasterized = spikes_aggregate(selection, streams=[range_stream],
                                  offset=offset, expand=False,
                                  aggregator=ds.sum('size'),
                                  spike_length=1).opts(raster_opts)
    filtered = selection.apply(xrange_filter, streams=[range_stream])
    return rasterized * filtered.opts(tools=[hover], position=offset)

This visualize_symbol function simply adds hover behavior to visualize_symbol_raster: you can now use the former to visualize the PINS and CHK stocks in exactly the same way as was demonstrated above.

Building a dynamic overlay of stocks#

The following overlay_symbols function is a trivial generalization of the prototype that overlays an arbitrary list of stocks according to their symbols. Each DynamicMap returned by visualize_symbol is collected into an Overlay and the corresponding yticks plot option is dynamically generated.

The only new concept is the call to .collate() which is necessary to convert an Overlay container of DynamicMaps into a DynamicMap of Overlays as required by the supported nesting hierarchy.

def overlay_symbols(symbols):
    items = []
    for offset, symbol in enumerate(symbols):
        items.append(visualize_symbol(symbol, offset))
    yticks = [(i+0.5,sym) for i,sym in enumerate(symbols)]
    return hv.Overlay(items).collate().opts(
        yticks=yticks, yaxis=True).redim(y='Stock Symbol')

The prototype example could now be replicated (with hover) by calling overlay_symbols(['PINS', 'CHK']).

Adding a selector widget with panel#

Using the panel library we can easily declare a cross-selector widget specified with the symbol_info dictionary, with our two top stocks selected by default:

import panel as pn
cross_selector = pn.widgets.CrossSelector(options=symbol_info,
                                          sizing_mode='stretch_width',
                                          value=['PINS','CHK'])

Now we can wrap our overlay_symbols function in visualize_symbols and decorate it with @pn.depends before displaying both the widgets and visualization in a panel Column:

@pn.depends(cross_selector)
def visualize_symbols(symbols):
    return overlay_symbols(symbols)

stock_selector = pn.Column(cross_selector, visualize_symbols)

We now have a handle called stock_selector on a visualization that allows you to zoom in to any time during the day and view the metadata for the selected stocks (once sufficiently zoomed in).

As a final step, we can build a small dashboard by adding the IEX logo and a short Markdown description to stock_selector and declaring it servable().

dashboard_info = ('This dashboard allows exploration of the top ten stocks by volume '
                  'on the [IEX exchange](https://iextrading.com/) on Monday 21st '
                  'of October 2019. To view the metadata of individual trades, '
                  'enable the Bokeh hover tool and zoom in until you can '
                  'view individual trades.')
pn.Column(
    pn.pane.SVG('./assets/IEX_Group_Logo.svg', width=100),
    dashboard_info, stock_selector).servable()

WARNING:param.dynamic_operation: Callable raised "TypeError("The DType <class 'numpy.dtype[datetime64]'> could not be promoted by <class 'numpy.dtype[float64]'>. This means that no common DType exists for the given inputs. For example they cannot be stored in a single array unless the dtype is `object`. The full list of DTypes is: (<class 'numpy.dtype[datetime64]'>, <class 'numpy.dtype[float64]'>)")".
Invoked as dynamic_operation(x_range=(nan, nan))

WARNING:param.dynamic_operation: Callable raised "TypeError("The DType <class 'numpy.dtype[datetime64]'> could not be promoted by <class 'numpy.dtype[float64]'>. This means that no common DType exists for the given inputs. For example they cannot be stored in a single array unless the dtype is `object`. The full list of DTypes is: (<class 'numpy.dtype[datetime64]'>, <class 'numpy.dtype[float64]'>)")".
Invoked as dynamic_operation(x_range=(nan, nan))

Traceback (most recent call last):
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/plotting/util.py", line 280, in get_plot_frame
    return map_obj[key]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 550, in __call__
    if not args and not kwargs and not any(kwarg_hash): return self.callable()
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1021, in resolve
    return key, map_obj[key]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 550, in __call__
    if not args and not kwargs and not any(kwarg_hash): return self.callable()
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1021, in resolve
    return key, map_obj[key]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 550, in __call__
    if not args and not kwargs and not any(kwarg_hash): return self.callable()
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1021, in resolve
    return key, map_obj[key]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 550, in __call__
    if not args and not kwargs and not any(kwarg_hash): return self.callable()
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 199, in dynamic_mul
    self_el = self.select(HoloMap, **key_map) if self.kdims else self[()]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 550, in __call__
    if not args and not kwargs and not any(kwarg_hash): return self.callable()
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 199, in dynamic_mul
    self_el = self.select(HoloMap, **key_map) if self.kdims else self[()]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 580, in __call__
    ret = self.callable(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1021, in resolve
    return key, map_obj[key]
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1213, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 980, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/spaces.py", line 580, in __call__
    ret = self.callable(*args, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1033, in dynamic_operation
    return apply(obj, *key, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1025, in apply
    processed = self._process(element, key, kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1007, in _process
    return self.p.operation.process_element(element, key, **kwargs)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/operation.py", line 194, in process_element
    return self._apply(element, key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/core/operation.py", line 141, in _apply
    ret = self._process(element, key)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/operation/datashader.py", line 589, in _process
    info = self._get_sampling(element, x, y, ndim=1, default=default)
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/holoviews/operation/resample.py", line 138, in _get_sampling
    x_range = (np.nanmin([np.nanmax([x0, ex0]), ex1]),
  File "<__array_function__ internals>", line 5, in nanmax
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/numpy/lib/nanfunctions.py", line 441, in nanmax
    res = np.amax(a, axis=axis, out=out, **kwargs)
  File "<__array_function__ internals>", line 5, in amax
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 2754, in amax
    return _wrapreduction(a, np.maximum, 'max', axis, None, out,
  File "/home/runner/work/examples/examples/iex_trading/envs/default/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
TypeError: The DType <class 'numpy.dtype[datetime64]'> could not be promoted by <class 'numpy.dtype[float64]'>. This means that no common DType exists for the given inputs. For example they cannot be stored in a single array unless the dtype is `object`. The full list of DTypes is: (<class 'numpy.dtype[datetime64]'>, <class 'numpy.dtype[float64]'>)

This dashboard can now be served using panel serve IEX_stock.ipynb.

Conclusion#

In this notebook, we have seen how a trade-by-trade stock explorer for arbitrarily large datasets can be built up incrementally using three of the HoloViz tools, namely by using the HoloViews Datashader API and Panel for the widgets.