VegaFusion Blog - Posts in Release

VegaFusion 2.0

2024-11-13T00:00:00-05:00

A new era: Building blocks for analyzing, accelerating, and scaling Vega visualizations:

By: Jon Mease

The VegaFusion team is excited to announce the release of VegaFusion 2.0. This major release represents a significant evolution in VegaFusion’s architecture and target audience. The focus has been on simplification - making VegaFusion easier to maintain and easier to integrate into larger Vega systems to accelerate and scale Vega visualizations.

Modernized Rust Core

The core Rust implementation has been significantly simplified by taking advantage of modern DataFusion functionality:

New get_column_usage() utility function for analyzing the column dependencies of datasets in a Vega spec.
Custom User Defined Functions (UDFs) have been largely replaced with DataFusion’s native capabilities.
Vega transforms are now implemented directly against the DataFusion DataFrame API, rather than using the circuitous SQL generation approach of VegaFusion 1.
Updated to DataFusion 43 and Arrow 52.3 for improved performance and bug fixes.

JavaScript / WebAssembly

The JavaScript integration has been simplified while maintaining full functionality:

Removed the separate vegafusion-embed package as the vegafusion-wasm package now renders charts using vega-embed.
The full VegaFusion runtime is now compiled to WebAssembly as part of the vegafusion-wasm pacakge, making it possible to run VegaFusion entirely in the browser without requiring an external server.
Connecting to external VegaFusion servers via gRPC-Web remains supported.

Python: Removal of Upstreamed Vega-Altair Integration

All the VegaFusion 1.x logic for integrating with Vega-Altair has been upstreamed into the Vega-Altair project itself. So this functionality has been removed in VegaFusion 2.0.

In particular:

To use VegaFusion with Vega-Altair, activate the "vegafusion" data transformer.
The vegafusion-jupyter package has been retired in favor of using Altair’s built-in JupyterChart for Altair charts and Vega-Lite specs. For Vega specs, the new AnyWidget-based VegaFusionWidget is included in the vegafusion package.
vf.enable() has been removed. Instead, use the "vegafusion" data transformer along with any of Altair’s built-in renderers
vf.save() has been removed. Instead, use Altair’s native chart.save() method with the "vegafusion" data transformer enabeld.
vf.transformed_data() has been removed. Instead, use Altair’s native chart.transformed_data() method.

Python: Enhanced DataFrame Compatibility

VegaFusion 2.0 brings improved DataFrame compatibility:

Support for any Narwhals-compatible DataFrame type as inline datasets.
Zero-copy sharing of Arrow data from Polars to VegaFusion’s DataFusion runtime is now supported thanks to the arro3 project..
The DuckDB SQL connection has been removed, but DuckDB relations are now directly supported as inline datasets, with automatic column pruning prior to conversion to Arrow.
Reduced Python dependencies. VegaFusion now requires only arrow3-core, packaging, and narwhals. In particular, pyarrow and pandas are no longer required dependencies.

New Documentation

VegaFusion 2.0 launches with completely revamped documentation:

New PyData Sphinx theme at vegafusion.io
Standalone examples in Python and Rust
Comprehensive support tables for Vega transforms and expressions

Learn more

Check out these resources to learn more:

VegaFusion 1.4

2023-08-21T00:00:00-04:00

Improved Vega coverage, external data source foundations, extended architecture support

By: Jon Mease

The VegaFusion team is happy to announce the release of version 1.4. Along with the usual bug fixes and updates to the core Arrow and DataFusion dependencies, this release improves coverage of Vega’s features by supporting q1/q3 aggregation functions and bitwise operators. It also lays important foundations for supporting external data sources and compute engines, and adds additional architectures for pip and conda packages.

Improved Vega coverage

VegaFusion 1.4 adds support for the q1 and q3 aggregation functions. This makes it possible for VegaFusion to evaluate all the transforms associated with a Vega-Lite boxplot. Here’s a Vega-Altair example:

import altair as alt
from vega_datasets import data
import vegafusion as vf
vf.enable()

source = data.cars()

chart = alt.Chart(source).mark_boxplot(extent="min-max").encode(
    alt.X("Miles_per_Gallon:Q").scale(zero=False),
    alt.Y("Origin:N"),
)
chart

An easy way to see that the transforms are supported is to extract the transformed data with vf.transformed_data.

vf.transformed_data(chart)

	Origin	lower_box_Miles_per_Gallon	upper_box_Miles_per_Gallon	mid_box_Miles_per_Gallon	lower_whisker_Miles_per_Gallon	upper_whisker_Miles_per_Gallon
0	USA	15	24	18.5	9	39
1	Europe	24	30.65	26.5	16.2	44.3
2	Japan	25.7	34.05	31.6	18	46.6

In addition, the full complement of bitwise operators are now supported in the Vega expression language including |, &, ^, <<, and >>.

External Data Source Foundations

VegaFusion 1.4 lays some important foundations toward the goal of supporting external data sources and compute engines. The vegafusion.dataset.sql.SqlDataset abstract class defines the interface for implementing SQL data sources in Python for any of VegaFusion’s 10 supported SQL dialects. Implementations for DuckDB and Snowpark are also provided. In addition, the vegafusion.dataset.DataFrameDataset abstract class defines the interface for implementing VegaFusion’s data transformations with external DataFrame libraries. Motivating examples include the future ability to dispatch VegaFusion data transformations to the Ibis and Polars Python libraries.

In the coming release of Vega-Altair 5.1, it will be possible to pass implementations of SqlDataset and DataFrameDataset to Altair Chart objects instead of pandas DataFrames. Stay tuned for more information and examples after the release of Altair 5.1!

Extended Architecture Support

VegaFusion wheels are now built and published to PyPI for the aarch64 Linux architecture. In addition, conda-forge packages are now published for the Apple Silicon architecture.

Updates to Arrow and DataFusion dependencies

VegaFusion 1.4 updates the dependency on arrow-rs to version 42.0.0 and DataFusion to version 27.0.0.

Looking ahead

The coming release of Vega-Altair 5.1 will include first-class integration with VegaFusion to support extracting transformed data from a chart with chart.transformed_data(). It will also include a "vegafusion" data transformer that will cause Altair to use VegaFusion to pre-evaluate data transformations and remove unused columns when saving or displaying charts. The timeline has not been decided on yet, but the plan is to eventually deprecate VegaFusion’s transformed_data and save_* functions and the VegaFusion mime renderer in favor of the integrations built into Altair.

Another near-term focus is on lowering the barrier to contributing to VegaFusion by adopting the Pixi environment manager.

Learn more

Check out these resources to learn more:

VegaFusion 1.3

2023-06-10T00:00:00-04:00

Transformed data for compound charts, Java library

By: Jon Mease

The VegaFusion team is happy to announce the release of version 1.3. Along with usual bug fixes and updates to the core Arrow and DataFusion dependencies, this release includes an enhancement to the transformed_data function to make it compatible with compound charts, adds DuckDB 0.8 support, and adds a new Java API.

Extract transformed data from compound Altair charts

The VegaFusion transformed_data function can be used to extract the transformed data from an Altair chart object as a pandas DataFrame. Prior to 1.3.0, transformed_data was only supported on simple charts that involve only a single mark. Now, compound layered and concatenated charts are supported as well. In the case of a compound chart, transformed_data will return a list of pandas DataFrame’s rather than single DataFrame.

Example: Histogram with a Global Mean Overlay

Here is an example based on the Histogram with a Global Mean Overlay example from the Altair gallery.

import altair as alt
import vegafusion as vf
from vega_datasets import data

vf.enable()

source = data.movies.url

base = alt.Chart(source)

bar = base.mark_bar().encode(
    alt.X('IMDB_Rating:Q').bin().axis(None),
    y='count()'
)

rule = base.mark_rule(color='red').encode(
    x='mean(IMDB_Rating):Q',
    size=alt.value(5)
)

chart = bar + rule
chart

Because chart is composed of two subcharts, the result of vf.transformed_data will be a list of two DataFrames.

[hist_df, rule_df] = vf.transformed_data(chart)
hist_df

|    |   bin_maxbins_10_IMDB_Rating |   bin_maxbins_10_IMDB_Rating_end |   __count |
|---:|-----------------------------:|---------------------------------:|----------:|
|  0 |                            6 |                                7 |       985 |
|  1 |                            3 |                                4 |       100 |
|  2 |                            7 |                                8 |       741 |
|  3 |                            5 |                                6 |       633 |
|  4 |                            8 |                                9 |       204 |
|  5 |                            2 |                                3 |        43 |
|  6 |                            4 |                                5 |       273 |
|  7 |                            9 |                               10 |         4 |
|  8 |                            1 |                                2 |         5 |

rule_df

|    |   mean_IMDB_Rating |
|---:|-------------------:|
|  0 |            6.28347 |

DuckDB 0.8 Support

VegaFusion’s DuckDB integration has been updated to support DuckDB 0.8.0

Java API

VegaFusion 1.3 may now be embedded in Java applications using the new io.vegafusion.vegafusion jar, which is published to Maven Central.

This was accomplished by wrapping the VegaFusion Rust API with a JNI interface using the jni-rs crate.

Updates to Arrow and DataFusion dependencies

VegaFusion 1.3 updates the dependency on arrow-rs to version 39.0 and DataFusion to version 25.0.

Learn more

Check out these resources to learn more:

VegaFusion 1.2

2023-04-12T00:00:00-04:00

Pre-transform and Save Altair Charts

By: Jon Mease

The VegaFusion team is happy to announce the release of version 1.2. Along with usual bug fixes and updates to the core Arrow and DataFusion dependencies, this release includes support for saving pre-transformed Altair charts to HTML, JSON, PNG, and SVG files.

Standard Altair save

Altair Charts provide a Chart.save() method that may be used to save Altair charts to HTML, JSON, or static image (PNG or SVG) files. When the chart references a pandas DataFrame, the full DataFrame is serialized to JSON and included in the chart specification that is saved. As dataset sizes get larger, the time it takes to save these files increases rapidly, and for the case of HTML and JSON formats, the file size increases rapidly as well.

VegaFusion’s new save functions improve the situation by pre-applying data transformations and removing unused columns before inlining the resulting data in the chart specification for saving.

HTML performance comparison

Here are two examples of the benefits of the new save_html function compared to the standard Altair Chart.save method.

With aggregations

Let’s save a one million row histogram chart to an HTML file. First create and display the histogram with the VegaFusion Mime Renderer enabled.

import pandas as pd
import altair as alt
import vegafusion as vf
vf.enable()

flights = pd.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/flights_1m.parquet"
)

delay_hist = alt.Chart(flights).mark_bar().encode(
    alt.X("delay", bin=alt.Bin(maxbins=30)),
    alt.Y("count()")
)
delay_hist

Next, save the chart to an HTML file with the standard Altair Chart.save method.

%%time
delay_hist.save("delay_hist_standard.html")

CPU times: user 5.72 s, sys: 388 ms, total: 6.11 s
Wall time: 6.18 s

This results in a ~116MB file, as all columns from the entire million row dataset are included in the HTML file. It also takes over 6 seconds to perform the save. Now, use VegaFusion’s save_html function.

%%time
vf.save_html(delay_hist, "delay_hist_vf.html")

CPU times: user 227 ms, sys: 119 ms, total: 347 ms
Wall time: 427 ms

The resulting file is now only ~5KB, because only ~30 rows have been included (one per histogram bin). It also takes under half a second to perform the save.

Without aggregations

The benefits of VegaFusion’s new save functions are most dramatic for charts that use aggregations. Even so, VegaFusion’s ability to remove unused columns still results in smaller file sizes for unaggregated charts, especially when the input datasets have many unused columns.

Here’s an example scatter chart using the movies dataset from the vega-datasets package, which has 3201 rows and 16 columns.

import vegafusion as vf
import altair as alt
from vega_datasets import data

source = data.movies()

scatter_chart = alt.Chart(source).mark_point().encode(
    alt.X("IMDB_Rating:Q"),
    alt.Y("Rotten_Tomatoes_Rating:Q"),
)
scatter_chart

Save the chart to an HTML file with Altair’s standard Chart.save method.

%%time
scatter_chart.save("scatter_standard.html")

CPU times: user 69.7 ms, sys: 14.7 ms, total: 84.4 ms
Wall time: 85 ms

This results in a 1.4MB file and takes ~80ms.

Now, save the file with VegaFusion’s save_html function.

%%time
vf.save_html(scatter_chart, "scatter_vf.html")

CPU times: user 59.8 ms, sys: 15.2 ms, total: 75 ms
Wall time: 57.1 ms

This results in a 125K file and takes under 60ms. The file is more than 10x smaller because it only includes data for the two columns that are referenced by the scatter plot.

Updates to Arrow and DataFusion dependencies

VegaFusion 1.2 updates the dependency on arrow-rs to version 36.0 and DataFusion to version 22.0.

Learn more

Check out these resources if you’d like to learn more:

VegaFusion 1.1

2023-03-25T00:00:00-04:00

DuckDB and Polars support, Altair 5 compatibility, and lots of bug fixes

By: Jon Mease

The VegaFusion team is happy to announce the release of version 1.1. In addition to the usual slew of bug fixes, this release includes support for evaluating Vega transforms in DuckDB, support for Polars and the DataFrame Interchange Protocol, and forward compatibility with the coming release of Altair 5.

DuckDB support

DuckDB is an in-process SQL OLAP query engine and database that provides bindings for a wide variety of languages, including Python. The VegaFusion 1.1 Python library now includes two forms of integration with DuckDB. First, it can use DuckDB in place of DataFusion to power Vega transforms over the pandas DataFrames that are referenced by Altair charts. Second, VegaFusion 1.1 makes it possible to reference externally defined DuckDB tables and views in Altair charts.

Vega transforms with DuckDB

VegaFusion can now be configured to evaluate Vega transforms using the DuckDB Python library by calling vegafusion.runtime.set_connection("duckdb").

Once the DuckDB connection is enabled, pandas DataFrames referenced by Altair charts are automatically registered with DuckDB and Vega transforms are translated into DuckDB SQL queries. Here is a full example, adapted from the 2D Histogram Heatmap Altair gallery example.

import vegafusion as vf
import pandas as pd
import altair as alt

# Configure DuckDB connection
vf.runtime.set_connection("duckdb")

# Enable Mime Renderer
vf.enable()

# Load 201k row version of the Vega movies dataset with pandas
movies = pd.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/movies_201k.parquet"
)

# Create an Altair chart from the pandas DataFrame as usual.
# Binning and aggregation will be evaluated against the DataFrame with DuckDB
chart = alt.Chart(movies).mark_rect().encode(
    alt.X('IMDB_Rating:Q', bin=alt.Bin(maxbins=60)),
    alt.Y('Rotten_Tomatoes_Rating:Q', bin=alt.Bin(maxbins=40)),
    alt.Color('count():Q', scale=alt.Scale(scheme='greenblue'))
)
chart

Additionally, the transformed data can be extracted as usual:

vf.transformed_data(chart, row_limit=5).T

	0	1	2	3	4
bin_maxbins_60_IMDB_Rating	3.4	5.8	7	7	7.4
bin_maxbins_60_IMDB_Rating_end	3.6	6	7.2	7.2	7.6
bin_maxbins_40_Rotten_Tomatoes_Rating	60	25	85	80	80
bin_maxbins_40_Rotten_Tomatoes_Rating_end	65	30	90	85	85
__count	63	441	504	1260	1134

The default DataFusion connection can be re-enabled by calling vegafusion.runtime.set_connection("datafusion")

Performance benefit

DuckDB has the ability to perform queries against Pandas DataFrames without first serializing them to Arrow. This can result in 10x+ speedups when visualizing large Pandas DataFrames with the VegaFusion Mime Renderer. Presently, the VegaFusion Widget Renderer always serializes DataFrames to Arrow and writes them to disk, so significant performance gains are not expected for the Widget Renderer.

Access DuckDB tables

VegaFusion 1.1 also supports integration with external DuckDB connections, making it possible to reference DuckDB tables and views from Altair charts. To begin, import duckdb and create a new connection.

import duckdb
conn = duckdb.connect()

Pass this DuckDB connection to vegafusion.runtime.set_connection (instead of the string "duckdb" as in the previous example).

vf.runtime.set_connection(conn)

Next, use the DuckDB connection to create a table or view. Here, the DuckDB read_parquet method is used to load the 201k row movies dataset, and a DuckDB query is used to filter NULL values. The result of this query is registered as a table named movies.

relation = conn.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/movies_201k.parquet"
)

relation.query("tbl", """
    SELECT * FROM tbl 
    WHERE Rotten_Tomatoes_Rating IS NOT NULL AND Imdb_Rating IS NOT NULL
""").to_table("movies")

DuckDB tables and views registered with conn may be referenced from Altair charts with a special URL syntax using table:// as the prefix. To reference the DuckDB table named movies, the Altair chart should be passed the url string "table://movies". Here is a full example

import duckdb
import vegafusion as vf
import pandas as pd
import altair as alt

# Create DuckDB connection
conn = duckdb.connect()

# Pass DuckDB connection to VegaFusion's set_connection method
vf.runtime.set_connection(conn)

# Enable Mime Renderer
vf.enable()

# Read parquet file using the DuckDB connection
relation = conn.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/movies_201k.parquet"
)

# Filter NULL values and register the result as a table named "movies"
relation.query("tbl", """
    SELECT * FROM tbl 
    WHERE Rotten_Tomatoes_Rating IS NOT NULL AND Imdb_Rating IS NOT NULL
""").to_table("movies")

# Create an Altair chart that references the registered DuckDB table
chart = alt.Chart("table://movies").mark_rect().encode(
    alt.X('IMDB_Rating:Q', bin=alt.Bin(maxbins=60)),
    alt.Y('Rotten_Tomatoes_Rating:Q', bin=alt.Bin(maxbins=40)),
    alt.Color('count():Q', scale=alt.Scale(scheme='greenblue'))
)
chart

See the DuckDB connection docs for more information.

Polars support

Polars describes itself as a “Lightning-fast DataFrame library for Rust and Python”. Polars has quickly gained popularity as a faster alternative to pandas that also supports larger datasets.

VegaFusion’s new Polars integration makes it possible to input and output Polars DataFrames without conversion through pandas.

Polars as Input

Here is a full example that uses Polars to read the 201k row movies dataset from a remote parquet file. The DataFrame’s filter method is then used to remove rows with NULL movie rating values. This filtered DataFrame is passed as the input to an Altair chart.

import polars as pl
import vegafusion as vf
import altair as alt

# Enable Mime Renderer
vf.enable()

# Load 201k movies parquet dataset
source = pl.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/movies_201k.parquet"
)

# Fitler out rows with null ratings
source = source.filter(
    pl.col("IMDB_Rating").is_not_null() & pl.col("Rotten_Tomatoes_Rating").is_not_null()
)

# Create chart
chart = alt.Chart(source).mark_circle().encode(
    alt.X('IMDB_Rating:Q', bin=True),
    alt.Y('Rotten_Tomatoes_Rating:Q', bin=True),
    size='count()'
)
chart

Polars as Output

When a Chart that references a Polars DataFrame is passed to vegafusion.transformed_data, the result will also be a Polars DataFrame. Here is a Polars version of the example from the Transformed Data section.

import altair as alt
import vegafusion as vf
import polars a pl

# Enable mime renderer
vf.enable()

# Load 201k movies parquet dataset with Polars
source = pl.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/movies_201k.parquet"
)

# Build chart
chart = (
    alt.Chart(source)
    .transform_aggregate(
        aggregate_gross='mean(Worldwide_Gross)',
        groupby=["Director"],
    ).transform_window(
        rank='row_number()',
        sort=[alt.SortField("aggregate_gross", order="descending")],
    ).transform_calculate(
        ranked_director="datum.rank < 10 ? datum.Director : 'All Others'"
    ).mark_bar().encode(
        x=alt.X("aggregate_gross:Q", aggregate="mean", title=None),
        y=alt.Y(
            "ranked_director:N",
            sort=alt.Sort(op="mean", field="aggregate_gross", order="descending"),
            title=None,
        ),
    )
).properties(
    title="Top Directors by Average Worldwide Gross",
)
chart

Now retrieve the chart’s transformed data as a Polars DataFrame.

transformed = vf.transformed_data(chart)
print(type(transformed))
transformed

<class 'polars.internals.dataframe.frame.DataFrame'>

shape: (10, 2)

ranked_director (str)	mean_aggregate_gross (f64)
David Yates	9.37984e+08
James Cameron	8.29781e+08
Carlos Saldanha	7.69293e+08
Pete Docter	7.31305e+08
Andrew Stanton	7.00319e+08
David Slade	6.88155e+08
George Lucas	6.73577e+08
Andrew Adamson	6.43134e+08
Peter Jackson	5.95566e+08
All Others	8.87602e+07

See the Polars Integration docs for more information.

DataFrame Interchange Protocol

The “Polars as Input” workflow above is powered by a more general abstraction: The DataFrame Interchange Protocol. Along with Altair 5, VegaFusion 1.1 adds support for inputting any DataFrame object that supports this protocol including pyarrow Tables, Vaex DataFrames, cuDF DataFrames, and more.

Altair 5 Compatibility

Along with Altair 5.0.0rc1, VegaFusion 1.1 updates Vega-Lite from 4.17.0 to 5.6.1. This is a significant update that enables many new features for Altair users. For more info, see the Altair 5 docs.

Learn more

Check out these resources if you’d like to learn more:

VegaFusion 1.0

2023-01-21T00:00:00-05:00

Mime renderer, transform evaluation, and a permissive license

By: Jon Mease

The VegaFusion team is happy to announce the release of VegaFusion version 1.0. This release is the culmination of several months of work on both VegaFusion itself and its new dependency: VlConvert. This version introduces a brand new mimetype based renderer that is compatible with many notebook and compute environments without requiring custom extensions. It also includes a new transformed_data() function that makes it possible to evaluate the data transforms associated with an Altair Chart and return the result as a pandas DataFrame.

Finally, we’re thrilled to announce that VegaFusion has been adopted by Hex and is now licensed under the same permissive BSD-3 license used by Vega, Vega-Lite, and Altair.

Background

VegaFusion scales Vega, Vega-Lite, and Altair visualizations by moving data transformations from the browser to a server (e.g. a Python kernel).

The initial focus was on supporting complex interactive visualizations that require a live two-way connection between the client and server. This approach works well, but it relies on a custom Jupyter widget extension which adds complexity and limits the environments it can be deployed in.

To reduce complexity while addressing the more common scenario of non-interactive charts, VegaFusion 1.0 introduces an entirely new renderer that does not require any custom extensions but instead leverages the built-in rendering capabilities of modern notebooks and compute environments.

VegaFusion mime renderer

Altair comes with a default limit of 5,000 input rows to avoid crashing the user’s browser. Because all data transformations are applied in the browser by the Vega JavaScript library, the full input dataset must be loaded into the browser. The VegaFusion mime renderer performs most common Vega transforms in the Python kernel, and so the row limit is applied to the transformed data rather than the input data. When the chart includes any form of aggregation, the transformed dataset typically has many fewer rows than the input dataset.

To use the VegaFusion mime renderer, first install the vegafusion Python package with the embed extras enabled:

pip install -U "vegafusion[embed]"

Then open a Jupyter notebook (either the classic notebook or a notebook inside JupyterLab), and create an Altair histogram of a 1 million row flights dataset

import pandas as pd
import altair as alt

flights = pd.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/flights_1m.parquet"
)

delay_hist = alt.Chart(flights).mark_bar().encode(
    alt.X("delay", bin=alt.Bin(maxbins=30)),
    alt.Y("count()")
)
delay_hist

---------------------------------------------------------------------------
MaxRowsError                              Traceback (most recent call last)
...
MaxRowsError: The number of rows in your dataset is greater than the maximum allowed (5000). For information on how to plot larger datasets in Altair, see the documentation

As expected, this results in an Altair MaxRowsError. The VegaFusion mime renderer can be used to overcome this limitation by performing data intensive transforms (e.g. filtering, binning, aggregation, etc.) in the Python kernel before the resulting data is sent to the web browser.

Run these two lines to import and enable the new VegaFusion mime renderer:

import vegafusion as vf
vf.enable()

Now the chart displays quickly without errors:

delay_hist

Transform evaluation

VegaFusion now supports extracting the transformed data for an Altair Chart using the vegafusion.transformed_data() function. This is particularly useful when building a chart that includes a pipeline of transforms, as it’s now possible to see the intermediate results of each transform step.

Example: Top K

Here is an example, based on the Top-K plot with Others example from the Altair documentation, of how transformed_data() can be helpful when building a complex chart.

First, create an Altair Chart wrapping the data source URL.

import altair as alt
import vegafusion as vf

source = "https://cdn.jsdelivr.net/npm/vega-datasets@v1.29.0/data/movies.json"
chart = alt.Chart(source)

The transformed_data() function can be used on this empty chart to access a preview of the data that is available at the URL. Here the row_limit argument is used to limit the result to 3 rows and the DataFrame is transposed to make it easier to read.

vf.transformed_data(chart, row_limit=3).T

	0	1	2
Title	The Land Girls	First Love, Last Rites	I Married a Strange Person
US_Gross	146083	10876	203134
Worldwide_Gross	146083	10876	203134
Production_Budget	8000000	300000	250000
Release_Date	Jun 12 1998	Aug 07 1998	Aug 28 1998
MPAA_Rating	R	R
Distributor	Gramercy	Strand	Lionsgate
IMDB_Rating	6.1	6.9	6.8
IMDB_Votes	1071.0	207.0	865.0
Major_Genre		Drama	Comedy
Rotten_Tomatoes_Rating	nan	nan	nan
Source
Creative_Type
Director
US_DVD_Sales	nan	nan	nan
Running_Time_min	nan	nan	nan

The first step of making this chart is to compute the average worldwide gross of all the movies for each director. This can be accomplished with the Altair Aggregate Transform.

chart = (
    alt.Chart(source)
    .transform_aggregate(
        aggregate_gross='mean(Worldwide_Gross)',
        groupby=["Director"],
    )
)
vf.transformed_data(chart, row_limit=5)

	Director	aggregate_gross
0		3.59284e+07
1	Christopher Nolan	3.44251e+08
2	Roman Polanski	5.13407e+07
3	Richard Fleischer	2.27635e+07
4	Blake Edwards	5e+06

Next, the directors are ranked by average gross in descending order. This can be accomplished with the Altair Window Transform

chart = (
    alt.Chart(source)
    .transform_aggregate(
        aggregate_gross='mean(Worldwide_Gross)',
        groupby=["Director"],
    ).transform_window(
        rank='row_number()',
        sort=[alt.SortField("aggregate_gross", order="descending")],
    )
)
vf.transformed_data(chart, row_limit=5)

	Director	aggregate_gross	rank
0	David Yates	9.37984e+08	1
1	James Cameron	8.29781e+08	2
2	Carlos Saldanha	7.69293e+08	3
3	Pete Docter	7.31305e+08	4
4	Andrew Stanton	7.00319e+08	5

Then, a new column is added that contains the director’s name for the top 9 ranked directors and “All Others” for the remaining directors. This can be accomplished using the Altair Calculate Transform.

chart = (
    alt.Chart(source)
    .transform_aggregate(
        aggregate_gross='mean(Worldwide_Gross)',
        groupby=["Director"],
    ).transform_window(
        rank='row_number()',
        sort=[alt.SortField("aggregate_gross", order="descending")],
    ).transform_calculate(
        ranked_director="datum.rank < 10 ? datum.Director : 'All Others'"
    )
)
vf.transformed_data(chart, row_limit=12)

	Director	aggregate_gross	rank	ranked_director
0	David Yates	9.37984e+08	1	David Yates
1	James Cameron	8.29781e+08	2	James Cameron
2	Carlos Saldanha	7.69293e+08	3	Carlos Saldanha
3	Pete Docter	7.31305e+08	4	Pete Docter
4	Andrew Stanton	7.00319e+08	5	Andrew Stanton
5	David Slade	6.88155e+08	6	David Slade
6	George Lucas	6.73577e+08	7	George Lucas
7	Andrew Adamson	6.43134e+08	8	Andrew Adamson
8	Peter Jackson	5.95566e+08	9	Peter Jackson
9	Richard Marquand	5.727e+08	10	All Others
10	Eric Darnell	5.66099e+08	11	All Others
11	Roland Emmerich	4.5506e+08	12	All Others

Finally, this dataset is ready to be encoded as a bar mark:

import altair as alt
import vegafusion as vf

vf.enable()

source = "https://cdn.jsdelivr.net/npm/vega-datasets@v1.29.0/data/movies.json"
chart = (
    alt.Chart(source)
    .transform_aggregate(
        aggregate_gross='mean(Worldwide_Gross)',
        groupby=["Director"],
    ).transform_window(
        rank='row_number()',
        sort=[alt.SortField("aggregate_gross", order="descending")],
    ).transform_calculate(
        ranked_director="datum.rank < 10 ? datum.Director : 'All Others'"
    ).mark_bar().encode(
        x=alt.X("aggregate_gross:Q", aggregate="mean", title=None),
        y=alt.Y(
            "ranked_director:N",
            sort=alt.Sort(op="mean", field="aggregate_gross", order="descending"),
            title=None,
        ),
    )
)
chart

The exact value of each bar can be accessed by applying transformed_data() to the final chart (which includes the implicit transforms in the bar mark encoding).

vf.transformed_data(chart)

	ranked_director	mean_aggregate_gross
0	David Yates	9.37984e+08
1	James Cameron	8.29781e+08
2	Carlos Saldanha	7.69293e+08
3	Pete Docter	7.31305e+08
4	Andrew Stanton	7.00319e+08
5	David Slade	6.88155e+08
6	George Lucas	6.73577e+08
7	Andrew Adamson	6.43134e+08
8	Peter Jackson	5.95566e+08
9	All Others	8.87602e+07

Increased transform coverage

VegaFusion’s coverage of Vega-Lite’s transforms is not yet complete, but it’s growing with each release. Support for the Pivot, Impute, Fold, and Sequence transforms was added in this release. Here is the full set of supported Vega-Lite/Vega transforms:

Leveraging VlConvert

Both the new mime renderer and the transfomed_data() function require the ability to convert Vega-Lite specifications (as produced by Altair) to Vega specifications (as consumed by VegaFusion). This functionality is provided by the Vega-Lite JavaScript library, which presents a challenge: How can we perform this conversion in Python without a web browser?

To meet this challenge, we developed a new Python library called VlConvert. You can read all about VlConvert in its announcement post, but in short it embeds the Deno JavaScript runtime in a Python library for the purpose of running the Vega-Lite JavaScript library.

While motivated by the needs of VegaFusion, VlConvert turned out to be a great foundation for implementing static image export, as it is a simple Python package that doesn’t require external dependencies on a web browser or Node.js. In fact, VlConvert has been adopted by the Vega community and will be the default image export engine for the coming release of Altair 5!

A new partnership and a new license

VegaFusion was first released in January 2022 under the copyleft AGPL license. I chose to use the AGPL license initially because I wanted to share the work with the Vega community, but I also wanted to leave open the possibility of building a company with a dual-license business model around the project.

The day of the initial release, Barry McCardel (the CEO of Hex Technologies) reached out to me, and we started talking about how VegaFusion could help scale the Hex chart cell (which is itself built on Vega-Lite). I granted them a commercial license exception, and started working with the Hex team to integrate VegaFusion into Hex. After months of working together with Barry and the Hex team, and lots of conversations around the future of the project, we decided that the best path forward was for me to continue working on VegaFusion as a member of Hex team, and for VegaFusion to be re-licensed to a permissive license that is compatible with the rest of the Vega ecosystem.

The end result is that VegaFusion 1.0 is licensed under the BSD-3 license. Also, to make it easier to take advantage of Hex’s testing infrastructure the VegaFusion repositories have been moved under the hex-inc GitHub organization.

Learn more

Check out these resources if you’d like to learn more:

VegaFusion 0.1

2022-01-27T00:00:00-05:00

Server-side acceleration for the Vega visualization grammar with Rust and WebAssembly

By: Jon Mease

Today, I’m happy to announce a new open source project I’ve been working on since stepping down as Plotly’s Chief Scientist this past fall. It’s called VegaFusion.

VegaFusion provides server-side acceleration for the Vega visualization grammar. While not limited to Python, the initial application of VegaFusion is the acceleration of the Altair Python interface to Vega-Lite. Its design was motivated by years of experience working on visualization libraries (Plotly, HoloViews, and Datashader) and dashboard technologies (Dash, Jupyter Widgets, and Panel).

Quickstart: Accelerate Altair in Jupyter

VegaFusion can be used to provide server-side acceleration for Altair visualizations when displayed in Jupyter contexts (Classic notebook, JupyterLab, and Voila). It also eliminates the Altair MaxRowsError when generating a chart with over 5000 rows.

First, install the vegafusion-jupyter package, along with vega-datasets for the example below.

$ pip install "vegafusion-jupyter[embed]" vega-datasets

Then, open a Jupyter notebook (either the classic notebook, or a notebook inside JupyterLab), and run these two lines to import and enable the VegaFusion Altair renderer.

import vegafusion as vf
vf.jupyter.enable()

VegaFusion will now be used to accelerate any Altair chart. For example, here’s the interactive average Altair gallery example.

import altair as alt
from vega_datasets import data

source = data.seattle_weather()
brush = alt.selection(type='interval', encodings=['x'])

bars = alt.Chart().mark_bar().encode(
    x='month(date):O',
    y='mean(precipitation):Q',
    opacity=alt.condition(brush, alt.OpacityValue(1), alt.OpacityValue(0.7)),
).add_selection(
    brush
)

line = alt.Chart().mark_rule(color='firebrick').encode(
    y='mean(precipitation):Q',
    size=alt.SizeValue(3)
).transform_filter(
    brush
)

chart = alt.layer(bars, line, data=source)
chart

This browser does not support the video tag.

Histogram binning, aggregation, selection filtering, and average calculations will now be evaluated in the Python kernel process with efficient parallelization, rather than in the single-threaded browser context.

You can see that VegaFusion acceleration is working by noticing that the Python kernel is running as the selection region is created or moved. You can also notice the VegaFusion logo in the dropdown menu button.

Vega Ecosystem Background

VegaFusion is designed to complement the Vega visualization ecosystem. In particular, the Vega, Vega-Lite, and Altair projects. If you’re not familiar with these projects, it will be helpful to take a few minutes to browse their documentation as background for understanding what VegaFusion adds.

Transforms and Signals

One powerful feature of the Vega visualization grammar is that it includes a rich collection of data manipulation functions called transforms. Transforms have functionality that is similar to that provided by SQL queries or Pandas DataFrame operations, but they are specifically designed to cover data preprocessing tasks that are useful in constructing data visualizations.

For additional flexibility, Vega also provides the concept of signals. These are scalar variables that can be constructed using the Vega expression language, which is a subset of JavaScript. Transforms can accept and produce signals.

There are at least two significant advantages to having data transformations and signals included in a visualization specification.

First, it makes it possible for a visualization to accept raw data files as input and then perform its own data cleaning and manipulation. This often removes the need to generate temporary intermediary data files.
Second, it enables higher-level libraries like Vega-Lite to automate the creation of rich interactive visualizations with features like cross filtering and drill down.

Motivation for VegaFusion

Vega makes it possible to create declarative JSON specifications of rich interactive visualizations that are fully self-contained. They can run entirely in a web browser without requiring access to an external database or a Python library like Pandas.

For datasets of a few tens of thousands rows or fewer, this architecture often results in extremely smooth and responsive interactivity. However, this architecture doesn’t scale very well to datasets of hundreds of thousands of rows or more. This is the gap VegaFusion aims to fill.

How VegaFusion Works

VegaFusion has two primary components: The Planner and the Runtime.

Planner

The Planner starts with an arbitrary Vega specification (typically generated by Vega-Lite, but this is not a requirement). The Planner’s job is to partition the specification into two valid Vega specifications, one that will execute in the browser with the Vega JavaScript library, and one that will execute on the server with the VegaFusion Runtime.

VegaFusion does not (yet) provide full coverage of all of Vega’s transforms and all of the features of the Vega expression language. The planner uses information about which transforms and expression functions are supported to make decisions about which parts of the original Vega specification can be included in the resulting server specification. The advantage of this approach is that VegaFusion can accept any Vega specification, and as more support is added over time, more of the input specification will be eligible for inclusion in the server specification.

Along with the client and server specifications, the planner also creates a communication plan. The communication plan determines which datasets and signals must be passed from server to client, and from client to server in order for the interactive behavior of the original specification to be preserved.

Runtime

After planning, the server specification is compiled into a VegaFusion task graph. The job of the Runtime is to calculate the value of requested nodes within a task graph.

A task graph includes the values of the root nodes (those with no parents), but it does not include the values of any of the interior nodes (those with parents). Each node in the task graph is a pure function of the values of its parents. This enables the Runtime to calculate the value of any node in the Task graph from the specification, while keeping the overall task graph size small enough to be efficiently transferred between the client and the server. The Runtime uses fingerprinting and precise caching to avoid repeated calculations of the same nodes. The cache is “precise” in the sense that cached values can be shared across visualizations that share a common substructure, even if the full specifications are not identical.

Technology

VegaFusion Technology Stack

VegaFusion uses a fairly diverse technology stack. The Planner and Runtime are both implemented in Rust.

In the context of vegafusion-jupyter, the Planner is compiled to WebAssembly using wasm-pack and wrapped in a TypeScript API using wasm-bindgen. This TypeScript API is used to integrate the WebAssembly library into the VegaFusion Jupyter Widget.

The Runtime is wrapped in a Python API using PyO3, resulting in the vegafusion-python package. The vegafusion-jupyter package is used to integrate vegafusion-python with Altair and the Python portion of VegaFusion Jupyter Widget.

The Task Graph specifications are defined as protocol buffer messages. The prost library is used to generate Rust data structures from these protocol buffer messages. When Arrow tables appear as task graph root values, they are serialized inside the protocol buffer specification using the Apache Arrow IPC format. The binary representation of the task graph protocol buffer message is what is transferred across the Jupyter Comms protocol.

DataFusion integration

Apache Arrow DataFusion is an SQL compatible query engine that integrates with the Rust implementation of Apache Arrow. VegaFusion uses DataFusion to implement many of the Vega transforms, and it compiles the Vega expression language directly into the DataFusion expression language. In addition to being really fast, a particularly powerful characteristic of DataFusion is that it provides many interfaces that can be extended with custom Rust logic. For example, VegaFusion defines many custom UDFs that are designed to implement the precise semantics of the Vega expression language and the Vega expression functions.

Examples

VegaFusion is compatible with all of the examples in the Altair gallery. Here are a few additional examples that demonstrate the benefits of server-side acceleration.

Large Altair Gallery Examples

The following examples are modifications of Altair Gallery examples that use larger datasets. Each example displays the same plot with plain Altair and with Altair + VegaFusion to make is easy to compare performance. You can try them out using the binder links below.

Interactive Crossfilter

Notebook file
- Voila Dashboard
- Jupyterlab

This browser does not support the video tag.

Interactive Average

Notebook file
- Voila Dashboard
- JupyterLab

This browser does not suppor class="video-border"t the video tag.

10 Million Taxi Rides

This example creates a dashboard to explore 10 million taxi rides in New York City from January 2015. It supports zoom-based rasterization of the pickup locations (similar to Datashader), and combines that with cross filtering across a histogram of trip distances and a heatmap of tip rate by hour and day of the week.

See the PyViz documentation example for more information on the dataset.

Notebook file

This browser does not support the video tag.

About the Name

There are two meanings behind the name “VegaFusion”

It’s a reference to the Apache Arrow DataFusion library described above
Vega and Altair are named after stars, and stars are powered by nuclear fusion

Licensing and Funding

At least until a sustainable funding model is established, VegaFusion will be developed under the AGPLv3 license. This is a copy-left license in the GPL family of licenses. As with all OSI approved licenses, there are no restrictions on what code licensed under the AGPL can be used for. However, the requirements for what must be shared publicly are greater than for licenses that are more commonly used in the Python ecosystem like Apache-2, MIT, and BSD-3.

Roadmap

This is only the beginning for VegaFusion. There are a lot of possible future directions, and many of them are detailed in the roadmap.

Learn more

Here are some ways that you can get in touch to learn more.

Check out the code on GitHub
Start a discussion on GitHub Discussions
And of course, if something doesn’t work please open an issue

Thanks!

VegaFusion 0.0.1

2022-01-19T00:00:00-05:00

The first public version of VegaFusion has been released as 0.0.1. Full project announcement to follow!