Mark's Dev Blog

Random musings on React, Redux, and more, by Redux maintainer Mark "acemarke" Erikson
 Sponsor @markerikson

Declaratively Rendering Earth in 3D, Part 1: Building a Cesium + React App with Webpack

Posted on
#javascript #react #webpack #cesium #create-react-app

This is a post in the Declaratively Rendering Earth in 3D series.

Create-React-App ejection, Webpack tweaks, and use of Webpack's DllPlugin

Intro 🔗︎

Cesium.js is a powerful Javascript library for rendering a 3D globe. It can display a wide variety of geospatial visualizations, including icons and text labels, vector geometries, 3D models, and much more, all on top of a fully-viewable 3D Earth with digital 3D terrain and imagery layers.

Cesium is very similar to Google Earth in concept and capabilities. However, Cesium has many advantages over the now-deprecated Google Earth browser plugin. The Google Earth Plugin was a binary plugin that had to be installed on client systems, and while it had a Javascript API, the internal implementation was a black box of behavior. Also, while GEP could be customized to work with a "Google Earth Enterprise" self-hosted globe data server, GEE licensing was expensive.

On the other hand, while Cesium is primarily developed by employees of AGI, it's a fully open-source Javascript library. Cesium does not require a browser plugin, just the WebGL 3D rendering capability that is built into modern browsers. Cesium also supports a wide variety of imagery and terrain data sources, and can work with both publicly hosted and self-hosted globe data servers.

I've used Cesium in several geospatial applications at work, and have become very familiar with its capabilities. In the process, I've used it with multiple client frameworks, including GWT, Backbone, and now React.

Cesium is a complex toolkit, and dates back to early 2012. Because of that, its architecture has several intricacies that make it somewhat difficult to use in a modern React+Webpack application. While there is existing documentation on how to use Cesium in a Webpack-based application, I haven't yet seen any discussion of how to use it with React. In addition, I've been able to leverage some of Webpack's advanced capabilities to improve the development and deployment process.

In this two-part series, I'll show you how to :

  • Set up a basic React app that loads Cesium
  • Configure Webpack for faster build times and deployment of an application that uses Cesium using DllPlugin for code splitting
  • Use React components to declaratively control rendering of Cesium primitives through Cesium's imperative APIs

This series assumes some familiarity with Cesium, React, and Webpack, and isn't intended to teach the basics for them.

The code for the sample project accompanying this series is on Github at github.com/markerikson/cesium-react-webpack-demo. The commits I made for this post can be seen in PR #1: Configure Webpack I'll be linking to many of the commits as I go through the post, as well as specific files in those commits. I won't paste every changed file in here or show every single changed line, to save space, but rather try to show the most relevant changes for each commit as appropriate.

Update, 2020 🔗︎

While the techniques shown here are still usable, I've since learned and would recommend newer approaches. In particular, I recommend:

In addition, CRA's current vendor chunking works well enough that I no longer see a need to create a separate DLL bundle for the Cesium code.

Table of Contents 🔗︎

Creating a Basic React+Cesium App 🔗︎

Initial Setup 🔗︎

We're going to start by using the excellent Create-React-App tool to create our new project. To keep things simple, we'll skip the details of the initial setup steps, but here's a summary:

  • Use create-react-app to create the project and commit it to Git
  • Use Yarn to set up a lockfile for the project's dependencies
  • Clean out the existing <App> component so it only renders the text "Empty"
  • Use Yarn to add Cesium as a dependency to the project

At the time of writing, the current versions are CRA 0.9.2 and Cesium 1.31.

Configuring Webpack to Use Cesium 🔗︎

As mentioned, Cesium has many complexities in its architecture that make it difficult to use with Webpack out of the box. This includes:

  • Cesium is currently written using the AMD module format for its source files
  • It includes some pre-bundled AMD-based third-party libraries
  • Cesium makes heavy use of web workers
  • Some of the code uses multi-line strings

The primary documentation on using Cesium with Webpack comes from the article Cesium and Webpack on Cesium's blog, and the sample repo mmacaula/cesium-webpack, both written by Mike Macaulay. In his tutorial, he discusses two ways to use Cesium: using the pre-built Cesium bundle, and using Cesium's source directly. We're going to use the "source" approach to start with.

As the blog post points out, we need to set two specific Webpack config options for this to work right. We need to set sourcePrefix: '' to fix Webpack indenting multi-line strings improperly, and also set unknownContextCritical: false to stop Webpack from printing warnings about certain libraries being loaded.

Unfortunately, since we're using Creact-React-App, we don't have direct access to the Webpack config files, since CRA hides them from us by default. So, we're going to have to use CRA's "escape hatch": the npm run eject command. This will copy all of CRA's config files into our own project, and update project.json to include all the individual tool dependencies instead of the single react-scripts dependency. After we eject, everything will still run exactly the same, but we can no longer simply upgrade the react-scripts dependency to get the latest build system updates - we now "own" all the build config ourselves.

After ejecting and committing the newly visible config files, we can make the necessary config tweaks.

First, per the instructions in the "Cesium and Webpack" tutorial, we need to copy Cesium's pre-built worker files and assets to our public folder. This is a manual step, and doesn't involve any commits. Browse into $PROJECT/node_modules/cesium/Build/. You should see two folders, Cesium and CesiumUnminified. Copy the entire Cesium folder over to PROJECT/public/, and rename it to cesium. Then, delete the Cesium.js file that's just inside. You should now have the following folder structure:

- $PROJECT
  - public
    - cesium
      - Assets
      - ThirdParty
      - Widgets
      - Workers

Since we don't want to actually commit these files, it's a good idea to add a line to the project's .gitignore file to ignore these:

Commit c47204c: Ignore the public/cesium folder containing the copied Cesium output

.gitignore

# production
/build
+/public/cesium/

Then, we'll add the needed config tweaks to the Webpack config files:

Commit 2015273: Add Webpack config options needed to run Cesium

config/webpack.config.dev.js

    publicPath: publicPath,
+   sourcePrefix : '',

// Skip ahead

  module: {
+   unknownContextCritical : false,

This should be done for both the development and production configs.

Loading Cesium Into the App 🔗︎

Before we can actually load the Cesium Viewer widget into our application, we have to configure Cesium so it knows how to construct the URLs for all of its assets. That requires a call to the buildModuleUrl() function that Cesium provides. Once that's done, we can load the Viewer instance and have things work correctly.

Commit 17479d3: Add initial Cesium viewer to the app

src/index.js

import App from './App';
import './index.css';

+import "cesium/Source/Widgets/widgets.css";
+
+import buildModuleUrl from "cesium/Source/Core/buildModuleUrl";
+buildModuleUrl.setBaseUrl('./cesium/');

Note that we set the base URL to "./cesium/", which corresponds to the pre-built Cesium folder we copied earlier into $PROJECT/public.

From there, all we have to do is use the standard React approach for interacting with code that needs to access a DOM element directly. We render a div that will serve as the container for our Cesium widget, and use a "callback ref" to save a reference to the real DOM element. Then, in componentDidMount, we create the Cesium.Viewer instance, and give it the div reference to serve as its parent:

src/App.js

import React, { Component } from 'react';

import Viewer from "cesium/Source/Widgets/Viewer/Viewer";

class App extends Component {
    componentDidMount() {
        this.viewer = new Viewer(this.cesiumContainer);
    }

    render() {
        return (
            <div>
                <div id="cesiumContainer" ref={ element => this.cesiumContainer = element }/>
            </div>
        );
    }
}

Notice that we're importing pieces from inside Cesium's folder structure, rather than just import Cesium from "cesium".

If we run the project, here's what we should see:

Success! We've loaded Cesium, and are viewing a 3D globe with the Cesium Viewer in its default configuration. (If you squint carefully at the image, you'll see that the viewer doesn't actually fill the whole screen - there's a bunch of blank space below. We'll fix that in the next part.)

That wraps up the basic application setup. Let's move on to setting up a proper production build.

Optimizing Cesium App Deployment 🔗︎

Thus far, we've been simply importing Cesium directly into our main application. That's okay as a starting point, but it also means that our main application bundle is going to include pretty much all of Cesium. Unfortunately, Cesium is a big toolkit. Let's run a production build of the application:

$ node scripts/build.js
Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  532.54 KB  build\static\js\main.304c45dc.js
  4.78 KB    build\static\css\main.c5310e60.css

Done in 48.13s.

Hmm. "532KB after gzip?" That seems big. Let's check out what goes into the bundle, using the source-map-explorer to view the contents of the minified bundle:

EEEP! Our production app bundle is over 2MB, and Cesium itself is taking up 1.8MB of minified Javascript!!! That's... not good, to say the least. Meanwhile, Cesium's various web workers and assets, while loaded separately, are also a significant chunk of space as well. Yes, gzipping the bundle will take that down to about 532KB, but that's still way more than we'd like to load at once.

The bad news is that there's only so much we can do to solve this. Cesium is simply a very large, very complex toolkit, and there's nothing we can do to fully shrink down the size of Cesium itself. Any app using Cesium will never win awards for "Smallest Total Loading Size".

However, the good news is that we can separate Cesium out from our application bundle. We do still need to load it, and in fact we'll need to load it before the app bundle, but by separating it out we can at least make it cached for the next time the user loads the app. Also, by pre-building the Cesium bundle, we can cut down on production build time for our application's bundle.

Serving Cesium Assets in Development 🔗︎

Before we get to building Cesium as a separate bundle, we can make an improvement to our development process. All this time, we've had a copy of Cesium's pre-built output folder sitting in our $PROJECT/public/cesium/ folder, because those files need to be loaded by Cesium at runtime from the server. We can remove the need for that by modifying the CRA dev server setup so that it serves those files directly out of node_modules/cesium/.

First, we'll add a few more entries to CRA's list of pre-defined paths:

Commit 73bdbe2: Add additional paths for use in build config

config/paths.js

  ownNodeModules: resolveApp('node_modules'),
+  app : resolveApp('.'),
+  appConfig : resolveApp('config'),
+  cesiumDebugBuild : resolveApp('node_modules/cesium/Build/CesiumUnminified/'),
+  cesiumProdBuild : resolveApp('node_modules/cesium/Build/Cesium/'),
+  cesiumSourceFolder : resolveApp('node_modules/cesium/Source/'),
  nodePaths: nodePaths,

Then, we'll make a small edit to CRA's dev server setup to tell it to serve static files directly out of the right folder:

Commit eabdcf5: Modify CRA dev server to serve Cesium files from node_modules

scripts/start.js

var openBrowser = require('react-dev-utils/openBrowser');
var prompt = require('react-dev-utils/prompt');
+var express = require("express");

// Skip ahead

function addMiddleware(devServer) {

+  // Handle requests for Cesium static assets that we want to
+  // serve up direct from /node_modules/cesium/.
+  devServer.use("/cesium", express.static(paths.cesiumDebugBuild));
+

We should now be able to delete the cesium/ folder from $PROJECT/public/, at least in terms of development. CRA automatically copies everything inside of /public/ to the build output folder when it does a production build, so we'll need to add some custom logic to handle copying Cesium's assets for production builds. We'll come back to that in a bit.

Building a Cesium Bundle using DllPlugin 🔗︎

Webpack has several ways to create bundles and chunks. The most widely used methods are declaring bundle entry points in the entry section of a Webpack config, and using the CommonsChunkPlugin to extract files that are shared between multiple chunks into a separate bundle.

There's another Webpack plugin that's less well known: DllPlugin. Named after the idea of a native code "Dynamic Link Library" from Windows (or .so for you *nix users), the idea is that you pre-build a bundle containing the reusable or shared portions of your code, and also create a metadata file that describes what's inside the bundle. Then, when you build your application itself, you add a reference to the metadata so that Webpack knows what code should already be available. Webpack will then skip including those pieces in your app bundle.

Building a DLL bundle requires creating a new Webpack config that's separate from the application configs. Here's what the config looks like:

Commit 4ddc26a: Add a Webpack config to pre-build a Cesium bundle

config/webpack.cesium.dll.config.js

"use strict";

const path = require("path");
const webpack = require("webpack");

const paths = require("./paths");
const env = require("./env");

const outputPath = path.join(paths.app, "distdll");

const webpackConfig = {
    entry : {
        cesiumDll : ["cesium/Source/Cesium.js"],
    },
    devtool : "#source-map",
    output : {
        path : outputPath,
        filename : "[name].js",
        library : "[name]_[hash]",
        sourcePrefix: "",
    },
    plugins : [
        new webpack.DllPlugin({
            path : path.join(outputPath, "[name]-manifest.json"),
            name : "[name]_[hash]",
            context : paths.cesiumSourceFolder,
        }),

        new webpack.DefinePlugin({
            'process.env.NODE_ENV': JSON.stringify("production")
        }),

        new webpack.optimize.UglifyJsPlugin({
            compress: {
                warnings: false
            }
        })

    ],
    module : {
        unknownContextCritical : false,
        loaders : [
            { test : /\.css$/, loader: "style!css" },
            {
                test : /\.(png|gif|jpg|jpeg)$/,
                loader : "file-loader",
            },
        ],
    },
};

Some key things to note here. First, we're building this in production mode by setting process.env.NODE_ENV="production" and minifying it. We've also included the couple specific config settings we had to tweak earlier.

The really important part is that we're including the DllPlugin in the config. The path option tells Webpack where to write the metadata file, name is the name of the bundle file itself, and I think context has to do with how the imported source files are looked up. Roughly. :)

Now that we have the config file, we need to actually run Webpack with this config to generate our Cesium DLL bundle. I've written a couple new script files to control this process. One encapsulates the work of running Webpack with a certain config and printing out some readable stats, and the other just calls the Webpack compiler script with the Cesium DLL config.

Commit b55dbd5: Add scripts to build a Cesium bundle with DllPlugin

We'll skip looking at the script contents, but here's the output of running the DLL build script:

$ node ./scripts/buildCesiumDLL.js
Compiling: cesium
  build [====================] 100% (35.5 seconds) ()

Build completed in 35.47s

Hash: 0829da3ac0fb7ef638b5
Version: webpack 1.14.0
Time: 35472ms
           Asset     Size  Chunks             Chunk Names
    cesiumDll.js  2.13 MB       0  [emitted]  cesiumDll
cesiumDll.js.map    19 MB       0  [emitted]  cesiumDll
Done in 37.33s.

If we look in $PROJECT/distdll, we should see three files: cesiumDll.js, cesiumDll.js.map, and cesiumDll-manifest.json. Cesium is still a really hefty chunk of code, but at least now it's a separate file.

Using a DLL Bundle in the Application 🔗︎

Happily, the hard part is done. Now that we have the DLL bundle generated, we can point our production Webpack config at the manifest file:

Commit 6db616f: Update Webpack prod config to use the Cesium DLL bundle

config/webpack/config.prod.js

+var path = require('path');
var autoprefixer = require('autoprefixer');

// Skip ahead

  plugins: [
+     new webpack.DllReferencePlugin({
+       context : paths.cesiumSourceFolder,
+         manifest: require(path.join(paths.app, "distdll/cesiumDLL-manifest.json")),
+     }),

We just add DllReferencePlugin to the production config, set the manifest option to point to the manifest file that was generated earlier, and use the same context option we used when we generated the DLL (to make sure that source file references match before and after).

Let's re-run a production build of the app and see how things look:

$ node scripts/build.js
Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  46.09 KB  build\static\js\main.f984614f.js
  4.78 KB   build\static\css\main.c5310e60.css

Done in 10.43s.

That looks much better! The compile time dropped from 48 seconds to about 10 seconds, and the main bundle is down to 46KB gzipped. Let's take another look at the contents of the bundle using source-map-explorer:

The overall application bundle is now down to 150KB. We're now looking at just the application code plus the non-Cesium libraries. There's more we could do to optimize, but this is sufficient for the concepts I wanted to show off.

Including Cesium in Production 🔗︎

We now need to actually include Cesium into the production build output. Since we deleted the hand-copied public/cesium folder, Cesium's assets aren't being copied to /build/ any more. We can add some logic to CRA's existing build script to do that. It'll just do some glob searching to collect a list of files under node_modules/cesium/Build/Cesium/, and copy each one to the output folder, along with the Cesium DLL bundle we've created.

Commit 902ba04: Update build script to copy Cesium files to the output folder

The last step is to actually include the Cesium DLL bundle into our page. CRA uses the HtmlWebpackPlugin to insert the right script tags into the index.html template during the build process. We can customize the template so that it only includes the DLL bundle during a production build.

Commit c65e7bd: Conditionally include Cesium bundle in the HTML host page

config/webpack.config.prod.js

    new HtmlWebpackPlugin({
      inject: true,
      template: paths.appHtml,
+     production : true,

HtmlWebpackPlugin takes a number of specific options, but you can also add any other options you want, and it will pass them through. So, since we know we're doing a prod build when this config is being used, we just add a production : true option that we can use in the template.

public/index.html

  <body>
    <div id="root"></div>
+   <% if(htmlWebpackPlugin.options.production) { %>
+       <script src="cesium/cesiumDll.js"></script>
+   <% } %>
    <!--

HtmlWebpackPlugin uses Lodash templating, so we can insert a simple check for the production option we added, and only include the Cesium DLL script tag if it's a prod build.

Note that the Cesium DLL bundle must be loaded before the main application bundle! This is because the DLL bundle exposes its contents as a global variable (such as var cesiumDll_0829da3ac0fb7ef638b5 ), and the DllReferencePlugin adds logic to look for that specific variable.

With that, we should be able to run an HTTP static file server from our build folder, and see the application successfully load!

Wrapping Up Webpack Optimization 🔗︎

While I was working on this post, I had intended to demonstrate lazy-loading the DLL bundle using Webpack's code-splitting abilities. Unfortunately, I realized that what I'm doing in my own app right now doesn't actually qualify as lazy-loading. I've got some code-splitting in place, but as we just saw, bundles produced by DllPlugin still have to be loaded manually. Since a DLL bundle is usually vendor libs, it's reasonable to have a specific script tag that loads the DLL bundle first.

There's a couple threads in the Webpack issues tracker that discuss ways to actually load DLL bundles at runtime, so that they truly are lazy-loaded. The consensus seems to be that it's possible, but it requires use of some non-Webpack loading code to make it work. See Webpack issues #2592 and #3115 for discussion on the topic.

It's also important to note that DllPlugin doesn't actually reduce the amount of code that has to be loaded, but it does split things into more cacheable pieces.

Final Thoughts 🔗︎

Cesium is a complex and powerful library, and Webpack is a complex and powerful build tool. Getting them to play nicely together takes a bit of work, but is well worth the effort.

Next up in Part 2, we'll look at how to use React components to control Cesium's API. Be sure to check it out!

Further Information 🔗︎

This is a post in the Declaratively Rendering Earth in 3D series. Other posts in this series: