BlogSourceIssues

d3 MapRenderer

A python QGIS plugin to export of polygons, polylines and point vector layers from shapefiles to topojson for display within a web page using the d3.js JavaScript library, with additional options of popup information based on d3-tip, charts from c3.js and a legend.

The aim of the plugin is not to replicate all of the mapping functionality available within d3, nor is it to provide every type of chart or data visualisation. The plugin aims to provide you with a solid starting point to start hacking the JavaScript code. For instance, you may want to use a different charting library, remove the y or x axis from the current chart, use a different projection, and alter the map size or colour scheme.

The log messages panel in QGIS will also contain the topojson command used to convert the shapefile to topojson, and can be used as a template to hand craft your own conversions via the topojson command line interface.

Pre-requisites

Download and install QGIS.

Download and install Node.js.

Once Node.js is installed, you will need to install the topojson package which converts Geographic Information System shapefiles to d3's topojson format. This can be done by opening a command prompt or terminal (depending on your operating system), and typing:

npm install -g topojson

This command downloads the topojson package and registers it for use on the command line.

d3 relies heavily on JSON files for loading data, and topojson is no different. To see the results of an export via this plugin you will also need a web server, as restrictions in modern web browsers prevent json files being loaded from the local file system. So if you want to view the results of the export before uploading to a website in the cloud (recomended), you will need a web server.

If you haven't already got a web server already installed use http-server for Node.js. Install it by returning to that command line and typing:

npm install -g http-server

http-server is started and stopped via the command line, and it is as simple as changing to the directory where you are going to output your d3 maps (this will be the root of the local web site), and starting the server. For example, on Windows this would be the following two commands (obviously using your own directory path):

cd c:\Users\Simon\Documents
http-server

That wasn't so bad, was it?

Installation

Download from the QGIS plugin repository. Currently flagged as an experimental plugin.

If you want to try the bleeding edge version, download the zip direct from GitHub, and extract the contents into your QGIS python plugins directory. On Windows this is something along the lines of:

C:\Users\Simon\.qgis2\python\plugins

Rename the folder from d3MapRenderer-master to just d3MapRenderer and you will find the plugin listed along side the others in the QGIS plugin repository.

Using the plugin

Once installed, open a QGIS project and click the D3 button in the toolbar.

The plugin dialog uses a tabbed interface to split the d3 map rendering options into groups.

The Main tab is where the mandatory options are specified: the map title, width and height of the map, the main layer, id field, d3 projection and directory to output the map and associated files.

The function of the fields is as follows:

  • The "Title" field is the title for the web page, and page header (if chosen).
  • The "Include page heading?" field tells the plugin to use the title for the web page is included as a page heading element.
  • The "Width" field is the width in pixels of the d3 map. This is a JavaScript variable in the web page, and can be altered after output.
  • The "Height" field is the height in pixels of the d3 map. This is a JavaScript variable in the web page, and can be altered after output.
  • The "Main layer" field is the vector layer within the QGIS project that contains the information to be visualised. The plugin uses this vector layer as the source for any legend, popup information or visualizations, as well as the initial extent of the map.
  • The "ID" field is the unique ID field within the main layer attributes which can be used to uniquely identify the feature. It is vital that a unique ID field is available when using popup information or charts, as the ID is used to select the correct data from a CSV file. Not having a unique ID will result in the wrong information being displayed. If one is not present in the layer attributes simply add a new field with the QGIS Field Calculator and use the expression $id to add a unique numeric identifier.
  • The "Projection" combo box lists all the supported d3 projections available to users of the plugin. More projections exist and these will be added over the course of further development of the plugin, but you can easily change the JavaScript code yourself to switch to a different projection, or even write one yourself.
    Conic projections such as Albers (Conic Equal Area) or Lambert Conic Conformal are best suited for small areas.
    Albers USA is a d3 specific projection best for United States visualisations, including Alaska and Hawaii as well as the lower 48 states in a compact layout.
  • The "Simplification" field allows control of the simplification level used in the creation of the topojson file. The simplification is from zero (no simplification), 1e-12 (very little) through to 9e-3 (extreme) and is measured in steradians. Simplification reduces the points within polylines and polygons in the shapefile, significantly reducing file size but increasing distortion. If you're producing a map of building footprints you would not want to use any simplification.
  • The "Output directory" field sets the location to store the output files. This location must already exist. A sub-directory with a time stamped name is created under the specified location to ensure uniqueness.
    On Windows this is limited to ascii charcaters only due to problems calling the topojson package via the command line (the Windows command prompt does not accept unicode characters). No such limitation on Linux installations.
  • The "Allow zoom and pan?" field sets whether the end user to zoom and pan around the d3 map. The default zoom is to 40 times the original scale, though this can be altered in the web page. Outlines are redrawn during the zoom process.
  • The "Include a legend?" flag determines whether to display a legend based on the categorised or graduated symbols selected for the main layer.
  • The "Legend position" combo box specifies the display position of any legend in one of the four corners of the visualisation, or in a html container external to the map, allowing the user to alter the position of the legend according to the design of their hosting page.

The Extra tab allows further layers to be added to the map. All layers are output in the legend order displayed in QGIS, maintaining background and foreground layer order.

The "Include other vector layers?" flag enables the "Vector layers" tree widget, allowing the selection of any other vector layer for inclusion in the d3 map. In the current version, layer groups within the QGIS legend are not supported, instead the layers are flattened. Output of the layers is performed in the reverse order of the legend display, so the background layers are displayed in the background, and the foreground layers on top in the resulting visualisation.

The Popup tab allows the creation of an information popup from attributes within the main layer.

The majority of the fields on the Popup tab are disabled by default, becoming enabled when the user checks the "Include information popup?" flag, allowing the display of attribute information when the user clicks on an object from the main layer. Once enables, the fields function as follows:

  • The "Popup position" field provides a choice of the popup position with options of "Bubble" or "External". Bubble creates a popup tip which will be familiar to users of Google maps, external displays the information in an html element external to the map, allowing the position of the information according to the design of the hosting website.
  • The "Popup fields" tree widget displays all the attribute fields from the main layer for inclusion in the information popup. Attributes are added to a list in the order they are selected. The values for the data is loaded into an info.csv file in the output directory where the values are dynamically looked up by the main layers id when the user selects an object within the visualisation.
  • The "Preview" field offers a preview of the popup information which will be output by the plugin. The column name of the attribute and the value are output as pairs of tables columns.
    The column title and layout of the html can be altered after the plugin has finished outputting the web page and associated files. The attribute value (the bit between the curly braces {MYCOLUMN}) should not be altered as the html template is dynamically parsed for these tags. For example:
<table>
<tr><td>MYCOLUMN</td><td>{MYCOLUMN}</td></tr>
</table>

The Viz tab allows for c3.js charts to be created from numeric attributes within the main layer.

Again the majority of fields are disabled by default, only becoming enabled when the user checks the "Include viz in popup?" flag. This allows specification of a chart based on numeric attributes within the chosen main layer, and uses the popup defined on the previous tab as the host. Once enabled the fields function as follows:

  • The "Chart type" combo box displays the list of supported C3.js charts. The settings of the chart can be altered by changing the JavaScript after the export is complete.
  • The "Width" field is the width in pixels of the chart. This is a JavaScript variable in the web page, and can be altered after output.
  • The "Height" The height in pixels of the chart. This is a JavaScript variable in the web page, and can be altered after output. Choose the width and height to fit the orientation of the chart.
  • The height in pixels of the chart. Choose the width and height to fit the orientation of the chart.
  • The "Viz fields" tree widget displays all the numeric attributes from the main layer for inclusion as a data range along the X-axis. Once attributes have been selected a data range can be added by using the add button located to the right of the field. A prompt is given for a name for the data range, which is used in the c3.js chart legend.
  • Multiple data ranges can be added by repeating this process, just ensure the data ranges have the same amount of fields, otherwise the chart will not function correctly and will need the JavaScript code. Attributes are added to the data range in the order they are selected. The values for the data are loaded into an info.csv file in the output directory where the values are dynamically looked up by the main layers id when the user selects an object within the visualisation.
  • The "Data ranges" field is a readonly comma separated preview of the data range(s), with the specified name and attribute field(s). Each row is a separate data range for the X-axis. The Minus button to the right can be used to remove the last added data range from the list.
  • The "X-axis labels" is a comma separated list of numeric labels for the X-axis, limited to 10 numeric characters for each label. If not all labels are specified, no labels will be output and c3.js will default to a series of numeric values. You may want to hide the axis completely, by changing the JavaScript that is output.

The buttons Cancel and OK, quit the plugin and run the output process respectively.

Output

Once the output process is started, the output folder is created, ensuring a unique name if any duplicate folder name is found, and ancillary folders and files are copied across.

The plugin then loops through the list of selected layers in the reverse order of the QGIS legend, exporting the background layers first and ending with the top most layer, in order to reproduce the same stacking of layers in the d3 map. Each layer is saved using the WSG84 Co-ordinate Reference System to the output folder, ensuring and the shapefile is not currently locked for editing, filtering is preserved and that a CRS is being used that topojson understands.

The saved shapefile is then re-opened and a new column added to store the CSS classname for each feature in the shapefile. This allows the flexibility of using CSS to style the features in the map.

The plugin then reproduces as best it can the QGIS renderer and symbology. At the time of writing only Single symbol, Graduated and Categorized renderers are supported, an error is deliberately thrown if, for example, a Rule-based renderer is encountered. Once the symbology has been read, it is appended to a stylesheet.

Each layer is then converted to topojson format. Only the unique ID and CSS layer attributes are stored in the topojson file. Although this goes against the design of topojson, it ensures a further reduction in file size. A future version of the plugin will provide the option of embedding all selected attributes in the topojson file, as well as storing them externally.

CSV files are produced for the legend and information popup (if required).

The shapefiles which form the source of the topojson file(s) are zipped. These can be quite large so don't upload to a webserver if you have storage restrictions. However, this is useful as a means of distributing your work for reproduction by others.

Once this is complete the default web browser is opened to display the resulting map. v0.7 assumes the http-server Node.js package is being used and has been started with the chosen output folder as its root. A future version will add settings to better support other web servers and urls.

The resulting folder and files are structures as follows:

  • css
    • c3.min.css - Stylesheet for c3.js charts
    • color.css - Stylesheet for map background and elements
    • legend.css - Stylesheet for the built in legend element
    • map.css - Stylesheet for basic map styling and outline
    • tip.css - Stylesheet for the d3-tip
  • data
    • info.csv - CSV file containing all the attributes required for the chosen chart or information popup
    • legend.csv - CSV file describing the legend elements
  • js
    • c3.min.js - Minified JavaScript file for c3.js
    • map.legend.js - JavaScript file for the legend element
    • map.tip.js - Customised version of d3-tip to work on mouse click events on map elements
  • shp
    • source.zip - Zipped shapefiles
  • topo
    • *.json - topojson file(s). There will be one topojson file for each layer chosen for output
  • index.html - The main content

Logging issues and feature requests

The issue log is to be used for reporting problems with the plugin and for feature requests and ideas. It is not a support forum. Nor is it the place to log problems with topojson, d3 or c3.

Before reporting an issue, please do the following:

  1. Search for existing issues to ensure you're not creating a duplicate.
  2. When posting the issue, please use a descriptive title, include the plugin version and relevant details about what was trying to be achieved with the plugin.
  3. Also, please post the details from the Log Messages Panel within QGIS, and if possible a link to download the pertinent shapefiles. A sample of the log messages for the plugin:
2015-08-12T14:56:58 0 model 2.10.1-Pisa
2015-08-12T14:56:58 0 model 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]
2015-08-12T14:56:58 0 topo Windows
2015-08-12T14:56:58 0 winHelper node.js found at C:\Program Files\nodejs\node.exe
2015-08-12T14:56:58 0 winHelper User environment variables: C:\Users\Simon\AppData\Roaming\npm
2015-08-12T14:56:58 0 winHelper topojson found at C:\Users\Simon\AppData\Roaming\npm\node_modules\topojson\bin\topojson
2015-08-12T15:00:50 0 model EXPORT start ==================================================
2015-08-12T15:00:50 0 model        Title = [English IMD]
2015-08-12T15:00:50 0 model        Header = [True]
2015-08-12T15:00:50 0 model        Width = [600]
2015-08-12T15:00:50 0 model        Height = [900]
2015-08-12T15:00:50 0 model        Main layer = [LSOA_2001_IMD]
2015-08-12T15:00:50 0 model        IDField = [LSOA01CD]
2015-08-12T15:00:50 0 model        Projection = [Albers]
2015-08-12T15:00:50 0 model        Simplify = [1e-9]
2015-08-12T15:00:50 0 model        Output = [D:\Downloads\Temp]
2015-08-12T15:00:50 0 model        Zoom/Pan = [True]
2015-08-12T15:00:50 0 model        Legend = [True]
2015-08-12T15:00:50 0 model        LegendPos = [Top Left]
2015-08-12T15:00:50 0 model        IncExtras = [True]
2015-08-12T15:00:50 0 model        Extras = [UK_IRE_POLY, LSOA_2001_IMD]
2015-08-12T15:00:50 0 model        IncPopup = [True]
2015-08-12T15:00:50 0 model        PopupPos = [Bubble]
2015-08-12T15:00:50 0 model        Popup = [<table>
<tr><td>GORNAME</td><td>{GORNAME}</td></tr>
<tr><td>LANAME</td><td>{LANAME}</td></tr>
<tr><td>LACODE</td><td>{LACODE}</td></tr>
</table><div id="chart" style="width: 240px; height: 240px"></div>]
2015-08-12T15:00:50 0 model        IncViz = [True]
2015-08-12T15:00:50 0 model        Chart = [Spline Chart]
2015-08-12T15:00:50 0 model        VizWidth = [240]
2015-08-12T15:00:50 0 model        DataRanges = [Overview:  OVRK2004   , OVRK2007   , OVRK2010   
Income:    INCRK2004  , INCRK2007  , INCRK2010  
Employment:EMPRK2004  , EMPRK2007  , EMPRK2010  
Health:    HLTHRK2004 , HLTHRK2007 , HLTHRK2010 
Education: EDRK2004   , EDRK2007   , EDRK2010   
Barriers to Housing and Services:HOURK2004  , HOUSRK2007 , HOUSRK2010 
Crime:     CRMRK2004  , CRMRK2007  , CRMRK2010  
Living Environment:LENVRK2004 , LENVRK2007 , LENVRK2010 
]
2015-08-12T15:00:50 0 model        Labels = [2004, 2007, 2010]
2015-08-12T15:00:50 0 model EXPORT copying folders and files
2015-08-12T15:00:50 0 model EXPORT UK_IRE_POLY
2015-08-12T15:00:51 0 model SINGLE: FILL SYMBOL (1 layers) color 217,217,217,255
2015-08-12T15:00:51 0 model setSingleSymbol
2015-08-12T15:00:51 0 model Filter: 
2015-08-12T15:00:52 0 winHelper C:\Program Files\nodejs\node.exe C:\Users\Simon\AppData\Roaming\npm\node_modules\topojson\bin\topojson -o C:\Users\Simon\Documents\EnglishIMD\topo\UKIREPOLY.json --id-property LSOA01CD -p d3Css -s 1e-9 -- l0=C:\Users\Simon\Documents\EnglishIMD\shp\UK_IRE_POLY.shp
2015-08-12T15:00:56 0 winHelper topojson result 
bounds: -10.625570682933017 49.168431613190194 1.7610498126253136 60.86076643797765 (spherical)
pre-quantization: 1.38m (0.0000124°) 1.30m (0.0000117°)
topology: 3394 arcs, 2037677 points
post-quantization: 138m (0.00124°) 130m (0.00117°)
simplification: retained 46554 / 391204 points (12%)
prune: retained 868 / 3394 arcs (26%)
2015-08-12T15:00:56 0 model EXPORT LSOA_2001_IMD
2015-08-12T15:00:58 0 model GRADUATED: attr OVRK2010
1 - 3249.1::Most deprived::FILL SYMBOL (1 layers) color 179,0,0,255
3249.1 - 6497.2::||::FILL SYMBOL (1 layers) color 227,74,51,255
6497.2 - 9745.3::|||::FILL SYMBOL (1 layers) color 252,141,89,255
9745.3 - 12993.4::||||::FILL SYMBOL (1 layers) color 253,204,138,255
12993.4 - 16241.5::|||||::FILL SYMBOL (1 layers) color 254,240,217,255
16241.5 - 19489.6::||||||::FILL SYMBOL (1 layers) color 255,255,204,255
19489.6 - 22737.7::|||||||::FILL SYMBOL (1 layers) color 161,218,180,255
22737.7 - 25985.8::||||||||::FILL SYMBOL (1 layers) color 65,182,196,255
25985.8 - 29233.9::|||||||||::FILL SYMBOL (1 layers) color 44,127,184,255
29233.9 - 32482::Least deprived::FILL SYMBOL (1 layers) color 37,52,148,255
2015-08-12T15:00:58 0 model setGraduatedSymbol
2015-08-12T15:00:58 0 model Filter: "OVRK2010" >= 1.0 and "OVRK2010" <= 3249.1
2015-08-12T15:00:59 0 model Filter: "OVRK2010" >= 3249.1 and "OVRK2010" <= 6497.2
2015-08-12T15:01:00 0 model Filter: "OVRK2010" >= 6497.2 and "OVRK2010" <= 9745.3
2015-08-12T15:01:01 0 model Filter: "OVRK2010" >= 9745.3 and "OVRK2010" <= 12993.4
2015-08-12T15:01:02 0 model Filter: "OVRK2010" >= 12993.4 and "OVRK2010" <= 16241.5
2015-08-12T15:01:03 0 model Filter: "OVRK2010" >= 16241.5 and "OVRK2010" <= 19489.6
2015-08-12T15:01:04 0 model Filter: "OVRK2010" >= 19489.6 and "OVRK2010" <= 22737.7
2015-08-12T15:01:05 0 model Filter: "OVRK2010" >= 22737.7 and "OVRK2010" <= 25985.8
2015-08-12T15:01:06 0 model Filter: "OVRK2010" >= 25985.8 and "OVRK2010" <= 29233.9
2015-08-12T15:01:07 0 model Filter: "OVRK2010" >= 29233.9 and "OVRK2010" <= 32482.0
2015-08-12T15:01:10 0 winHelper C:\Program Files\nodejs\node.exe C:\Users\Simon\AppData\Roaming\npm\node_modules\topojson\bin\topojson -o C:\Users\Simon\Documents\EnglishIMD\topo\LSOA2001IMD.json --id-property LSOA01CD -p d3Css -s 1e-9 -- l1=C:\Users\Simon\Documents\EnglishIMD\shp\LSOA_2001_IMD.shp
2015-08-12T15:01:17 0 winHelper topojson result 
bounds: -6.3649789650534805 49.886367784112075 1.762763270374323 55.81105303680311 (spherical)
pre-quantization: 0.904m (0.00000813°) 0.659m (0.00000593°)
topology: 96721 arcs, 271892 points
post-quantization: 90.4m (0.000813°) 65.9m (0.000593°)
simplification: retained 269975 / 271792 points (99%)
prune: retained 96720 / 96721 arcs (100%)
2015-08-12T15:01:17 0 bounds Max bounds: -10.6255706829 49.8863677841 1.76276327037 60.860766438
2015-08-12T15:01:17 0 model d3.geo.albers()
     .center([0, 55])
     .rotate([4.0, 0])
     .parallels([49, 61])
     .scale(1000)
     .translate([width / 2, height / 2])
2015-08-12T15:01:21 0 model EXPORT popup data
2015-08-12T15:01:29 0 model EXPORT complete =========================================================

Versions

  • v 0.9
    Additional projections. Better support for Conic Conformal projection (and Albers). Support for no brush and no pen. Fix for binding of first SVG path element.
  • v 0.8
    Unicode support added. Validation added to check the output directory exists. Sub-directory for output files now created with a time stamped name.
    Linux now supports non-ASCII chars in the output directory, title, legend and popup.
    Windows subprocess command on python 2.7 will not accept unicode characters (it has it's own limited encoding). On Windows only, the output directory is limited to ascii characters. Title, legend and popup support unicode.
  • v 0.7
    Initial version