No Clocks

Optimization 2: Streamable Hilbert-sorted RTree Index

The second exciting piece of flatgeobuf, and honestly what makes the whole thing worth the price of admission alone, is its approach to indexing. Including a pre-built index with geospatial data isn’t new — Shapefile and Geopackage support this, and there’s always spatialite if you want to pack your data into a sqlite file.

To do this, the format combines an RTree index (the trusty workhorse of all geospatial indexing) with a compact and streamable encoding and a particular spatial ordering of the geometry data (using a Hilbert Curve) to make the whole thing more efficient. Here’s how it works.

To write an indexed FGB file, you first sort your feature data along a Hilbert Curve. Space Filling Curves are a fascinating and brain-melting topic that merit a full post on their own, but in short they allow a linear encoding of multiple dimensions into 1 dimension (in our case, sorting a 2-dimensional xy space along a 1-dimensional curve). This is an optimization used in many spatial indexing systems to ensure that data with high storage locality also has high spatial locality.

The point is that both the index and the underlying feature data are designed to be loaded lazily — a client can stream the portions of the index they need to satisfy a bounding-box query, then use those results to stream the relevant portions of the Feature section. This can all be done either locally (reading from a potentially large file) or remotely over HTTP using Range requests.

Downsides? Complexity — No denying the format is pretty complicated. Between sophisticated binary encodings and optimized indexing strategies there is a lot going on in these bytes. Small errors in encoding or indexing can render files unreadable. To become more mainstream, the format will need better linting tools to validate structural correctness of without a lot of painful hexdump-ing. Not human readable — Especially compared to simpler formats like GeoJSON and WKT this is always a downside. Of course the benefits are huge (smaller storage footprint, faster serde), but it will take really polished tooling to ease people off of text-based formats. On the other hand Shapefiles aren’t human-readable and ESRI has managed to jam those down everyone’s throats for several decades, so maybe there’s hope. Not designed for updates — Because of the intricacies of data layout and indexing, updates aren’t really a thing for FGB. It’s best used for static datasets. Non-spatial indexing — Currently the format doesn’t really have a way to add indexing on other dimensions besides the spatial one. Maybe this will evolve in the future (FGB’s first sidecar?)

All R functions are technically closures (they all carry an enclosing environment), but the term earns its keep when a function captures variables from a parent that isn’t the global environment. Here is a counter that actually counts:

<<- is the only way to create mutable state in R’s functional world, because it searches parent environments for an existing binding and modifies it there. If it doesn’t find the variable in any parent, it creates one in the global environment. That is almost always a mistake.

R descends from the lambda calculus, by way of Scheme and a Bell Labs language called S. That ancestry shaped every part of the language. All of it traces back to a single design decision made in the 1970s.

Geospatial Data Preparation

Open options Open options can be specified in command-line tools using the syntax -oo <NAME>=<VALUE> or by providing the appropriate arguments to GDALOpenEx() (C) or gdal.OpenEx (Python). The following open options are supported: ENCODING=[<encoding_name>​/​""]: Override the encoding interpretation of the shapefile with any encoding supported by CPLRecode or to "" to avoid any recoding. DBF_DATE_LAST_UPDATE=YYYY-MM-DD: Modification date to write in DBF header with year-month-day format. If not specified, current date is used. ADJUST_TYPE=[YES​/​NO]: Defaults to NO. Set to YES to read the whole .dbf to adjust Real->Integer/Integer64 or Integer64->Integer field types when possible. This can be used when field widths are ambiguous and that by default OGR would select the larger data type. For example, a numeric column with 0 decimal figures and with width of 10/11 character may hold Integer or Integer64, and with width 19/20 may hold Integer64 or larger integer (hold as Real) ADJUST_GEOM_TYPE=[NO​/​FIRST_SHAPE​/​ALL_SHAPES]: Defaults to FIRST_SHAPE. Defines how layer geometry type is computed, in particular to distinguish shapefiles that have shapes with significant values in the M dimension from the ones where the M values are set to the nodata value. By default (FIRST_SHAPE), the driver will look at the first shape and if it has M values it will expose the layer as having a M dimension. By specifying ALL_SHAPES, the driver will iterate over features until a shape with a valid M value is found to decide the appropriate layer type. AUTO_REPACK=[YES​/​NO]: Defaults to YES. Whether the shapefile should be automatically repacked when needed, at dataset closing or at FlushCache()/SyncToDisk() time. DBF_EOF_CHAR=[YES​/​NO]: Defaults to YES. Whether the .DBF should be terminated by a 0x1A end-of-file character, as in the DBF spec and done by other software vendors. Previous GDAL versions did not write one.