No Clocks

Like other relational databases, GeoPackages contain a number of tables. These tables fall into two categories, user-defined data tables and metadata tables. GeoPackages contain two mandatory metadata tables, gpkg_contents and gpkg_spatial_ref_sys. The presence of other metadata tables is dictated by the content being stored (see Content Types). The name of the user-defined data table is the primary key for gpkg_contents and generally is a foreign key for content-specific metadata tables.

Vector feature data are geographic entities including conceptual ones such as districts, real world objects such as roads and rivers, and observations. (An observation is an act that results in the estimation of the value of a feature property, and involves application of a specified procedure, such as a sensor, instrument, algorithm or process chain. A temperature at a given geographic location provided by a sensor is an example of an observation.) For vector feature data, there is one additional required table: gpkg_geometry_columns. Features are stored in the user-defined data tables identified by the table_name values in gpkg_contents (one table per row).

Use zstd for compression, and set the compression level to 15.

Be sure to include the bbox covering, and use GeoParquet version 1.1.

Spatially order the data within the file.

Set the maximum row group size between 50,000 and 150,000 per row.

If the data is larger than ~2 gigabytes consider spatially partitioning the files.

Use STAC Metadata metadata to describe the data.

Parquet has built in compression, enabling users to directly use files that are similar in size to the zipped versions of other formats. You can easily change the compression algorithm, and new ones continue to be added. The default for most Parquet libraries is snappy, which excels at speed and gets good compression. More recently the zstd library has been added to the Parquet ecosystem, and it achieves a better compression ratio with similar speeds to snappy. So it is recommended to use zstd, since at this point most all Parquet libraries support zstd and because better compression makes for faster downloads.

zstd does have a nice ability to control compression, with options ranging up to 22. The cool thing is that decompression times are pretty constant with zstd, so if you're distributing data then it makes a lot of sense to spend a bit more time up to do a higher compression level. Then downloads will go faster, but it won't take clients longer to decompress. Many tools default to one of the lowest compression levels, indeed the core Apache Arrow library that many tools use defaults to 1. So our recommendation is generally to increase the compression level, particularly if you're making data for distribution. But don't bother to go all the way to 22 - the consensus seems to be that the levels 17 and above take way longer, but the size gains are less than one percent. There is more research needed on this topic, but the current recommendation is to aim for something between 11 and 16.

The other new feature is the Native geometry encodings, based on GeoArrow. Using these will enable the same types of speed ups as the bbox covering, but will store the data more efficiently. Parquet readers will be able to use the min/max statistics directly from the geometry column, instead of needing the bbox column. For points this will be a big win, since the bbox for a point adds some overhead. But we do not yet recommend using the native encodings, since the tool support isn't yet extensive enough to be sure that most clients will understand them. But as the ecosystem matures this will be a great option.

GeoParquet 1.1 included a couple new features that help with spatial indexing and querying. The easiest one to use is the bbox covering, which adds a column called bbox that contains the bounding box of each geometry as a native Parquet 'struct' of four values. This enables Parquet readers to quickly filter rows based on the bounding box, and thus greatly increasing the performance of spatial queries. The bbox column by itself is not sufficient to speed up spatial queries - for that you'll need to be sure to follow the next two recommendations. But be sure to include it. It is possible for some tools to make use of the bbox column even if the GeoParquet version is not 1.1, but it's best to actually distribute the files with GeoParquet 1.1 to ensure all tools know they can use the bbox column.

It is essential to make sure that the data is spatially ordered in some way within the file, in order for the bbox column to be used effectively. If the GeoParquet data was converted from a GIS format like GeoPackage or Shapefile then often it will already by spatially ordered. One way to check this is to open the file in a GIS tool and see if the data loads all the spatial data for an area in chunks, or if data for the whole are appears and continues to load everywhere.

GeoParquet itself does not have a specific spatial index like other formats (R-tree in GeoPackage, Packed Hilbert R-tree in FlatGeobuf). Instead data can be ordered in any way, and then Parquet's Row Group statistics will be used to speed up spatial queries (when using bbox covering or native arrow types). Most tools that provide GeoParquet writers have some ability to apply a spatial ordering. The examples below will show how to do this for a few common tools.

A row group in Parquet is 'a logical horizontal partitioning of the data into rows', and there's some good explanation in this article. It ends up being important to get this right, since it will impact the performance of spatial queries. If the row group size is too big then the GeoParquet reader will not be able to 'skip' over large chunks of data and if it's too small then the file metadata can get large, which can slow things down if there are a lot of files.

Unfortunately there's no single 'best' size for row groups, and it will depend on the size of the data and the access patterns. The community is still learning what works best, so there are no solid recommendations at this point - hopefully we'll learn more and update this section in the future. As of this writing there are several larger global datasets that are being distributed with row group sizes of 50,000 to 200,000 rows, so that's what we recommend as a starting point.

Most geospatial tools give you the ability to set the maximum number of rows per row group, but some tools may let you set the byte size for the row group. The core thing that really matters is the byte size for the row group, as that will be the amount of data that needs to be read (and moved over the network in cloud-native geo access patterns). So if your data rows are large then you'll want to set a smaller row group size, and if your rows are small it could make sense to go to the larger end of the spectrum. If you can set the byte size for row groups a common recommendation is to aim for 128MB - 256MB per row group.

One of the useful features of Parquet is the ability to partition a large dataset into multiple files, as most readers can be pointed at a folder of files and it will read them as a single dataset. The reader will use the row group statistics to quickly figure out if a given file needs to be read, and multiple files can be read in parallel. So with spatial data, where most every query contains a spatial filter, partioning the data spatially can greatly accelerate the performance.

Similar to the row group size, the community is still figuring out the best way to spatially partition the data, and the overall query performance will depend on both row group size and the size of the partitioned files, along with the nature of the data. Hopefully someone will do a set of robust testing to help inform more definitive recommendations. For now the recommendation is to spatially partition your data 'in some way', at least if the dataset is larger than a couple gigabytes. If it's smaller than that then the additional overhead of splitting it up is likely not worth it. There was some great discussion on the topic, and an nice blog post with some further experimentation. The leading approach at the moment is to use a K-dimensional tree (KD-tree), which will enable balancing of the file sizes and spatial separation; however, sorts based on S2, GeoHash or R-tree can all work. Partitioning based on admin boundaries is another approach that works and is used in the Google-Microsoft-OSM Buildings - combined by VIDA dataset.

Out of the box GDAL/OGR defaults to snappy compression, with max row group size of 65536. Version 3.9 and later will write out the bbox column by default. And there is a built-in option to spatially order the data that works by creating a temporary GeoPackage file and using its r-tree spatial index. It defaults to false since it can be an intensive operation, and GDAL is usually translating from formats that already have spatial indexes.

GDAL/OGR with recommended settings These examples are done with the `ogr2ogr command-line tool, but the layer creation options will be the same calling from C or Python. You can easily control the compression and the max row group size, and the following command is sufficient if your source data is already spatially ordered in a file format with a spatial index (like FlatGeobuf or GeoPackage): ogr2ogr out.parquet -lco "COMPRESSION=ZSTD" -lco "MAX_ROW_GROUP_SIZE=100000" in.fgb GDAL 3.12 and above introduces COMPRESSION_LEVEL as a Parquet layer creation option. So if you're working with that then you should definitely use it (along with the new gdal CLI, which is used here. gdal vector convert vegetation.fgb vegetation.parquet --lco compression=zstd --lco compression_level=15 If you want to be sure that the output is spatially ordered then you can add SORT_BY_BBOX=YES, like in the following example: ogr2ogr out.parquet -lco SORT_BY_BBOX=YES -lco "COMPRESSION=ZSTD" in.geojson This operation writes the data to a GeoPackage as an interim step, so it can take additional storage and computation, especially with large files, so it's not enabled by default.

None of the tools to write GeoParquet currently write out STAC Metadata, but that makes sense, as they don't write out other metadata formats either. To write STAC metadata you can write it by hand if you've just got one or two GeoParquet files. If you've got more then the best option is to use something like rustac or pystac to do it a bit more programmatically. You should be able to populate some of the STAC fields like bbox from the GeoParquet files directly.