No Clocks

Now we’ll talk a bit about where the real performance gains happen: vectorization, parallelization, and memory management.

Hardware utilization refers to how code leverages computational resources. For example, vectorization and parallelization help us squeeze every last drop of juice from our CPUs, while in-place modification, object size pre-allocation, and on-demand data access are useful to manage memory usage.

Vectorization refers to the application of an operation to multiple elements simultaneously.

At the hardware level, vectorization is enabled by an architectural feature known as Single Instruction Multiple Data (SIMD). SIMD operations can, for example, sum 16 pairs of vector elements simultaneously within a single core, offering substantial speed-ups. However, only compiled languages (C, C++, Fortran, etc) can leverage SIMD instructions via specific compiler optimizations.

At the software level, many languages implement vectorized semantics. Think of adding two vectors b and c with the expression a = b + c. This abstraction makes code concise, and can also unlock performance gains in different ways. In compiled languages like Fortran, such expressions are typically optimized for SIMD vectorization

In interpreted languages like R, many vectorized functions are backed by compiled code. For instance, primitives like + are implemented as fast C loops, that may or may not be optimized for SIMD by the compiler (see the section R side: how can R possibly use SIMD? in this excellent StackOverflow answer for details). In contrast, matrix operations rely on blazing-fast matrix algebra backends such as BLAS and LAPACK, which explicitly exploit SIMD vectorization (and parallelization!).

However, it’s not uncommon to find vectorized semantics without performance gains. This is the case with R functions like apply(), lapply(), purrr::map(), and the likes, which are essentially loops in a trenchcoat.

By combining SIMD vectorization for raw performance with semantics-level vectorization for expressiveness, we maximize hardware utilization while keeping our code clean and efficient

GDAL supports the ability to hold datasets in a temporary in-memory format. This is primarily useful for temporary datasets in scripts or internal to applications. It is not generally of any use to application end-users. This page documents its vector capabilities. Starting with GDAL 3.11, this driver has been unified with the long-time existing Memory driver to offer both raster and vector capabilities. This driver implements read and write access layers of features contained entirely in memory. This is primarily useful as a high performance, and highly malleable working data store. All update options, geometry types, and field types are supported. There is no way to open an existing vector MEM dataset. It must be created with Create(name, 0, 0, 0, GDT_Unknown) and populated and used from that handle. When the dataset is closed all contents are freed and destroyed. The driver does not implement spatial or attribute indexing, so spatial and attribute queries are still evaluated against all features. Fetching features by feature id should be very fast (just an array lookup and feature copy).

Arrow VSI file system Added in version 3.10. Starting with GDAL 3.10 and Arrow 16.0, any GDAL Virtual File System can be used (in a read-only context) wherever the Arrow C++ library expects a URI, in particular outside of the context of the OGR Arrow driver, by: loading the libgdal.so/dll library (or the ogr_Arrow.so/dll plugin library if the Arrow driver is built as a library) with the arrow::fs::LoadFileSystemFactories() function (cf Defining new filesystems) Note: if the Arrow driver is fully loaded, e.g. by querying GetGDALDriverManager()->GetDriverByName("ARROW")->GetMetadata(), the Arrow VSI file system will be also registered. Prefixing any GDAL file name with the gdalvsi:// URI scheme prefix. In addition to any potential vsi prefix in the GDAL file name. So the /vsicurl/http://example.com GDAL file name becomes the gdalvsi:///vsicurl/http://example.com Arrow URI.

Starting with GDAL 3.12, gdal vector info can be used as the last step of a pipeline.

-f, --of, --format, --output-format json|text

Which output format to use. Default is JSON, and starting with GDAL 3.12, text when invoked from command line.

-l, --layer, --input-layer <INPUT-LAYER>

Name of one or more layers to inspect. If no layer names are passed and --sql is not specified, then all layers will be selected.

--summary

Print a summary with the list of layers and the geometry type of each layer. This option is mutually exclusive with the --features option.

--features

List all features by default, unless limited with --limit. Beware of RAM consumption on large layers when using JSON output. This option is mutually exclusive with the --summary option.

--limit <FEATURE-COUNT>

Limit the number of features reported per layer. When set, this implies --features.

--sql <statement>|@<filename>

Execute the indicated SQL statement and return the result. The @<filename> syntax can be used to indicate that the content is in the pointed filename (e.g @my_select.txt where my_select.txt is a file in the current directory). Data can also be edited with SQL INSERT, UPDATE, DELETE, DROP TABLE, ALTER TABLE etc if the dataset is opened in update mode. Editing capabilities depend on the selected dialect with --dialect.

--where <WHERE>|@<filename>

An attribute query in a restricted form of the queries used in the SQL WHERE statement. Only features matching the attribute query will be reported. The @<filename> syntax can be used to indicate that the content is in the pointed filename.

Example of --where and quoting: --where "\"Corner Point Identifier\" LIKE '%__00_00'"

This option is mutually exclusive with the --sql option.

SQL dialect. In some cases can be used to use (unoptimized) OGR SQL dialect instead of the native SQL of an RDBMS by passing the OGRSQL dialect value. The SQL SQLite dialect can be selected with the SQLITE and INDIRECT_SQLITE dialect values, and this can be used with any datasource.