No Clocks

gdal vsi sync

Usage: gdal vsi sync [OPTIONS] <SOURCE> <DESTINATION> Synchronize source and target file/directory located on GDAL Virtual System Interface (VSI). Positional arguments: --source <SOURCE> Source file or directory name [required] --destination <DESTINATION> Destination file or directory name [required] Common Options: -h, --help Display help message and exit --json-usage Display usage as JSON document and exit --config <KEY>=<VALUE> Configuration option [may be repeated] -q, --quiet Quiet mode (no progress bar or warning message) [not available in pipelines] Options: -r, --recursive Synchronize recursively --strategy <STRATEGY> Synchronization strategy. STRATEGY=timestamp|ETag|overwrite (default: timestamp) -j, --num-threads <NUM-THREADS> Number of jobs (or ALL_CPUS)

gdal vsi sync synchronize files and directories located on GDAL Virtual File Systems (compressed, network hosted, etc...): /vsimem, /vsizip, /vsitar, /vsicurl, ....

This is an analog to the UNIX rsync command. In the current implementation, rsync would be more efficient for local file copying, but gdal vsi sync main interest is when the source or target is a remote file system like /vsis3/ or /vsigs/, in which case it can take into account the timestamps of the files (or optionally the ETag/MD5Sum) to avoid unneeded copy operations.

This is implemented efficiently for: local filesystem <--> remote filesystem. remote filesystem <--> remote filesystem, where the source and target remote filesystems are the same and one of /vsis3/, /vsigs/ or /vsiaz/. Or when the target is /vsiaz/ and the source is /vsis3/, /vsigs/, /vsiadls/ or /vsicurl/

Similarly to rsync behavior, if the source filename ends with a slash, it means that the content of the directory must be copied, but not the directory name. For example, assuming /home/even/foo contains a file bar, gdal vsi sync -r /home/even/foo/ /mnt/media will create a /mnt/media/bar file. Whereas gdal vsi sync -r /home/even/foo /mnt/media will create a /mnt/media/foo directory which contains a bar file.

Options -r, --recursive Synchronize recursively. --strategy timestamp|ETag|overwrite Determines which criterion is used to determine if a target file must be replaced when it already exists and has the same file size as the source. Only applies for a source or target being a network filesystem. The default is timestamp (similarly to how aws s3 sync works). For an upload operation, a remote file is replaced if it has a different size or if it is older than the source. For a download operation, a local file is replaced if it has a different size or if it is newer than the remote file. The ETag strategy assumes that the ETag metadata of the remote file is the MD5Sum of the file content, which is only true in the case of /vsis3/ for files not using KMS server side encryption and uploaded in a single PUT operation (so smaller than 50 MB given the default used by GDAL). Only to be used for /vsis3/, /vsigs/ or other filesystems using a MD5Sum as ETAG. The overwrite strategy will always overwrite the target file with the source one. -j, --num-threads <value> Number of jobs to run at once

gdal vsi sync -r my_directory/ /vsis3/bucket/my_directory

GDALVector provides an interface for accessing a vector layer in a GDAL dataset and calling methods on the underlying OGRLayer object. An object of class GDALVector persists an open connection to the dataset, and exposes methods to: retrieve layer information set attribute and spatial filters set a list of ignored or selected columns traverse and read feature data by traditional row-based cursor, including an analog of DBI::dbFetch() read via column-oriented Arrow Array stream write new features in a layer edit/overwrite existing features upsert delete features perform data manipulation within transactions.

GDALVector is a C++ class exposed directly to R (via RCPP_EXPOSED_CLASS). Fields and methods of the class are accessed using the $ operator. Note that all arguments to class methods are required and must be given in the order documented. Most GDALVector methods take zero or one argument, so this is usually not an issue. Class constructors are the main exception. Naming the arguments is optional but may be preferred for readability.

Arguments dsn Character string containing the data source name (DSN), usually a filename or database connection string. layer Character string containing the name of a layer within the data source. May also be given as an SQL SELECT statement to be executed against the data source, defining a layer as the result set. read_only Logical scalar. TRUE to open the layer read-only (the default), or FALSE to open with write access. open_options Optional character vector of NAME=VALUE pairs specifying dataset open options. spatial_filter Optional character string containing a geometry in Well Known Text (WKT) format which represents a spatial filter. dialect Optional character string to control the statement dialect when SQL is used to define the layer. By default, the OGR SQL engine will be used, except for RDBMS drivers that will use their dedicated SQL engine, unless "OGRSQL" is explicitly passed as the dialect. The "SQLITE" dialect can also be used.

Value An object of class GDALVector, which contains pointers to the opened layer and the GDAL dataset that owns it. Class methods that operate on the layer are described in Details, along with a set of writable fields for per-object settings. Values may be assigned to the class fields as needed during the lifetime of the object (i.e., by regular <- or = assignment).

## Constructors # for single-layer file formats such as shapefile lyr <- new(GDALVector, dsn) # specifying the layer name, or SQL statement defining the layer lyr <- new(GDALVector, dsn, layer) # for update access lyr <- new(GDALVector, dsn, layer, read_only = FALSE) # using dataset open options lyr <- new(GDALVector, dsn, layer, read_only, open_options) # setting a spatial filter and/or specifying the SQL dialect lyr <- new(GDALVector, dsn, layer, read_only, open_options, spatial_filter, dialect) ## Read/write fields (per-object settings) lyr$defaultGeomColName lyr$returnGeomAs lyr$promoteToMulti lyr$convertToLinear lyr$wkbByteOrder lyr$arrowStreamOptions lyr$quiet lyr$transactionsForce ## Methods lyr$open(read_only) lyr$isOpen() lyr$isReadOnly() lyr$getDsn() lyr$getFileList() lyr$info() lyr$getDriverShortName() lyr$getDriverLongName() lyr$getName() lyr$getFieldNames() lyr$testCapability() lyr$getFIDColumn() lyr$getGeomType() lyr$getGeometryColumn() lyr$getSpatialRef() lyr$bbox() lyr$getLayerDefn() lyr$getFieldDomain(domain_name) lyr$setAttributeFilter(query) lyr$getAttributeFilter() lyr$setIgnoredFields(fields) lyr$setSelectedFields(fields) lyr$getIgnoredFields() lyr$setSpatialFilter(wkt) lyr$setSpatialFilterRect(bbox) lyr$getSpatialFilter() lyr$clearSpatialFilter() lyr$getFeatureCount() lyr$getNextFeature() lyr$setNextByIndex(i) lyr$getFeature(fid) lyr$resetReading() lyr$fetch(n) lyr$getArrowStream() lyr$releaseArrowStream() lyr$setFeature(feature) lyr$createFeature(feature) lyr$batchCreateFeature(feature_set) lyr$upsertFeature(feature) lyr$getLastWriteFID() lyr$deleteFeature(fid) lyr$syncToDisk() lyr$startTransaction() lyr$commitTransaction() lyr$rollbackTransaction() lyr$getMetadata() lyr$setMetadata(metadata) lyr$getMetadataItem(mdi_name) lyr$close()

Constructors new(GDALVector, dsn) The first layer by index is assumed if the layer argument is omitted, so this form of the constructor might be used for single-layer formats like shapefile. new(GDALVector, dsn, layer) Constructor specifying the name of a layer to open. The layer argument may also be given as an SQL SELECT statement to define a layer as the result set. new(GDALVector, dsn, layer, read_only) Constructor specifying read/write access (read_only = TRUE|FALSE). The layer argument is required in this form of the constructor, but may be given as empty string (""), in which case the first layer by index will be assumed. new(GDALVector, dsn, layer, read_only, open_options) Constructor specifying dataset open options as a character vector of NAME=VALUE pairs. new(GDALVector, dsn, layer, read_only, open_options, spatial_filter, dialect)) Constructor to specify a spatial filter and/or SQL dialect. All arguments are required in this form of the constructor, but open_options may be NULL, and spatial_filter or dialect may be an empty string ("").

$defaultGeomColName Character string specifying a name to use for returned columns when the geometry column name in the source layer is empty, like with shapefiles etc. Defaults to "geom".

$returnGeomAs Character string specifying the return format of feature geometries. Must be one of WKB (the default), WKB_ISO, WKT, WKT_ISO, BBOX, or NONE. Using WKB/WKT exports as 99-402 extended dimension (Z) types for Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon and GeometryCollection. For other geometry types, it is equivalent to using WKB_ISO/WKT_ISO (see https://libgeos.org/specifications/wkb/). Using BBOX exports as a list of numeric vectors, each of length 4 with values xmin, ymin, xmax, ymax. If an empty geometry is encountered these values will be NA_real_ in the corresponding location. Using NONE will result in no geometry value being present in the feature returned.

$promoteToMulti A logical value specifying whether to automatically promote geometries from Polygon to MultiPolygon, Point to MultiPoint, or LineString to MultiLineString during read operations (i.e., with methods $getFeature(), $getNextFeature(), $fetch()). Defaults to FALSE. Setting to TRUE may be useful when reading from layers such as shapefiles that mix Polygons and MultiPolygons.

$wkbByteOrder Character string specifying the byte order for WKB geometries. Must be either LSB (Least Significant Byte first, the default) or MSB (Most Significant Byte first).

$arrowStreamOptions Character vector of "NAME=VALUE" pairs giving options used by the $getArrowStream() method (see below). The available options may be driver and GDAL version specific. Options available as of GDAL 3.8 are listed below. For more information about options for Arrow stream, see the GDAL API documentation for OGR_L_GetArrowStream(). INCLUDE_FID=YES/NO. Defaults to YES. MAX_FEATURES_IN_BATCH=integer. Maximum number of features to retrieve in an ArrowArray batch. Defaults to 65536. TIMEZONE=unknown/UTC/(+|:)HH:MM or any other value supported by Arrow (GDAL >= 3.8). GEOMETRY_METADATA_ENCODING=OGC/GEOARROW (GDAL >= 3.8). The GDAL default is OGC if not specified. GEOMETRY_ENCODING=WKB (Arrow/Parquet drivers). To force a fallback to the generic implementation when the native geometry encoding is not WKB. Otherwise the geometry will be returned with its native Arrow encoding (possibly using GeoArrow encoding).

$quiet A logical value, FALSE by default. Set to TRUE to suppress various messages and warnings.

$transactionsForce A logical value, FALSE by default. Affects the behavior of attempted transactions on the layer (see the $startTransaction() method below). By default, only "efficient" transactions will be attempted. Some drivers may offer an emulation of transactions, but sometimes with significant overhead, in which case the user must explicitly allow for such an emulation by first setting $transactionsForce <- TRUE.

$open(read_only) (Re-)opens the vector layer on the existing DSN. Use this method to open a layer that has been closed using $close(). May be used to re-open a layer with a different read/write access (read_only set to TRUE or FALSE). The method will first close an open dataset, so it is not required to call $close() explicitly in this case. No return value, called for side effects.

$isOpen() Returns a logical value indicating whether the vector dataset is open.

$getDsn() Returns a character string containing the dsn associated with this GDALVector object (dsn originally used to open the layer).

$isReadOnly()

Class encapsulating a vector layer in a GDAL dataset Source: R/gdalvector.R GDALVector provides an interface for accessing a vector layer in a GDAL dataset and calling methods on the underlying OGRLayer object. An object of class GDALVector persists an open connection to the dataset, and exposes methods to: retrieve layer information set attribute and spatial filters set a list of ignored or selected columns traverse and read feature data by traditional row-based cursor, including an analog of DBI::dbFetch() read via column-oriented Arrow Array stream write new features in a layer edit/overwrite existing features upsert delete features perform data manipulation within transactions. GDALVector is a C++ class exposed directly to R (via RCPP_EXPOSED_CLASS). Fields and methods of the class are accessed using the $ operator. Note that all arguments to class methods are required and must be given in the order documented. Most GDALVector methods take zero or one argument, so this is usually not an issue. Class constructors are the main exception. Naming the arguments is optional but may be preferred for readability. Arguments dsn Character string containing the data source name (DSN), usually a filename or database connection string. layer Character string containing the name of a layer within the data source. May also be given as an SQL SELECT statement to be executed against the data source, defining a layer as the result set. read_only Logical scalar. TRUE to open the layer read-only (the default), or FALSE to open with write access. open_options Optional character vector of NAME=VALUE pairs specifying dataset open options. spatial_filter Optional character string containing a geometry in Well Known Text (WKT) format which represents a spatial filter. dialect Optional character string to control the statement dialect when SQL is used to define the layer. By default, the OGR SQL engine will be used, except for RDBMS drivers that will use their dedicated SQL engine, unless "OGRSQL" is explicitly passed as the dialect. The "SQLITE" dialect can also be used. Value An object of class GDALVector, which contains pointers to the opened layer and the GDAL dataset that owns it. Class methods that operate on the layer are described in Details, along with a set of writable fields for per-object settings. Values may be assigned to the class fields as needed during the lifetime of the object (i.e., by regular <- or = assignment). Usage (see Details) ## Constructors # for single-layer file formats such as shapefile lyr <- new(GDALVector, dsn) # specifying the layer name, or SQL statement defining the layer lyr <- new(GDALVector, dsn, layer) # for update access lyr <- new(GDALVector, dsn, layer, read_only = FALSE) # using dataset open options lyr <- new(GDALVector, dsn, layer, read_only, open_options) # setting a spatial filter and/or specifying the SQL dialect lyr <- new(GDALVector, dsn, layer, read_only, open_options, spatial_filter, dialect) ## Read/write fields (per-object settings) lyr$defaultGeomColName lyr$returnGeomAs lyr$promoteToMulti lyr$convertToLinear lyr$wkbByteOrder lyr$arrowStreamOptions lyr$quiet lyr$transactionsForce ## Methods lyr$open(read_only) lyr$isOpen() lyr$isReadOnly() lyr$getDsn() lyr$getFileList() lyr$info() lyr$getDriverShortName() lyr$getDriverLongName() lyr$getName() lyr$getFieldNames() lyr$testCapability() lyr$getFIDColumn() lyr$getGeomType() lyr$getGeometryColumn() lyr$getSpatialRef() lyr$bbox() lyr$getLayerDefn() lyr$getFieldDomain(domain_name) lyr$setAttributeFilter(query) lyr$getAttributeFilter() lyr$setIgnoredFields(fields) lyr$setSelectedFields(fields) lyr$getIgnoredFields() lyr$setSpatialFilter(wkt) lyr$setSpatialFilterRect(bbox) lyr$getSpatialFilter() lyr$clearSpatialFilter() lyr$getFeatureCount() lyr$getNextFeature() lyr$setNextByIndex(i) lyr$getFeature(fid) lyr$resetReading() lyr$fetch(n) lyr$getArrowStream() lyr$releaseArrowStream() lyr$setFeature(feature) lyr$createFeature(feature) lyr$batchCreateFeature(feature_set) lyr$upsertFeature(feature) lyr$getLastWriteFID() lyr$deleteFeature(fid) lyr$syncToDisk() lyr$startTransaction() lyr$commitTransaction() lyr$rollbackTransaction() lyr$getMetadata() lyr$setMetadata(metadata) lyr$getMetadataItem(mdi_name) lyr$close() Details Constructors new(GDALVector, dsn) The first layer by index is assumed if the layer argument is omitted, so this form of the constructor might be used for single-layer formats like shapefile. new(GDALVector, dsn, layer) Constructor specifying the name of a layer to open. The layer argument may also be given as an SQL SELECT statement to define a layer as the result set. new(GDALVector, dsn, layer, read_only) Constructor specifying read/write access (read_only = TRUE|FALSE). The layer argument is required in this form of the constructor, but may be given as empty string (""), in which case the first layer by index will be assumed. new(GDALVector, dsn, layer, read_only, open_opti

$isReadOnly() Returns a logical value, TRUE if the vector dataset has read-only access or FALSE if it is writable.

$getFileList() Returns a character vector of files believed to be part of the data source. If it returns an empty string ("") it means there is believed to be no local file system files associated with the dataset (e.g., a virtual file system). The returned filenames will normally be relative or absolute paths depending on the path used to originally open the dataset.

$info() Prints information about the vector layer to the console (no return value, called for that side effect only). For non-SQL DSN/layer, calls ogrinfo() passing the command options cl_arg = c("-so", "-nomd"), and for layers open with a SQL statement, calls ogrinfo() passing the command options cl_arg = c("-so", "-nomd", "-sql", <statement>).