No Clocks

Where the “package 1”, “package 2”, etc. folders are actually be git submodules pointing to separate repositories. The renv folder defines the package versions used by the workflows and the depends/imports/suggests used by the various packages, ensuring consistent behavior across the workflows. Technically you don’t need to have the submodules at all (you could just install the packages in the project environment using renv::install() and the repository url) but I found myself developing the packages and workflows side-by-side, so it was convenient to have everything in one place and tell renv to install from the package folders (e.g., renv::install("./package 1")).

Pipelines run by targets create a variety of intermediate files that capture target outputs and identify their completeness and currentness. This works fine for smaller projects, but we store model inputs and outputs in a series of Microsoft SQL Server databases. Furthermore, the organizational structure of the database can be pretty different from how we would organize data if we were able to maintain R objects, so some of our targets need to write multiple tables to the database. This presents a problem, since targets needs to check the return value of each target and produce a hash. My solution was to create a function that generates a checksum of a database table (or a list of checksums for a list of tables):

The obvious and more well known SQL way to keep some transient state is via temp tables! They give some nice data type guarantees, performance, editor happiness to name a few benefits. But - don’t use them for high frequency use cases! A few temp tables per second might already be too much and a disaster might be waiting to happen…Because CREATE TEMP TABLE actually writes into system catalogs behind the scenes, which might not be directly obvious… And in cases of violent mis-use - think frequent, short-lived temp tables with a lot of columns, plus unoptimized and overloaded Autovacuum together with long-running queries - can lead to extreme catalog bloat (mostly on pg_attribute) and unnecessary IO for each session start / relcache filling / query planning. And it’s also hard to recover from without some full locking - so that for critical high velocity DB’s it might be a good idea to revoke temp table privileges altogether - for app / mortal users at least (not possible for superusers).

The 2nd most obvious way to keep some DB-side session state around would probably be to use more persistent normal tables, right? Already better than temp tables as no danger of bloating the system catalog, right? NO. Pushing transient data though WAL (including replicas and backup systems) is pretty bad and pointless and only to be recommended for tiny use cases. In the Postgres world, exactly for these kinds of transient use cases, special UNLOGGED tables should be used! Which can relieve the IO pressure on the system / whole cluster considerably. One of course just needs to account for the semi-persistent nature - and the fact that they won’t be private anymore. Meaning usage of RLS in case of secret data or just using some random enough keys to avoid collisions.

Stop playing “diagnostics ping-pong” with your users. This post explores why the _sitrep() (situation report) pattern — popularized by the usethis package — is a game-changer for R packages wrapping APIs or external software. Learn how to build structured validation functions that power both early error-handling and comprehensive system reports, featuring real-world implementation examples from the meetupr and freesurfer packages.

Geospatial APIs are software abstraction layers that provide standardized methods to query, analyze, and visualize spatial data from diverse sources. They support essential functionalities including 2D/3D map rendering, geocoding, coordinate transforms, and real-time sensor data integration. Modern designs employ RESTful architectures and OGC standards to enhance interoperability, performance, and scalability across geospatial applications.

A geospatial Application Programming Interface (API) is a software abstraction layer—typically a web service or client library—that exposes standardized operations for querying, rendering, analyzing, and modeling spatial data, including vector features, raster coverages, multi-dimensional sensor observations, and geospatial attributes. Geospatial APIs are foundational for scientific computing, urban analytics, planetary research, public health surveillance, and geospatial-AI workflows, enabling programmable access to distributed spatial resources and seamless integration across data repositories, sensor infrastructures, visualization platforms, and analytic pipelines.