Dbdbml
tabde/R/utils-sfmisc.R at master · s-fleck/tabde
Work with table designs (col_names, col_types) in csv format - s-fleck/tabde
tabde/R/as_sql.R at master · s-fleck/tabde
Work with table designs (col_names, col_types) in csv format - s-fleck/tabde
From API to Database: A Step-by-Step Guide on Efficient Data Integration
A helpful step-by-step guide to learn about the benefits of integrating APIs with databases and how to do it.
The integration process typically follows a request-response cycle where client applications send HTTP requests to API endpoints, which then execute corresponding database operations. Middleware layers often handle authentication, rate limiting, and data transformation between API responses and database schemas.
Modern integrations also support batch processing for high-volume operations and implement caching strategies to reduce database load while maintaining data freshness.
Command Query Responsibility Segregation (CQRS)
CQRS separates read and write operations into distinct models, optimizing each for its specific purpose. In API-database integrations, this pattern allows you to design write-optimized database schemas for data ingestion while maintaining read-optimized views for query performance.
Event-Driven Architecture
Event-driven patterns complement traditional request-response API integrations by enabling asynchronous processing and loose coupling between systems. Instead of direct API calls to databases, events trigger database operations, allowing for better fault tolerance and scalability.
Microservices Database Patterns
The database-per-service pattern ensures each microservice maintains its own database, preventing tight coupling while enabling independent scaling and deployment. When implementing API-database integrations in microservices architectures, each service exposes its own API for data access rather than allowing direct database connections.
How Do You Implement Change Data Capture and Real-Time Synchronization?
Change Data Capture (CDC) represents a fundamental shift from traditional batch processing to real-time data synchronization, enabling organizations to react immediately to data changes and maintain current information across systems.
Storing External Requests
I’ve worked in payment systems for a long time, and one common theme is that they make a lot of 3rd party API calls. For example, there are numerous payment networks and banks, fraud and risk checks, onboarding and KYC/KYB (Know Your Customer/Business) services, and more. And then there are the inbound calls as well, such as webhooks and callbacks from integration partners, e-commerce stores, and other outside interactions.
The first step is storing all inbound and outbound API calls. People will often use logs for this, but I think it’s far more valuable to put them in the database instead. This makes them easier to query, easier to aggregate, and easier to join against other data.
You can use one table for both inbound and outbound or separate them depending on preferences, but generally it’s useful to store most of the available information, such as:
URL of the request
Datetime of the request
Request body
Response body
Response status/code (e.g. 200, 500)
Total time spent on the request
Request headers
Response headers
Metadata
For metadata, I like to use a JSON column and add in any metadata that links this request to an entity in the system. For example, it could include user_id, order_id, request_id, etc. As a JSON column, you can include more than one, and even include other complex nested information.
Response headers often include extra debugging information, such as request or trace Ids, version numbers, etc. It’s common when asking for 3rd party support to provide these values so they can go look in their own logging to find your requests.
Rather than try to write code for every API call, it’s often better to hook into the request/response lifecycle in one place and instrument all calls.
The way you do this depends on the language and libraries, but they are generally called interceptors.
Where possible, I like to record the request fields before the outbound call is started (e.g. request body, request headers) and then go back and update the row to store the response fields once the call is completed. There are several advantages over a single write at the end of the request cycle:
feddelegrand7/rlowdb: Lightweight, JSON-based database for R 🦩🦩🦩
Lightweight, JSON-based database for R 🦩🦩🦩
Customising condition messages — topic-condition-customisation
Various aspects of the condition messages displayed by abort(), warn(), and inform() can be customised using options from the cli package.
mkquartodocs
mkquartodocs extension
GIS
Understanding Slope and Aspect (Terrain Analysis)
Slope and Aspect (Terrain Analysis)
Slope 101
Slope is an important data point in insurance, enabling companies and agents to accurately price their policies.
PostGIS Performance: pg_stat_statements and... | Crunchy Data Blog
PostGIS performance basics. Second post in a series covering pg_stat_statements, shared buffers, work_mem, and parallel queries.
Using Cloud Rasters with PostGIS | Crunchy Data Blog
Paul shows you how to access raster data stored in the cloud or object storage for PostGIS using cloud optimized GeoTIFF (aka COG) files. He also includes some functions for working with raster elevation.
How to Determine Land Value: A Comprehensive Guide | AcreValue
Learn how to determine land value with key factors and methods. Explore how to utilize informed decisions with data-driven insights.
FEMA Flood Zones: Deep Dive | AcreValue
Explore AcreValue’s FEMA Flood Zones map layer to assess flood risk, protect investments, and make informed land decisions with real-time floodplain data.
Farmland Values, Soil Survey, & GIS Maps | AcreValue
View maps featuring agricultural data, sold land, and land for sale. Discover farmland values, soil productivity, crop mix, and parcel ownership.
R: Calculate slope and aspect from elevation data.
Digital Soil Mapping with R – digital-terrain-analysis
USGS 3DEP LiDAR Point Clouds - Registry of Open Data on AWS
OpenTopography for Developers | OpenTopography
APIsAPI access to USGS 3DEP rasters now available | OpenTopography
https://apps.nationalmap.gov/downloader/#/
AcuGIS | GIS Hosting Services
Complete GIS Hosting for PostGIS, GeoServer, Lizmap, GeoNode, QWC2, and much more. Your choice of 21 hosting locations worldwide. Everything you need to build, deploy, and host your GIS applications. Hosting individuals, organizations, and leading academic and research institutions in over 80 counties worldwide.
Tidy design principles - Error constructors
An Example of the DRY/DAMP Principles for Package Tests
rOpenSci’s second cohort of Champions has been onboarded!
Their training first started with a session on code style, was followed by three sessions on the basics of R package development, and ended with a session on advanced R package development, which consisted of a potpourri of tips with discussion, followed by time for applying these principles to the participants’ packages.
Here, I want to share one of the topics covered: Package testing, and in particular, the DRY (“don’t repeat yourself”) and DAMP (“descriptive and meaningful phrases”) principles.
For this topic, we used a GitHub repository, containing an R package whose different commits illustrate the two principles. In each step we’ll share a commit or diff illustrating the changes made.
S3 vectors
Data Dictionary | BatchData
The Data Dictionary is a collection of all of the fields, descriptions, and data types found in BatchData services. Powered by Stoplight.
batchdata | Stoplight
The API Design Management Platform powering the world's leading API first companies. Powered by Stoplight.
Data — DIVA-GIS