Before we can integrate, publish, or analyze datasets, we need to understand what we are working with. This involves answering questions such as: Does the dataset contain Personally Identifiable Information (PII)? What is its size on disk? Is it spatially enabled? What Coordinate Reference System (CRS) does it use? What is the update frequency? What are the spatial and temporal resolutions? What file format is it stored in? Who is responsible for the dataset?

File size, file format, and in some structured formats, the CRS can be determined through direct inspection.

Enable Cross-origin resource sharing (CORS)

The adapter approach is more expensive to build and dramatically easier to maintain. Each platform’s raw API response flows through its adapter and emerges as a normalized LeagueData object with a consistent shape: league settings, team rosters, schedule, draft history, scoring categories, roster slot counts, and metadata.

The analytics layer never touches the raw platform response. It only reads the normalized output. Every player object carries the same fields regardless of platform: positions, platformId, draftRank, proTeam, dateOfBirth, percentOwned. Every roster entry carries slot (a position string like ‘SP’ or ‘WR’ or ‘BN’) and acquisitionType. The raw platform shape is fully consumed inside the adapter for normal operation. ESPN’s settings retry is the explicit exception: when the initial settings response is incomplete, the raw ESPN payload is held briefly under S._rawEspn so a second fetch can merge richer fields and the adapter can re-run against the augmented payload.

This separates the data translation problem from the analysis problem. When Yahoo changes its API response format, one function changes. When a new platform is added, a new adapter is written and the entire pipeline inherits it without modification. The normalization logic is also auditable in isolation, which matters for verifying that the tool is reading the data correctly.

Yahoo’s adapter has its own dialect. Yahoo’s stat IDs disagree with ESPN’s, so the adapter maps them to the canonical set the pipeline uses. Yahoo provides positions as strings rather than slot integers, which is one less translation. Yahoo’s OAuth flow introduces a lifecycle problem the ESPN connection doesn’t have: access tokens expire after one hour, so the server-side proxy silently refreshes them using the stored refresh token and passes new credentials back to the client. If a user returns after the four-hour window has elapsed, Yahoo asks them to authorize again, which is the price of using OAuth that takes its expiry seriously.

All four adapters stamp a platform field on the normalized output. Every function downstream that needs platform-specific behavior reads S.league.platform rather than maintaining its own detection. The source of truth is in one place.

A source is any input that assigns ranked positions or projected statistics to players. FantasyPros ECR is a source. A FanGraphs Steamer CSV is a source. The connected platform’s internal rankings are a source. NFBC ADP is a source. Each measures something slightly different.

ECR is aggregated expert opinion, a consensus of analysts who have thought carefully about how the season will go, adjusted for positional context and draft format. Projection systems like Steamer are statistical models built on historical performance, aging curves, park factors, and regression to mean. They know nothing about narrative or reputation. ADP is a market price, what actual participants in real drafts have been paying for players. It carries behavioral patterns, recency bias, league-size effects, and the accumulated heuristics of the fantasy-playing population. These three things diverge for structural reasons. Experts may underweight regression toward the mean on breakout candidates. Projection models don’t capture role uncertainty or injury history that isn’t yet in the stats. ADP reflects what the market believes, which is often a lagged version of what the analysis shows.

Z-scores normalize player projections across scoring categories by measuring each player’s projected contribution as a deviation from the position-eligible player mean, in units of standard deviation. The normalization is necessary because raw counting stats aren’t comparable across categories. A projection of 30 home runs and a projection of 40 stolen bases can’t be directly summed into a value number without first converting them to a common scale.

Shapefiles frequently fail during bulk ingestion due to implicit encoding mismatches, malformed .prj definitions, and untyped attribute columns. Execute a deterministic validation gate before triggering conversion jobs.

Extract Metadata Deterministically

Parse featureCount, geometryType, CRS, and field definitions from the JSON. Reject datasets where featureCount is -1 or unknown; these indicate a corrupted .shx index, which you regenerate by rewriting the dataset (ogr2ogr regenerated.shp input.shp).

Enforce CRS Synchronization: Missing or legacy WKT1 .prj files cause downstream projection drift. Normalize explicitly:

If the output is ambiguous, force EPSG:4326 or a project-specific projected CRS using ogr2ogr -t_srs EPSG:XXXX. Store the resolved EPSG code directly in the GeoParquet geo metadata block. Do not rely on implicit CRS inference. Refer to CRS Synchronization in Pipelines for standardized projection registries.

Map DBF Types to Arrow Primitives: DBF lacks native boolean, date, or 64-bit integer support. Apply explicit type coercion during ingestion: | DBF Type | Arrow Primitive | Coercion Logic | |----------|----------------|----------------| | String(254) | large_string | Truncate with audit log if >254 chars | | Numeric(10,2) | float64 | Preserve precision; reject NaN unless explicitly allowed | | Date(YYYYMMDD) | date32 | Parse via pd.to_datetime(..., format='%Y%m%d') | | Logical | boolean | Map T/F/Y/N/1/0 → True/False |

Log any field exceeding 254 characters to a compliance manifest before truncation. Reject implicit type promotion to prevent silent data loss.

Monolithic ogr2ogr invocations exhaust memory and stall on terabyte-scale archives. Implement a chunked, parallelized pipeline with strict resource boundaries.

export GDAL_NUM_THREADS=ALL_CPUS export OGR_MAX_BUFFER_SIZE=512000000 export CPL_DEBUG=ON export SHAPE_ENCODING=UTF-8 export GDAL_CACHEMAX=2048