Repository navigation
refactor: read nflverse through nflreadpy - #13
Merged
Merged
Conversation
nfl_data_py is archived and directs its readers to nflreadpy. Its final release predates the nflverse release rename that left `import_weekly_data` reading `player_stats`, whose assets stop at 2024, so every season after that reached the play-by-play reconstruction rather than the weekly stats nflverse publishes. nflreadpy is nflverse's own package, returns polars frames, and reads the releases nflverse publishes to. ─── The seven loaders ─── `load_pbp`, `load_snap_counts`, `load_player_stats`, `load_rosters`, `load_injuries`, `load_schedules` and `load_ff_playerids` replace the nfl_data_py importers one for one. `load_ff_playerids` reads the same dynastyprocess table `import_ids` did, so the pfr_id and gsis_id pairs `build_id_crosswalk` resolves are unchanged. Declining the participation sidecar is no longer a thing a caller can do or needs to: `load_pbp` reads the play-by-play asset alone. ─── What a loader answers with ─── `_nflverse_season` maps three shapes onto one dropped season. A missing asset and a failed transfer both arrive as the `ConnectionError` nflreadpy raises around an HTTP failure, an `OSError`, and the same one either way — which is what keeps a cause out of the message. Some loaders range-check a season before requesting anything and raise `ValueError` for one their release does not reach. The ceiling moves with the calendar and is not the cut `seasons.season_for` makes, so a season this repository resolves can sit above it for the opening days of a September. `load_player_stats` and `load_schedules` do not check, and answer the same season with a 404 and an empty frame instead. nflreadpy's downloader raises `ValueError` for anything that goes wrong once a response has arrived as well. That is a failure to read what was fetched rather than a season the source will not give, so the catch reads the range check's message and carries the rest to the caller: a season silently missing is worse than a load that stops. A loader whose release covers every season in one asset filters it rather than failing, so a season it does not hold returns no rows. A season that yields no rows is a season not obtained, and raising keeps an empty frame out of the cache so the next request asks the source again. ─── Column names ─── nflreadpy spells a column as the release spells it, and the loader returning a frame renames the two that disagree with the frames they are joined to. The roster asset's `gsis_id` becomes `player_id`, which play-by-play, weekly stats and every engine joining them already use. The weekly asset's `team` becomes `recent_team`. Position comes from the weekly asset's `position_group`, not its `position`, which labels a fullback FB and a two-way player by the side of the ball they are listed on — neither of which `OFFENSIVE_POSITIONS` holds. `position_group` resolves both to the side they play and travels in the frame that carries them, so the published path fetches nothing beside the asset and a roster failure cannot empty a season the weekly asset covers in full. ─── Weekly stats ─── The weekly asset updates through a season rather than landing complete at the end of one, so a season being played is read from it. `compute_weekly_stats_from_pbp` covers a season the asset does not carry at all, and reads the rosters for the position play-by-play does not name. `stats.attach_position` carries that join, which the derivation had inline. The published frame is the wider one: the engines read eleven columns off it and it carries a hundred and fifty, including the zero-touch weeks the reconstruction had no rows for. ─── Caching and deadlines ─── nflreadpy holds what it downloaded in memory, keyed by URL. That sits beside `cache`, which stores by season: a release covering every season in one asset is downloaded once for a request naming several, where a per-season store cannot know the two seasons came from one file. Only the memory mode is on, so nothing is written to disk twice. Its per-request timeout is set to the deadline `_download` already holds, which leaves that the only one that fires — a transfer that stalls is reported as a stall rather than as a season the request could not obtain. Both settings are written in code, so neither the environment nor a `.env` beside the process can change how a season is fetched. ─── Dependencies ─── `nflreadpy` is pinned rather than floored: it is pre-1.0 and has moved a positional argument between patch releases. pandas, numpy, fastparquet, cramjam, appdirs and fsspec leave the tree with nfl_data_py, and the pandas constraint under `[tool.uv]` goes with them. ─── Tests ─── `FakeNflreadpy` stands in for the module rather than for the function that reads through it, so the exception mapping, the empty-frame rule and all three renames run under test instead of being replaced by it. `published_rosters` carries the asset's shape, before the loader renames it. A loader is named by a string that both the source and its stand-in can spell the same wrong way, so `test_every_loader_reached_for_exists_on_the_module` reads the names off the source's own syntax tree and checks them against the module. It is the only test a rename landing in both files fails. The empty-frame rule is pinned on the cache key it does not write, since the rows a dropped season contributes are none either way. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
nfl_data_pyis archived and directs its readers tonflreadpy. Its final release predates the nflverse release rename that leftimport_weekly_datareadingplayer_stats, whose assets stop at 2024 — so every season after that reached the play-by-play reconstruction rather than the weekly stats nflverse publishes.nflreadpyis nflverse's own package, returns polars frames, and reads the releases nflverse publishes to.The seven loaders
load_pbp,load_snap_counts,load_player_stats,load_rosters,load_injuries,load_schedulesandload_ff_playeridsreplace the nfl_data_py importers one for one.load_ff_playeridsreads the same dynastyprocess tableimport_idsdid, so the crosswalkbuild_id_crosswalkresolves is unchanged. The participation sidecar the previous commit worked around is not somethingload_pbpreads at all.What a loader answers with
_nflverse_seasonmaps three shapes onto one dropped season: theConnectionErrornflreadpy raises around any HTTP failure; theValueErrorsome loaders raise for a season their release does not reach, whose ceiling moves with the calendar and is not the cutseasons.season_formakes; and a frame with no rows, from a loader whose release covers every season in one asset and filters it.nflreadpy's downloader also raises
ValueErrorfor anything that goes wrong after a response arrives. That is a failure to read what was fetched rather than a season the source will not give, so the catch reads the range check's message and carries the rest to the caller.Column names and position
The roster asset's
gsis_idbecomesplayer_id; the weekly asset'steambecomesrecent_team. Position comes fromposition_group, notposition, which labels a fullback FB and a two-way player by the side of the ball they are listed on — neither of whichOFFENSIVE_POSITIONSholds.position_groupresolves both and travels in the frame that carries them, so the published path fetches nothing beside the asset and a roster failure cannot empty a season the weekly asset covers in full.Caching
nflreadpy's memory cache sits beside
cache: a release covering every season in one asset is downloaded once for a request naming several, where a per-season store cannot know the two seasons came from one file. Only the memory mode is on. Its per-request timeout is set to the deadline_downloadalready holds, so a stalled transfer is reported as a stall.Dependencies
nflreadpyis pinned rather than floored — pre-1.0, and it has moved a positional argument between patch releases. pandas, numpy, fastparquet, cramjam, appdirs and fsspec leave the tree, and the pandas constraint under[tool.uv]goes with them.Verification
ruff check,ruff format --checkandmypy srcclean.FBlabel survives into the frame the engines filter.