AB_SILC_construction_files - #188
Merged
Merged
Conversation
…5_2023_panel/.DS_Store
…5_2023_panel/do_files/.DS_Store
…5_2023_panel/data/.DS_Store
…ths/SimPathsEU into AB_SILC_construction_files
Updated README to improve formatting and clarity of the data construction process for SimPaths using EU-SILC data.
…5_2023_panel/data/EU-SILC/.DS_Store
There was a problem hiding this comment.
Pull request overview
This PR adds Stata do-files and READMEs to support constructing EU‑SILC longitudinal panel data (2005–2023) and building/validating Poland (PL) UID inputs for SimPaths EU.
Changes:
- Added an extended EU‑SILC 2005–2023 panel construction workflow (master + masterH/R/P + weight rescaling).
- Added PL UID construction do-files for benefit-unit creation, reweighting, slicing/refactoring, and yearly checks.
- Added helper scripts/docs for merging selected variables from the 2005–2020 panel into the 2005–2023 panel, plus workflow READMEs.
Reviewed changes
Copilot reviewed 9 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| input_processing/regression_estimates/PL/00_master_regression_estimates_PL.do | Adds a note clarifying prerequisites (conditions master). |
| input_processing/data_construction/UID_construction/PL/extra_var_info/vars_05_20_PL2.do | New helper to extract legacy variables from 2005–2020 panel for later merge. |
| input_processing/data_construction/UID_construction/PL/extra_var_info/prepare_pooled_data _05_20.do | New helper to compile a 2005–2020 pooled dataset used for legacy-variable extraction. |
| input_processing/data_construction/UID_construction/UID_construction/PL/07_version_comparison_PL.do | New diagnostic script to compare “new vs old” versions and investigate drops. |
| input_processing/data_construction/UID_construction/PL/06_check_yearly_data_PL.do | New script to generate yearly summary stats / checks for PL initial populations. |
| input_processing/data_construction/UID_construction/PL/05_drop_hholds_slice_and_refactoring_PL.do | New script to drop incomplete households, adjust weights to population totals, slice to cross-sections, and rename to “refactored” codebook variables. |
| input_processing/data_construction/UID_construction/PL/04_reweight_PL.do | New script to reweight using inverse-probability model of household completeness and slice “pre-drop” yearly files. |
| input_processing/data_construction/UID_construction/PL/03_create_benefit_units_PL.do | New core script to construct benefit units, enforce consistency rules, flag/drop invalid records, and update derived variables. |
| input_processing/data_construction/UID_construction/PL/01_prepare_pooled_data_PL.do | New script to build the pooled PL panel dataset by merging R/P/D/H master files. |
| input_processing/data_construction/UID_construction/PL/00_master_data_set_construction_PL.do | Updates notes to point to PL-specific conditions master. |
| input_processing/data_construction/SILC_panel_construction/README_SILC_panel_construction.txt | New documentation describing why/ how the 2005–2020 + 2005–2023 panels are combined. |
| input_processing/data_construction/SILC_panel_construction/2005_2023_panel/do_files/05_weights.do | New weight-rescaling do-file (includes optional cleanup of intermediates). |
| input_processing/data_construction/SILC_panel_construction/2005_2023_panel/do_files/04_create_masterP.do | New script to build masterP across releases (2005–2023). |
| input_processing/data_construction/SILC_panel_construction/2005_2023_panel/do_files/03_create_masterR.do | New script to build masterR across releases (2005–2023). |
| input_processing/data_construction/SILC_panel_construction/2005_2023_panel/do_files/02_create_masterH.do | New script to build masterH across releases (2005–2023). |
| input_processing/data_construction/SILC_panel_construction/2005_2023_panel/do_files/00_master_construct_eusilc_panel.do | New master script to run the end-to-end EU‑SILC panel build (imports + master build + weights). |
| input_processing/data_construction/README_data_construction.txt | New high-level documentation for SimPaths input dataset construction workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| do "$do_files_dir/Long_allcountries_Rfiles_`i'.do" | ||
| } | ||
| */ |
| * DEFINE DIRECTORIES | ||
| *******************************************************************************/ | ||
|
|
||
| global main_dir "/Users/aburdett/Library/CloudStorage/Box-Box/ESPON - OVERLAP/_countries/Cumulative Longitudional Dataset (all countries)/2005_2023_panel" // master folder for processing |
Comment on lines
+35
to
+39
| /* Erasing superflous files from the disc */ | ||
|
|
||
| erase "${datapath}/EU-SILC/19 L-2023/2023D.dta" | ||
| // erase "${datapath}/EU-SILC/19 L-2023/2023H.dta" | ||
| erase "${datapath}/EU-SILC/19 L-2023/2023P.dta" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added: