Skip to content
This repository was archived by the owner on Nov 5, 2018. It is now read-only.

GPII-3250: Pre-warm the data loaded by dataloader #7

Merged
seg10 merged 3 commits into from
Sep 6, 2018
Merged

GPII-3250: Pre-warm the data loaded by dataloader #7

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
be6ec65
GPII-3250: Pre-warm the data loaded by dataloader
Sep 4, 2018
4468a3e
Decide what to pre-warm based on querying available views
Sep 5, 2018
52387ec
Install jq as part of the docker image
Sep 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions loadData.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ log() {
echo "$(date +'%Y-%m-%d %H:%M:%S') - $1"
}

warm(){
VIEW=$1

curl -fsS $COUCHDB_URL/_design/views/_view/$VIEW >/dev/null

@natarajaya natarajaya Sep 4, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very expensive query, since it executes view function against ALL data in DB. I think this works now only because we have small number of records.

@seg10 seg10 Sep 5, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One option to reduce the cost of this query might be to run it immediately after the creation of the view when. This would require a higher degree of coupling with the test data however and I'm not sure I like the tradeoff considering that load data on a new DB should be pretty infrequent.

}

loadData() {
log "Loading data from $1"

Expand Down Expand Up @@ -47,3 +53,10 @@ fi
# Submit data
loadData $STATIC_DATA_DIR
loadData $BUILD_DATA_DIR

# Warm Data
log "Warming indices..."
warm findPrefsSafeByGpiiKey
Comment thread
seg10 marked this conversation as resolved.
Outdated
warm findClientByOauth2ClientId
warm findAuthorizationByAccessToken
log "Finished warming indices..."