EDSC-3612, make search results export asynchronous by DanielJDufour · Pull Request #1588 · nasa/earthdata-search

DanielJDufour · 2023-02-13T22:16:05Z

Overview

What is the feature?

Please summarize the feature or fix.

What is the Solution?

Summarize what you changed.

What areas of the application does this impact?

List impacted areas.

Testing

Reproduction steps

Environment for testing:
Collection to test with:

Step 1
Step 2...

Attachments

Please include relevant screenshots or files that would be helpful in reviewing and verifying this change.

Checklist

I have added automated tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings

… message

…chRequest

…sts__/handler.test.js

…-sqs, adding scripts for installing and running sqs and s3 simulations

…t S3 buckets, used by search exporting

…ss-configs/aws-resources.yml

…port functions

…ults

…ocking of randomUUID

…d moto

…onnections while running tests in parallel

…test

…ng local development

DanielJDufour · 2023-02-27T15:40:09Z


+  exportSearchRequest:
+    handler: serverless/src/exportSearchRequest/handler.default
+    timeout: 300 # this lambda isn't restricted to 30s because it's triggered by SQS not API Gateway


move comment down

DanielJDufour · 2023-02-27T15:40:30Z

+  exportSearchRequest:
+    handler: serverless/src/exportSearchRequest/handler.default
+    timeout: 300 # this lambda isn't restricted to 30s because it's triggered by SQS not API Gateway
+    memorySize: 256


keep memory default

DanielJDufour · 2023-02-27T15:42:20Z

+    Properties:
+      QueueName: ${self:custom.siteName}-SearchExportQueue
+      ReceiveMessageWaitTimeSeconds: 20
+      VisibilityTimeout: 300


make larger than lambda timeout

DanielJDufour · 2023-02-27T15:50:40Z

    const authOptionalPaths = [
      '/autocomplete',
-      '/opensearch/granules',
-      '/collections/export'


put /collections/export back in

DanielJDufour · 2023-02-27T15:51:46Z

+const MOCK_KEY = '00000000-0000-0000-0000-000000000000' // see /__mocks__/crypto.js
+
+// over-rides randomUUID to return the MOCK_KEY
+jest.mock('crypto')


remove, for uuid package

DanielJDufour · 2023-02-27T16:03:16Z

@@ -0,0 +1,124 @@
+import { randomUUID } from 'crypto' // if we do 'node:crypto', jest.mock won't work


use older library

DanielJDufour · 2023-02-27T16:04:47Z

+        const earthdataEnvironment = determineEarthdataEnvironment(headers)
+
+        const jwt = getJwtToken(event)
+        if (!jwt) throw Error("missing jwt")


make jwt optional

DanielJDufour · 2023-02-27T16:05:01Z

+
+        const { id: userId } = getVerifiedJwtToken(jwt, earthdataEnvironment)
+
+        if (!userId) throw Error("failed getting userId from jwt")


auth is optional now

DanielJDufour · 2023-02-27T16:06:38Z

+            variables
+        }
+
+        const key = randomUUID()


use uuid lib

DanielJDufour · 2023-02-27T16:09:26Z

+
+        const dbConnection = await getDbConnection()
+
+        await dbConnection('exports').insert({


DanielJDufour · 2023-02-27T16:10:10Z

+        const searchExportQueueUrl = getSearchExportQueueUrl()
+        console.log('searchExportQueueUrl:', searchExportQueueUrl)
+
+        await sqs.sendMessage({


DanielJDufour · 2023-02-27T16:11:25Z

+            QueueUrl: searchExportQueueUrl,
+            MessageBody: messageBody
+        }).promise()
+        console.log('posted to search export queue')


prefix with lambda name, proper caps

and requestId!

DanielJDufour · 2023-02-27T16:13:03Z

+            statusCode: 200,
+            headers: {
+                ...defaultResponseHeaders,
+                'jwt-token': jwt


DanielJDufour · 2023-02-27T16:13:42Z

@@ -1,125 +1,189 @@
+import crypto from 'node:crypto'


uuid lib instead

DanielJDufour · 2023-02-27T16:14:08Z

  context.callbackWaitsForEmptyEventLoop = false

-  const { body, headers } = event
+  if (process.env.IS_OFFLINE || process.env.JEST_WORKER_ID) {


re-evaluate

DanielJDufour · 2023-02-27T16:14:42Z

+  if (!filename) throw new Error("missing filename")
+  if (!key) throw new Error("missing key")
+  if (!requestId) throw new Error("missing requestId")
+  if (!userId) throw new Error("missing userId")


DanielJDufour · 2023-02-27T16:15:13Z

+  if (!userId) throw new Error("missing userId")

-  const { data, requestId } = JSON.parse(body)
+  const updateState = (state) => dbConnection('exports').where({ user_id: userId, key }).update({ state })


userId optional

macrouch · 2023-02-27T16:41:14Z

+    "s3": "EXTRA_CORS_ALLOWED_ORIGINS=http://localhost:5000 MOTO_ALLOW_NONEXISTENT_REGION=True moto_server -H 0.0.0.0",
+    "s3:install": "brew install moto",
+    "s3:reset": "curl -X POST http://localhost:5000/moto-api/reset",
+    "sqs": "java -jar elasticmq-server-1.3.9.jar",


I'd rather have sqs offline set up the same way as other projects our team manages, with the docker compose file.

I hate the idea of running extra processes in order to run EDSC locally. This PR for serverless offline sqs shows that it is possible to disable sqs offline. So we'd get the benefit of queues locally, but only if the dev wants them. I don't think that is happening, but it is easy enough to open a new PR that is up to date and install the forked version of the plugin

DanielJDufour · 2023-05-02T12:45:16Z

superseded by #1630

DanielJDufour added 30 commits February 13, 2023 15:55

EDSC-3612: moved and refactored jsonToCsv.js

ef5786e

EDSC-3612: removed old version of jsonToCsv

73eefad

EDSC-3612: added exportSearchRequest

e57d59b

EDSC-3612: added deleteBucket.js

59018aa

EDSC-3612: refactored exportSearch to only work when triggered by SQS…

b1aa728

… message

EDSC-3612: added papaparse and sendero to dependencies

2831db4

EDSC-3612: added database support to exportSearchRequest

ba41042

EDSC-3612: added pg support to exportSearch and cleaned up exportSear…

0246617

…chRequest

EDSC-3612: added clearBucket.js

b23ef3a

EDSC-3612: added doesObjectExist.js to serverless/src/util/aws

f495642

EDSC-3612: updated deleteBucket to use clearBucket

dbefb40

EDSC-3612: added exportSearchCheck lambda

2a604fc

EDSC-3612: removed extra import from serverless/src/exportSearch/__te…

56050f6

…sts__/handler.test.js

EDSC-3612: updated package.json, adding pg-mem and serverless-offline…

5785311

…-sqs, adding scripts for installing and running sqs and s3 simulations

EDSC-3612: added elasticmq-server-1.3.9.jar to .gitignore

461830e

EDSC-3612: added sleep and retry

a501304

EDSC-3612: updated README.md with S3 and SQS simulations

8a7de7f

EDSC-3612: added doesBucketExist

8b5c505

EDSC-3612: added serverless/src/util/aws/getSearchExportBucket.js

91fd4b1

EDSC-3612: added serverless/src/util/aws/getSearchExportQueueUrl.js

fddda4e

EDSC-3612: added getSearchExportS3Endpoint

3bde7a4

EDSC-3612: added getS3Config.js

4eb6a26

EDSC-3612: added custom serverless-offline plugin for creating defaul…

4d8f1a5

…t S3 buckets, used by search exporting

EDSC-3612: created migration for search exports table

bf4fcd3

EDSC-3612: added exportStatusRefreshTime to static.config.json

5177805

EDSC-3612: added SearchExportBucket and SearchExportQueue to serverle…

53a2046

…ss-configs/aws-resources.yml

EDSC-3612: updated serveless-configs/aws-functions.yml with search ex…

54305ee

…port functions

EDSC-3612: require user to be logged-in in order to export search res…

666aeaa

…ults

EDSC-3612: updated GraphQlRequest search function to accept extra params

450dc53

EDSC-3612: updated exportSearchRequest handler and tests, including m…

7cb67e5

…ocking of randomUUID

DanielJDufour added 6 commits February 24, 2023 14:36

EDSC-3612: updated github ci workflow to install and run ElasticMQ an…

4171a9f

…d moto

EDSC-3612: install moto with pip

9951e41

EDSC-3612: hopefully fixed edge cases around disabling/enabling net c…

384c9ff

…onnections while running tests in parallel

EDSC-3612: clear nock interceptors before each search export-related …

7336562

…test

EDSC-3612: updated getSearchExportQueueUrl to return 0.0.0.0 when doi…

c9bd85c

…ng local development

EDSC-3612: make sure testSearchExportQueueUrl uses 0.0.0.0 not localhost

7ad1d17

DanielJDufour changed the title ~~Work in Progress: EDSC-3612, make search results export asynchronous~~ EDSC-3612, make search results export asynchronous Feb 27, 2023

Merge branch 'master' into EDSC-3612

7a2b558

DanielJDufour marked this pull request as ready for review February 27, 2023 14:38

DanielJDufour commented Feb 27, 2023

View reviewed changes

macrouch reviewed Feb 27, 2023

View reviewed changes

DanielJDufour closed this May 2, 2023

macrouch deleted the EDSC-3612 branch June 28, 2023 17:14

		@@ -0,0 +1,124 @@
		import { randomUUID } from 'crypto' // if we do 'node:crypto', jest.mock won't work


		const { id: userId } = getVerifiedJwtToken(jwt, earthdataEnvironment)

		if (!userId) throw Error("failed getting userId from jwt")


		const dbConnection = await getDbConnection()

		await dbConnection('exports').insert({

Uh oh!

Conversation

DanielJDufour commented Feb 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What is the feature?

What is the Solution?

What areas of the application does this impact?

Testing

Reproduction steps

Attachments

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanielJDufour commented May 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DanielJDufour commented Feb 13, 2023 •

edited

Loading