Terraform, Lambda, and Frontend Changes

.gitignore

@@ -55,5 +55,32 @@ AWS/Repo-information.eml
 # Local data and files
 **/local/
 **/tmp/
+
+# Local .terraform directories
+.terraform/
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Ignore transient lock info files created by terraform apply
+.terraform.tfstate.lock.info
+
+.terraform.lock.hcl
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
 fp-info-cache
-WIBL-backups/
+WIBL-backups/

wibl-python/requirements-lambda.txt

@@ -7,3 +7,4 @@ csbschema~=1.1.2
 #   urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with
 #   OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
 urllib3==1.26.18
+deepmerge

wibl-python/scripts/AWSCloudSetup.md

@@ -1,3 +1,7 @@
+# IMPORTANT: 
+This file describes the deprecated scripting build approach. The intended build method is to use Terraform. Follow the 
+instructions in the README.md located in the Terraform folder, which is located here [Terraform_README](./cloud/AWS/Terraform/README.md)
+
 # Packaging and deploying AWS lambdas for DCDB processing
 There are a number of steps required to set up the AWS Lambdas, the S3 buckets, and associated triggers and permissions in order to make processing work in the cloud.  The steps are covered in detail below, and a corresponding set of scripts are available in the `wibl-python/AWS-setup` directory in the repository.  These scripts should *mostly* work, but will likely need some modficiation for a local configuration before being fully executable.  Consider the `configuration-parameters.sh` file first for this.
 

wibl-python/scripts/cloud/AWS/README.md

@@ -1,3 +1,7 @@
+# Important 
+The instructions inside this file describe a deprecated build approach. The intended build method is to use Terraform. 
+Follow the README.md inside the Terraform folder for more information.
+
 # WIBL AWS processing setup instructions and usage
 The following instructions describe how to setup WIBL processing lambdas and WIBL manager
 web service in AWS.

wibl-python/scripts/cloud/AWS/Terraform/README.md

@@ -0,0 +1,48 @@
+# Terraform AWS Build Instructions
+
+## Setup
+The following software items must be installed and configured before continuing
+
+- Docker
+- AWS CLI
+- Terraform
+
+### 1. Change default configurations
+Edit the `terraform.tfvars` file as needed. Due to the nature of the cloud, all AWS components must have name unique to
+its region and type. Thus, the default names provided will most likely already be taken. 
+
+#### Important Variables To Consider:
+- region: Must be a valid AWS region code. Consider this list for available regions. [AWS_Region_List](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html)
+- auth_file_name: Like it describes in the configuration file, it must be set to a valid file name that exists in the 
+`auth` folder. The easiest solution is to copy and paste your string into the `default_auth.txt` file.
+- frontend_secret_key: Following the directions is the configuration file to generate a Django secret is highly 
+recommended. The script is located [here](./generate_secret.sh)
+- origin_secret: This password phrase adds an extra level of security between the CloudFront module and the 
+frontend's load balancer. Run the generate_secret script again to create a new value.
+- DCDB_provider_id: Replace with your given DCDB provider id.
+- DCDB_mode: By default the test mode is active. To switch to using the production URL, set this variable to 1.
+
+### 2. Building The Lambda Package
+This step only works if `docker` is currently running on your system. Inside the outer `AWS` folder is a script called 
+`build-lambda.sh`, run this script. Even if the script is successful, there may still be a lingering `build` or `package`
+folder, pay these no mind and continue to the next step.
+
+### 3. Building The System With Terraform
+This step also requires `docker` to work, so ensure it is running. First, run the `plan.sh` script inside the Terraform 
+folder. This is where any you will be alerted of any misconfigured or missing variables. If script says "Plan: 125 to 
+add, 0 to change, 0 to destroy." without returning any errors, run the next script `build.sh`. If you do experience any 
+errors, you need to then run the `destroy.sh` script before you can attempt another build. 
+
+#### IMPORTANT: The system can take upwards of 15 to 25 minutes to fully build or destroy.
+
+### How To Tear Down The System
+#### WARNING: This script destroys ALL resources, meaning all data put into the system will be unrecoverable.
+To destroy the resources in the cloud, including all AWS and docker resources, run the `destroy.sh` script. 
+The following resources tend to take longer to destroy than their counter-parts. 
+- module.configure-manager-ecs.aws_internet_gateway.ig_public
+- module.configure-manager-ecs.aws_cloudfront_distribution.frontend
+
+This is mostly due to a bug with AWS's resource state management system. AWS believes select lambda functions are still 
+running, which can often leave multiple components hanging on each other waiting for the other to be destroyed. If it 
+seems like this might be the case, using ^c to cancel the script and rerunning it may solve the issue.
+

wibl-python/scripts/cloud/AWS/Terraform/auth/default_auth.txt

@@ -0,0 +1 @@
+Replace with your DCDB auth string

wibl-python/scripts/cloud/AWS/Terraform/buckets/main.tf

@@ -0,0 +1,52 @@
+resource "aws_s3_bucket" "incoming_bucket" {
+    bucket = var.incoming_bucket
+    force_destroy = true
+}
+
+resource "aws_s3_bucket" "staging_bucket" {
+    bucket = var.staging_bucket
+    force_destroy = true
+}
+
+resource "aws_s3_bucket" "viz_bucket" {
+    bucket = var.viz_bucket
+    force_destroy = true
+}
+
+resource "aws_s3_bucket" "static_bucket" {
+    bucket = var.static_bucket
+    force_destroy = true
+}
+
+data "aws_iam_policy_document" "static_bucket_policy" {
+  statement {
+    actions = ["s3:GetObject"]
+
+    resources = [
+      "${aws_s3_bucket.static_bucket.arn}/static/*"
+    ]
+
+    principals {
+      type        = "AWS"
+      identifiers = [var.oai_iam_arn]
+    }
+  }
+}
+
+resource "aws_s3_bucket_policy" "static" {
+  bucket = aws_s3_bucket.static_bucket.id
+  policy = data.aws_iam_policy_document.static_bucket_policy.json
+}
+
+resource "aws_s3_bucket_cors_configuration" "static_files" {
+  bucket = aws_s3_bucket.static_bucket.id
+
+  cors_rule {
+    allowed_methods = ["GET", "HEAD"]
+    allowed_origins = [var.alb_url]
+    allowed_headers = ["*"]
+    max_age_seconds = 3000
+  }
+  depends_on = [aws_s3_bucket.static_bucket]
+}
+

wibl-python/scripts/cloud/AWS/Terraform/buckets/outputs.tf

@@ -0,0 +1,18 @@
+output "incoming_bucket_arn" {
+    value = aws_s3_bucket.incoming_bucket.arn
+}
+
+output "incoming_bucket_id" {
+    value = aws_s3_bucket.incoming_bucket.id
+}
+output "staging_bucket_arn"  {
+    value = aws_s3_bucket.staging_bucket.arn
+}
+
+output "viz_bucket_arn" {
+    value = aws_s3_bucket.viz_bucket.arn
+}
+
+output "static_bucket_regional_dns_name" {
+    value = aws_s3_bucket.static_bucket.bucket_regional_domain_name
+}

wibl-python/scripts/cloud/AWS/Terraform/buckets/variables.tf

@@ -0,0 +1,28 @@
+variable "incoming_bucket" {
+    description = "Name of the incoming s3 bucket"
+    type = string
+}
+
+variable "staging_bucket" {
+    description = "Name of the staging s3 bucket"
+    type = string
+}
+
+variable "viz_bucket" {
+    description = "Name of the viz s3 bucket"
+    type = string
+}
+
+variable "static_bucket" {
+    description = "Name of the static bucket"
+    type = string
+}
+
+variable "alb_url" {
+    description = "Frontend cloudfront distribution url"
+    type = string
+}
+
+variable "oai_iam_arn" {
+    type = string
+}

wibl-python/scripts/cloud/AWS/Terraform/build.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+WIBL_BUILD_LOCATION=$(git rev-parse --show-toplevel)/wibl-python/awsbuild
+ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
+SRC_PATH=$(git rev-parse --show-toplevel)/wibl-python
+TF_LOG=DEBUG
+terraform init && terraform apply -auto-approve -var="src_path=${SRC_PATH}" -var="wibl_build_path=${WIBL_BUILD_LOCATION}" -var="account_number=${ACCOUNT_NUMBER}"

wibl-python/scripts/cloud/AWS/Terraform/destroy.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+WIBL_BUILD_LOCATION=$(git rev-parse --show-toplevel)/wibl-python/awsbuild
+ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
+SRC_PATH=$(git rev-parse --show-toplevel)/wibl-python
+terraform destroy -auto-approve -var="wibl_build_path=${WIBL_BUILD_LOCATION}" -var="src_path=${SRC_PATH}" -var="account_number=${ACCOUNT_NUMBER}"

wibl-python/scripts/cloud/AWS/Terraform/generate_secret.sh

@@ -0,0 +1 @@
+dd if=/dev/random bs=4096 count=1 status=none | sha512sum --quiet

wibl-python/scripts/cloud/AWS/Terraform/lambda/configure_roles.tf

@@ -0,0 +1,99 @@
+resource "aws_iam_role" "conversion_lambda_role" {
+  name = var.conversion_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "validation_lambda_role" {
+  name = var.validation_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "submission_lambda_role" {
+  name = var.submission_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "conversion_start_lambda_role" {
+  name = var.conversion_start_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "viz_lambda_role" {
+  name = var.viz_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_policy_attachment" "lambda_basic_policy_attach" {
+  name       = "lambda_basic_policy_attach"
+
+  roles      = [aws_iam_role.conversion_lambda_role.name, aws_iam_role.submission_lambda_role.name,
+  aws_iam_role.validation_lambda_role.name, aws_iam_role.viz_lambda_role.name, aws_iam_role.conversion_start_lambda_role.name]
+
+  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
+
+}
+
+resource "aws_iam_policy_attachment" "lambda_nic_policy_attach" {
+  name       = "lambda_nic_policy_attach"
+
+  roles      = [aws_iam_role.conversion_lambda_role.name, aws_iam_role.submission_lambda_role.name,
+  aws_iam_role.validation_lambda_role.name, aws_iam_role.viz_lambda_role.name, aws_iam_role.conversion_start_lambda_role.name]
+
+  policy_arn = aws_iam_policy.lambda-nic-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_lambda_s3_access_policy_attach" {
+  name       = "lambda_s3_access_policy_attach"
+
+  roles      = [aws_iam_role.conversion_lambda_role.name]
+
+  policy_arn = aws_iam_policy.lambda-s3-access-all-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_lambda_sns_validation_policy_attach" {
+    name = "lambda_sns_validation_policy_attach"
+    roles = [aws_iam_role.conversion_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-validation.arn
+}
+
+resource "aws_iam_policy_attachment" "validation_lambda_s3_access_policy_attach" {
+  name       = "validation_lambda_s3_access_policy_attach"
+  roles = [aws_iam_role.validation_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-s3-access-staging-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "validation_lambda_sns_submission_policy_attach" {
+    name = "validation_lambda_sns_submission_policy_attach"
+    roles = [aws_iam_role.validation_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-submission.arn
+}
+
+resource "aws_iam_policy_attachment" "submission_lambda_s3_access_policy_attach" {
+  name       = "submission_lambda_s3_access_policy_attach"
+  roles = [aws_iam_role.submission_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-s3-access-staging-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "submission_lambda_sns_submitted_policy_attach" {
+    name = "submission_lambda_sns_submitted_policy_attach"
+    roles = [aws_iam_role.submission_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-submitted.arn
+}
+
+resource "aws_iam_policy_attachment" "submission_lambda_ssm_policy_attach" {
+  name       = "submission_lambda_ssm_policy_attach"
+  roles      = [aws_iam_role.submission_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-ssm-access-submission.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_start_lambda_s3_access_policy_attach" {
+  name       = "conversion_start_lambda_s3_access_policy_attach"
+  roles = [aws_iam_role.conversion_start_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-s3-access-incoming-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_start_lambda_sns_conversion_policy_attach" {
+    name = "conversion_start_lambda_sns_conversion_policy_attach"
+    roles = [aws_iam_role.conversion_start_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-conversion.arn
+}