Terraform, Lambda, and Frontend Changes

.gitignore

@@ -55,5 +55,32 @@ AWS/Repo-information.eml
 # Local data and files
 **/local/
 **/tmp/
+
+# Local .terraform directories
+.terraform/
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Ignore transient lock info files created by terraform apply
+.terraform.tfstate.lock.info
+
+.terraform.lock.hcl
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
 fp-info-cache
-WIBL-backups/
+WIBL-backups/

wibl-python/requirements-lambda.txt

@@ -7,3 +7,4 @@ csbschema~=1.1.2
 #   urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with
 #   OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
 urllib3==1.26.18
+deepmerge

wibl-python/scripts/AWSCloudSetup.md

@@ -1,3 +1,7 @@
+# IMPORTANT: 
+This file describes the deprecated scripting build approach. The intended build method is to use Terraform. Follow the 
+instructions in the README.md located in the Terraform folder, which is located here [Terraform_README](./cloud/AWS/Terraform/README.md)
+
 # Packaging and deploying AWS lambdas for DCDB processing
 There are a number of steps required to set up the AWS Lambdas, the S3 buckets, and associated triggers and permissions in order to make processing work in the cloud.  The steps are covered in detail below, and a corresponding set of scripts are available in the `wibl-python/AWS-setup` directory in the repository.  These scripts should *mostly* work, but will likely need some modficiation for a local configuration before being fully executable.  Consider the `configuration-parameters.sh` file first for this.
 

wibl-python/scripts/cloud/AWS/README.md

@@ -1,3 +1,7 @@
+# Important 
+The instructions inside this file describe a deprecated build approach. The intended build method is to use Terraform. 
+Follow the README.md inside the Terraform folder for more information.
+
 # WIBL AWS processing setup instructions and usage
 The following instructions describe how to setup WIBL processing lambdas and WIBL manager
 web service in AWS.

wibl-python/scripts/cloud/AWS/Terraform/README.md

@@ -0,0 +1,106 @@
+# Terraform AWS Build Instructions
+
+## Setup
+The following software items must be installed and configured before continuing
+
+- Docker
+- AWS CLI
+- Terraform
+
+### 1. Change default configurations
+Edit the `terraform.tfvars` file as needed. Due to the nature of the cloud, all AWS components must have name unique to
+its region and type. Thus, the default names provided will most likely already be taken. 
+
+#### Important Variables To Consider:
+- region: Must be a valid AWS region code. Consider this list for available regions. [AWS_Region_List](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html)
+- auth_file_name: Like it describes in the configuration file, it must be set to a valid file name that exists in the 
+`auth` folder. The easiest solution is to copy and paste your string into the `default_auth.txt` file.
+- frontend_secret_key: Following the directions is the configuration file to generate a Django secret is highly 
+recommended. The script is located [here](./generate_secret.sh)
+- origin_secret: This password phrase adds an extra level of security between the CloudFront module and the 
+frontend's load balancer. Run the generate_secret script again to create a new value.
+- DCDB_provider_id: Replace with your given DCDB provider id.
+- DCDB_mode: By default the test mode is active. To switch to using the production URL, set this variable to 1.
+
+#### Mandatory Replacement Areas
+In [main.tf](./main.tf) in the section near the top labeled `backend "s3"`, the variables inside must be **manually updated**. 
+They will not follow what is located inside the tfvars file. The variables that need to be updated are 
+`terraform-state-bucket`, `region`, and `terraform-state-key`. 
+
+In [configuration-paramters.sh](../configuration-parameters.sh) the variable `AUTHKEY` must be replaced with the relative
+path to your document containing your DCDB authentication key. If the file is located inside the `Terraform/auth` folder,
+than the path will look something like this 
+```
+AUTHKEY=`cat ./Terraform/auth/default_auth.txt`
+```
+### 2. Building The Lambda Package
+This step only works if `docker` is currently running on your system. Inside the outer `AWS` folder is a script called 
+`build-lambda.sh`, run this script. Even if the script is successful, there may still be a lingering `build` or `package`
+folder, pay these no mind and continue to the next step.
+
+### 3. Bootstrapping Terraform
+Terraform is able to modify and delete computing resources that were created using Terraform. To do this, Terraform must 
+store state information about what resources have been created. This state information can be stored locally on a single 
+compute. However, if the locally stored state information is lost, then you lose the ability to automatically update or 
+delete the computing resources originally created by Terraform. To avoid this problem, Terraform state can be stored 
+remotely in S3. In addition to providing a disaster recovery solution, storing state in S3 allows multiple 
+people/computers to manage a WIBL upload-server created with Terraform, and to do so asynchronously, without having to
+worry about each person's updates overwriting each other.
+
+You can use the script terraform-bootstrap.bash to create an S3 bucket in which to store Terraform state. Before doing 
+so, change the name of the bucket and state object key by editing terraform_state_bucket and terraform_state_key in 
+terraform.tfvars.
+
+Running terraform-bootstrap.bash will look something like this:
+
+```
+$ ./scripts/cloud/Terraform/aws/terraform-bootstrap.bash
+CONTENT_ROOT: /Users/JANE_USER/repos/WIBL/UploadServer
+Using AWS_TF_ROOT: /Users/JANE_USER/repos/WIBL/UploadServer/scripts/cloud/Terraform/aws
+Using ARCHITECTURE: arm64
+Using AWS_BUILD: aws-build
+Using BUILD_DEST: /Users/JANE_USER/repos/WIBL/UploadServer/aws-build
+Using CERTS_DEST: /Users/JANE_USER/repos/WIBL/UploadServer/aws-build/certs
+Using WIBL_UPLOAD_BINARY: upload-server
+Using WIBL_UPLOAD_BINARY_PATH: /Users/JANE_USER/repos/WIBL/UploadServer/aws-build/upload-server
+Using ADD_LOGGER_BINARY: add-logger
+Using ADD_LOGGER_BINARY_PATH: /Users/JANE_USER/repos/WIBL/UploadServer/aws-build/add-logger
+Using WIBL_UPLOAD_CONFIG_PROTO: /Users/JANE_USER/repos/WIBL/UploadServer/scripts/cloud/Terraform/aws/config-aws.json.proto
+Using WIBL_UPLOAD_CONFIG_PATH: /Users/JANE_USER/repos/WIBL/UploadServer/aws-build/config.json
+Using AWS_PROFILE: wibl-upload-server
+Using TF_VARS: /Users/JANE_USER/repos/WIBL/UploadServer/scripts/cloud/Terraform/aws/terraform.tfvars
+Using WIBL_UPLOAD_SERVER_PORT: 443
+Using AWS_REGION: us-east-2
+Using TF_STATE_BUCKET: unhjhc-wibl-tf-state
+Using TF_STATE_KEY: terraform/state/wibl-upload-server-deploy.tfstate
+Using WIBL_UPLOAD_BUCKET_NAME: unhjhc-wibl-upload-server-incoming
+Using WIBL_UPLOAD_SNS_TOPIC_NAME: unhjhc-wibl-upload-server-conversion
+Using AWS_CLI: aws --profile wibl-upload-server --region us-east-2
+Using AWS_ACCOUNT_NUMBER: XXXXXXXXXXXX
+Using WIBL_UPLOAD_SNS_TOPIC_ARN: arn:aws:sns:us-east-2:XXXXXXXXXXXX:unhjhc-wibl-upload-server-conversion
+Creating terraform state bucket unhjhc-wibl-tf-state in AWS region us-east-2...
+{
+    "Location": "http://unhjhc-wibl-tf-state.s3.amazonaws.com/"
+}
+Enabling bucket versioning in terraform state bucket unhjhc-wibl-tf-state...
+Done.
+```
+### 4. Building The System With Terraform
+This step also requires `docker` to work, so ensure it is running. First, run the `plan.sh` script inside the Terraform 
+folder. This is where any you will be alerted of any misconfigured or missing variables. If script says "Plan: 125 to 
+add, 0 to change, 0 to destroy." without returning any errors, run the next script `build.sh`. If you do experience any 
+errors, you need to then run the `destroy.sh` script before you can attempt another build. 
+
+#### IMPORTANT: The system can take upwards of 15 to 25 minutes to fully build or destroy.
+
+### How To Tear Down The System
+#### WARNING: This script destroys ALL resources, meaning all data put into the system will be unrecoverable.
+To destroy the resources in the cloud, including all AWS and docker resources, run the `destroy.sh` script. 
+The following resources tend to take longer to destroy than their counter-parts. 
+- module.configure-manager-ecs.aws_internet_gateway.ig_public
+- module.configure-manager-ecs.aws_cloudfront_distribution.frontend
+
+This is mostly due to a bug with AWS's resource state management system. AWS believes select lambda functions are still 
+running, which can often leave multiple components hanging on each other waiting for the other to be destroyed. If it 
+seems like this might be the case, using ^c to cancel the script and rerunning it may solve the issue.
+

wibl-python/scripts/cloud/AWS/Terraform/auth/default_auth.txt

@@ -0,0 +1 @@
+Replace with your DCDB auth string

wibl-python/scripts/cloud/AWS/Terraform/buckets/main.tf

@@ -0,0 +1,52 @@
+resource "aws_s3_bucket" "incoming_bucket" {
+    bucket = var.incoming_bucket
+    force_destroy = true
+}
+
+resource "aws_s3_bucket" "staging_bucket" {
+    bucket = var.staging_bucket
+    force_destroy = true
+}
+
+resource "aws_s3_bucket" "viz_bucket" {
+    bucket = var.viz_bucket
+    force_destroy = true
+}
+
+resource "aws_s3_bucket" "static_bucket" {
+    bucket = var.static_bucket
+    force_destroy = true
+}
+
+data "aws_iam_policy_document" "static_bucket_policy" {
+  statement {
+    actions = ["s3:GetObject"]
+
+    resources = [
+      "${aws_s3_bucket.static_bucket.arn}/static/*"
+    ]
+
+    principals {
+      type        = "AWS"
+      identifiers = [var.oai_iam_arn]
+    }
+  }
+}
+
+resource "aws_s3_bucket_policy" "static" {
+  bucket = aws_s3_bucket.static_bucket.id
+  policy = data.aws_iam_policy_document.static_bucket_policy.json
+}
+
+resource "aws_s3_bucket_cors_configuration" "static_files" {
+  bucket = aws_s3_bucket.static_bucket.id
+
+  cors_rule {
+    allowed_methods = ["GET", "HEAD"]
+    allowed_origins = [var.alb_url]
+    allowed_headers = ["*"]
+    max_age_seconds = 3000
+  }
+  depends_on = [aws_s3_bucket.static_bucket]
+}
+

wibl-python/scripts/cloud/AWS/Terraform/buckets/outputs.tf

@@ -0,0 +1,18 @@
+output "incoming_bucket_arn" {
+    value = aws_s3_bucket.incoming_bucket.arn
+}
+
+output "incoming_bucket_id" {
+    value = aws_s3_bucket.incoming_bucket.id
+}
+output "staging_bucket_arn"  {
+    value = aws_s3_bucket.staging_bucket.arn
+}
+
+output "viz_bucket_arn" {
+    value = aws_s3_bucket.viz_bucket.arn
+}
+
+output "static_bucket_regional_dns_name" {
+    value = aws_s3_bucket.static_bucket.bucket_regional_domain_name
+}

wibl-python/scripts/cloud/AWS/Terraform/buckets/variables.tf

@@ -0,0 +1,28 @@
+variable "incoming_bucket" {
+    description = "Name of the incoming s3 bucket"
+    type = string
+}
+
+variable "staging_bucket" {
+    description = "Name of the staging s3 bucket"
+    type = string
+}
+
+variable "viz_bucket" {
+    description = "Name of the viz s3 bucket"
+    type = string
+}
+
+variable "static_bucket" {
+    description = "Name of the static bucket"
+    type = string
+}
+
+variable "alb_url" {
+    description = "Frontend cloudfront distribution url"
+    type = string
+}
+
+variable "oai_iam_arn" {
+    type = string
+}

wibl-python/scripts/cloud/AWS/Terraform/build.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+WIBL_BUILD_LOCATION=$(git rev-parse --show-toplevel)/wibl-python/awsbuild
+ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
+SRC_PATH=$(git rev-parse --show-toplevel)/wibl-python
+TF_LOG=DEBUG
+terraform init && terraform apply -auto-approve -var="src_path=${SRC_PATH}" -var="wibl_build_path=${WIBL_BUILD_LOCATION}" -var="account_number=${ACCOUNT_NUMBER}"

wibl-python/scripts/cloud/AWS/Terraform/destroy.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+WIBL_BUILD_LOCATION=$(git rev-parse --show-toplevel)/wibl-python/awsbuild
+ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
+SRC_PATH=$(git rev-parse --show-toplevel)/wibl-python
+terraform destroy -auto-approve -var="wibl_build_path=${WIBL_BUILD_LOCATION}" -var="src_path=${SRC_PATH}" -var="account_number=${ACCOUNT_NUMBER}"

wibl-python/scripts/cloud/AWS/Terraform/generate_secret.sh

@@ -0,0 +1 @@
+dd if=/dev/random bs=4096 count=1 status=none | sha512sum --quiet

wibl-python/scripts/cloud/AWS/Terraform/lambda/configure_roles.tf

@@ -0,0 +1,99 @@
+resource "aws_iam_role" "conversion_lambda_role" {
+  name = var.conversion_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "validation_lambda_role" {
+  name = var.validation_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "submission_lambda_role" {
+  name = var.submission_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "conversion_start_lambda_role" {
+  name = var.conversion_start_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_role" "viz_lambda_role" {
+  name = var.viz_lambda_role_name
+  assume_role_policy = data.aws_iam_policy_document.lambda-trust-policy-doc.json
+}
+
+resource "aws_iam_policy_attachment" "lambda_basic_policy_attach" {
+  name       = "lambda_basic_policy_attach"
+
+  roles      = [aws_iam_role.conversion_lambda_role.name, aws_iam_role.submission_lambda_role.name,
+  aws_iam_role.validation_lambda_role.name, aws_iam_role.viz_lambda_role.name, aws_iam_role.conversion_start_lambda_role.name]
+
+  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
+
+}
+
+resource "aws_iam_policy_attachment" "lambda_nic_policy_attach" {
+  name       = "lambda_nic_policy_attach"
+
+  roles      = [aws_iam_role.conversion_lambda_role.name, aws_iam_role.submission_lambda_role.name,
+  aws_iam_role.validation_lambda_role.name, aws_iam_role.viz_lambda_role.name, aws_iam_role.conversion_start_lambda_role.name]
+
+  policy_arn = aws_iam_policy.lambda-nic-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_lambda_s3_access_policy_attach" {
+  name       = "lambda_s3_access_policy_attach"
+
+  roles      = [aws_iam_role.conversion_lambda_role.name]
+
+  policy_arn = aws_iam_policy.lambda-s3-access-all-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_lambda_sns_validation_policy_attach" {
+    name = "lambda_sns_validation_policy_attach"
+    roles = [aws_iam_role.conversion_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-validation.arn
+}
+
+resource "aws_iam_policy_attachment" "validation_lambda_s3_access_policy_attach" {
+  name       = "validation_lambda_s3_access_policy_attach"
+  roles = [aws_iam_role.validation_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-s3-access-staging-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "validation_lambda_sns_submission_policy_attach" {
+    name = "validation_lambda_sns_submission_policy_attach"
+    roles = [aws_iam_role.validation_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-submission.arn
+}
+
+resource "aws_iam_policy_attachment" "submission_lambda_s3_access_policy_attach" {
+  name       = "submission_lambda_s3_access_policy_attach"
+  roles = [aws_iam_role.submission_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-s3-access-staging-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "submission_lambda_sns_submitted_policy_attach" {
+    name = "submission_lambda_sns_submitted_policy_attach"
+    roles = [aws_iam_role.submission_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-submitted.arn
+}
+
+resource "aws_iam_policy_attachment" "submission_lambda_ssm_policy_attach" {
+  name       = "submission_lambda_ssm_policy_attach"
+  roles      = [aws_iam_role.submission_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-ssm-access-submission.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_start_lambda_s3_access_policy_attach" {
+  name       = "conversion_start_lambda_s3_access_policy_attach"
+  roles = [aws_iam_role.conversion_start_lambda_role.name]
+  policy_arn = aws_iam_policy.lambda-s3-access-incoming-policy.arn
+}
+
+resource "aws_iam_policy_attachment" "conversion_start_lambda_sns_conversion_policy_attach" {
+    name = "conversion_start_lambda_sns_conversion_policy_attach"
+    roles = [aws_iam_role.conversion_start_lambda_role.name]
+    policy_arn = aws_iam_policy.lambda-sns-access-conversion.arn
+}