Incorporate comments

nhatnghiho · nhatnghiho · commit dc77eb23ffc3 · 2025-05-12T08:06:25.000-07:00
diff --git a/tests/ci/cdk/README.md b/tests/ci/cdk/README.md
@@ -36,6 +36,7 @@ AWS-LC CI uses AWS CDK to define and deploy AWS resources (e.g. AWS CodeBuild, E
     * step 4: click **Connect using OAuth** and **Connect to GitHub**.
     * step 5: follow the OAuth app to grant access.
 * Setup Python environment:
+
   * From `aws-lc/tests/ci` run:
 ```shell
 python -m pip install -r requirements.txt
@@ -83,75 +84,91 @@ To setup or update the CI in your account you will need the following IAM permis
   * secretsmanager:GetSecretValue
 
 ### Pipeline Commands
-Use these commands to deploy the CI pipeline. Any changes to the CI or Docker images will be updated automatically after the pipeline is deployed. 
-
-These commands are run from `aws-lc/tests/ci/cdk`.
-
-[SKIP IF NO CROSS-ACCOUNT DEPLOYMENT] Give the pipeline account administrator access to the deployment account's CloudFormation. Repeat this step depending on how many deployment environment there are. You only need to run this step once when the pipeline is deploying to a new account for the first time.
-```
-cdk bootstrap aws://${DEPLOY_ACCOUNT_ID}/us-west-2 --trust ${PIPELINE_ACCOUNT_ID} --trust-for-lookup ${PIPELINE_ACCOUNT_ID} --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess
-```
-
-If not done previously, bootstrap cdk for the pipeline account before running the next commands.
-```
-cdk bootstrap aws://${PIPELINE_ACCOUNT_ID}/us-west-2
-```
-
-To deploy dev pipeline to the same account as your CI:
-```
-./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --github-source-version ${GITHUB_SOURCE_VERSION} --deploy-account ${DEPLOY_ACCOUNT_ID} --action deploy-dev-pipeline
-```
-
-To deploy dev pipeline but pipeline is hosted in a separate account:
-```
-./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --github-source-version ${GITHUB_SOURCE_VERSION} --pipeline-account ${PIPELINE_ACCOUNT_ID} --deploy-account ${DEPLOY_ACCOUNT_ID} --action deploy-dev-pipeline
-```
-
-To deploy production pipeline using default parameters:
-```
-./run-cdk.sh --action deploy-production-pipeline
-```
+Use the following commands to deploy the CI pipeline. Any changes to the CI or Docker images will be updated automatically after the pipeline is deployed. 
+
+1. Ensure you are in `aws-lc/tests/ci/cdk`
+2. Export the relevant environment variables:
+   - `PIPELINE_ACCOUNT_ID`: the AWS account to host your pipeline
+   - `DEPLOY_ACCOUNT_ID`: the AWS account to deploy Docker images and CodeBuild CI tests to.
+   - `GITHUB_REPO_OWNER`: GitHub repo targeted by the pipeline (i.e, your personal Github account)
+   - `GITHUB_SOURCE_VERSION`: Git branch holding the latest pipeline code (default: main)
+
+3. [SKIP IF NO CROSS-ACCOUNT DEPLOYMENT] Give the pipeline account administrator access to the deployment account's CloudFormation. Repeat this step depending on how many deployment environment there are. You only need to run this step once when the pipeline is deploying to a new account for the first time.
+    ```shell
+    cdk bootstrap aws://${DEPLOY_ACCOUNT_ID}/us-west-2 --trust ${PIPELINE_ACCOUNT_ID} --trust-for-lookup ${PIPELINE_ACCOUNT_ID} --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess
+    ```
+4. If not done previously, bootstrap cdk for the pipeline account before running the next commands.
+    ```shell
+    cdk bootstrap aws://${PIPELINE_ACCOUNT_ID}/us-west-2
+    ```
+5. You may also need to request an increase to certain account quotas:
+    ```shell
+    open https://${DEPLOY_REGION}.console.aws.amazon.com/servicequotas/home/services/ec2/quotas
+    ```
+   Set EC2-VPC Elastic IPs = 20 (default is only 5)
+
+
+6. Choose 1 of the following options to deploy the pipeline:
+   - To deploy dev pipeline to the same account as your CI
+    ```shell
+    ./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --github-source-version ${GITHUB_SOURCE_VERSION} --deploy-account ${PIPELINE_ACCOUNT_ID} --action deploy-dev-pipeline
+    ```
+   - To deploy dev pipeline but pipeline is hosted in a separate account:
+    ```shell
+    ./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --github-source-version ${GITHUB_SOURCE_VERSION} --pipeline-account ${PIPELINE_ACCOUNT_ID} --deploy-account ${DEPLOY_ACCOUNT_ID} --action deploy-dev-pipeline
+    ```
+   - To deploy production pipeline using default parameters:
+    ```shell
+    ./run-cdk.sh --action deploy-production-pipeline
+    ```
+
+**Note**: If this is your first time deploying the pipeline and it's failing on the Source stage, this is normal and expected, since you haven't given CodePipeline access to your repo.
+To fix this:
+1. Go to your Console > CodePipeline > Settings > Connections. You should see a pending connection named `AwsLcCiPipelineGitHubConnection`. Click on it.
+2. Click on `Update Pending Connection.`
+3. On the pop up, you would see an `App Installation - optional`. Click on `Install a new app` (or choose an existing app if you have one). This step is REQUIRED to allow CodePipeline to detect new events from your repo.
+4. Click `Connect`. The connection status should become `Available` now
 
 ### CI Commands
 Use these commands if you wish to deploy individual stacks instead of the entire pipeline.
 
-These commands are run from `aws-lc/tests/ci/cdk`.
-
-If not done previously, bootstrap cdk before running the commands below. Make sure that AWS_ACCOUNT_ID is the AWS account you wish to deploy the CI stacks to.
-
-```shell
-cdk bootstrap aws://${AWS_ACCOUNT_ID}/us-west-2
-```
-
-You may also need to request an increase to certain account quotas:
-```shell
-open https://${DEPLOY_REGION}.console.aws.amazon.com/servicequotas/home/services/ec2/quotas
-```
-* **EC2-VPC Elastic IPs** = 20
-
-Note: `GITHUB_REPO_OWNER` specifies the GitHub repo targeted by this CI setup.
-* https://github.com/${GITHUB_REPO_OWNER}/aws-lc.git
-
-To set up AWS-LC CI, run command:
-```
-./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action deploy-ci --deploy-account ${AWS_ACCOUNT_ID}
-```
-
-To update AWS-LC CI, run command:
-```
-./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action update-ci --deploy-account ${AWS_ACCOUNT_ID}
-```
-
-To create/update Linux Docker images, run command:
-```
-./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action build-linux-img --deploy-account ${AWS_ACCOUNT_ID}
-```
-
-To destroy AWS-LC CI resources created above, run command:
-```
-# NOTE: this command will destroy all resources (AWS CodeBuild and ECR).
-./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action destroy-ci --deploy-account ${AWS_ACCOUNT_ID}
-```
+1. Ensure you are in `aws-lc/tests/ci/cdk`
+2. Export the relevant environment variables:
+  - `DEPLOY_ACCOUNT_ID`: AWS account you wish to deploy the CI stacks to
+  - `GITHUB_REPO_OWNER`: the GitHub repo targeted by this CI setup.
+
+2. If not done previously, bootstrap cdk before running the commands below.
+    ```shell
+    cdk bootstrap aws://${DEPLOY_ACCOUNT_ID}/us-west-2
+    ```
+
+3. You may also need to request an increase to certain account quotas:
+    ```shell
+    open https://${DEPLOY_REGION}.console.aws.amazon.com/servicequotas/home/services/ec2/quotas
+    ```
+    Set EC2-VPC Elastic IPs = 20 (default is only 5)
+
+
+4. Choose 1 of the following command options:
+   - To set up AWS-LC CI, run command:
+    ```shell
+    ./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action deploy-ci --deploy-account ${DEPLOY_ACCOUNT_ID}
+    ```
+
+   - To update AWS-LC CI, run command:
+    ```shell
+    ./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action update-ci --deploy-account ${DEPLOY_ACCOUNT_ID}
+    ```
+   - To create/update Linux Docker images, run command:
+    ```shell
+    ./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action build-linux-img --deploy-account ${DEPLOY_ACCOUNT_ID}
+    ```
+
+   - To destroy AWS-LC CI resources created above, run command:
+    ```shell
+    ./run-cdk.sh --github-repo-owner ${GITHUB_REPO_OWNER} --action destroy-ci --deploy-account ${DEPLOY_ACCOUNT_ID}
+    ```
+   NOTE: this command will destroy all resources (AWS CodeBuild and ECR).
 
 For help, run command:
 ```
diff --git a/tests/ci/cdk/pipeline/ci_stage.py b/tests/ci/cdk/pipeline/ci_stage.py
@@ -80,8 +80,8 @@ def add_stage_to_pipeline(
                     "git remote add upstream https://github.com/aws/aws-lc.git",
                     "git fetch upstream",
                     "git checkout main",
-                    "git merge upstream/main",
-                    # "git push origin main",
+                    "git merge --ff-only upstream/main",
+                    "git push origin main",
                 ],
                 env={
                     "STAGING_GITHUB_REPO_OWNER": STAGING_GITHUB_REPO_OWNER,
@@ -98,7 +98,6 @@ def add_stage_to_pipeline(
             input=input,
             commands=[
                 "cd tests/ci/cdk/pipeline/scripts",
-                "chmod +x check_trigger_conditions.sh",
                 'trigger_conditions=$(./check_trigger_conditions.sh --build-type ci --stacks "${STACKS}")',
                 "export NEED_REBUILD=$(echo $trigger_conditions | sed -n 's/.*\(NEED_REBUILD=[0-9]*\).*/\\1/p' | cut -d'=' -f2 )",
             ],
@@ -133,7 +132,6 @@ def add_stage_to_pipeline(
             input=input,
             commands=[
                 "cd tests/ci/cdk/pipeline/scripts",
-                "chmod +x build_target.sh",
                 "./build_target.sh --build-type ci --project ${PROJECT} --max-retry ${MAX_RETRY} --timeout ${TIMEOUT}",
             ],
             role=role,
diff --git a/tests/ci/cdk/pipeline/linux_docker_image_build_stage.py b/tests/ci/cdk/pipeline/linux_docker_image_build_stage.py
@@ -81,7 +81,6 @@ def add_stage_to_wave(
             input=input,
             commands=[
                 "cd tests/ci/cdk/pipeline/scripts",
-                "chmod +x cleanup_orphaned_images.sh check_trigger_conditions.sh build_target.sh",
                 './cleanup_orphaned_images.sh --repos "${ECR_REPOS}"',
                 'trigger_conditions=$(./check_trigger_conditions.sh --build-type docker --platform linux --stacks "${STACKS}")',
                 "export NEED_REBUILD=$(echo $trigger_conditions | sed -n -e 's/.*\(NEED_REBUILD=[0-9]*\).*/\\1/p' | cut -d'=' -f2 )",
diff --git a/tests/ci/cdk/pipeline/pipeline_stack.py b/tests/ci/cdk/pipeline/pipeline_stack.py
@@ -285,7 +285,6 @@ def deploy_to_environment(
                 input=source,
                 commands=[
                     "cd tests/ci/cdk/pipeline/scripts",
-                    "chmod +x finalize_images.sh",
                     './finalize_images.sh --repos "${ECR_REPOS}"',
                 ],
                 env={
diff --git a/tests/ci/cdk/pipeline/scripts/check_trigger_conditions.sh b/tests/ci/cdk/pipeline/scripts/check_trigger_conditions.sh
diff --git a/tests/ci/cdk/pipeline/scripts/cleanup_orphaned_images.sh b/tests/ci/cdk/pipeline/scripts/cleanup_orphaned_images.sh
diff --git a/tests/ci/cdk/pipeline/scripts/finalize_images.sh b/tests/ci/cdk/pipeline/scripts/finalize_images.sh
diff --git a/tests/ci/cdk/pipeline/scripts/util.sh b/tests/ci/cdk/pipeline/scripts/util.sh
@@ -11,6 +11,7 @@ else
 fi
 
 function assume_role() {
+  set +x
   if [[ -z ${CROSS_ACCOUNT_BUILD_ROLE_ARN} ]]; then
     echo "No role arn provided"
     return 1
@@ -21,6 +22,7 @@ function assume_role() {
   export AWS_ACCESS_KEY_ID=$(echo $CREDENTIALS | jq -r .Credentials.AccessKeyId)
   export AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIALS | jq -r .Credentials.SecretAccessKey)
   export AWS_SESSION_TOKEN=$(echo $CREDENTIALS | jq -r .Credentials.SessionToken)
+  set -x
 }
 
 function refresh_session() {
diff --git a/tests/ci/cdk/pipeline/windows_docker_image_build_stage.py b/tests/ci/cdk/pipeline/windows_docker_image_build_stage.py
@@ -77,7 +77,6 @@ def add_stage_to_wave(
             input=input,
             commands=[
                 "cd tests/ci/cdk/pipeline/scripts",
-                "chmod +x cleanup_orphaned_images.sh check_trigger_conditions.sh build_target.sh",
                 './cleanup_orphaned_images.sh --repos "${ECR_REPOS}"',
                 'trigger_conditions=$(./check_trigger_conditions.sh --build-type docker --platform windows --stacks "${STACKS}")',
                 "export NEED_REBUILD=$(echo $trigger_conditions | sed -n -e 's/.*\(NEED_REBUILD=[0-9]*\).*/\\1/p' | cut -d'=' -f2 )",
diff --git a/tests/ci/cdk/run-cdk.sh b/tests/ci/cdk/run-cdk.sh
@@ -16,11 +16,7 @@ function delete_s3_buckets() {
   aws s3api list-buckets --query "Buckets[].Name" | jq '.[]' | while read -r i; do
     bucket_name=$(echo "${i}" | tr -d '"')
     # Delete the bucket if its name uses AWS_LC_S3_BUCKET_PREFIX.
-#    if [[ "${bucket_name}" == *"${AWS_LC_S3_BUCKET_PREFIX}"* ]]; then
-#      aws s3 rm "s3://${bucket_name}" --recursive
-#      aws s3api delete-bucket --bucket "${bucket_name}"
-    # Delete bm-framework buckets if we're not on the team account
-    if [[ "${DEPLOY_ACCOUNT}" != "620771051181" ]] && [[ "${bucket_name}" == *"${aws-lc-ci-bm-framework}"* ]]; then
+    if [[ "${bucket_name}" == *"${S3_FOR_WIN_DOCKER_IMG_BUILD}"* ]]; then
       aws s3 rm "s3://${bucket_name}" --recursive
       aws s3api delete-bucket --bucket "${bucket_name}"
     fi
@@ -41,15 +37,15 @@ function delete_container_repositories() {
 }
 
 function destroy_ci() {
-  if [[ "${DEPLOY_ACCOUNT}" == "620771051181" ]]; then
+  if [[ "${DEPLOY_ACCOUNT}" == "620771051181" || "${DEPLOY_ACCOUNT}" == "351119683581" ]]; then
     echo "destroy_ci should not be executed on team account."
     exit 1
   fi
   cdk destroy 'aws-lc-*' --force
   # CDK stack destroy does not delete s3 bucket automatically.
-#  delete_s3_buckets
+  delete_s3_buckets
   # CDK stack destroy does not delete ecr automatically.
-#  delete_container_repositories
+  delete_container_repositories
 }
 
 function destroy_docker_img_build_stack() {
@@ -115,7 +111,7 @@ function run_windows_img_build() {
         --document-name "${WIN_DOCKER_BUILD_SSM_DOCUMENT}" \
         --output-s3-bucket-name "${S3_FOR_WIN_DOCKER_IMG_BUILD}" \
         --output-s3-key-prefix 'runcommand' \
-        --parameters "TriggerType=[\"pipeline\"]" | \
+        --parameters "TriggerType=[\"manual\"]" | \
         jq -r '.Command.CommandId')
       # Export for checking command run status.
       export WINDOWS_DOCKER_IMG_BUILD_COMMAND_ID="${command_id}"
@@ -177,9 +173,6 @@ function win_docker_img_build_status_check() {
 }
 
 function build_linux_docker_images() {
-  # Always destroy docker build stacks (which include EC2 instance) on EXIT.
-#  trap destroy_docker_img_build_stack EXIT
-
   # Create/update aws-ecr repo.
   cdk deploy 'aws-lc-ecr-linux-*' --require-approval never
 
@@ -195,9 +188,6 @@ function build_linux_docker_images() {
 }
 
 function build_win_docker_images() {
- # Always destroy docker build stacks (which include EC2 instance) on EXIT.
-# trap destroy_docker_img_build_stack EXIT
-
  # Create/update aws-ecr repo.
  cdk deploy 'aws-lc-ecr-windows-*' --require-approval never
 
diff --git a/tests/ci/cdk/util/metadata.py b/tests/ci/cdk/util/metadata.py
@@ -4,9 +4,6 @@
 # SPDX-License-Identifier: Apache-2.0 OR ISC
 
 from util.env_util import EnvUtil
-from datetime import datetime
-
-# timestamp = datetime.now().strftime('%Y-%m-%d-%H-%M')
 
 # Used when AWS CDK defines AWS resources.
 PROD_ACCOUNT = "620771051181"
