AWS Architecture diagrams created:
Kafka stream application in the private subnet can upload the images to the s3
•The Kafka Cluster is placed within the VPC, specifically in the private subnet.
•It can upload images to S3 (which is in the same AWS environment) via the NAT Gateway for outbound internet access. This setup allows the Kafka stream to operate within the same cloud environment as the rest of the architecture.
•The Kafka Cluster is in the private subnet, which enhances security.
•Engineers can access and manage the Kafka stream through a secure connection via AWS Direct Connect/VPN
•The NAT Gateway allows the Kafka Cluster to receive updates and patches from the internet, facilitating management.
Image processing code to be stored in the ecs cluster, and images and data to be stored in s3
•The ECS Cluster in the private subnet is ideal for hosting this image processing code.
•ECS allows for containerized applications, making it easy to deploy and scale the image processing code.
•Being in the private subnet enhances security while still allowing access to necessary resources.
•S3 is used for storing the images and metadata.
•S3 supports lifecycle policies that can automatically delete objects after a specified time (in this case, 7 days).
•This fulfills both the storage and automatic purging requirements for compliance and privacy.
Business intelligence resource will be Quicksight
•QuickSight is included in the architecture as the Business Intelligence tool.
•It’s placed outside the VPC as it’s a managed service, but within the AWS Cloud.
•QuickSight can access data stored in S3, allowing analysts to perform analysis on the image data and metadata.
Purpose of the components: ALB, ECS, S3
1.Application Load Balancer (ALB):
a)The ALB in the public subnet is the entry point for user requests.
b)It receives the API calls for image uploads from users.
2.ECS Cluster:
a)The ECS (Elastic Container Service) Cluster in the private subnet hosts the web application.
b)This is where the API for image uploads would be implemented.
c)The web application running in ECS containers would handle the logic for receiving uploads and storing them.
3.S3 (Simple Storage Service):
a)Supports lifecycle policies for automatic deletion after 7 days.
b)The web application would use S3 to store the uploaded images.
c)S3 is the cloud storage where the images are ultimately saved.
d)Stores images and metadata securely.
Purpose of: NAT gateway, IAM, ROUTE 53
•NAT Gateway in public subnet:
•To allow resources in the private subnet to access the internet.
•IAM (Identity and Access Management):
•Manages access to AWS services and resources securely.
•Implements the principle of least privilege for all components
•Manages permissions for the web application to access S3 and other necessary resources.
•Route 53:
•Provides DNS services for routing users to the application.
•Can be used for health checks and failover routing.
Purpose of cloudfront, s3, elasticache
•CloudFront:
•Acts as a CDN to cache and deliver content from edge locations, reducing latency.
•Provides an additional layer of security against DDoS attacks.
•Could be used to provide a content delivery network for faster upload speeds from different geographic locations.
•ElastiCache:
•Provides in-memory caching to improve application performance and reduce database load.
PUROSE OF THE ECS CLUSTER, QUICKSIGHT, KAFA STREAM AND CLOUDWATCH
•ECS Cluster:
•Hosts containerized applications, including the web application and image processing code.
•Allows for easy scaling and management of containerized workloads.
•QuickSight:
•Provides business intelligence and data visualization capabilities for analysts.
•Kafka Stream:
•It provides a stream of images to be uploaded and processed.
•CloudWatch:
•Monitors the entire infrastructure, providing logs, metrics, and alarms.
•Helps in maintaining the health and performance of the system.
Process flow for image processing
1.Users -> ALB: Users send API requests to upload images.
2.ALB -> ECS Cluster: The ALB routes these requests to the web application running in the ECS Cluster.
3.ECS Cluster -> S3: The web application processes the upload requests and stores the images in S3.
●
For images from the Kafka stream, they are routed to the application in the ECS cluster before being stored in the S3
Conclusion of strategy
•This architecture addresses the key requirements:
•Secure access: Implemented through IAM, VPC design, and private subnets.
•Data security: Ensured by S3 encryption, VPC isolation, and secure transit via HTTPS.
•Scalability: Achieved through ECS, ALB, and CloudFront.
•Ease of maintenance: Facilitated by ECS for containerized deployments and managed services like RDS and ElastiCache.
•High Availability: Provided by multi-AZ deployments and services like ALB and RDS.
•Disaster Recovery: Supported by S3 cross-region replication and RDS backups.
•Efficiency and Low Latency: Addressed by ElastiCache and CloudFront.
•Least Privilege: Implemented through IAM roles and policies.
Leave a Reply