Designing a Scalable Media Storage Solution Using AWS EFS and EC2 Instances

 





Creating a Centralized and Scalable Media Storage Solution with AWS EFS and EC2 Instances

A media production company needs a centralized storage solution that is both scalable and accessible to multiple EC2 instances across different Availability Zones (AZs). The company manages large volumes of video files and other media assets, requiring efficient sharing among various production teams. Their key requirements include:

  • High Availability: The storage must be accessible at all times.
  • Seamless Scalability: It should handle fluctuating storage demands without manual intervention.
  • Top-Notch Performance: Fast access to large files is critical.
  • Data Security: Ensuring secure transmission between servers is a priority.

To meet these needs, the company chooses AWS Elastic File System (EFS). EFS provides a fully managed, elastic, and highly available shared file system. It enables multiple EC2 instances to access the same file system simultaneously, regardless of their location within the same AWS Region. This approach eliminates storage bottlenecks, allowing teams to collaborate in real time while working on shared media assets efficiently.




Step 1: Launching EC2 Instances in Multiple AZs

Goal: Deploy two EC2 instances in different AZs to ensure high availability and redundancy. Both instances will share access to an AWS Elastic File System (EFS).

Steps

  1. Create a Security Group: Define a security group for controlling access.


    aws ec2 create-security-group --group-name StorageLabs --description "Security group for EFS storage"

  2. Add an Inbound SSH Rule: Enable SSH access for administrative purposes.


    aws ec2 authorize-security-group-ingress --group-name StorageLabs --protocol tcp --port 22 --cidr 0.0.0.0/0

  3. Launch EC2 Instance in AZ us-east-1a: Deploy the first instance in AZ us-east-1a.


    aws ec2 run-instances --image-id ami-0866a3c8686eaeeba --instance-type t2.micro --placement AvailabilityZone=us-east-1a --security-group-ids sg-00055e6ae2bc876ce

  4. Launch EC2 Instance in AZ us-east-1b: Deploy the second instance in AZ us-east-1b.


    aws ec2 run-instances --image-id ami-0866a3c8686eaeeba --instance-type t2.micro --placement AvailabilityZone=us-east-1b --security-group-ids sg-00055e6ae2bc876ce

With these steps, you now have two EC2 instances in different AZs, ready for shared storage access.

Step 2: Creating and Configuring AWS EFS

Goal: Set up an AWS Elastic File System accessible by both EC2 instances.

Steps

  1. Create an EFS File System: Use the AWS Management Console to create a new EFS. Associate it with the security group created earlier. Default settings suffice for most use cases.

  2. Enable NFS Protocol: Add a rule to the security group to allow communication between EFS and the EC2 instances.


    aws ec2 authorize-security-group-ingress --group-id sg-00055e6ae2bc876ce --protocol tcp --port 2049 --source-group sg-00055e6ae2bc876ce

  3. Mount EFS on EC2 Instances:

    • Install the NFS client:


      sudo yum -y install nfs-utils
    • Create a mount point directory:

      mkdir ~/efs-mount-point
    • Mount the EFS:

      sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS-DNS>: ~/efs-mount-point

  4. Verify File Sharing: Create a file on one instance and confirm it appears on the other. This validates the shared access functionality.

Step 3: Enabling Encryption in Transit

Goal: Secure all data transfers between EC2 instances and the EFS using encryption in transit.

Steps

  1. Update File System Policy: Use the AWS Management Console to enforce in-transit encryption for the EFS.

  2. Unmount the File System:


    sudo umount ~/efs-mount-point

  3. Install Amazon EFS Utils:
    The default NFS client does not support in-transit encryption, so install EFS Utils to enable it.



    sudo yum install -y amazon-efs-utils

  4. Mount with Encryption:
    Remount the EFS with encryption enabled.


    sudo mount -t efs -o tls <EFS-DNS>: ~/efs-mount-point

Outcome

  • High Availability: Both EC2 instances in different AZs access the same EFS seamlessly.
  • Scalability: The EFS automatically scales to accommodate growing storage needs.
  • Performance: Optimized for low latency and high throughput.
  • Security: All data transfers are encrypted in transit, ensuring compliance with security standards.

This setup demonstrates how AWS EFS can provide a robust, secure, and scalable solution for media production teams needing efficient file sharing and collaboration across multiple AZs.


Comments

Post a Comment