AWS S3 (Simple Sorage Service)

-

  S3 (Simple Sorage Service)



S3 Bucket - 

  • AWS S3 allows people to store objects in buckets
  • Buckets must have globally unique (across all regions all account)
  • Buckets are defined at a region level
  • S3 looks like a global service but buckets are created in region
  • Naming Convention - 
  • No uppercase , No underscore
  • 3-63 character long
  • Not an IP
  • Must start with lowercase letter or number 
  • Must not start with prefix xn--
  • Must not end with suffix -s3alias

S3 Object - 

  • Object have a key 
  • The key is th full path
S3://my-bucket/myfile.txt

S3://my-bucket/myfolder1/another_folder/myfile.txt/

  • There is no concept of directories within buckets
  • Just keys with very long names that contains slashes ("/")

AWS S3 Security - 

  • User based
  • IAM Policies - which calls should be allowed for a specific user from IAM 
  • Resource-based - 
  • Bucket Policies - Bucket wide rules from S3 consol-allows cross account 
  • Object Access Control List (ACL) - finer grain (can be disabled)
  • Bucket Access Control List (ACL) - less common (can be disbaled)
  • Note - an IAM principal can access an S3 object if 
                 -The user IAM permissions ALLOW it OR the resource policy ALLOWS it

                 -AND there is no explicit deny

  • ENCRYPTION : encrypt objects in Amazon S3 using encryption keys

AWS S3 - Bucket Policies - 

  • JSON Based Policy : Resource:buckets and objects, Effects:Allow / Deny, Actions: set of API to Allow or Deny, Principal - The account or user to apply user
  • Use S3 bucket for policy to : Grant public access to the bucket, Force objects to be encrypted at upload, Grant access to another account to another (cross account)

AWS S3 - Static Web Hosting - 
  • S3 can host static websites and have them accessible on
  • The website URL will be (depending on the region)
  • If you get a 403 forbidden error, make sure the bucket
  • Policy allows public reads

AWS S3 - Version - 

  • You can version files in amazon S3
  • It is enabled at the bucket level 
  • Some key overwrite will change the "version": 1,2,3....
  • It is best practice to version your buckets 
  • Protect against unintended deletes
  • Easy roll back to previous version 
  • Any file that is not versioned prior to enabling versioning will have version "null"
  • Suspending versioning does not delete the previous version

AWS S3 - Replication (CRR & SRR) - 

  • Must Enable versioning in source and destination bucket 
  • cross region replication (CRR)
  • some region replication (SRR)
  • Buckets can be in different AWS account 
  • Copy is asychronous 
  • Must give proper IAM permissions to S3
  • Use Cases - 
  • CRR - compliance, lower latency access, replication across accounts
  • SRR - log aggregation, live replication between production and test accounts
AWS S3 Replication (notes) - 

  • After you enable replication, only new objects are  replicated
  • Optionally, you can replicate existing objects using S3 batch replication
  • Replicates existing objects and objects that failed replication 
  • For delete operation - 
  • Can replicate delete markers from source to target 
  • Deletation with a version id are not replicate 
  • There is no chaining of replication 
  • If bucket one has replication into bucket 2 which has replication into bucket 3
  • then objects created in bucket 1 are not replicated to bucket 3

AWS S3 Durability and  Availability - 

  • Durability -
  • High durability (99.999999999) 11 9s of objects across multiple AZ
  • If you store 1cr objects in amazon S3, you can on avarage expect to incure a loss of a single object once every 10k years 
  • Same for all storage classes 
  • Availabilty - 
  • Measures how readily available a service is
  • Varies depending on storage class
  • Example : S3 standard has 99.99% availabilty=not available 53minutes a year

S3 Storage classes - 

S3 standard - General Purpose - 

  • 99.99% availabilty 
  • Used for frequently accessed data
  • Low latency and high throughput 
  • Sustain 2 concurrent facility failures
  • Uses cases - 
  • Big data analytics, mobile and gaming application, content distribution
S3 Storage Classes - Infrequent Access -  

  • For data is less frequently accessed,but requires rapid access when needed
  • Lower cost that S3 standard
  • 99% availability
  • Use cases - disaster recovery, backups
S3 Storage Classes -One Zone-Infrequent Access -

  • High durability (99.999999999) in a single AZ, data is lost when AZ is destroyed
  • 99.5% availability
  • Use cases - Storing secondary backup copies of On-premises data, or data you can recreate


AWS S3 Glacier Storage Classes - 

  • Low-cost object storage meant for archiving/backup
  • Pricing: price of storage + object retrieval cost
  • Amazon S3 Glacier Instanst retrieval - 
  • Millisecond retrieval, data accessed once a quarter
  • Minimum storage of 90 days
  • Amazon S3 Glacier Flexible Retrieval - 
  • Expedited (1 to 5 mins), standard(3 to 5 hrs), bulk (5 to 11 hrs)
  • Minimum storage of 90 days
  • Amazon S3 Glacier Deep Archive - 
  • standard(12hrs), bulk(48hrs)
  • Minimum storage duration of 180 days

AWS S3 Intelligent-Tiering - 
  • Small monthly monitoring and auto-tiering fee
  • Moves objects automatically between access tiers based on usage 
  • There are no retrieval charges 
  • Frequent Access Tier : default tier
  • Infrequent Access Tier : Objects not accessed for 30 days
  • Archieve Instant Access Tier : Objects not Accesssed for 90 days
  • Archieve Access Tier : configurable from 90 days to 700+ days
  • Deep Archieve Access Tier : configurable from 180 days 700+ days



AWS S3 - Lifecycle Rules - 
  • Transition Actions - Configure objects to transition to another storage class 
  • Move objects to standard IA class 60 days after creation
  • Move to glacier for archieving after 6 months 
  • Expiration Action - configure objects to expire after same time 
  • Access log files can be set to delete after 365 days
  • Can be used to delete old versions of files
  • Can be used to delete incomplete multi-part uploads
  • Rules can be created for a certain prefix 
  • Rules can be created for certain objects tags 

AWS S3 Analytics - Storage Class Analysis - 
  • Help you decide when to transition objects to the right storage class
  • Recommendation for standard and standard IA
  • Does not work for one-zoned IA and glacier 
  • Report is updated daily
  • 24 to 48 hrs to start seeing data analysis 
  • Good first step to put together lifecycle rules

AWS S3 Performance - 
  • Multi-part upload - 
  • Recommend for files > 100mb, must use for files >5gb 
  • Can help parallelize uploads 
  • S3 Transfer Acceleration - 
  • Increase tranfer speed by tranferring files to an AWS edge location which will forward the data to the S3 bucket in the target region 
  • Compativle with multi part upload


AWS S3 Batch Operation - 

Performs bulk operations on existing S3 objects with a single request. For Ex. - 
  • Modify object metadata and properties
  • Copy objects between  S3 buckets
  • Encrypt un-encypted objects
  • Modify ACls,tags
  • Restore objects from S3 glaciers
  • Invoke Lambda Function to perform custom action on each object
  • A job consists of a list of objects, the actions to perform and optional parameters 
  • S3 batch operation manages retries, track progress, sends completion notification, generate reports 
  • You can use S3 inventory to get object list and use S3 select to filter your objects
 
  

Comments

Post a Comment

Popular posts from this blog

AWS Instance Store

-
Image
 AWS Instance Store  AWS Instance Store -  Hardware storage directly attached to EC2 instance amd cannot be detached and attached to another instance  Highest IOPS for any available storage Ephemiral Storage (Loses data when instance is terminated, stopped or hibernated) Good to buffer/cache/scratch data/temporary content  AMI created from an instance does not its instance store volume preserved  You can specify the instance store volumes to an instance when you launch the instance. You can't attach instance store volumes to an instance after you’ve launched it. Raid -  Raid 0  Improve performance of a storage volume by distributing reads and write in a stripes across attached volumes  If you add a storage volume you get the straight addition of throughput and iops For high performance applications Raid 1   Improve data availability by mirroring in multiple volumes For critical applications
Read more

AWS Identity and Access Management

-
Image
  IAM Introduction to IAM -  AWS identity and management is a web service that helps you to securely control access to AWS resources You use IAM to control who is authenticated and authorized to use resources IAM allows you to create and manage AWS users and groups , and use permissions to allow or deny their access to AWS resources IAM is a fundamental part of AWS security, as it allows you to implement the principle of least privilege , ensuring that users and applications have only the permissions they need to perform their tasks IAM also provides features such as multi-factor authentication (MFA) , password policies, and access keys to enhance the security of your AWS environment Features of IAM -  IAM User  -   IAM allows you to create and manage users in your AWS account Each IAM user has a unique name and security credentials, which are used to authenticate and authorize their access to AWS resources IAM users have security credentials that are us...
Read more

Elastic Block Storage (EBS)

-
Image
 Elastic Block Storage (EBS) Elastic Block Storage (EBS) -  AWS Elastic Block Storage is a web service that provides block level storage volumes for use with amazon EC2 instances Amazon EBS volumes are highly available and reliable storage volumes that can be attached to any running instance and used like hard drive Amazon EBS allows you to create storage volumes and attach them to EC2 instances   It is used for various databses (both relational and non relational) as well as a wide range of application such as software testing and development  EBS provides a high performance option for many use cases it is directly attached to the instance Features of EBS  -  Amazon Elastic Block Storage  Region and availability zones are multiple physical location for your resources such as instances and EBS volumes Using security groups, you can specify the protocols, ports and source IP ranges that can reach your instances Elastic IP addresses are static IPv4 ...
Read more