AWS Kinesis

-

 AWS Kinesis




AWS Kinesis - 
  • Real-time data steaming service 
  • Used in digest data in real time directly from source 
  • Capacity Modes - 
  • Provisioned -
  • Publishing - 1mb/s per shard or 1000 msg/s per shard 
  • Consuming - 
  • 2mb/s per shard (throughput shared between all consumer)
  • Enhanced Fanout - 2mb/s per shard consumer (dedicated throughput for each consumer )
  • Throughput scales with shards (manual scaling)
  • Pay per shard provisioned per hour 
  • On-demand - 
  • No need to provision or manage the capacity
  •  Default capacity provisioned - 4mb/s or 4000 record/s
  • Scales automatically based on observed throughput peak during the last 30 days 
  • Pay per stream per hour and data in/out per GB

g
  • Not serverless
  • Data retention: 1day (default) to 365 days
  • A record consist of a partition key ( used to partition data coming from multiple publisher) and data blob (max 1mb)
  • Record will be ordered in each shard 
  • Producers SDK, kinesis producer library (KPL), or kinesis agent to publish records
  • Once data inserted in kinesis, it can't me modified or deleted 
  • Ability to reprocess (replay) data


Kinesis Data FireHouse -  
  • Used to load streaming data into target location
  • Writes data in batches efficiently (near real time) - 
  • Buffer size (size of the batch ) - 1 mb to 128 mb (default 5mb)
  • Buffer interval (how long to wait for buffer to fill up) - 60s to 900s (default 300s)
  • Greater the buffer size, higher the write efficiency, longer it will take to fill the buffer 
  • Can ingest data in real time directly from source 
  • Auto scaling
  • Serverless
  •  Pay for data going through firehouse (no provisioning)
  • Supports custom data transformation using lamda function  
  • No replay capabilities (does not store data like KDS)




Kinesis Data Analytics - 
  • Perform real-time analytics on kinesis streams using SQL 
  • Creates streams from SQL query response
  • Cannot ingest data directly from source (ingest data from KDS or KDF)
  • Auto scaling
  • Serverless
  • Pay for the data processed (no provision)
  • Use Cases - 
  • Time-series analytics
  • Real-time dashboard 
  • Real-time metrics 



Kinesis Video Streams - 
  • Capture, process, and store video streams


Comments

Post a Comment

Popular posts from this blog

AWS Instance Store

-
Image
 AWS Instance Store  AWS Instance Store -  Hardware storage directly attached to EC2 instance amd cannot be detached and attached to another instance  Highest IOPS for any available storage Ephemiral Storage (Loses data when instance is terminated, stopped or hibernated) Good to buffer/cache/scratch data/temporary content  AMI created from an instance does not its instance store volume preserved  You can specify the instance store volumes to an instance when you launch the instance. You can't attach instance store volumes to an instance after you’ve launched it. Raid -  Raid 0  Improve performance of a storage volume by distributing reads and write in a stripes across attached volumes  If you add a storage volume you get the straight addition of throughput and iops For high performance applications Raid 1   Improve data availability by mirroring in multiple volumes For critical applications
Read more

AWS Identity and Access Management

-
Image
  IAM Introduction to IAM -  AWS identity and management is a web service that helps you to securely control access to AWS resources You use IAM to control who is authenticated and authorized to use resources IAM allows you to create and manage AWS users and groups , and use permissions to allow or deny their access to AWS resources IAM is a fundamental part of AWS security, as it allows you to implement the principle of least privilege , ensuring that users and applications have only the permissions they need to perform their tasks IAM also provides features such as multi-factor authentication (MFA) , password policies, and access keys to enhance the security of your AWS environment Features of IAM -  IAM User  -   IAM allows you to create and manage users in your AWS account Each IAM user has a unique name and security credentials, which are used to authenticate and authorize their access to AWS resources IAM users have security credentials that are us...
Read more

Elastic Block Storage (EBS)

-
Image
 Elastic Block Storage (EBS) Elastic Block Storage (EBS) -  AWS Elastic Block Storage is a web service that provides block level storage volumes for use with amazon EC2 instances Amazon EBS volumes are highly available and reliable storage volumes that can be attached to any running instance and used like hard drive Amazon EBS allows you to create storage volumes and attach them to EC2 instances   It is used for various databses (both relational and non relational) as well as a wide range of application such as software testing and development  EBS provides a high performance option for many use cases it is directly attached to the instance Features of EBS  -  Amazon Elastic Block Storage  Region and availability zones are multiple physical location for your resources such as instances and EBS volumes Using security groups, you can specify the protocols, ports and source IP ranges that can reach your instances Elastic IP addresses are static IPv4 ...
Read more