AWS SQS

 AWS SQS



AWS SQS - 

  • Used to asychronously decouple application 
  • Suppports multiple producers and consumers 
  • The consumer polls the queue for messages. Once a consumer processes a messages, it deletes it from the queue using DeleteMessage API
  • Unlimited number of messages in queue
  • Max 10 messages recieved per batch (configured using MaxNumberOfMessages parameter in the ReceiveMessage API)
  • Max message size - 256kb
  •  Consumers could be EC2 instances or Lamda function
  • SQS cannot ingest data, it must be sent to the queue by the producer (use kinesis - kinesis data stream kds instead)

Queue Types - 

Standard queue - 
  • Unlimited throughput (publish any number of message per second into the queue)
  • Low latency (<10 ms on publish and recieve)
  • Can have duplicate messages (at least once delivery)
  • Can have out of order messages (best effort ording)
Fifo queue -
  • Limited throughput - 
  • 300 msg/s without batching (batch size=1)
  • 3000 msg/s withour batching (batch size=10)
  • The queue name must end with .fifo to be considered a fifo queue 
  • Message De-duplication - 
  • De-duplication interval - 5min (duplicate messages will be discarded only if they are sent less than 5 mins apart)
  • De-duplication methods - 
  • Content-based-de-duplication : computes the hash of the messages body and compares
  • Using a message de-duplication ID - messages with the same de-duplication ID considered duplicates
  • Message Grouping - 
  • Group messages based on MessageGroupID to send them to different consumers
  • Same valur for MessageGroupID
  • All the messages are in order 
  • Single consumer 
  • Different values for MessageGroupID
  • Message will be ordered for each group ID 
  • Ordering across groups is not guaranteed 
  • Each group ID an have different consumer (parallel processing)
  • Max number of consumers = number of unique group IDs


Consumer Auto Scaling - 
  • We can attach an ASG to the consumer instances which will scale based on the CW metrics=Queue length/number of EC2 instances. CW alarms can be triggered to step the scale the consumer application

Encryption - 
  • In-flight encryption using HTTPS API
  • At-rest server-side encyption :
  • SSE-SQS: keys managed by SQS
  • SSE-KMS: keys managed by KMS
  • Client-side encryption

Configuration - 

Message visibility Timeout - 
  • Once a message is polled by a consumer, it becomes invisible to other consumers for the duration of message visibility timeout, After the message visibility timeout is over, the message become visible in the queue if it is not deleted by the consumer 
  • If a consumer dies while processing the message, it will be visible in the queue after the visibility timeout 
  • If a message is not processed within the visibility timeout, it will be processed again (by another consumer). However, a consumer could call the ChangeMessageVisibility API to change the visibility timeout for that specific message. This will give the consumer more time to process the message
  • Default-30s
  • Can be configured for the entire queue
  • High-if the consumer crashes,re-processing will take long 
  • Low-may get duplicate processing of messages



Dead Letter Queue - 
  • An SQS queue used to store failed to be processe3d messages in another queue
  • After the MaximumReceives threshold is exceeded, the message goes into the DLQ
  • Redrive to source - once the bug in the consumer has been resolved, messages in the DLQ can be sent back to the queue (original queue or a citom queue) for processing
  • Prevents resource wastage 
  • Recommend to set high retention period for 14 days 

Queue Delay - 
  • Consumer see the message after some delay
  • Default: 0(max 15min)
  • Can be set at the queue level 
  • Can override the default queue delay for a specific message using the DelaySecond parameter in the SendMessage API

Long polling - 
  • Poll the queue for long
  • Decreases the number of API calls made to SQS 
  • Reduces latency 
  • Polling time: 1 sec to 20 sec
  • Long polling is prefered over Short polling
  • Can be enabled to the queue level or at the consumer level by using WaitTimeSecond parameter in API 

Comments

Popular posts from this blog

AWS Instance Store

AWS Identity and Access Management

Elastic Block Storage (EBS)