AWS SQS on local

We come across asynchronous use cases where the program runs for certain amount of time before giving the output. REST APIs certainly have an SLA to follow (Ex: Its 60 secs in our product)

Tasks which goes beyond 60 secs are considered asynchronous jobs and are handled with event driven systems

Typical Event driven systems have queues as part of their design pattern.
Whatever queues suite your use case.

AWS SQS is one such managed service which is easier to start with.

AWS Localstack allows you to work with sqs queues on your local machine for free.
https://docs.localstack.cloud/overview/

Installation:

I prefer doing most of the installations via docker

docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack

This will download the latest localstack image and runs it

You will get “Ready” in the output if it successfully runs

and you have passed the port 4566 for it to run. So it runs on http://localhost:4566

Install aws-cli (https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
or
awslocal( https://github.com/localstack/awscli-local)

if you want to manage queues, messages from cli

I am using sqs api in python using boto3. So you can go ahead install the python library boto3 to use it

pip install boto3

You are good to go!

Creating Queues:

There are 2 types of queues you can create

1. Standard queues
2. FIFO queues

I am using aws-cli to create the queue

sudo aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name queue_name --region eu-central-1

Note that the region is a dummy region. You bucket this queue inside that region.

On successful creation it gives an output something like

{
    "QueueUrl": "http://localhost:4566/000000000000/queue_name"
}

FIFO queue:

create a FIFO queue by adding the attributes like below

sudo aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name queue_name --region eu-central-1 --attributes "FifoQueue=true"

You will see the similar output to standard queue on successful creation

{
    "QueueUrl": "http://localhost:4566/000000000000/queue_name"
}

Sending and Recieving message to and from the queue:

1. Initialize an sqs client

import boto3
sqs_client = boto3.client("sqs",
                        endpoint_url="http://localhost:4566",
                        region_name="eu-central-1")

2.Send and Recieive messages to queue like below

Send message to a standard queue:

message_payload = {"name": "some name", "age": "some age"}
resp = sqs_client.send_message(
                QueueUrl="http://localhost:4566/000000000000/queue_name",
                MessageBody=json.dumps(message_payload)
                )

Send message to a FIFO queue:

import uuid
message_payload = {"name": "some name", "age": "some age"}
resp = sqs_client.send_message(
                QueueUrl="http://localhost:4566/000000000000/queue_name",
                MessageBody=json.dumps(message_payload),
                MessageGroupId="some-group",
                MessageDeduplicationId=str(uuid.uuid4()))

You can recieve the message like below for both types of queues

while True:
  message = sqs_client.receive_message(
                      QueueUrl=QUEUE_URL,
                      MaxNumberOfMessages=1,
                      WaitTimeSeconds=2)
  
  message = message.get("Messages", [])
  if message:
    message = messages[0]
    body = json.loads(message.get("Body"))
    receiptHandle = message.get("ReceiptHandle")

    #process message further here
  
    sqs_client.delete_message(QueueUrl=EVENTS_SQS.QUEUE_URL,
                                        ReceiptHandle=receiptHandle)