Building Scalable Messaging Systems with AWS SQS & SNS

Introduction

Modern applications require robust messaging systems to handle asynchronous communication, decouple components, and scale effectively. Amazon Web Services provides two powerful messaging services: Simple Queue Service (SQS) for message queuing and Simple Notification Service (SNS) for publish-subscribe patterns.

This comprehensive tutorial will guide you through building a complete messaging application using AWS SQS and SNS. We'll explore core concepts, implement real-world patterns, and demonstrate how these services work together to create scalable, resilient architectures.

Understanding AWS Messaging Services

AWS SQS and SNS serve different but complementary purposes in distributed systems. SQS provides reliable message queuing for point-to-point communication, while SNS enables fan-out messaging through publish-subscribe patterns.

Core SQS Concepts

Message Queue Fundamentals

SQS provides a managed message queue service that enables asynchronous communication between distributed application components. Messages are stored reliably until they are processed and deleted by consumers.

// Sending a message to SQS
const params = {
  QueueUrl: process.env.SQS_QUEUE_URL,
  MessageBody: JSON.stringify({
    orderId: '12345',
    customerId: 'user-456',
    items: ['product-1', 'product-2'],
    timestamp: new Date().toISOString()
  }),
  MessageAttributes: {
    'MessageType': {
      DataType: 'String',
      StringValue: 'ORDER_CREATED'
    }
  }
};

const result = await sqsClient.send(new SendMessageCommand(params));

Message Processing Patterns

SQS supports both short polling and long polling for message retrieval. Long polling reduces costs and improves efficiency by waiting for messages to arrive rather than immediately returning empty responses.

// Receiving messages with long polling
const receiveParams = {
  QueueUrl: process.env.SQS_QUEUE_URL,
  MaxNumberOfMessages: 10,
  WaitTimeSeconds: 20, // Long polling
  MessageAttributeNames: ['All']
};

const messages = await sqsClient.send(
  new ReceiveMessageCommand(receiveParams)
);

// Process each message
for (const message of messages.Messages || []) {
  try {
    // Process the message
    await processMessage(JSON.parse(message.Body));
    
    // Delete the message after successful processing
    await sqsClient.send(new DeleteMessageCommand({
      QueueUrl: process.env.SQS_QUEUE_URL,
      ReceiptHandle: message.ReceiptHandle
    }));
  } catch (error) {
    console.error('Message processing failed:', error);
    // Message will become visible again after visibility timeout
  }
}

Core SNS Concepts

Topics and Subscriptions

SNS topics act as communication channels where publishers send messages and subscribers receive them. A single message published to a topic is delivered to all active subscriptions.

// Publishing a message to SNS topic
const publishParams = {
  TopicArn: process.env.SNS_TOPIC_ARN,
  Message: JSON.stringify({
    event: 'USER_REGISTERED',
    userId: 'user-789',
    email: 'user@example.com',
    timestamp: new Date().toISOString()
  }),
  MessageAttributes: {
    'event_type': {
      DataType: 'String',
      StringValue: 'USER_REGISTERED'
    },
    'priority': {
      DataType: 'String',
      StringValue: 'high'
    }
  }
};

const result = await snsClient.send(new PublishCommand(publishParams));

Subscription Management

SNS supports multiple subscription protocols including email, SMS, HTTP/HTTPS endpoints, and SQS queues. This flexibility allows you to integrate with various systems and notification channels.

// Creating subscriptions for different protocols
const subscriptions = [
  {
    Protocol: 'email',
    Endpoint: 'admin@example.com'
  },
  {
    Protocol: 'sqs',
    Endpoint: 'arn:aws:sqs:us-east-1:123456789012:notification-queue'
  },
  {
    Protocol: 'https',
    Endpoint: 'https://api.example.com/webhooks/notifications'
  }
];

for (const subscription of subscriptions) {
  await snsClient.send(new SubscribeCommand({
    TopicArn: process.env.SNS_TOPIC_ARN,
    Protocol: subscription.Protocol,
    Endpoint: subscription.Endpoint
  }));
}

Building the Demo Application

Our demo application showcases practical implementations of both SQS and SNS using Next.js 16 with the App Router. The application demonstrates real-world messaging patterns and best practices for AWS integration.

Application Architecture

The demo follows a modern full-stack architecture with clear separation between frontend components, API routes, and AWS service integrations:

Setting Up AWS Services

The application requires proper AWS configuration including IAM permissions, SQS queue creation, and SNS topic setup. The repository includes detailed setup instructions:

// AWS SDK v3 Configuration
import { SQSClient } from '@aws-sdk/client-sqs';
import { SNSClient } from '@aws-sdk/client-sns';

const awsConfig = {
  region: process.env.AWS_REGION || 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
};

export const sqsClient = new SQSClient(awsConfig);
export const snsClient = new SNSClient(awsConfig);

Component Architecture

The application uses a modular component structure with custom hooks for state management and API interactions. This approach provides clean separation of concerns and reusable logic.

// Custom hook for SQS operations
export function useSQS() {
  const [messages, setMessages] = useState<SQSMessage[]>([]);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);

  const sendMessage = async (messageBody: string) => {
    setLoading(true);
    setError(null);
    
    try {
      const response = await fetch('/api/sqs/send', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ message: messageBody }),
      });
      
      if (!response.ok) throw new Error('Failed to send message');
      
      const result = await response.json();
      return result;
    } catch (err) {
      setError(err instanceof Error ? err.message : 'Unknown error');
      throw err;
    } finally {
      setLoading(false);
    }
  };

  const receiveMessages = async () => {
    // Implementation for receiving messages
  };

  return { messages, loading, error, sendMessage, receiveMessages };
}

Advanced Messaging Patterns

SQS + SNS Integration

Combining SQS and SNS creates powerful messaging architectures. SNS can fan out messages to multiple SQS queues, enabling parallel processing of the same event by different services.

Error Handling and Retry Logic

Proper error handling is crucial for reliable messaging systems. The demo implements comprehensive error handling with exponential backoff and dead letter queue patterns.

Message Filtering and Routing

SNS message filtering allows subscribers to receive only relevant messages based on message attributes. This reduces processing overhead and improves system efficiency.

Key Benefits of AWS Messaging

Getting Started

Ready to build scalable messaging systems with AWS? Follow these steps to get the tutorial project running on your local machine:

Learning Outcomes

By completing this tutorial, you will have gained hands-on experience with:

• • Setting up and configuring AWS SQS queues and SNS topics
• • Implementing message sending and receiving with proper error handling
• • Building publish-subscribe patterns with SNS subscriptions
• • Integrating AWS SDK v3 with Next.js applications
• • Managing AWS credentials and IAM permissions securely
• • Implementing custom React hooks for AWS service interactions
• • Understanding message queuing and pub/sub architectural patterns
• • Best practices for scalable, decoupled application design

Conclusion

AWS SQS and SNS provide the foundation for building scalable, resilient messaging systems. By decoupling application components and enabling asynchronous communication, these services help create applications that can handle varying loads and recover gracefully from failures.

The demo application demonstrates practical implementation patterns that you can adapt for your own projects. From simple message queuing to complex event-driven architectures, these concepts form the backbone of modern cloud-native applications.

About the Author

Wayne Cheng is the founder and AI app developer at Audoir, LLC. Prior to founding Audoir, he worked as a hardware design engineer for Silicon Valley startups and an audio engineer for creative organizations. He holds an MSEE from UC Davis and a Music Technology degree from Foothill College.

Further Exploration

To continue your AWS messaging journey, explore the complete tutorial repository and experiment with advanced features like dead letter queues, message filtering, and Lambda integration. Consider implementing cross-service communication patterns to deepen your understanding of event-driven architectures.

For more AI-powered development tools and tutorials, visit Audoir .