Please wait..

Azure Messaging and Event Services

YASH Blog

Publish date August 2, 2019

Sourabh Babaji Nighot

Sourabh Babaji Nighot Sr. Software Engineer, Innovation Group – Cloud|Azure

RSS FeedRss Feed

Microsoft Azure offers four services for messaging and events in Azure: Service Bus, Storage Queues, Event Hubs, and Event Grid.

Out of these four two are pecifically for events – Event Hubs and Event Grid, and the other two Service Bus and Storage queues  for Azure messaging . Before discussing these Azure Services – let’s describe what are events and messages.

Messages Events
A message is raw data produced by a service to be consumed or stored elsewhere. An event is a lightweight notification of a state change.
The publisher of the message expects how the consumer handles the message. The publisher of the event does not expect how the event is handled
E.g. The file sent as a message containing the data. E.g. An event notifies consumers that a file is created. It may contain general information about the file, but it doesn’t have the file itself.

 

When to use which?

Service Purpose Type When to use
Event Grid Reactive programming Event distribution (discrete) React to status changes
Event Hubs Big data pipeline Event streaming (series) Telemetry and distributed data streaming
Service Bus High-value enterprise messaging Message Order processing and financial transactions
Storage queue Standard queuing scenarios, load leveling, and building process workflows Message  Applications which need to store large sizes of messages in a queue.

 

Azure Event Grid

What is Azure Event Grid?

Azure Event Grid is used where there is a requirement for building applications with event-based architectures. You can select the azure services for which you want to receive events and select the endpoint to send them.

Built-in support is provided by Event Grid for events coming from Azure services, like storage blobs and resource groups. Event grid also has support for your events in your application using custom topics.

Event grid also provides filters for the event so that you can filter out the required events for processing. Filters can also be used to route particular events to different endpoints, multicast to multiple endpoints, and ensure your events are reliably delivered.

Five concepts in Azure Event Grid let you get going:

  • Events– What happened?
  • Event sources– Where the event took place?
  • Topics– The endpoint where publishers send events.
  • Event subscriptions– The endpoint or built-in mechanism to route events, sometimes to more than one handler. Subscriptions are also used by handlers to filter incoming events intelligently.
  • Event handlers– The app or service reacting to the event.
Azure Event Grid

Where to use Event Grid?

Serverless application architectures
Data sources and event handlers are connected by Event. For example, use Event Grid to trigger a serverless function that analyzes images when added to a blob storage container.
Ops Automation
Automation process can be speeded up by using Event Grid. For example, use Event Grid to notify Azure Automation when a virtual machine or SQL database is created. Use the events to automatically check/verify that service configurations are compliant, put metadata into operations tools, tag virtual machines, or file work items.
Application integration
The application can be connected with other services using Event Grid. For example, create a custom topic to send your app’s event data to Event Grid, and take advantage of its reliable delivery, advanced routing, and direct integration with Azure.

Use Case:

There is a Content Management System application, and thousands of media files are uploaded daily. Admin wants to get notified whenever a new file is added/modified in azure storage via email. So we can create a Logic App that monitors the changes to storage account and sends emails about those changes.

When you create a logic app with an event subscription for an Azure resource, events flow from that resource through an event grid to the logic app. Also, we can apply various filters for the events e.g., Send the events when only a file is added or modified.

Azure Event Hubs

Azure Event Hubs

What is Event Hubs?

Azure Event Hubs is a hyper-scale telemetry ingestion service that collects, transforms, and stores millions of events. As a distributed streaming platform, it gives you low latency and configurable time retention, which enables you to ingress massive amounts of telemetry into the cloud and read the data from multiple applications using publish-subscribe semantics.

It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.

Why Event Hubs?

Event Hubs are perfect for analyzing big data. This amount of data can be huge and impossible to crunch, without a managed event service that has elastic scale capabilities. By the term “elastic,” we mean that it can handle spikes when they occur and it works with load profiles of different sizes.

Other than this, the key features are:

  • Fully managed PaaS
  • Support for real-time and batch processing
  • Scalable

Key architecture components:

Event Hubs contains the following key components:

  • Event producers: Any entity that sends data to an event hub. Event publishers can publish events using HTTPS or AMQP 1.0 or Apache Kafka (1.0 and above)
  • Partitions: Each consumer only reads a specific subset, or partition, of the message stream.
  • Consumer groups: A view (state, position, or offset) of an entire event hub. Consumer groups enable consuming applications to each have a separate view of the event stream. They read the stream independently at their own pace and with their own offsets.
  • Throughput units: Pre-purchased units of capacity that controls the throughput capacity of Event Hubs.
  • Event receivers: Any entity that reads event data from an event hub. All Event Hubs consumers connect via the AMQP 1.0 session. The Event Hubs service delivers events through a session as they become available. All Kafka consumers connect via the Kafka protocol 1.0 and later.

The following figure shows the Event Hubs stream processing architecture:

Event Hubs stream processing architecture

Use Case:

Several IoT devices are sending weather-related data, and it is required, to store the streaming data in storage and also stream the data with stream analytics so that it can be shown on the dashboard. The data can be ingested into event hubs and then from event hubs to storage and stream analytics.

stream analytics

Azure Service Bus 

What is Azure Service Bus?

Azure Service Bus is a messaging service on cloud used to connect any applications, devices, and services in the cloud to any other applications or services. It acts as a messaging backbone for applications in the cloud or across any devices.

Service Bus Concepts:
Some typical common messaging scenarios are:

  • Messaging: transfer business data, such as sales or purchase orders, journals, or inventory movements.
  • Decouple applications: improve reliability and scalability of applications and services (client and service do not have to be online at the same time).
  • Topics and subscriptions: enable 1:n relationships between publishers and subscribers.
  • Message sessions: implement workflows that require message ordering or message deferral.

The critical  sercapabilities of the service bus are:

  • Queues: Queues are offering an asynchronous messaging capability with first-in-first-out (FIFO) message delivery. Only one consumer receives each message.

Here a sender will send a message to a queue and the receiver will collect it in initial or later stage from that queue. Besides this, queues enable us to store the message until the receiver is available to receive and process them. The user will receive the message upon request.

Message Queue
  • Topics and subscriptions: Topics and subscriptions offer a similar capability as queues. However, there could be more consumers for messages sent to a topic. A topic has one or more subscriptions and they can have filters, and thus, various messages to a topic can end up in different subscriptions belonging to the topic.
Topics and subscriptions

  • Relay: In Service Bus Relay, we have an endpoint that is hosted in the cloud, but the difference would be the fact that our on-premise service will make an outbound connection to register with the Service Bus relay endpoint. At a conceptual level, we send a message to an endpoint in the cloud from where it will be forwarded to an on-premise service. As a result, it gives us the advantage to connect from one organization to another organization without having to set VPNs.

Service Bus Relay can be used to solve problems in scenarios like,

  1. Information passed between two data centers.
  2. Building integration architecture through SAP to forward messages into microservice architecture using Service Bus Relay.

Advanced features:

Message sessions
Sessions are used to realize a first-in, first-out (FIFO) guarantee in Service Bus. Message sessions enable joint and ordered handling of unbounded sequences of related messages.

Auto-forwarding
The auto-forwarding feature enables you to chain a queue or subscription to another queue or topic in the same namespace.

Dead-lettering
Service Bus supports a dead-letter queue (DLQ) to hold messages that cannot be delivered to any receiver, or messages that cannot be processed.

Scheduled delivery
You can submit messages to a queue or topic for delayed processing; for example, to schedule a job to become available for processing by a system at a specific time.

Message deferral
When a queue or subscription client receives a message that it is willing to process, but due to special circumstances within the applicationfor such processing is not currently possible , the entity has the option to  and can defer retrieval of the message to a later point. The message remains in the queue or subscription, but it is set aside.

Batching
Client-side batching enables a queue or topic client to delay sending a message for a specified period. If the client sends additional messages during this period, it transmits the messages in a single batch.

Transactions
transaction groups two or more operations into an execution scope. Service Bus supports grouping operations against a single messaging entity (queue, topic, subscription) within the scope of a transaction.

Filtering and actions
Subscribers can define which messages  to receive from a topic. These messages are specified in the form of one or more named subscription rules. For each matching rule condition, the subscription produces a copy of the message, which may be differently annotated for each matching rule.

Auto-delete on idle
Auto-delete on idle enables you to configure an idle interval after which the queue is automatically deleted. The minimum duration for idle interval is 5 minutes.

Duplicate detection
If an error occurs that causes the client to doubt the outcome of a send operation, duplicate detection helps clear that doubt by enabling the sender to re-send the same message, and the queue or topic discards any duplicate copies.

Use Case:

One of the utility bill payments company wants to build an application, which will do the mobile recharge for end users. The application should support all the operators which are available in the market. The flow is, the user sends SMS in predefined format and application should parse the message and do the recharge accordingly.

Now, the foremost challenge is that the application unable to process  a vast number of recharge requests it gets during the peak hours, so the company is looking for a solution where all the requests are parked when they come and processed one by one.

To meet the business requirement, use the queue mechanism where all the requests are added in the queue, and on the another side queue listener will process the message by reading from the queue.

Azure Storage Queues

What is Azure storage queue?

Azure Queue storage is a service for storing large numbers of messages. Messages can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message size can be up to 64 KB. A queue may contain messages up to the total capacity limit of a storage account.

Where to use storage queues?

  • Creating a backlog of work to process asynchronously
  • Passing messages from an Azure web role to an Azure worker role

Use Case:

A content management system application where files are uploaded by a number of users daily. The files need to be processed asynchronously by a background process which filters out the media files and sends them for transcoding.

When the users upload the file, it is pushed to queue, and the background process will process the files from the queue asynchronously.

When to use Storage Queue and Service bus Queue?

Storage Queue Service bus Queue
Your application should store over 80 GB of messages in a queue. Your solution should be able to receive messages without having to poll the queue
Your application needs to track progress for processing a message inside of the queue. This is useful if the worker processing message crashes. A subsequent worker can then use that information to continue from where the prior worker left off. Your solution needs the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You require server-side logs of all of the transactions executed against your queues. Your solution must support automatic duplicate detection.
No role-based access supported You deal with a requirement to provide a role-based access model to the queues, and different rights or permissions for senders and receivers.
No dead lettering in storage queue. Dead lettering, supported only  by Service Bus queues, can be useful for isolating messages that cannot be processed successfully by the receiving application
Maximum message size is 64Kb Maximum message size 256 KB or 1 MB

 
References:

https://docs.microsoft.com
https://www.serverless360.com

Comments

No Comments

Add Comments

Type in a topic service or offering and then hit enter to search

Thank you for your message. It has been sent.
X