Publish date August 2, 2019
Microsoft Azure offers four services for messaging and events in Azure: Service Bus, Storage Queues, Event Hubs, and Event Grid.
Out of these four two are pecifically for events – Event Hubs and Event Grid, and the other two Service Bus and Storage queues for Azure messaging . Before discussing these Azure Services – let’s describe what are events and messages.
|A message is raw data produced by a service to be consumed or stored elsewhere.||An event is a lightweight notification of a state change.|
|The publisher of the message expects how the consumer handles the message.||The publisher of the event does not expect how the event is handled|
|E.g. The file sent as a message containing the data.||E.g. An event notifies consumers that a file is created. It may contain general information about the file, but it doesn’t have the file itself.|
When to use which?
|Service||Purpose||Type||When to use|
|Event Grid||Reactive programming||Event distribution (discrete)||React to status changes|
|Event Hubs||Big data pipeline||Event streaming (series)||Telemetry and distributed data streaming|
|Service Bus||High-value enterprise messaging||Message||Order processing and financial transactions|
|Storage queue||Standard queuing scenarios, load leveling, and building process workflows||Message||Applications which need to store large sizes of messages in a queue.|
Azure Event Grid
What is Azure Event Grid?
Azure Event Grid is used where there is a requirement for building applications with event-based architectures. You can select the azure services for which you want to receive events and select the endpoint to send them.
Built-in support is provided by Event Grid for events coming from Azure services, like storage blobs and resource groups. Event grid also has support for your events in your application using custom topics.
Event grid also provides filters for the event so that you can filter out the required events for processing. Filters can also be used to route particular events to different endpoints, multicast to multiple endpoints, and ensure your events are reliably delivered.
Five concepts in Azure Event Grid let you get going:
Where to use Event Grid?
Serverless application architectures
Data sources and event handlers are connected by Event. For example, use Event Grid to trigger a serverless function that analyzes images when added to a blob storage container.
Automation process can be speeded up by using Event Grid. For example, use Event Grid to notify Azure Automation when a virtual machine or SQL database is created. Use the events to automatically check/verify that service configurations are compliant, put metadata into operations tools, tag virtual machines, or file work items.
The application can be connected with other services using Event Grid. For example, create a custom topic to send your app’s event data to Event Grid, and take advantage of its reliable delivery, advanced routing, and direct integration with Azure.
There is a Content Management System application, and thousands of media files are uploaded daily. Admin wants to get notified whenever a new file is added/modified in azure storage via email. So we can create a Logic App that monitors the changes to storage account and sends emails about those changes.
When you create a logic app with an event subscription for an Azure resource, events flow from that resource through an event grid to the logic app. Also, we can apply various filters for the events e.g., Send the events when only a file is added or modified.
Azure Event Hubs
What is Event Hubs?
Azure Event Hubs is a hyper-scale telemetry ingestion service that collects, transforms, and stores millions of events. As a distributed streaming platform, it gives you low latency and configurable time retention, which enables you to ingress massive amounts of telemetry into the cloud and read the data from multiple applications using publish-subscribe semantics.
It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.
Why Event Hubs?
Event Hubs are perfect for analyzing big data. This amount of data can be huge and impossible to crunch, without a managed event service that has elastic scale capabilities. By the term “elastic,” we mean that it can handle spikes when they occur and it works with load profiles of different sizes.
Other than this, the key features are:
Key architecture components:
Event Hubs contains the following key components:
The following figure shows the Event Hubs stream processing architecture:
Several IoT devices are sending weather-related data, and it is required, to store the streaming data in storage and also stream the data with stream analytics so that it can be shown on the dashboard. The data can be ingested into event hubs and then from event hubs to storage and stream analytics.
Azure Service Bus
What is Azure Service Bus?
Azure Service Bus is a messaging service on cloud used to connect any applications, devices, and services in the cloud to any other applications or services. It acts as a messaging backbone for applications in the cloud or across any devices.
Service Bus Concepts:
Some typical common messaging scenarios are:
The critical sercapabilities of the service bus are:
Here a sender will send a message to a queue and the receiver will collect it in initial or later stage from that queue. Besides this, queues enable us to store the message until the receiver is available to receive and process them. The user will receive the message upon request.
Service Bus Relay can be used to solve problems in scenarios like,
Sessions are used to realize a first-in, first-out (FIFO) guarantee in Service Bus. Message sessions enable joint and ordered handling of unbounded sequences of related messages.
The auto-forwarding feature enables you to chain a queue or subscription to another queue or topic in the same namespace.
Service Bus supports a dead-letter queue (DLQ) to hold messages that cannot be delivered to any receiver, or messages that cannot be processed.
You can submit messages to a queue or topic for delayed processing; for example, to schedule a job to become available for processing by a system at a specific time.
When a queue or subscription client receives a message that it is willing to process, but due to special circumstances within the applicationfor such processing is not currently possible , the entity has the option to and can defer retrieval of the message to a later point. The message remains in the queue or subscription, but it is set aside.
Client-side batching enables a queue or topic client to delay sending a message for a specified period. If the client sends additional messages during this period, it transmits the messages in a single batch.
A transaction groups two or more operations into an execution scope. Service Bus supports grouping operations against a single messaging entity (queue, topic, subscription) within the scope of a transaction.
Filtering and actions
Subscribers can define which messages to receive from a topic. These messages are specified in the form of one or more named subscription rules. For each matching rule condition, the subscription produces a copy of the message, which may be differently annotated for each matching rule.
Auto-delete on idle
Auto-delete on idle enables you to configure an idle interval after which the queue is automatically deleted. The minimum duration for idle interval is 5 minutes.
If an error occurs that causes the client to doubt the outcome of a send operation, duplicate detection helps clear that doubt by enabling the sender to re-send the same message, and the queue or topic discards any duplicate copies.
One of the utility bill payments company wants to build an application, which will do the mobile recharge for end users. The application should support all the operators which are available in the market. The flow is, the user sends SMS in predefined format and application should parse the message and do the recharge accordingly.
Now, the foremost challenge is that the application unable to process a vast number of recharge requests it gets during the peak hours, so the company is looking for a solution where all the requests are parked when they come and processed one by one.
To meet the business requirement, use the queue mechanism where all the requests are added in the queue, and on the another side queue listener will process the message by reading from the queue.
Azure Storage Queues
What is Azure storage queue?
Azure Queue storage is a service for storing large numbers of messages. Messages can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message size can be up to 64 KB. A queue may contain messages up to the total capacity limit of a storage account.
Where to use storage queues?
A content management system application where files are uploaded by a number of users daily. The files need to be processed asynchronously by a background process which filters out the media files and sends them for transcoding.
When the users upload the file, it is pushed to queue, and the background process will process the files from the queue asynchronously.
When to use Storage Queue and Service bus Queue?
|Storage Queue||Service bus Queue|
|Your application should store over 80 GB of messages in a queue.||Your solution should be able to receive messages without having to poll the queue|
|Your application needs to track progress for processing a message inside of the queue. This is useful if the worker processing message crashes. A subsequent worker can then use that information to continue from where the prior worker left off.||Your solution needs the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.|
|You require server-side logs of all of the transactions executed against your queues.||Your solution must support automatic duplicate detection.|
|No role-based access supported||You deal with a requirement to provide a role-based access model to the queues, and different rights or permissions for senders and receivers.|
|No dead lettering in storage queue.||Dead lettering, supported only by Service Bus queues, can be useful for isolating messages that cannot be processed successfully by the receiving application|
|Maximum message size is 64Kb||Maximum message size 256 KB or 1 MB|
Type in a topic service or offering and then hit enter to search