Will Microsoft Azure meet your real-time analytic needs?
Sara Handel, one of our Melbourne consultants, recently undertook a Proof of Concept for real-time streaming analytics on network sensor data for a utility client using Microsoft Azure Streaming. In this blog Sara shares some of her key takeaways from this project.
Microsoft Azure Streaming is straightforward and you can quickly setup a streaming solution using an Event Hub, Stream Analytics, and Power BI. It is worth noting that with the different services still maturing, investing some initial time in a proof of concept to optimise the configuration before embarking on a full scale solution will be beneficial.
Below are my experiences from a recent proof of concept for real-time streaming analytics on network sensor data for a utility client delivered over two, two-week sprints. The rapidly prototyped solution used the following Microsoft Azure services:
• Event Hubs to get data from the network devices,
• Stream Analytics to analyse the data from the Event Hub in real time,
• Power BI to display real-time data from Stream Analytics and
• Azure SQL Database for data collection.
Each service works well on its own, but challenges come in when you combine the services for a full solution. I will cover the key challenges I encountered below to help you best leverage the services.
I quickly and easily setup an Event Hub. It took a bit of doing to get the device data connected to the Event Hub, but luckily there are a lot of code samples here.
I, again, quickly and easily, added on the Stream Analytics service. However, this is where things got a bit trickier. Stream Analytics requires an input, output and a query that gets data from the input and sends it to the output. Inputs can only be reference data stored in Azure blob storage or streaming data from an Event Hub (or IoT Hub*). This means if you want to add any non-real-time data to your real-time stream, such as device reference data, you must load it into blob storage and create a way to maintain that data.
For the Stream Analytics outputs, I used Power BI for real-time streaming, and Azure SQL database for data collection. The setup for both is straightforward, but troubleshooting Stream Analytics can be difficult. Azure provides a test tool in the query editor. However, one significant limit of the test tool is that it does not behave the same way as it would on streaming data since the test is on a static set of data. This makes it challenging to check query logic that includes windowing functions for time series aggregation. Having the same data streaming to a SQL database and Power BI helped sanity check things, but still did not fully enable the type of query troubleshooting you may be used to. For query logic help, I turned to the Stream Analytics query language reference and examples as well as the forums which I found useful for finding complex examples. (While you are out exploring technology, another streaming option available in Microsoft Azure as part of HDInsight is Apache Storm – check out the comparison to Stream Analytics.)
The most challenging piece of the puzzle turned out to be Power BI. Once the data streamed successfully from the devices to the Event Hub through Stream Analytics, I needed to display it in Power BI for the customer to get business value. The data from Stream Analytics can only be displayed using streaming tiles in the online version of Power BI. This means that many features available in Power BI desktop cannot be used. Also, only a single dataset is supported per tile which means all the data needs to be as you want it coming out of Stream Analytics. Microsoft is actively working on enhancing these features so these limitations may not impact future projects, but be aware that active changes also means that features can disappear overnight. If you decide to use this technology be sure to stay active in the Power BI community (search for ASA posts on the Power BI blog) and contact ASAtoPBIStreaming@microsoft.com if strange things happen.
Again, it is straightforward to quickly setup a streaming solution using an Event Hub, Stream Analytics, and Power BI which makes the stack ideal for rapid prototyping. However, as the different services are still maturing, it is worth investing some initial time in a proof of concept before embarking on a full scale solution.
To find out more about how the Microsoft Azure streaming solutions can benefit your organisation, please contact us.
*It is worth noting that Microsoft is pushing people toward their IoT Hubs offering which support the same capabilities of Event Hubs with the additional ability to manage IoT devices. Click here for a comparison between IoT and Event Hubs.