IoT Experience: Building a Sensor Data Tube with Azure Services and Snowflake Cloud Data Warehouse

Collecting and utilizing data has become very common in the business world as well as in everyday life. Storing data on cloud services like Microsoft's Azure or Amazon's AWS has become very simple and cheap. For example, Microsoft offers Office365, together with data reporting tools like PowerBI and Excel. With these tools alone, an individual can build a relatively effective way of managing, editing and analyzing his/her own data, and it doesn't even cost much. The biggest benefit of cloud data warehouses is that you no longer need to worry about ending storage. This makes it possible for an individual to collect large amounts of data and use his/her own historical data. In addition, data is available almost anywhere if the person has access to the Internet. From these starting points, I started to build an IoT (Internet of Things) demo.

The idea of the demo was to build a data tube among Power BI, Snowflake, Azure services and sensors in Raspberry Pi (sensors measured the temperature, humidity and lightness of our office). The aim of this entity was to create a data tube that can store data, make tables or data models required, and finally report the results either by traditional methods or by visualizing data in real time. Such an entity may seem relatively large to someone, but as I mentioned earlier, anyone can create such a system if only interest is found.

It is best to start the project by getting to know the Raspberry Pi and the sensors (or whatever data is being collected). The good side of Raspberry Pi is that it has a lot of information and guidance on the Internet. When configuring Raspberry Pi, it is not required that a person need experience in coding. Because ready-made instructions for creating scripts can be found in the cloud. 

Next, get to know Azure and the services you want to use in the demo. All services in this demo can be enabled by a click of the mouse, so this intermediate step does not require any coding background. Full documentation of all Azure's operations are available, as well as demos, and you can follow them well. Creating connections between different devices is the most time consuming part in this demo, in order to get data to flow as desired. Because each device has its own configuration, logic and reasoning can be utilized in small scale. In addition, there are calculators for all the services you can use to see how much your device would cost. 

Main Azure Services used in Demo:

  • IoT hub
  • Storage account / Blob storage
  • Logic app
  • Stream analytics 

Iot Demo flowchat

When Azure and Raspberry Pi are in order, data transfer can start for preservation. The intention is that data can also be modeled and analyzed. For this, the data must be available for to be queried to get the desired information, for example, in table or view formats. At this stage of the project, Snowflake, a modern cloud repository, was utilized. So by using Snowflake, you can modify raw data (JSON, AVRO, etc.) and create tables and views that are stored in your database. This step requires a bit of understanding of data formats and queries (SQL), but reading Snowflake documentation and selectively learning should not be too challenging. Personally, I managed to learn the basic logic in a day. The integration of Snowflake into the Azure data warehouse is also very easy. To do this, only one access key is required, after which you can open the connection. In addition, the Snowflake interface is very clear and well structured, which helps you to learn the software.

After receiving the data from Azure and Snowflake you are ready to improve or stream your data in real time. By importing historical data from Snowflake to Power BI or Excel and you can analyze the data as soon as you are ready to create the desired tables or views in Snowflake. Using the Azure stream analytical is also very easy. Stream analytics is placed for unnecessary processing, after that you can get information about the model or graphics in Power BI. 

This IoT demo was, in my opinion, a very instructive and interesting way to focus on a similar matter. Through this type of demo, one can learn a lot of basics, which are required to handle “big data” and, in general, what it is possible to do today. It has helped me greatly to learn the logic of Azure and how it works. In other words, making such a demo is a good way to understand the platforms of different service providers. In addition, once you get the first demo, it is very easy to start developing, for example, adding more automation, making different types of alarms, or improving your queries.

Iot demo