Graph database solution for real-time sales and chemical composition analytics at scale.

• ChallengeUnstructured production data with complex parent-child database relations making data querying resource-intensive
• SolutionAnalytical solution for real-time sales, product analytics with Graph database to maintain complex relation, built on Azure
• Technologies and toolsAzure Databricks, CosmosDB, ADLS/Data lake service

Client

The client is a UK-based chemical manufacturer. The company was looking for a more efficient way to analyze the chemical composition of mixtures and products as well as their test results. The sales data of a particular chemical product also remained untapped requiring a robust analytical solution.

The client wanted to focus on a graph-based querying data solution that retrieves streaming data on sales and product chemical composition. The insights should then be visualized for recording and scoring information as well as becoming a reliable source of data-driven critical business decisions.

InData Labs was chosen by a client as a tech partner with seasoned Big data analytics solutions and Machine Learning Consulting services as well as experience in deploying AI in chemical industry.

Challenge: unstructured production data with complex parent-child database relations making data querying resource-intensive

The main issue was that the client’s existing data storage approach revolved around unstructured production data that came in various stages and forms. Besides, the data had complex parent-child relations that required to be maintained for querying and follow up analysis.

A new compiling process was required that would include newly created metadata and database relations to request information using a graph database without utilizing much computing power as well within a time constraint of 5 seconds.

Solution: analytical solution for real-time sales, product analytics with Graph database to maintain complex relation, built on Azure

Our Big data engineering team performed a thorough analysis of the client’s requirements and bottlenecks. The following scope of work was performed to transform the data:

• Imported, cleaned, and transformed data from CSV files using Azure Databricks.
• Defined relations and transformed data in the form of graphs in Azure Databricks Pushed it in Cosmos DB and queried data and graph using Azure Cosmos graph.
• Established interoperability with a graph-based query on Azure Cosmos DB.
• Set up real-time streaming in Power BI for recording and scoring.

The whole transformation process was carried out by three Big data specialists, including a data architect, data engineer, and BI expert. The business intelligence project was completed in 6 months.

Here is the solution architecture that demonstrates this case study of the chemical industry and data lifecycle:

Result: harnessing chemical and sales data with BI & reporting solution

As a result of our collaboration, we have transformed the client’s data stack and delivered a tailored advanced data analytics solution that drastically improved decision-making, real-time analytics, and insight extraction. The company now has a comprehensive data-driven overview of chemical composition analysis as well as historical customer data that is visualized within a scalable platform for self-service and enterprise business intelligence.