Our client is a leading artificial intelligence company that offers innovative AI solutions to various industries, including aerospace, defense, energy, and finance. Their solutions are designed to help businesses improve efficiency, reduce costs, and enhance their overall performance.They are known for leveraging advanced machine learning and deep learning techniques to solve complex, industry-specific challenges.
Client’s Asset ETL Suite was composed of redundant coding strategies and ineffective code pipelines over Serverless Architecture. For the Onboarding of any new asset to the ETL suite, we had to code Its respective dependencies. In addition, there was no way to import any time-series data either from CSV or through sensor data. Ability to import and expose data through various data sources.
The Proposed Solution was to disintegrate the code, which involved separating the common ETL code from the Asset Systems code. This would enable the common ETL code to be applied to all the Asset Systems, while still allowing for differentiation between individual Asset Systems.
The ETL code was split and a common utility layer was created, which the lambda functions could inherit. This approach simplified the process of creating new ETL functions for different Asset Systems.
The team deployed the solution using the serverless framework and was successful in deploying both the pre-processing and post-processing stages of the ETL process.
To ensure efficient interaction with influxdb, a custom Python package in the form of a TSDS client was created. This client provided additional features to connect to Amazon Web Services (AWS) Time Stream, which is an important component for storing and managing large volumes of data. The custom Python package enabled other ETL services to interact with influxdb more easily and efficiently.
Overall, the solution provided an efficient and scalable way of disintegrating the code and ensuring that the common ETL code could be applied across all Asset Systems. The custom Python package provided an additional layer of functionality to ensure efficient interaction with AWS Time Stream and influxdb.
.png)
Tech stack used: Python, Django, Influxdb, AWS RDS, AWS Lambda, AWS S3, AWS IoT SiteWise, Serverless.
-Reduced asset onboarding time by 50% through reusable, modular ETL components.
-Enabled code reusability across multiple asset systems, minimizing redundancy and improving maintainability.
-Achieved seamless ingestion of time-series data from CSV files and real-time sensor sources.
-Improved ETL pipeline scalability and flexibility using a serverless architecture.
-Enhanced data processing performance through a custom Python TSDS client for InfluxDB and AWS Timestream.
-Delivered a fully serverless, cloud-native solution optimized for real-time data integration and analytics.
