Automating data migration through a hybrid cloud strategy and loo-footprint clusters.
We increased the readily available data of a leading MedTech company by 50% and implemented cutting-edge monitoring and analytics tools, improving the system’s stability and allowing faster onboarding of clients and significant scaling out.
As a part of the overall scale-out operation the company had been preparing for due to a foreseeable significant increase in traffic and clients, it had been struggling with an error-prone and manual process of data ingestion and acquisition pipeline, which in turn, led to high operational cost and an increased risk to its sensitive data. Moreover, the combination of their lack of monitoring that led to poor detection and resolution times of errors across critical core services, along with their inefficient usage of their limited compute resources have contributed to its severe instability and load issues that have had a direct connection to their product's accuracy and availability to the customers.
ProdOps entered the picture with an innovative design of an automated data migration process, using a hybrid cloud strategy and low-footprint clusters. ProdOps managed to reduce the company’s operational costs and to increase by 50% its readily available data. Furthermore, ProdOps implemented the usage of cutting-edge monitoring and visualization tools, resulting in a significant reduction of fail rate and recovery time.
ProdOps solutions allowed the company to scale out dramatically and achieve faster time to market.
- Manual and error-prone process of data migration from distant clients’ on-premise servers to the company’s servers for de-identification of sensitive data-information of patients, including long travel of teams and extensive operational cost.
- Constant system fails due to low stability and complete dependence on data lead for data management.
- Unawareness of failures, due to lack of monitoring tools over the data team’s data ingestion pipeline.
- Manual onboarding of clients, done only by one technological executive.
- Design and Implementation of hybrid cloud strategy, and setup of low-footprint and lean Kubernetes clusters on the company’s data centers to maximize the hardware resources and support optimal streaming performances, and to support the creation of a steady stream of data from customers’ data centers to the company’s servers.
- Creation of a visual monitoring tool for quick identification of failures, based on Prometheus time-series DB and Grafana visualization and analytics dashboard.
- Design and implementation of a real-time monitoring solution of On-premise servers and additional monitoring of the process as a whole.
- The transition from Redis cluster to monitored and scalable Kafka cluster, for easier onboarding of users and enhanced performance with the added ability to replay messages in a queue, permitting centralized and secure management of data.
- Significant improvement of resources usage, and reduction of reserve usage from 80% to 20%, while maintaining the same actual usage rate.
- 50% increase in the volume of readily available data, thus improving the medical Vision tool’s accuracy.
- Faster data processing (from low speed to top speed) and time to market, allowing significant and quick scaling out.
- Improvement of the system’s stability, including constant monitoring of On-premise servers, leading to reduced fail rate and recovery time.
- Cost reduction and faster onboarding process for new clients, done in a couple of hours instead of a couple of days, and that can be made by additional team members.