{"id":2143524169,"date":"2022-08-09T00:00:00","date_gmt":"2022-08-09T00:00:00","guid":{"rendered":"https:\/\/newgensoft-v2.wordpoets.com\/resources\/blog\/2022\/08\/09\/adapting-spark-distributed-compute-framework\/"},"modified":"2026-06-03T10:38:31","modified_gmt":"2026-06-03T10:38:31","slug":"adapting-spark-distributed-compute-framework","status":"publish","type":"post","link":"https:\/\/newgensoft.com\/resources\/blog\/adapting-spark-distributed-compute-framework\/","title":{"rendered":"Adapting Spark for a Distributed Compute Framework"},"content":{"rendered":"\n

The Beginning of Our Data Science Platform Development Journey<\/strong><\/h2>\n\n\n\n

When we started developing the Data Science Platform to help data engineers and citizen and expert data scientists manage the end-to-end Data Science model lifecycle at the enterprise level, the key question was to choose the<\/strong> right compute framework<\/strong>. The fundamental need in AI\/ML model development<\/a> is to personalize the decision at the individual level instead of micro\/macro segment level. As such, AI\/ML models require large iterative compute processes at the time of model training and development. Therefore, the memory computation helps a lot in expediting the training process, leading to up to 100x improvement in turnaround time compared to classical Disk I\/O.<\/p>\n\n\n\n

<\/span>Embracing and Enhancing Spark<\/strong><\/h2>\n\n\n\n

We selected Open Source Spark for our compute engine and used their existing Spark ML base to build the platform. At that time, Spark had just released 1.x version. We analyzed the gap in the existing Spark version and started working to fill those gaps based on our own platform\u2019s requirements. Therefore, we had to make our own derived version of the Spark ML component as per our need, where we did not make any significant changes to the compute part. For seamless integration with different versions of Spark distribution (Cloud-based like Azure, AWS, etc. and non-cloud-based like Cloudera, MapR one), our team has made enhancements to the Livy module that provides integration with Spark Rest-based API.<\/p>\n\n\n\n

<\/span>A glimpse into 10 notable enhancements we have done in Spark for our AI\/ML platform<\/strong><\/h2>\n\n\n\n
    \n
  1. Deep Learning Algorithms:<\/strong>\u00a0Spark ML was found lacking in deep learning algorithms capability. We added some of the commonly used deep learning algorithms, such as CNN (Convolutional Neural Network) and RBM (Restricted Boltzmann Machine) which are crucial in solving\u00a0many\u00a0use cases\u00a0of deep\u00a0learning model approaches.<\/li>\n\n\n\n
  2. Use of SMOTE:<\/strong>\u00a0One of the major problems Data Scientists faces is the requirement to use SMOTE kind of algorithms to tackle data imbalances. This happens in use cases like lending where data is imbalanced. We have extended Spark ML with SMOTE through KNN implementation.<\/li>\n\n\n\n
  3. Mechanism for Solving Timeseries Algorithms:<\/strong>\u00a0Given that there was no support for the timeseries\u00a0algorithms on Spark, our team fixed the gap by putting ARIMA and ARIMAX on Spark. This has given our platform the capability to solve timeseries problems like sales forecast, price forecast etc.<\/li>\n\n\n\n
  4. Ability to Handle Missing Data Scenarios:<\/strong>\u00a0One of the major problems for solving any AI\/ML modeling<\/a> is handling the missing data. The missing data can be numeric like age or can be a string like address, city, or type of retail, etc. We have added many different scenarios of missing value data handling which helps data scientists to solve user cases properly.<\/li>\n\n\n\n
  5. Field-aware Factorization Machine (FFM):<\/strong>\u00a0There are many scenarios where datasets used for AI\/ML modeling are in the form of\u00a0the sparse matrix. For example, one of the use cases where you need to predict CTR (Click through rate), one needs to use factorization process to be able to use that kind of dataset. We have extended Spark ML with FFM to solve these kinds of use cases.<\/li>\n\n\n\n
  6. AutoML Functionality:<\/strong>\u00a0As platform users have grown, many citizen data scientists\u00a0started asking for AutoML support. Using AutoML, they can try different algorithms in one go without trying algorithms one-by-one with different parameters. It really helps to try different AI\/ML algos in one go and also saves the time to try different individual experiments.\u00a0We have added AutoML to our platform.<\/li>\n\n\n\n
  7. Model Evaluation Metrics:<\/strong>\u00a0We have added different model evaluation metrics which are not present in Spark ML into the various algorithms, such as regression, clustering, classification, etc. This helps the data science personnel evaluate the model better against multiple metrics at the time of auto retraining of the AI\/ML models.<\/li>\n\n\n\n
  8. Memory Consumption Optimization:<\/strong>\u00a0We optimize the memory consumption as the scale of data increases and compute changes corresponding to data values or operation.<\/li>\n\n\n\n
  9. Enhanced Data Presentation for Recommendation Scenarios:<\/strong>\u00a0Typically, algorithms like ALS provide the basic capability for recommendation. We have added advanced capability to check all the products identified for recommendation and built data presentation components to control the presentation basis segments of users.<\/li>\n\n\n\n
  10. Addition of Vector Disassembler:<\/strong>\u00a0We have added vector disassembler for many use-cases like SVD, PCA, One Hot Encoding, etc. It also helps in understanding each dimension of the vector properly through data dumps.<\/li>\n<\/ol>\n\n\n\n

    We understand that the product journey is constantly evolving. To stay ahead of the curve, we keep adding the relevant components in Spark with every release. Here are some of the product roadmap components that we plan to add to our existing release of Spark\/Spark ML in the AI Cloud platform<\/a>:<\/strong><\/p>\n\n\n\n