Software Engineering Intern | Machine Learning Server
Microsoft Artificial Intelligence & Research
In Summer 2017, I worked as a Software Engineering Intern in the Machine Learning Server Team within Microsoft’s AI & Research division, for 12 weeks. My project essentially tried to answer the question, “How can we help our users optimize their Azure HDInsight Hadoop cluster resources?” and I did this by developing the first version of the Spark Compute Context Configuration Suggestion Tool. This tool loads preconfigured scripts and predefined data into a customer Hadoop cluster, runs a batch of iterations to benchmark the cluster and then suggests optimal or near-optimal Spark CPU & memory configuration for future task deployment by the user. To make the tool smarter, I incorporated a probabilistic technique, for approximating the global optimum of the total-runtime function, called Simulated Annealing. This helped my tool find the optimal/ near-optimal Spark configurations that provided the best runtime for task deployment & processing, with greater accuracy.
(HDInsight Cluster Usage During Batch Iterations)
This internship was a great learning experience for me as I got to write shell scripts and program in Python, R and HTML while exploring the resource management and job scheduling frameworks used by our team as components and utilities on HDInsight Clusters, like YARN and platforms like Ambari which help us manage, secure and monitor our Hadoop clusters.
The internship ended on a memorable note, with Microsoft’s Intern Signature Event which featured performances by The Chainsmokers and Daya at the Boeing Factory and also Xbox One S as a gift for every intern. I am really looking forward to return to my team next summer!
