Yadong Xie / 谢亚东
ProjectsLabWritingSaved|GithubLinkedInX

Alibaba Cloud E-MapReduce

Staff Engineer, product experience · Alibaba Cloud · 2019 – 2023

Alibaba Cloud E-MapReduce screenshot

E-MapReduce is Alibaba Cloud's managed platform for the open-source big data stack: Hadoop, Hive, Spark, StarRocks, Doris, Presto, and Trino, running on ECS, on Kubernetes, or serverless. I led the web engineering for its product experience.

The hardest part was buying a cluster. E-MapReduce has the most complex purchase console on Alibaba Cloud: a cluster is defined by its deployment type, the software stack and its versions, the services to install, node groups with their instance types, disks and counts, networking, and metadata storage, and many of those choices constrain each other. The aim was that someone could choose a working configuration without reading a wiki. That meant checking prerequisites up front instead of failing on submit, showing what a choice would produce before it was made, and leaving no defaults on settings that cost money or can't be changed after the cluster is created.

The rest of the work was about everyday tasks: scaling a cluster while jobs are running, and seeing what a running cluster is doing. Many actions on an infrastructure console can't be undone, so we spent a lot of design effort on the moments right before a destructive action, and on showing system state clearly enough that people could act with confidence.

I wrote more about this in 10 Essential Details for Enhancing Cloud Computing User Experience.

E-MapReduce

←Realtime Compute for Apache FlinkNG-ZORRO→