Data processing framework is the core element of Big Data. It provided good abstraction for computing resource and logic. In this talk, I will use google mapreduce (written in python) to introduce some key components such as message queue, pipeline, object collection, fault tolerance and task flow which used in common data processing frameworks. Then use a micro framework written in django to demo how data processing works.
GliaCloud founder and coorganizer of GCPUG. Gliacloud is a startup focus on AI and data analysis.
I like coding, ramen. and skiing. Most of time, I involve in python coding in Google Cloud Platform for big data processing and cloud architect designing. It is my pleasure to join pycon with lots of passion community members.