湾区同学技术沙龙
(Bay Area) The Evolution of Big Data APIs in Spark (Reynold Xin)
23 July 2016
1:30PM ~ 4:00PM, 07/23//2016, Saturday
Registration
- Registration link: tech-meetup.com/events/07-23-2016/
- Event link: /the-evolution-of-big-data-apis-in-spark
Event Info
- Time: 1:30PM ~ 4:00PM, 07/23//2016, Saturday
- Location: 1069 East Meadow Circle, Sofia U Auditorium, Palo Alto, CA, 94303
Agenda:
- 1:30pm - 2:00pm: Registration and social time
- 2:00pm - 3:30pm: Talk and QA
- 3:30pm - 4:00pm: offline networking
Abstract:
Apache Spark has aimed from the beginning to provide a high-level API for diverse big data workloads. In the past two years, Spark has seen some of the largest API additions since it began in order to meet this goal. In particular, while Spark started out with a functional API based on collections of Java/Python objects, a new series of "structured APIs" including DataFrames, Spark SQL and Structured Streaming are introducing a much more declarative layer that is simultaneously more optimizable. Underneath these APIs, the engine can understand the structure of the data and of user queries, and applies rich optimizations to both computation and storage that are typically not common big data processing frameworks, such as relational optimizations, columnar storage, and code generation. Moreover, these optimizations automatically apply across Spark libraries, including third-party data sources and packages. I will discuss the rationale and design of these APIs, as well as recent additions in this area, such as the incremental Structured Streaming API that provides a simple way to reason about out-of-order data, connections with external systems, and failures.
Speaker’ bio:
Reynold Xin is a co-founder and Chief Architect at Databricks. He is also an Apache Spark PMC member and release manager for Spark's 2.0 release. Prior to Databricks, he was pursuing PhD research at the UC Berkeley AMPLab, where he worked on large-scale data processing.
- Weibo: hashjoin
- Twitter: rxin
Language:
Chinese
主办
协办
- 南京大学硅谷校友会
- 硅谷清华联网
- 索菲亚大学(Sofia University, Palo Alto, CA)
- 瀚海硅谷科技园
- 中国科技大学校友会创业俱乐部
- 浙江大学校友会海纳创新创业俱乐部
- 北京大学北加州校友会
- 北京航空航天大学硅谷校友会
- 武汉大学北加州校友会
- 东南大学硅谷校友会
- 吉林大学硅谷校友会
- 复旦大学北加州校友会
- 北加州华中科技大学校友会
- 华人事业互助会
Related articles
- (Bay Area) Snowflake / Databricks / OceanBase
- (Bay Area) 云端数据中台:数据编排与平台运维
- (Bay Area) Google Doc 是如何炼成的 - 深入浅出协同编辑/Deep Dive Collaborative Editing
- (Bay Area) An introduction of Analytics Zoo and how to use it at Uber
- (Bay Area) Tensorflow.JS: Bringing Machine Learning To The Web And Beyond
- (Bay Area) Weakly Supervised Natural Language Understanding / 基于弱监督学习的自然语言理解 By Mosaix.ai
- (Bay Area) Data Extraction Revolution in Bloomberg, From Human Typing To Deep Learning Excerpting
- (Bay Area) Next-Generation AI Powered Operation System
- (Bay Area) Power Blockchain with Hardware Innovations
- (Bay Area) 区块链产业现状及技术发展(阿里巴巴技术日)
- (Bay Area) Anatomizing Blockchain through Many Views(区块链折叠)
- (Bay Area) Deep Dive of Alluxio and Google gVisor
- (Bay Area) 技术创造新商业:阿里巴巴搜索推荐&计算平台事业部硅谷开放日
- (Bay Area) Google Translate助力自然语言理解
- (Bay Area) Alibaba Tech Open Day – AI, Cloud, Infrastructure and More
- (Bay Area) 通向区块链3.0的未来之路
- (Bay Area) Alibaba New Retail / Hema Tech Day (盒马生鲜技术日)
- (Bay Area) exGoogle Leaders, leap.ai co-founders share their career stories & insights (Richard Liu, Yunkai Zhou)
- (Bay Area) Augmented Intelligence to Improve Health Care Consumer Experience
- (Bay Area) GrowingIO 湾区技术同学见面会
- (Bay Area) Alibaba Technology Forum, Stanford University
- (Bay Area) How Pinterest Perfected New User Onboarding
- (Bay Area) Tencent Tech Day - Silicon Valley
- (Bay Area) Deep dive of DeepMap (Wei Luo)
- (Bay Area) Apache Kafka: The Rise of Real-time
- (Bay Area) 苏宁机器学习平台及Buddy AI人工智能自动客服系统技术分享
- (Bay Area) JD.com Tech Day - Leverage Technology to empower business intelligence
- (Shanghai) 采用超低功耗AI技术的小MU机器人的实现与应用
- (Bay Area) Transwarp(星环科技) && DistributedLog
- (Bay Area) AI in Service robotics and Mini Robot
- (Shanghai) Google SRE如何管理数据中心
- (Bay Area) 如何用1/6000的训练数据击败深度学习——文字识别实验讨论
- (Shanghai) Twitter Heron Streaming at Scale
- (Bay Area) AI大牛谈深度学习最新进展
- (Bay Area) 新一代创新搜索技术架构讨论专场
- (Bay Area) CAINIAO Technology Forum, Silicon Valley
- (Bay Area) How to build a NewSQL database? (Qi Liu)
- (Bay Area) TensorFlow: A Large-Scale Machine Learning System (Zhifeng Chen)
- (Bay Area) Ant Financial Tech Forum (2016蚂蚁金服技术湾区论坛)
- (Bay Area) Espresso: LinkedIn’s Distributed Database (Yun Sun)
- (Bay Area) Virtual Reality & Augmented Reality (Guodong Rong)
- (Bay Area) Etcd: A key-value store Open Source for Data consistency, Data persistency, Data synchronization in Distributed system (Xiang Li)
- (Bay Area) Introduction To OpenStack (Weidong Shao & Xin Wu)
- (Bay Area) A Journey of AI: from Silicon Valley to Beijing, from Big Name to Startup (Kai Yu)
- (Bay Area) Borg: Large-scale Cluster Management at Google (Xiao Zhang)
- (Bay Area) CoreOS rkt, a Container Runtime (Yifan Gu)
- (Bay Area) Spark MLlib: Past, Present and Future (Xiangrui Meng)
- (Bay Area) Cassandra: an open source distributed database (Charles Cao)
- (Bay Area) Tachyon: an open source memory-centric distributed storage system (Bin Fan / Shaoshan Liu / Haoyuan Li)
- (Bay Area) Apache Samza: a distributed stream processing framework (Yi Pan)
- (Bay Area) 大数据时代的金融服务创新 (Li Cheng)
- (Bay Area) 大数据人工智能 (Kai Yu)
- (Bay Area) Photon: Fault-tolerant and scalable joining of continuous data streams (Tianhao Qiu)
- (Bay Area) Large-scale data science and engineering with Spark (Reynold Xin)
- (Bay Area) Building a real time data platform with Apache Kafka (Jun Rao)
- (Bay Area) Kubernetes: Google’s secret weapon for Cloud computing (Dawn Chen)
- (Bay Area) Tachyon: A Reliable Memory-Centric Distributed Storage System