Apache Hadoop YARN: Moving Beyond MapReduce and Batch Processing with Apache Hadoop 2 -

. .

Auteur(s) : Collectif

Editeur : Addison Wesley Pub Co Inc

Langue : Anglais

Parution : 01/03/2014

Format : Moyen, de 350g à 1kg

Nombre de pages : 304

Expédition : 530

Dimensions : 22.8 x 17.7 x 2.2

ISBN : 0321934504

Résumé :

In Apache Hadoop YARN, key YARN developer Arun Murthy shows how to get existing code to run on Apache Hadoop 2, and develop new applications that take absolutely full advantage of Hadoop clusters. Drawing on insights from the entire Apache Hadoop 2 team, Murthy and Dr. Douglas Eadline review Apache Hadoop YARN's goals, design, architecture, and components, guide the reader thrugh migrating existing MapReduce applications, identify the functional requirements for each element of an Apache Hadoop 2 application, walk the reader through a sample appliation project, and offer multiple examples and case studies drawn from their cutting-edge experience.

Biographie:
Arun Murthy (California) has contributed to Apache Hadoop full-time since the inception of the project in early 2006. He is a long-term Hadoop Committer and a member of the Apache Hadoop Project Management Committee. Previously, he was the architect and lead of the Yahoo Hadoop Map-Reduce development team and was ultimately responsible, technically, for providing Hadoop Map-Reduce as a service for all of Yahoo - currently running on nearly 50,000 machines! Arun is the Founder and Architect of the Hortonworks Inc., a software company that is helping to accelerate the development and adoption of Apache Hadoop. Hortonworks was formed by the key architects and core Hadoop committers from the Yahoo! Hadoop software engineering team in June 2011 in order to accelerate the development and adoption of Apache Hadoop. Funded by Yahoo! and Benchmark Capital, one of the preeminent technology investors, their goal is to ensure that Apache Hadoop becomes the standard platform for storing, processing, managing and analyzing big data. He lives in Silicon Valley in California. Douglas Eadline (Pennsylvania), PhD, began his career as a practitioner and a chronicler of the Linux Cluster HPC revolution and now documents big data analytics. Starting with the first Beowulf How To document, Dr. Eadline has written hundreds of articles, white papers, and instructional documents covering virtually all aspects of HPC computing. Prior to starting and editing the popular ClusterMonkey.net web site in 2005, he served as Editorinchief for ClusterWorld Magazine, and was Senior HPC Editor for Linux Magazine. Currently, he is a consultant to the HPC industry and writes a monthly column in HPC Admin Magazine. Both clients and readers have recognized Dr. Eadline's ability to present a technological value proposition in a clear and accurate style. He has practical hands on experience in many aspects of HPC including, hardware and software design, benchmarking, storage, GPU, cloud, and parallel computing.

Sommaire:
Foreword by Raymie Stata xiii Foreword by Paul Dix xv Preface xvii Acknowledgments xxi About the Authors xxv Chapter 1: Apache Hadoop YARN: A Brief History and Rationale 1 Introduction 1 Apache Hadoop 2 Phase 0: The Era of Ad Hoc Clusters 3 Phase 1: Hadoop on Demand 3 Phase 2: Dawn of the Shared Compute Clusters 9 Phase 3: Emergence of YARN 18 Conclusion 20 Chapter 2: Apache Hadoop YARN Install Quick Start 21 Getting Started 22 Steps to Configure a Single-Node YARN Cluster 22 Run Sample MapReduce Examples 30 Wrap-up 31 Chapter 3: Apache Hadoop YARN Core Concepts 33 Beyond MapReduce 33 Apache Hadoop MapReduce 35 Apache Hadoop YARN 38 YARN Components 39 Wrap-up 42 Chapter 4: Functional Overview of YARN Components 43 Architecture Overview 43 ResourceManager 45 YARN Scheduling Components 46 Containers 49 NodeManager 49 ApplicationMaster 50 YARN Resource Model 50 Managing Application Dependencies 53 Wrap-up 57 Chapter 5: Installing Apache Hadoop YARN 59 The Basics 59 System Preparation 60 Script-based Installation of Hadoop 2 62 Script-based Uninstall 68 Configuration File Processing 68 Configuration File Settings 68 Start-up Scripts 71 Installing Hadoop with Apache Ambari 71 Wrap-up 84 Chapter 6: Apache Hadoop YARN Administration 85 Script-based Configuration 85 Monitoring Cluster Health: Nagios 90 Real-time Monitoring: Ganglia 97 Administration with Ambari 99 JVM Analysis 103 Basic YARN Administration 106 Wrap-up 114 Chapter 7: Apache Hadoop YARN Architecture Guide 115 Overview 115 ResourceManager 117 NodeManager 127 ApplicationMaster 138 YARN Containers 148 Summary for Application-writers 150 Wrap-up 151 Chapter 8: Capacity Scheduler in YARN 153 Introduction to the Capacity Scheduler 153 Capacity Scheduler Configuration 155 Queues 156 Hierarchical Queues 156 Queue Access Control 159 Capacity Management with Queues 160 User Limits 163 Reservations 166 State of the Queues 167 Limits on Applications 168 User Interface 169 Wrap-up 169 Chapter 9: MapReduce with Apache Hadoop YARN 171 Running Hadoop YARN MapReduce Examples 171 MapReduce Compatibility 181 The MapReduce ApplicationMaster 181 Calculating the Capacity of a Node 182 Changes to the Shuffle Service 184 Running Existing Hadoop Version 1 Applications 184 Running MapReduce Version 1 Existing Code 187 Advanced Features 188 Wrap-up 190 Chapter 10: Apache Hadoop YARN Application Example 191 The YARN Client 191 The ApplicationMaster 208 Wrap-up 226 Chapter 11: Using Apache Hadoop YARN Distributed-Shell 227 Using the YARN Distributed-Shell 227 Internals of the Distributed-Shell 232 Wrap-up 240 Chapter 12: Apache Hadoop YARN Frameworks 241 Distributed-Shell 241 Hadoop MapReduce 241 Apache Tez 242 Apache Giraph 242 Hoya: HBase on YARN 243 Dryad on YARN 243 Apache Spark 244 Apache Storm 244 REEF: Retainable Evaluator Execution Framework 245 Hamster: Hadoop and MPI on the Same Cluster 245 Wrap-up 245 Appendix A: Supplemental Content and Code Downloads 247 Available Downloads 247 Appendix B: YARN Installation Scripts 249 install-hadoop2.sh 249 uninstall-hadoop2.sh 256 hadoop-xml-conf.sh 258 Appendix C: YARN Administration Scripts 263 configure-hadoop2.sh 263 Appendix D: Nagios Modules 269 check_resource_manager.sh 269 check_data_node.sh 271 check_resource_manager_old_space_pct.sh 272 Appendix E: Resources and Additional Information 277 Appendix F: HDFS Quick Reference 279 Quick Command Reference 279 Index 287