Details

Professional Hadoop Solutions


Professional Hadoop Solutions


1. Aufl.

von: Boris Lublinsky, Kevin T. Smith, Alexey Yakubovich

38,99 €

Verlag: Wiley
Format: PDF
Veröffentl.: 30.08.2013
ISBN/EAN: 9781118612545
Sprache: englisch
Anzahl Seiten: 504

DRM-geschütztes eBook, Sie benötigen z.B. Adobe Digital Editions und eine Adobe ID zum Lesen.

Beschreibungen

<p><b>The go-to guidebook for deploying Big Data solutions with Hadoop</b></p> <p>Today's enterprise architects need to understand how the Hadoop frameworks and APIs fit together, and how they can be integrated to deliver real-world solutions. This book is a practical, detailed guide to building and implementing those solutions, with code-level instruction in the popular Wrox tradition. It covers storing data with HDFS and Hbase, processing data with MapReduce, and automating data processing with Oozie. Hadoop security, running Hadoop with Amazon Web Services, best practices, and automating Hadoop processes in real time are also covered in depth.</p> <p>With in-depth code examples in Java and XML and the latest on recent additions to the Hadoop ecosystem, this complete resource also covers the use of APIs, exposing their inner workings and allowing architects and developers to better leverage and customize them.</p> <ul> <li>The ultimate guide for developers, designers, and architects who need to build and deploy Hadoop applications</li> <li>Covers storing and processing data with various technologies, automating data processing, Hadoop security, and delivering real-time solutions</li> <li>Includes detailed, real-world examples and code-level guidelines</li> <li>Explains when, why, and how to use these tools effectively</li> <li>Written by a team of Hadoop experts in the programmer-to-programmer Wrox style</li> </ul> <p><i>Professional Hadoop Solutions</i> is the reference enterprise architects and developers need to maximize the power of Hadoop.</p>
Introduction xvii<br /> <br /> <b>Chapter 1: Big Data and the Hadoop Ecosystem 1</b><br /> <br /> Big Data Meets Hadoop 2<br /> <br /> Hadoop: Meeting the Big Data Challenge 3<br /> <br /> Data Science in the Business World 5<br /> <br /> The Hadoop Ecosystem 7<br /> <br /> Hadoop Core Components 7<br /> <br /> Hadoop Distributions 10<br /> <br /> Developing Enterprise Applications with Hadoop 12<br /> <br /> Summary 16<br /> <br /> <b>Chapter 2: Storing Data in Hadoop 19</b><br /> <br /> HDFS 19<br /> <br /> HDFS Architecture 20<br /> <br /> Using HDFS Files 24<br /> <br /> Hadoop-Specific File Types 26<br /> <br /> HDFS Federation and High Availability 32<br /> <br /> HBase 34<br /> <br /> HBase Architecture 34<br /> <br /> HBase Schema Design 40<br /> <br /> Programming for HBase 42<br /> <br /> New HBase Features 50<br /> <br /> Combining HDFS and HBase for Effective Data Storage 53<br /> <br /> Using Apache Avro 53<br /> <br /> Managing Metadata with HCatalog 58<br /> <br /> Choosing an Appropriate Hadoop Data Organization for Your Applications 60<br /> <br /> Summary 62<br /> <br /> <b>Chapter 3: Processing Your Data with MapReduce 63</b><br /> <br /> Getting to Know MapReduce 63<br /> <br /> MapReduce Execution Pipeline 65<br /> <br /> Runtime Coordination and Task Management in MapReduce 68<br /> <br /> Your First MapReduce Application 70<br /> <br /> Building and Executing MapReduce Programs 74<br /> <br /> Designing MapReduce Implementations 78<br /> <br /> Using MapReduce as a Framework for Parallel Processing 79<br /> <br /> Simple Data Processing with MapReduce 81<br /> <br /> Building Joins with MapReduce 82<br /> <br /> Building Iterative MapReduce Applications 88<br /> <br /> To MapReduce or Not to MapReduce? 94<br /> <br /> Common MapReduce Design Gotchas 95<br /> <br /> Summary 96<br /> <br /> <b>Chapter 4: Customizing MapReduce Execution 97</b><br /> <br /> Controlling MapReduce Execution with InputFormat 98<br /> <br /> Implementing InputFormat for Compute-Intensive Applications 100<br /> <br /> Implementing InputFormat to Control the Number of Maps 106<br /> <br /> Implementing InputFormat for Multiple HBase Tables 112<br /> <br /> Reading Data Your Way with Custom RecordReaders 116<br /> <br /> Implementing a Queue-Based RecordReader 116<br /> <br /> Implementing RecordReader for XML Data 119<br /> <br /> Organizing Output Data with Custom Output Formats 123<br /> <br /> Implementing OutputFormat for Splitting MapReduce<br /> <br /> Job’s Output into Multiple Directories 124<br /> <br /> Writing Data Your Way with Custom RecordWriters 133<br /> <br /> Implementing a RecordWriter to Produce Outputtar Files 133<br /> <br /> Optimizing Your MapReduce Execution with a Combiner 135<br /> <br /> Controlling Reducer Execution with Partitioners 139<br /> <br /> Implementing a Custom Partitioner for One-to-Many Joins 140<br /> <br /> Using Non-Java Code with Hadoop 143<br /> <br /> Pipes 143<br /> <br /> Hadoop Streaming 143<br /> <br /> Using JNI 144<br /> <br /> Summary 146<br /> <br /> <b>Chapter 5: Building Reliable MapReduce Apps 147</b><br /> <br /> Unit Testing MapReduce Applications 147<br /> <br /> Testing Mappers 150<br /> <br /> Testing Reducers 151<br /> <br /> Integration Testing 152<br /> <br /> Local Application Testing with Eclipse 154<br /> <br /> Using Logging for Hadoop Testing 156<br /> <br /> Processing Applications Logs 160<br /> <br /> Reporting Metrics with Job Counters 162<br /> <br /> Defensive Programming in MapReduce 165<br /> <br /> Summary 166<br /> <br /> <b>Chapter 6: Automating Data Processing with Oozie 167</b><br /> <br /> Getting to Know Oozie 168<br /> <br /> Oozie Workflow 170<br /> <br /> Executing Asynchronous Activities in Oozie Workflow 173<br /> <br /> Oozie Recovery Capabilities 179<br /> <br /> Oozie Workflow Job Life Cycle 180<br /> <br /> Oozie Coordinator 181<br /> <br /> Oozie Bundle 187<br /> <br /> Oozie Parameterization with Expression Language 191<br /> <br /> Workflow Functions 192<br /> <br /> Coordinator Functions 192<br /> <br /> Bundle Functions 193<br /> <br /> Other EL Functions 193<br /> <br /> Oozie Job Execution Model 193<br /> <br /> Accessing Oozie 197<br /> <br /> Oozie SLA 199<br /> <br /> Summary 203<br /> <br /> <b>Chapter 7: Using Oozie 205</b><br /> <br /> Validating Information about Places Using Probes 206<br /> <br /> Designing Place Validation Based on Probes 207<br /> <br /> Designing Oozie Workflows 208<br /> <br /> Implementing Oozie Workflow Applications 211<br /> <br /> Implementing the Data Preparation Workflow 212<br /> <br /> Implementing Attendance Index and Cluster Strands<br /> <br /> Workflows 220<br /> <br /> Implementing Workflow Activities 222<br /> <br /> Populating the Execution Context from a java Action 223<br /> <br /> Using MapReduce Jobs in Oozie Workflows 223<br /> <br /> Implementing Oozie Coordinator Applications 226<br /> <br /> Implementing Oozie Bundle Applications 231<br /> <br /> Deploying, Testing, and Executing Oozie Applications 232<br /> <br /> Deploying Oozie Applications 232<br /> <br /> Using the Oozie CLI for Execution of an Oozie Application 234<br /> <br /> Passing Arguments to Oozie Jobs 237<br /> <br /> Using the Oozie Console to Get Information about Oozie<br /> <br /> Applications 240<br /> <br /> Getting to Know the Oozie Console Screens 240<br /> <br /> Getting Information about a Coordinator Job 245<br /> <br /> Summary 247<br /> <br /> <b>Chapter 8: Advanced Oozie FEATURES 249</b><br /> <br /> Building Custom Oozie Workflow Actions 250<br /> <br /> Implementing a Custom Oozie Workflow Action 251<br /> <br /> Deploying Oozie Custom Workflow Actions 255<br /> <br /> Adding Dynamic Execution to Oozie Workflows 257<br /> <br /> Overall Implementation Approach 257<br /> <br /> A Machine Learning Model, Parameters, and Algorithm 261<br /> <br /> Defining a Workflow for an Iterative Process 262<br /> <br /> Dynamic Workflow Generation 265<br /> <br /> Using the Oozie Java API 268<br /> <br /> Using Uber Jars with Oozie Applications 272<br /> <br /> Data Ingestion Conveyer 276<br /> <br /> Summary 283<br /> <br /> <b>Chapter 9: Real-Time Hadoop 285</b><br /> <br /> Real-Time Applications in the Real World 286<br /> <br /> Using HBase for Implementing Real-Time Applications 287<br /> <br /> Using HBase as a Picture Management System 289<br /> <br /> Using HBase as a Lucene Back End 296<br /> <br /> Using Specialized Real-Time Hadoop Query Systems 317<br /> <br /> Apache Drill 319<br /> <br /> Impala 320<br /> <br /> Comparing Real-Time Queries to MapReduce 323<br /> <br /> Using Hadoop-Based Event-Processing Systems 323<br /> <br /> HFlame 324<br /> <br /> Storm 326<br /> <br /> Comparing Event Processing to MapReduce 329<br /> <br /> Summary 330<br /> <br /> <b>Chapter 10: Hadoop Security 331</b><br /> <br /> A Brief History: Understanding Hadoop Security Challenges 333<br /> <br /> Authentication 334<br /> <br /> Kerberos Authentication 334<br /> <br /> Delegated Security Credentials 344<br /> <br /> Authorization 350<br /> <br /> HDFS File Permissions 350<br /> <br /> Service-Level Authorization 354<br /> <br /> Job Authorization 356<br /> <br /> Oozie Authentication and Authorization 356<br /> <br /> Network Encryption 358<br /> <br /> Security Enhancements with Project Rhino 360<br /> <br /> HDFS Disk-Level Encryption 361<br /> <br /> Token-Based Authentication and Unified Authorization Framework 361<br /> <br /> HBase Cell-Level Security 362<br /> <br /> Putting it All Together — Best Practices for Securing Hadoop 362<br /> <br /> Authentication 363<br /> <br /> Authorization 364<br /> <br /> Network Encryption 364<br /> <br /> Stay Tuned for Hadoop Enhancements 365<br /> <br /> Summary 365<br /> <br /> <b>Chapter 11: Running Hadoop Applications on AWS 367</b><br /> <br /> Getting to Know AWS 368<br /> <br /> Options for Running Hadoop on AWS 369<br /> <br /> Custom Installation using EC2 Instances 369<br /> <br /> Elastic MapReduce 370<br /> <br /> Additional Considerations before Making Your Choice 370<br /> <br /> Understanding the EMR-Hadoop Relationship 370<br /> <br /> EMR Architecture 372<br /> <br /> Using S3 Storage 373<br /> <br /> Maximizing Your Use of EMR 374<br /> <br /> Utilizing CloudWatch and Other AWS Components 376<br /> <br /> Accessing and Using EMR 377<br /> <br /> Using AWS S3 383<br /> <br /> Understanding the Use of Buckets 383<br /> <br /> Content Browsing with the Console 386<br /> <br /> Programmatically Accessing Files in S3 387<br /> <br /> Using MapReduce to Upload Multiple Files to S3 397<br /> <br /> Automating EMR Job Flow Creation and Job Execution 399<br /> <br /> Orchestrating Job Execution in EMR 404<br /> <br /> Using Oozie on an EMR Cluster 404<br /> <br /> AWS Simple Workflow 407<br /> <br /> AWS Data Pipeline 408<br /> <br /> Summary 409<br /> <br /> <b>Chapter 12: Building Enterprise Security Solutions for Hadoop Implementations 411</b><br /> <br /> Security Concerns for Enterprise Applications 412<br /> <br /> Authentication 414<br /> <br /> Authorization 414<br /> <br /> Confidentiality 415<br /> <br /> Integrity 415<br /> <br /> Auditing 416<br /> <br /> What Hadoop Security Doesn’t Natively Provide for Enterprise Applications 416<br /> <br /> Data-Oriented Access Control 416<br /> <br /> Differential Privacy 417<br /> <br /> Encrypted Data at Rest 419<br /> <br /> Enterprise Security Integration 419<br /> <br /> Approaches for Securing Enterprise Applications Using Hadoop 419<br /> <br /> Access Control Protection with Accumulo 420<br /> <br /> Encryption at Rest 430<br /> <br /> Network Isolation and Separation Approaches 430<br /> <br /> Summary 434<br /> <br /> Chapter 13: Hadoop’s Future 435<br /> <br /> Simplifying MapReduce Programming with DSLs 436<br /> <br /> What Are DSLs? 436<br /> <br /> DSLs for Hadoop 437<br /> <br /> Faster, More Scalable Processing 449<br /> <br /> Apache YARN 449<br /> <br /> Tez 452<br /> <br /> Security Enhancements 452<br /> <br /> Emerging Trends 453<br /> <br /> Summary 454<br /> <br /> APPENDIX : Useful Reading 455<br /> <br /> Index 463
<p><b>Boris Lublinsky</b> is principal architect at Nokia and an author of more than 70 publications, including <i>Applied SOA: Service-Oriented Architecture and Design Strategies.</i></p> <p><b>Kevin T. Smith</b> is Director of Technology Solutions for the AMS division of Novetta Solutions, where he builds highly secure, data-oriented solutions for customers.</p> <p><b>Alexey Yakubovich</b> is a system architect at Hortonworks and a member of the Object Management Group SIG on SOA governance and model-driven architecture.</p>
<p><b>Implement Hadoop Solutions throughout the enterprise</b></p> <p>If you're ready to make the most out of massively scalable analytics, you need to know how to take advantage of Hadoop technologies. With this guide, you'll learn how! It focuses on the architecture and approaches for building advanced, Hadoop-based enterprise applications, taking a deeper, code-level dive into delivering real-world solutions. It also walks you through the data design and how it impacts implementation. It explains how MapReduce works, and shows you how to reformulate specific business problems in MapReduce. Throughout the pages, you'll find in-depth Java code examples that you can use, derived from applications that have been successfully built and deployed.</p> <p>Professional Hadoop Solutions:</p> <ul> <li>Explores the MapReduce architecture, its main components, and the MapReduce programming model</li> <li>Discusses how to create reliable MapReduce applications, including testing and debugging, as well as how to use built-in MapReduce facilities</li> <li>Explains how to extend Oozie and use it to integrate Hadoop-based implementations with other enterprise applications</li> <li>Describes how to build real-time Hadoop-based applications and how to leverage real-time Hadoop queries</li> <li>Shows how to extend Hadoop's security capabilities, including encryption, authentication, authorization, Single-Sign-On (SSO), and auditing</li> <li>Covers different approaches to running Hadoop in the Amazon Web Services (AWS) cloud</li> </ul> <p>Programmer Forums</p> <p>Join our Programmer to Programmer forums to ask and answer programming questions about this book, join discussions on the hottest topics in the industry, and connect with fellow programmers from around the world.</p> <p>Code Downloads</p> <p>Take advantage of free code samples from this book, as well as code samples from hundreds of other books, all ready to use.</p> <p><b>Read More</b></p> <p>Find articles, e-books, sample chapters, and tables of contents for hundreds of books, and more reference resources on programming topics that matter to you.</p> <p><b>Wrox Professional guides</b> are written by working developers to address everyday needs. They provide examples, practical solutions, and expert education in new technologies, all designed to help programmers do a better job.</p>

Diese Produkte könnten Sie auch interessieren:

MDX Solutions
MDX Solutions
von: George Spofford, Sivakumar Harinath, Christopher Webb, Dylan Hai Huang, Francesco Civardi
PDF ebook
53,99 €
Concept Data Analysis
Concept Data Analysis
von: Claudio Carpineto, Giovanni Romano
PDF ebook
107,99 €
Handbook of Virtual Humans
Handbook of Virtual Humans
von: Nadia Magnenat-Thalmann, Daniel Thalmann
PDF ebook
150,99 €