Franklin

Apache Spark 2.x for Java Developers.

Author/Creator:
Gulati, Sourav.
Edition:
1st ed.
Publication:
Birmingham : Packt Publishing, Limited, 2017.
Format/Description:
Book
1 online resource (338 pages)
Subjects:
Application program interfaces (Computer software).
Form/Genre:
Electronic books.
Contents:
Cover
Copyright
Credits
Foreword
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Table of Contents
Preface
Chapter 1: Introduction to Spark
Dimensions of big data
What makes Hadoop so revolutionary?
Defining HDFS
NameNode
HDFS I/O
YARN
Processing the flow of application submission in YARN
Overview of MapReduce
Why Apache Spark?
RDD - the first citizen of Spark
Operations on RDD
Lazy evaluation
Benefits of RDD
Exploring the Spark ecosystem
What's new in Spark 2.X?
References
Summary
Chapter 2: Revisiting Java
Why use Java for Spark?
Generics
Creating your own generic type
Interfaces
Static method in an interface
Default method in interface
What if a class implements two interfaces which have default methods with same name and signature?
Anonymous inner classes
Lambda expressions
Functional interface
Syntax of Lambda expressions
Lexical scoping
Method reference
Understanding closures
Streams
Generating streams
Intermediate operations
Working with intermediate operations
Terminal operations
Working with terminal operations
String collectors
Collection collectors
Map collectors
Groupings
Partitioning
Matching
Finding elements
Summary
Chapter 3: Let Us Spark
Getting started with Spark
Spark REPL also known as CLI
Some basic exercises using Spark shell
Checking Spark version
Creating and filtering RDD
Word count on RDD
Finding the sum of all even numbers in an RDD of integers
Counting the number of words in a file
Spark components
Spark Driver Web UI
Jobs
Stages
Storage
Environment
Executors
SQL
Streaming
Spark job configuration and submission
Spark REST APIs
Summary.
Chapter 4: Understanding the Spark Programming Model
Hello Spark
Prerequisites
Common RDD transformations
Map
Filter
flatMap
mapToPair
flatMapToPair
union
Intersection
Distinct
Cartesian
groupByKey
reduceByKey
sortByKey
Join
CoGroup
Common RDD actions
isEmpty
collect
collectAsMap
count
countByKey
countByValue
Max
Min
First
Take
takeOrdered
takeSample
top
reduce
Fold
aggregate
forEach
saveAsTextFile
saveAsObjectFile
RDD persistence and cache
Summary
Chapter 5: Working with Data and Storage
Interaction with external storage systems
Interaction with local filesystem
Interaction with Amazon S3
Interaction with HDFS
Interaction with Cassandra
Working with different data formats
Plain and specially formatted text
Working with CSV data
Working with JSON data
Working with XML Data
References
Summary
Chapter 6: Spark on Cluster
Spark application in distributed-mode
Driver program
Executor program
Cluster managers
Spark standalone
Installation of Spark standalone cluster
Start master
Start slave
Stop master and slaves
Deploying applications on Spark standalone cluster
Client mode
Cluster mode
Useful job configurations
Useful cluster level configurations (Spark standalone)
Yet Another Resource Negotiator (YARN)
YARN client
YARN cluster
Useful job configuration
Summary
Chapter 7: Spark Programming Model - Advanced
RDD partitioning
Repartitioning
How Spark calculates the partition count for transformations with shuffling (wide transformations )
Partitioner
Hash Partitioner
Range Partitioner
Custom Partitioner
Advanced transformations
mapPartitions
mapPartitionsWithIndex
mapPartitionsToPair
mapValues.
flatMapValues
repartitionAndSortWithinPartitions
coalesce
foldByKey
aggregateByKey
combineByKey
Advanced actions
Approximate actions
Asynchronous actions
Miscellaneous actions
Shared variable
Broadcast variable
Properties of the broadcast variable
Lifecycle of a broadcast variable
Map-side join using broadcast variable
Accumulators
Driver program
Summary
Chapter 8: Working with Spark SQL
SQLContext and HiveContext
Initializing SparkSession
Reading CSV using SparkSession
Dataframe and dataset
SchemaRDD
Dataframe
Dataset
Creating a dataset using encoders
Creating a dataset using StructType
Unified dataframe and dataset API
Data persistence
Spark SQL operations
Untyped dataset operation
Temporary view
Global temporary view
Spark UDF
Spark UDAF
Untyped UDAF
Type-safe UDAF:
Hive integration
Table Persistence
Summary
Chapter 9: Near Real-Time Processing with Spark Streaming
Introducing Spark Streaming
Understanding micro batching
Getting started with Spark Streaming jobs
Streaming sources
fileStream
Kafka
Streaming transformations
Stateless transformation
Stateful transformation
Checkpointing
Windowing
Transform operation
Fault tolerance and reliability
Data receiver stage
File streams
Advanced streaming sources
Transformation stage
Output stage
Structured Streaming
Recap of the use case
Structured streaming - programming model
Built-in input sources and sinks
Input sources
Built-in Sinks
Summary
Chapter 10: Machine Learning Analytics with Spark MLlib
Introduction to machine learning
Concepts of machine learning
Datatypes
Machine learning work flow
Pipelines
Operations on feature vectors
Feature extractors
Feature transformers.
Feature selectors
Summary
Chapter 11: Learning Spark GraphX
Introduction to GraphX
Introduction to Property Graph
Getting started with the GraphX API
Using vertex and edge RDDs
From edges
EdgeTriplet
Graph operations
mapVertices
mapEdges
mapTriplets
reverse
subgraph
aggregateMessages
outerJoinVertices
Graph algorithms
PageRank
Static PageRank
Dynamic PageRank
Triangle counting
Connected components
Summary
Index.
Notes:
Description based on publisher supplied metadata and other sources.
Local notes:
Electronic reproduction. Ann Arbor, Michigan : ProQuest Ebook Central, 2021. Available via World Wide Web. Access may be limited to ProQuest Ebook Central affiliated libraries.
Contributor:
Kumar, Sumit.
Other format:
Print version: Gulati, Sourav Apache Spark 2.x for Java Developers
ISBN:
9781787129429
OCLC:
999653462
Loading...
Location Notes Your Loan Policy
Description Status Barcode Your Loan Policy