Home       Trees       Indices       Help   
Spark 1.0.0 Python API Docs
Package pyspark :: Module rdd :: Class RDD
[frames] | no frames]

Class RDD

source code

object --+
         |
        RDD

A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable, partitioned collection of elements that can be operated on in parallel.

Instance Methods
 
__init__(self, jrdd, ctx, jrdd_deserializer)
x.__init__(...) initializes x; see help(type(x)) for signature		 source code
 
id(self)
A unique ID for this RDD (within its SparkContext).		 source code
 
__repr__(self)
repr(x)		 source code
 
context(self)
The SparkContext that this RDD was created on.		 source code
 
cache(self)
Persist this RDD with the default storage level (MEMORY_ONLY).		 source code
 
persist(self, storageLevel)
Set this RDD's storage level to persist its values across operations after the first time it is computed.		 source code
 
unpersist(self)
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.		 source code
 
checkpoint(self)
Mark this RDD for checkpointing.		 source code
 
isCheckpointed(self)
Return whether this RDD has been checkpointed or not		 source code
 
getCheckpointFile(self)
Gets the name of the file to which this RDD was checkpointed		 source code
 
map(self, f, preservesPartitioning=False)
Return a new RDD by applying a function to each element of this RDD.		 source code
 
flatMap(self, f, preservesPartitioning=False)
Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.		 source code
 
mapPartitions(self, f, preservesPartitioning=False)
Return a new RDD by applying a function to each partition of this RDD.		 source code
 
mapPartitionsWithIndex(self, f, preservesPartitioning=False)
Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.		 source code
 
mapPartitionsWithSplit(self, f, preservesPartitioning=False)
Deprecated: use mapPartitionsWithIndex instead.		 source code
 
filter(self, f)
Return a new RDD containing only the elements that satisfy a predicate.		 source code
 
distinct(self)
Return a new RDD containing the distinct elements in this RDD.		 source code
 
sample(self, withReplacement, fraction, seed=None)
Return a sampled subset of this RDD (relies on numpy and falls back on default random generator if numpy is unavailable).		 source code
 
takeSample(self, withReplacement, num, seed=None)
Return a fixed-size sampled subset of this RDD (currently requires numpy).		 source code
 
union(self, other)
Return the union of this RDD and another one.		 source code
 
intersection(self, other)
Return the intersection of this RDD and another one.		 source code
 
__add__(self, other)
Return the union of this RDD and another one.		 source code
 
sortByKey(self, ascending=True, numPartitions=None, keyfunc=lambda x: x)
Sorts this RDD, which is assumed to consist of (key, value) pairs.		 source code
 
glom(self)
Return an RDD created by coalescing all elements within each partition into a list.		 source code
 
cartesian(self, other)
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in self and b is in other.		 source code
 
groupBy(self, f, numPartitions=None)
Return an RDD of grouped items.		 source code
 
pipe(self, command, env={})
Return an RDD created by piping elements to a forked external process.		 source code
 
foreach(self, f)
Applies a function to all elements of this RDD.		 source code
 
foreachPartition(self, f)
Applies a function to each partition of this RDD.		 source code
 
collect(self)
Return a list that contains all of the elements in this RDD.		 source code
 
reduce(self, f)
Reduces the elements of this RDD using the specified commutative and associative binary operator.		 source code
 
fold(self, zeroValue, op)
Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value."		 source code
 
aggregate(self, zeroValue, seqOp, combOp)
Aggregate the elements of each partition, and then the results for all the partitions, using a given combine functions and a neutral "zero value."		 source code
 
max(self)
Find the maximum item in this RDD.		 source code
 
min(self)
Find the maximum item in this RDD.		 source code
 
sum(self)
Add up the elements in this RDD.		 source code
 
count(