:: Experimental :: Add an archive to be downloaded and unpacked with this Spark job on every node.

If an archive is added during execution, it will not be available until the next TaskSet starts.

Add a file to be downloaded with this Spark job on every node.

If a file is added during execution, it will not be available until the next TaskSet starts.

Add a file to be downloaded with this Spark job on every node.

If a file is added during execution, it will not be available until the next TaskSet starts.

Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.

If a jar is added during execution, it will not be available until the next TaskSet starts.

Add a tag to be assigned to all the jobs started by this thread.

Often, a unit of execution in an application consists of multiple Spark actions or jobs. Application programmers can use this method to group all those jobs together and give a group tag. The application can use org.apache.spark.sql.SparkSession.interruptTag to cancel all running executions with this tag. For example:

// In the main thread:
sc.addJobTag("myjobs")
sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()

// In a separate thread:
spark.cancelJobsWithTag("myjobs")

There may be multiple tags present at the same time, so different parts of application may use different tags to perform cancellation at different levels of granularity.

Add multiple tags to be assigned to all the jobs started by this thread. See addJobTag for more details.

:: DeveloperApi :: Register a listener to receive up-calls from events that happen during execution.

A unique identifier for the Spark application. Its format depends on the scheduler implementation. (i.e. in case of local spark app something like 'local-1433865536131' in case of YARN something like 'application_1433865536131_34483' )

Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file (useful for binary data)

For example, if you have the following files:

hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn

Do val rdd = sparkContext.binaryFiles("hdfs://a-hdfs-path"),

then rdd contains

(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)

Load data from a flat binary file, assuming the length of each record is constant.

Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions. The variable will be sent to each executor only once.

Cancel a given job if it's scheduled or running.

Cancel a given job if it's scheduled or running.

Cancel active jobs for the specified group. See org.apache.spark.SparkContext.setJobGroup for more information.

Cancel active jobs for the specified group. See org.apache.spark.SparkContext.setJobGroup for more information.

Cancel active jobs for the specified group, as well as the future jobs in this job group. Note: the maximum number of job groups that can be tracked is set by 'spark.scheduler.numCancelledJobGroupsToTrack'. Once the limit is reached and a new job group is to be added, the oldest job group tracked will be discarded.

Cancel active jobs for the specified group, as well as the future jobs in this job group. Note: the maximum number of job groups that can be tracked is set by 'spark.scheduler.numCancelledJobGroupsToTrack'. Once the limit is reached and a new job group is to be added, the oldest job group tracked will be discarded.

Cancel active jobs that have the specified tag. See org.apache.spark.SparkContext.addJobTag.

Cancel active jobs that have the specified tag. See org.apache.spark.SparkContext.addJobTag.

Cancel a given stage and all jobs associated with it.

Cancel a given stage and all jobs associated with it.

Clear the current thread's job tags.

Default min number of partitions for Hadoop RDDs when not given by user Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2. For large files, the Hadoop InputFormat library always creates more partitions even though defaultMinPartitions is 2. For small files, it can be good to process small files quickly. However, usually when Spark joins a small table with a big one, we'll still spend most of time on the map part of the big one anyway.

Default level of parallelism to use when not given by user (e.g. parallelize and makeRDD).

:: DeveloperApi :: Return pools for fair scheduler

Return a copy of this SparkContext's configuration. The configuration cannot be changed at runtime.

Get the tags that are currently set to be assigned to all the jobs started by this thread.

Get a local property set in this thread, or null if it is missing. See org.apache.spark.SparkContext.setLocalProperty.

Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.

:: DeveloperApi :: Return the pool associated with the given name, if one exists

:: DeveloperApi :: Return information about what RDDs are cached, if they are in mem or on disk, how much space they take, etc.

Get a read-only reference to the spark conf. This is preferred version over getConf.

A default Hadoop Configuration for the Hadoop code (e.g. file systems) that we reuse.

Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly. Instead, callers can just write, for example,

val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)

Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly. Instead, callers can just write, for example,

val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minPartitions)

Get an RDD for a Hadoop file with an arbitrary InputFormat

Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable), using the older MapReduce API (org.apache.hadoop.mapred).

:: DeveloperApi :: Request that the cluster manager kill the specified executor.

:: DeveloperApi :: Request that the cluster manager kill the specified executors.

This is not supported when dynamic allocation is turned on.

Kill and reschedule the given task attempt. Task ids can be obtained from the Spark UI or through SparkListener.onTaskStart.

:: Experimental :: Returns a list of archive paths that are added to resources.

Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object. Create a new partition for each collection item.

Distribute a local Scala collection to form an RDD.

This method is identical to parallelize.

Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

Smarter version of newApiHadoopFile that uses class tags to figure out the classes of keys, values and the org.apache.hadoop.mapreduce.InputFormat (new MapReduce API) so that user don't need to pass them directly. Instead, callers can just write, for example: val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)

Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

Distribute a local Scala collection to form an RDD.

Creates a new RDD[Long] containing elements from start to end(exclusive), increased by step every element.

Register the given accumulator with given name.

Register the given accumulator.

Remove a tag previously added to be assigned to all the jobs started by this thread. Noop if such a tag was not added earlier.

Remove multiple tags to be assigned to all the jobs started by this thread. See removeJobTag for more details.

:: DeveloperApi :: Deregister the listener from Spark's listener bus.

:: DeveloperApi :: Request an additional number of executors from the cluster manager.

Update the cluster manager on our scheduling needs. Three bits of information are included to help it make decisions. This applies to the default ResourceProfile.

:: DeveloperApi :: Run a job that can return approximate results.

Run a job on all partitions in an RDD and pass the results to a handler function.

Run a job on all partitions in an RDD and pass the results to a handler function. The function that is run against each partition additionally takes TaskContext argument.

Run a job on all partitions in an RDD and return the results in an array.

Run a job on all partitions in an RDD and return the results in an array. The function that is run against each partition additionally takes TaskContext argument.

Run a function on a given set of partitions in an RDD and return the results as an array.

Run a function on a given set of partitions in an RDD and return the results as an array. The function that is run against each partition additionally takes TaskContext argument.

Run a function on a given set of partitions in an RDD and pass the results to the given handler function. This is the main entry point for all actions in Spark.

Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter. For example, to access a SequenceFile where the keys are Text and the values are IntWritable, you could simply write

sparkContext.sequenceFile[String, Int](path, ...)

WritableConverters are provided in a somewhat strange way (by an implicit function) to support both subclasses of Writable and types for which we define a converter (e.g. Int to IntWritable). The most natural thing would've been to have implicit objects for the converters, but then we couldn't have an object for every subclass of Writable (you can't have a parameterized singleton object). We use functions instead to create a new converter for the appropriate type. In addition, we pass the converter a ClassTag of its type to allow it to figure out the Writable class to use in the subclass case.

Get an RDD for a Hadoop SequenceFile with given key and value types.

Get an RDD for a Hadoop SequenceFile with given key and value types.

Set the directory under which RDDs are going to be checkpointed.

Set the behavior of job cancellation from jobs started in this thread.

Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.

Often, a unit of execution in an application consists of multiple Spark actions or jobs. Application programmers can use this method to group all those jobs together and give a group description. Once set, the Spark web UI will associate such jobs with this group.

The application can also use org.apache.spark.SparkContext.cancelJobGroup to cancel all running jobs in this group. For example,

// In the main thread:
sc.setJobGroup("some_job_to_cancel", "some job description")
sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()

// In a separate thread:
sc.cancelJobGroup("some_job_to_cancel")

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. User-defined properties may also be set here. These properties are propagated through to worker tasks and can be accessed there via org.apache.spark.TaskContext#getLocalProperty.

These properties are inherited by child threads spawned from this thread. This may have unexpected consequences when working with thread pools. The standard java implementation of thread pools have worker threads spawn other worker threads. As a result, local properties may propagate unpredictably.

To remove/unset property simply set value to null e.g. sc.setLocalProperty("key", null)

Control our logLevel. This overrides any user-defined log settings.

Shut down the SparkContext with exit code that will passed to scheduler backend. In client mode, client side may call SparkContext.stop() to clean up but exit with code not equal to 0. This behavior cause resource scheduler such as ApplicationMaster exit with success status but client side exited with failed status. Spark can call this method to stop SparkContext and pass client side correct exit code to scheduler backend. Then scheduler backend should send the exit code to corresponding resource scheduler to keep consistent.

Submit a job for execution and return a FutureJob holding the result.

Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings. The text files must be encoded as UTF-8.

Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file. The text files must be encoded as UTF-8.

For example, if you have the following files:

hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn

Do val rdd = sparkContext.wholeTextFile("hdfs://a-hdfs-path"),

then rdd contains

(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)