如何用命令行的方式运行Spark平台的wordcount项目，针对这个问题，这篇文章详细介绍了相对应的分析和解答，希望可以帮助更多想解决这个问题的小伙伴找到更简单易行的方法。单机模式运行，即local模式local模式运行非常简单，只要运行以下命令即可，假设当前目录是$SPARK_HOME
MASTER=local bin/spark-shell
“MASTER=local”就是表明当前运行在单机模式
scala> val textFile = sc.textFile(“README.md”)
val textFile = sc.textFile(“jerry.test”)
15/08/08 19:14:32 INFO MemoryStore: ensureFreeSpace(182712) called with curMem=664070, maxMem=278302556
15/08/08 19:14:32 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 178.4 KB, free 264.6 MB)
15/08/08 19:14:32 INFO MemoryStore: ensureFreeSpace(17237) called with curMem=846782, maxMem=278302556
15/08/08 19:14:32 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 16.8 KB, free 264.6 MB)
15/08/08 19:14:32 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:37219 (size: 16.8 KB, free: 265.3 MB)
15/08/08 19:14:32 INFO SparkContext: Created broadcast 7 from textFile at :21
textFile: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[12] at textFile at :21
then: textFile.filter(.contains(“Spark”)).count
or textFile.flatMap(.split(” “)).map((_, 1))
15/08/08 19:16:27 INFO FileInputFormat: Total input paths to process : 1
15/08/08 19:16:27 INFO SparkContext: Starting job: count at :24
15/08/08 19:16:27 INFO DAGScheduler: Got job 0 (count at :24) with 1 output partitions (allowLocal=false)
15/08/08 19:16:27 INFO DAGScheduler: Final stage: ResultStage 0(count at :24)
15/08/08 19:16:27 INFO DAGScheduler: Parents of final stage: List()
15/08/08 19:16:27 INFO DAGScheduler: Missing parents: List()
15/08/08 19:16:27 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at filter at :24), which has no missing parents
15/08/08 19:16:27 INFO MemoryStore: ensureFreeSpace(3184) called with curMem=156473, maxMem=278302556
15/08/08 19:16:27 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.1 KB, free 265.3 MB)
15/08/08 19:16:27 INFO MemoryStore: ensureFreeSpace(1855) called with curMem=159657, maxMem=278302556
15/08/08 19:16:27 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1855.0 B, free 265.3 MB)
15/08/08 19:16:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:42648 (size: 1855.0 B, free: 265.4 MB)
15/08/08 19:16:27 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
15/08/08 19:16:27 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at filter at :24)
15/08/08 19:16:27 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/08/08 19:16:27 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1415 bytes)
15/08/08 19:16:27 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
15/08/08 19:16:27 INFO HadoopRDD: Input split: file:/root/devExpert/spark-1.4.1/README.md:0+3624
15/08/08 19:16:27 INFO deprecation: mapred.tip.id is deprecated. Instead, 香港云主机use mapreduce.task.id
15/08/08 19:16:27 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/08/08 19:16:27 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/08/08 19:16:27 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/08/08 19:16:27 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/08/08 19:16:27 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1830 bytes result sent to driver
15/08/08 19:16:27 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 80 ms on localhost (1/1)
15/08/08 19:16:27 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/08/08 19:16:27 INFO DAGScheduler: ResultStage 0 (count at :24) finished in 0.093 s
15/08/08 19:16:27 INFO DAGScheduler: Job 0 finished: count at :24, took 0.176689 s
res0: Long = 19关于如何用命令行的方式运行Spark平台的wordcount项目问题的解答就分享到这里了，希望以上内容可以对大家有一定的帮助，如果你还有很多疑惑没有解开，可以关注开发云行业资讯频道了解更多相关知识。
相关推荐: golang中怎么利用leetcode实现Z字形变换
golang中怎么利用leetcode实现字形变换，针对这个问题，这篇文章详细介绍了相对应的分析和解答，希望可以帮助更多想解决这个问题的小伙伴找到更简单易行的方法。将一个给定字符串根据给定的行数，以从上往下、从左到右进行 Z 字形排列。比如输入字符串为 “LE…
免责声明：本站发布的图片视频文字，以转载和分享为主，文章观点不代表本站立场，本站不承担相关法律责任；如果涉及侵权请联系邮箱：360163164@qq.com举报，并提供相关证据，经查实将立刻删除涉嫌侵权内容。
微信扫一扫 
