下面介绍如何在Windows8.1上搭建hadoop2.7.2的本地模式开发环境，为后期做mapreduce的开发做准备。在搭建开发环境之前，首先选择开发工具，就是大家都很熟悉的Eclipse（本人这次使用的是eclipse4.4.2版本），Eclipse提供了hadoop的插件，我们通过这个插件，就可以在eclipse中编写mapreduce。但是，这个插件可能会随着hadoop的版本升级或者eclipse的版本升级，而需要相应的去进行编译。所以，在我们开发之前，学会编译这个eclipse的hadoop插件至关重要，编译eclipse插件使用ant工具，ant工具不在本次的介绍范围内。1、首先通过sourcetree获取hadoop2x-eclipse-plugin插件。1.1、插件地址在github上:https://github.com/winghc/hadoop2x-eclipse-plugin.git上下载1.2、将下载的插件hadoop2x-eclipse-plugin-master.zip在本地磁盘进行解压，解压之后的目录结构如下：1.3、接着修改F:Hadoopeclipsechajianhadoop2x-eclipse-plugin-mastersrccontribeclipse-plugin目录下的build.xml文件由于网站上都是基于hadoop2.6版本进行的编译，2.7.2版本对于build.xml需要修改如下：
找到
以上这些jar包在编译hadoop2.7.2 eclipse插件的时候需要用到，如果不添加就会报错，所以，我们在ant编译之前先添加进来。1.4、然后再找到
lib/commons-io-${commons-io.version}.jar,并将lib/htrace-core-${htrace.version}.jar替换为lib/htrace-core-${htrace.version}-incubating.jar
1.5、再修改hadoop2x-eclipse-pluginsrcivylibraries.properties文件，这个文件配置了ant构建需要用到各个jar包的版本，以及构建hadoop的版本，由于下载的这个插件是编译hadoop2.6.0的，所以我们需要修改以下配置，更改下列属性和其值使其对应hadoop2.7.2和当前环境的jar包版本： hadoop.version=2.7.2
apacheant.version=1.9.7
commons-collections.version=3.2.2
commons-httpclient.version=3.1
commons-logging.version=1.1.3
commons-io.version=2.4
slf4j-api.version=1.7.10
slf4j-log4j12.version=1.7.10其实在ant构建的时候，会选择本地hadoop2.7.2目录中的jar包版本(hadoop-2.7.2sharehadoopcommon)，所以只要将版本号改成对应的版本号即可如下图：
1.6、最后修改hadoop2x-eclipse-pluginivylibraries.properties文件，文件的的版本如上图版本修改一样，但是还有一个版本需要修改的就是htrace.version的版本要改成3.1.0，htrace.version=3.1.01.7、然后cd到F:Hadoopeclipsechajianhadoop2x-eclipse-plugin-mastersrccontribeclipse-plugin目录
执行以下命令：ant jar -Dversion=2.7.2 -Declipse.home=D:eclipse_hadoop -Dhadoop.home=F:Hadoophadoop-2.7.2解释下这个命令：-Dversion是指这个插件的版本，Declipse.home是指eclipse的安装目录，-Dhadoop.home指本地文件中hadoop-2.7.2的安装目录。命令执行成功之后就可以在hadoop2x-eclipse-pluginbuildcontribeclipse-plugin目录下面找到hadoop-eclipse-plugin-2.7.2.jar 包，这个包就是编译好的eclipse hadoop2.7.2插件，把这个插件放到eclipse安装目录的plugins目录下，我们就可以进入eclipse然后找到一个叫mapreduce的视图，就可以开始尝试编写mapreduce程序了。1.8、下载eclipse并配置JDK去http://www.eclipse.org/downloads/ 下载你需要的版本，我们这里下载的是win64位版。直接解压到目录中。进行简单设置，根据你的开发需要，选择jdk的版本1.9、设置hadoop插件在eclipse菜单中选择，window -preferences，打开设置菜单至此Eclipse开发环境搭建完成，下面将搭建hadoop的运行环境，hadoop项目是需要将程序提交到hadoop运行环境上面运行的。
2、Eclipse插件编译好之后，就需要安装Hadoop2.7.2hadoop环境搭建相对麻烦，需要安装虚拟机或者着cygwin什么的，但是通过查官方资料和摸索，在window上搭建了本地模式，可以不需要虚拟机和cygwin依赖，而且官网明确指出cygwin已经不支持hadoop2.x。Windows下搭建Hadoop本地模式运行环境参考：http://wiki.apache.org/hadoop/Hadoop2OnWindows 下面配置windows环境：2.1、Java JDK ：我采用的是1.8的，配置JAVA_HOME，如果默认安装，会安装在C:Program FilesJavajdk1.8.0_51。此目录存在空格，启动hadoop时将报错，JAVA_HOME is incorrect …此时需要将环境变量JAVA_HOME值修改为：C:Progra~1Javajdk1.8.0_51，Program Files可以有Progra~代替。2.2、Hadoop 环境变量： 新建HADOOP_HOME，指向hadoop解压目录，如：F:Hadoophadoop-2.7.2。然后在path环境变量中增加：%HADOOP_HOME%bin;。2.3、Hadoop 依赖库：winutils相关，hadoop在windows上运行需要winutils支持和hadoop.dll等文件，下载地址：http://download.csdn.net/detail/fly_leopard/9503059注意hadoop.dll等文件不要与hadoop冲突。为了不出现依赖性错误可以将hadoop.dll放到c:/windows/System32下一份，然后重启计算机。2.4、hadoop环境测试：起一个cmd窗口，切换到hadoop-2.7.2bin下，执行hadoop version命令，显示如下：2.5、hadoop基本文件配置：hadoop配置文件位于：hadoop-2.7.2etchadoop下
core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml
core-site.xml:
hdfs-site.xml:
mapred-site.xml:
yarn-site.xml:
2.6、格式化系统文件：hadoop-2.7.2/bin下执行 hdfs namenode -format待执行完毕即可，不要重复format容易出现异常。
2.7、格式化完成后到hadoop-2.7.2/sbin下执行 start-dfs.cmd启动hadoop
访问：http://localhost:50070
2.8、在hadoop-2.7.2/sbin下执行start-yarn.cmd启动yarn，访问http://localhost:8088可查看 资源、节点管理
至此表示hadoop2.7.2运行环境搭建完成。
3、结合Eclipse创建MR项目并使用本地系统进行hadoop本地模式开发我在者使用Eclipse开发使用的是本地文件系统，没有使用HDFS，HDFS在完全分布式下介绍的多，在这就不用介绍，另外使用Eclipse开发并不是很多文章介绍一定要配置DFS Locations（这个不影响开发），这个是用来查看集群上的HDFS文件系统的（我目前是这样理解），反正我使用这个连接本地windows8.1上启动的hadoop（本地模式），一直没练成功过，报下面的错误：java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilderat org.apache.hadoop.hdfs.DFSClient.
好了，下面进入使用Eclipse开发hadoop的介绍3.1、上面环境搭建完成之后，下面开始讲如何进行开发，我们使用hadoop的wordcount来做测试。创建mr项目设置项目名称创建类设置类属性创建完成后，将hadoop-2.7.2-srchadoop-mapreduce-projecthadoop-mapreduce-examplessrcmainjavaorgapachehadoopexamples目录下的WordCount.java文件内容，copy到刚创建的文件中。3.2接下来创建配置环境在项目中创建一个名为resources的Source Floder，然后将F:Hadoophadoop-2.7.2etchadoop下的配置文件全部copy到该目录下。3.3、运行WordCount程序以上完成后，即完成开发环境配置，接下来试试运行是否成功。上图中红圈标注的是重点，配置的是wordcount的输入输出路径，因为本地模式我使用的是本地文件系统而不是HDFS，所以该地方是使用的file:///而不是hdfs://（需要特别注意）。 然后点击Run按钮，hadoop就可运行了。当出现下面情况，则表示运行成功：16/09/15 22:18:37 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:803216/09/15 22:18:39 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).16/09/15 22:18:39 INFO input.FileInputFormat: Total input paths to process : 216/09/15 22:18:40 INFO mapreduce.JobSubmitter: number of splits:216/09/15 22:18:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1473949101198_000116/09/15 22:18:41 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.16/09/15 22:18:41 INFO impl.YarnClientImpl: Submitted application application_1473949101198_000116/09/15 22:18:41 INFO mapreduce.Job: The url to track the job: http://Lenovo-PC:8088/proxy/application_1473949101198_0001/16/09/15 22:18:41 INFO mapreduce.Job: Running job: job_1473949101198_000116/09/15 22:18:53 INFO mapreduce.Job: Job job_1473949101198_0001 running in uber mode : false16/09/15 22:18:53 INFO mapreduce.Job: map 0% reduce 0%16/09/15 22:19:03 INFO mapreduce.Job: map 100% reduce 0%16/09/15 22:19:10 INFO mapreduce.Job: map 100% reduce 100%16/09/15 22:19:11 INFO mapreduce.Job: Job job_1473949101198_0001 completed successfully16/09/15 22:19:12 INFO mapreduce.Job: Counters: 50File System CountersFILE: Number of bytes read=119FILE: Number of bytes written=359444FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=194HDFS: Number of bytes written=0HDFS: Numb 香港云主机er of read operations=2HDFS: Number of large read operations=0HDFS: Number of write operations=0Job CountersKilled map tasks=1Launched map tasks=2Launched reduce tasks=1Rack-local map tasks=2Total time spent by all maps in occupied slots (ms)=12156Total time spent by all reduces in occupied slots (ms)=4734Total time spent by all map tasks (ms)=12156Total time spent by all reduce tasks (ms)=4734Total vcore-milliseconds taken by all map tasks=12156Total vcore-milliseconds taken by all reduce tasks=4734Total megabyte-milliseconds taken by all map tasks=12447744Total megabyte-milliseconds taken by all reduce tasks=4847616Map-Reduce FrameworkMap input records=2Map output records=8Map output bytes=78Map output materialized bytes=81Input split bytes=194Combine input records=8Combine output records=6Reduce input groups=4Reduce shuffle bytes=81Reduce input records=6Reduce output records=4Spilled Records=12Shuffled Maps =2Failed Shuffles=0Merged Map outputs=2GC time elapsed (ms)=187CPU time spent (ms)=1733Physical memory (bytes) snapshot=630702080Virtual memory (bytes) snapshot=834060288Total committed heap usage (bytes)=484966400Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format CountersBytes Read=44File Output Format CountersBytes Written=43
然后在输出路径（运行中配置的输出路径）中查看运行结果：
运行当中可能出现如下问题：1）、问题1：16/09/15 22:12:08 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032Exception in thread “main” java.net.ConnectException: Call From Lenovo-PC/192.168.1.105 to 0.0.0.0:9000 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefusedat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)at org.apache.hadoop.ipc.Client.call(Client.java:1479)at org.apache.hadoop.ipc.Client.call(Client.java:1412)at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)Caused by: java.net.ConnectException: Connection refused: no further informationat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)at org.apache.hadoop.ipc.Client.call(Client.java:1451)… 27 more
出现上述问题是由于项目中的core-site.xml中和本地安装的hadoop配置文件core-site.xml中的端口不一致，请修改成一致。
2）、问题2：16/09/15 22:14:45 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:803216/09/15 22:14:48 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)16/09/15 22:14:50 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)16/09/15 22:14:52 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)16/09/15 22:14:54 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
如果出现上述问题表示yarn没有启动，请启动yarn。
3）、问题3：16/09/15 22:16:00 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:803216/09/15 22:16:02 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).16/09/15 22:16:02 INFO input.FileInputFormat: Total input paths to process : 216/09/15 22:16:03 INFO mapreduce.JobSubmitter: number of splits:216/09/15 22:16:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1473948945298_000116/09/15 22:16:04 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.16/09/15 22:16:04 INFO impl.YarnClientImpl: Submitted application application_1473948945298_000116/09/15 22:16:04 INFO mapreduce.Job: The url to track the job: http://Lenovo-PC:8088/proxy/application_1473948945298_0001/16/09/15 22:16:04 INFO mapreduce.Job: Running job: job_1473948945298_000116/09/15 22:16:08 INFO mapreduce.Job: Job job_1473948945298_0001 running in uber mode : false16/09/15 22:16:08 INFO mapreduce.Job: map 0% reduce 0%16/09/15 22:16:08 INFO mapreduce.Job: Job job_1473948945298_0001 failed with state FAILED due to: Application application_1473948945298_0001 failed 2 times due to AM Container for appattempt_1473948945298_0001_000002 exited with exitCode: -1000For more detailed output, check application tracking page:http://Lenovo-PC:8088/cluster/app/application_1473948945298_0001Then, click on links to logs of each attempt.Diagnostics: Could not find any valid local directory for nmPrivate/container_1473948945298_0001_02_000001.tokensFailing this attempt. Failing the application.16/09/15 22:16:08 INFO mapreduce.Job: Counters: 0
如果出现上述问题，表示你没有使用管理员权限启动hadoop，请使用管理员权限启动hadoop。
这篇文章主要为大家展示了“win8系统网络受限故障怎么处理”，内容简而易懂，条理清晰，希望能够帮助大家解决疑惑，下面让小编带领大家一起研究并学习一下“win8系统网络受限故障怎么处理”这篇文章吧。1、使用快捷键按下“win键+x键”，并选择“命令提示符(管理员…
免责声明：本站发布的图片视频文字，以转载和分享为主，文章观点不代表本站立场，本站不承担相关法律责任；如果涉及侵权请联系邮箱：360163164@qq.com举报，并提供相关证据，经查实将立刻删除涉嫌侵权内容。
微信扫一扫 
