网站建设公司浙江华企,祝明电子商务网站建设实验报告,四川省城乡和住房建设厅网站,网红商城自助下单app前言
Hadoop是一个由Apache基金会所开发的分布式系统基础架构#xff0c;主要解决海量数据的存储和海量数据的分析计算问题#xff0c;广义上来说#xff0c;Hadoop通常是指一个更广泛的概念–hadoop生态圈
Hadoop优缺点#xff1a; 优点#xff1a; 1、高可靠性#x…前言
Hadoop是一个由Apache基金会所开发的分布式系统基础架构主要解决海量数据的存储和海量数据的分析计算问题广义上来说Hadoop通常是指一个更广泛的概念–hadoop生态圈
Hadoop优缺点 优点 1、高可靠性Hadoop底层维护多个数据版本所以即使Hadoop某个计算元素或者存储出现故障也不会导致数据的丢失 2、高扩展性在集群间分配任务数据可方便的扩展到数以千计的节点上 3、高效性在MapReduce的思想下Hadoop是并行工作的以加快任务处理速度 4、高容错性能够自动将失败的任务重新分配 缺点 1、不适合低延时数据访问毫秒级的数据访问 2、无法高效对大量小文件进行存储存储大量小文件的话会占用NameNode大量的内存来存储文件目录和块信息NameNode的内存总是有限的小文件的存储的寻址时间会超过读取时间违反了HDFS的设计目标 3、不支持并发写入、文件随机修改一个文件只能有一个写不允许多个线程同时写仅支持数据追加不支持文件的随机修改
参考链接 https://blog.csdn.net/weixin_43842853/article/details/123007306 https://blog.csdn.net/weixin_52112640/article/details/124907147
一、安装启动
配置java环境 yum install java-1.8.0-openjdk-devel echo export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk /etc/profile source /etc/profile
创建密钥 ssh-keygen -t rsa -P ‘’ -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys
下载安装包 mkdir -p /usr/local/hadoop wget https://mirrors.sonic.net/apache/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz -P /usr/local/hadoop cd /usr/local/hadoop tar -xvf hadoop-3.3.4.tar.gz
#配置核心组件core-site.xml
cat - EOF /usr/local/hadoop/hadoop-3.3.4/etc/hadoop/core-site.xml
configurationpropertynamefs.defaultFS/namevaluehdfs://bogon:9000/value/property
/configuration
EOF#配置文件系统配置文件hdfs-site.xml
cat - EOF /usr/local/hadoop/hadoop-3.3.4/etc/hadoop/hdfs-site.xml
configurationpropertynamedfs.replication/namevalue1/value/property
/configuration
EOF#配置env定义JAVA_HOME路径
sed -i s!# export JAVA_HOME!export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk! /usr/local/hadoop/hadoop-3.3.4/etc/hadoop/hadoop-env.sh#hadoop-3.x为了提升安全性需要指定操作hadoop进程的用户
sed -i 2i HDFS_DATANODE_USERroot\nHDFS_DATANODE_SECURE_USERhdfs\nHDFS_NAMENODE_USERroot\nHDFS_SECONDARYNAMENODE_USERroot /usr/local/hadoop/hadoop-3.3.4/sbin/start-dfs.sh
sed -i 2i YARN_RESOURCEMANAGER_USERroot\nHADOOP_SECURE_DN_USERyarn\nYARN_NODEMANAGER_USERroot /usr/local/hadoop/hadoop-3.3.4/sbin/start-yarn.sh#格式化文件系统 cd /usr/local/hadoop/hadoop-3.3.4/ bin/hdfs namenode -format 会看到类似如下的输出
2023-03-07 16:10:47,309 INFO namenode.FSImage: Allocated new BlockPoolId: BP-512421437-10.130.0.73-1678176647285
2023-03-07 16:10:47,333 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
2023-03-07 16:10:47,402 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2023-03-07 16:10:47,657 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 396 bytes saved in 0 seconds .
2023-03-07 16:10:47,689 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid 0
2023-03-07 16:10:47,739 INFO namenode.FSNamesystem: Stopping services started for active state
2023-03-07 16:10:47,740 INFO namenode.FSNamesystem: Stopping services started for standby state
2023-03-07 16:10:47,747 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid0 when meet shutdown.
2023-03-07 16:10:47,748 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at bogon/10.130.0.73
************************************************************/启动服务 [rootbogon hadoop-3.3.4]# sbin/start-dfs.sh Starting namenodes on [bogon] Starting datanodes Starting secondary namenodes [bogon] 2023-03-07 17:17:19,371 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[rootbogon hadoop-3.3.4]# sbin/start-yarn.sh Starting resourcemanager Starting nodemanagers
二、查看进程
#如果显示SecondaryNameNode、ResourceManager、NameNode、NodeManager、DataNode 进程代表hadoop服务启动成功
[rootbogon hadoop-3.3.4]# jps
127968 NameNode
128672 ResourceManager
128110 DataNode
128816 NodeManager
128306 SecondaryNameNode
129183 Jpsweb端访问 http://本机ip:9870
