大数据环境搭建-Ambari图形化环境配置工具

发表于 更新于 分类于
大数据环境搭建-Ambari图形化环境配置工具

准备环境

https://www.psvmc.cn/article/2022-03-31-bigdata-environment.html

配置安装YUM源

在（master上）

1
mkdir -p /var/www/html

ambari依赖包准备，上传到主节点master的/var/www/html路径下

（本次选择的ambari是比较新的版本2.7.4，HDP3.1.4，Mysql5.7，JDK1.8）

1、安装http服务器（在 master上）

需要安装的包（systemctl status httpd.service检查有的话就不用装了）

2、修改端口

1
vi /etc/httpd/conf/httpd.conf

可以修改/var/www/html至其他目录

最好修改端口 这里使用8081，不用默认80

1
systemctl start httpd.service

3、将Ambari、HDP、HDP-GPL、HDP-UTIL放到master的/var/www/html目录下并解压

链接：https://pan.baidu.com/s/1zs04Kg-mktmA9r60cUwa5w
提取码：psvm

4、并修改repo文件

1
cd /var/www/html/ambari/centos7/2.7.4.0-118

修改repo文件

1
2
3
4
5
6
7
[ambari-2.7.4.0]
name=ambari Version - ambari-2.7.4.0
baseurl=http://192.168.7.101:8081/ambari/centos7/2.x/updates/2.7.4.0
gpgcheck=1
gpgkey=http://192.168.7.101:8081/ambari/centos7/2.x/updates/2.7.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

复制

1
2
cp ambari.repo /etc/yum.repos.d/
cd /var/www/html/HDP/centos7/3.1.4.0-315

修改hdp.repo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#VERSION_NUMBER=3.1.4.0-315

[HDP-3.1.4.0]

name=HDP Version - HDP-3.1.4.0
baseurl=http://192.168.7.101:8081/HDP/centos7/3.x/updates/3.1.4.0
gpgcheck=1
gpgkey=http://192.168.7.101:8081/HDP/centos7/3.x/updates/3.1.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.7.101:8081/HDP-UTILS-1.1.0.22/repos/centos7
gpgcheck=1
gpgkey=http://192.168.7.101:8081/HDP/centos7/3.x/updates/3.1.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

修改

1
2
cp hdp.repo /etc/yum.repos.d/
cd /var/www/html/HDP-GPL/centos7/3.1.4.0-315

编辑

1
vi hdp.gpl.repo

内容

1
2
3
4
5
6
7
8
9
#VERSION_NUMBER=3.1.4.0-315

[HDP-GPL-3.1.4.0]
name=HDP-GPL Version - HDP-GPL-3.1.4.0
baseurl=http://192.168.7.101:8081/HDP-GPL/centos7/3.x/updates/3.1.4.0
gpgcheck=1
gpgkey=http://192.168.7.101:8081/HDP-GPL/centos7/3.x/updates/3.1.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

复制配置

1
cp hdp.gpl.repo /etc/yum.repos.d/

所有服务器上，把在线的备份、删除

1
2
tar zcvf centos_bak.tar.gz CentOS-*.repo
ll

复制到另外两台服务器上

1
2
scp /etc/yum.repos.d/*.repo root@ slave1:/etc/yum.repos.d/
scp /etc/yum.repos.d/*.repo root@ slave2:/etc/yum.repos.d/

所有服务器上，执行：

1
2
3
yum clean all
yum makecache
yum repolist all

安装Mysql( master上)

安装

1
2
3
4
5
6
7
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache
yum install wget

wget -i https://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
yum localinstall mysql57-community-release-el7-10.noarch.rpm
yum repolist enabled | grep mysql

安装mysql 服务器 命令：

1
yum install -y mysql-community-server --nogpgcheck

启动mysql 命令:

1
service mysqld start

查看mysql是否自启动,并且设置开启自启动 命令:

1
2
chkconfig --list | grep mysqld
chkconfig mysqld on

查询初始密码

1
cat /var/log/mysqld.log|grep password

连接

1
2
3
4
mysql -u root -p
set password=password('psvmc123'); 
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'psvmc123' WITH GRANT OPTION;
flush privileges;

设置mysql5.7允许简单密码

1
2
3
4
5
set global validate_password_policy=0; 
set global validate_password_mixed_case_count=0; 
set global validate_password_number_count=3;
set global validate_password_special_char_count=0; 
set global validate_password_length=3;

（可忽略）删除MySQL用户

1
2
3
use mysql;
Delete FROM user Where User='test';
flush privileges;

创建MySQL ambari用户

1
2
3
4
5
6
7
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; 
CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
CREATE USER 'ambari'@' master' IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@' master';
FLUSH PRIVILEGES;

创建MySQL hive用户

1
2
3
4
5
6
7
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost'; 
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
CREATE USER 'hive'@' master' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@' master';
FLUSH PRIVILEGES;

安装ambari（master上执行）

1、拷贝mysql连接驱动，路径可以自定义，创建ambari时用

1
cp mysql-connector-java-5.1.43.jar /usr/share/java/

2、yum install ambari-server

1
yum install ambari-server

3、ambari-server setup（注意以下标红的地方，需要手动输入）

1
ambari-server setup

显示如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
Using python  /usr/bin/python

Setup ambari-server

Checking SELinux...

SELinux status is 'enabled'

SELinux mode is 'permissive'

WARNING: SELinux is set to 'permissive' mode and temporarily disabled.

OK to continue [y/n] (y)? y

Customize user account for ambari-server daemon [y/n] (n)? y

Enter user account for ambari-server daemon (root):

Adjusting ambari-server permissions and ownership...

Checking firewall status...

Checking JDK...

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Custom JDK

==============================================================================

Enter choice (1): 2

WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.

WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.

Path to JAVA_HOME: /usr/java/jdk1.8

Validating JDK on Ambari Server...done.

Check JDK version for Ambari Server...

JDK version found: 8

Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.

Checking GPL software agreement...

GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html

Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

==============================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL / MariaDB

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

[6] - SQL Anywhere

[7] - BDB

==============================================================================

Enter choice (1): 3

Hostname (localhost): 192.168.7.101

Port (3306):

Database name (ambari):

Username (ambari):

Enter Database Password (bigdata):

Re-enter password:

Configuring ambari database...

Enter full path to custom jdbc driver: /usr/share/java/mysql-connector-java-5.1.43.jar

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceed with configuring remote database connection properties [y/n] (y)? y

Extracting system views...

ambari-admin-2.7.4.0.118.jar

....

Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...

Adjusting ambari-server permissions and ownership...

Ambari Server 'setup' completed successfully.

4、如果选择mysql,需要执行mysql连接

1
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.43.jar

5、进mysql执行

1
2
3
4
create database ambari;
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
create database hive;

6、（所有服务器上）关闭DNS服务: lsof -i:53

卸载dns服务

1
2
3
4
5
rpm -qa|grep dnsmasq

rpm -ev <rpm包名> --nodeps

rpm -ev dnsmasq-2.76-9.el7.x86_64 --nodeps

7、修改ambari-server默认端口

1
vi /etc/ambari-server/conf/ambari.properties

如下

1
client.api.port=8089

8、（master上）启动ambari服务:

1
ambari-server start

9、（所有服务器）启动chronyd服务

1
systemctl start chronyd.service

系统配置

1、hive配置beeline，在$HIVE_HOME/conf下添加文件beeline-hs2-connection.xml

1
2
3
4
5
6
7
8
9
10
11
12
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>beeline.hs2.connection.user</name>
    <value>username</value>
  </property>
  <property>
    <name>beeline.hs2.connection.password</name>
    <value>password</value>
  </property>
</configuration>

2、Hive用户配置管理员权限，在hive-site.xml中添加配置：

1
2
3
4
<property> 
  <name>hive.users.in.admin.role</name>  
  <value>hive</value>  
</property>

安装配置部署HDP集群

1、访问Ambari web页面

默认端口8080，我前面改成8089，所以访问（http://192.168.7.101:8089）

admin/admin登陆

2、 开始集群安装

点击启用安装向导，点击开始安装

3、 配置集群名称

注意不要用下划线！！！（一开始用了下划线无法安装，下列截图出现下划线请忽略）

设置：

1
2
3
master
slave1
slave2

4、版本选择

5、 添加需要纳入集群的节点

从master服务器上取免密文件

选择加密文件id_rsa

（也可以选择右边的，不用免密）

6、主机确认

7、选择需要安装的组件（根据自己需要选择）

8、 资源节点分配（合理分配资源）

9、分配各节点需要安装的服务客户端

10、 配置信息、设置密码、集群服务文件路径

可以查看Review模块：下载集群节点服务部署信息

11、 开始安装（不一定会顺利完成，一个个问题解决完再往下一步）

耐心等待。。。

12、安装成功