Securing Hadoop Cluster
part-2
KERBEROS SETUP
Contents
Kerberos: 1
Kerberos Installation and setup: 2
Kerberos KDC server setup. 2
Kerberos Client Setup: 8
Create service principal and keytabs for Hadoop Services. 8
Update the configuration files for each Hadoop service. 10
Kerberos:
–a secured netowrk authentication system developed by MIT in mid 1990.
–KDC (key distribution centre)
— AS (Authntication server)
— Ticket Granting Service
Pre-requisites:
I have used Centos 6.2 for all the VM to setup Hadoop Cluster, hence all the steps mentioned below will be with reference to Centos; steps might vary a little for other OS.
If you are using CentOS/Red Hat Enterprise Linux 5.6 or later, or Ubuntu, which use AES-256 encryption by default for tickets, you must install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File on all cluster and Hadoop user machines. For JCE Policy File installation instructions, see the README.txt file included in the jce_policy-x.zip file.
Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File can be downloaded from below link.
http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
Download the file and then copy the local_policy.jar file and US_export_policy.jar file to the jre security folde as shown below.
[root@dn1 bin]# ll /usr/java/latest/jre/lib/security/
total 144
-rw-r–r– 1 root root 4054 Apr 10 2015 blacklist
-rw-r–r– 1 root root 98626 Apr 10 2015 cacerts
-rw-r–r– 1 root root 158 Mar 16 2015 javafx.policy
-rw-r–r– 1 root root 2593 Apr 10 2015 java.policy
-rw-r–r– 1 root root 18033 Apr 10 2015 java.security
-rw-r–r– 1 root root 98 Apr 10 2015 javaws.policy
-rw-r–r– 1 root root 2500 Jan 29 23:51 local_policy.jar
-rw-r–r– 1 root root 0 Apr 10 2015 trusted.libraries
-rw-r–r– 1 root root 2487 Jan 29 23:51 US_export_policy.jar
[root@dn1 bin]#
In the following section we will first perform the Kerberos setup. And for this demo I will be using my master node nn1.hadoop.com as KDC server.
Kerberos Installation and setup:
Kerberos KDC server setup
1. For Kerberos server Install krb-server, krb5-libs and krb5-workstation
# yum install krb5-server krb5-libs krb5-workstation
It should show the following output (enter Yes when prompted) :
[root@NN1 ~]#
[root@NN1 ~]# yum install –skip-broken krb5-server krb5-libs krb5-workstation
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
* base: centos.mirror.net.in
* extras: mirror.nbrc.ac.in
* updates: centos.mirror.net.in
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package krb5-libs.x86_64 0:1.9-22.el6 will be updated
—> Package krb5-libs.x86_64 0:1.10.3-42.el6 will be an update
—> Package krb5-server.x86_64 0:1.10.3-42.el6 will be installed
updates/filelists_db | 2.5 MB 00:04
—> Package krb5-workstation.x86_64 0:1.10.3-42.el6 will be installed
–> Processing Conflict: krb5-server-1.10.3-42.el6.x86_64 conflicts selinux-policy < 3.7.19-177.el6
–> Restarting Dependency Resolution with new changes.
–> Running transaction check
—> Package selinux-policy.noarch 0:3.7.19-126.el6 will be updated
–> Processing Dependency: selinux-policy = 3.7.19-126.el6 for package: selinux-policy-targeted-3.7.19-126.el6.noarch
–> Processing Dependency: selinux-policy = 3.7.19-126.el6 for package: selinux-policy-targeted-3.7.19-126.el6.noarch
—> Package selinux-policy.noarch 0:3.7.19-279.el6_7.8 will be an update
–> Running transaction check
—> Package selinux-policy-targeted.noarch 0:3.7.19-126.el6 will be updated
—> Package selinux-policy-targeted.noarch 0:3.7.19-279.el6_7.8 will be an update
–> Processing Conflict: krb5-libs-1.10.3-42.el6.x86_64 conflicts libsmbclient < 3.5.10-124
–> Restarting Dependency Resolution with new changes.
–> Running transaction check
—> Package libsmbclient.x86_64 0:3.5.10-114.el6 will be updated
—> Package libsmbclient.x86_64 0:3.6.23-24.el6_7 will be an update
–> Processing Dependency: samba-winbind-clients = 3.6.23-24.el6_7 for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Processing Dependency: libtevent.so.0(TEVENT_0.9.9)(64bit) for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Processing Dependency: libtdb.so.1(TDB_1.2.5)(64bit) for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Processing Dependency: libtdb.so.1(TDB_1.2.2)(64bit) for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Processing Dependency: libtdb.so.1(TDB_1.2.1)(64bit) for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Processing Dependency: libtalloc.so.2(TALLOC_2.0.2)(64bit) for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Processing Dependency: libtevent.so.0()(64bit) for package: libsmbclient-3.6.23-24.el6_7.x86_64
–> Running transaction check
—> Package libtalloc.x86_64 0:2.0.1-1.1.el6 will be updated
—> Package libtalloc.x86_64 0:2.0.7-2.el6 will be an update
—> Package libtdb.x86_64 0:1.2.1-3.el6 will be updated
—> Package libtdb.x86_64 0:1.2.10-1.el6 will be an update
—> Package libtevent.x86_64 0:0.9.18-3.el6 will be installed
—> Package samba-winbind-clients.x86_64 0:3.5.10-114.el6 will be updated
—> Package samba-winbind-clients.x86_64 0:3.6.23-24.el6_7 will be an update
–> Processing Dependency: samba-winbind = 3.6.23-24.el6_7 for package: samba-winbind-clients-3.6.23-24.el6_7.x86_64
–> Running transaction check
—> Package samba-winbind.x86_64 0:3.6.23-24.el6_7 will be installed
–> Processing Dependency: samba-common = 3.6.23-24.el6_7 for package: samba-winbind-3.6.23-24.el6_7.x86_64
–> Running transaction check
—> Package samba-common.x86_64 0:3.6.23-24.el6_7 will be installed
–> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
krb5-server x86_64 1.10.3-42.el6 base 2.0 M
krb5-workstation x86_64 1.10.3-42.el6 base 811 k
Updating:
krb5-libs x86_64 1.10.3-42.el6 base 768 k
libsmbclient x86_64 3.6.23-24.el6_7 updates 1.6 M
selinux-policy noarch 3.7.19-279.el6_7.8 updates 882 k
Installing for dependencies:
libtevent x86_64 0.9.18-3.el6 base 26 k
samba-common x86_64 3.6.23-24.el6_7 updates 10 M
samba-winbind x86_64 3.6.23-24.el6_7 updates 2.2 M
Updating for dependencies:
libtalloc x86_64 2.0.7-2.el6 base 20 k
libtdb x86_64 1.2.10-1.el6 base 33 k
samba-winbind-clients x86_64 3.6.23-24.el6_7 updates 2.0 M
selinux-policy-targeted noarch 3.7.19-279.el6_7.8 updates 3.1 M
Transaction Summary
================================================================================
Install 5 Package(s)
Upgrade 7 Package(s)
Total download size: 23 M
Is this ok [y/N]: y
Downloading Packages:
(1/12): krb5-libs-1.10.3-42.el6.x86_64.rpm | 768 kB 00:01
(2/12): krb5-server-1.10.3-42.el6.x86_64.rpm | 2.0 MB 00:05
(3/12): krb5-workstation-1.10.3-42.el6.x86_64.rpm | 811 kB 00:02
(4/12): libsmbclient-3.6.23-24.el6_7.x86_64.rpm | 1.6 MB 00:04
(5/12): libtalloc-2.0.7-2.el6.x86_64.rpm | 20 kB 00:00
(6/12): libtdb-1.2.10-1.el6.x86_64.rpm | 33 kB 00:00
(7/12): libtevent-0.9.18-3.el6.x86_64.rpm | 26 kB 00:00
(8/12): samba-common-3.6.23-24.el6_7.x86_64.rpm | 10 MB 00:14
(9/12): samba-winbind-3.6.23-24.el6_7.x86_64.rpm | 2.2 MB 00:05
(10/12): samba-winbind-clients-3.6.23-24.el6_7.x86_64.rp | 2.0 MB 00:05
(11/12): selinux-policy-3.7.19-279.el6_7.8.noarch.rpm | 882 kB 00:03
(12/12): selinux-policy-targeted-3.7.19-279.el6_7.8.noar | 3.1 MB 00:08
——————————————————————————-
Total 444 kB/s | 23 MB 00:53
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
Updating : krb5-libs-1.10.3-42.el6.x86_64 1/19
Updating : libtalloc-2.0.7-2.el6.x86_64 2/19
Installing : libtevent-0.9.18-3.el6.x86_64 3/19
Updating : libtdb-1.2.10-1.el6.x86_64 4/19
Updating : samba-winbind-clients-3.6.23-24.el6_7.x86_64 5/19
Installing : samba-common-3.6.23-24.el6_7.x86_64 6/19
Installing : samba-winbind-3.6.23-24.el6_7.x86_64 7/19
Updating : selinux-policy-3.7.19-279.el6_7.8.noarch 8/19
Updating : selinux-policy-targeted-3.7.19-279.el6_7.8.noarch 9/19
Updating : libsmbclient-3.6.23-24.el6_7.x86_64 10/19
Installing : krb5-server-1.10.3-42.el6.x86_64 11/19
Installing : krb5-workstation-1.10.3-42.el6.x86_64 12/19
Cleanup : selinux-policy-targeted-3.7.19-126.el6.noarch 13/19
Cleanup : libsmbclient-3.5.10-114.el6.x86_64 14/19
Cleanup : selinux-policy-3.7.19-126.el6.noarch 15/19
Cleanup : samba-winbind-clients-3.5.10-114.el6.x86_64 16/19
Cleanup : krb5-libs-1.9-22.el6.x86_64 17/19
Cleanup : libtalloc-2.0.1-1.1.el6.x86_64 18/19
Cleanup : libtdb-1.2.1-3.el6.x86_64 19/19
Installed:
krb5-server.x86_64 0:1.10.3-42.el6 krb5-workstation.x86_64 0:1.10.3-42.el6
Dependency Installed:
libtevent.x86_64 0:0.9.18-3.el6 samba-common.x86_64 0:3.6.23-24.el6_7
samba-winbind.x86_64 0:3.6.23-24.el6_7
Updated:
krb5-libs.x86_64 0:1.10.3-42.el6
libsmbclient.x86_64 0:3.6.23-24.el6_7
selinux-policy.noarch 0:3.7.19-279.el6_7.8
Dependency Updated:
libtalloc.x86_64 0:2.0.7-2.el6
libtdb.x86_64 0:1.2.10-1.el6
samba-winbind-clients.x86_64 0:3.6.23-24.el6_7
selinux-policy-targeted.noarch 0:3.7.19-279.el6_7.8
Complete!
[root@NN1 ~]#
**************************
Once the package are installed successfully, edit /etc/krb.conf file
1. Edit /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = HADOOP.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
HADOOP.COM = {
kdc = nn1.hadoop.com
admin_server = nn1.hadoop.com
}
[domain_realm]
.hadoop.com = HADOOP.COM
hadoop.com = HADOOP.COM
================
By default the krb5.conf file will have the following entry, replace the content as shown above
[realms]
EXAMPLE.COM = {
kdc = kerberos.example.com
admin_server = kerberos.example.com
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
====================================================================
2. Edit “ kdc.conf” file
# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
HADOOP.COM = {
#master_key_type = aes256-cts
max_renewable_life = 10d 0h 0m 0s
default_principal_flags = +postdateable, +forwardable, +tgt-based, +renewable, +proxiable, +dup-skey, +allow-tickets, +service, +preauth
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
}
3. Create KDC Database
In order to create database please run command “kdb5_util create –s”. it will prompt you to enter a passphrase for “KDC database master key”.
[root@NN1 ~]# kdb5_util create -s
Loading random data
Initializing database ‘/var/kerberos/krb5kdc/principal’ for realm ‘HADOOP.COM’,
master key name ‘K/M@LINUXPROBLEMS.ORG’
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:
[root@NN1 ~]#
Note: in order to delete/destroy a KDC database use following command .
kdb5_util destroy
e.g # kdb5_util -r HADOOP.COM destroy
4. Add Admini user for KDC database.
Update “/var/kerberos/krb5kdc/kadm5.acl” to add principals who will have administrative access to the Kerberos database.
[root@NN1 ~]# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@HADOOP.COM *
[root@NN1 ~]#
5. Start kadmin service
[root@NN1 ~]# service kadmin start
Starting Kerberos 5 Admin Server: [ OK ]
[root@NN1 ~]#
6. Create the first administrator principal:
When creating the Kerberos principals and keytabs, you can use kadmin.local or kadmin depending on your access and account:
If you have root access to the KDC machine, but you don’t have a Kerberos admin account, use kadmin.local.
If you don’t have root access to the KDC machine, but you do have a Kerberos admin account, use kadmin.
If you have both root access to the KDC machine and a Kerberos admin account, you can use either one.
In the following example we will be creating an admin principal with name “hadoop/admin”. It will prompt you to enter a password for this account. Please take a note of the password for future use.
[root@NN1 ~]# kadmin.local -q “addprinc hadoop/admin”
Authenticating as principal root/admin@HADOOP.COM with password.
WARNING: no policy specified for hadoop/admin@HADOOP.COM; defaulting to no policy
Enter password for principal “hadoop/admin@HADOOP.COM”:
Re-enter password for principal “hadoop/admin@HADOOP.COM”:
Principal “hadoop/admin@HADOOP.COM” created.
[root@NN1 ~]#
Hint: the command to create a principal and keytab is as below (you can run the following command form kadminlocal or kadmin shell)
To add principal: “addprinc -randkey <principalname/host@realm>”
To create a keytab for existing principal:
“xst -norandkey –k <keytab file path and name> <principalname/host@realm>
To delete principal “delete_principal <principalname/host@realm>
Now create a keytab for the admin principal (hadoop/admin@HADOOP.COM) created above (so that we can use the keytab for all further activity instead of providing password each time)
[root@NN1 ~]# kadmin.local
Authenticating as principal hadoop/admin@HADOOP.COM with password.
kadmin.local:
kadmin.local: xst -k /root/hadoop_admin.keytab hadoop/admin@HADOOP.COM
Entry for principal hadoop/admin@HADOOP.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/root/hadoop_admin.keytab.
Entry for principal hadoop/admin@HADOOP.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/root/hadoop_admin.keytab.
Entry for principal hadoop/admin@HADOOP.COM with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/root/hadoop_admin.keytab.
Entry for principal hadoop/admin@HADOOP.COM with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/root/hadoop_admin.keytab.
Entry for principal hadoop/admin@HADOOP.COM with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/root/hadoop_admin.keytab.
Entry for principal hadoop/admin@HADOOP.COM with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/root/hadoop_admin.keytab.
kadmin.local: quit
It will create the keytab in following path “/root/hadoop_admin.keytab”
Note: to exit from kadmin.local or kadmin shell type “quit”.
7. Start KDC server
Now start KDC server using following command:
[root@NN1 ~]# service krb5kdc start
Starting Kerberos 5 KDC: [ OK ]
[root@NN1 ~]#
With this we have competed the Kerberos (KDC) server setup. Now we need to setup the client on each node of hadoop cluster.
Kerberos Client Setup:
1. On client servers install krb5-libs and krb5-workstation
# yum install krb5-libs and krb5-workstation
Ensure the packages are installed successfully.
2. Update “/etc/krb5.conf” file to point to the correct KDC server.
# vi /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = HADOOP.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
HADOOP.COM = {
kdc = NN1.HADOOP.COM
admin_server = centos.linuxproblems.org
}
[domain_realm]
.hadoop.com = HADOOP.COM
hadoop.com = HADOOP.COM
Repeat the above two steps for all nodes.
Create service principal and keytabs for Hadoop Services
1. Create principal for hdfs, yarn, mapred and HTTP for each host.
Note: In the following I have created the principal and keytab for nn1 host please repeat the same steps to create the principal for other hosts as well (by replacing the host name in principal).
# kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “addprinc -randkey yarn/nn1.hadoop.com@HADOOP.COM”
# kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “addprinc -randkey HTTP/nn1.hadoop.com@HADOOP.COM”
# kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “addprinc -randkey hdfs/nn1.hadoop.com@HADOOP.COM”
# kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “addprinc -randkey mapred/nn1.hadoop.com@HADOOP.COM”
2. Create keytab for the principals created above.
kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “xst -norandkey -k /opt/keytabs/hdfs.nn1.keytab hdfs/nn1.hadoop.com@HADOOP.COM HTTP/nn1.hadoop.com@HADOOP.COM”
kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “xst -norandkey -k /opt/keytabs/yarn.nn1.keytab yarn/nn1.hadoop.com@HADOOP.COM HTTP/nn1.hadoop.com@HADOOP.COM”
kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “xst -norandkey -k /opt/keytabs/mapred.nn1.keytab mapred/nn1.hadoop.com@HADOOP.COM HTTP/nn1.hadoop.com@HADOOP.COM”
kadmin -p hadoop/admin -k -t /root/hadoop_admin.keytab -q “xst -norandkey -k /opt/keytabs/HTTP.nn1.keytab HTTP/nn1.hadoop.com@HADOOP.COM”
3. create OS users for each service
we will create OS user for each service and add them to a common group”hadoop”
adduser hdfs -g hadoop
adduser mapred -g hadoop
adduser yarn -g hadoop
4. copy keytabs for respectiveprincipal to each machine
Copy the keytab file for respective host using SCP. Each host should keep the keytab with its host name in principal, and we will keep the keytab names same on each machine to avoid updating it on each host configuration files.
On nn1:
cp /opt/keytabs/hdfs.nn1.keytab /opt/keytabs/hdfs.keytab
cp /opt/keytabs/yarn.nn1.keytab /opt/keytabs/yarn.keytab
cp /opt/keytabs/mapred.nn1.keytab /opt/keytabs/mapred.keytab
cp /opt/keytabs/HTTP.nn1.keytab /opt/keytabs/HTTP.keytab
similarly to copy to other host (e.g. for dn1):
scp /opt/keytabs/hdfs.dn1.keytab dn1:/opt/keytabs/hdfs.keytab
scp /opt/keytabs/yarn.dn1.keytab dn1:/opt/keytabs/yarn.keytab
scp /opt/keytabs/mapred.dn1.keytab dn1:/opt/keytabs/mapred.keytab
scp /opt/keytabs/HTTP.dn1.keytab dn1:/opt/keytabs/HTTP.keytab
And change the permission as below:
chown hdfs:hadoop /opt/keytabs/hdfs.keytab
chown hdfs:hadoop /opt/keytabs/HTTP.keytab
chown mapred:hadoop /opt/keytabs/mapred.keytab
chown yarn:hadoop /opt/keytabs/yarn.keytab
Update the configuration files for each Hadoop service
Properties for core-site.xml
Please add the following in addition to properties already present in core-site.xml
<!–property for Kerberos–>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value> <!– A value of “simple” would disable security. –>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
===============================================================
Propertied for hdfs-site.xml
Please add the following in addition to properties already present in hdfs-site.xml (including the properties added for SSL)
<!–Kerberos–>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<!– NameNode security config –>
<property>
<name>dfs.namenode.keytab.file</name>
<value>/opt/keytabs/hdfs.keytab</value>
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/_HOST@HADOOP.COM</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@HADOOP.COM</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.keytab</name>
<value>/opt/keytabs/hdfs.keytab</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@YOUR-REALM.COM</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>/opt/keytabs/HTTP.keytab</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<!– Secondary NameNode security config –>
<property>
<name>dfs.secondary.namenode.keytab.file</name>
<value>/opt/keytabs/hdfs.keytab</value> <!– path to the HDFS keytab –>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.principal</name>
<value>hdfs/_HOST@HADOOP.COM</value>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@HADOOP.COM</value>
</property>
<!– DataNode security config –>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>hdfs/_HOST@HADOOP.COM</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>/opt/keytabs/hdfs.keytab</value>
</property>
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
=======================================================
Property for mapred-site.xml
Please add the following in addition to properties already present in hdfs-site.xml (including the properties added for SSL)
<!–kerberos–>
<property>
<name>mapreduce.jobhistory.keytab</name>
<value>/opt/keytabs/mapred.keytab</value>
</property>
<property>
<name>mapreduce.jobhistory.principal</name>
<value>mapred/_HOST@HADOOP.COM</value>
</property>
——————————————–
property for yarn-site.xml
Please add the following in addition to properties already present in hdfs-site.xml (including the properties added for SSL)
<!– resource manager secure configuration info –>
<property>
<name>yarn.resourcemanager.principal</name>
<value>yarn/_HOST@HADOOP.COM</value>
</property>
<property>
<name>yarn.resourcemanager.keytab</name>
<value>/opt/keytabs/yarn.keytab</value>
</property>
<!– these (next four) need only to be set on node manager nodes –>
<property>
<name>yarn.nodemanager.principal</name>
<value>yarn/_HOST@HADOOP.COM</value>
</property>
<property>
<name>yarn.nodemanager.keytab</name>
<value>/opt/keytabs/yarn.keytab</value>
</property>
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>hadoop</value>
</property>
<!– OPTIONAL – set these to enable secure proxy server node –>
<property>
<name>yarn.web-proxy.keytab</name>
<value>/opt/keytabs/yarn.keytab</value>
</property>
<property>
<name>yarn.web-proxy.principal</name>
<value>yarn/_HOST@HADOOP.COM</value>
</property>
———————————————
5. Modifying the Container and Script Configure the Linux Container
Update “container-executor.cfg “ file
yarn@dn1 bin]$
[yarn@dn1 bin]$ cat /opt/hadoop/hadoop-2.7.1/etc/hadoop/container-executor.cfg
yarn.nodemanager.linux-container-executor.group=hadoop
yarn.nodemanager.local-dirs=/opt/hadoop/yarn
yarn.nodemanager.log-dirs=/opt/hadoop/yarn/logs
banned.users=hdfs
min.user.id=1000 #Prevent other super-users
allowed.system.users=user1 ##comma separated list of system users who CAN run applications
[yarn@dn1 bin]$
———————
Create “yarn.nodemanager.local-dirs” and “yarn.nodemanager.log-dirs”
mkdir -p /opt/hadoop/yarn/logs
chown hadoop:hadoop -R /opt/hadoop/yarn/
chmod 755 -R /opt/hadoop/yarn/
change the permission for container-executor
[root@dn1 ~]# cd /opt/hadoop/hadoop-2.7.1/bin/
[root@dn1 bin]#
[root@dn1 bin]# chown root:hadoop container-executor
[root@dn1 bin]# chmod 6050 container-executor
[root@dn1 bin]# ll container-executor
—Sr-s— 1 root hadoop 160127 Jun 28 2015 container-executor
[root@dn1 bin]#
File /opt/hadoop/hadoop-2.7.1/etc/hadoop/container-executor.cfg must be owned by root,
ile /opt/hadoop/hadoop-2.7.1/etc/hadoop/container-executor.cfg must not be world or group writable
chown root:hadoop /opt/hadoop/hadoop-2.7.1/etc/hadoop/container-executor.cfg
chown root:hadoop /opt/hadoop/hadoop-2.7.1/etc/hadoop/
chown root:hadoop /opt/hadoop/hadoop-2.7.1/etc/
chown root:hadoop /opt/hadoop/hadoop-2.7.1/etc/bin
chown root:hadoop /opt/hadoop/hadoop-2.7.1/
chown root:hadoop /opt/hadoop/
chown root:hadoop /opt/
chmod 650 /opt/hadoop/hadoop-2.7.1/etc/hadoop/
chmod 650 /opt/hadoop/hadoop-2.7.1/etc/
chmod 650 /opt/hadoop/hadoop-2.7.1/bin
chmod 650 /opt/hadoop/hadoop-2.7.1/
chmod 650 /opt/hadoop/
chmod 650 /opt/
————————————————
verify execution:
./container-executor –checksetup
If no error returns while running the above command then the container-executor is setup correctly.
Start all the services.
Start-dfs.sh
Start-yarn.sh
Now try to access HDFS with user1 (without a valid Kerberos ticket)
error without kerberos ticket:
[user1@nn1 ~]$ hdfs dfs -ls /
16/01/30 09:48:11 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: “nn1.hadoop.com/192.168.1.20”; destination host is: “nn1.hadoop.com”:8020;
[user1@nn1 ~]$
[user1@nn1 ~]$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_500)
[user1@nn1 ~]$
Now perform kinit and then try to access HDFS directory or file:
[user1@nn1 ~]$ kinit
Password for user1@HADOOP.COM:
[user1@nn1 ~]$
[user1@nn1 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: user1@HADOOP.COM
Valid starting Expires Service principal
01/30/16 09:50:50 01/31/16 09:50:45 krbtgt/HADOOP.COM@HADOOP.COM
renew until 01/30/16 09:50:50
[user1@nn1 ~]$
[user1@nn1 ~]$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x – hadoop supergroup 0 2016-01-24 08:49 /test
[user1@nn1 ~]$
==============================================
Some Known error and solution:
- Error:
Illegal principal name hdfs/dn2.hadoop.com@HADOOP.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule
check : /etc/krb5.conf file
could not connect from name node to datanode:
check: java security policy file
error:
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /data/hdfs/dn :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /data/hdfs/dn
check permission if owner has been changed..
- Error:
016-01-30 11:17:22,743 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
2016-01-30 11:17:23,843 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
2016-01-30 11:17:24,815 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container executor initialization is : 1
ExitCodeException exitCode=1: /opt/hadoop/hadoop-2.7.1/bin/container-executor: /lib64/libc.so.6: version `GLIBC_2.14′ not found (required by /opt/hadoop/hadoop-2.7.1/bin/container-executor)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:185)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:216)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:485)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:533)
2016-01-30 11:17:24,817 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
@
Reson/Soltion: check GLIBC version in your OS, if it is older than that .. then it will not work.
- Error:
016-01-30 11:21:28,253 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2016-01-30 11:21:28,253 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server’s Kerberos principal name; Host Details : local host is: “dn1.hadoop.com/192.168.1.30”; destination host is: “nn1.hadoop.com”:8031;
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:202)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:271)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:486)
Reason/solution: verify the resource manager kerberos principal and keytab properties is present as well in nodemanager yarn-site.xml
- Error:
016-01-30 23:18:53,728 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
2016-01-30 23:18:54,694 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:218)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:485)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:533)
Caused by: java.io.IOException: Cannot run program “/opt/hadoop/hadoop-2.7.1/bin/container-executor”: error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:185)
Reason /solution: verify permission on container-executor and directorystructure .
./container-executor –checksetup
Leave a Reply