Resolve permission issue among DataNodes with NameNode to establish Secure Shell /SSH without a passphrase
Sometimes it has been observed that when we configure and deploy multi-node Hadoop cluster or add new DataNodes, there is an SSH permission issue in communication with Hadoop daemons. In a fully distributed environment when the cluster is alive and running, the NameNode (Hadoop core services like NodeManager, YARN, etc) uses SSH for communication with DataNodes very frequently. Simply, in other words, we can say monitoring the heartbeats of every configured slaves or DataNodes. The error in the terminal console appears as “Permission Denied (public key, password)” once we start the cluster.
Most of the time we suspect that there was an issue in public-private RSA key pair generation followed by granting accurate permissions. And we keep repeating those steps to resolve the issue.
Even though key-pair generation and permission grant were correct to connect via SSH
$ ssh-keygen -t rsa -P ‘’ -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
and able to ssh to all the systems (designated for DataNodes) without a passphrase from NameNode terminal (without starting Hadoop daemons), there could be an issue of permission denied as mentioned above. This issue ideally might come upon unknowing modification/changes in the sshd_config file in any DataNode or Secondary NameNode (if configured in a separate system ) in the cluster. This file is available in /etc/ssh/ in Ubuntu 14.04. Here are the following parameters in the sshd_config that we need to be careful.
- “PubKeyAuthentication” key should be uncommented with value “yes”
2. “PasswordAuthentication” key should be uncommented with value “yes”
3. The key “UsePAM” should be uncommented with value “no”
After verification with necessary corrections, restart the ssh service or reboot the systems.
sudo service network-manager restart
sudo service ssh restart
And finally, restart the cluster after the successful format of NameNode. The error will disappear and successfully starts all the DataNode in the cluster. We used Ubuntu 14.04 as OS in the multi-node cluster.
