Unix & Linux
ssh scripting parallelism
Updated Sun, 24 Jul 2022 13:42:06 GMT

Automatically run commands over SSH on many servers


There is a list of IP addresses in a .txt file, ex.:

1.1.1.1
2.2.2.2
3.3.3.3

Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is in the known_hosts list (on my PC, Ubuntu 10.04 LTS/bash).

How can I run commands on these servers, and collect the output?

Ideally, I'd like to run the commands in parallel on all the servers.

I'll be using public key authentication on all the servers.

Here are some potential pitfalls:

  • The ssh prompts me to put the given servers ssh key to my known_hosts file.
  • The given commands might return a nonzero exit code, indicating that the output is potentially invalid. I need to recognize that.
  • A connection might fail to be established to a given server, for example because of a network error.
  • There should be a timeout, in case the command runs for longer than expected or the server goes down while running the command.

The servers are AIX/ksh (but I think that doesn't really matter.

Solution

Assuming that you are not able to get pssh or others installed, you could do something similar to:

tmpdir=${TMPDIR:-/tmp}/pssh.$$
mkdir -p $tmpdir
count=0
while IFS= read -r userhost; do
    ssh -n -o BatchMode=yes ${userhost} 'uname -a' > ${tmpdir}/${userhost} 2>&1 &
    count=`expr $count + 1`
done < userhost.lst
while [ $count -gt 0 ]; do
    wait $pids
    count=`expr $count - 1`
done
echo "Output for hosts are in $tmpdir"




Comments (5)

  • +1 – what does exactly wait do in this script?? ty! — Aug 19, 2011 at 15:43  
  • +0 – what happens if a servers key is not in my knows_hosts file? — Aug 19, 2011 at 15:50  
  • +5 – putting scripts into the background creates child processes. When a child process exits, the process 'slot' and resources stay in the system until the parent process exits or the parent process 'waits' for the child. These 'terminated by still present' processes are called 'zombie' processes. It is good behavior to clean up after child proceses and reclaim resources. — Aug 19, 2011 at 19:38  
  • +1 – If not known, then that might screw it up, but you can add -o StrictHostKeyChecking=no to avoid this. However it is better to scan all the servers from a command-line first so the host keys can be added. — Aug 19, 2011 at 19:40  
  • +1 – As I mentioned, after the ssh program is put into the background and after it exits, resources are still kept. The wait command reclaims those resources, including the process' return code (how the processs exited). Then if later in the program, you put another process, lets say a compression, in the background and needs to wait for it, then without this loop, the wait will reclaim one of the completed ssh programs, not the compression - which may still be running. You would get an return status on the wrong child process. It is good to clean up after yourself. — Aug 28, 2011 at 18:23  

References