GNU/Parallel – Run multiple commands simultaneously

GNU/Parallel is an open source tiny executable yet so powerful to run / start multiple commands at the same time. This is a handy tool when we are trying to run multiple scripts to maintain consistency of timestamp across the board.

gnu-parallel

Below is a simple file named as REST to execute Python RESTAPI scripts simultaneously

date > start.$today.array;python sym_perf_stats_v4.py &>> /vin/array.$today.log; date > end.$today.array
date > start.$today.fe;python fe_perf_stats_v3.py &>> /vin/fe.$today.log; date > end.$today.fe
date > start.$today.be; python be_perf_stats_v3.py &>> /vin/be.$today.log; date > end.$today.be
date > start.$today.diskgrp ; python dg_perf_stats_v3.py &>> /vin/diskgrp.$today.log; date > end.$today.diskgrp
date > start.$today.tp ; python tp_perf_stats_v3.py &>> /vin/tp.$today.log; date > end.$today.tp

to run the above scripts in parallel we need to execute the below command

#parallel -j 5 <REST

-j switch used to run ‘n’ number of commands in parallel from the same console

***UPDATE *** As per suggestion from oletange. Above file ‘REST’ can be better written as below

parallel ‘date > start.$today.{};python {}_perf_stats_v3.py &>> /vin/{}.$today.log; date > end.$today.{}’ ::: sym fe be dg tp

To install GNU/Parallel on Debian / Debian based OS

#apt-get install parallel

 

Advertisements

2 thoughts on “GNU/Parallel – Run multiple commands simultaneously

  1. oletange

    Shorter:

    parallel ‘date > start.$today.{1};python {2}_perf_stats_v3.py &>> /vin/{1}.$today.log; date > end.$today.{1}’ ::: array fe be diskgrp tp :::+ sym fe be dg tp

    If you can rename ‘array’ and ‘diskgrp’:

    parallel ‘date > start.$today.{};python {}_perf_stats_v3.py &>> /vin/{}.$today.log; date > end.$today.{}’ ::: sym fe be dg tp

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s