Scripting

Monitor Log for pattern match and trigger an action

Linux




For real time log monitoring in which an action should be triggered, “awk program execution” can be used. An AWK program consists of a sequence of pattern-action statements and optional function definitions.

pattern { action statements }

From man page:

For each record in the input, gawk tests to see if it matches any pattern in the AWK program. For each pattern that the record matches, the associated action is executed. The patterns are tested in the order they occur in the program.

Examples to utilize this on a command line would look like this:

~$ tail -fn0 logfile | awk '/pattern/ { print }'

where print will ouput the pattern the console. print can be replaced with any command, even to call another shell script. If the command contains spaces, use double quotes, for example:

~$ tail -fn0 logfile | awk '/pattern/ { "echo ${VARIABLE}" }'

This method works great if there is a need for continuous log monitoring and action triggering any time pattern is found. But what to do if there is a need to run this inside a shell script? I attempted to do varous things from using the action statement “exit”, to command concatenation like “print && exit“ or “print; exit“, but every time tail would not recognize the pipe had been broken and therefore would not exit. This precluded the bash script from moving onto the next block of code to execute once pattern was found.

So how do we do this?

How to terminate tail -f inside a shell script

To reach a solution, I began by observing the differences in running the following tail commands with the action statement set to “exit” via command line, and from a shell script:

tail -fn0 server.log | awk '/\(JBoss Shutdown Hook\) Shutdown complete/ { exit }'

ps -ef output when running from command line:
reese 5922 4181 0 13:04 pts/1 00:00:00 tail -fn0 server.log

ps -ef output when running from shell script:

reese    5591  4100  0 12:59 pts/0    00:00:00 /bin/bash ./deployEar.sh  
reese    5641  5591  0 12:59 pts/0    00:00:00 tail -fn0 /opt/Jboss_5.1.2/server/JbossServer1/log/server.log  
reese    5642  5591  0 12:59 pts/0    00:00:00 awk /\(JBoss Shutdown Hook\) Shutdown complete/ { exit }

Interesting that awk doesn’t show up as a separate process when running from command line. This must be why tail exits once the pattern is logged. However, from the shell script, only the awk process exits when “exit” is executed, leaving tail to run and therefore hanging up the shell script from further code execution.

To solve this issue, the tail man page has the following option:

--pid=PID  
              with -f, terminate after process ID, PID dies

Awesome! So let’s use this to tell it to terminate tail -f after awk dies, like this!

tail -fn0 --pid=$(pidof awk) server.log | awk '/\(JBoss Shutdown Hook\) Shutdown complete/ { exit }'

Works like a charm. I hope this helps.

An example of what I used this for was to monitor an application log for a shutdown trigger. If stopped, sleep 2 seconds then start the application.

${BINPATH}/application.sh stop >/dev/null 2>&1
	tail -fn0 --pid=$(pidof awk) ${APPLOG} | awk '/\(JBoss Shutdown Hook\) Shutdown complete/ { exit }'
	tac ${APPLOG} | grep -m1 "Shutdown complete" | awk '{ print $1" "$2" "$5" "$6" "$7" "$8" "$9 }'

	sleep 2

	echo -e "\nStarting ${Target} on ${HOSTNAME}.  This will take a few minutes..."
	${BINPATH}/application.sh start >/dev/null 2>&1
	tail -fn0 --pid=$(pidof awk) ${APPLOG} | awk '/Started in/ { exit }'
	tac ${APPLOG} | grep -m1 "Started in" | awk '{ print $1" "$2" "$12" "$13" "$14 }'

If using as an alias within .bash_profile, escape $ where appropriate. For example:

alias applog="tail -fn0 --pid=\$(pidof awk) ${APPLOG} | awk '/Started in/ { exit }'; tac ${APPLOG} | grep -m1 'Started in' | awk '{ print \$1\" \"\$2\" \"\$12\" \"\$13\" \"\$14 }'"

or wrap it into a function within .bash_profile:

function app()
{
  ENV=$(hostname | awk 'BEGIN {FS="scm"}{print $1}')
  ENV=$(echo $ENV | awk '{ print toupper($0)}')

  if [ "$1" = "sct" ]; then
  	#LOG="/scmlogs/${ENV}/${HOSTNAME}/SCTServer1/log/server.log"
  	LOG="/opt/Jboss_5.1.2/server/SCTServer1/log/server.log"
	CMD="/opt/Jboss_5.1.2/bin/sct.sh"
  elif [ "$1" = "clm" ]; then
  	#LOG="/scmlogs/${ENV}/${HOSTNAME}/CLMServer1/log/server.log"
	LOG="/opt/Jboss_5.1.2/server/CLMServer1/log/server.log"
	CMD="/opt/Jboss_5.1.2/bin/clm.sh"
  elif [ "$1" = "sctbatch" ]; then
  	#LOG="/scmlogs/${ENV}/${HOSTNAME}/CLMServer1/log/server.log"
	LOG="/opt/Jboss_5.1.2/server/SCTBatchServer/log/server.log"
	CMD="/opt/Jboss_5.1.2/bin/sctbatch.sh"
  else
	echo "Usage: app [sct|clm|sctbatch] [start|stop|status] [tail]"
	return 1;
  fi

  case $2 in
    start)
	  eval ${CMD} start
	  if [ ! -z $3 ]; then
	    tail -f ${LOG}
	  else
	    tail -fn0 --pid=$(pidof awk) ${LOG} | awk '/Started in/ { exit }'
	  fi
	  tac ${LOG} | grep -m1 'Started in' | awk '{ print $1" "$2" "$12" "$13" "$14 }'
	  ;;
     stop)
	  eval ${CMD} stop
	  if [ ! -z $3 ]; then
	    tail -f ${LOG}
	  else
	    tail -fn0 --pid=$(pidof awk) ${LOG} | awk '/\(JBoss Shutdown Hook\) Shutdown complete/ { exit }'
	  fi
	  echo "Shutdown Complete"
	  ;;
   status)
	  eval ${CMD} status
	  ;;
        *)
	  echo "Usage: app [sct|clm|sctbatch] [start|stop|status] [tail]"
	  ;;
  esac
}

END

Share