Tailing Multiple Log Files with Twisted

In my last entry, I described a mechanism to follow a growing log file using the Twisted framework. Now, what if you wanted to follow multiple log files simultaneously?

You could try something like this:

#!/usr/bin/env python  

from twisted.internet import reactor  
from Support.followtail import FollowTail  

def lineReceived_file1( line ):  
  print "FILE1: " + line  

def lineReceived_file2( line ):  
  print "FILE2: " + line  

if __name__ == "__main__":  
  tailer1 = FollowTail( "/home/mohit/var/log/access-log" )  
  tailer2 = FollowTail( "/home/mohit/var/log/error-log" )  

  tailer1.lineReceived = lineReceived_file1  
  tailer2.lineReceived = lineReceived_file2  

  tailer1.start()  
  tailer2.start()  

  reactor.run()

Sure, that works fine. But there's a scaling issue above. What if you wanted to follow an unknown number of files, and track which file each line came from? Since a function callback is required for each file, it's quite difficult to use FollowTail for more "dynamic" applications.

Here's a simple solution: Subclass FollowTail, and override lineReceived to have it supply the filename along with the line.

#!/usr/bin/env python  

from twisted.internet import reactor  
from Support.followtail import FollowTail  

import sys  

class MultiFollowTail( FollowTail ):  
  def lineReceived( self, line ):  
    self.onFileLine( self.filename, line )  

  def onFileLine( self, filename, line ):  
    pass  

def lineReceived( filename, line ):  
  print filename + ": " + line  

if __name__ == "__main__":  
  for arg in sys.argv[1:]:  
    print "Monitoring: " + arg  
    tailer = MultiFollowTail( arg )  
    tailer.onFileLine = lineReceived  
    tailer.start()  

  if len( sys.argv ) > 1:    
    reactor.run()

Above, we created a new MultiFollowTail to do just that. Since the filename is included with each callback, we only need a single function to monitor multiple files.

And there you have it, a simple easy-to-use mechanism to monitor multiple logfiles with Python.