Processing log files

Before we can process and display all our log files, we need to organize them and the parsing formats in the main controller of the application.

Let’s create an array of logs in the controller of our application and add the processor expressions to each log type. Don’t worry if the parser and map attributes seem unfamiliar to you; I will explain them right after this page:

/* src/app.js */
...
$scope.logs = [
{
  name: 'apache.access',
  path: 'var/log/apache/access.log',
  parser: {
    line: "\n",
    word: /[-"]/gi,
    rem: /["\[\]]/gi
  },
  map: function(d) {
    var format = d3.time.format("%d/%b/%Y:%H:%M:%S %Z");
    return {
      ip: d[0], time: +format.parse(d[2]), request: d[3], status: d[4], agent: d[8]
    }
  },
  data: []
},
{
  name: 'mysql.slow-queries',
  path: 'var/log/mysql/slow-queries.log',
  parser: {
    line: /# Time:/,
    word: /\n/gi,
    rem: /[#"\[\]]/gi
  },
  map: function(d) {
    var format = d3.time.format("%y%m%d %H:%M:%S");
    return {
      time: +format.parse(d[0]), host: d[1], query: d[2]
    }
  },
  data: []
}
...
];

In the preceding code, we see that this is a very clean way to structure our log files and specify the format to parse them. The only thing missing is to actually fill the data attributes with data and change them if the log data changes. However, this is no problem with Socket.IO and our previously developed monitor server. We simply have to add watchers for every log file:

/* src/app.js */
angular.forEach($scope.logs, function(log){

  socket.emit('watch', {
    name: log.name,
    path: log.path
  });

  socket.on(log.name, function(data){
    console.log("Received: " + log.name);

    // Now we can really process all the data
  });
});

In the preceding code, we register every log file in the monitor server via the watch event; therefore, we automatically receive real-time updated data. Thanks to the watcher on the data attribute of the chart directive, the chart will be redrawn automatically when the data is updated. Now, I want to show how to process these files with the tools that we implemented in the previous tutorials with two different log files. The goal is to generate a grouped array of entry objects from a big string of log entries. Let’s recall the StringParser and the Classifier services that we wrote in tutorial 5, Loading and Parsing Data, and apply them in this example to process the log files:

/* src/app.js */
...
socket.on(log.name, function(data){

  // The data log as string
  var responseDataStr = data;

  // 1:
  // Parse string to an array of datum arrays
  var parsed = StringParser(responseDataStr, log.parser.line, log.parser.word, log.parser.rem);

  // 2:
  // Map each datum array to object
  var mapped = parsed.map(log.map);

  // 3:
  // Filter the data
  var filtered = mapped.filter(function(d){
    return !isNaN(d.time);
  });

  // 4:
  // Group the dataset by time
  var grouped = Classifier(filtered, function(d) {
    var coeff = 1000 * 60 * $scope.groupByMinutes;
    return Math.round(d.time / coeff) * coeff;
  });

  // Use the grouped data for the chart
  log.data = grouped;
});