create real-time server monitoring app with angularjs and nodejs

In this Tutorial, we implement the server application. This application should be able to monitor files and communicate with multiple clients at the same time. These are the perfect requirements to use an event-driven, nonblocking I/O application platform such as Node.js. The fact that we can also code the server application in JavaScript is another huge advantage. Additionally, the built-in package manager npm provides a variety of useful and easy to use packages.

Preview of our app

 

Setting up a Node.js application

First, we need to install Node.js on our development machine. We can download and install the latest binaries from http://nodejs.org/. Node.js automatically installs its npm package manager, which we will use to install all the required packages for the server application.

To create a Node.js application, we will first create a package.json file, a file that contains all metainformation about the application (such as name, version number, and dependencies):

{
  "name": "webserver-monitor",
  "version": "0.0.1",
  "description": "Webserver Monitor Application",
  "dependencies": {
  }

}

For now, we leave the dependencies section empty and add the dependencies automatically when we install packages via npm and the --save flag.

Setting up a web server that serves static files

In this book, we will use the excellent Node.js web framework Express and the serve-static package to kick off a web server in under 10 lines of code. I will not go into the details about Express, but we will see that it’s very easy and straightforward to use for our purpose.

So, let’s add both the packages to our application. To do so, we open the terminal in the root directory of the project and execute the following commands in the terminal:

npm install --save express
npm install --save serve-static

These commands will add both the packages and their current versions as dependencies to the package.json file and download and install them in the npm_packages directory. The native Node.js function require(), which imports packages and modules, will automatically look for third-party libraries in this folder.

We can now implement the web server, so we create a server.js file:

/* server.js */var app = require('express')();
var http = require('http').Server(app);
var serveStatic = require('serve-static');

// Serve all files of the root directory
app.use(serveStatic('.', {'index': ['index.html']}));

// Listen on port 3000
http.listen(3000, function(){
  console.log('listening on 0.0.0.0:3000');
});

I will quickly guide you through the steps in the preceding code. First, we load the express package and initialize it by calling (). It returns the app reference for the web framework. In the second line, we load the internal http module and use the .Server() method to handle all requests with the Express framework. The third line loads the serve-static package. In the following step, we create an instance of the serve-static module and add it as a middleware to Express. We do this to tell Express to look for every URL for static files in the root directory and it’s subfolders. The last line finally starts the server application on port 3000.

We can now run the server by calling node server.js and navigate to http://localhost:3000/ to open the client application (we will do this a little bit later).

 

Adding server push with WebSockets

In the title of this tutorial, we referred to a real-time application, meaning changes in log files should be available on the client in real time/immediately. Thus, we cannot use standard http requests anymore because they are unidirectional from the client to the server. It’s not possible to notify the client about data changes.

Real-time applications are usually implemented with a bidirectional communication between the web server and the client; thus, the WebSockets technology is exactly what we need. WebSocket is a standardized implementation of a bidirectional TCP connection for the Web. We do not want to deal with low-level protocols or compatibility issues, so we will use the awesome Socket.IO library, a wrapper for WebSockets with an extra compatibility layer.

Note

Socket.IO uses long polling to simulate a server push behavior in older browsers.

Let’s add Socket.IO to our server application and package.json file:

npm install --save socket.io

We can now add the Socket.IO module to our server.js file:

/* server.js */var app = require('express')();
var http = require('http').Server(app);
var serveStatic = require('serve-static');
var io = require('socket.io')(http);

// Serve all files of the root directory
app.use(serveStatic('.', {'index': ['index.html']}));

// Wait for socket connection
io.on('connection', function(socket){
  // do while a client is connected

  socket.on('disconnect', function(){
    // do when client disconnects
  });
});

http.listen(3000, function(){
    console.log('listening on 0.0.0.0:3000');
});

Besides initializing the socket.io module with the http server object, we implement the two event listeners: .on('connection', callback) and .on('disconnect', callback) in the preceding code. These let us execute functions whenever a client connects to the server through WebSockets and lets us cleans up everything when the client disconnects again.

We already saw that Socket.IO waits for events that are triggered by clients (such as connecting or disconnecting clients). The same principle as the .on(type, callback) function can be used to listen for custom events that are triggered by the client, for example, to transfer data from the client to the server. In the callback function, the data that was sent by the client can be accessed with the first argument. To send data from the server to the client, we use the .emit(type, data) function. This function takes an event type and a message object data as arguments.

 

Reading logs and pushing them to the client

Now, it’s time to send some useful data through the WebSockets connection. Therefore, we add the native file system module, fs, to the application in order to read a file and push its content to the client:

var fs = require('fs');
var app = require('express')();
...
// Wait for socket connection
io.on('connection', function(socket){

  // Send the content of a file to the client
  var sendFile = function(name, path) {
    // Read the file
    fs.readFile(path, 'utf8', function (err, data) {
      // Emit the content of the file
      io.emit(name, data);
    });
  };

  // Wait for events on socket
  socket.on('watch', function(obj){
    sendFile(obj.name, obj.path);
  });

  socket.on('disconnect', function(){
    // do when client disconnects
  });
});

In the preceding example, we implement the sendFile()function inside the connection handler. In this function, we call the readFile() function. This function reads a file asynchronously and—once it is finished—pushes the content to the client via .emit() with an event type of the filename. Then, we set up a listener for an event watch whose message object should contain the name and path of a file and return the content of the file.

Now, we can implement a simple client that emits a watch event with the name and path of a log file and listens to an event with the name of the file. This client will look like the following code:

/* example/of/a/client.js */socket.emit('watch', {
  name: 'nginx.error',
  path: '/var/log/nginx/error.log'
});

socket.on('nginx.error', function(data){
  console.log("Received: " + data);
});

We will use a very similar implementation later for the client. For now, let’s continue with the final step.

 

 

Watching files for changes

We want to add a file watcher for every file requested by a client. This watcher should detect file changes and automatically push them to the client. Also, once a client disconnects, we need to clean up and remove all file watchers. To watch files for changes, we will use the asynchronous watchFile() function from the fs module:

/* server.js */var fs = require('fs');
var app = require('express')();
var http = require('http').Server(app);
var serveStatic = require('serve-static');
var io = require('socket.io')(http);

// Serve all files from the root directory
app.use(serveStatic('.', {'index': ['index.html']}));

// Wait for socket connection
io.on('connection', function(socket){

  var watchers = [];

  // Send the content of a file to the client
  var sendFile = function(name, path) {
    // Read the file
    fs.readFile(path, 'utf8', function (err, data) {
      // Emit the content of the file
      io.emit(name, data);
    });
  };

  // Wait for events on socket
  socket.on('watch', function(obj){

    if (!watchers.hasOwnProperty(obj.name)){

       console.log("Watching " + obj.name);
       watchers[obj.name] = obj;
       sendFile(obj.name, obj.path);

       // Watch the file for changes
       fs.watchFile(obj.path, function (curr, prev) {

        sendFile(obj.name, obj.path);
      });
    }
  });

  socket.on('disconnect', function(){
    watchers.forEach(function(obj) {
      fs.unwatchFile(obj.path);
    });
  });
});

http.listen(3000, function(){
  console.log('listening on 0.0.0.0:3000');
});

In the preceding code, we add a watchers array that contains all the current file watchers. This makes it easy to clean up and unwatch all the files in the disconnect handler via unwatchFile() once the connection is closed. In the connection handler, we add the watchFile() function. This function pushes the content of a file to the clients once it’s changed.

These few lines are all the magic that we need to monitor files and push them to the client when they are updated, pretty cool! Also, keep in mind that we completely neglected proper error handling in this simple example.

Finally, we run the server via the node server.js command and open http://localhost:3000/ for the client application.

Bootstrapping a template with angularjs and Socket.IO

Let’s create a HTML page for our client application; we need to load the JavaScript libraries (D3.js, angularjs, Socket.IO, the CSS layout Bootstrap, and all application files). Due to the usage of Socket.IO on the server side, it can be referenced on the client side with the /socket.io/socket.io.js pseudo location; all other third-party libraries are loaded from the bower_components directory.

We create the index.html page in the root directory of the project, add all libraries, and set up a very simple Bootstrap layout:

<!-- index.html -->
<html ng-app="myApp">
  <head>
    <!-- Include 3rd party libraries -->
    <script src="bower_components/d3/d3.js" charset="UTF-8"></script>
    <script src="bower_components/angular/angular.js" charset="UTF-8"></script>

    <!-- Include Socket.io -->
    <script src="/socket.io/socket.io.js"></script>

    <!-- Include the application files -->
    <script src="src/app.js"></script>
    <link href="src/app.css" rel="stylesheet">

    <!-- Include Bootstrap -->
    <link href="bower_components/bootstrap/dist/css/bootstrap.css" rel="stylesheet">

    <!-- Include the files of the chart component -->
    <script src="src/chart.js"></script>
    <link href="src/chart.css" rel="stylesheet">

  </head>
  <body ng-controller="MainCtrl">

    <div class="container">

      <nav class="navbar navbar-default">
        <!-- header goes here -->
      </nav>

      <div class="row">
        <!-- visualization goes here -->
      </div>
    </div>
  </body>
</html>

 

 

Using Socket.IO with angularjs

As we did with D3.js, we want to integrate Socket.IO properly into the client application. In other words, encapsulate it as a service and make it injectable. Therefore, we create a new factory for Socket.IO in the app.js file:

/* src/app.js */app.factory('socket', function () {
  var socketio = io.connect();
  return socketio;
});

In our example, we will use the .on() method to listen for events propagated from the server and the .emit() method to propagate events to the server. To inform angularjs about changes on the scope (outside of the angularjs application), we need to call $scope.$apply() to trigger a digest circle that updates all scope variables. Let’s write a wrapper for the .on() and .emit() functions that automatically update $rootScope. and thereby all scope variables of the application:

/* src/app.js */angular.module('myApp', ['myChart'])
// Socket.IO Wrapper
.factory('socket', ["$rootScope",
  function($rootScope) {
    var socketio = io.connect();
    return {
      on: function (e, callback) {
        socketio.on(e, function() {
          var args = arguments;
          $rootScope.$apply(function() {
            callback.apply(socketio, args);
          });
        });
      },
      emit: function (e, data, callback) {
        socketio.emit(e, data, function() {
          var args = arguments;
          $rootScope.$apply(function() {
            if (callback) {
              callback.apply(socketio, args);
            }
          });
        });
      }
    };
  }
])

The preceding implementation checks and updates the state of $rootScope on every callback of the .on() and .emit() function automatically.

Now, we can inject Socket.IO into the controller and send and receive data; let’s try it:

/* src/app.js */...
.controller('MainCtrl', ["$scope", "socket",
  function ($scope, socket) {
    $scope.logs = [{
      name: 'apache.access',
      path: 'var/log/apache/access.log'
    }];

    angular.forEach($scope.logs, function(log){

      socket.emit('watch', {
        name: log.name,
        path: log.path
      });

      socket.on(log.name, function(data){
        console.log("Received: " + data);

        // Now we can process the data
      });
    });
  }
]);

Although my browser has to struggle a little to display all the content from the Apache access log, we see that it works. This means that we receive the string of the correct data log from the server if Apache is running and the access log is updated; also, the file is reloaded. Perfect. Now, we can already think of processing the log file. Keep in mind that in a more advanced scenario, we will just transfer the small changes of the log files instead of 5 MB, of logs. It’s worth mentioning that you should implement security mechanism for the HTTP connection and for the WebSockets connection as well.

 

Processing log files

Before we can process and display all our log files, we need to organize them and the parsing formats in the main controller of the application.

Let’s create an array of logs in the controller of our application and add the processor expressions to each log type. Don’t worry if the parser and map attributes seem unfamiliar to you; I will explain them right after this page:

/* src/app.js */...
$scope.logs = [
{
  name: 'apache.access',
  path: 'var/log/apache/access.log',
  parser: {
    line: "\n",
    word: /[-"]/gi,
    rem: /["\[\]]/gi
  },
  map: function(d) {
    var format = d3.time.format("%d/%b/%Y:%H:%M:%S %Z");
    return {
      ip: d[0], time: +format.parse(d[2]), request: d[3], status: d[4], agent: d[8]
    }
  },
  data: []
},
{
  name: 'mysql.slow-queries',
  path: 'var/log/mysql/slow-queries.log',
  parser: {
    line: /# Time:/,
    word: /\n/gi,
    rem: /[#"\[\]]/gi
  },
  map: function(d) {
    var format = d3.time.format("%y%m%d %H:%M:%S");
    return {
      time: +format.parse(d[0]), host: d[1], query: d[2]
    }
  },
  data: []
}
...
];

In the preceding code, we see that this is a very clean way to structure our log files and specify the format to parse them. The only thing missing is to actually fill the data attributes with data and change them if the log data changes. However, this is no problem with Socket.IO and our previously developed monitor server. We simply have to add watchers for every log file:

/* src/app.js */angular.forEach($scope.logs, function(log){

  socket.emit('watch', {
    name: log.name,
    path: log.path
  });

  socket.on(log.name, function(data){
    console.log("Received: " + log.name);

    // Now we can really process all the data
  });
});

In the preceding code, we register every log file in the monitor server via the watch event; therefore, we automatically receive real-time updated data. Thanks to the watcher on the data attribute of the chart directive, the chart will be redrawn automatically when the data is updated. Now, I want to show how to process these files with the tools that we implemented in the previous tutorials with two different log files. The goal is to generate a grouped array of entry objects from a big string of log entries. Let’s recall the StringParser and the Classifier services that we wrote in tutorial 5, Loading and Parsing Data, and apply them in this example to process the log files:

/* src/app.js */...
socket.on(log.name, function(data){

  // The data log as string
  var responseDataStr = data;

  // 1:
  // Parse string to an array of datum arrays
  var parsed = StringParser(responseDataStr, log.parser.line, log.parser.word, log.parser.rem);

  // 2:
  // Map each datum array to object
  var mapped = parsed.map(log.map);

  // 3:
  // Filter the data
  var filtered = mapped.filter(function(d){
    return !isNaN(d.time);
  });

  // 4:
  // Group the dataset by time
  var grouped = Classifier(filtered, function(d) {
    var coeff = 1000 * 60 * $scope.groupByMinutes;
    return Math.round(d.time / coeff) * coeff;
  });

  // Use the grouped data for the chart
  log.data = grouped;
});

 

 

Let’s view the preceding code step by step:

  1. We parse the log string into an array of lines where every line contains an array of strings. This means that we need to find a separator that splits the lines and a separator that splits a line into segments.
  2. We map the array of segments from each line to an object. This helps us to identify the different parts of the log message (such as date, error message, ip address, and so on). We also convert the time string to a timestamp of a JavaScript Date object.
  3. We discard all rows that don’t have a valid time attribute.
  4. We group the data logs by an interval of minutes. From the preceding points, point 1 is the most difficult point; therefore, I will explain it systematically with two example logs.

First, we will use a MySQL slow query log from the var/log/mysql directory with the following structure:

# Time: 141129 17:24:37
# User@Host: root[root] @ server.com [172.14.26.38]
# Query_time: 2.240000  Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 2560674
SET timestamp=1334841877;
SELECT  ...;
# Time: 141129 17:24:39
# User@Host: root[root] @ server.com [172.14.26.38]
# Query_time: 1.896000  Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 2560674
SET timestamp=1334841879;
SELECT  ...;

First, we can split the log string using the /# Time:/regular expression to generate an array of log entries:

Array[
 '141129 17:24:37
  # User@Host: root[root] @ server.com [172.14.26.38]
  # Query_time: 2.240000  Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 2560674
  SET timestamp=1334841877;
  SELECT  ...;',
 '141129 17:24:39
  # User@Host: root[root] @ server.com [172.14.26.38]
  # Query_time: 1.896000  Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 2560674
  SET timestamp=1334841879;
  SELECT  ...;'
];

Then, using the newline symbol, every single log entry can be split via the /\n/ regular expression into single segments:

Array[
  Array['141129 17:24:37',
  '# User@Host: root[root] @ server.com [172.14.26.38]',
  '# Query_time: 2.240000  Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 2560674',
  'SET timestamp=1334841877;',
  'SELECT  ...;'],
  Array['141129 17:24:39',
  '# User@Host: root[root] @ server.com [172.14.26.38]',
  '# Query_time: 1.896000  Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 2560674',
  'SET timestamp=1334841879;',
  'SELECT  ...;']
];

To make the dataset more readable, we can also remove some characters (such as # from the log entries). As a last step, we need to convert the DateTime string to a JavaScript Date Object. We can do this here by using the %y%m%d %H:%M:%S D3.js formatter. Now, we have a beautiful dataset with valid JavaScript dates. We can easily display it in a chart, for example, as a histogram.

Let’s try it once more and parse a NginX error log with the following structure:

2014/11/29 11:13:53 [alert] 6976#8040: could not respawn worker
2014/11/29 11:14:24 [emerg] 6488#2952: unknown directive "concat" in /etc/nginx/conf/nginx.conf:76

Splitting the lines is very easy because every log entry starts on a new line; thus, we can use the /\n/ regular expression to split them:

Array[
  '2014/11/29 11:13:53 [alert] 6976#8040: could not respawn worker',
  '2014/11/29 11:14:24 [emerg] 6488#2952: unknown directive "concat" in /etc/nginx/conf/nginx.conf:76'
];

In the next step, we will divide every line into segments by splitting it with the [ and ] characters with the /\[|\]/ regular expression:

Array[
  Array['2014/11/29 11:13:53', 'alert', '6976#8040: could not respawn worker'],
  Array['2014/11/29 11:14:24', 'emerg', '6488#2952: unknown directive "concat" in /etc/nginx/conf/nginx.conf:76']
];

Again, as a last step, we need to convert the date string into a JavaScript Date object; this can be done with the %Y/%m/%d %H:%M:%S formatter in this example.

 

 

 

One last step needs to be done to finally see the charts of the log files, that is, to add the chart directives to the index.html page:

<!-- index.html -->
<div class="col-lg-6" ng-repeat="log in logs">
  <h3>{{ log.name }}</h3>

  <bar-chart class="chart blue" data="log.data" start-date="time.startDateTime" end-date="time.endDateTime" cur-date="time.currentDateTime">
  </bar-chart>
</div>

To better understand the cursor position and current zooming level, we will output the current value of the cursor and the first and last date from the current filter to the navigation bar:

<nav class="navbar navbar-default">
  <div class="navbar-text">
    <span>Date/Time Filter:</span>
    <span>{{ time.startDateTime | date : 'dd.MM.yyyy HH:mm' }}</span> -
    <span>{{ time.endDateTime | date : 'dd.MM.yyyy HH:mm' }}</span>
  </div>
  <div class="navbar-text">
    <i>Current Date/Time: {{ time.currentDateTime | date : 'dd.MM.yyyy HH:mm' }}</i>
  </div>
</nav>

In the preceding code, we use the built-in angularjs date filter to create a more readable output of the DateTime object.

If we now run the server and open the application, we will see four charts that are automatically updating in real time. We can see that zooming or panning in one chart also affects the current zoom and panning level in other charts. The reason for this is that all chart directives use the same reference for the startDate and endDate filter attributes as well as the curDate attribute. Therefore, the cursor moves and highlights in all charts simultaneously because of angularjs‘ two-way-binding. Pretty neat, isn’t it?

We used Bootstrap not only for it’s nice visual template, but also for the built-in grid system. As a logical step, we want to make our charts responsive. This can be easily achieved by watching the window size in the chart directive and redrawing each chart when the window size changes:

/* src/chart.js → Chart Directive */// Watch the window for resizing
angular.element($window).bind('resize', function(){

  // Set the width of the chart to the width of the parent element
  chart.width(element[0].parentElement.offsetWidth);
  // Redraw the chart
  chart.redraw();
});

Nice, now the charts adapt automatically according to the column size of the grid.

Let’s open the application and take a look. The filters and cursors now play together throughout the whole application. Thanks to angularjs, a web designer with no JavaScript knowledge can easily arrange the charts and write the HTML code for this page:

Deven Rathore

Deven is an Entrepreneur, and Full-stack developer, Constantly learning and experiencing new things. He currently runs CodeSource.io and Dunebook.com.

Published by
Deven Rathore
Tags: angularjs

Recent Posts

3 Ways to Get the Most Out of Your University’s Virtual Computer Lab

IT is more important than ever in the world of higher education, and yet with…

20 hours ago

Top Tips for Learning Java Programming

If you’re here for the top tips, we assume you’re ahead of the “how to…

1 day ago

Neural Networks for Creating Blog Texts

The world is progressing at unprecedented rates at the current moment, especially in terms of…

2 days ago

Top 20 Opensource Python Tkinter Projects

This article will highlight the Top 20 Opensource Python Tkinter Projects which we believe will…

4 days ago

Beginners guide to Sneaker Proxies

With their numerous applications in streamlining the data flow, securing both the servers and the…

1 week ago

Top 20 Node.js dashboard templates

In this article, We will be looking at some of the top Node.js dashboard templates.…

1 week ago