Error Emitted => SELF_SIGNED_CERT_IN_CHAIN

While using elasticdump to dump an elasticsearch index to a JSON file, I got the following error message:

Error Emitted => SELF_SIGNED_CERT_IN_CHAIN

This basically means that we are accessing elasticsearch using an HTTPS connection and the certificate it gets is self-signed (and thus cannot be verified).

Googling for this issue I mostly found questions from people who get this error message when using NPM. But all answers were basically aimed at making it work in NPM by disabling the strict SSL rule in the NPM config (npm config set strict-ssl false), setting an HTTP URL as HTTPS proxy (npm config set https-proxy “http://:8080”), using an HTTP URL for the registry (npm config set registry=”http://registry.npmjs.org/”) or having npm use known registrars (npm config set ca=””).

But none of this could help me since I do not have an issue using NPM but using another Node application. The only think I eventually found was to set an environment variable so that NodeJS would not reject self signed certificates:

export NODE_TLS_REJECT_UNAUTHORIZED=0

After that elasticdump was working fine. But keep in mind that using this method for temporarily using a NodejS software is fine, but using this setting in production is not a good idea.

C#: Understanding CLOSE_WAIT and FIN_WAIT_2

Here’s a short C# program which can be used to better understand what the TCP states CLOSE_WAIT and FIN_WAIT_2 are and why you sometimes see connections stuck in these states:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;

namespace TcpTester
{
    internal static class Program
    {
        private const int Port = 15000;
        private const string Hostname = "127.0.0.1";

        private static void Main(string[] args)
        {
            if (args.Length > 0 && args[0] == "client")
            {
                // Started in client mode
                var tcpClient = new TcpClient();
                tcpClient.Connect(Hostname, Port);
                Console.WriteLine("Connected to {0}:{1}", Hostname, Port);
                PrintState();
                Console.WriteLine("Press any key to close the connection from this side.");
                Console.ReadKey();
                tcpClient.Close();
                PrintState();
            }
            else
            {
                // Started in server mode
                var tcpListener = new TcpListener(IPAddress.Parse(Hostname), Port);
                tcpListener.Start();
                Console.WriteLine("Listening on {0}:{1}", Hostname, Port);
                TcpClient tcpClient = tcpListener.AcceptTcpClient();
                tcpListener.Stop();
                Console.WriteLine("Client connected on {0}:{1}", Hostname, Port);
                PrintState();
                Console.WriteLine("Press any key to close the connection from this side.");
                Console.ReadKey();
                tcpClient.Close();
                PrintState();
            }
        }

        private static void PrintState()
        {
            IEnumerable<TcpConnectionInformation> activeTcpConnections =
                IPGlobalProperties.GetIPGlobalProperties().GetActiveTcpConnections()
                    .Where(c => c.LocalEndPoint.Port == Port || c.RemoteEndPoint.Port == Port);
            foreach (TcpConnectionInformation connection in activeTcpConnections)
            {
                Console.WriteLine("{0} {1} {2}", connection.LocalEndPoint, connection.RemoteEndPoint, connection.State);
            }
        }
    }
}

You can start the program without parameters to start a server and with the parameter “client” to start a client (guess it was kind of obvious…).

The server listens to 127.0.01:15000 and the clients connects to it. First start the server. The following will be written to the console:

Listening on 127.0.0.1:15000

Then start the client in another window. The following will appear in the client window:

Connected to 127.0.0.1:15000
127.0.0.1:15000 127.0.0.1:57663 Established
127.0.0.1:57663 127.0.0.1:15000 Established

This tells you that the client is connected from port 57663 (this will change every time you run this test) to port 15000 (where the server is listening).

In the server window, you will see that it got a client connection and the same information regarding port and connection states.

Then press any key on the server console and the following will be displayed:

127.0.0.1:15000 127.0.0.1:57663 FinWait2
127.0.0.1:57663 127.0.0.1:15000 CloseWait

So once the server closed the connection, the connection on the server side went to FIN_WAIT_2 and the one on the client side went to CLOSE_WAIT.

Then press any key in the client console to get the following displayed:

127.0.0.1:15000 127.0.0.1:57663 TimeWait

The connection will stay in TIME_WAIT state for some time. If you really wait a long time before pressing a key in the client console, this last line will not be displayed at all.

So, this should make it easier to understand what the TCP states CLOSE_WAIT and FIN_WAIT_2 are: When the connection has been closed locally but not yet remotely, the local connection is the state FIN_WAIT and the remote one in CLOSE_WAIT.

For more details about the different TCP states, please refer to TCP: About FIN_WAIT_2, TIME_WAIT and CLOSE_WAIT.

OWIN: Serving static files from an external directory

I am working on an application with a self-hosted OWIN server where the UI is running in an embedded browser and the backend part of the application is implemented using WebApi. When I generate files in the backend, I store them in a subfolder of the application (called “uploads”) configure my application so that files from this folder are served statically:

appBuilder.UseStaticFiles("/uploads");

It all worked fine until an installer was created for the application which installed it in c:\Program Files. Unfortunately, the application is not able to write to the uploads subfolder, so it broke this type of functionality. Obviously, the solution is to be a good Windows citizen and store files created by the application in the LocalAppData directory e.g. instead of using:

Path.Combine(AppDomain.CurrentDomain.SetupInformation.ApplicationBase, "uploads")

use:

Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), @"MyCompany\uploads")

This solves the issue writing to the folder. All that is now missing is to tell OWIN to serve files from this folder whether the “/uploads” virtual path is accessed:

var staticFilesOptions = new StaticFileOptions();
staticFilesOptions.RequestPath = new PathString("/uploads");
var uploadPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), @"MyCompany\uploads");
Directory.CreateDirectory(uploadPath);
staticFilesOptions.FileSystem = new PhysicalFileSystem(uploadPath);
appBuilder.UseStaticFiles(staticFilesOptions);

Note that the folder needs to exist before you use UseStaticFiles hence the Directory.CreateDirectory call.

Asynchronously pre-loading scripts with AngularJS and RequireJS

As your web application grows, the number and size of JavaScript files you will have to load grows as well. If you are using AngularJS and RequireJS, you might well reach a level where the initial loading of such resources takes so long that you need to start looking into better ways to handle loading these dependencies.

Initial Loading

Your starting point when working with AngularJS is that everything is loaded in the beginning:

initial loading

The advantages of this approach are that it’s very easy to handle and switching to the second or third view is extremely fast as everything was already loaded upfront.

Lazy loading

As described in this article, lazy loading can help reducing the initial loading time by loading resources on demand when the users moves to another view:

lazy loading

So with this approach the time required to switch from the initial view to another view is increasing (because you now need to load some resources first) but the initial load time is decreasing.

Asynchronous pre-loading

In order to get a low initial loading time and not require any loading time for the second and third view, you need to implement some asynchronous solution which relies on the fact that files which have already been loaded will not be loaded again and that the user usually doesn’t immediately switch to the second or third view.

The delay introduced by the user could be because he needs to enter some credential in a login page. Or the initial view is some kind of dashboard and the user will first review all displayed information before going to more detailed views.

asynchronous preloading

When the initial view is displayed, we require scripts needed for the other two views to be loaded asynchronously. So while the user is interacting with the initial view, the scripts are loaded in the background and once the user activates one of the other views, the scripts will not be loaded again.

Asynchronous pre-loading with AngularJS and RequireJS

In order to load JavaScript files asynchronously in the background, we’ll need to use the async version of require:

require(["module_name"])

There are basically 3 places where you could trigger this:

  1. Immediately
  2. In the resolve function of your route definition
  3. In the require callback of your view

The problem with the first option is that since it triggers the asynchronous background loading of the files immediately, these load operations will compete with the loading of other resources you are waiting for. This means that it will use some of the bandwidth you need for the synchronous loading of files and it will also use HTTP connections which might cause the other load operations to have to wait.

If you are dynamically loading files for the new view in your resolve function, putting the code to asynchronously load files you will need in the future has the same effect. It’s just maybe not that bad because the files which are lazy loaded might be less or smaller than the ones which are required for all views and loaded upfront.

So the solution I went for is the third. Since the background loading is triggered once the required files for the view have been loaded, it has no impact on other load operations.

I’ve updated the pluggableViews provider I’ve already described in a previous post to have an additional optional parameter called preload. It’s basically a function called in the callback of the require function call during lazy loading of the view. The default value for this function is an empty function:

if (!viewConfig.preload) {
    viewConfig.preload = function () {
    };
}

And it is called in the callback of the require function:

$routeProvider.when(viewConfig.viewUrl, {
	templateUrl: viewConfig.templateUrl,
	controller: viewConfig.controller,
	resolve: {
		resolver: ['$q', '$timeout', function ($q, $timeout) {

			var deferred = $q.defer();
			if (angular.element("#" + viewConfig.cssId).length === 0) {
				var link = document.createElement('link');
				link.id = viewConfig.cssId;
				link.rel = "stylesheet";
				link.type = "text/css";
				link.href = viewConfig.cssUrl;
				angular.element('head').append(link);
			}
			require([viewConfig.requirejsName], function () {
				pluggableViews.registerModule(viewConfig.moduleName);
				$timeout(function () {
					deferred.resolve();
					viewConfig.preload();
				});
			});
			return deferred.promise;
		}]
	}
});

I can then use it this way:

$pluggableViewsProvider.registerView({
	ID: 'walls',
	moduleName: "cards.walls",
	requirejsConfig: {paths: {'walls': '../views/walls/walls'}},
	preload: function () {
		require(['reports'], function () {
			console.log("reports loaded");
		});
		require(['admin'], function () {
			console.log("admin loaded");
		});
	}
});

Conclusion

Of course, introducing lazy loading and asynchronous background pre-loading increases the complexity of your application. And it is not trivial to introduce once you already have a large application which dependencies are a mess because you never need to clean them before (since by default AngularJS causes all files to be loaded upfront).

But if you load lots of files (especially JavaScript libraries), then making sure that files are loaded when needed and are not loaded at a point in time where this would slow down the application, will definitely help making your application look thinner and faster (from a user perspective).

Blocking all BlazingFast IP address blocks (ranges)

Over the past few weeks, I’ve had some issues with my site sometimes not being available or loading very slowly. Checking on the server I could see a high number of Apache processes and a memory usage about 5GB higher than usual. Issuing a netstat I could see that there were many connections from the same IP address: 185.62.189.162.

A whois on this address shows that this IP address belongs to a hosting company in Kiev, Ukraine called BlazingFast. I first blocked this IP address using iptables:

/sbin/iptables -A INPUT -s 185.62.189.162  -j DROP

Since I have a monitoring script checking intrusion attempts and blocking IP addresses, I end up having lots of DROP rules in iptables. So once a week I clean them automatically. Usually hackers do not spend more than a week trying if they see that their traffic to my server is blocked anyway.

Here it was different. As soon as the rules where cleared, it started again with the exact same address. Of course, I immediately blocked this IP address again and sent an email to their abuse email address. But as expected never got an answer. Instead, the same thing happened again but coming from another similar IP address: 185.62.190.221. Whois shows that this address also belongs to the same Ukrainian hosting company.

So, since it was now clear that I’ll keep having problems with IP addresses belonging to this company, I decided to block all traffic coming for the IP ranges owned by them. First I checked what was their ASN on https://who.is/whois-ip/ip-address/185.62.190.221: AS60033. Then looked up their IP address blocks on https://ipinfo.io/AS60033.

Then all I had to do is use iptables to block traffic from these IP address blocks (and make sure that these rules stay in there):

/sbin/iptables -A INPUT -s 185.11.144.0/22 -j DROP
/sbin/iptables -A INPUT -s 185.11.145.0/24 -j DROP
/sbin/iptables -A INPUT -s 185.11.146.0/24 -j DROP
/sbin/iptables -A INPUT -s 185.11.147.0/24 -j DROP
/sbin/iptables -A INPUT -s 185.61.136.0/22 -j DROP
/sbin/iptables -A INPUT -s 185.62.188.0/23 -j DROP
/sbin/iptables -A INPUT -s 185.62.188.0/24 -j DROP
/sbin/iptables -A INPUT -s 185.62.189.0/24 -j DROP
/sbin/iptables -A INPUT -s 185.62.190.0/23 -j DROP
/sbin/iptables -A INPUT -s 185.62.190.0/24 -j DROP
/sbin/iptables -A INPUT -s 185.62.191.0/24 -j DROP
/sbin/iptables -A INPUT -s 188.209.48.0/23 -j DROP
/sbin/iptables -A INPUT -s 188.209.48.0/24 -j DROP
/sbin/iptables -A INPUT -s 188.209.49.0/24 -j DROP
/sbin/iptables -A INPUT -s 188.209.50.0/23 -j DROP
/sbin/iptables -A INPUT -s 188.209.52.0/23 -j DROP
/sbin/iptables -A INPUT -s 188.209.52.0/24 -j DROP
/sbin/iptables -A INPUT -s 188.209.53.0/24 -j DROP

So now the load on the server is fine again and unlike the past few weeks the hosted websites are always accessible and load fast.

It’s interesting to see that BlazingFast is advertizing with the DDOS protection service on hand and actually seem to have customers performing brute force attacks from their servers on the other. If you look up their ASN on the fail2ban reporting service, you will see that a few of their IP addresses are being blocked. So I am not the only one who’s been hit by this. Maybe they should not only focus on protecting their customers from DDOS attacks but should also prevent them from performing attacks.

This post on stackexchange also shows that it’s not something new but it looks like there were already attacks originating from one of their IP addresses in May. The answers to this post will also give you some alternative solutions to block them using the Apache .htaccess file, the Cisco firewall, Nginx, a Microsoft IIS Web Server rule, netsh ADVFirewall or CSF firewall.

I know it’s more difficult to identify attacks originating from one of your IP addresses than attacks targeting your network. As a hosting company, you definitely do not want to have to many false positive and block legitimate traffic created by your customers. But I’m still pretty mad having to waste so much time taking care of this kind of things…

Update 18/07/2015: I’ve ended up also blocking all IP blocks of the following companies: ISPsystem, cjsc and Lekosport-Kharkov LLC which were wasting my time and there’s trying to hack wp-login.php.

Update 20/07/2015: Today I’ve blocked additional IP blocks belonging to Kyivstar PJSC. Slowly I’m starting to think that I’ll have to block access to complete regions in order to be able to sleep at night without worrying…

About lazy loading AngularJS modules

I recently wrote a post about creating pluggable AngularJS views using lazy loading of Angular modules. After discussing this topic with colleagues, I realized I did provide a technical solution but not much background as to why it is useful, what should be lazy loaded and when does lazy loading make sense. Hence this post…

Does lazy loading always make sense?

First, as general rule, optimization should only be performed when you actually have a performance problem. Definitely not in a prophylactic way. This is a common error made by many software engineers and causing complexity to unnecessarily increase at the beginning of a project. If you identify a bottleneck while loading specific resources, you should start thinking about introducing lazy loading in order to speed up your app e.g. during initial loading.

But just lazy loading everything without first having a good reason to introduce it can very well result in worse performance. You need to understand that lazy loading could actually increase the overall cost of loading resources. By loading all resources upfront, you get a chance to combine your resources and reduce the number of HTTP requests to your server. By splitting the loading of resources, you then actually have less possibilities to reduce the overall resource consumption by combination. Especially when you are loading many small resources, the overhead of loading them individually increases. Since JavaScript code is relatively small compared to static assets like images, lazy loading could cause a higher request overhead.

Also loading resources on demand means that when the user of your application activates a view which has not yet been loaded, he first has to wait until the additional resources are lazy loaded and properly registered. When you load everything upfront, the user has a longer wait time on application load time but can immediately access all views in your application without delay.

So using lazy loading always means having a tradeoff between initial load time of the application and subsequent activation of parts of your application.

RequireJS vs. AngularJS Dependency Injection

RequireJS is currently the state of the art in modular script loader and asynchronous and lazy loading of JavaScript files. It is based upon the AMD (Asynchronous Module Definition) API. It handles loading your script files in the browser based on a description of dependencies between your modules.

This may sound similar to AngularJS dependency injection mechanism. But RequireJS and AngularJS DI work on two completely different levels. AngularJS DI handles runtime artifacts (AngularJS components). In order to work properly, AngularJS DI requires all of the JavaScript code to have been loaded and registered by the framework. It injects controllers, directives, factories, services, etc. which have been previously loaded.

Since there is no direct integration of RequireJS and AngularJS and AngularJS has a single initialization phase where the definition of modules is interpreted, when using both of them, RequireJS will need to first load the complete dependency tree before AngularJS DI can be used. This effectively means that your complete JavaScript code will need to be fetched on initial load.

So we cannot just lazy load all AngularJS modules JavaScript files with RequireJS after an AngularJS application has started because the AngularJS DI container wouldn’t handle the components created as a result of loading the JavaScript files. Instead, you need to manually handle all these components which have been loaded after the startup phase of your application.

What’s the point of lazy loading ?

Lazy loading components generally improves an application’s load time. If your web application takes many seconds to load all components ever required to interact with user, you will most probably experience a high bounce rate with users just giving up before they even get a chance to see how great your application is.

By lazy loading parts of your application, you can make sure that the views in your application which are used immediately by most users are available immediately. If a given user decides to use other (e.g. more advanced views), the application starts loading required components on demand.

Especially if you are building a web application which can be used on phone and tablets (which represents about 60% of the total web traffic now), you have to consider that most users will not have a 4G mobile connection and initial load times can become prohibitive in an environment where the download bandwidth is limited.

Loading time is a major contributing factor to page abandonment. The average user has no patience for a page to take too long to load. Slower page response time results in an increase in page abandonment. Nearly half of web users expect a site to load in 2 seconds or less, and they tend to abandon a site that isn’t loaded within 3 seconds.

So improving the initial load time of your web application is critical. And this is the main use case for lazy loading.

When and what to lazy load ?

As explained above lazy loading makes sense when you want to reduce the initial load times and are ready to accept that loading additional views might not be instantaneous. Ideally, at some point in time during the run time of your application, you would be able to determine that the user will need a specific view and load it asynchronously in the background. Unfortunately, this kind of smart lazy loading is very difficult to implement for two reasons. First, it is not so easy to predict that a user will need a specific view. Second, asynchronicity introduces additional problems which increase the complexity of your application.

This is why lazy loading is mostly implemented in such a way that when a user activates a view (or a sub-part) of your application which hasn’t yet been loaded, it is loaded on the fly before displaying it.

A loading mechanism can be implemented on different levels. Having a fine granular lazy loading mechanism, reduces the maximum wait time for the user when something is loaded. But the complexity of your application grows (potentially exponentially) as well, the more fine granular it is. Since our general rule is not to optimize upfront but only find a solution to problems you are actually facing, this requires a strategy along these lines:

  1. First load everything upfront (this is how most desktop applications work). If you face problems because of high initial load time, continue optimize. Otherwise you are done.
  2. Lazy load individual modules of the application. This is the way I handle lazy loading in the application I used as a basis for my previous post. If the load times are acceptable, then stop optimizing. “Acceptable” could either mean that the load times for all parts of the application are good enough or that the load times are good for 95% of the use cases and the user only has a longer wait time for rare use cases.
  3. Keep reducing the granularity of lazy loaded resources…
  4. Until the performance is acceptable.

AngularJS components vs. Module loading

If you google for AngularJS lazy loading, you will find many resources. Most of them teach you how to lazy load controllers, directives, etc. The difference between these approaches and i.e. the one described in my previous post is basically that when you just lazy load controllers and such, you have a single module which is initialized at the beginning and for which you register additional components on the fly. This approach has two drawbacks:

  1. You cannot organize your application in multiple AngularJS modules.
  2. This doesn’t work well for third-party party AngularJS modules.

Both of these drawbacks have the same root cause. Since AngularJS does not only rely on JavaScript files to be loaded but also need to have the modules properly registered in order to inject them using DI, if this step is missing because the files were loaded later on, new AngularJS modules will not be available.

That’s why you need to additionally handle the invoke queue, config blocks and run blocks when lazy loading new AngularJS modules.

Also note that whenever the term “module” is used in this context, it can mean two different things:

  1. AMD module as used in RequireJS. These modules just encapsulate a piece of code which has load dependencies to other modules. These are designed for asynchronous loading.
  2. AngularJS modules. These modules are basically containers for controllers, services, filters, directives…

When I reference modules in this article, I mean the second kind of modules.

Conclusion

I hope that with this article I made it clearer, why lazy loading AngularJS modules is not a stupid idea but should be handled carefully. You need to make sure that you choose the right level on which to lazy load components. And if you need to split your application in multiple modules or use third-party modules, it is definitely not sufficient to use most of the mechanisms you will find by quickly googling for lazy loading in AngularJS. Using RequireJS is definitely a first step in the right direction but you need to make sure that the loaded scripts are also made available to the AngularJS dependency injection container.

 

AngularJS: Stopping event propagation on ng-click

Let’s assume you have a link containing an additional icon e.g.:

<a href="" ng-click="doSomething()">
	<span class="glyphicon glyphicon-th-list" ng-click="doSomethingElse()"></span>
</a>

(this whole article of course only make sense if your <a> tag contains more than this or is displayed in such a way that it’s larger than the icon)

When you click outside of the <span> doSomething() will be called. But when you click on the <span> both doSomethingElse() and doSomething() will be called. This is because of event propagation. The click is processed by doSomethingElse and then the event propagates to the parent of the <span> tag i.e. the <a> tag and triggers doSomething().

Actually the fact that it’s an <a> tag doesn’t make much of a difference. If you replace the <a> tag by a <div> tag, you’ll get the same results.

In plain old JavaScript, you’d return false in your click handler to prevent event propagation. Doing this in doSomethingElse() unfortunately doesn’t help. doSomething() will still be called.

The code defining the ngClick directive looks like this (in AngularJS 1.3.15):

forEach(
  'click dblclick mousedown mouseup mouseover mouseout mousemove mouseenter mouseleave keydown keyup keypress submit focus blur copy cut paste'.split(' '),
  function(eventName) {
    var directiveName = directiveNormalize('ng-' + eventName);
    ngEventDirectives[directiveName] = ['$parse', '$rootScope', function($parse, $rootScope) {
      return {
        restrict: 'A',
        compile: function($element, attr) {
          // We expose the powerful $event object on the scope that provides access to the Window,
          // etc. that isn't protected by the fast paths in $parse.  We explicitly request better
          // checks at the cost of speed since event handler expressions are not executed as
          // frequently as regular change detection.
          var fn = $parse(attr[directiveName], /* interceptorFn */ null, /* expensiveChecks */ true);
          return function ngEventHandler(scope, element) {
            element.on(eventName, function(event) {
              var callback = function() {
                fn(scope, {$event:event});
              };
              if (forceAsyncEvents[eventName] && $rootScope.$$phase) {
                scope.$evalAsync(callback);
              } else {
                scope.$apply(callback);
              }
            });
          };
        }
      };
    }];
  }
);

So it’s not returning the return value of the function used in ng-click. So we have two ways of solving this.

The first solution is to pass the $event parameter to your function and call both stopPropagation() and preventDefault():

<a href="" ng-click="doSomething()">
	<span class="glyphicon glyphicon-th-list" ng-click="doSomethingElse($event)"></span>
</a>

And:

this.doSomethingElse = function($event) {
	// do something else
	$event.stopPropagation();
	$event.preventDefault();
};

If you do not want to add this code in all ngClick handlers, you can also use a directive which will install a click handler on the element and return false:

<a href="" ng-click="doSomething()">
	<span class="glyphicon glyphicon-th-list" ng-click="doSomethingElse()" stop-propagation></span>
</a>

And:

.directive('stopPropagation', function () {
	return {
		restrict: 'A',
		link: function (scope, element) {
			element.bind('click', function (e) {
				return false;
			});
		}
	};
});

Both ways work. I ended up using the first one because it doesn’t involve going around AngularJS to get this solved but it does pollute your JavaScript code. So use the second one if it’s a problem for you and you’d rather only add an attribute to your HTML code. The advantage of the second approach is that you can easily add this behavior to any element by just adding the attribute.

AngularJS: RequireJS, dynamic loading and pluggable views

I’ve been using AngularJS for some time now and find declarative templates, the testability and dependency injection great.What I especially like is that the fact that AngularJS prescribes and dictates quite a lot, you are forced to bring some structure to the code of your application. This tends to reduce the velocity of the degeneration you often observe in larger JavaScript projects, where chaos slowly takes over.

But one of the things which was bothering me was that I always ended up adding some libraries/modules and forgetting to add them to my index.html. And sometimes after the list of loaded scripts grew quite a lot, figuring out where to insert this new script tag is really a pain.

Introducing RequireJS

So I decided some time ago to start using RequireJS to bring order in what was slowly becoming a mess. RequireJS does exactly what’s missing when you only rely on the AngularJS dependency injection: a file loader which supports dependencies and a way to define dependencies non-angular scripts (plain old JavaScript files).

Defining your AngularJS modules as RequireJS AMD modules with dependencies just means wrapping them in  a call to the define function e.g.:

define([
    'angular',
    'moment'
], function (angular, moment) {
    return angular.module('admin', [])
    	.controller('AdminController', ...);
});

Instead of loading all the scripts, you can replace all script tags by one of them loading RequireJS and referencing your main JavaScript file where RequireJS will be configured:

<script src="vendor/requirejs/require.js" data-main="scripts/main"></script>

main.js will contain the RequireJS configuration e.g.:

require.config({
    paths: {
        jquery: "../vendor/jquery/dist/jquery.min",
        jqueryui: "../vendor/jquery-ui/jquery-ui.min",
        angular: "../vendor/angular/angular.min",
        ngRoute: "../vendor/angular-route/angular-route.min",
    },
    shim: {
        angular: {exports: 'angular', deps: ['jquery']},
        jqueryui: {deps: ['jquery']},
        jquery: {exports: '$'},
        ngRoute: ["angular"],
    }
});

You also need to manually bootstrap your application to make sure that RequireJS loads all required dependency first. This is done by removing the ng-app attribute from your index.html and instead calling angular.bootstrap in main.js e.g.:

require.config({
	...
});

require([
        'angular',
        'scripts/app'
    ], function (angular, app) {
        angular.element(document).ready(function () {
            angular.bootstrap(document, ['cards']);
        });
    }
);

This will load AngularJS, then load your app and then bootstrap your application.

Introducing RequireJS is actually quite some work (if all your files and modules are already there) but not really complex. The main issue I have faced was that there is no explicit dependencies between AngularJS and JQuery, so they might be loaded in any random order. If AngularJS is loaded before jQuery, it will revert to using jqLite which is definitely not sufficient for jQuery UI. This will lead to strange errors occuring. So what you need to do is make sure that jQuery is loaded before AngularJS. This can either be done by setting priorities in your RequireJS configuration (only works with RequireJS 1.x) or by making jQuery a dependency of AngularJS (in the shim section of RequireJS 2.x – see example above).

OK, after all small problem were solved and my application was running again, I looked back at what I had done and realized that even though I didn’t ever had to add a script tag to my main HTML file, it wasn’t yet as great as I thought it would be before I started moving to RequireJS. First, I still have to manage dependencies on 3 levels:

  1. Dependencies between files i.e. file load order
  2. Dependencies between AngularJS modules
  3. Dependencies to individual controllers or providers

The first kind of dependencies is now managed by RequireJS instead of having me manage it in index.html. The other two have not changed.

Moreover, my application still looked like a big monolithic application at runtime since even though the file load order was now computed by RequireJS based on dependencies, I was still loading all files and interpreting them at application start. No matter whether some modules, controllers, directives… might only be needed much later or for specific users not at all.

Introducing Lazy Loading

So I started looking into lazy loading. Luckily I quickly realized that the step introducing RequireJS in my application hadn’t been useless. It is indeed possible to lazy load controllers, directives and such in AngularJS without resorting to using RequireJS. But you then need to manage the JavaScript files containing them and their dependencies to external libraries manually. With RequireJS, all you need to do to have all required files loaded before registering the lazy loaded controllers, views and such is to wrap it all in a require call.

There are different levels of lazy loading which can be achieved in AngularJS (having different levels of difficulty and restrictions).

Dynamically Loading Controllers and Views

The first level of lazy loading is dynamically loading controllers and views to an already loaded AngularJS module. To associate views with controllers, you would typically put your routing code in a module’s config function:

$routeProvider
     .when('/customers',
        {
            controller: 'CustomersController',
            templateUrl: 'views/customers.html'
        })
    .when('/orders',
        {
            controller: 'OrdersController',
            templateUrl: 'views/orders.html'
        })
    .otherwise({ redirectTo: '/customers' });

This means that everything will be loaded at once at runtime using the $routeProvider object. The referenced controller must have already been registered or you will get an error while defining the routing. Lazy loading the views (i.e. the referenced template URL) is supported out of the box but the JavaScript code for your controllers needs to be loading upfront.

To allow lazy loading, you will need to use the resolve property (additionally to the templateUrl) and also make sure that your controllers are properly registered once the JavaScript file is loaded. So using RequireJS to load the scripts and their dependencies, your code would look like this:

$routeProvider.when('/customers', {
    controller: 'CustomersController',
    templateUrl: 'views/customers.html',
    resolve: {
        resolver: ['$q','$rootScope', function($q, $rootScope)
        {
            var deferred = $q.defer();
            require(['views/customers'], function()
            {
                $rootScope.$apply(function()
                {
                    deferred.resolve();
                });
            });
            return deferred.promise;
        }]
    }
});

This makes sure that your controller doesn’t need to present immediately but that the resolver function will be called when this route is activated. The resolver function returns a promise we create using $q.defer. In order to load all required files and AMD modules, we wrap the logic in this function in a require block. Once the loading of the script and all dependencies is done, our callback is called which just resolves the deferred object.

Now, there is still a problem. All files will be loaded but you won’t be able to use the new controllers because they were created after startup. In order to have them properly registered, you will need to overwrite a few functions of your angular module to use compiler providers (such as $controllerProvider) instead (this needs to be done before using the $routeProvider). So you’re application js file would look like this:

define([
    'angular'
], function (angular) {
    var app = angular.module('app', []);

    app.config(['$routeProvider',
        '$controllerProvider',
        '$compileProvider',
        '$filterProvider',
        '$provide',
        function ($routeProvider, $controllerProvider, $compileProvider, $filterProvider, $provide) {
            app.controller = $controllerProvider.register;
            app.directive = $compileProvider.directive;
            app.filter = $filterProvider.register;
            app.factory = $provide.factory;
            app.service = $provide.service;

            $routeProvider.when(...);

            $routeProvider.otherwise(...);
        }]);

    return app;
});

Now you will see that when you activate this view, not only the HTML template file will be loaded on the fly, but also the JavaScript file containing your controller. And your controller will be accessible from the view.

This approach works well and doesn’t require much additional code but it only works if the controller you’re lazy loading is controller on an existing module. And even if it is the case, you’ll notice it doesn’t work that well, when the controllers you are lazy loading bring their own dependencies (other modules with controllers and directives).

Registering AngularJS modules dynamically

So to have a more robust and versatile solution, we need to be able to register and activate AngularJS modules dynamically (and the modules on which they are dependent).

In order to do it, we first need to understand what happens when you load a new module at startup. A module contains the following data which are relevant when you want to dynamically activate it:

  • A list of dependencies: module.requires
  • An invoke queue: module.invokeQueue
  • A list of config blocks to be executed on module load: module.configBlocks
  • A list of run blocks to be executed after injector creation: module.runBlocks

When a module is registered, AngularJS checks whether all referenced modules are available.

Whenever you call a function on your module (e.g. config, run, controller, directive, service…), it just pushes the provided function to a queue. In case of config, the queue is module.configBlocks. For run, it goes to module.runBlocks. For the others, it’s module.invokeQueue.

On module load, AngularJS will concatenate the run blocks (but without executing them yet), run the invoke queue and then run the config blocks. Once the modules are loaded, all run blocks will be executed.

When loading modules dynamically, the phase when modules are usually loaded is over, so even though the JavaScript files are loaded by RequireJS, the modules will not be properly initialized. So we just need to perform manually what’s usually done automatically by AngularJS.

Module Dependencies

So the first step is to make sure that modules on which this module is dependent are activated first. So the function to load a module would start like this:

this.registerModule = function (moduleName) {
    var module = angular.module(moduleName);

    if (module.requires) {
        for (var i = 0; i < module.requires.length; i++) {
            this.registerModule(module.requires[i]);
        }
    }

    ...
};

The Invoke Queue

Each entry in the invoke queue is an array with three entries:

  1. A provider
  2. A method
  3. Arguments

So in order to process it, we need to have a list of providers:

var providers = {
    $controllerProvider: $controllerProvider,
    $compileProvider: $compileProvider,
    $filterProvider: $filterProvider,
    $provide: $provide
};

And invoke the appropriate method with the provided arguments:

angular.forEach(module._invokeQueue, function(invokeArgs) {
    var provider = providers[invokeArgs[0]];
    provider[invokeArgs[1]].apply(provider, invokeArgs[2]);
});

The Config and Run Blocks

To execute the config and run blocks, you can just rely on $injector:

angular.forEach(module._configBlocks, function (fn) {
    $injector.invoke(fn);
});
angular.forEach(module._runBlocks, function (fn) {
    $injector.invoke(fn);
});

The registerModule function

So the complete registerModule function would look like this:

this.registerModule = function (moduleName) {
    var module = angular.module(moduleName);

    if (module.requires) {
        for (var i = 0; i < module.requires.length; i++) {
            this.registerModule(module.requires[i]);
        }
    }

    angular.forEach(module._invokeQueue, function(invokeArgs) {
        var provider = providers[invokeArgs[0]];
        provider[invokeArgs[1]].apply(provider, invokeArgs[2]);
    });
    angular.forEach(module._configBlocks, function (fn) {
        $injector.invoke(fn);
    });
    angular.forEach(module._runBlocks, function (fn) {
        $injector.invoke(fn);
    });
};

Registering pluggable views

Now that we can we can register modules dynamically, we’re only one step away from defining views which can be plugged into our application. Imagine you have an application, with multiple views showing different (possibly unrelated) data. If you want others to be able to extend your application, you need to provide a way to define a pluggable view and load all such views setting the appropriate routes and providing some model which can be used to display the corresponding navigation links.

In order to store all data required for configuring the routing and creating the navigation links, we’ll be using an object (viewConfig). The routing (assuming you’re using ngRoute) is then configured this way:

    $routeProvider.when(viewConfig.viewUrl, {
        templateUrl: viewConfig.templateUrl,
        controller: viewConfig.controller,
        resolve: {
            resolver: ['$q', '$timeout', function ($q, $timeout) {

                var deferred = $q.defer();
                if (angular.element("#"+viewConfig.cssId).length == 0) {
                    var link = document.createElement('link');
                    link.id = viewConfig.cssId;
                    link.rel = "stylesheet";
                    link.type = "text/css";
                    link.href = viewConfig.cssUrl;
                    angular.element('head').append(link);
                }
                if (viewConfig.requirejsConfig) {
                    require.config(viewConfig.requirejsConfig);
                }
                require([viewConfig.requirejsName], function () {
                    pluggableViews.registerModule(viewConfig.moduleName);
                    $timeout(function() {
                        deferred.resolve();
                    });
                });
                return deferred.promise;
            }]
        }
    });
};

It provides the following configuration possibilities:

viewConfig.viewUrl: it’s the relative routing URL e.g. “/admin”
viewConfig.templateUrl: it’s the relative URL for the HTML template of the view e.g. “views/admin/admin.html”
viewConfig.controller: it’s the name of the controller for the view e.g. “AdminController”
viewConfig.navigationText: it’s the text displayed on the navigation link e.g. “Administration”
viewConfig.requirejsName: it’s the name of the RequireJS AMD module e.g. “admin”
viewConfig.requirejsConfig: it’s the object to be added to the RequireJS configuration e.g. { paths: { ‘admin’: ‘views/admin/admin’ } }
viewConfig.moduleName: it’s the name of the module being loaded e.g. “app.admin”
viewConfig.cssId: it’s the ID of the link tag created to load the CSS stylesheet e.g. “admin-css”
viewConfig.cssUrl: it’s the relative URL of the CSS stylesheet file e.g. “views/admin/admin.css”

In order not to have to define all these parameters every time we call the function to register the view, we’ll first define some defaults to be set when some of these parameters are not explicitly set:

if (!viewConfig.viewUrl) {
    viewConfig.viewUrl = '/' + viewConfig.ID;
}
if (!viewConfig.templateUrl) {
    viewConfig.templateUrl = 'views/' + viewConfig.ID + '/' + viewConfig.ID + '.html';
}
if (!viewConfig.controller) {
    viewConfig.controller = this.toTitleCase(viewConfig.ID) + 'Controller';
}
if (!viewConfig.navigationText) {
    viewConfig.navigationText = this.toTitleCase(viewConfig.ID);
}
if (!viewConfig.requirejsName) {
    viewConfig.requirejsName = viewConfig.ID;
}
if (!viewConfig.moduleName) {
    viewConfig.moduleName = viewConfig.ID;
}
if (!viewConfig.cssId) {
    viewConfig.cssId = viewConfig.ID + "-css";
}
if (!viewConfig.cssUrl) {
    viewConfig.cssUrl = 'views/' + viewConfig.ID + '/' + viewConfig.ID + '.css';
}

Using this, it’s sufficient to call our provider function like this in order to have a link added for our administration view:

$pluggableViewsProvider.registerView({ ID: 'admin', moduleName: "admin", requirejsConfig: { paths: { 'admin': 'views/admin/admin' } } });

Since the way the navigation links are displayed pretty much depends on how you do it in your markup, our provider will not do it itself but just store the corresponding information and provide it through the provider:

this.views = [];
...
this.views.push(viewConfig);

So the complete module for our provider looks like this:

(function () {
    'use strict';

    define([
        'angular'
    ], function (angular) {
        return angular.module('pluggableViews', [])
            .provider('$pluggableViews', [
                '$controllerProvider',
                '$compileProvider',
                '$filterProvider',
                '$provide',
                '$injector',
                '$routeProvider',
                function ($controllerProvider, $compileProvider, $filterProvider, $provide, $injector, $routeProvider) {
                    var providers = {
                        $compileProvider: $compileProvider,
                        $controllerProvider: $controllerProvider,
                        $filterProvider: $filterProvider,
                        $provide: $provide
                    };
                    this.views = [];

                    var pluggableViews = this;

                    this.registerModule = function (moduleName) {
                        var module = angular.module(moduleName);

                        if (module.requires) {
                            for (var i = 0; i < module.requires.length; i++) {
                                this.registerModule(module.requires[i]);
                            }
                        }

                        angular.forEach(module._invokeQueue, function(invokeArgs) {
                            var provider = providers[invokeArgs[0]];
                            provider[invokeArgs[1]].apply(provider, invokeArgs[2]);
                        });
                        angular.forEach(module._configBlocks, function (fn) {
                            $injector.invoke(fn);
                        });
                        angular.forEach(module._runBlocks, function (fn) {
                            $injector.invoke(fn);
                        });
                    };

                    this.toTitleCase = function (str)
                    {
                        return str.replace(/\w\S*/g, function(txt){return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();});
                    };

                    this.registerView = function (viewConfig) {
                        if (!viewConfig.viewUrl) {
                            viewConfig.viewUrl = '/' + viewConfig.ID;
                        }
                        if (!viewConfig.templateUrl) {
                            viewConfig.templateUrl = 'views/' + viewConfig.ID + '/' + viewConfig.ID + '.html';
                        }
                        if (!viewConfig.controller) {
                            viewConfig.controller = this.toTitleCase(viewConfig.ID) + 'Controller';
                        }
                        if (!viewConfig.navigationText) {
                            viewConfig.navigationText = this.toTitleCase(viewConfig.ID);
                        }
                        if (!viewConfig.requirejsName) {
                            viewConfig.requirejsName = viewConfig.ID;
                        }
                        if (!viewConfig.moduleName) {
                            viewConfig.moduleName = viewConfig.ID;
                        }
                        if (!viewConfig.cssId) {
                            viewConfig.cssId = viewConfig.ID + "-css";
                        }
                        if (!viewConfig.cssUrl) {
                            viewConfig.cssUrl = 'views/' + viewConfig.ID + '/' + viewConfig.ID + '.css';
                        }

                        this.views.push(viewConfig);

                        $routeProvider.when(viewConfig.viewUrl, {
                            templateUrl: viewConfig.templateUrl,
                            controller: viewConfig.controller,
                            resolve: {
                                resolver: ['$q', '$timeout', function ($q, $timeout) {

                                    var deferred = $q.defer();
                                    if (angular.element("#"+viewConfig.cssId).length == 0) {
                                        var link = document.createElement('link');
                                        link.id = viewConfig.cssId;
                                        link.rel = "stylesheet";
                                        link.type = "text/css";
                                        link.href = viewConfig.cssUrl;
                                        angular.element('head').append(link);
                                    }
                                    if (viewConfig.requirejsConfig) {
                                        require.config(viewConfig.requirejsConfig);
                                    }
                                    require([viewConfig.requirejsName], function () {
                                        pluggableViews.registerModule(viewConfig.moduleName);
                                        $timeout(function() {
                                            deferred.resolve();
                                        });
                                    });
                                    return deferred.promise;
                                }]
                            }
                        });
                    };
                    this.$get = function () {
                        return {
                            views: pluggableViews.views,
                            registerModule: pluggableViews.registerModule,
                            registerView: pluggableViews.registerView,
                            toTitleCase: pluggableViews.toTitleCase
                        };
                    }
                }]);
    });
}());

This is an example of how to use this provider in order to register pluggable views:

(function () {
    'use strict';

    define([
        'angular',
        'ngRoute',
        'pluggableViews',
        'views/nav/nav'
    ], function (angular) {
        var app = angular.module('cards', [
            'ngRoute',
            'pluggableViews',
            'cards.nav'
        ]);
        app.directive('navbar', function () {
            return {
                restrict: 'E',
                templateUrl: '../views/nav/nav.html'
            };
        });
        app.config(['$routeProvider',
            '$pluggableViewsProvider',
            function ($routeProvider, $pluggableViewsProvider) {
                $pluggableViewsProvider.registerView({
                    ID: 'walls',
                    moduleName: "cards.walls",
                    requirejsConfig: {paths: {'walls': 'views/walls/walls'}}
                });
                $pluggableViewsProvider.registerView({
                    ID: 'admin',
                    moduleName: "cards.admin",
                    requirejsConfig: {paths: {'admin': 'views/admin/admin'}}
                });
                $pluggableViewsProvider.registerView({
                    ID: 'reports',
                    moduleName: "cards.reports",
                    requirejsConfig: {paths: {'reports': 'views/reports/reports'}}
                });

                $routeProvider.otherwise({redirectTo: '/walls'});
            }]);

        return app;
    });
}());

In each view (which needs the navigation bar), I use the navbar directive which template contains the following:

<ul class="nav navbar-nav" ng-controller="NavigationController">
    <li ng-repeat="view in views track by $index" ng-class="navClass(view.ID)"><a href='#{{view.viewUrl}}'>{{view.navigationText}}</a></li>
</ul>

And the NavigationController gives us access to the views configured:

'use strict';
define([
    'angular',
    'pluggableViews'
], function(angular) {
    angular.module('cards.nav', [
        'pluggableViews'
    ])
        .controller('NavigationController', ['$scope', '$location', '$pluggableViews', function ($scope, $location, $pluggableViews) {
            $scope.navClass = function (page) {
                var currentRoute = $location.path().substring(1) || 'home';
                return page === currentRoute ? 'active' : '';
            };

            $scope.views = $pluggableViews.views;
        }]);
});

In this example, the pluggable views registered are hardcoded in my application but having a separate file containing the view configuration is fairly easy and would be a way to make your views truly work as plugins.

So that’s it for this post. We’ve build an provider which allows us to register views on the fly considering their dependencies and allowing the navigation bar or panel to be extended. Of course, you’ll still need to add some error handling, some logic to make sure that modules you depend on do not get activated multiple times… But this is all for a next post.

CSS: centering an elemenent vertically and horizontally

I had already written an article about this quite some time ago. The problem with this approach is that it only works with elements having a fix size. So I’ve written an article specific to the horizontal centering of variable width DIVs last year. The problem with this approach is that although it works I have no clue why and feel it might break some time in the future. Also this approach only works for centering horizontally. Luckily, since then, I’ve able to use another approach which seems cleaner and also doesn’t require my components to have a fix size and works both vertically and horizontally, without using any strange hacks.

First let’s start with a non-centered element:

<html>
	<body>
		<div style="background-color:green;width:35%;height:35%">
	</body>
</html>

It’s green and it has a height and width of 35% (so no fixed size). As expected, it’s displayed in the upper left corner:

not centered

Obviously, the first thing you’d try and is giving it a top and left margin of 50% to center it:

<html>
	<head>
		<style>
		.centered {
			margin-left: 50%;
			margin-top: 50%;
		}
		</style>
	</head>
	<body>
		<div class="centered" style="background-color:green;width:35%;height:35%">
	</body>
</html>

Unfortunately, this will move the element past the center. You’ll also notice that it’s off by a lot vertically but less horizontally:

margins 50 percent

This is because margin-top doesn’t use the height of the container to compute percentages to pixels but just like margin-left, it uses the width. So if your container is square, you won’t see a difference but otherwise, it’s a problem. So using margin-top is no option. Instead, we’ll set the position to relative and set top to 50%. Ok, we’d implement it without hacks but this one not really a bad one…

<html>
	<head>
		<style>
		.centered {
			margin-left: 50%;
			top: 50%;
			position: relative;
		}
		</style>
	</head>
	<body>
		<div class="centered" style="background-color:green;width:35%;height:35%">
	</body>
</html>

This now looks better but it’s still not properly centered:

better 50 percent

As you can probably guess now, the problem is that the upper left corner of the element is centered both vertically and horizontally. Not the center of the element. In order to correct this, we need some CSS properties which unlike the margin-xxx or top do not take into account the dimensions of the container but the dimensions of the element. Actually there aren’t so many properties like this. Luckily CSS transformations do work with the dimensions of the element. So a translation of -50% both vertically and horizontally will center the element:

<html>
	<head>
		<style>
		.centered {
			margin-left: 50%;
			top: 50%;
			position: relative;
			-webkit-transform: translate(-50%, -50%);
			-moz-transform: translate(-50%, -50%);
			-o-transform: translate(-50%, -50%);
			-ms-transform: translate(-50%, -50%);
			transform: translate(-50%, -50%);
		}
		</style>
	</head>
	<body>
		<div class="centered" style="background-color:green;width:35%;height:35%">
	</body>
</html>

Now it looks exactly the way it’s supposed to:

centered

AngularJS: Running a directive after data has been fetched

In one of my AngularJS projects, I am using the angular-split-pane directive which is basically wrapping the split-pane JQuery plugin. This directive finds out how to split the panes and position the slider by reading the height and width attributes of the HTML tag.

I needed to add a functionality to my application so that the position of the slider is written to some session file on the server and reused next time the application is started to reposition the sliders at the same position, basically setting the height of the lower pane in percentage.

The previous position is fetched using $http. The problem is that it runs asynchronously and by the time the results are there, the directive has already been processed and changing the height tag has no effect anymore.

You can define the order in which directives are run by setting the priority property. But I needed a way to have the directive processed after the results from the server were available, not after or before another directive is processed.

After searching for a solution for quite some time, I finally find a way of implementing it late at night. This is actually pretty simple but for some reason it took me forever to think about such a solution.

When I fetch the data from the server I set some variable in my scope e.g.:

$http.get("../api/Config/").success(function(data) {
	sessionModel.ui = data;
});

Since I only want to execute the directive once the data is fetched, this means I want to have it executed once sessionModel.ui has a value. So all I need to do is add an ng-if attribute to my tag. Since the variable is initially not set, AngularJS will remove the element from the DOM and once the value is set, it will recreate it having the directive executed. So my tag would now look like this:

<split-pane ng-if="sessionModel.ui">

Of course the small disadvantage of this solution is that my UI is first shortly rendered without the split pane and when the data has been fetched, the split-pane is also rendered. In order not to have a partial rendering of the UI, you could of course move the ng-if attribute higher in your HTML code so that the whole UI is only rendered once the data from the server is available.

If your directive is not replacing the DOM element but just manipulating them, another solution would be to have the logic run asynchronously using $timeout and repeat until the data from the server is available. Or having it applied once the data is available by using $watch.