Calling NAG C Lib from Python & Welcome!

Welcome

As this is my first post, I thought I should write a few introductory words. This blog is where I hope to share some of my personal quant trading projects. This page is not aimed at professional investors, but rather amateur quants i.e. anyone with an interest in financial markets and a bit of programming and/or mathematical finance skills. It's a platform for me to share the stuff that I work on, primarily to get feedback, but with the hope that others will find this useful. I encourage you to post your thoughts on the forum!

Onto the good stuff

I'm going to start off with a brief discussion about calling NAG C Libraries from python. These are a tried & tested numerical libraries (+20 years I think) which are predominantly used in science & engineering; I've been introduced to these through the CQF.

I began using these for my personal trading projects, which I've been writing in Python, as I found the SciPy-provided optimizers somewhat limited for my purposes (mainly fitting model parameters via MLE and portfolio optimization). I wanted 'industrial strength' optimization routines with a consistent interface, and since I had a student license for the NAG libraries, I decided to use these.
The main drawback was that there are no direct bindings for Python. There are however several ways to interface with NAG. See for example here or here. I decided to write my own wrapper for the C NAG lib, so that they look like 'native' calls. Then it's a simple as:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

#!/usr/bin/python import matplotlib.pyplot as plt import numpy as np import nag print("Testing Non-Linear Programming") def f(x): return x[0]*x[3]*(x[0] + x[1] + x[2]) + x[2] def c1(x): return np.sum(x**2) def c2(x): return np.prod(x) bounds = {"bl": np.array([1.]*4 + [np.NINF] + [np.NINF] + [25.]), "bu": np.array([5.]*4 + [20.] + [40.] + [np.PINF])} A = np.ones(4) x = np.array([1,5,5,1],dtype="double") try: result = nag.opt_nlp(f,x,A,(c1,c2),tol=1e-8,quiet=True,**bounds) print("The result is {}".format(result)) except nag.NagException as e: print("Exception: {}, code {:f}".format(e,e.code))

You can see that there is no boilerplate code here: all that is required is

import nag

And the actual call is the first line of the try block. So the first advantage of this method is that all that cython nastiness is hidden away in the nag package. This means that there is no mixing of C(ython) and python, which makes the code much more maintainable. I personally have a lot of trouble working with cython because it mixes programming styles in one file. For example, I often forget to declare C variables, as their python counterparts are automagically instantiated; this is not to say I don't know how to code in C - it's just the mixing of two paradigms that I find unpleasant. Some of the approaches used in the links above suffer from this problem.

The second advantage, which arguably stems from the first, is that we can use higher-level data structures directly as arguments. For example NumPy arrays or the wonderful pandas dataframes. These are the bread and butter of time-series data structures. Also we don't have to worry about pointers and callbacks as that is all taken care of behind the scenes. Take a look at the NAG call:

result = nag.opt_nlp(f,x,A,(c1,c2),tol=1e-8,quiet=True,**bounds)

Here x, A and bounds are all NumPy arrays, and f is a native python method. We haven't done any special conversion or initialization for these. You could almost use opt_nlp as a drop-in replacement for SciPy optimization routines.

The main drawback of all this is of course the amount of wrapper code required (240 lines for two methods - see nag.pyx), and the fact that each routine has to be hand-coded; there is no good way to automate the conversion. This is fine, if like in my case you rely heavily on only a handful of routines. But I can imagine that it would become very time-consuming if you wanted to call a whole subsection of the library.
As with every other approach this one has other problems - no solution is perfect. You need to try out the different methods and figure out which one works for you!

Try it out!

So how do you get started if you want to use these wrappers in your own projects? Well, first you will obviously need a Python environment, Cython installed (which requires a working C compiler) and last but not least a licensed and working NAG C library on your system. Then, head over to the code section and download the 'nag' folder. The README explains the compilation procedure (you might need to amend the include paths for linking), but you really want to look at test.py. In that file, the examples correspond to the examples in the NAG library documentation and source distribution.

For example, the code excerpt above, which is part of the test.py file, has a corresponding entry in the NAG library e04ucc documentation, which also explains how to compile and run it. Thus you can build both and check that the results match.

Next Time

I hope someone will find this code useful - please feel free to leave feedback via the forum, and I'll do my best to comment. This is my first blog post so I would also welcome any thoughts on SourceForge and the blog format in addition to its content. Finally, you can also find me on LinkedIn if you want to know more about me!

Next week I'm planning to share some more Python wrappers I wrote, but this time for the Interactive Brokers trading platform. I'll be focusing particularly on fetching price data for back-testing.