Backtesting System V2 Thoughts

During the last few months, I've been studying how to retrofit my backtesting system to to work as plug-in strategy engines for a live trading system.  Commercial algorithmic systems have had a big impact on the design and architecture of the new system.

A table structure for storing the back test results was one of the first focus areas.  At this time, all of the back test results are automatically dumped to CSV files at the end of each run.  Three files are generated:

• Trade summary - performance metrics for each trade at trade closure.
• For example, if one multi-legged trade is entered per monthly expiration over a two year period, then there will be 24 summary lines in this file.
• Trade summary by day - metrics for each trade by day.
• For example, if the days in trade (DIT) is 30 then each rade will have 30 lines in this file.  For two years of monthly expiration trades, there will be 24 x 30 lines in this file.
• Trade blotter - list of all of the transactions for each trade (all legs - entry, adjustment, exit).

The first iteration for storing the results of both live and backtested strategy performance looked like the ERM diagram below:

Backtesting System V1

Most of my focus during the last year, was spent on the backtesting engine for multi-legged options strategies.  Backtesting of six legged options strategies with start dates at approximately 60 DTE will run in less than two minutes for nine years worth of trades.  This application:

• Is written in Java (1.7), using the following libraries:
• commons-collections-3.2.1.jar
• commons-configuration-1.9.jar
• commons-dbutils-1.5.jar
• commons-io-2.4.jar
• commons-lang-2.6.jar
• commons-logging-1.1.1.jar
• guava-14.0-rc1.jar
• joda-time-2.1.jar
• log4j-1.2.17.jar
• mysql-connector-java-5.1.22-bin.jar

• Runs in Eclipse Juno (4.2) and Keppler (4.3) 
• Uses property files for strategy and database settings
• Uses data stored in MySQL (5.5.29):
• Because of the amount of historical options data, there is a table for each symbol:
• ivolatility_rut (end-of-day options data from iVolatility.com)
• ivolatility_spx (end-of-day options data from iVolatility.com)
• ...
• livevol_rut (15-minute options data from LiveVol.com)
• livevol_spx (15-minute options data from LiveVol.com)
• ...
• hod_iwm (end-of-day options data from HistoricalOptionData.com)
• hod_spy (end-of-day options data from HistoricalOptionData.com)
• ...
• Contains two utility tables:
• market_holidays
• List of market holidays: Jan-1-2000 to Dec-31-2014
• optionsexpirations
• List of monthly expirations: Jan-1-2000 to Dec-31-2014
• The historical data from iVolatility is the most current, with options data spanning from Jan-4-2006 to Dec-20-2013.

Node, Ubuntu Security POC

For my Ubuntu configuration of Node.js, there were six parts...listed below.  

The basic flow is for a router to forward to the port where the proxy is listening.  The iptables configuration allows non-LAN originated traffic to connect only to this proxy port.  This port is configured only for SSL.  

When the user hits the proxy port the user's browser is requested to provide its SSL client certificate.  If successful, the user attempts to authenticate with the primary Node.js application using the Node.js Passport module.  Authentication will be discussed in a later post.


1.  Node-http-proxy:

• Here's the link: https://github.com/nodejitsu/node-http-proxy
• http://stackoverflow.com/questions/15411042/how-to-run-node-js-app-on-port-80-using-http-proxy
• Make sure the port that Node.js will be running on is open in iptables!

2.  SSL:

• I found this link very helpful in creating the SSL components:
  http://www.akadia.com/services/ssh_test_certificate.html
• This site and the associated Github files were very useful:
  http://www.gettingcirrius.com/2012/06/automating-creation-of-certificate.html
  https://github.com/berico-rclayton/certificate-automation/

3.  Node SSL configuration:

• This link outlines how to use the certificates, etc (created in the prior step) with the Node-http-proxy:
  http://www.gettingcirrius.com/2012/06/securing-nodejs-and-express-with-ssl.html

4.  Browser SSL configuration:

• The client certificates from step 2 will need to be manually loaded into each user's browser.  On the iPad, it seems that the certificates are only recognized by the Safari browser and not Chrome.

5.  Upstart Script:

• The following links were useful in configuring upstart to launch both the Node-http-proxy process as well as the primary Node processes:
• http://caolanmcmahon.com/posts/deploying_node_js_with_upstart/
• https://gist.github.com/louischatriot/3385102
• http://stackoverflow.com/questions/13982257/upstart-script-for-node-js-app
• http://clock.co.uk/tech-blogs/upstart-and-nodejs
• I decided not to use the Node.js forever (https://github.com/nodejitsu/forever) tool with my code.
• I did have my upstart scripts send email when they were invoked...I used the information at this link to help with the configuration:
  http://zackreed.me/articles/39-send-system-email-with-gmail-and-ssmtp
• See my upstart script on my Upstart Script page in the nav bar.

6.  Monit Script:

• To monitor the server, the Node-http-proxy, and the primary Node.js processes, I used Monit.  The instructions at the following link were helpful:
  http://howtonode.org/deploying-node-upstart-monit
• See my Monit script on my Monit Script page in the nav bar.