Comparing the APIs: what's the best way to access Betfair from Python?

(Just a quick break from the blog series on building a Betfair bot: this is a quick comparison of the various options for accessing the API through Python.)

In the examples I've used to date, I've used either Betfair's own Excel plugin, or I've used Resolver Systems' own Betfair plugin (which I've modified to add additional features to...) But Resolver's own plugin may not be the best for accessing Betfair – even from within Resolver One – and it does come with a number of disadvantages: it is Windows only, for instance. But what are the other options:

Well, there's the Python Betfair Library, hosted on Google Code, and last updated in mid-2008. The readme suggests it will not be completely simple to install under Windows, but as I'm using my Ubuntu box at the moment, that isn't likely to be a serious problem. So, OK, let's download and untar it. Looking into the example code, it looks like it operates in a very similar way to the Resolver libraries:

exchange = Exchange()

exchange.login(username, password)

# How to call get_account_funds

funds = exchange.get_account_funds(Exchange.EXCHANGE_UK)

Yep: all similar here. If it works, and is easily cross-platform, this might be a goer. OK: let's run the example code:

python example.py

Uh oh:

ImportError: No module named ZSI

OK: this may not be quite as simple as I thought. ZSI is the Python Web Services project (Zolera SOAP Infrastructure). It's no big deal to install this on my box, but it does explain with these libraries might struggle to be easily cross-platform. Is ZSI/pywebsvc in the Ubuntu packages? Yep, it's python-zsi (as you might expect). A simple sudo apt-get install python-zsi is all that's needed.

Let's run again. It's asking me for my username and password – this is good. And now I wait... and I wait...

...lots of errors followed by...

socket.gaierror: [Errno -2] Name or service not known

Grr: let's run again. It's always hard to know whether the problem is Betfair's (often down, particularly for the free service, API) or whether it's because the PythonBetfairLibrary is out of date.

This time it's better: it's giving me lots of information on markets that I don't have positions in. Then it craps out on a 'MARKET_CLOSED' error, that I can at least trap. Initial impression: promising (the syntax looks fine), but it's not been updated for a long while, and it doesn't seem to trap errors very well.

Let's try a slightly more complex test: I have a Python app for simple market making, that I've optimized for golf tournaments. Normally, I run this in Resolver. But there's no reason it cannot run it as a standalone application – with the proviso that I lose the pretty display.

[25 minutes later]

Well, that was simpler than I thought. The Python Betfair Library works pretty well, with two major provisos. Firstly, it doesn't trap errors particularly well. I may do a little bit of work to allow it to throw out more useful exceptions. Secondly, it's sloooooooooooooow. (This can be an advantage, as you are less likely to accidentally exceed 60 API requests per minutes.) Perhaps it's because I'm used to the Windows native Betfair DLL that backs up the Resolver libraries, but taking 20-30 seconds to execute and output the results of GetAllMarkets (against less than 2 for the native DLL) is pretty slow. It was hard to market-make more than 6-7 golfers in an efficient manner, against to 30-40 I manage using the Resolver libraries. There is nothing intrinsically slow about using pure Python libraries, so I shall do a little investigation, and see if it's anything about my setup that is causing the slow performance...

Which bring us to the library at BespokeBets.com. This can be got here: http://bespokebots.com/showme.php?id=betfair_api_python.zip. It's a pretty simple library, that uses Python's built-in url2 library to do web service queries:

bot = API() # create new api object

print "Login:", bot.login(username, password)
bot.get_account_funds()

The code behind the library is pretty simple: this is a very thin wrapper around the Betfair API. (I was impressed to note that there is no Betfair DLL dependence, and everything fits into a few hundred lines of code. Nice work.) And this thin-ness brings a cost: it implements only a limited subset of the API – there is no GetDetailAvailableMktDepth or GetMarketProfitAndLoss for example. That said, extending this library would be simple, and it certainly seems capable of 4-5 simple API calls a second, which makes it quite a lot faster than the PBL.

I am not going to re-implement my market making system, because a small part of the algorithm relies on GetMarketProfitAndLoss. But in theory, working around that would not be difficult.

For now, I am going to stick with the Resolver libraries – which are, of course, just a thin interface round Betfair's own .NET ones: but for those in a non-Windows environment, both the Python Betfair Library and BespokeBot's library will work well. For anyone playing with building my bot, very few code changes should be necessary...

And now I need to get back to work...

Building Our First Betfair Bot IV - Talking to the API and storing your bets

*EDIT: In this piece, I originally said you needed to use the paid API to place bets; this is no longer the case*

In the last missive, I wrote about how we want to store data. In theory, we could use Betfair itself to store data – and pull everything we need on an ad hoc basis. So: if we want to see which bets are open, we'd query Betfair, and it would return a list of betIDs. But, this comes with two major problems: speed and cost. Betfair can charge as much as 0.5p for certain data intensive API calls: you need some pretty amazing programming and trading skills to overcome those costs.

So, our 'bot needs to store as much information locally as possible. And this means a (very simple) database – in our case SQLite. In today's missive, we're going to put a bet on, and then store it in our database.

The first thing we'll need is a programming language. I am going to be a little Windows-centric and use IronPython – the Microsoft implementation of Python 2.6. You'll also need this zip file: betfairdatabase.zip. This contains the following files:

betfair.py

lib.py

betfair.dll

→ these are the IronPython / Resolver Betfair libraries

SystemData.SQLite.dll

→ The .NET ADO SQLite connector

bettinghistory.s3db

→ An empty SQLite database

database.py

→ our database library

What we'd like to achieve (ultimately) is to put bets on, and then put them in a table that looks something like this:

After you've installed IronPython, it's time to have a play. For simplicities sake, copy the contents of the above folder into your IronPython directory. Now, run 'ipy.exe', which you'll find in the selfsame directory. You should get something that looks like this:

OK: Let's import the database module:

>>> import database

And create a new database connection object:

>>> dc = database.database()

Open the database file, so we can interact with it:

>>> dc.open('bettinghistory.s3db')

OK: so, what can we do with our database connection? Well, firstly we can record our betting activities. Let's set-up and record a bet:

>>> import betfair

>>> betfairGateway = betfair.BetfairGateway('username', 'password', productId = 22)

OK

(This assumes you have a Betfair paid API subscription. If you don't, you'll be reading, rather than doing.)

>>> betfairMarket = betfairGateway.GetMarketById(20692870)

We've set up a betfairMarket object for the winner of the World Cup 2010 (which, slightly surprisingly, Spain is the favourite to win.)

Shall we see who the runners are?

>>> for runner in betfairMarket.GetRunners()

... print runner, runner.Id, runner.Back

...

The screen should look something like this:

Now, I'm working on the theory that – if England moves out to 10-1 (11.0 in Betfair speak) – then I want some of that action. (Of course, the current price is 7.0, so that isn't very likely right now... but still...) To do this we have to get hold of the Betfair runner (selection) object:

>>> selection = betfairMarket.getRunner(27)

(We've gotten the selectionID by looking at the print-out above, and entering it manually. You'd probabaly want to iterate over GetRunners() until you got the object you wanted... but this will do for now.) And let's check we have the right object:

>>> selection, selection.Back

(England, 7.0)

Perfect: that looks like the right object, and we need to pass this object to the betfairGateway if we want to put a bet on:

>>> response = betfairGateway.Back(selection, 11, 2)

OK

The first thing we pass is the selection object, the second the desired odds, and finally the amount (in this case, the £2 minimum).

Now, the Betfair bet response object is a complex beast, but the only bit that is really of interest to us is the array of results. Let's pull it out:

>>> betsArray = response.betResults

And see how many bet objects it contains:

>>> len(betsArray)

1

No great surprise here: the bet we've put into the system is so far out the money I'd be surprised if there was more than one object: that of an unmatched bet. (If you were to offer to back England at 6, with £100,000, then you would get an awful lot of bet objects in return: one covering the bets matched at 7.0, one at 6.9, etc.

But let's take a look at the bet object we've gotten here:

>>> bet = betsArray[0]

Ha ha! Now we (finally) have an object with the details of our bet. Let's see if it was succesfully placed into the system (i.e. that the market is still live, and we still have money in our account):

>>> bet.success

True

Excellent. We've placed the bet. That's a good start. Was it matched?

>>> bet.sizeMatched

0.0

No surprise: it wasn't matched. It's sitting in the Betfair system waiting for someone to offer 10-1 on England. Finally, before we call it a wrap for the evening, let's store this bet in our database:

>>> dc.openBet(bet, 20692870, 27, 200, 11, 'BACK')

The openBet method takes the following paramaters: bet object, marketID, selectionID, amount (in pence), odds (a float), type (either 'BACK' or 'LAY').

Tomorrow – or in a few days – we'll play a little more with putting bets into the system, and reading the responses. But, hey, pat yourself on the back, you've just put some money on England to win the World Cup :-)

Building Our First Bot III - Storing Our Data

(Apologies for the delay in the posting of this… I foolishly decided to move my work PC to Ubuntu. In the process, I discovered GNOME does not handle multiple monitors on multiple graphics cards particularly gracefully.)

Amyway, back to the Betfair ‘bot: it's funny; if I wasn't writing about this as I build it, I'd get out a text editor, hack up some Python, chuck it into Resolver, and see how it did. (With a little bit of testing... probably... to make sure that the idea and the code is not completely hairbrained. Heck, when I wrote my first marketmaking 'bot, that's exactly what I did.)

But when I'm writing for an audience, even a pretty small one, I feel the need to do things at least vaguely properly. No 'magic numbers', relatively well structured code, not using object orientation unless it is strictly necessary.

I'm also starting to think a little about 'bot program structure. And what I've been thinking about is the importance of keeping our own record of bets (aside from Betfair's own one). That is, my spreadsheet should contain a transaction list, and I should be able to find the betIDs of open bets, without querying Betfair.

Why? Well, there are three good reasons:

1. Querying Betfair is expensive. If we do more than a certain number of API calls a month, we'll get charged extra. If our 'bot is low margin (as this first 'bot certainly will be...), this could be a fatal flaw.

2. Querying Betfair is slow. Let's say we want to run this strategy on 200 markets, with 4 selections apiece, then that means we'll be doing close to 1,000 API calls a minute: and that's somewhere between mildly and completely impossible, given how slow the Betfair API can be.

3. We'll want to do analysis of where and why our bot worked, and where and why it didn't. Having easily – machine readable – logs of everything we've done is a clear positive. Yes, we can pull data out of Betfair. But if we don't have to, that's preferable.

So: how to record our activity?

Well: we could use a table/worksheet in Excel or Resolver. But that has a number of problems: it’s difficult to collate across multiple different strategies, markets and selections. It also makes for an often unwieldy and slow spreadsheet. (Spreadsheets aren’t really meant to be used as databases…)

Or we could use flatfiles or self generated XML. But then we’re building a complex interface, and this isn’t really hierarchical data.

No, what we want is a simple (but proper) database of all open and closed transactions, that can be queried extremely quickly and efficiently. Which brings us immediately to the question of which database: mySQL, PostgreSQL, Access, SQL Server or… SQLite.

And it really is no contest: SQLite is the option. Simple, fast to deploy, multi-platform, works with Excel, Resolver, Python, PHP, VBA, .NET, etc. In addition: an SQLite database, sitting in a Dropbox directory, can be used by different instances and computers in different locations, running atop different platforms. Nice.

Now, before we delve into the technical details, let’s ask what we’re going to store in this database:

• key – simple auto-incrementing integer
• betID - integer
• marketID - integer
• selectionID - integer
• amount – integer (we’ll store stuff in pence)
• back – Boolean (if True, we’ve put a back bet into the system, if false, we’ve put a lay)
• open – Boolean (is the bet open, or was it taken)
• cancelled – flag
• odds – float
• strategy – integer (which will be the primary key on a table of strategies. We’ll set this as 1 for now)
• timeopened – time
• timeclosed - time

This is not a complex selection for now; no great need for joins. This is a rapidly searchable data store, not an ERP system. Relationality is for convenience only.

Inside our ‘bot, we’ll have a worksheet that reflects the activity of that strategy in that selection on that market. And it will get this data out through the simple use of SQLite. (Best of all, this datastore will be useful for all your ‘bots going forwards.)

Right: that’s all the theory for this post. Next one, we’ll implement this store.

Off The Shelf Bots

There are an extraordinary number of off-the shelf pieces of software that promise extraordinary trading gains. All you have to do is hand over a small amount of money... load up your Betfair account... and... profit!

I am naturally suspicious of all get rich quick schemes. This blog will not make you rich. It will, I hope, give you the tools that will enable you to write sensible Betfair bots. It will, I hope, improve the quality of your betting, and perhaps your enjoyment of it.

But there are no 'guaranteed winner' Betfair bots out there. As soon as more than one person knows about a strategy, they will ruthlessly exploit it until the gains no longer exist. A year ago there were plenty of opportunities to make money though the arbitrage explained in this post. Now, the opportunities are extremely limited, if they exist at all - because when the smallest gap between bet and lay prices opens up, it is immediately exploited. Two years ago, uberbot sought out thousands of opportunities to make £2 bets at 100/1, which were really 70/1 shots. For an individual, exploiting this was all but impossible. For a bot it was a winning strategy. But that opportunity has been and gone: the uberbot team has had to find other ways to make a turn.

Natural suspicions aside, what's in these 'bots? What does my £25 or $50 or $1,000 buy me? What kind of strategy does it run? Colour me intrigued, if cautious. A little bit of Googling is in order, I think:

Well, first up is the (aptly named) Betfair Bot. This is a "is a brand new Automated Betfair Laying Software which will supercharge your betting activities even while you're away at work or simply want to spend time with your family." There is the obligatory quote about a chap called Adam who "made over £100,000 profit on Betfair from a starting bank of £200 with no knowledge of gambling by trading on Betfair!". Well, I don't know about you, but I'm considering giving up my day job to cash in on this £100,000 bonanza. OK, so this is a system that automatically lays things on Betfair. In other words, it's a super-simplified market making application of some description - I can see that would work. I'll download and review at some point.

Then there is the wonderfully titled Make Money on Betfair, which proudly advertises "the proof is in the video", and claims that the system - in one two hour period - made £200. If all I need (to sell a 'bot) is a video of a making money on Betfair, then I think this should be pretty simple. Even the world's worst system - given vaguely efficient Betfair markets - should occasionally make money, even if only briefly. The site continually harps on about "100% Proof". Sign me up!

And then there is Bet Angel, which has by far the most professional web page. A cursory glance makes it look a pretty useful tool: it seems to introduce a lot of financial world concepts to gambling and Betfair, dressed up into wonderful terms such as a 'riskometer' (which I suspect is a combination of a NAV calculator, with some simple variance calculations). The charts it seems to produce look very Bloomberg - and I wouldn't be surprised if the people involved in this project worked in Financial Services at one point or another. The pricing mechanism also seems very reasonable: I like the way it charges by the month, etc. Most of all, I like the way Bet Angel doesn't promise instant riches: for that reason, I give the first G&T Thumbs Up.

Well: this has been a diversion (caused by lack of access to Betfair from the hotel I'm staying in!) I'll continue with the Betfair 'bot on Monday or Tuesday.

Building Our First Bot - Part II, Framework

Last post, we looked at how out 'bot would make money. This post, we'll put the framework in place. The framework is the outline of the code we'll be using. One of the nice things about this framework is that it is pretty language-independent. With just a little work, it can be converted to pretty much any language: Ruby, VBA, Java, etc. I shall, of course, be using Python - and will probably implement it in Resolver down the road.

The main program loop will look something like this:

Evaluate Status

if No Open or Matched Bets:
 Offer To Back
else if Unmatched Offer to Back:
 See if Conditions for Backing Still Exist
 Move or Cancel Bet if they do not
else if Unmatched Offer to Lay:
 Do Nothing For Now (we're doing this simply, remember)
else if Matched Offer to Back:
 Record the Matched Bet
 Create new Offer to Lay
else if Matched Offer to Lay:
 Record the Matched Bet
 Offer To Back

This is the simplest kind of pseudo-code. Moving it into more real Python gets us this:

import betfair
betfairGateway = betfair.BetfairGateway(username, password, productID)

marketID = xx
selectionID = xx

while not MarketInPlay():

status = EvaluateStatus()

if status == 'No Open or Matched Bets':
  if ConditionsExistToBack():
    OfferToBack()

elif status == 'Unmatched Offer to Back':
  if not MyBestBestBet():
    CancelBets()
  
    if ConditionsExistToBack():
      OfferToBack()

elif status == 'Unmatched Offer to Lay':
  # Do Nothing For Now (we're doing this simply, remember)
  pass

elif status == 'Matched Offer to Back':
  RecordMatchedBackBet()
  OfferToLay()

elif status == 'Matched Offer to Lay':
  RecordMatchedLayBet()
  if ConditionsExistToBack():
    OfferToBack()

Now, while we're not quite at a working, trading 'bot, we have the outline in place. Tomorrow (or in the next few days), I'll fill in the missing methods, and transition the 'bot to Resolver One.

Next week (when I'm back in the UK, with reliable Internet access again...), I'll set it up and running.

Building Our First Bot - Part I, Theory

OK: we've done a lot of theory. Now, let's actually build something that can actually trade. What we're going to do is a very, very simple version of a market-making bot. Why so simple? Because we want to minimise risk during this learning experience, and also because it'll be easier to understand what we're doing. We'll build out this 'bot over the next couple of weeks and months, but this should get you started.

First things first: what's this bot going to do?

Well, it's going to act as a very simple market maker. It will offer liquidity, for a given selection, at a slightly better price than the best price currently offered. Every few minutes it will check whether it is still the best price, and will move it if price change around it. Should our meagre offer be taken, it will jump to the other side of the trade, offering up a price for someone else to take.

Simple? Well, let's explain it using some pictures. A typical Betfair selection looks like this:

Tottenham Hotspur vs Arsenal

	Back	Lay
Tottenham	2.2	2.4
Arsenal	1.5	1.7

Now, for the purposes of our super-simple 'bot, let's simply look at one selection – Tottenham. So, we have an option to back them at 2.2, and to lay them at 2.4.

What out 'bot will do is this: it will offer to back Tottenham with a £2 (the minimum) bet at 2.38.

Tottenham Hotspur vs Arsenal

	Back	Lay
Tottenham	2.2	2.38
Arsenal	1.5	1.7

Now, the market will look like this. (I've highlighted the bet that is ours.) Our 'bot will then monitor the market every 30 seconds or so. If our bet is taken (and nothing else happens), then the market will look like the first figure again. And we will have a profit of £2.76, should Tottenham win.

But we're not interested in whether Tottenham wins or not – our goal is to make money by supplying liquidity to the market. So, what we'll do now is offer to lay Tottenham at 2.22.

Tottenham Hotspur vs Arsenal

	Back	Lay
Tottenham	2.22	2.4
Arsenal	1.5	1.7

Should someone take our bet, then we'll have a (£0.32) profit irrespective of whoever wins.

This will be the simplest Betfair 'bot in the world: it will monitor somewhere between 1 and 100 selections, and will attempt to grab a small chunk of the action by acting as a (very limited) market maker.

But what happens if no-one takes my offered back bet, or if the market moves away from me? Well, no great loss. You can just reset the level of your bet, so that it is inside the best lay price available. The 'bot needs to be smart enough to only put bets on when there is a reasonable (say 3%) gap between the back and lay prices.

And what happens if someone takes my offered back bet, but then no-one takes my offered lay, or the market moves away from me so I am making a guaranteed loss? Well, the 'bot has two possible choices: it can always stay inside the best back price, and so sometimes it will take small losses. Or it can leave the bet in place and hope the market returns. Because the amounts we're playing with are so small, the worst that happens is that we end up with a small uncovered back bet on Tottenham.

This 'bot will work best on markets where the odds move around a lot. And it will work least well on odds which will tend to trend for long-periods in certain directions. It will also work very poorly for in-play markets – not least because it will only be checking on prices every 30 seconds or so. The 'bot needs a reasonably, but not excessively, liquid markets: we need to have a gap between the back and the lay prices, otherwise we're not offering anything useful. (The aim with this 'bot is to get paid for the provision of liquidity.) We'll work on some algorithms that find appropriate markets and selections down the road.

In the second part of this post – which will hopefully be tomorrow, but may be Friday – we'll actually program this 'bot, and get it trading on some real markets.

Four Trading Strategies

There are probably a hundred different, already implemented, trading strategies on Betfair. Someone will have - I'm sure - hooked the Fink Tank football predictor up to Betfair. Some of these will be fiendishly complex, others will simple. Some will make money for a short while, others will remain profitable for a long time, many will never consistently make money.

All trading systems will fall into one of four broad categories:

Algorithms that are better at calculating probabilities than most gamblers. I know of one system that trades only cricket matches, and it relies on the fact that many people over-react when a wicket goes down. Apparently, one wicket falling has less impact than many gamblers expect (in the short run, they over-estimate its importance). By immediately buying the side who has lost the wicket, assuming the odds have swung enough, the system has had significantly positive returns over the last several years. Another - also cricket related - system was based on the fact that people significantly over-estimate the probability of draws. For a long-while this was also pretty profitable - until other gamblers noted that Betfair odds were out of line with history. Each of these systems is based upon a better ability to calculate true odds than most punters.

Arbitrage related algorithms. The system described in this post, was an arbitrage related one. Arbitrage - executed correctly - is riskless profits. The principle is always roughly the same: there will be multiple markets or selections whose results are highly correlated in one way or another - and whose prices should move in lock-step. To take a political example, the Betfair markets on the individual seats should match up pretty accurately to the party seats line (a 50/50 bet on the number of seats individual parties will achieve). Frequently it does not. During the US Primaries, Hillary Clinton was still the favourite, even when the state betting was strongly favouring Barack Obama. There was arbitrage opportunity backing Hillary in the individual States, and backing Barack to be nominee. Some arbitrage happens between Betfair and Betdaq, and/or the spreadbetting firms and/or traditional bookmakers. For example, bookmakers typically give each-way odds based on a simple 1/4 formula. It used to be possible to look for races with strong favourites, back the second or third horses each way with bookmakers, and then sell the win portion of the bet on Betfair. This was enormously profitable - until the bookmakers realised their mistakes.

Flow of Funds Systems. This is what people tend to think of when they think of financial algorithmic trading; it is systems that are plugged into price feeds, monitoring movements, and detecting where prices are going to go. Flow of funds trading is unique because it requires limited information about the underlying market: you don't need to be a horse racing expert, just a maths whizz. The simplest flow of funds trading system will be the one we implement on here first: a simple market making system. Market making involves providing liquidity to a market: i.e., being prepared to take other peoples' bets. In return for this, prices you put into the system include an over-round (that is, the raw implied probabilities add up to more than 100%). If you are able to correctly judge the market so that you get roughly similar numbers of people backing each of the runners, then you will profit.

Informational Advantage Systems. "The most valuable commodity," said Gordon Gecko to Bud Fox, "is information". The first person to see that a cricket wicket has fallen, or a goal scored, is able to adjust prices that bit quicker. (To encourage market making, Betfair has a 5 second delay on bet placing - but this still gives opportunities when a foolish cricket market maker goes on a bathroom break at just the wrong moment.) Given limited resources, it will be extremely hard for you or I to know things other people do not: however, it is possible we will be able to react quicker if we leave the trading in the hands of a 'bot. There are a number of Python Bayesian analysis tools that might be useful to automatically divine of a piece of information is likely to move a market... before human traders can react.

Now, I have never written a bot based around types one or two (although I have made money with manual trading of this type.) I currently run a reasonably successful marketmaking bot that prices up perhaps 20-30 markets, and makes c. £300-400/month. (One of the impetuses behind this blog was that if I wrote about it, I might get around to improving it!) I am very keen to deploy a system based on informational advantage - and after we cover building a market making 'bot, I think we'll go straight onto Bayesian analysis.

So: until next time...