Did it ever strike you as odd – after you lost a match on Galaxy.com – that your Performance Rating was better than your opponent’s and that you were supposed to have had the better luck as well?
Well, it’s struck me – several times even. I’m a newcomer to the game, and my head is full of questions but void of answers – questions that I imagine more experienced players have asked and answered years ago.
I cut my trade on Bridge, a game where luck is present but invariably plays a smaller part. I wanted to get a comparison of sorts; how luck compares to skill, and which plays a greater part. The information seemed readily available to me on Galaxy, so I went through my last 100 games and picked out the PR and the Luck Rating (LR). This is what I found;
The winner in those 100 games had:
- Better PR in 68% of the games – worse in 32%
- Better LR in 75% of the games – worse in 25%
This would, at first glance, indicate that luck plays a marginally greater role than PR.
The winner had better PR and better LR in 48% of the games, worse PR but better luck in 27% of the games, worse LR but better PR in 20% and, going back to my initial question, worse PR and worse LR in 5% of the games.
Of course, I haven’t compared the relative size of the PR or LR but, hopefully, over one hundred games they’ll have cancelled each other out enough to give a reasonably correct estimate.
I felt like I at least had an idea of the individual importance between skill and luck; if one player plays really well and the other is really lucky – the luck takes it.
But then, none of this matters of course; I don’t know what luck is, or even if it really exists? Some would say that it’s a feeling or a figment of our imagination – an energy that goes through our souls and minds perhaps – and others would swear that it’s real. It’s abstract, obviously, but so is love and few would say that that doesn’t exist.
My own favourite definition is that there is no luck – only skill – but it’s a skill that takes on different shapes and forms; occasionally warped like the universe itself. Sometimes it’s stretched thinly like an elastic piece of string that gets snagged on a corner somewhere, but then it comes loose with a generous bounce and we forget instantly how bad things were just a moment ago. One thing I’m fairly certain about: we notice bad luck a lot more than good. I’ve never encountered a games-player who thought he was luckier than average, but plenty who claim to have worse luck than most.
So, is there anything we can do about it then? Well, to my knowledge, it doesn’t help to use the same underwear, the same pen or the same chair, but I have no quarrel with people who do. It also doesn’t help – and this may come as a shock to millions of games-players – it doesn’t help to moan to others about it.
So, if we don’t know if luck actually exists, and if we can’t do anything about it, then why do both XG and Galaxy display it? And shouldn’t it be right?
Coming from a game where luck – in my opinion – plays a smaller part, I find it amusing when you try to put a number on it, but both XG and Galaxy do.
I’ve played a fair bit on Galaxy in the last year, and suspicions arose very early about the accuracy of the luck ratings given. At first, I assumed that I just couldn’t recognise smaller lucky rolls and was just fixating on the game-busting jokers, but then I noticed that many others seemed to express disbelief in them as well. I remembered that a friend had suggested that I download the file from Galaxy to my own XG and compare them. So I waited a match or two until one came along where the numbers didn’t look right, and I fed it into XG.
Galaxy thought that:
But XG had other ideas:
Now that’s quite a difference – it’s miles out! How could that be?
We notice here that Galaxy and XG display luck in different ways, and we know that, on Galaxy, you cannot look up the luck value of a specific roll, like you can on XG. It’s therefore impossible to make a straight comparison, which I suppose is very convenient for Galaxy.
This was a revelation! Soon I was downloading every other match to compare the numbers, but I wasn’t sure I was using the right settings.
When downloading a match to XG, you can choose how deeply you wish to analyse, and you can choose a different level for each player. This makes a lot of sense if you wish to look at your own game but don’t care so much how your opponent played. I selected a random match and asked XG to look at it in all different settings:
XG Very Quick
XG World Class
We can see that the relative difference between the settings is nothing much to worry about: my PR fluctuates between 5.22 and 5.78, and his between 2.05 and 2.49, both in relative proximity to Galaxy’s own numbers.
We also get the idea that – on this occasion – the luck ratings seem to be both rather consistent in themselves, and also remind us In some way of those coming out of Galaxy; XG thinks the difference in luck is somewhere around 4.2, and Galaxy says It’s 2.9. Not bad!
I ran the analysis again to see if anything changed, but XG stood its ground on all settings except “World Class” where my PR changed from 5.56 to 5.55, giving us perhaps some insight into how complicated these calculations are.
Galaxy tells us they’re using an external server to do the analysis, and that it’s set to the default setting “Fast”, so that’s the one I’m going with from here on in.
Now that I knew how to check the numbers against XG, I was eager to see how often – and by how much – the numbers were out. I played another match where the numbers were very off, and then, just ten or so matches later, it became ridiculous.
I played a 7-pointer against a fellow called “Burento” and lost 7-2. We both played reasonably, but I had no chance whatsoever; it was a hailstorm of jokers – I mean brutal! Luckily, I was prepared for what Galaxy would say, so I wasn’t too surprised when it claimed that I had been the lucky one.
Well, I was happy that my rating was unchanged, but I knew the luck rating was absurd. I asked XG.
-7.40 in a 7-point match! That’s by far the worst I’ve ever seen and I don’t think many will be able to show me anything similar for some time to come. It doesn’t even seem possible in a 7-point match, but then it occurred to me that we actually played for nine points (7-2). In essence, he could have moved his checkers at random and I’d still have had no chance.
But what do we think of Galaxy’s performance? Its luck rating is out – not by inches, metres or even miles – it’s out by lightyears! It appears to be nothing more than a random guess.
I needed to find out more.
I wanted to attempt to find out if there was any kind of consistency in Galaxy’s numbers or if they were just randomly taken out of the air. I decided to take a look at my last ten games. The results are displayed like this; the luck is displayed from Player A’s perspective in the first column; if he was deemed to be -0.5, and his opponent +1.5, the figure shown would be -2.0. The second column is XG’s analysis, the third column shows the difference between the two, and the last column will have a “Yes” if Galaxy and XG agreed, and a “No” if they didn’t.
In this recent batch of ten matches, the average difference between Galaxy’s and XG’s luck ratings was 1.37 and only on one occasion did they disagree on who was the luckier player. Some of the numbers were pretty spot on, but one differed by a massive 3.54 equity points.
I’d say it’s fair to conclude that Galaxy is not taking its numbers randomly out of thin air – they’re just out by a very large margin from time to time. Why should that be?
At this point I had an idea. This may sound a bit weird but bear with me here…
I play Bridge on an online site called Bridge Base Online, or BBO for short. On BBO, the largest number of players we had seen, until a few months ago, was about 12,000 when the World Champ final was screened live. Last week when I logged on, there were 54,000 people online!
What happened with that many players online was that the site couldn’t function properly; you were unable to start games or move freely to watch them, you couldn’t find friends, messages wouldn’t go through etc etc. One evening it got so bad that, suddenly, everything just crashed and we lost connection. If it disconnected all 54,000 of us or not, I don’t know, but certainly everyone I was playing with at the time.
We can see that the number of players and matches on Galaxy has increased manifold during the lockdown, and I habitually get disconnected from a table and have to find my way back before my time runs out. Is it possible that, due to the large numbers, Galaxy does not “have time” to work out the luck ratings properly?
Just to get a comparison of sorts, I went back to look at ten more matches that I knew were played before the lockdown – around early autumn 2019.
Well, the numbers are different; much less volatile. They differ by less than one point in all cases but one, and XG and Galaxy agree in all ten cases. The average difference between the two computers is pretty much half of that of the more recent batch.
So, does this prove anything at all?
Of course not; in order to make a scientific study, you‘d have to use a lot more data than I have done, and probably also know something about maths and probabilities.
But then, I’m not out to prove anything – I’m just looking for answers. The one thing I’ve found out, for sure, is that we should all take the numbers coming out of Galaxy.com with a fairly large pinch of salt.
Since I wrote the first draft of this little memo and sent it to some friends to see if they had an opinion (well, they were friends before I sent them it to them anyway), it has been mistaken for an article – it’s not.
These are just the muddled thoughts of a newcomer who’s keen on learning more about the game but not sure where to find the answers. And guess what – I have found some! Just hours after it was circulated amongst some top players, I received very instructive pieces of information, and they’re much appreciated.
It’s not shameful to be ignorant – especially when you’re new to something – and I’m not afraid to look stupid. The only thing that’s stupid, in my opinion, is not asking questions when there’s something you want to know.