It bothers me that so many of us developers are annoyed by managers saying stupid stuff like "shouldn't be too hard" about things they don't understand, but then other developers walk into the same trap.
Yes, it looks simple at the surface. It most likely isn't. The problem is not that it looks simple, the problem is that you assume it is simple, probably because you don't have the full context.
Googling gave me this result, which sounds about right for what I remember. https://www.gfinityesports.com/grand-theft-auto/gta-online-h...
I’m not certain it’s still a valid workaround, and it’s not nearly as sophisticated as the OP method, but at least everyone can do it :)
Ahh. Modern software rocks.
I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.
This online gamemode alone made $1 billion in 2017 alone.
Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.
Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.
The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.
(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)
But, you know, premature optimization yadda yadda.
Parsing JSON?! I thought it was some network game logic finding session magic. If this is true that's the biggest WTF I saw in the last few years and we've just finished 2020.
Stunning work just having binary at hand. But how could R* not do this? GTAV is so full of great engineering. But if it was a CPU bottleneck then who works there that wouldn't just be irked to try to nail it? I mean it seems like a natural thing to try to understand what's going on inside when time is much higher than expected even in the case where performance is not crucial. It was crucial here. Almost directly translates to profits. Unbelievable.
There's a "but" though: you might end up getting banned from GTA Online altogether if this DLL injection is detected by some anti-cheat routine. The developers should really fix it on their end.
will r* fix it? maybe, especially since some person literally did half of the work for them. But given R* is a large company, this probably wont be fixed for a long time, and GTAO is probably being maintained by the lowest bid contractor group.
They probably have all of their full time assets working on the next iteration of GTA.
Something tells me this "problem" was discovered long ago at Rockstar HQ, and quietly deemed not to be a problem worth solving.
Overwatch is another an incredibly popular game with obvious bugs (the matchmaking time) front and center. And gamers are quick to excuse it as some sort of incredibly sophisticated matchmaking - just like the gamers mentioned in OP.
It's easy to to say it's something about gamers / gaming / fandom - but I have a locked down laptop issued by my bigcorp which is unbelievably slow. I'd bet a dollar there's a bug in the enterprise management software that spins the CPU. A lot of software just sucks and people use it anyway.
Another reason why C styled null terminated strings suck. Use a class or structure and store both the string pointer and its length.
I have seen other programs where strlen was gobbling up 95% of execution time.
So much to get from this.
Even if you don't have the source, you can make a change if you are annoyed enough.
If you don't like something, and the source code is out there, really go contribute.
Performance matters, know how to profile and if using an external dependency, then figure out their implementation details.
Algorithms & Data structures matter, often I see devs talking about how it doesn't matter much but the difference between using a hashmap vs array is evident.
Attentive code reviews matter, chances are they gave this to a junior dev/intern, and it worked with a small dataset and no one noticed.
Such silly mistakes causing such a huge delay for millions of gamers over 6 years.
Another pet peeve of mine is Civ 6's loading time for a saved game is atrocious. I'm sure there's a O(n^2) loop in there somewhere.
So has that JSON always been 10mb or has it grown a lot over time?
Has the load time crept up since launch or has it gone down?
I've logged in exactly twice. Load times like that may be worth it to a hardcore gamer, but I have no patience for it. There's no shortage of entertainment available to someone with a PC, a reasonable internet connection, and a modicum of disposable income. Waste my time and I'll go elsewhere for my entertainment.
Can anyone recommend a youtube video where I can watch (not necessarily learn) people doing this sort of work? I'm in the mood for some vicarious hacking :-)
But this seems pretty damn clean. And it’s egregious that no one at R* took a week to investigate and fix “correctly”.
> To be fair I had no idea most sscanf implementations called strlen so I can’t blame the developer who wrote this.
Is this true? Is sscanf really O(N) on the size of the string? Why does it need to call strlen in the first place?
I really enjoyed this article, but I do have a question about this part. Why would a single function be listed at mutliple addresses?
Is 1 min loading time even normal? Why did it take so long? I never played GTA Online so could someone explain?
Maybe set up a donation page? I’d be more than happy to send some beer money your way for your time spent on this!
How it remained undetected for so long is really weird though. Surely they must've had a massive amount of complaints about the loading times. I completely stopped playing because of them. How could they not have investigated the root cause, atleast once, in six years?
Yeah I know it sounds stupid, but I suspect real Sherlock Holmes was inspired by a true story like this one too, and at least some contemporary detectives started to enjoy reading them.
The first hurdle is the technical skill required: there has always been closed source software, but these days the software is so much more complex, often even obfuscated, that the level of knowledge necessary to diagnose an issue and fix it has gone up significantly. It used to be that you could hold and entire program in your head and fix it by patching a couple bytes, but these days things have many, many libraries and you may have to do patching at much stranger boundaries (“function level but when the arguments are these values and the call stack is this”). And that’s not to say anything of increasing codesigning/DRM schemes that raise the bar for this kind of thing anyways.
The other thing I’ve been seeing is that the separation between the perceived skills of software authors and software users has increased, which I think has discouraged people from trying to make sense of the systems they use. Software authors are generally large, well funded teams, and together they can make “formidable” programs, which leads many to forget that code is written by individuals who write bugs like individuals. So even when you put in the work to find the bug there will be people who refuse to believe you know what you are doing on account of “how could you possibly know better than $GIANT_CORPORATION”.
If you’re looking for ways to improve this, as a user you should strive to understand how the things you use work and how you might respond to it not working–in my experience this is a perpetually undervalued skill. As a software author you should look to make your software introspectable, be that providing debug symbols or instructions on how users can diagnose issues. And from both sides, more debugging tools and stories like this one :)
* There’s a single thread CPU bottleneck while starting up GTA Online
* It turns out GTA struggles to parse a 10MB JSON file
* The JSON parser itself is poorly built / naive and
* After parsing there’s a slow item de-duplication routine
(Copied from the END of the article)
Never tried consoles before, but figured the point was that it would be fast and hassle free.
GTA was fun, but after completing it I never load it, just to play for fun, because PS load times + GTA load times are too much.
Now I just play 20 year old games under wine on Linux.. that's less of a hassle, and I don't have to touch windows.
It offends my taste as a software engineer and I always fix these.
I would have expected libc to take that use case in consideration and use a different algorithm when the string exceeds a certain size. Even if the GTA parser is not optimal, I would blame libc here. The worst part is that some machines may have an optimized libc and others don't, making the problem apparent only in some configuration.
I believe standard libraries should always have a reasonable worst case by default. It doesn't have to be perfectly optimized, but I think it is important to have the best reasonable complexity class, to avoid these kinds of problems. The best implementations usually have several algorithms for different cases. For example, a sort function may do insertion (n^2, good for small n) -> quicksort (avg. nlog(n), worst n^2, good overall) -> heapsort (guaranteed nlog(n), slower than quicksort except in edge cases). This way, you will never hit n^2 but not at the cost of slow algorithm for the most common cases.
The pseudo hash table is all GTA devs fault though.
I did not ever consider inspecting obfuscated assembly and hooking functions to fix the problem myself. Very impressive work!
For reference, I just ran a 10MB file through the JSON parser I use in Python and it finished in 0.047s (vs 190s)
But honestly, on a large enough team with a large enough backlog much worse things slip through the cracks. Doesn't excuse this at all though.
Also kudos on the write up! Enjoyed reading it
It's also the reason the DOM is so slow, in my view.
I remember spotting a lot of fast JSON parsers around, but again, there doesn't seem to be any popular, open, flexible, binary file format out there.
Meanwhile, games are always larger and larger, machine learning requires more and more data.
There is ALWAYS money, battery and silicon to be saved by improving performance.
I've had to revisit production code from time to time to discover my or my colleague assumptions about library functions were not actually true.
But for someone in the possession of the source code this looks like _minutes_ worth of profiling time! With a stellar technical accomplishment that GTA5 truly is, you would expect tons of really talented devs that could have traced this one down on their lunch break.
The fact that they didn't speaks volumes about organisational structure within R*. I think some higher up managers _really_ need some stern talking to!
I wonder if it might even be an incompatible OSS license?
From a game programming perspective, I'm sure at launch there were very few extras to obtain, so this method ran fast and didn't raise any red flags.
But as time has worn on they've added a ton of extra items and it's become a real problem. What it does show is that probably most of their team are working on creating new things they can sell, vs testing and maintaining the codebase for the last N years.
First, a disclaimer: I have no idea how much Rockstar employees are paid, nor how their workdays look like. I don't know what their team sizes are, where they worked before, or who manages who and in what fashion. I actually don't know anything about the Rockstar engineering organization at all.
I am also not a GTA player (or, more accurately, haven't been since San Andreas came out many moons ago). This is my perspective as someone who has worked for various organizations in his life ( SWE-centered ones but also other, more "traditional" ones too).
We're all familiar with technical debt - it's a well established concept by now. Reducing it is part of the normal workload for well-functioning organizations, and (good) tech leads think about it often.
What isn't talked about as often is the "organizational" debt. Some things are "verboten" - you don't touch them, you don't talk about them, and you run like hell if you're ever assigned to deal with them.
Every large enough company has (at least) one of those. It might be a critical service written in a language nobody in the team knows anymore. Maybe it's a central piece of infra that somebody wrote and then left, and it's seen patch after patch ever since. These are things that end your career at the company if the BLAME points to you after you attempted to fix them.
I have a gut feeling - not based on anything concrete, as I mentioned - that the loading part for GTA Online might be one of those things. If someone breaks the process that loads the game - that's a big deal. No one would be able to play. Money would be lost, and business-folk will come knocking.
So sure, there might be some mitigations in place - if some part fails to load they allow the game to load anyways, and then attempt to fix it mid-fly. It's not black and white. But it feels like one of those things, and so people might have just been running like hell from it for years and years. Teams change. Projects change hands. People leave, more people join. It's life in the industry.
I would be REALLY interested in learning how software orgs deal with these types of behemoths in their projects. I have yet to find someone who knows how to - methodically and repetitively - break these apart when they appear.
Old time: 6 minutes (360 seconds). New time: 1 minute and 50 seconds (110 seconds). Speed-up calculated by article author: 1-110/360=0.694 (69.4% saved). Speed-up calculated by me: 360/110=3.27 (3 times faster).
Please calculate it the other way around. It makes great difference when you say you made something 10× faster than when you say you saved 90% of work even if both mean exactly the same thing.
Bruce Dawson has great article about this: https://randomascii.wordpress.com/2018/02/04/what-we-talk-ab...
millions of players, billions of minutes wasted
when you scale things, every bit of optimizations IS MENDATORY!
#saveOurPlanet
Its now at least 4 month old.
So whats the problem? When you tab+alt out of fullscreen, to change your soundvolumen everytime you start the game, you have to rechange the graphics configuration as well.
I solved it by .net code i found on github which would run fur 5 minutes and would put the volumne up as soon as it ifinds the rdd2 process...
What's worth noting is that the city overview can hang for 20-30 minutes during connecting. At the same time, suspending the GTAV.exe for 20-30 seconds allows you to immediately throw the player into an empty public session. It is unlikely caused by slow parsing of json, more like the suspend causes udp timeout when connecting to the slow session hosts.
Also the post where he mentions adding the branch with your fix is here: https://www.unknowncheats.me/forum/3078353-post937.html
I wonder who t0st is. He lives in Moscow according to Twitter. This type of work is usually done to find vulnerability in code e.g. buffer overflows. Injecting a piece of code to program machine code is a technique used in system hacking or software piracy. I can bet t0st is working or has worked for FSB. Anyway, not that is matters... it is still fun to think about it
Comments: