Regarding the impact of the crashcode put into ConfigLib and the needed clarification and explanation of the code for layman understanding

[Diff]

Welcome to the forums! Pr0fesseur's assessment is accurate. Right now we're aware of 2 mods that were caught in the crossfire, the caves n caverns you mentioned already and the url radio mod.

Edited Sunday at 11:35 AM by Diff

[reg8]

I agree. Now it's leaking outside the VS space by itself through Reddit and through others discussing it elsewhere. Especially about the Reddit bit, every comment and post will show up in search results, possibly whenever anyone tries to get more information about the game before buying. Further responses can't be delayed, as more people make up their mind over time and lose trust. I personally won't take anything the team says at face value anymore, unless a new statement addresses everything people are unhappy about and explains why some previous decisions were made.

[Michaloid]

Wouldn't it be funny if Penguin0 covered this mess 

[reg8]

11 minutes ago, Michaloid said:

Wouldn't it be funny if Penguin0 covered this mess 

welcome to slop liive

[Sadi89]

okay so take this with a grain of salt since I'm not proficient with programming. Please confirm my following understanding.

 

 

Enumerates all assemblies loaded into the application's AppDomain.
--> scans all game files loaded? but does not look through the pc's files

Fingerprints loaded assemblies using hard-coded CRC32 values.
knows hash values of certain mods and has them saved on a list to detect them when looking through the loaded mods 
---> targeted specifc mods the author deems "bad". possibility of false positives? what is it looking for exactly? 
can we have the list?

Performs an additional heuristic check for obfuscated assemblies by measuring the percentage of type and member names shorter than three characters.
When its detection conditions are satisfied, schedules execution after a pseudo-random delay.
--> reads like it look for shortened values in the code, like instead of the variable named caveindexmultiplier its just named "ab". 
a simple way to hide what the specifc part of code is about to the untrained eye. Does not sound bad and sounds more like a privacy filter. 
It feels like looking through a code block and flagging it if a lot of short variabels are found - it flags it.

--> second part is about how the dll then proceeds to crash the game after a random length of time 
when one of the two search methods gave a positive reading, which is maybe good for keeping it hidden 
but doesnt really help people getting abused by said cheats if the crash can take several minutes to execute.

What is the time frame coded into the dll?


Deliberately modifies unrelated internal Vintage Story client state in ways capable of causing crashes, disconnects, 
broken networking, corrupted UI state, and other intermittent failures.
Silently catches and discards exceptions throughout the detection path, making the resulting failures substantially more difficult to diagnose.
--> Those alterations can cause crashes, disconnects, networking problems, UI problems, and other unpredictable behavior. 
    Not just simple crashes

--> hides/alters error logs to make you unaware what caused said DCs networking issues or crashes, 
blaming other mods, server owners, you ISPs or your own hardware.

So this is how it is presented to me at this point which does not line up with the official statement.

Edited Monday at 11:10 AM by Sadi89

[MKMoose]

33 minutes ago, Sadi89 said:

can we have the list?

It targets not mods, but type names, to be specific. Apparently the CRC32s that it checks against correspond to "Aimbot", "InstantShot", "KillAura", "MapCursorTeleport", "Nuker", "VintageClient", "BlockFinder" and "Esp", though I'm not certain about the credibility of the source for this information (it was written by an LLM and posted by someone on Discord, and I don't even know what the original source is at this point). The type names need to start with this string, plus there are also additional checks against the length of the type name for "BlockFinder" and "Esp" which need to have at least one additional character after that. If at least three of these are found, the detector trips.

 

33 minutes ago, Sadi89 said:

What is the time frame coded into the dll?

It's (5 ± 4) * 92439 ms, so anywhere between ~92 s and nearly 14 minutes.

 

33 minutes ago, Sadi89 said:

hides/alters error logs to make you unaware what caused said DCs networking issues or crashes,

I have no idea where this comes from, as I haven't personally seen any evidence of it hiding or altering any logs in spite of seemingly quite a lot of people claiming that it did. It randomly modifies some part of the game to cause it to work incorrectly or crash instead of directly causing a crash by itself, which means that it avoids taking the blame in the stack trace, but it doesn't obscure the cause for the crash further beyond this.

Edited Sunday at 01:18 PM by MKMoose

[Gandorf]

3 hours ago, MKMoose said:

I'm not certain why the "gameplay affected" group doesn't include the cheat client? That could be seen as implying that the anti-cheat didn't actually trigger on the intended target, but it very much did.

 

One thing that is really missing from this whole discussion for me is actual analysis into how many mods tripped the detection. If we could run it on all mods on ModDB or even from outside of ModDB (separately on current mods and on mods from around when that code was first added) to see how many of them trip it, and while at it ideally also while controlling for whether the mod lists any of Maltiez' affected mods as a soft or hard dependency, then we would could justifiably say what the risk of false positives actually was at the time that the code was added as well as currently. I would do that myself but don't really have the time for it at the moment.

I've not yet come across any confirmation as to whether any mods trigger the detection besides:

• URL Radio (YT) - released ~1.5 months ago, currently 1349 downloads,
• Caves and Caverns - released ~one month ago, currently 1466 downloads,
• and one other mod, but no idea which - from several months ago, whose author apparently removed obfuscation after Maltiez contacted them.

Given how popular and necessary ConfigLib is, possibly a few hundred to over a thousand players of those two mods could have been affected. Maybe some math wizard can give us probabilities? But the potential for false positives seems to be bigger than ~100 affected cheaters. I think not acknowledging it in an official statement is one of the major points of frustration for half of the community outside the case: should Maltiez stay or not

[LadyWYT]

3 hours ago, MKMoose said:

• URL Radio (YT) - released ~1.5 months ago, currently 1349 downloads,
• Caves and Caverns - released ~one month ago, currently 1466 downloads

I'm guessing this is total downloads, and not unique downloads? I'm not even sure there's a way to get the total unique download number, outside of being on Anego's end of things, but I do feel it's an important distinction to make. Ten people downloading the mod once is a lot broader scope than one guy who downloaded it ten different times for whatever reason.

 

4 hours ago, Vratislav said:

I did not put the numbers into the plot, but I assume that ConfigLib with the payload was installed by up to 100 000 players, and the players who used mods triggering false positives were hundreds or thousands in maximum.

I do feel this is a pretty safe guesstimate though.

[Diff]

7 minutes ago, Gandorf said:

Given how popular and necessary ConfigLib is, possibly a few hundred to over a thousand players of those two mods could have been affected. Maybe some math wizard can give us probabilities? But the potential for false positives seems to be bigger than ~100 affected cheaters. I think not acknowledging it in an official statement is one of the major points of frustration for half of the community outside the case: should Maltiez stay or not

IMO, likely more than ~100. Really doubt we hit 1000 false positives just based on vibes.

Downloads are cumulative across versions, they're not unique users. If we pretend the highest number of downloads on any single version is a proxy for unique downloads, it's 1200 for the both of them combined. Caves N Caverns likely has a substantial population of single player users. URL Radio sounds like it'd be almost completely multiplayer users. Just based on vibes still, let's say 650 (all of em) from URL and 250 (half) from CNC are downloads for servers.

So that's 850 opportunities pulled straight out of my assumptions, however common ConfigLib is on your average server is what's going to matter for the final made up body count. I don't know enough about that to even try to guess though. 

[LadyWYT]

6 hours ago, Mushroomancer said:

I'm loathe to say it but whatever happens to Maltiez, I think the lesson that gets taken from this is to separate from the community and discourage any direct emotional contact or parasocial bonding as best as you can. It's not worth it to thread the needle between professional corporate conduct and leading community members especially when you're going to be criticized for being too corporate or too emotionally invested either way.

Is that really necessary though? I mean it is one solution, but I'm not sure that becoming cold and distant from the community is a very healthy way to handle it. A better option, I think, is to perhaps re-assess whatever process is used to vet potential hires, and have some better documentation/training regarding public communications. Which it sounds like Anego Studios is looking into the training part. 

It's probably also a good idea to carefully consider the personality of who you're hiring as well. Several users say Maltiez is a jerk. I'm not sure how accurate those claims are, since I'm not familiar with the guy's personality to say to what extent that's true. Pretty much anyone can come across as a jerk if you catch them on the wrong day or otherwise talk to the right people, and being a massive jerk doesn't mean that you aren't capable of good work. However, an abrasive personality is also difficult to work with, and a liability when it comes to public relations, so more oversight is probably advisable when it comes to individuals like that so that problems can be caught and fixed before they spiral out of control.

[LadyWYT]

8 minutes ago, Diff said:

So that's 850 opportunities pulled straight out of my assumptions, however common ConfigLib is on your average server is what's going to matter for the final made up body count. I don't know enough about that to even try to guess though. 

That seems a pretty safe bet.

One thing that just crossed my mind though, regarding users blaming ConfigLib for their crashes. Do we have any kind of community guide when it comes to troubleshooting issues with mods? I know several of us are experienced enough with modding that troubleshooting is second nature at this point, but there's also a decent chunk of the community that's not nearly as familiar with the technical side of things. It seems a pretty good idea to have some sort of guide that players can refer to when they have trouble, as well as have some of the more technically adept community members inspect mods that seem to be a common denominator when it comes to crashes(especially if the crash reason isn't obvious). To be fair, this could be wishful thinking on my part, but it seems like at best issues like ConfigLib could be found and dealt with much sooner, and at worst mod authors get some additional help when it comes to fixing their own code.

[tiggerbiggo]

8 minutes ago, LadyWYT said:

That seems a pretty safe bet.

One thing that just crossed my mind though, regarding users blaming ConfigLib for their crashes. Do we have any kind of community guide when it comes to troubleshooting issues with mods? I know several of us are experienced enough with modding that troubleshooting is second nature at this point, but there's also a decent chunk of the community that's not nearly as familiar with the technical side of things. It seems a pretty good idea to have some sort of guide that players can refer to when they have trouble, as well as have some of the more technically adept community members inspect mods that seem to be a common denominator when it comes to crashes(especially if the crash reason isn't obvious). To be fair, this could be wishful thinking on my part, but it seems like at best issues like ConfigLib could be found and dealt with much sooner, and at worst mod authors get some additional help when it comes to fixing their own code.

From my perspective there's (at least) one big problem with the way things are done right now that mean a guide (while useful) wouldn't really have stopped this from taking longer than normal to uncover;

Releases (Compiled binaries) are uploaded by the mod author separately from the code, so there's nothing stopping a mod author from misdirecting people by making the release act differently to the source code. Usually if you're working with open source mods, if there's a problem you can go to the source, but in this case the easiest solution would have surfaced nothing since the source wasn't malicious.

One possible way to solve this would be to require mods to be compiled be a trusted entity. So, modders develop and publish the source code, and then instead of publishing a release themselves, the mod portal itself would automatically compile it and serve it. If the mod author cannot push their own binary, there's no possibility for the source code to differ from the binary.

This does introduce the problem of forcing all mods to be open source, which wouldn't be a problem for most people but I guess some people are protective of mod code. I don't understand it personally since it kinda goes against the spirit to me. That feels like writing code on top of someone else's creation and not wanting anyone to modify yours in kind, but that's beside the point.

Edited Sunday at 02:24 PM by tiggerbiggo

[cjc813]

Lol.

I randomly decide to check the forums, and people are melting down over what is, at worst, a dick move.

[LadyWYT]

7 minutes ago, tiggerbiggo said:

From my perspective there's (at least) one big problem with the way things are done right now that mean a guide (while useful) wouldn't really have stopped this from taking longer than normal to uncover;

Releases (Compiled binaries) are uploaded by the mod author separately from the code, so there's nothing stopping a mod author from misdirecting people by making the release act differently to the source code. Usually if you're working with open source mods, if there's a problem you can go to the source, but in this case the easiest solution would have surfaced nothing since the source wasn't malicious.

Well, that's also why I suggest that when a mod starts causing issues with a lot of other things and there's not an obvious why beyond "it just doesn't work and removing the mod fixes the problem", it might not be a bad idea for some of the more technically adept members of the community to have a peek at the code if they have time to do so. Fishy code could potentially be caught a bit faster and reported, while at worst the mod author in question gets some extra help trouble-shooting the issue.

It's definitely not a foolproof plan, but it seems a proactive way for the community itself to grow stronger. And it seems a little less invasive than trying to tighten the modDB rules. Mind you, I'm not saying the modDB rules shouldn't be looked at, but additional rules aren't guaranteed to prevent future problems either, and rules that are too strict can certainly hurt a community more than they help.

[pigfood]

1 hour ago, cjc813 said:

I randomly decide to check the forums, and people are melting down over what is, at worst, a dick move.

It goes way beyond that.

maltiez didn't merely target cheat mods with his malware. He targeted any mod using a lot of short type names/identifiers in its code (as a half-assed heuristic to identify obfuscated code). It's absolutely certain that he knew that legitimate mods would very likely be impacted by his malware.

A lot of people wasted their time trying to troubleshoot mysterious crashes, because he maliciously randomly targeted non-cheat mods. He probably caused damages in people overall wasting thousands of hours.

I have invested at least 50 hours contributing to Vintage Story mods (bug fixes and updates for VS version changes). Unless Anego Studios dramatically changes course and honestly deals with the situation, I will stay away from VS and will actively encourage others to do the same.

Tyron is downright lying in his official statement in that he doesn't acknowledge that legitimate mods were impacted by the obfuscated code heuristics:

Quote

What was the impact?

Maltiez’s code was designed to crash the game client when it detected the cheat. It did not harm anyone’s hardware or touch personal data in any way, shape or form. Fewer than 100 cheat mod users were affected by the deliberate targeting.

The malware code is quite simple and very small. Any competent dotnet developer can figure out in 10 minutes, what it does (once it is found). 

Edited Sunday at 04:25 PM by pigfood

[cjc813]

3 minutes ago, pigfood said:

It's absolutely certain that he knew that legitimate mods would very likely be impacted by his malware.

A lot of people wasted their time trying to troubleshoot mysterious crashes, because he maliciously randomly targeted non-cheat mods. He probably caused damages in people overall wasting thousands of hours.

He wasted an unknown amount of people's time, and it was probably a lot of people's time. I can agree to that. That really sucks, and I mean it. I'm not being sarcastic.

Should he be completely separated from Vintage Story and its community? Maybe, idk.

I keep playing the game just as often either way.

Does the offense warrant a community-wide meltdown?

Nah. I really don't think so.

[Vratislav]

6 hours ago, MKMoose said:

I'm not certain why the "gameplay affected" group doesn't include the cheat client?

Originally, I had a title "Victims" and users of the hacked clients were labeled "No mercy." Then I considered it too subjective and changed it this way, without actually redrawing the first group. I should, yes.

[Spear and Fang]

2 hours ago, LadyWYT said:

Several users say Maltiez is a jerk

Having been in and around conversations with Maltiez frequently, I would describe him as relatively emotionless, admittedly having zero sense of humor, and always straight to the point (which was often borderline offensive, especially to those who were unfamiliar with his personality).  Add to this his acknowledgement of using ai at times to work though some difficult hurdles. So yeah, he rubbed a lot of people the wrong way.  

I'd be curious to see a venn diagram of the Maltiez is a jerk crowd, the anti-AI crowd, and the fire Maltiez crowd.

[icesharkk]

9 minutes ago, Spear and Fang said:

Having been in and around conversations with Maltiez frequently, I would describe him as relatively emotionless, admittedly having zero sense of humor, and always straight to the point (which was often borderline offensive, especially to those who were unfamiliar with his personality).  Add to this his acknowledgement of using ai at times to work though some difficult hurdles. So yeah, he rubbed a lot of people the wrong way.  

I'd be curious to see a venn diagram of the Maltiez is a jerk crowd, the anti-AI crowd, and the fire Maltiez crowd.

anecdotal personal evidence for your consideration: could not care less about AI use in coding, Once paid patreon subscriber to maltiez (ended months before this), absolutely consider his community interactions to indicate high levels of disdain towards the community, absolutely think that retaining and defending a confirmed malware developer and supply chain risk with a proven track record of violating your own policies to distribute malicious code using your own infrastructure is the wrong call on many levels.  

[icesharkk]

6 hours ago, Endeavour said:

This is a very good summary.
I would like to add:
The cheat mod was paid and designed to grief and destroy servers and there were some seedy practices around luring in customers. It was a cheat mod made with the goal of profiting off those with malicious intent.
The anti cheat worked by detecting whether a mod had more than 22% of its code obfuscated and would crash the game within 1-15 minutes with crash logs that hid the real reason of the crash
The anti-cheat was hidden inside a popular library mod.
True extent of collateral damage is unknown but it is presumed to be minor.

the cheat could be countered with server rollback and standard moderation procedure but the collateral damage could not be. the crashcodes could not troubleshot successfully until caves and caverns discovered the malicious code and many hundreds to thousands of man hours have been spent chasing these ghost crashes prior to that. I am hesitant to say the collateral damage is presumed minor. collateral damage is unknown but presumed significantly higher than cheater impact. 

people keep willfully ignoring the wasted time spent troubleshooting and chasing the crashcodes.

[Jack_Black]

Not sure how to reply here but I see the reply to obfuscated code was "Its not obfuscating anything- it just pointed the gun at a random mod in your crashlogs to make you think something other than itself was responsible. You wasted all those hours figuring out why a grab bag of mods was crashing because Maltiez felt he was righteous and prioritized his own ideologically driven ideal over keeping the mod out of controversial grounds. Nothing was hidden, nothing was actively attempting to hide itself in the crash logs, it's totally transparent to the enduser!" When it's not and I'm not tolerating lazy bad faith like this because I'm past my mid 30s and done with children.

Ridiculously pedantic.

 

You know exactly what's being said and what the problem is. It's not going away. The nature of the issue is trust and code that shouldn't have ran running on likely a five digit number of PCs. Dev response? "He's hired!"

Worst can of worms I've ever seen get cracked open and stood beside in a while but not ever. Games good, people care about it for that reason, so this is an especially shitty situation.

 

I won't be asking for a refund. Every time I think of this game I'll think of what I ran on my Pterodactyl panel and infrastructure, and how close to losing peace of mind I was lest it wasn't on a containerized VM and then I'll correspondingly recommend against buying this game off its own independent site and tell them the tale of Who Maltiez and his corroborating moderators were, What they were deflecting from and what the devteam decided was a character trait worth hiring, When the earliest deflections and ban waves began and Why people are so upset about it alongside why Malthiez himself claimed he did it.

That should really get them to put their card details into some thirdparty site that hired the guy. My firm recommendation will always be to torrent this game now.

Edited Sunday at 04:59 PM by Jack_Black

[icesharkk]

34 minutes ago, Vratislav said:

Originally, I had a title "Victims" and users of the hacked clients were labeled "No mercy." Then I considered it too subjective and changed it this way, without actually redrawing the first group. I should, yes.

its probably fine the way it is. the bad faith arguers already claim the upset parties are all just the impacted cheaters. no need to make their argument easier.

[icesharkk]

2 minutes ago, Jack_Black said:

Not sure how to reply here but I see the reply to obfuscated code was "Its not obfuscating anything- it just pointed the gun at a random mod in your crashlogs to make you think something kther than itself was responsivble."

Ridiculously pedantic.

its the only argument they have. between this and tyring to debate the definition of malware in increasingly convoluted ways. The incident as a whole is indifensible so they have to resort to small technicalities where "winning" the technicality can make the passive observer feel like the arguer won the argument. 

let me put this another way to outline a common problem with human nature in large debates like this. if you give someone 10 reasons why they should wear sunscreen at the beach a bad faith opposing party will not tackle your most convincing argument. they will pick your weakest argument, defeat it with rhetoric or hair splitting and then declare that you reasoning is flawed. a less canny audience will then feel that they have a point and your debate position has weakened far more than the simple loss of your weakest argument. Something like" well you said the suns rays are harmful but the suns rays are our primary source of vitamin D. Vitamin D is critical vitamin that your body needs to survive, without it you will die!

[Jack_Black]

8 minutes ago, icesharkk said:

let me put this another way to outline a common problem with human nature in large debates like this. if you give someone 10 reasons why they should wear sunscreen at the beach a bad faith opposing party will not tackle your most convincing argument. they will pick your weakest argument, defeat it with rhetoric or hair splitting and then declare that you reasoning is flawed. a less canny audience will then feel that they have a point and your debate position has weakened far more than the simple loss of your weakest argument. Something like" well you said the suns rays are harmful but the suns rays are our primary source of vitamin D. Vitamin D is critical vitamin that your body needs to survive, without it you will die!

I figured it out, guys! How to reply anyways.

 

This here is the real logical problem I see and why I don't step into these kinds of debates beyond saying what I have to say in a concise matter, why I believe what I do, and an appeal to see eye to eye- its going to be met by some smart-ass in bad faith trying to make a nonargument, sophistry and entirely pedantic takes that attack use of vernacular over meaning and what actually happened.

 

I'm a scientist. I don't deal with these people nor do I encounter them short of volunteering myself to the experience. We deal with facts of matter and foundationals. We do not handle those eager to abstract well and thus I choose not to participate in most of these types of drama. In this one however, coding ethics are near and dear to my heart and what Malth chose to do particularly on "feeling its right and justified- the only effective way to handle it" sees him nonperson'd in our field on a routine basis.

 

Anego, you have hired a proven outstanding example of how to get fired doing the one rookie move beginners with an ego lack the self restraint to stop themselves from doing in the first place. Trying to have your Maltware and eat it too isn't going to end well longterm. You're losing sales.

Edited Sunday at 05:07 PM by Jack_Black

[l33tmaan]

9 hours ago, Mushroomancer said:

The fact that we can't determine the scope of the issue is probably a big part of the reason that Anego isn't making a statement. Tyron is not just a member of the community and Anego is not working on some obscure bespoke software project, they're a legal entity and have to use language that reflects that. Of course, that's speculation. It's also speculation to say Anego supports malware and is going to turn Vintage Story into a smokescreen for malware. It's also speculation to say they rushed a statement and didn't cover their bases under the community pressure they were feeling. It's also speculation to say that they're carefully picking their language for legal reasons until they have consulted with counsel and have a clear path forward.

We can read in whatever level of charity we want all day.

This is unfortunately what happens when a company stops being transparent and open with its customer base - it invites baseless speculation and fiery hearsay. Of course, Anego should try and cover their asses from any legal liability here, but I just don't understand why they would do so for a guy who constantly talked down to his userbase and was completely unrepentant about all this reputational damage he caused the studio. Maybe he apologized behind closed doors and maybe Tyron actually believed him? For their sake, I hope his code is REALLY good and they announce that they will not be renewing his contract once all this is said and done. 

In practical steps forward, not only should the modDB REQUIRE hash matching for open-source mods, not only should it have a warning posted on closed-source mod pages, not only should all of Maltiez's mods be forked and worked on by the community instead of this limp-wristed 'custodianship account', but there should be an independent third party group of experienced cybersecurity experts who go through and ruthlessly tear mods apart to provide them with a public-use safety rating.
Is this excessive? Yes, of course it is. But this was an excessive breach of trust that necessitates swift, decisive action and I'm simply not seeing that yet.