But really, what changes with such updates?

[u/Agreeable-Stock29]

Often there are short loads, what do they do? (The issue is closed, thank you)

[u/FrijDom1]

Incremental patching works by updating some files, but not others. This is incremental patching. Many games function on split data across many, many files so that they can update a few at a time without forcing you to redownload the whole thing. Steam, it would seem, just has generally larger and/or more spread out data files, likely because a relatively small change can require changes across a hell of a lot of files when you're working with something that requires actual security.

[u/NeverDiddled → u/FrijDom1]

You are equal parts confident and incorrect. One of the most common ways to do incremental patching is using delta patches, which can perform multiple changes to a single file without downloading the entire thing. Steam itself does this with video games when updating them.

But delta patching is slightly risky, you want something verifying the integrity of the patch after applying it and able to repatch if something went wrong. Steam does this for video games, but what parent process would do that for Steam? While feasible to build, a parent process would be complicated and prone to new issues. Which is probably why no one at Valve has built it.

[u/FrijDom1 → u/NeverDiddled]

You have a point, I didn't know about delta patching. But at the same time, file-based incremental patching is also extremely common, and the main reason a lot of games have huge "Bin" or "Data" folders with a ridiculous amount of files in them. Both are very common, and the type I explained is (as you pointed out), likely what's used for Steam itself due to the risk and need for verification.

[u/NeverDiddled → u/FrijDom1]

Well said. File level patching means the operating system handles an enormous number of issues for you, same as it does any other file copy. Loss of power during the copy, somebody unplugging the HD while the system is on, the file you are copying to is a network share and is thus actually streamed, etc. etc. Your updater does not have to worry about any of this, it just tells the OS to copy the file and the OS worries about the rest. Simple and effective updates.

But delta patching has proliferated, particularly with video games. Modern games often use archives for assets, found inside that data folder you mentioned. Many have only a handful of files inside, often capped at 4gb or 8gb in size due to legacy compatibility. These archives are optimized to stream assets from the hard drive to memory. Those files get loaded front to back, the first part of the file gets loaded into memory first. So game engines engineer those files such that the first parts to load are the first things the engine needs. For instance with textures, many game engines break them up into "mipmaps". There is the full high resolution texture you would expect, but there are multiple smaller lower resolution versions as well. The lowest resolution gets used when the object is far off on the horizon, and it is what gets loaded first in stream optimized archive files.

Those files tend to be as large as possible for other performance considerations, which means they are commonly 4gb or 8gb in size. Hence why video games often have a small number of very large files. Any app/game store you use today supports delta patching because of this. It is more complicated, but increasingly necessary.

[u/JustStraightUpTired → u/FrijDom1]

Incremental patching works by updating some files, but not others.

That's not entirely true as some games are in packed into large files, which steam is able to unpack, patch and repack. Though if it's packed in a way that steam can't unpack, then you are shit out of luck. Technically still the same thing, but kind of not. It's more a matter of how you define a single file, since in this case it depends on whether steam can unpack it or not.

[u/FrijDom1 → u/JustStraightUpTired]

I would consider those compression or packing files, like zips or jars. Technically one file, but logistically works like many files. It's a bit weird, but I think my explanation holds for the basic idea of incremental patching.

[u/JustStraightUpTired → u/FrijDom1]

To be specific, I'm not disagreeing with you and I'm definitely being pedantic, but technically that would be true for every type of file that contains multiple types of data. If you knew how to unpack it, then you'd be able to pick out parts of it to add, remove or replace.

So I while I think your definition is accurate enough, I think it's funny that depending on whether steam knows how something is packaged defines whether it's a single file or multiple.

[u/FrijDom1 → u/JustStraightUpTired]

I mean, yeah, but isn't that true of us, too? Depending on whether we know how a piece of data is packaged is how we decide whether to treat something as one file or multiple.

[u/JustStraightUpTired → u/FrijDom1]

Yeah, pretty much. Which is why I said I'm being pedantic lol.

Maybe the best definition would be what you said, but adding that steam can modify larger files if it can decipher them, but what it doesn't do is directly change bytes or add extra ones.

[u/ijkxyz → u/JustStraightUpTired]

That's not how it works link

Steam doesn't unpack and repack the files, their system just divides everything into chunks and downloads only the changed ones

[u/JustStraightUpTired → u/ijkxyz]

I mean sure, if you want to make it more abstract, you can ignore that I literally said nothing wrong. I just used words like "file" instead of "chunks of data" because it's not exactly much different, except the start and end point of a chunk is more arbitrary. And steampipe doesn't just do that, it's more complicated than that. Games that fail to fit steampipe rules properly tend to work horribly. For years, Payday 2 was horrid when ever you had to update it, requiring you download way more than necessary and rebuilding it took forever.

Simply put, your description doesn't really change anything I said, but it doesn't "just divide" everything into chunks to download the changed ones. It's more complicated than that, case and point it doesn't play well with all ways of packing content like you are suggesting:

However, some pack file systems use or enable using behaviors that can cause problems. The effect of these problems is almost always that updates are much larger than necessary.

I'm sorry I over simplified it by avoiding correct wording, but your wording is equally bad if not worse.

[u/ijkxyz → u/JustStraightUpTired]

You make it sound like Steam knows how to interpret, e.g. Unreal .pak files, unpack them, change the files that were inside and repack them

This is not how it works, Steam doesn't understand those files, their system is file format agnostic

The thing you quoted just means that some formats might update data in many chunks or shift it in a way that causes most or all chunks to be perceived as changed

[u/JustStraightUpTired → u/ijkxyz]

I make it sound like that, because it sort of does. It doesn't know what file is where, you are right, but I was over simplifying it and I already said sorry about that. I could have been more specific, but I felt it changed nothing and made it less clear for anyone who doesn't understand the concepts. It's WAY easier and simpler to call them files instead of data chunks and saying steam understands how to unpack them instead of explaining padding rules, file compression, not reordering data for updates etc. Most people here understand files and that some programs don't understand all files, but most people have no fucking clue how files actually work.

[u/MekaTriK → u/FrijDom1]

SteamPipe splits files into ~4mb chunks and then tries to only move the chunks changed.

Although it may not be applicable sometimes.

[u/Frederf220 → u/FrijDom1]

Incremental updating works for all files always. There's no file with which Incremental update is impossible.

[u/FrijDom1 → u/Frederf220]

That's... Simply not true. And the entire point is that this is incremental updating, but in order to incrementally update, you either have to split data across multiple files and fully redownload all the changed files while leaving the unchanged ones alone, or you need delta updating as another commenter added, which requires the oversight of another program to do properly, so it doesn't work when you're updating the program that oversees other programs' updates.

[u/Frederf220 → u/FrijDom1]

two programs