Main » Discussion » It's not a bug, it's a feature! » New reply
    Alert
    You are about to bump an old thread. This is usually a very bad idea. Please think about what you are about to do before you press the Post button.
    New reply
    Post help

    Presentation

    [b]…[/b] — bold type
    [i]…[/i] — italic
    [u]…[/u] — underlined
    [s]…[/s] — strikethrough
    [code]…[/code] — code block
    [spoiler]…[/spoiler] — spoiler block
    [spoiler=…]…[/spoiler]
    [source]…[/source] — colorcoded block, assuming C#
    [source=…]…[/source] — colorcoded block, specific language[which?]
    [abbr=…]…[/abbr] — abbreviation
    [color=…]…[/color] — set text color
    [jest]…[/jest] — you're kidding
    [sarcasm]…[/sarcasm] — you're not kidding

    Links

    [img]http://…[/img] — insert image
    [url]http://…[/url]
    [url=http://…]…[/url]
    >>… — link to post by ID
    [user=##] — link to user's profile by ID

    Quotations

    [quote]…[/quote] — untitled quote
    [quote=…]…[/quote] — "Posted by …"
    [quote="…" id="…"]…[/quote] — ""Post by …" with link by post ID

    Embeds

    [youtube]…[/youtube] — video ID only please
    Thread review
    Kakashi When did you change from 1GB to 1.5GB and was there a noticeable change?
    Screwtape Back when I was researching delta-patching for byuu's BPS patch format, I had a workstation with 3GB of RAM, and delta-patching got pretty RAM intensive, especially when implemented in Python. Very quickly I learned to turn off swap, because the OOM-killer killing my out-of-control test program returned my computer to a functioning state much, much more quickly than trying to kill it manually, or even just hard-rebooting.

    Unfortunately, modern Linux kernels seem to try a lot harder to fit the working set into memory, paging every memory-mapped file (like executables) in and out of memory before it will resort to invoking the OOM-killer. That's probably an improvement for workloads whose working set just creeps up above available RAM occasionally, but my workloads tend to be "absolutely stable" or "exponentially climbing to infinity", so I miss the OOM-killer. I've heard it might be possible to get back the old behaviour by tweaking a parameter named "swappiness", but I haven't tried it.

    > And that's purging. Very destructive, and that's why we only do it for immutable data.

    Purging is just a trivial case of paging: when a page of memory has not been modified since it was read from disk, it can just be discarded. If the page *has* been modified, it needs to be written out before a new page can replace it. Both cases still need another disk read to fetch the replacement page, though, so they still make the computer slow.

    > Disabling the Swap File should be resulting in Windows closing programs as you run out of memory/programs crashing, with the entire purpose of the Swap File being to avoid that behaviour.

    Windows and Linux behave somewhat differently here. Most applications request a bit more memory than they actually need, just in case. Windows refuses the allocation request if there is not enough room in RAM + the swap file for it, while Linux just always says "yes". Odds are the app will never use all the RAM it asks for, so Linux figures nobody will call its bluff. For example, on my Linux system apparently Firefox has "allocated" 26GiB of RAM, but this machine only has 8GiB of RAM and 1.5GiB of swap. In comparison, Firefox seems to be actually using about 2.5GiB.
    Nicholas Steel Yeah I was confused at first too. Disabling the Swap File should be resulting in Windows closing programs as you run out of memory/programs crashing, with the entire purpose of the Swap File being to avoid that behaviour.

    Purging is typically what video games do, even if you have plenty of RAM to store everything in because devs weirdly code an upper limit to how much RAM a program can use or something? (I can understand purging stuff from VRAM as needed since it's more difficult and more expensive to upgrade the capacity of VRAM)
    CaptainJistuce
    Posted by sureanem
    The whole point of the thread was that the setup fixes a whole unrelated issue through a very ugly method, so then I'd get those problems back.

    You know, most people consider the problem to be "the applications I'm using randomly crash" as opposed to "the applications I'm using are stable and do what I ask of them"
    BearOso
    Posted by funkyass
    the MBTF for modern SSD is like on the order of millions of hours. my nvme is 2 million hours.

    so enable swap, it'll be more proactive with swapping long-untouched pages to disk, and you'll get less IO spikes. Hell all modern OS's work on the assumption of a swapfile

    MTBF is not the figure you're looking for. You want write endurance. Swap files are terribly random, and contain tons of small writes, so the write amplification on a SSD will be an order of magnitude greater than normal access.

    Swap is a really bad solution with terrible performance, and its use even in the early days when RAM was smaller was non-ideal. Nowadays RAM is plentiful and difficult to keep full. Swapping makes computing miserable.
    funkyass the MBTF for modern SSD is like on the order of millions of hours. my nvme is 2 million hours.

    so enable swap, it'll be more proactive with swapping long-untouched pages to disk, and you'll get less IO spikes. Hell all modern OS's work on the assumption of a swapfile
    ‮strfry("emanresu") Well, I just said they got paged out, never that they got swapped, didn't I?
    > it pages the applications in and out like crazy
    Or, well, shit, I did actually say that, although not in my original post.
    > They don't get swapped out, but the static code/data segments do.

    Yes, you're right, that's correctly called purging. It causes trouble just the same, since Linux would sooner evict all the caches and executable segments than kick OOM-killer into drive. Which is understandable, except if you happen to be a desktop user, since you can't kill runaway processes if that happens. (except with the sysrq key that ships disabled)
    Kawaoneechan To/from the original EXE on disk, you say?

    That's not "swapping" though. That's purging. I see it in my favorite early 90s DOS game engine SCI all the time: when the game runs short on memory (of which it has a limited amount easily measured in megabytes), it'll start unloading assets that were previously loaded. The next time that particular asset is required again, the dance repeats. Importantly, these assets do not change. Simply unloading them loses the game nothing and earns it a few precious KB to fit the stuff it needs right now. Now, script assets are a slightly different matter considering they have variables. That's why their variables are stored in a totally different piece of memory and have to be manually unloaded by the script. I think Windows can do the same trick with program resources and I hope to Fruit Jesus you understand what I mean by "resources".

    And that's purging. Very destructive, and that's why we only do it for immutable data. What you describe is removing the static code and data segments of applications -- the parts that don't change -- from RAM, and having them reloaded when needed. That's purging.

    Swapping is taking entire pages of RAM, writing them to the swapfile with some bookkeeping notes ("which page # was it" and such) to make room for new stuff, be it all-new allocations, or pages that have to be swapped in. The contents of these pages worth of RAM don't matter much -- they could be program images including their BSS segments, they could be mallocs, it doesn't matter. If a page isn't locked, it's eligible for swapping.

    Are we clear on the long-established meanings of these words?
    ‮strfry("emanresu") They don't get swapped out, but the static code/data segments do.
    Kawaoneechan
    Posted by sureanem
    To where they're stored on the disk. These are applications, so its layout in RAM can be perfectly reconstructed from the executable.
    I'm sorry, what? What about variable data? What about dynamically-allocated memory?

    Actually, don't bother trying to answer that.
    ‮strfry("emanresu") To where they're stored on the disk. These are applications, so its layout in RAM can be perfectly reconstructed from the executable.
    The issue occurs when someone wants to use that library but the OS gave away its physical RAM. Then it has to tell another library to piss off back into the hard drive so the first library can get loaded. Then the second library needs to get used... Imagine a bunch of executable files playing musical chairs, that's the general gist of it.

    Dude, just enable the goddamned swap. ... Unless you're using noname Chinaware junk with bottom-of-the-barrel flash ICs that were rejected for a reason, your average SSD will be able to stand a few petabytes of writes before giving up the ghost

    Can't it reach those numbers if you swap stuff in and out rapidly though?

    Eh, I'd rather not play the partition game. That stuff makes me sweat bullets, honestly. The whole point of the thread was that the setup fixes a whole unrelated issue through a very ugly method, so then I'd get those problems back.

    And also, whenever it does hit swap it tends to make everything sluggish just the same, so I'm not even sure the gains are that big. Then I could just enable the OOM-killer sysrq command and call it a day without having to mess around with partitions.
    Kawaoneechan Good question. Where *does* it page the applications into and out of like crazy?
    funkyass wait, if you dont have swap enabled, then where are the applications being swapped to?

    BearOso
    Posted by sureanem
    (Enabling swap isn't recommended on SSDs, and I could just head down to the store and buy $50 of RAM instead of messing around with partitions)

    I agree. If you're regularly hitting the RAM cap, you need to buy more RAM, not mess around with swap. RAM's very cheap right now anyway.
    tomman Dude, just enable the goddamned swap. Dunno about your beloved *BSDs, but on Linux I've been using swap FILES since I learned you can actually have unholy Windows-isms on your Linux.

    Sure, you can always buy more RAM, but still swap is there for very good reasons.

    > Enabling swap isn't recommended on SSDs

    Ah, that old myth that seems to still be around (this used to be true with early SSDs which were nothing but USB sticks with an ATA controlled bolted on, basically). Unless you're using noname Chinaware junk with bottom-of-the-barrel flash ICs that were rejected for a reason, your average SSD will be able to stand a few petabytes of writes before giving up the ghost.

    I can't get you guys, what the hell do you do with your computerizers!?!?!??!
    ‮strfry("emanresu") You joke, but it exists.

    (Enabling swap isn't recommended on SSDs, and I could just head down to the store and buy $50 of RAM instead of messing around with partitions)
    creaothceann Put it on a RAM drive!
    CaptainJistuce
    Posted by Nicholas Steel
    Sounds to me like you should enable the Swap File.
    QFT, as the kids say.
    Nicholas Steel Sounds to me like you should enable the Swap File.
    ‮strfry("emanresu") On my current system, I only have four gigabytes of RAM. As you might imagine, this causes all sorts of trouble when the RAM runs out. Since I don't use swap, it pages the applications in and out like crazy. This causes inputs to get processed extremely slowly, which in turn makes it impossible to kill processes. Since the memory isn't _technically_ exhausted, the OOM killer doesn't trigger.

    One day, I got fed-up with this order of things and downloaded earlyoom. When your free RAM drops below 10%, it kills a big process at random. You might point out this is dreadful, since it can close your Firefox with 20 tabs without any warning.

    But on the flip side, it also closes your Firefox with 20 tabs without any warning. This solves the whole tab explosion problem, and makes you a lot more wary about opening links you don't need to read. Overall, it's been a great boon to my productivity, even though it's an ugly hack fixing an ugly hack fixing an ugly hack fixing a configuration that should not exist and could be trivially fixed by just going out and buying more RAM.

    So what's your stories of horribly broken bugs turning into features?
      Main » Discussion » It's not a bug, it's a feature! » New reply
      Get an ad blocker.