File Fragmentation, Purposeful, or?

[Ceyarrecks]

First off, I am unsure where this question ought be posted, Hardware, or Software, as it seems the underlying problem could reside in both.
Now to answer the fact of "Fragmentation" and "Purposeful" in the same sentence, I would refer one to the attached picture of a recent Vopt defragmentation session:

ok! I /was/ going to attach here a whoopingly super-huge 1.5MB .jpg picture; but, something vaguely and ambiguously went wrong with the error: "Oops! We ran into some problems." gee, i wonder WHAT those problems could be,--mmmmmmm--nope, mind reading did not help. Since Cookies and JS are both allowed to function on this site, there is probably issue with site since there was no admittance to what the "problem" was; had their been, maybe I could have investigated further to find if i am at issue or advised of what was found to those that could have done something about it.

https://imgur.com/SABynVb

Anyway.
as noted by the above image a relatively speaking-small 9 "square" file was initially sprayed over the entire drive taking up MULTIPLE sectors!

Something (or someone) made the choice to "get every file onto the drive as fast as possible! quick! panic!" as opposed to taking a whole one second of time, reviewing the drive for the last file contiguously copied file, and then placing the new file on the next following or shared sector available.

Which of course, the whole industry, and all whom work in the IT field know full well the benefit of an unfragmented or contiguous drive.
Rather akin to the benefit in time of having your entire Sunday outfit in ONE location as opposed to one sock in sock drawer, other sock somewhere in living room, shirt on floor in basement, pant somewhere in garage, tie,.. tie,.. now where it the color-matched tie?! oh forget it, will just wear a purple poka-dotted tie since it is right here on the lamp shade,...

As the fragmentation of SSDs problem noted in the recent SN! #807 episode, so similar to HDD fragmentation;

The question, then is, WHY the policy of "panic and write ANYWHERE"? Why is there not a policy of "review drive, find last contiguously written file, then place new file immediately after"?

Certainly even those who want their file RIGHT! NOW! can wait that one second for the obviously grand benefit of maintained performance? right?
or
am I expecting too much again?

 

[rfrazier]

Here's what I think. It may even be factual. Somebody can jump in if I munge it up.

Fragmentation should have little effect on an SSD regardless of what a defrag utility says. In fact, my understanding is that you should almost never defrag an SSD. The block or sector numbers don't have any constant association with the individual flash cells. And, an SSD can read any block or sector equally as fast as any other. And, jumping to file fragments has minimal overhead. The SSD has to erase an entire page at a time to write, so that may affect how data is laid out when it's written.

But, with a HDD, it's different. Reading and writing a HDD is much faster at the front of the drive due to the physical geometry. So, the OS will preferentially try to put files there first as far as I know. It especially will try to put the Windows files at the front. So, say you use of the 1st 20% of your HDD installing the OS. Then you put a bunch of videos on and use the next 30%. Then you add games and use another 10%. Then you add email and documents and use another 20%. So, you're up to 80% usage. Then, say you delete some games and videos and put others in of different sizes. Or say you download a bunch of new game data and the size of some of the games expand. Let's say you get lots of email every day and the size of your email database expands, until you compact it and it shrinks. As you can imagine, there may be times when you delete data and make a hole, but then you add more data. The system may plug the hole but the file gets fragmented and part of it is put in another place. Now imagine these processes going on every day dozens or hundreds of times. That's essentially how the drive gets fragmented.

Bottom line, I think, is that the system will preferentially place data toward the front of the drive, and if that ends up fragmenting files, then at least some of the data will be near the front. Also, even if it only ever wrote to free contiguous space, there would eventually be holes of unused space which you would then have to use or sacrifice.

So, it's an almost unavoidable consequence of running the machine. Others feel free to jump in and clarify.

Disk Defragmentation Explained - Defrag Hard Drive - Speed Up PC

Disk Defragmentation & Drive Optimization as Fast As Possible

Fragmentation and Defragmentation

May your bits be stable and your interfaces be fast. Ron

 

[PHolder]

In most OSes if you only ever wrote files and never deleted any, there would be virtually no fragmentation. I say virtually none, because some OSes decide to place the file system control data in the center of the disk assuming that is more efficient than placing it at the beginning or end. Also the file system control data grows as the number of files grow, so it would end up being written in multiple chunks. Additionally, even if the user never deleted a single file, the file system control data will need to have changes as more files get added, and it might delete some of that after it writes it to a new location, because write and then "link old to new" is safer for your data than "overwrite in place."

Based on what I just wrote, and knowing that the OS uses many temporary files, and most users also make changes, it becomes clear that fragmentation is an inevitability. Most users make many deletions without realizing it while using the PC. "Save often" creates a new file which leads to the old file being renamed to a backup and the older file being deleted. It's better if the OS has a plan to deal with HDD [de]fragmentation than it is to assume it just never happens. Luckily, on SSDs, there isn't much penalty for fragmentation, so the OS doesn't have to plan for any optimization there.

 

[Tig77]

I would recommend that you give PerfectDisk a try - https://www.raxco.com/home/products/perfectdisk-pro

It has a technology that runs in the background that prevents most fragmentation.

From their website:

Fragmentation Prevention with OptiWrite®​

PerfectDisk's OptiWrite technology prevents most fragmentation on your drive before it occurs, which prevents your system from slowing down. OptiWrite detects when Windows is going to fragment files and intelligently redirects I/O to stop the fragmentation from occuring. System performance is maintained and the need to utilize CPU and Disk I/O resources to defragment files is greatly reduced. Because less defragmentation is needed, there is a direct benefit in the form of energy saved in both CPU usage and in the reading and writing to a disk. It saves users both in energy costs and in the time required to defragment a file system.

I tried it myself for the 30 day free trial and it does run seamlessly in the background and takes almost no memory and doesn't slow down your system at all. It keeps track of how many fragments it prevents too. Worth checking out.

I would think that this kind of technology should be built-in to modern OSes, but I don't think it is.

Shawn

 

[pmikep]

I have removed my content

 

[PHolder]

I doubt this supposed fragmentation is THE issue

I haven't listened to the issue, but there is thinking that running a defragmentation pass on an SSD invokes TRIM on sectors and that the TRIM pass can be helpful for SSD performance (especially helpful for fuller SSDs.)

 

[rfrazier]

if I recall correctly significant improvement was observed after only reading too.

This comment prompted me to look back at the 1st page of the threads in the Readspeed Results forum. I've been following some of the threads and contributed to some, but certainly don't remember all of them. I also didn't have time to read every thread again. But my very small and unscientific survey showed that 5 threads from that 1st page mentioned reading and writing for balky SSD remediation (IE Spinrite Level 3 or 4). 3 threads mentioned read only for remediation. Some threads mentioned both. But, it appears that there is evidence that both reading as well as writing can have beneficial effects.

I'm not an SSD expert and don't claim to be. The idea that writing helps doesn't really surprise me, especially when all the trillions of little capacitors (what a scary thought) drain anyway and the drive's firmware resets the "threshold" voltages to reconsider what is a 1 or 0. It makes sense that writing would reset everything and top off the capacitor with some nice juicy electrons and nice fresh thresholds. Why reading has an effect (not whether it does affect [hope I got those two words right] ) is somewhat more of a mystery. But, as @Steve always says about SpinRite in general, maybe it jars the controller into realizing the error levels are too high when attempting to read and it does SOMETHING. Another thing I find scary (when I found out a few years ago) is that drives do not verify after write. If an app wants to know a sector was written properly, it has to read it and, hopefully, the drive will correct any errors.

May your bits be stable and your interfaces be fast. Ron

 

[pmikep]

I have removed my content

 

[rfrazier]

We discussed this in another thread a while back the title of which I don't remember. But, AFAIK defrag in Windows 7 is not smart enough to know that it should "optimize" IE run TRIM rather than defrag an SSD. Therefore, the scheduled defrag should be manually turned off if it's not automatically turned off. Some SSD makers provide monitoring utilities, IE Samsung Magician, which will allow you to trigger an optimize cycle. I believe Windows 8 and 10 are better about this. I don't know about Linux and Mac.

May your bits be stable and your interfaces be fast. Ron

 

[DanR]

AFAIK defrag in Windows 7 is not smart enough to know that it should "optimize" IE run TRIM rather than defrag an SSD

Actually, it supposedly does: https://www.google.com/search?client=firefox-b-1-d&q=does+Win+7+defrag+SSD

That said, it may not be flawlessly perfect at not defragging a SSD.

 

[rfrazier]

Yes ... BUT ... I decided to do a little test. The C drive on this Windows 7 laptop is an SSD. I started Disk Defragmenter. The text on the program says "Only disks that can be defragmented are shown." It showed the C drive. I told it to analyze the drive. It reported 22% fragmentation. I started defragmentation and it obediently let me. I immediately stopped it. There is no option to optimize anywhere in sight.

SO, I recommend that all Windows 7 users go into Disk Defragmenter and turn off scheduled defragmentation for any SSD's you have. If you want to optimize or trim your drive, you may have to figure out a different way to do that. If your defrag utility is newer or smarter than mine, you may have other options.

May your bits be stable and your interfaces be fast. Ron

 

[DanR]

Ron, there is a notable difference between a manual defrag and a scheduled defrag, at least on my Win 7 Pro box.

Disk Defragmenter shows C: (SSD), E: ( Spinner), and G: (USB Spinner).

C is shown as never run; E: 0% fragmented; G: 1% fragmented; with E: and G: last run 2/24/2021 (the schedule is once a week).

The scheduler only shows the E: and G: spinner drives as schedule-able.

So in my case:
- I could manually defrag the C: SSD (I started and immediately stopped a manual defrag of the C: SSD to verify). But I would not!
- I cannot schedule a defrag of the SSD; I can only schedule the spinners for defrag.

The Scheduler keeps the spinners defragged, but it does not touch the SSD. It is doing just what it should!

 

[Barry Wallis]

I thought this discussion in Tom's Hardware from 2017 was interesting.

 

[rfrazier]

@DanR You make an interesting point. I went to my desktop which has an SSD and some spinners. I looked at the defrag schedule which was on. I clicked on select disks and there was a check box which says select all which was on. But, the SSD was not listed as you mentioned. So, yes, it looks like it's catching the spinners only. Nice call. Still, that discussion that @Barry Wallis mentioned says that Windows doesn't always detect the SSD's properly. So, It wouldn't be a bad idea to check your defrag schedule and make sure that SSD's are excluded. I still don't think that Windows 7 defragger will optimize / trim the SSD.

May your bits be stable and your interfaces be fast. Ron

 

[DanR]

I thought this discussion in Tom's Hardware from 2017 was interesting.

I agree! Good valuable reading there

Windows doesn't always detect the SSD's properly. So, It wouldn't be a bad idea to check your defrag schedule and make sure that SSD's are excluded.

Agreed! Win 7 is supposed to work this way. But, per what I have been reading, it may not always. So checking is wise.

 

[Dr.Flay]

Shame nobody really dealt with the question.

Q: How / Why does file fragmentation happen ?
A: Modern filesystems will try to allocate a space big enough to put a file if it knows how big it is, such as copying from one media to another.
However when downloading, extracting or writing a new file, the OS does not know how big it will be, so just puts data in the first available space until filled then moves on.

The common setup of 1 partition to put everything in makes it unavoidable.
Usually you get a download of software to install from a site, so you have a fragmented installer file in the cache then moved to your DL folder.
You then run the installer which extracts or downloads a load of other files which now become fragmented.
The installer finishes placing all these still fragmented files into the programs folder, and then quits but likely leaves the extracted files in a windows temp folder.
When moving files around the same partition, the filesystem does not move the file but changes where it says it lives.

You can reduce the amount your system becomes fragmented, by using a dedicated TEMP data partition where all browser caches and windows temp folders etc. are put, and also having your main software install location also be its own dedicated partition.
Temp cache files that are moved or copied over to the program folder are of a known size, so if that folder is a different partition it will try to use the first available space that can fit the whole file.

 

[AlanD]

It's an interesting thought that, given that SSD's can do multiple reads almost simultaneously, would the response times be faster for a (nicely) fragmented file where parallelism could be a benefit in retrieving multiple concurrent pages rather than having to wait for each block to be read sequentially.