but legit 10G cables for Thunderbolt 3 drives, even
Make sure you are not using Thunderbolt cables .. some will support USB as slow speeds but TB and USB are different technologies and the cables are significantly different (one of the problems with the stupid decision of putting TB and USB on the same connector).
I doubt switching file systems etc will make any difference here, nor physical drives. Its probably that you just have a LOT of music files compared to others. I have approx 6500 songs on my Samsung T5 SSD (formatted as regular FAT32) and it has always worked perfectly.
I suggest removing everything from the drive (reformat/repartion is probably best), then adding songs 5000 at a time, and see if/when you notice things getting unstable. It sucks, but you might have to make a subset of your music collection (or be prepared to swap drives around).