My guess is that we're not allocating buffers for the network transfer in the filesystem case. Because the Loader object is passed off directly to the displaylist, it never really gets deallocated, and we can't garbage collect it. That's why making an exact copy of the object with LoadBytes doesn't incur the same overhead.
I've been chatting with the Scout team about it, and I gave them a copy to see if they can glean some more insight into what those Uncategorized buffers are.