I understamd, The snippet of code I wrote is just to describe how I am getting it, we are not using that in the specific, in a real case I would be loading different files, not the same one over and over. I believe the solution to the mistery is in what you said: "I think what you're seeing is that we're optimizing in the case where we're grabbing the object off the local filesystem.".
So what I am seeing is not an increase in memory for remote files, but actually, a lower memory usage for assets loaded locally due to optimization for that particular case.
Thanks again for double checking on this.