[pacman-dev] mirrors.kernel.org serves chunked transfer downloads

[Dave Reisner]

On Mon, Jan 26, 2015 at 08:29:51AM -0500, Konstantin Ryabitsev wrote:
> On 25/01/15 11:53 PM, Dave Reisner wrote:
> > Would it be possible to turn off chunked transfer so that nginx serves a
> > Content-Length header? This is highly preferrable -- the overhead in
> > calculating the response size is that of a simple stat syscall. In
> > addition, knowing the response body size up front potentially allows
> > downloaders to match the remote file size against local metadata, as a
> > method of detecting corrupted or tampered-with files.
> 
> Thanks for the suggestion -- I turned it off. It doesn't make sense to
> have it on a static-only site.
> 

Great, thanks!

> > Also, I offhandedly highlight that your cache varies on querystring. Do
> > you really need to do this for static content? This actually works
> > against you in a the case of a DoS attack -- a malicious user could
> > potentially evict a large amount of the cache by flooding it with
> > variations on a single large blob. If mirrors.kernel.org shares a cache
> > with other sites, it might be a Bad Thing™. Actually, if the Varnish
> > instance used for mirrors.kernel.org is shared with other subdomains,
> > you might consider disabling it entirely for files below
> > mirrors.kernel.org. Relying on the kernel's page cache alone seems like
> > a better strategy.
> 
> Using varnish is a temporary but, unfortunately, necessary measure as we
> work with upstream to fix FS corruption problems we're seeing with
> dm-cache, libvirt and xfs.
> 
> https://plus.google.com/+KonstantinRyabitsev/posts/6YRFhcKKipP
> 
> Varnish+ssd is helping us last things out until the FS corruption is fixed.

Understood.

Thanks again for the quick response!

Cheers,
dR