[pacman-dev] bug in pacman 3.3.2 retrieving of repos?

Mon Oct 12 12:00:48 EDT 2009

Am Montag, den 12.10.2009, 17:06 +0200 schrieb Xavier:
> On Mon, Oct 12, 2009 at 4:54 PM, Marc - A. Dahlhaus [ Administration |
> Westermann GmbH ] <mad at wol.de> wrote:
> > Am Montag, den 12.10.2009, 16:36 +0200 schrieb Xavier:
> >> On Mon, Oct 12, 2009 at 4:12 PM, Marc - A. Dahlhaus [ Administration |
> >> Westermann GmbH ] <mad at wol.de> wrote:
> >> > Well it could be a local problem with my environment of course, but
> >> > shouldn't 3.3.0 then suffer from the same problem then? As there were
> >> > some fiddling with the libfetch download code in pacman on version 3.3.1
> >> > and 3.3.2 i thought it might be related to that.
> >> >
> >>
> >> I already answered this question, see my previous mail.
> >> And from libfetch man page :
> >>      If the `i' (if-modified-since) flag is specified, the library will try to
> >>      fetch the content only if it is newer than last_modified.  For HTTP an
> >>      If-Modified-Since HTTP header is sent.  For FTP a MTDM command is sent
> >>      first and compared locally.  For FILE the source file is compared.
> >>
> >> You the http header you need to check is If Modified Since
> >
> > Had the mail in response to Dan open the whole testing and spooted your
> > mail after i send.
> >
> > I checked the lighttpd behaviour again:
> >
> > 1. try:
> > Last-Modified: Mon, 12 Oct 2009 13:39:50 GMT
> >
> > a touch on the file later on 2. try:
> > Last-Modified: Mon, 12 Oct 2009 14:41:48 GMT
> >
> > Looks sane.
> >
> > Might be that lighttpd headers are distinct from apache httpd. The only
> > change is that lighttpd sends the local time header "Date" after the
> > "Last-Modified" header. Could it be a bug regarding ordering of Header
> > lines in libfetch?
> >
> > Have no gdb ready on this box, but will look into it some more tomorrow
> > when i get to it.
> >
> > Marc
> >
> >
> >
> 
> >From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
> 
>  The purpose of this feature is to allow efficient updates of cached
> information with a minimum amount of transaction overhead.
> 
>       Note: The Range request-header field modifies the meaning of If-
>       Modified-Since; see section 14.35 for full details.
> 
>       Note: If-Modified-Since times are interpreted by the server, whose
>       clock might not be synchronized with the client.
> 
>       Note: When handling an If-Modified-Since header field, some
>       servers will use an exact date comparison function, rather than a
>       less-than function, for deciding whether to send a 304 (Not
>       Modified) response. To get best results when sending an If-
>       Modified-Since header field for cache validation, clients are
>       advised to use the exact date string received in a previous Last-
>       Modified header field whenever possible.
> 
>       Note: If a client uses an arbitrary date in the If-Modified-Since
>       header instead of a date taken from the Last-Modified header for
>       the same request, the client should be aware of the fact that this
>       date is interpreted in the server's understanding of time. The
>       client should consider unsynchronized clocks and rounding problems
>       due to the different encodings of time between the client and
>       server. This includes the possibility of race conditions if the
>       document has changed between the time it was first requested and
>       the If-Modified-Since date of a subsequent request, and the
> 
>       possibility of clock-skew-related problems if the If-Modified-
>       Since date is derived from the client's clock without correction
>       to the server's clock. Corrections for different time bases
>       between client and server are at best approximate due to network
>       latency.
> 
> 
> Are your client and server on two different box ? If so, check their clocks.

They are on different hosts and different carrier networks.
They get time updates via ntp from the same source.
System times are sane on this boxes.

> Anyway we are using ust.mtime from libfetch. I would think this comes
> from http Last-Modified and thus should not cause problems.
> But I am not sure. I am still trying to understand how this stuff works :)

Thanks Xavier, this information was what was missing in this puzzle for
me. I sniffed the http headers with tshark and checked what pacman asks
for per header and what lighttpd is sending back to pacman in response.
This is definitely a lighttpd bug. It answers with "304 not modified"
response even if the file was modified because of some missing checks. 

Sorry for the noise.

If anybody is interested, this is the corresponding fix for
lighttpd-1.4.23 which will be in 1.4.24 when it gets released:

http://redmine.lighttpd.net/projects/lighttpd/repository/revisions/2643/diff/branches/lighttpd-1.4.x/src/http-header-glue.c?format=diff&rev_to=2408

Marc