Thanks for extensive explanation. What do you thing would be the best? Is
it asking IETF for comment? Eventually it would be nice to match browsers
behavior but not at expense of missing RFC guidelines. It's unclear whether
to fix curl or major browsers.

On Tue, Sep 27, 2016 at 1:30 PM, Ray Satiro via curl-library <
curl-library_at_cool.haxx.se> wrote:

> On 9/26/2016 9:34 PM, Sergei Kuzmin wrote:
>
> I recently tried to automate login into av.by. After login site sets .
> av.by cookies (note leading dot). However these cookies are stored as
> av.by (strict domain) so it doesn't work for sub-domain in consecutive
> call. Is it some security measure? Is there some well known workaround? At
> present I postfix the cookiejar after the call:
> sed -i -r 's/^av.by\tFALSE/.av.by\tTRUE/' av_by.cookiejar
>
> Interesting parts are in yellow:
>
> curl http://av.by/login -H 'Origin: http://av.by' -H 'Content-Type:
> application/x-www-form-urlencoded' -H 'Cache-Control: max-age=0' -H
> 'Referer: http://av.by/login' --data 'login_form_csrf=1111111111111
> 11111-1111111111111111111111111111&login=11111&password=1111
> 111111&submit=%D0%92%D0%BE%D0%B9%D1%82%D0%B8&remember=0&remember=1' -H
> 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language:
> en-US,en;q=0.8,ru;q=0.6,pl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H
> 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36' -H 'Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
> -H 'Connection: keep-alive' --compressed -b av_by.cookiejar -c
> av_by.cookiejar -v
>
> * Connected to av.by (86.57.246.106) port 80 (#0)
> > POST /login HTTP/1.1
> > Host: av.by
> > Cookie: PHPSESSID=22222222222222222222222
> > Origin: http://av.by
> > Content-Type: application/x-www-form-urlencoded
> > Cache-Control: max-age=0
> > Referer: http://av.by/login
> > Accept-Encoding: gzip, deflate, sdch
> > Accept-Language: en-US,en;q=0.8,ru;q=0.6,pl;q=0.4
> > Upgrade-Insecure-Requests: 1
> > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
> > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/
> webp,*/*;q=0.8
> > Connection: keep-alive
> > Content-Length: 111
> >
> } [111 bytes data]
> * upload completely sent off: 111 out of 111 bytes
> < HTTP/1.1 302 Found
> < Server: nginx
> < Content-Type: text/html
> < Transfer-Encoding: chunked
> < Connection: keep-alive
> < Expires: Thu, 1 Nov 1981 01:01:01 GMT
> < Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
> pre-check=0
> < Pragma: no-cache
> * Added cookie avby_id="11111" for domain av.by, path /, expire
> 11111111111
> < Set-Cookie: avby_id=11111; expires=Mon, 1-Jan-2026 01:01:01 GMT; path=/;
> domain=.av.by
> * Added cookie avby_hash="123456789123456789123456789" for domain av.by,
> path /, expire 1111111111
> < Set-Cookie: avby_hash=123456789123456789123456789; expires=Mon,
> 1-Jan-2026 01:01:01 GMT; path=/; domain=.av.by
> * Replaced cookie avby_id="11111" for domain av.by, path /, expire
> 123512345 Set-Cookie: avby_id=11111; expires=Mon, 1-Jan-2026 01:01:01;
> path=/
> * Replaced cookie avby_hash="123456789123456789123456789" for domain av.by,
> path /, expire 1790298555
> < Set-Cookie: avby_hash=123456789123456789123456789; expires=Mon,
> 1-Jan-2026 01:01:01; GMT; path=/
> * Replaced cookie PHPSESSID="deleted" for domain av.by, path /, expire 1
> < Set-Cookie: PHPSESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT
> * Added cookie PHPSESSID="1234512345" for domain av.by, path /, expire 0
> < Set-Cookie: PHPSESSID=1234512345
> < Location: http://av.by/profile
>
> Cookie jar after execution
> av.by FALSE / FALSE 11111111111 avby_id 11111
> av.by FALSE / FALSE 11111111111 avby_hash
> 123456789123456789123456789
> av.by FALSE / FALSE 0 PHPSESSID 1234512345
>
> First two lines are expected to start with ".av.by TRUE"
>
> Then site redirects to cars.av.by which doesn't have required cookies.
>
>
> What is happening is your website is sending two Set-Cookie lines for
> those cookies, one with the domain (ie yes subdomains) and one without the
> domain (ie no subdomains). For example avby_id, Firefox and Chrome will
> keep both cookies and send both on a future request to av.by. libcurl
> however will replace the first cookie with a second cookie of the same
> name, even if the 'allow subdomains' flag is different.
>
> curld -v -b cookies.txt -c cookies.txt "http://httpbin.org/response-
> headers?Set-Cookie=%20foo=bar;%20domain=.httpbin.org&Set-
> Cookie=%20foo=qux;"
> <http://httpbin.org/response-headers?Set-Cookie=%20foo=bar;%20domain=.httpbin.org&Set-Cookie=%20foo=qux;>
>
> * Added cookie foo="bar" for domain httpbin.org, path /, expire 0
> < Set-Cookie: foo=bar; domain=.httpbin.org
> * Replaced cookie foo="qux" for domain httpbin.org, path /, expire 0
> < Set-Cookie: foo=qux;
>
> And that results in a single cookie that does not include subdomains
> (second field is 'FALSE'):
> httpbin.org FALSE / FALSE 0 foo qux
>
> So is this correct behavior? I don't know. RFC 6265 says for the server in
> 4.1.1 Syntax [1]:
>
> "Servers SHOULD NOT include more than one Set-Cookie header field in the
> same response with the same cookie-name. (See Section 5.2 for how user
> agents handle this case.)"
>
> and later, in 4.1.2 Semantics [2]:
>
> "If the user agent receives a new cookie with the same cookie-name,
> domain-value, and path-value as a cookie that it has already stored, the
> existing cookie is evicted and replaced with the new cookie."
>
> ... and later, in 5.3 Storage Model [3] if there is no domain attribute
> "Set the cookie's domain to the canonicalized request-host."
>
> So that could be interpreted as the domain value being the same (like
> Domain=foo is domain-value 'foo' and no domain is also domain-value 'foo'
> if the server is foo). I've noticed in the browsers though when there's a
> domain they store it as '.foo' and when there's no domain they store it as
> 'foo'. I must have missed that part in the RFC? But in that case I see an
> argument that the domain-value wouldn't be the same. Thoughts?
>
> Also, a side note: when Firefox or Chrome send two cookies of the same
> name but one allowing subdomains how does a server know which is which? I'm
> looking at RFC 6265 and I don't see any guidance for it. I did some
> experiment in Chrome just now, expecting the order to be the subdomain
> cookie always comes first but it doesn't, instead it's FIFO style.
>
> [1]: https://tools.ietf.org/html/rfc6265#page-10
> [2]: https://tools.ietf.org/html/rfc6265#section-4.1.2
> [3]: https://tools.ietf.org/html/rfc6265#page-23
>
>
> -------------------------------------------------------------------
> List admin: https://cool.haxx.se/list/listinfo/curl-library
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-09-28