My actual purporse for crawling webpage
Some web pages get last modified date and some webpage not get last modified date
Not Getting Modified Date
Accept-Ranges:bytes
Age:0
Content-Encoding:gzip
Content-Length:12458
Content-Type:text/html; charset=UTF-8
Date:Thu, 03 Aug 2017 09:00:51 GMT
Vary:Accept-Encoding, User-Agent
X-Backend:all_requests
X-Cache:uncached
X-Cache-Hit:MISS
X-Cacheable:YES:Forced
X-Port:port_10128
Getting Modified Date
Accept-Ranges:bytes
Age:545944
Cache-Control:max-age=31622400
Connection:keep-alive
Content-Encoding:gzip
Content-Length:67742
Content-Type:text/css
Date:Fri, 28 Jul 2017 01:24:13 GMT
Etag:W/"597a0925-66a34"
Expires:Sun, 29 Jul 2018 01:24:13 GMT
Last-Modified:Thu, 27 Jul 2017 15:39:17 GMT
Aucun commentaire:
Enregistrer un commentaire