Сообщения за Апрель, 2017

Проверка Вашего сайта на битые ссылки как это делают поисковые роботы (для Linux). Check your site for broken links as Crawler or Spider (useful for Linux)



Вот плюшки (buns: ) ):
The "Spider" or "Crawler"

- HTTP/1.1 compliant with persistent connections and cookies support
- HTTP Basic authentication supported
- HTTP Proxy support (basic authentication included)
- Crawl customisable through many configuration attributes which let the user
limit the digging on URLs pattern matchings and distance ("hops") from the first URL.
- MySQL databases directly created by the spider
- MySQL connections through user or general option files as defined by the
database system (/etc/my.cnf or ~/.my.cnf)

А вот ложка дёгтя (oil tar: ):
No support for Javascript and other protocols like HTTPS, FTP, NNTP and local files.