爬蟲程式存取目標網站的過程會消耗目標系統資源，導致目標系統效能下降，嚴

爬蟲程式存取目標網站的過程會消耗目標系統資源，導致目標系統效能下降，嚴重時更可能造成服務癱瘓，因此爬蟲程式需要考慮到抓取策略規劃 (包括抓取頻率、抓取內容、對目標的負載、目標的限制)，機器人排除協議則由目標管理者來決定爬蟲程式可否進行資料收集或可到達哪些網頁的協議。不願意被爬蟲存取的公開網站管理者，可以使用robots.txt檔案設定的方法避免存取。

結果 (英文) 1:

Process reptile program to access the target site of the target will consume system resources, resulting in the target system performance degradation, more likely to cause severe paralysis of service, so the program needs to be considered reptiles crawl strategic planning (including crawl frequency, crawling content, the target the load limit of the target), the robot exclusion protocol target by the manager to decide whether the reptile program or data collection protocol which pages are reachable. Not want to be publicly accessible website crawler managers can use a robots.txt file to avoid setting access.

結果 (英文) 2:

The process of accessing the target website by the crawler will consume the target system resources, resulting in a decrease in the performance of the target system, which in severe cases may cause service paralysis, so the crawler needs to take into account the crawl strategy planning (including crawl frequency, crawl content, the target load, target restrictions), The robot exclusion protocol is up to the target manager to decide which pages the crawler can collect data or reach. Public web managers who do not want to be accessed by crawlers can use the robots.txt file-set method to avoid access.

結果 (英文) 3:

The process of accessing the target website by the crawler will consume the target system resources, resulting in the performance degradation of the target system, and even more likely to cause service paralysis when it is serious. Therefore, the crawler needs to consider the capture strategy planning (including the frequency of crawling, the content of crawling, the load of the target and the limitation of the target). The robot exclusion protocol is a protocol that the target manager decides whether the crawler can collect data or which pages it can reach. The public website manager who is not willing to be accessed by the crawler can use the method set by the robots.txt file to avoid access.