Robust and efficient web crawler
This presentation will introduce the knowledge and skills of using Python for web crawler development in five aspects: crawling, parsing, storing, anti-climbing, and acceleration. It introduces how to take different measures to efficiently capture data in different scenarios, including Web crawling, App crawling, data storage, agent purchasing, verification code cracking, distributed crawling and management, intelligent parsing, etc., and also introduce some commonly used toolkits in combination with different scenarios. All of the speech content are the summary of the experience of the speaker since the web crawler research process.
Qingcai Cui, master of Beihang University, author of 《Python3网络爬虫开发实战》, Blog bloggers of cuiqingcai.com, the number of blog reading about Crawler has exceeded one million, Big Data Engineer of Microsoft China, lecturer of Tianshan Intelligent and Netease Cloud Classroom, and currently engaged in the research of conversational chat direction.