Golang高并发抓取HTML图片

待兔 等级 1054 0 0

版权所有,转载请注明:http://www.lenggirl.com/language/go-picture.html

使用准备

1.安装Golang

2.下载爬虫包

go get -v github.com/hunterhug/marmot/expert
go get -v github.com/hunterhug/marmot/miner
go get -v github.com/hunterhug/parrot/util 

程序

该程序只能抓取HTML中src="http"中的图片, 必须带有协议头http(s), 其他如data-src和混淆在JS中的无法抓取

See: https://github.com/hunterhug/marmot/blob/master/example/lesson/lesson6.go

package main

import (
    "errors"
    "fmt"
    "net/url"
    "strings"

    "github.com/hunterhug/marmot/expert"
    "github.com/hunterhug/marmot/miner"
    "github.com/hunterhug/parrot/util"
)

// Num of miner, We can run it at the same time to crawl data fast
var MinerNum = 5

// You can update this decide whether to proxy
var ProxyAddress interface{}

func main() {
    // You can Proxy!
    // ProxyAddress = "socks5://127.0.0.1:1080"

    fmt.Println(`Welcome: Input "url" and picture keep "dir"`)
    for {
        fmt.Println("---------------------------------------------")
        url := util.Input(`URL(Like: "http://publicdomainarchive.com")`, "http://publicdomainarchive.com")
        dir := util.Input(`DIR(Default: "./picture")`, "./picture")
        fmt.Printf("You will keep %s picture in dir %s\n", url, dir)
        fmt.Println("---------------------------------------------")

        // Start Catch
        err := CatchPicture(url, dir)
        if err != nil {
            fmt.Println("Error:" + err.Error())
        }
    }
}

// Come on!
func CatchPicture(picture_url string, dir string) error {
    // Check valid
    _, err := url.Parse(picture_url)
    if err != nil {
        return err
    }

    // Make dir!
    err = util.MakeDir(dir)
    if err != nil {
        return err
    }

    // New a worker to get url
    worker, _ := miner.New(ProxyAddress)

    result, err := worker.SetUrl(picture_url).SetUa(miner.RandomUa()).Get()
    if err != nil {
        return err
    }

    // Find all picture
    pictures := expert.FindPicture(string(result))

    // Empty, What a pity!
    if len(pictures) == 0 {
        return errors.New("empty")
    }

    // Devide pictures into several worker
    xxx, _ := util.DevideStringList(pictures, MinerNum)

    // Chanel to info exchange
    chs := make(chan int, len(pictures))

    // Go at the same time
    for num, imgs := range xxx {

        // Get pool miner
        worker_picture, ok := miner.Pool.Get(util.IS(num))
        if !ok {
            // No? set one!
            worker_temp, _ := miner.New(ProxyAddress)
            worker_picture = worker_temp
            worker_temp.SetUa(miner.RandomUa())
            miner.Pool.Set(util.IS(num), worker_temp)
        }

        // Go save picture!
        go func(imgs []string, worker *miner.Worker, num int) {
            for _, img := range imgs {

                // Check, May be Pass
                _, err := url.Parse(img)
                if err != nil {
                    continue
                }

                // Change Name of our picture
                filename := strings.Replace(util.ValidFileName(img), "#", "_", -1)

                // Exist?
                if util.FileExist(dir + "/" + filename) {
                    fmt.Println("File Exist:" + dir + "/" + filename)
                    chs <- 0
                } else {

                    // Not Exsit?
                    imgsrc, e := worker.SetUrl(img).Get()
                    if e != nil {
                        fmt.Println("Download " + img + " error:" + e.Error())
                        chs <- 0
                        return
                    }

                    // Save it!
                    e = util.SaveToFile(dir+"/"+filename, imgsrc)
                    if e == nil {
                        fmt.Printf("SP%d: Keep in %s/%s\n", num, dir, filename)
                    }
                    chs <- 1
                }
            }
        }(imgs, worker_picture, num)
    }

    // Every picture should return
    for i := 0; i < len(pictures); i++ {
        <-chs
    }

    return nil
} 

解释均写, 运行后:

superpika@superpika-chen-110:~/code/src/github.com/hunterhug/marmot/example/lesson$ go run lesson6.go 

        Welcome: Input "url" and picture keep "dir"



---------------------------------------------
URL(Like: "http://publicdomainarchive.com")

DIR(Default: "./picture")

You will keep http://publicdomainarchive.com picture in dir ./picture
---------------------------------------------
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_02_03_modern.jpg
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_03_03_google_dark.png
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-003-667x1000-192684_667x675.jpg

superpika@superpika-chen-110:~/code/src/github.com/hunterhug/marmot/example/lesson$ go run lesson6.go 

        Welcome: Input "url" and picture keep "dir"



---------------------------------------------
URL(Like: "http://publicdomainarchive.com")

DIR(Default: "./picture")

You will keep http://publicdomainarchive.com picture in dir ./picture
---------------------------------------------
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_02_03_modern.jpg
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_03_03_google_dark.png
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-003-667x1000-192684_667x675.jpg
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_05_03_powered-by-wp-engine.png
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_05_03_divi.png
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-002-1000x667.jpg
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_05_03_public-domain-mark.png
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_01_03_public-domain-images-free-stock-photos008-1000x625.jpg
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_09_03_Weekly.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_11_03_free-stock-photos-public-domain-images-054-1000x667.jpg
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_10_03_instagram_dark.png
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_02_03_vintage.jpg
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-001-1000x667.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_11_03_free-stock-photos-public-domain-images-070-1000x667.jpg
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_03_03_twitter02_dark.png
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_01_03_public-domain-images-free-stock-photos001-1000x750-167066_1000x675.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-035-1000x667.jpg
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_03_03_facebook_dark.png
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_11_03_free-stock-photos-public-domain-images-060-1000x667.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-013-1000x667.jpg
---------------------------------------------
URL(Like: "http://publicdomainarchive.com")



SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_05_03_powered-by-wp-engine.png
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_05_03_divi.png
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-002-1000x667.jpg
SP4: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_05_03_public-domain-mark.png
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_01_03_public-domain-images-free-stock-photos008-1000x625.jpg
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_09_03_Weekly.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_11_03_free-stock-photos-public-domain-images-054-1000x667.jpg
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_10_03_instagram_dark.png
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_02_03_vintage.jpg
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-001-1000x667.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_11_03_free-stock-photos-public-domain-images-070-1000x667.jpg
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_03_03_twitter02_dark.png
SP2: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_01_03_public-domain-images-free-stock-photos001-1000x750-167066_1000x675.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-035-1000x667.jpg
SP3: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2014_03_03_03_facebook_dark.png
SP0: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_11_03_free-stock-photos-public-domain-images-060-1000x667.jpg
SP1: Keep in ./picture/http_04__03__03_publicdomainarchive.com_03_wp-content_03_uploads_03_2017_03_09_03_free-stock-photos-public-domain-images-013-1000x667.jpg
---------------------------------------------
URL(Like: "http://publicdomainarchive.com") 

[图片上传失败...(image-685ed8-1557639583356)]

收藏
评论区

相关推荐

Mac安装Golang和vscode
Mac第一次安装golang和vscode一起使用,遇到了不少的坑,下面介绍一下正确的安装方式。 1、使用brew安装Golang 如果不知道brew是什么,或怎么安装请看这里 brew官网(https://brew.sh/index_zhcn) brew install golang 安装完成后可以使用
golang 分析调试高阶技巧
layout: post title: “golang 调试高阶技巧” date: 2020603 1:44:09 0800 categories: golang GC 垃圾回收 golang 高阶调试 Golang tools nm compile
Go WEB入门
摘要 由于Golang优秀的并发处理,很多公司使用Golang编写微服务。对于Golang来说,只需要短短几行代码就可以实现一个简单的Http服务器。加上Golang的协程,这个服务器可以拥有极高的性能。然而,正是因为代码过于简单,我们才应该去研究他的底层实现,做到会用,也知道为什么这么用。 在本文中,会以自顶向下的方式,从如何使用,到如何实现,一点点的分
Go语言入门系列(一)之Go的安装和使用
1.安装环境 ====== 1. 进入[Golang官网](https://www.oschina.net/action/GoToLink?url=https%3A%2F%2Fgolang.org),进入下载页面。 (如果打不开可访问[Golang中国](https://www.oschina.net/action/GoToLink?u
Archlinux下Visual Studio Code配置Golang开发环境
一、Golang的安装 ----------- GoLang安装并验证一下: [cox@localhost ~]$ sudo pacman -S go [cox@localhost ~]$ go version go version go1.8.3 linux/amd64s 要注意,Golang的安装要确保两个环境变量,一个是G
GoLang基础数据类型
GoLang基础数据类型--->字符串处理大全 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任。 欢迎加入:   高级运维工程师之路               598432640 任何一门语言都有它的好处,Golang的长处就是在于它的高并发能力,所以现在练习好Golang的用法,有利于你找一份好的工作哟~在学习任何一本语言
Golang Gin实践 番外 请入门 Makefile
<h1>Golang Gin实践 番外 请入门 Makefile</h1> <p>原文地址:<a href="https://github.com/EDDYCJY/blog/blob/master/golang/gin/2018-08-26-Gin%E5%AE%9E%E8%B7%B5-%E7%95%AA%E5%A4%96-%E8%AF%B7%E5%85%A5
Golang In PingCAP
随着 Golang 在后端领域越来越流行,有越来越多的公司选择 Golang 作为主力开发语言。本次 GopherChina Beijing 2016 大会上,看到 Golang 在各家公司从人工智能到自动运维,从 Web 应用到基础架构都发挥着越来越多的作用。可以说 Golang 在这几年间,获得了长足的进步。 PingCAP 是一家由几名 Go
Golang 内存管理源码剖析
Golang 的内存管理基于 tcmalloc,可以说起点挺高的。但是 Golang 在实现的时候还做了很多优化,我们下面通过源码来看一下 Golang 的内存管理实现。下面的源码分析基于 go1.8rc3。 1.tcmalloc 介绍 ------------- 关于 tcmalloc 可以参考这篇文章 [tcmalloc 介绍](https://ww
Golang 开发环境搭建
Golang 是 Google 发布的开发语言,Go 编译的程序速度可以媲美 C/C++。 安装 -- sudo apt-get install golang sudo apt-get install golang-go.tools 使用 -- * 编译运行程序 go run main.go * 查看命令文
Golang 文章正文抽取(readability)
readability for golang Golang版本是根据[readabiliity for node.js](https://www.oschina.net/action/GoToLink?url=https%3A%2F%2Fgithub.com%2Fluin%2Freadability)以及[readability for python](h
Golang依赖管理工具:glide从入门到精通使用
介绍 -- 不论是开发Java还是你正在学习的Golang,都会遇到**依赖管理**问题。Java有牛逼轰轰的Maven和Gradle。 Golang亦有godep、govendor、glide、gvt、gopack等等,本文主要给大家介绍[gilde](https://www.oschina.net/action/GoToLink?url=https%3
Golang学习系列第一天:安装golang
0.  ssh连接linux(我用的centos7),略 ============================ 1.  golang下载 ============ 由于Golang官网[https://golang.org/](https://www.oschina.net/action/GoToLink?url=https%3A%2F%2Fgola
Golang(四)正则表达式使用
0\. 前言 ====== * 最近用到了 regexp 包,下面整理下正则表达式相关用法 * 参考 [基础知识 - Golang 中的正则表达式](https://www.oschina.net/action/GoToLink?url=https%3A%2F%2Fwww.cnblogs.com%2Fgolove%2Fp%2F3269099.htm
Sentinel
![9.28头图.png](https://ucc.alicdn.com/pic/developer-ecology/af7ab6c27c3c4c3aa5dc2cce3c9e8ab9.png) > \*\*导读:\*\*2020年,Sentinel 推出 Go 原生版本[Sentinel-Golang](https://www.oschina.net/ac