python爬豆瓣250王能获取数据但是爬出的.csv文件没有数据
代码：items.py：
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.
import scrapy
class DoubanmainItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()#电影名
star = scrapy.Field()#评分
critical = scrapy.Field()#评分人数
pass
settings.py：
# Scrapy settings for doubanmain project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.
# https://docs.
# https://docs.
BOT_NAME = 'doubanmain'
SPIDER_MODULES = ['doubanmain.spiders']
NEWSPIDER_MODULE = 'doubanmain.spiders'
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"
# Configure item pipelines
# See https://docs.
ITEM_PIPELINES = {
'doubanmain.pipelines.DoubanmainPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
pipelines.py：
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.
# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
class DoubanmainPipeline(object):
# def __init__(self):
# self.file = open("D:/test/doubanmain/douban_movie.csv","w")
def __init__(self):
self.file = open("D:/softwareforwork/Python32/Scripts/scrapytest/Scripts/doubanmain/douban_movie.csv","w")
def process_item(self, item, spider):
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
# str = item['title'].encode('utf-8')+','+item['star'].encode('utf-8')+','+item['critical'].encode('utf-8')
self.file.write(str+'\n')
# return item
def close_spider(self,spider):
self.file.close()
douban.py：
#coding:utf-8
import scrapy
from scrapy.spiders import CrawlSpider
from scrapy.http import Request
from scrapy.selector import Selector
from doubanmain.items import DoubanmainItem
class Douban(CrawlSpider):
name = "doubanmain"
allowed_domains = ['']
start_urls = ['https://movie.']
def start_requests(self):
for url in self.start_urls:
yield Request(url= url,callback= self.parse)
def parse(self, response):
item = DoubanmainItem()
selector = Selector(response)
Movies = selector.xpath('//div[@class="info"]')
for eachMovie in Movies:
title = eachMovie.xpath('div[@class="hd"]/a/span/text()').extract()[0]
star = eachMovie.xpath('div[@class="bd"]/div[@class="star"]/span[@class="rating_num"]/text()').extract()[0]
critical = eachMovie.xpath('div[@class="bd"]/div[@class="star"]/span/text()').extract()[1]
item['title'] = title
item['star'] = star
item['critical'] = critical
yield item
nextLink = selector.xpath('//span[@class="next"]/link/@href').extract()
if nextLink: #第十页是最后一页，没有下一页的链接
nextLink = nextLink[0]
yield Request(self.start_urls[0]+nextLink)
报错
D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\python.exe "D:\softwareforwork\PyCharm 2021.1.1\plugins\python\helpers\pydev\pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 63648 --file D:/softwareforwork/Python32/Scripts/scrapytest/Scripts/doubanmain/main.py
Connected to pydev debugger (build 211.7142.13)
2021-05-17 16:47:23 [scrapy.utils.log] INFO: Scrapy 2.5.0 started (bot: doubanmain)
2021-05-17 16:47:23 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:13:28) [MSC v.1928 32 bit (Intel)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k 25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19042-SP0
2021-05-17 16:47:23 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-05-17 16:47:23 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'doubanmain',
'NEWSPIDER_MODULE': 'doubanmain.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['doubanmain.spiders'],
'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
'(KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 '
'Edg/90.0.818.62'}
2021-05-17 16:47:23 [scrapy.extensions.telnet] INFO: Telnet Password: a0bd1cce9ca9132c
2021-05-17 16:47:23 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2021-05-17 16:47:23 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-05-17 16:47:23 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-05-17 16:47:23 [scrapy.middleware] INFO: Enabled item pipelines:
['doubanmain.pipelines.DoubanmainPipeline']
2021-05-17 16:47:23 [scrapy.core.engine] INFO: Spider opened
2021-05-17 16:47:23 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-05-17 16:47:23 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-05-17 16:47:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://movie. (referer: None)
2021-05-17 16:47:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://movie. (referer: None)
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '2351420人评价', 'star': '9.7', 'title': '肖申克的救赎'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1749765人评价', 'star': '9.6', 'title': '霸王别姬'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://movie. - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1771032人评价', 'star': '9.5', 'title': '阿甘正传'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1945571人评价', 'star': '9.4', 'title': '这个杀手不太冷'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1732228人评价', 'star': '9.4', 'title': '泰坦尼克号'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1089712人评价', 'star': '9.5', 'title': '美丽人生'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1848109人评价', 'star': '9.4', 'title': '千与千寻'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '904192人评价', 'star': '9.5', 'title': '辛德勒的名单'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1710151人评价', 'star': '9.3', 'title': '盗梦空间'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1174646人评价', 'star': '9.4', 'title': '忠犬八公的故事'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1382842人评价', 'star': '9.3', 'title': '星际穿越'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1298789人评价', 'star': '9.3', 'title': '楚门的世界'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1389152人评价', 'star': '9.3', 'title': '海上钢琴师'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1561074人评价', 'star': '9.2', 'title': '三傻大闹宝莱坞'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1098421人评价', 'star': '9.3', 'title': '机器人总动员'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1082274人评价', 'star': '9.3', 'title': '放牛班的春天'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1263509人评价', 'star': '9.2', 'title': '大话西游之大圣娶亲'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1051792人评价', 'star': '9.2', 'title': '无间道'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1527326人评价', 'star': '9.2', 'title': '疯狂动物城'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '767127人评价', 'star': '9.3', 'title': '熔炉'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '768630人评价', 'star': '9.3', 'title': '教父'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1254720人评价', 'star': '9.1', 'title': '当幸福来敲门'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1046757人评价', 'star': '9.2', 'title': '龙猫'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '1484091人评价', 'star': '9.1', 'title': '怦然心动'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.scraper] ERROR: Error processing {'critical': '367974人评价', 'star': '9.6', 'title': '控方证人'}
Traceback (most recent call last):
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\twisted\internet\defer.py", line 662, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\softwareforwork\Python32\Scripts\scrapytest\lib\site-packages\scrapy\utils\defer.py", line 150, in f
return deferred_from_coro(coro_f(*coro_args, **coro_kwargs))
File "D:\softwareforwork\Python32\Scripts\scrapytest\Scripts\doubanmain\doubanmain\pipelines.py", line 17, in process_item
str = item['title'].encode('utf-8') + ',' + item['star'].encode('utf-8') + ',' + item['critical'].encode('utf-8')
TypeError: can't concat str to bytes
2021-05-17 16:47:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://movie. (referer: https://movie.)
2021-05-17 16:47:24 [scrapy.core.engine] INFO: Closing spider (finished)
2021-05-17 16:47:24 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1070,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'downloader/response_bytes': 25878,
'downloader/response_count': 3,
'downloader/response_status_count/200': 3,
'dupefilter/filtered': 24,
'elapsed_time_seconds': 0.805056,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 5, 17, 8, 47, 24, 647444),
'httpcompression/response_bytes': 135516,
'httpcompression/response_count': 3,
'log_count/DEBUG': 4,
'log_count/ERROR': 25,
'log_count/INFO': 10,
'request_depth_max': 1,
'response_received_count': 3,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'start_time': datetime.datetime(2021, 5, 17, 8, 47, 23, 842388)}
2021-05-17 16:47:24 [scrapy.core.engine] INFO: Spider closed (finished)
Process finished with exit code -1
