阿布云

你所需要的，不仅仅是一个好用的代理。

抓取知乎百万用户信息之Redis篇

发表于 2018-03-16

<div id="cnblogs_post_body">             <a href="https://github.com/wangqifan/"><img src="http://images2015.cnblogs.com/blog/814953/201702/814953-20170227113456829-1370229104.png" alt="" width="50" height="50">点击我前往Github查看源代码</a>   别忘记star
本项目github地址：https://github.com/wangqifan/ZhiHu     
Redis安装
 Redis官方并没有推出windows版本，人家觉得linux已经够了，开发windows版本影响开发进度，还好微软有一个团队维持着Redis的windows版本，网上有很多介绍Redis安装的博客，大多数是敲各种命令行。这里有Redis的msi版本，只需要像安装普通软件一样点击下一步，下一步即可地址：https://github.com/MSOpenTech/redis/releases/download/win-3.2.100/Redis-x64-3.2.100.msi
RRedis配置
Redis配置文件详解 http://www.cnblogs.com/kreo/p/4423362.html
找到Redis.windowserver.conf
 
<img src="http://images2015.cnblogs.com/blog/814953/201701/814953-20170108112435581-179605722.png" alt="">
这里要注意的两点：1.远程连接
<div class="cnblogs_code"><div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div>
<pre>#
# ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
# internet, binding to all the interfaces is dangerous and will expose the
# instance to everybody on the internet. So by default we uncomment the
# following bind directive, that will force Redis to listen only into
# the IPv4 lookback interface address (this means Redis will be able to
# accept connections only from clients running into the same computer it
# is running).
#
# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
# JUST COMMENT THE FOLLOWING LINE.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bind 0.0.0.0
将bind 127.0.0.1 修改成bind 0.0.0.0这样redis可以接受远程连接</pre>
<div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div></div>
内存限制
<div class="cnblogs_code"><div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div>
<pre># NOTE: since Redis uses the system paging file to allocate the heap memory,
# the Working Set memory usage showed by the Windows Task Manager or by other
# tools such as ProcessExplorer will not always be accurate. For example, right
# after a background save of the RDB or the AOF files, the working set value
# may drop significantly. In order to check the correct amount of memory used
# by the redis-server to store the data, use the INFO client command. The INFO
# command shows only the memory used to store the redis data, not the extra
# memory used by the Windows process for its own requirements. Th3 extra amount
# of memory not reported by the INFO command can be calculated subtracting the
# Peak Working Set reported by the Windows Task Manager and the used_memory_peak
# reported by the INFO command.
#
maxmemory 2000mb
这里可以修改最大内存，建议放大点Redis比较还是吃内存的</pre>
<div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div></div>
连接Reids类的封装
Redis的C#驱动ServiceStack.Redis使用NuGet进行安装，由于这个类库已经商业化了，在4.0版本开始限制数量，每小时不得超过6000次，建议安装3.9版本
在这个爬虫系统中，开始时候我只使用一台电脑装Redis，后来发现这台电脑特别卡，后来换成三台电脑装Redis，一个负责hash表，一个负责UrlNext队列和Urltoken队列，一台负责User队列，由于实验室的电脑非常老旧，还是很卡。最后又加持2台电脑，实验室三台电脑负责hash表，我的电脑负责User队列，征用学妹电脑用作任务队列。
这个类命名为RedisCore
Ip地址列表
<div class="cnblogs_code"><div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div>
<pre>public static List<string> ips = new List<string>()

{

"59.74.169.54",

"59.74.169.57",

"59.74.169.52",

"59.74.169.58",

"59.74.169.39"

};</pre>
<div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div></div>
 
对插入队列的封装。
Redis队列是有list这个数据结构实现的，从右边插入，左边弹出就可以实现队列
插入
<div class="cnblogs_code"><div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div>
<pre>public static bool PushIntoList(int type, string key, string value)

{

bool Result = false;

using (RedisClient Redis = new RedisClient(ips[type - 1], 6379))

{

Redis.ConnectTimeout = 2000;

Result = Redis.RPush(key, Encoding.UTF8.GetBytes(value)) > 0;

}

return Result;

}</pre>
<div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div></div>
 
注意这个非托管资源要手动释放
弹出
<div class="cnblogs_code"><div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div>
<pre>public static string PopFromList(int type, string key)
{
string result = string.Empty;
try
{

using (RedisClient Redis = new RedisClient(ips[type - 1], 6379))
{
Redis.ConnectTimeout = 2000;
result = Encoding.UTF8.GetString(Redis.LPop(key));
}

}
catch
{

}
return result;
}</pre>
<div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div></div>
Hash表有三个电脑，到底放到那一台，首先对key进行hash运算，取绝对值，对3取余，为0 就放到3号机器，为1放到4号机器，为2 放到5号机器
，如果hash表已经存在就会插入失败返回false，不存在插入成功返回true
<div class="cnblogs_code"><div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div>
<pre> public static bool InsetIntoHash(int type, string hashid, string key, string value)
{
bool result = false;
try
{
using (RedisClient Redis = new RedisClient(ips[type - 1], 6379))
{
Redis.ConnectTimeout = 2000;
result = Redis.SetEntryInHashIfNotExists(hashid, key, value);
}
}
catch { }

return result;
}
</pre>
<div class="cnblogs_code_toolbar"><a href="javascript:void(0);" onclick="copyCnblogsCode(this)" title="复制代码"><img src="//common.cnblogs.com/images/copycode.gif" alt="复制代码"></a></div></div>
 </div>

反爬虫问题

抓取过于频繁，服务器返回429.这个时候需要切换代理IP了，推荐使用阿布云代理,阿布云代理IP,提供高匿代理,爬虫代理.

Keywords

新闻中心