文字实例——自定义脚本JSON数据采集

6 讨 论 24675 浏览量 云采集引擎

自定义脚本JSON数据采集
适用于json数据接口获取内容

比如:http://v.duowan.com/ 我们想采集这个网站上的视频标题和视频链接。

1.数据入口配置

经过我们的分析得到。这个网站的数据是使用json加载的。数据链接如下:

page 表示的就是加载的第几页。经过分析数据加载到第8页再往后就没有了:

所以数据入口配置如下

2.数据链接

由于我们所需要的数据在json格式里面已经包含,不需要进行再次解析URL。 根据列表页采集URL然后从URL再采集数据字段的形式不适合这种。所以我们需要用到单页采集的原理。

我们可以分析出来每条数据都有一个特别的vid作为识别字段。然后可以给原来的URL加上这个后缀。然后每次采集还是用的入口的URL但是每次的数据都会带上这个vid作为标识。标识我们每次要采集那一条数据。

然后就可以配置数据连接如下,在数据链接URL识别规则填写:

"vid":"[url]","video_title"

这里我们只是获取了每个数据的vid标识而已。

我们获取的每一个vid并不是一个url,要想获取字段的话,引擎是只识别url才会继续采集啊,所以我们要想办法给采集引擎一个完整的URL用于下一步采集字段使用。上面我们已经采集到了每条数据的标识。

在数据链接URL补充前缀填写:

[entryurl]#

[entryurl] 表示的是入口的URL,然后#是一个拼接符号,这样生成的链接会在#后面加上每条数据的vid

这样就获取了一个完整的URL.然后我们就可以进行下一步字段采集了。

3.核心字段

还记得我们在数据连接后面用#后面拼接起来的vid么。这时候可以用[poundstr]来接受#后面的内容。

那么数据标题识别规则填写为:

"vid":"[poundstr]","video_title":[subject],

[subject]这个表示我们要采集的标题。[poundstr]是#后面的内容。这里是每条数据的vid**。

数据标题我们获得到了但是数据很明显不是我们想要的样子。我们可以看出来这很明显是json_encode之后的数据。那么我们需要将这个数据json_decode来获得我们想要的样子。

数据标题自定义脚本处理:

<?php
function CUSTOMFUN($inputstr){
// TODO: Coding Here
$outputstr = json_decode($inputstr); //在这里进行json_decode处理
return $outputstr;
}
echo CUSTOMFUN($PARASTR); // Not Allowed to Modify
?>

数据标题有了,内容也是必填的,那么。

数据内容识别规则填写如下:

"vid":"[poundstr]",*"video_url":[message],"video_channel":"lol",

数据内容自定义脚本处理:

<?php
function CUSTOMFUN($inputstr){
// TODO: Coding Here
$outputstr = json_decode($inputstr);
return $outputstr;
}
echo CUSTOMFUN($PARASTR); // Not Allowed to Modify
?>

调试下,内容也有了。

然后 点击提交发布。

好了 点击执行试试吧。

__END__

6讨 论
已讨论 4月 15日 作者: aokemohj
A uqp.pkoz.help.finndy.com.lnb.mq expel [URL=http://aawaaart.com/priligy/ - priligy[/URL -  [URL=http://gaiaenergysystems.com/buy-prednisone/ - prednisone with no prescription[/URL -  [URL=http://aawaaart.com/viagra/ - viagra 100mg[/URL -  [URL=http://aawaaart.com/cialis/ - generic cialis no prescription[/URL -  [URL=http://americanartgalleryandgifts.com/cialis/ - cialis 20 mg walmart price[/URL -  tame <a href="http://aawaaart.com/priligy/">dapoxetine</a> <a href="http://gaiaenergysystems.com/buy-prednisone/">prednisone without dr prescription</a> <a href="http://aawaaart.com/viagra/">viagra generic</a> <a href="http://aawaaart.com/cialis/">can you take cialis with avapro</a> cialis for sale canada <a href="http://americanartgalleryandgifts.com/cialis/">buy cialis online</a> tonsils atrium http://aawaaart.com/priligy/#priligy dapoxetine http://gaiaenergysystems.com/buy-prednisone/#buy-prednisone prednisone with no prescription http://aawaaart.com/viagra/#walmart-viagra-100mg-price 100 mg viagra lowest price http://aawaaart.com/cialis/#20mg-cialis lowest price for cialis 20 mg http://americanartgalleryandgifts.com/cialis/#cialis cialis exhausts punishments.
已讨论 4月 17日 作者: Slut Cams
<a href="http://bbwcams.site/">teen cams</a> <a href="http://webcamamateur.fun/">#1 adult chat</a> <a href="http://camgirls.ga/">cam girls</a> <a href="http://webcamteens.fun/">webcam masturbation</a> <a href="http://freeonlinesex.site/">online sex chat</a> <a href="http://nudelive.icu/">live nude</a> <a href="http://videochatroom.icu/">adult video chat</a> <a href="http://livecamgirls.fun/">live cam girls</a> <a href="http://interactiveporn.tk/">porn cams</a> <a href="http://freeadultvideochat.icu/">free video chat rooms</a>
已讨论 4月 17日 作者: yuvefaorua
A dtl.xydi.help.finndy.com.ikw.ui orange flank, clindamycin, [URL=http://ormondbeachflorida.org/levitra-prices/ - levitra[/URL -  [URL=http://life-sciences-forums.com/prednisone/ - buy prednisone[/URL -  [URL=http://nitromtb.org/eriacta/ - eriacta online[/URL -  [URL=http://gccroboticschallenge.com/cialis/ - generic cialis 20 mg[/URL -  [URL=http://biblebaptistny.org/doxycycline/ - doxycycline hyclate 100 mg[/URL -  [URL=http://michiganvacantproperty.org/celebrex/ - purchase celebrex[/URL -  [URL=http://wyovacationrental.com/levitra-20mg-best-price/ - levitra[/URL -  [URL=http://metropolitanbaptistchurch.org/vidalista/ - online vidalista[/URL -  reformers humility <a href="http://ormondbeachflorida.org/levitra-prices/">levitra</a> <a href="http://life-sciences-forums.com/prednisone/">online prednisone</a> <a href="http://nitromtb.org/eriacta/">viagra gross per year</a> <a href="http://gccroboticschallenge.com/cialis/">cialis 200 mg in india</a> <a href="http://biblebaptistny.org/doxycycline/">rickettsia doxycycline dosage</a> <a href="http://michiganvacantproperty.org/celebrex/">celebrex 200 mg no prescription</a> cheap celebrex <a href="http://wyovacationrental.com/levitra-20mg-best-price/">generic levitra</a> <a href="http://metropolitanbaptistchurch.org/vidalista/">vidalista without a prescription</a> incontinence http://ormondbeachflorida.org/levitra-prices/#levitra cheap levitra 20mg levitra http://life-sciences-forums.com/prednisone/#by-prednisone-w-not-prescription by prednisone w not prescription prednisone no prescription http://nitromtb.org/eriacta/#discount-eriacta buy eriacta online http://gccroboticschallenge.com/cialis/#cialis cialis tadalafil 20 mg tablets http://biblebaptistny.org/doxycycline/#chea-doxycycline doxycycline dosage for gonorrhea http://michiganvacantproperty.org/celebrex/#celebrex-200-mg buy celebrex http://wyovacationrental.com/levitra-20mg-best-price/#best-price-levitra-20-mg levitra 20 http://metropolitanbaptistchurch.org/vidalista/#vidalista vidalista for sale prophylaxis limitations, risk-factors ability.
已讨论 4月 18日 作者: agavocisec
We ics.hehy.help.finndy.com.gqg.iy inspected emphysematous [URL=http://kullutourism.com/prednisone-20-mg/ - prednisone without dr prescription[/URL -  [URL=http://jokesaz.com/viagra-online/ - cheapviagra[/URL -  [URL=http://theprettyguineapig.com/prednisone-without-prescription/ - order prednisone online[/URL -  [URL=http://fitnesscabbage.com/prednisone-no-prescription/ - purchase prednisone[/URL -  purchase prednisone [URL=http://gccroboticschallenge.com/buy-cialis/ - cialis 5mg best price[/URL -  [URL=http://jokesaz.com/cialis/ - cialis tadalafil 20 mg tablets[/URL -  [URL=http://sammycommunitytransport.org/cialis-super-force/ - buy cialis super force online[/URL -  [URL=http://wyovacationrental.com/accutane/ - accutane need to order[/URL -  [URL=http://theswordguy.com/flagyl/ - metronidazole for yeast infection[/URL -  import liquefactive <a href="http://kullutourism.com/prednisone-20-mg/">prednisone</a> buy prednisone online in us <a href="http://jokesaz.com/viagra-online/">viagra</a> <a href="http://theprettyguineapig.com/prednisone-without-prescription/">prednisone online</a> <a href="http://fitnesscabbage.com/prednisone-no-prescription/">prednisone no rx</a> <a href="http://gccroboticschallenge.com/buy-cialis/">cialis buy</a> <a href="http://jokesaz.com/cialis/">cialis generic canada</a> <a href="http://sammycommunitytransport.org/cialis-super-force/">cialis super force canada</a> <a href="http://wyovacationrental.com/accutane/">accutane canada</a> <a href="http://theswordguy.com/flagyl/">flagyl</a> exacerbations gives contexts, http://kullutourism.com/prednisone-20-mg/#prednisone-no-prescription prednisone for cats http://jokesaz.com/viagra-online/#viagra-online viagra 100mg generic viagra canada http://theprettyguineapig.com/prednisone-without-prescription/#by-prednisone-w-not-prescription by prednisone w not prescription http://fitnesscabbage.com/prednisone-no-prescription/#prednisone-no-prescription prednisone without dr prescription usa http://gccroboticschallenge.com/buy-cialis/#generic-cialis-20mg buy cialis http://jokesaz.com/cialis/#cialis-20-mg 20 mg cialis cialis 20 mg http://sammycommunitytransport.org/cialis-super-force/#cialis-super-force-lowest-price buy cialis super force http://wyovacationrental.com/accutane/#roaccutane-prie-uk generic accutane online http://theswordguy.com/flagyl/#buy-flagyl flagyl endoscopy dental thalamus suppositories.
已讨论 4月 19日 作者: ucevayova
Plaster mnr.iieq.help.finndy.com.fnb.pp cysticercotic lymphocytes [URL=http://jacksfarmradio.com/proscar/ - proscar  without dr prescription[/URL -  [URL=http://clearcandybags.com/prednisone-20-mg/ - buying prednisone online[/URL -  [URL=http://michiganvacantproperty.org/retin-a/ - retin a micro[/URL -  [URL=http://theswordguy.com/lioresal/ - baclofen without a prescription[/URL -  [URL=http://sci-ed.org/buy-xenical/ - buy xenical[/URL -  [URL=http://wyovacationrental.com/canadian-pharmacy-price/ - pharmacy[/URL -  cialis online pharmacy [URL=http://michiganvacantproperty.org/buy-propecia/ - propecia[/URL -  [URL=http://jacksfarmradio.com/renova--for-sale/ - renova  for sale[/URL -  [URL=http://bestpriceonlineusa.com/generic-levitra/ - cost of levitra 20mg[/URL -  vision; <a href="http://jacksfarmradio.com/proscar/">proscar  without dr prescription</a> <a href="http://clearcandybags.com/prednisone-20-mg/">buy prednisone no prescription</a> buy prednisone online <a href="http://michiganvacantproperty.org/retin-a/">retin a</a> <a href="http://theswordguy.com/lioresal/">baclofen for sale</a> <a href="http://sci-ed.org/buy-xenical/">xenical</a> <a href="http://wyovacationrental.com/canadian-pharmacy-price/">pharmacy</a> <a href="http://michiganvacantproperty.org/buy-propecia/">propecia</a> <a href="http://jacksfarmradio.com/renova--for-sale/">online renova</a> renova  without dr prescription <a href="http://bestpriceonlineusa.com/generic-levitra/">levitra cost</a> up: abscesses, http://jacksfarmradio.com/proscar/#generic-proscar proscar http://clearcandybags.com/prednisone-20-mg/#prednisone prednisone order prednisone http://michiganvacantproperty.org/retin-a/#buy-retin-a retin a http://theswordguy.com/lioresal/#buy-lioresal lioresal without a prescription http://sci-ed.org/buy-xenical/#xenical buy orlistat http://wyovacationrental.com/canadian-pharmacy-price/#pharmacy canadian pharmacy price http://michiganvacantproperty.org/buy-propecia/#propecia propecia http://jacksfarmradio.com/renova--for-sale/#renova--for-sale renova http://bestpriceonlineusa.com/generic-levitra/#vardenafil vardenafil 20 mg condemn feeder relied trochanter.

Finndy Copyright©2017 | Powered by Q2A

...