标题:又来求助各位老师了,这个网页的数据如何采集?
只看楼主
jinanshui
Rank: 2
等 级:论坛游民
帖 子:274
专家分:64
注 册:2009-10-6
得分:0 
回复 10楼 sdta
谢谢老师,真心感谢
2021-06-13 14:48
jinanshui
Rank: 2
等 级:论坛游民
帖 子:274
专家分:64
注 册:2009-10-6
得分:0 
老师,我想先看一个学科的,老师请看看我的代码有那些错误,我是个大菜鸟,请多笑话
CREATE CURSOR 学校 (h1 C(4),h2 C(4), h3 C(5), h4 C(20),h5 C(4))
lcUrl = "https://souky.
oHTTP=NEWOBJECT("MSXML2.XMLHTTP")
oHTTP.Open("GET", lcUrl, .F.)
oHTTP.SETREQUESTHEADER("Content-Type", "text/html;charset=UTF-8")
oHTTP.Send()
lcText = STRCONV(oHTTP.ResponseBody, 11)
lyText = STREXTRACT  (lcText,"</span> <span id='name'>","</span></div>")
laText = STREXTRACT  (lcText,"评选结果</td>","</table>")
FOR  k =1  TO  65
lxText="<td>"-ALLTRIM(STR(k))+"<td>"
lbText = STREXTRACT(laText,lxText,"</tr>")
? lbText
endfor
2021-06-13 15:33
吹水佬
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
等 级:版主
威 望:432
帖 子:10064
专家分:41463
注 册:2014-5-20
得分:0 
以下是引用sdta在2021-6-13 14:46:02的发言:

楼主提供的网址,下载一次后,网页2小时打不开,

可能刷得快了
2021-06-13 15:33
sdta
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
来 自:江苏省连云港市
等 级:版主
威 望:323
帖 子:9621
专家分:26174
注 册:2012-2-5
得分:0 
回复 12楼 jinanshui
这个网页源码中有一段JSON格式的文档
    var arr=[
{"code":"0101","name":"哲学","total":"91","unitcode":"80000","unitname":"中共中央党

校","result":"B","percent":"20%~30%"},
{"code":"0101","name":"哲学","total":"91","unitcode":"14430","unitname":"中国科学院大

学","result":"C+","percent":"40%~50%"},
{"code":"0101","name":"哲学","total":"91","unitcode":"87903","unitname":"上海社会科学

院","result":"C+","percent":"40%~50%"},
{"code":"0201","name":"理论经济学","total":"95","unitcode":"87903","unitname":"上海社会科学

院","result":"B","percent":"20%~30%"},
{"code":"0201","name":"理论经济学","total":"95","unitcode":"80000","unitname":"中共中央党校

","result":"B-","percent":"30%~40%"},
{"code":"0201","name":"理论经济学","total":"95","unitcode":"82405","unitname":"中国环境科学

研究院","result":"C-","percent":"60%~70%"},
{"code":"0202","name":"应用经济学","total":"160","unitcode":"87903","unitname":"上海社会科

学院","result":"C+","percent":"40%~50%"},
{"code":"0202","name":"应用经济学","total":"160","unitcode":"89101","unitname":"四川省社会

科学院","result":"C-","percent":"60%~70%"},
{"code":"0301","name":"法学","total":"147","unitcode":"80000","unitname":"中共中央党

校","result":"B-","percent":"30%~40%"},
{"code":"0301","name":"法学","total":"147","unitcode":"87903","unitname":"上海社会科学

院","result":"C+","percent":"40%~50%"},
{"code":"0301","name":"法学","total":"147","unitcode":"89101","unitname":"四川省社会科学

院","result":"C-","percent":"60%~70%"},
{"code":"0302","name":"政治学","total":"84","unitcode":"80000","unitname":"中共中央党

校","result":"B+","percent":"10%~20%"},
{"code":"0302","name":"政治学","total":"84","unitcode":"87903","unitname":"上海社会科学

院","result":"C","percent":"50%~60%"},
{"code":"0302","name":"政治学","total":"84","unitcode":"87802","unitname":"黑龙江省社会科学

院","result":"C-","percent":"60%~70%"},
{"code":"0302","name":"政治学","total":"84","unitcode":"89101","unitname":"四川省社会科学院

","result":"C-","percent":"60%~70%"},
{"code":"0303","name":"社会学","total":"69","unitcode":"87903","unitname":"上海社会科学

院","result":"C+","percent":"40%~50%"},
{"code":"0303","name":"社会学","total":"69","unitcode":"80000","unitname":"中共中央党

校","result":"C-","percent":"60%~70%"},
{"code":"0305","name":"马克思主义理论","total":"235","unitcode":"80000","unitname":"中共中

央党校","result":"B+","percent":"10%~20%"},
{"code":"0305","name":"马克思主义理论","total":"235","unitcode":"87903","unitname":"上海社

会科学院","result":"B-","percent":"30%~40%"},
{"code":"0402","name":"心理学","total":"52","unitcode":"14430","unitname":"中国科学院大

学","result":"A-","percent":"5%~10%"},
{"code":"0602","name":"中国史","total":"85","unitcode":"87903","unitname":"上海社会科学

院","result":"C","percent":"50%~60%"},
{"code":"0603","name":"世界史","total":"60","unitcode":"87802","unitname":"黑龙江省社会科学

院","result":"C-","percent":"60%~70%"},
{"code":"0701","name":"数学","total":"183","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0702","name":"物理学","total":"129","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0702","name":"物理学","total":"129","unitcode":"82801","unitname":"中国原子能科学

研究院","result":"B","percent":"20%~30%"},
{"code":"0703","name":"化学","total":"152","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0703","name":"化学","total":"152","unitcode":"82801","unitname":"中国原子能科学研

究院","result":"C","percent":"50%~60%"},
{"code":"0704","name":"天文学","total":"9","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0705","name":"地理学","total":"61","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0706","name":"大气科学","total":"16","unitcode":"14430","unitname":"中国科学院大学

","result":"A+","percent":"前2%"},
{"code":"0707","name":"海洋科学","total":"25","unitcode":"14430","unitname":"中国科学院大学

","result":"A+","percent":"前2%"},
{"code":"0707","name":"海洋科学","total":"25","unitcode":"85302","unitname":"国家海洋局第二

海洋研究所","result":"B+","percent":"10%~20%"},
{"code":"0707","name":"海洋科学","total":"25","unitcode":"85301","unitname":"国家海洋局第一

海洋研究所","result":"B","percent":"20%~30%"},
{"code":"0708","name":"地球物理学","total":"14","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0709","name":"地质学","total":"23","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0710","name":"生物学","total":"164","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0710","name":"生物学","total":"164","unitcode":"82101","unitname":"中国农业科学

院","result":"A-","percent":"5%~10%"},
{"code":"0710","name":"生物学","total":"164","unitcode":"82405","unitname":"中国环境科学研

究院","result":"C-","percent":"60%~70%"},
{"code":"0711","name":"系统科学","total":"13","unitcode":"14430","unitname":"中国科学院大学

","result":"A+","percent":"前2%"},
{"code":"0712","name":"科学技术史","total":"22","unitcode":"14430","unitname":"中国科学院大

学","result":"B+","percent":"10%~20%"},
{"code":"0713","name":"生态学","total":"103","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0713","name":"生态学","total":"103","unitcode":"82101","unitname":"中国农业科学

院","result":"B+","percent":"10%~20%"},
{"code":"0713","name":"生态学","total":"103","unitcode":"82405","unitname":"中国环境科学研

究院","result":"B","percent":"20%~30%"},
{"code":"0714","name":"统计学","total":"121","unitcode":"14430","unitname":"中国科学院大

学","result":"A-","percent":"5%~10%"},
{"code":"0801","name":"力学","total":"81","unitcode":"14430","unitname":"中国科学院大

学","result":"A-","percent":"5%~10%"},
{"code":"0802","name":"机械工程","total":"191","unitcode":"14430","unitname":"中国科学院大

学","result":"B+","percent":"10%~20%"},
{"code":"0803","name":"光学工程","total":"81","unitcode":"14430","unitname":"中国科学院大学

","result":"A","percent":"2%~5%"},
{"code":"0804","name":"仪器科学与技术","total":"74","unitcode":"14430","unitname":"中国科学

院大学","result":"A-","percent":"5%~10%"},
{"code":"0805","name":"材料科学与工程","total":"174","unitcode":"14430","unitname":"中国科

学院大学","result":"A","percent":"2%~5%"},
{"code":"0805","name":"材料科学与工程","total":"174","unitcode":"86401","unitname":"昆明贵

金属研究所","result":"C","percent":"50%~60%"},
{"code":"0807","name":"动力工程及工程热物理","total":"86","unitcode":"14430","unitname":"中

国科学院大学","result":"B+","percent":"10%~20%"},
{"code":"0807","name":"动力工程及工程热物理","total":"86","unitcode":"82304","unitname":"西

安热工研究院","result":"C+","percent":"40%~50%"},
{"code":"0808","name":"电气工程","total":"86","unitcode":"14430","unitname":"中国科学院大学

","result":"B+","percent":"10%~20%"},
{"code":"0808","name":"电气工程","total":"86","unitcode":"82302","unitname":"中国电力科学研

究院","result":"B+","percent":"10%~20%"},
{"code":"0809","name":"电子科学与技术","total":"107","unitcode":"14430","unitname":"中国科

学院大学","result":"A+","percent":"前2%"},
{"code":"0810","name":"信息与通信工程","total":"138","unitcode":"14430","unitname":"中国科

学院大学","result":"A","percent":"2%~5%"},
{"code":"0811","name":"控制科学与工程","total":"163","unitcode":"14430","unitname":"中国科

学院大学","result":"A","percent":"2%~5%"},
{"code":"0812","name":"计算机科学与技术","total":"239","unitcode":"14430","unitname":"中国

科学院大学","result":"A+","percent":"前2%"},
{"code":"0814","name":"土木工程","total":"136","unitcode":"14430","unitname":"中国科学院大

学","result":"A-","percent":"5%~10%"},
{"code":"0816","name":"测绘科学与技术","total":"35","unitcode":"14430","unitname":"中国科学

院大学","result":"B+","percent":"10%~20%"},
{"code":"0817","name":"化学工程与技术","total":"147","unitcode":"14430","unitname":"中国科

学院大学","result":"A","percent":"2%~5%"},
{"code":"0817","name":"化学工程与技术","total":"147","unitcode":"83301","unitname":"煤炭科

学研究总院","result":"C+","percent":"40%~50%"},
{"code":"0818","name":"地质资源与地质工程","total":"41","unitcode":"14430","unitname":"中国

科学院大学","result":"B+","percent":"10%~20%"},
{"code":"0818","name":"地质资源与地质工程","total":"41","unitcode":"83301","unitname":"煤炭

科学研究总院","result":"B","percent":"20%~30%"},
{"code":"0819","name":"矿业工程","total":"29","unitcode":"83301","unitname":"煤炭科学研究总

院","result":"C","percent":"50%~60%"},
{"code":"0825","name":"航空宇航科学与技术","total":"26","unitcode":"14430","unitname":"中国

科学院大学","result":"C-","percent":"60%~70%"},
{"code":"0827","name":"核科学与技术","total":"22","unitcode":"14430","unitname":"中国科学院

大学","result":"B+","percent":"10%~20%"},
{"code":"0827","name":"核科学与技术","total":"22","unitcode":"82801","unitname":"中国原子能

科学研究院","result":"B+","percent":"10%~20%"},
{"code":"0828","name":"农业工程","total":"38","unitcode":"82101","unitname":"中国农业科学院

","result":"C+","percent":"40%~50%"},
{"code":"0830","name":"环境科学与工程","total":"159","unitcode":"14430","unitname":"中国科

学院大学","result":"A+","percent":"前2%"},
{"code":"0830","name":"环境科学与工程","total":"159","unitcode":"82405","unitname":"中国环

境科学研究院","result":"B+","percent":"10%~20%"},
{"code":"0830","name":"环境科学与工程","total":"159","unitcode":"82101","unitname":"中国农

业科学院","result":"B-","percent":"30%~40%"},
{"code":"0830","name":"环境科学与工程","total":"159","unitcode":"85301","unitname":"国家海

洋局第一海洋研究所","result":"C-","percent":"60%~70%"},
{"code":"0831","name":"生物医学工程","total":"71","unitcode":"14430","unitname":"中国科学院

大学","result":"B+","percent":"10%~20%"},
{"code":"0832","name":"食品科学与工程","total":"80","unitcode":"82101","unitname":"中国农业

科学院","result":"B+","percent":"10%~20%"},
{"code":"0835","name":"软件工程","total":"166","unitcode":"14430","unitname":"中国科学院大

学","result":"A-","percent":"5%~10%"},
{"code":"0837","name":"安全科学与工程","total":"53","unitcode":"83301","unitname":"煤炭科学

研究总院","result":"C","percent":"50%~60%"},
{"code":"0901","name":"作物学","total":"44","unitcode":"82101","unitname":"中国农业科学

院","result":"A+","percent":"前2%"},
{"code":"0901","name":"作物学","total":"44","unitcode":"14430","unitname":"中国科学院大

学","result":"C+","percent":"40%~50%"},
{"code":"0902","name":"园艺学","total":"37","unitcode":"82101","unitname":"中国农业科学

院","result":"B+","percent":"10%~20%"},
{"code":"0903","name":"农业资源与环境","total":"36","unitcode":"14430","unitname":"中国科学

院大学","result":"A+","percent":"前2%"},
{"code":"0903","name":"农业资源与环境","total":"36","unitcode":"82101","unitname":"中国农业

科学院","result":"A-","percent":"5%~10%"},
{"code":"0904","name":"植物保护","total":"36","unitcode":"82101","unitname":"中国农业科学院

","result":"A+","percent":"前2%"},
{"code":"0905","name":"畜牧学","total":"46","unitcode":"82101","unitname":"中国农业科学

院","result":"A+","percent":"前2%"},
{"code":"0905","name":"畜牧学","total":"46","unitcode":"14430","unitname":"中国科学院大

学","result":"B","percent":"20%~30%"},
{"code":"0906","name":"兽医学","total":"42","unitcode":"82101","unitname":"中国农业科学

院","result":"A+","percent":"前2%"},
{"code":"0907","name":"林学","total":"31","unitcode":"14430","unitname":"中国科学院大

学","result":"B","percent":"20%~30%"},
{"code":"0908","name":"水产","total":"20","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"0909","name":"草学","total":"27","unitcode":"82101","unitname":"中国农业科学

院","result":"B+","percent":"10%~20%"},
{"code":"1001","name":"基础医学","total":"80","unitcode":"14430","unitname":"中国科学院大学

","result":"B-","percent":"30%~40%"},
{"code":"1005","name":"中医学","total":"38","unitcode":"84502","unitname":"中国中医科学

院","result":"B+","percent":"10%~20%"},
{"code":"1006","name":"中西医结合","total":"55","unitcode":"84502","unitname":"中国中医科学

院","result":"A-","percent":"5%~10%"},
{"code":"1007","name":"药学","total":"105","unitcode":"14430","unitname":"中国科学院大

学","result":"A+","percent":"前2%"},
{"code":"1008","name":"中药学","total":"45","unitcode":"84502","unitname":"中国中医科学

院","result":"B+","percent":"10%~20%"},
{"code":"1008","name":"中药学","total":"45","unitcode":"14430","unitname":"中国科学院大

学","result":"C-","percent":"60%~70%"},
{"code":"1201","name":"管理科学与工程","total":"189","unitcode":"14430","unitname":"中国科

学院大学","result":"A","percent":"2%~5%"},
{"code":"1201","name":"管理科学与工程","total":"189","unitcode":"82101","unitname":"中国农

业科学院","result":"C","percent":"50%~60%"},
{"code":"1202","name":"工商管理","total":"241","unitcode":"14430","unitname":"中国科学院大

学","result":"B-","percent":"30%~40%"},
{"code":"1203","name":"农林经济管理","total":"41","unitcode":"82101","unitname":"中国农业科

学院","result":"B","percent":"20%~30%"},
{"code":"1204","name":"公共管理","total":"146","unitcode":"14430","unitname":"中国科学院大

学","result":"C+","percent":"40%~50%"},
{"code":"1205","name":"图书情报与档案管理","total":"40","unitcode":"14430","unitname":"中国

科学院大学","result":"B+","percent":"10%~20%"}
];

坚守VFP最后的阵地
2021-06-13 16:30
sdta
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
来 自:江苏省连云港市
等 级:版主
威 望:323
帖 子:9621
专家分:26174
注 册:2012-2-5
得分:0 
以下是引用吹水佬在2021-6-13 15:33:25的发言:


可能刷得快了

下载一次在1分钟左右,共5112条记录,打开95次网页,这个频率快吗

坚守VFP最后的阵地
2021-06-13 16:32
sdta
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
来 自:江苏省连云港市
等 级:版主
威 望:323
帖 子:9621
专家分:26174
注 册:2012-2-5
得分:0 
以下是引用jinanshui在2021-6-13 15:33:23的发言:

老师,我想先看一个学科的,老师请看看我的代码有那些错误,我是个大菜鸟,请多笑话
CREATE CURSOR 学校 (h1 C(4),h2 C(4), h3 C(5), h4 C(20),h5 C(4))
lcUrl = "https://souky.
oHTTP=NEWOBJECT("MSXML2.XMLHTTP")
oHTTP.Open("GET", lcUrl, .F.)
oHTTP.SETREQUESTHEADER("Content-Type", "text/html;charset=UTF-8")
oHTTP.Send()
lcText = STRCONV(oHTTP.ResponseBody, 11)
lyText = STREXTRACT  (lcText,"</span> <span id='name'>","</span></div>")
laText = STREXTRACT  (lcText,"评选结果</td>","</table>")
FOR  k =1  TO  65
lxText="<td>"-ALLTRIM(STR(k))+"<td>"
lbText = STREXTRACT(laText,lxText,"</tr>")
? lbText
endfor

有点问题,自己上机测试

坚守VFP最后的阵地
2021-06-13 16:40
吹水佬
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
等 级:版主
威 望:432
帖 子:10064
专家分:41463
注 册:2014-5-20
得分:0 
以下是引用sdta在2021-6-13 16:32:53的发言:


下载一次在1分钟左右,共5112条记录,打开95次网页,这个频率快吗

问题是人家认为你太快了,以为你在做什么被封禁2小时
2021-06-13 16:45
sdta
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
来 自:江苏省连云港市
等 级:版主
威 望:323
帖 子:9621
专家分:26174
注 册:2012-2-5
得分:0 
评选结果.rar (1.41 KB)

坚守VFP最后的阵地
2021-06-13 17:07
sdta
Rank: 20Rank: 20Rank: 20Rank: 20Rank: 20
来 自:江苏省连云港市
等 级:版主
威 望:323
帖 子:9621
专家分:26174
注 册:2012-2-5
得分:0 
学科fl.rar (1.17 KB)

坚守VFP最后的阵地
2021-06-13 17:23
jinanshui
Rank: 2
等 级:论坛游民
帖 子:274
专家分:64
注 册:2009-10-6
得分:0 
版主老师,能否把代码发上来,供我们学习一下?谢谢.
2021-06-13 19:20



参与讨论请移步原网站贴子:https://bbs.bccn.net/thread-506002-1-1.html




关于我们 | 广告合作 | 编程中国 | 清除Cookies | TOP | 手机版

编程中国 版权所有,并保留所有权利。
Powered by Discuz, Processed in 0.232904 second(s), 8 queries.
Copyright©2004-2024, BCCN.NET, All Rights Reserved