kazu634
/
blog
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
blog/content/post/2008/12/06/2008-12-06-00001087.md

44 KiB
Raw Blame History

title author date wordtwit_post_info categories
店舗検索結果のURLを一覧表示にする kazu634 2008-12-06
O:8:"stdClass":13:{s:6:"manual";b:0;s:11:"tweet_times";i:1;s:5:"delay";i:0;s:7:"enabled";i:1;s:10:"separation";s:2:"60";s:7:"version";s:3:"3.7";s:14:"tweet_template";b:0;s:6:"status";i:2;s:6:"result";a:0:{}s:13:"tweet_counter";i:2;s:13:"tweet_log_ids";a:1:{i:0;i:4423;}s:9:"hash_tags";a:0:{}s:8:"accounts";a:1:{i:0;s:7:"kazu634";}}
Perl
starbucks

「 Starbucksの店舗検索からスクレイピング – 武蔵の日記」の続きだよ。

この前までの復習と今日やったこと

「スターバックス コーヒー | 店舗検索」から各県の店舗検索結果の一ページ目を取得することができます。また、店舗検索結果のページから店舗情報を取得することもできます。

今回は店舗検索結果の一ページ目～最後のページまでのURLを取得することを目標とします。

具体的にやること

この「次の10件」というリンク先が存在すれば、そのURLを取得し、そのURL先へと飛んで同じように「次の10件」が存在するかどうかをチェックする…ということを繰り返します。再帰が使えそう。

f:id:sirocco634:20081206223124p:image

コード 

# === Libraries ===
use strict;
use warnings;
use Data::Dumper;
# character modules
use Encode;
use utf8;
# WWW::Mechanize
use WWW::Mechanize;
use URI;
binmode( STDOUT, ':raw :encoding(utf8)' );
binmode( STDERR, ':raw :encoding(utf8)' );
# ====================
# === Main Routine ===
# ====================
# creating the object of WWW::Mechanize
my $mech = WWW::Mechanize->new();
&list_url('http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7&SearchCity=&storelist=1');
sub list_url {
my $url=shift();
# Get the contents of the url
$mech->get($url);
# Show the error message, if it fails.
die $mech->response->status_line unless $mech->success;
my $html = decode( 'sjis', $mech->content );
if ( $html =~ /<a href="(.+)">次の10件<\/a>/ ) {
my $next = $1;
my $url = URI->new_abs( $next, $mech->base );
print "$url\n";
&list_url($url);
}
}

店舗検索トップページから一覧にして列挙

店舗検索から、各県の店舗検索結果の一ページ目を取得するスクリプトは作成済みなので、上のスクリプトと組み合わせてみました。 

# === Libraries ===
use strict;
use warnings;
# character modules
use Encode;
use utf8;
# WWW::Mechanize
use WWW::Mechanize;
use URI;
binmode( STDOUT, ':raw :encoding(utf8)' );
binmode( STDERR, ':raw :encoding(utf8)' );
# ====================
# === Main Routine ===
# ====================
my @url = &get_url_list();
foreach my $x (@url) {
&list_url($x);
}
exit;
# ====================
# === sub routines ===
# ====================
# Sub-routine for getting the list of the urls.
sub get_url_list {
# 検索結果のページURLを格納する配列
my @links;
# creating the object of WWW::Mechanize
my $mech = WWW::Mechanize->new();
# Get the contents of the url
$mech->get("http://www.starbucks.co.jp/search/index.html/");
# Show the error message, if it fails.
die $mech->response->status_line unless $mech->success;
# Get the list of urls of each Starbucks Stores
my @lnks = $mech->find_all_links( url_regex => qr/SearchPerfecture/ );
foreach my $url (@lnks) {
my $tmp = $url->url;
# Change the relative url into the absolute url
$tmp = URI->new_abs( $tmp, $url->base );
$tmp = decode( 'shift-jis', $tmp );
push( @links, $tmp );
}
return @links;
}
sub list_url {
my $mech = WWW::Mechanize->new();
my $url=shift();
print "$url\n";
# Get the contents of the url
$mech->get($url);
# Show the error message, if it fails.
die $mech->response->status_line unless $mech->success;
my $html = decode( 'sjis', $mech->content );
if ( $html =~ /<a href="(.+)">次の10件<\/a>/ ) {
my $next = $1;
my $url = URI->new_abs( $next, $mech->base );
&list_url($url);
}
}
実行結果

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%96k%8AC%93%B9

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%96k%8AC%93%B9&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%C2%90X%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8A%E2%8E%E8%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%7B%8F%E9%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%7B%8F%E9%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8FH%93c%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8ER%8C%60%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%93%87%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%EF%8F%E9%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%EF%8F%E9%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%C8%96%D8%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%C8%96%D8%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8CQ%94n%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%E9%8B%CA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%E9%8B%CA%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%E9%8B%CA%8C%A7&storelist=21

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%E9%8B%CA%8C%A7&storelist=31

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%E9%8B%CA%8C%A7&storelist=41

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%E7%97t%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%E7%97t%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%E7%97t%8C%A7&storelist=21

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%E7%97t%8C%A7&storelist=31

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%E7%97t%8C%A7&storelist=41

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8ER%97%9C%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%92%B7%96%EC%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90V%8A%83%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95x%8ER%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%CE%90%EC%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%88%E4%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8E%A0%89%EA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%9E%93s%95%7B

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%9E%93s%95%7B&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%BA%8C%C9%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%BA%8C%C9%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%BA%8C%C9%8C%A7&storelist=21

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%98a%89%CC%8ER%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%DE%97%C7%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%92%B9%8E%E6%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%87%8D%AA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%89%AA%8ER%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8DL%93%87%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8DL%93%87%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8ER%8C%FB%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%89%AA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%89%AA%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%89%AA%8C%A7&storelist=21

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%89%AA%8C%A7&storelist=31

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%B2%89%EA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%92%B7%8D%E8%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8CF%96%7B%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%91%E5%95%AA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%7B%8D%E8%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8E%AD%8E%99%93%87%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%89%AB%93%EA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%89%AB%93%EA%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%BF%93%87%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%81%90%EC%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%95Q%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%82%92m%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%C3%89%AA%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%C3%89%AA%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7&storelist=11

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7&storelist=21

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7&storelist=31

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7&storelist=41

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8A%F2%95%8C%8C%A7

http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8EO%8Fd%8C%A7