kazu634
/
blog
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
blog/content/post/2009/09/14/2009-09-14-00001221.md

22 KiB
Raw Blame History

title author date url wordtwit_post_info categories
フィルターの概念を覚えた kazu634 2009-09-14 /2009/09/14/_1327/
O:8:"stdClass":13:{s:6:"manual";b:0;s:11:"tweet_times";i:1;s:5:"delay";i:0;s:7:"enabled";i:1;s:10:"separation";s:2:"60";s:7:"version";s:3:"3.7";s:14:"tweet_template";b:0;s:6:"status";i:2;s:6:"result";a:0:{}s:13:"tweet_counter";i:2;s:13:"tweet_log_ids";a:1:{i:0;i:4779;}s:9:"hash_tags";a:0:{}s:8:"accounts";a:1:{i:0;s:7:"kazu634";}}
Perl

Web::Scraperにはフィルターというものがあります。要はスクレイピングした結果を anonymous subroutine に渡して、結果を操作できてしまう…というものです。

なんでそんなのが必要なの？

Starbucksの店舗情報をスクレイピングしていて、HTMLの文字コードはShift-JISでした。そして、リンクへのURLには日本語が付加されていました。

ここでWeb::Scraperを用いてURLを取り込むと、私の環境ではUTF-8でエンコードされたURLが取得され、そのURLでアクセスしてもリンク先へ飛ぶことが出来ませんでした。UTF-8で取り込まれてしまったエンコード済みURLをデコードし、Shift-JISに変換し、再度エンコードするという処理が必要だということがわかりました。

ソース

実際にやってみます: 

use strict;
use Web::Scraper;
use URI;
use URI::Escape;
use utf8;
use YAML;
use Perl6::Say;
use Encode;
# =================
# === Main Part ===
# =================
# =========================
# === 各県へのURLを取得 ===
# =========================
# starbucksのURLを指定
my $uri = URI->new("http://www.starbucks.co.jp/search/index.html");
# スクレイピングの設定を行う
my $scraper = scraper {
process '//area[@href=~/.+SearchPerfecture/]', 'prefs[]' => [
'@href',
        sub {
my $url         = $_->as_string;
my @url_split   = split( /=/, $url );
# utf8なURLをデコード
my $utf8_encode = pop(@url_split);
my $utf8        = uri_unescape($utf8_encode);
# UTF-8に変換
my $temp        = decode( 'utf8', $utf8 );
# Shift-JISに変換
$temp = encode( 'shiftjis', $temp );
# 再度エンコード
$temp = uri_escape($temp);
push( @url_split, $temp );
return {'shiftjis' => join( '=', @url_split ), 'utf-8' => $_->as_string};
}
];
process '//td[@class="SelectFromPlace"]//a', 'citys[]' => '@href';
};
# スクレイピングの実行
my $result = $scraper->scrape($uri);
say YAML::Dump($result);

実行例 

kazu634@srv634% perl test.pl                                                                   ~/work/tmp_perl/scrap [6054]
---
citys:
- !!perl/scalar:URI::http http://www.starbucks.co.jp/search/tokyo.php
- !!perl/scalar:URI::http http://www.starbucks.co.jp/search/kanagawa.php
- !!perl/scalar:URI::http http://www.starbucks.co.jp/search/osaka.php
prefs:
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%96k%8AC%93%B9
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%8C%97%E6%B5%B7%E9%81%93
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%C2%90X%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%9D%92%E6%A3%AE%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8A%E2%8E%E8%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B2%A9%E6%89%8B%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%7B%8F%E9%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%AE%AE%E5%9F%8E%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8FH%93c%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%A7%8B%E7%94%B0%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8ER%8C%60%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B1%B1%E5%BD%A2%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%93%87%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%A6%8F%E5%B3%B6%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%EF%8F%E9%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E8%8C%A8%E5%9F%8E%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%C8%96%D8%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E6%A0%83%E6%9C%A8%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8CQ%94n%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%BE%A4%E9%A6%AC%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%E9%8B%CA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%9F%BC%E7%8E%89%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%E7%97t%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%8D%83%E8%91%89%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8ER%97%9C%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B1%B1%E6%A2%A8%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%92%B7%96%EC%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%95%B7%E9%87%8E%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90V%8A%83%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E6%96%B0%E6%BD%9F%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95x%8ER%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%AF%8C%E5%B1%B1%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%CE%90%EC%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%9F%B3%E5%B7%9D%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%88%E4%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%A6%8F%E4%BA%95%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8E%A0%89%EA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E6%BB%8B%E8%B3%80%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%9E%93s%95%7B
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E4%BA%AC%E9%83%BD%E5%BA%9C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%BA%8C%C9%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%85%B5%E5%BA%AB%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%98a%89%CC%8ER%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%92%8C%E6%AD%8C%E5%B1%B1%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%DE%97%C7%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%A5%88%E8%89%AF%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%92%B9%8E%E6%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%B3%A5%E5%8F%96%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%87%8D%AA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B3%B6%E6%A0%B9%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%89%AA%8ER%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B2%A1%E5%B1%B1%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8DL%93%87%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%BA%83%E5%B3%B6%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8ER%8C%FB%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B1%B1%E5%8F%A3%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%95%9F%89%AA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%A6%8F%E5%B2%A1%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%B2%89%EA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E4%BD%90%E8%B3%80%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%92%B7%8D%E8%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%95%B7%E5%B4%8E%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8CF%96%7B%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E7%86%8A%E6%9C%AC%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%91%E5%95%AA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%A4%A7%E5%88%86%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8B%7B%8D%E8%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%AE%AE%E5%B4%8E%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8E%AD%8E%99%93%87%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%B9%BF%E5%85%90%E5%B3%B6%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%89%AB%93%EA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E6%B2%96%E7%B8%84%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%93%BF%93%87%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%BE%B3%E5%B3%B6%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%81%90%EC%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%A6%99%E5%B7%9D%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%95Q%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E6%84%9B%E5%AA%9B%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8D%82%92m%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%AB%98%E7%9F%A5%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%90%C3%89%AA%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E9%9D%99%E5%B2%A1%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%88%A4%92m%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E6%84%9B%E7%9F%A5%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8A%F2%95%8C%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E5%B2%90%E9%98%9C%E7%9C%8C
- shiftjis: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%8EO%8Fd%8C%A7
utf-8: http://www.starbucks.co.jp/search/result_city2.php?SearchPerfecture=%E4%B8%89%E9%87%8D%E7%9C%8C

shift-jisとutf8でまったくURLが異なっていることに気づきましたか？shift-jisだと、正常なページにリンクされますが、utf8なURLだと正常ではないページに飛んでいきます。

こんな機能があるだなんて、便利だ。