kazu634
/
blog
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
blog/content/post/1970-01-01-00000003.md

32 KiB
Raw Blame History

title author date url wordtwit_post_info categories
Fluentd を試してみた kazu634 1969-12-31 /1970/01/01/_3/
O:8:"stdClass":13:{s:6:"manual";b:0;s:11:"tweet_times";i:1;s:5:"delay";i:0;s:7:"enabled";i:1;s:10:"separation";s:2:"60";s:7:"version";s:3:"3.7";s:14:"tweet_template";b:0;s:6:"status";i:2;s:6:"result";a:0:{}s:13:"tweet_counter";i:2;s:13:"tweet_log_ids";a:1:{i:0;i:5461;}s:9:"hash_tags";a:0:{}s:8:"accounts";a:1:{i:0;s:7:"kazu634";}}
つれづれ

仕事でログ管理サーバというものに日次でログを転送して、解析という処理を行なっていました。

しかしここで問題が発生。Windowsの更新日付を見て、前回日付から更新されていればログを転送するロジックなのに、ログが書き込まれても更新日付が更新されていなかった。。。原因は調査中。。。そういったトラブルに見舞われた結果、モダンなログ監視手法を勉強したいと思っていたのでした。

そんなときに読んだ『WEB+DB PRESS Vol.69』で紹介されていた Fluentd | Open Source Data Collector を試して見ることを決意したのでした。

fluentd のいいところ

WEB+DB PRESS Vol.69によれば既存のログ解析システムには以下の問題点があると指摘しています:

  1. データの即時性
  2. データコピー処理が不安定
  3. 障害対応の苦労

fluentd の tail プラグインを用いると、 tail -f して追記されたものを随時転送してくれるため、即時に転送されます。コピー処理についてもリトライを行なってくれます。また、ログ転送をほぼリアルタイムに行えるため、ログ解析処理を日中帯に実行でき、障害対応の苦労が軽減されます。

導入方法

ubuntu の deb パッケージ・td-agent を使用して導入してみました。 

kazu634@fluent-master:~$ cd /etc/apt
kazu634@fluent-master:/etc/apt$ ll
total 48
drwxr-xr-x  6 root root 4096  6月 28 00:02 ./
drwxr-xr-x 85 root root 4096  6月 28 00:01 ../
drwxr-xr-x  2 root root 4096  6月 24 13:13 apt.conf.d/
drwxr-xr-x  2 root root 4096  4月 20 19:21 preferences.d/
-rw-r--r--  1 root root 3377  6月 28 00:02 sources.list
drwxr-xr-x  2 root root 4096  4月 20 19:21 sources.list.d/
-rw-------  1 root root 1200  6月 24 13:00 trustdb.gpg
-rw-r--r--  1 root root 6713  6月 24 13:00 trusted.gpg
drwxr-xr-x  2 root root 4096  4月 20 19:21 trusted.gpg.d/
-rw-r--r--  1 root root 6713  6月 24 13:00 trusted.gpg~
kazu634@fluent-master:/etc/apt$ sudo cp -p sources.list sources.list.20120627
kazu634@fluent-master:/etc/apt$ sudo vi sources.list
kazu634@fluent-master:/etc/apt$ diff -u sources.list.20120628 sources.list
--- sources.list.20120628	2012-06-28 00:02:06.263023624 +0900
+++ sources.list	2012-06-28 00:02:50.683021714 +0900
@@ -59,3 +59,5 @@
 ## developers who want to ship their latest software.
 # deb http://extras.ubuntu.com/ubuntu precise main
 # deb-src http://extras.ubuntu.com/ubuntu precise main
+
+deb http://packages.treasure-data.com/debian/ lucid contrib
kazu634@fluent-master:/etc/apt$ sudo aptitude update
Ign http://ubuntutym.u-toyama.ac.jp precise InRelease
Ign http://ubuntutym.u-toyama.ac.jp precise-updates InRelease
Ign http://ubuntutym.u-toyama.ac.jp precise-backports InRelease
Get: 1 http://ubuntutym.u-toyama.ac.jp precise Release.gpg [198 B]
Get: 2 http://ubuntutym.u-toyama.ac.jp precise-updates Release.gpg [198 B]
Get: 3 http://ubuntutym.u-toyama.ac.jp precise-backports Release.gpg [198 B]
Get: 4 http://ubuntutym.u-toyama.ac.jp precise Release [49.6 kB]
Get: 5 http://ubuntutym.u-toyama.ac.jp precise-updates Release [49.6 kB]
Get: 6 http://ubuntutym.u-toyama.ac.jp precise-backports Release [49.6 kB]
Get: 7 http://ubuntutym.u-toyama.ac.jp precise/main Sources [934 kB]
Ign http://security.ubuntu.com precise-security InRelease
Get: 8 http://security.ubuntu.com precise-security Release.gpg [198 B]
Ign http://packages.treasure-data.com lucid InRelease
Get: 9 http://security.ubuntu.com precise-security Release [49.6 kB]
Ign http://packages.treasure-data.com lucid Release.gpg
Hit http://packages.treasure-data.com lucid Release
Ign http://packages.treasure-data.com lucid/contrib amd64 Packages/DiffIndex
Get: 10 http://ubuntutym.u-toyama.ac.jp precise/restricted Sources [5,470 B]
Get: 11 http://ubuntutym.u-toyama.ac.jp precise/universe Sources [5,019 kB]
Ign http://packages.treasure-data.com lucid/contrib i386 Packages/DiffIndex
Ign http://packages.treasure-data.com lucid/contrib TranslationIndex
Get: 12 http://security.ubuntu.com precise-security/main Sources [21.1 kB]
Get: 13 http://security.ubuntu.com precise-security/restricted Sources [14 B]
Get: 14 http://security.ubuntu.com precise-security/universe Sources [7,120 B]
Get: 15 http://security.ubuntu.com precise-security/multiverse Sources [713 B]
Get: 16 http://security.ubuntu.com precise-security/main amd64 Packages [64.3 kB]
Get: 17 http://security.ubuntu.com precise-security/restricted amd64 Packages [14 B]
Get: 18 http://security.ubuntu.com precise-security/universe amd64 Packages [17.2 kB]
Get: 19 http://security.ubuntu.com precise-security/multiverse amd64 Packages [1,155 B]
Get: 20 http://security.ubuntu.com precise-security/main i386 Packages [65.9 kB]
Hit http://packages.treasure-data.com lucid/contrib amd64 Packages
Hit http://packages.treasure-data.com lucid/contrib i386 Packages
Get: 21 http://security.ubuntu.com precise-security/restricted i386 Packages [14 B]
Get: 22 http://security.ubuntu.com precise-security/universe i386 Packages [17.4 kB]
Get: 23 http://security.ubuntu.com precise-security/multiverse i386 Packages [1,394 B]
Hit http://security.ubuntu.com precise-security/main TranslationIndex
Hit http://security.ubuntu.com precise-security/multiverse TranslationIndex
Hit http://security.ubuntu.com precise-security/restricted TranslationIndex
Hit http://security.ubuntu.com precise-security/universe TranslationIndex
Ign http://packages.treasure-data.com lucid/contrib Translation-ja_JP
Hit http://security.ubuntu.com precise-security/main Translation-en
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Hit http://security.ubuntu.com precise-security/restricted Translation-en
Ign http://packages.treasure-data.com lucid/contrib Translation-ja
Hit http://security.ubuntu.com precise-security/universe Translation-en
Ign http://packages.treasure-data.com lucid/contrib Translation-en
Get: 24 http://ubuntutym.u-toyama.ac.jp precise/multiverse Sources [155 kB]
Get: 25 http://ubuntutym.u-toyama.ac.jp precise/main amd64 Packages [1,273 kB]
Get: 26 http://ubuntutym.u-toyama.ac.jp precise/restricted amd64 Packages [8,452 B]
Get: 27 http://ubuntutym.u-toyama.ac.jp precise/universe amd64 Packages [4,786 kB]
Get: 28 http://ubuntutym.u-toyama.ac.jp precise/multiverse amd64 Packages [119 kB]
Get: 29 http://ubuntutym.u-toyama.ac.jp precise/main i386 Packages [1,274 kB]
Get: 30 http://ubuntutym.u-toyama.ac.jp precise/restricted i386 Packages [8,431 B]
Get: 31 http://ubuntutym.u-toyama.ac.jp precise/universe i386 Packages [4,796 kB]
Get: 32 http://ubuntutym.u-toyama.ac.jp precise/multiverse i386 Packages [121 kB]
Hit http://ubuntutym.u-toyama.ac.jp precise/main TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise/multiverse TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise/restricted TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise/universe TranslationIndex
Get: 33 http://ubuntutym.u-toyama.ac.jp precise-updates/main Sources [117 kB]
Get: 34 http://ubuntutym.u-toyama.ac.jp precise-updates/restricted Sources [1,379 B]
Get: 35 http://ubuntutym.u-toyama.ac.jp precise-updates/universe Sources [28.2 kB]
Get: 36 http://ubuntutym.u-toyama.ac.jp precise-updates/multiverse Sources [1,058 B]
Get: 37 http://ubuntutym.u-toyama.ac.jp precise-updates/main amd64 Packages [297 kB]
Get: 38 http://ubuntutym.u-toyama.ac.jp precise-updates/restricted amd64 Packages [2,417 B]
Get: 39 http://ubuntutym.u-toyama.ac.jp precise-updates/universe amd64 Packages [81.1 kB]
Get: 40 http://ubuntutym.u-toyama.ac.jp precise-updates/multiverse amd64 Packages [1,825 B]
Get: 41 http://ubuntutym.u-toyama.ac.jp precise-updates/main i386 Packages [299 kB]
Get: 42 http://ubuntutym.u-toyama.ac.jp precise-updates/restricted i386 Packages [2,439 B]
Get: 43 http://ubuntutym.u-toyama.ac.jp precise-updates/universe i386 Packages [81.6 kB]
Get: 44 http://ubuntutym.u-toyama.ac.jp precise-updates/multiverse i386 Packages [2,049 B]
Get: 45 http://ubuntutym.u-toyama.ac.jp precise-updates/main TranslationIndex [74 B]
Get: 46 http://ubuntutym.u-toyama.ac.jp precise-updates/multiverse TranslationIndex [71 B]
Get: 47 http://ubuntutym.u-toyama.ac.jp precise-updates/restricted TranslationIndex [71 B]
Get: 48 http://ubuntutym.u-toyama.ac.jp precise-updates/universe TranslationIndex [73 B]
Get: 49 http://ubuntutym.u-toyama.ac.jp precise-backports/main Sources [1,346 B]
Get: 50 http://ubuntutym.u-toyama.ac.jp precise-backports/restricted Sources [14 B]
Get: 51 http://ubuntutym.u-toyama.ac.jp precise-backports/universe Sources [6,873 B]
Get: 52 http://ubuntutym.u-toyama.ac.jp precise-backports/multiverse Sources [1,383 B]
Get: 53 http://ubuntutym.u-toyama.ac.jp precise-backports/main amd64 Packages [929 B]
Get: 54 http://ubuntutym.u-toyama.ac.jp precise-backports/restricted amd64 Packages [14 B]
Get: 55 http://ubuntutym.u-toyama.ac.jp precise-backports/universe amd64 Packages [6,114 B]
Get: 56 http://ubuntutym.u-toyama.ac.jp precise-backports/multiverse amd64 Packages [996 B]
Get: 57 http://ubuntutym.u-toyama.ac.jp precise-backports/main i386 Packages [929 B]
Get: 58 http://ubuntutym.u-toyama.ac.jp precise-backports/restricted i386 Packages [14 B]
Get: 59 http://ubuntutym.u-toyama.ac.jp precise-backports/universe i386 Packages [6,117 B]
Get: 60 http://ubuntutym.u-toyama.ac.jp precise-backports/multiverse i386 Packages [999 B]
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/main TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/multiverse TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/restricted TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/universe TranslationIndex
Hit http://ubuntutym.u-toyama.ac.jp precise/main Translation-ja
Hit http://ubuntutym.u-toyama.ac.jp precise/main Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise/multiverse Translation-ja
Hit http://ubuntutym.u-toyama.ac.jp precise/multiverse Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise/restricted Translation-ja
Hit http://ubuntutym.u-toyama.ac.jp precise/restricted Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise/universe Translation-ja
Hit http://ubuntutym.u-toyama.ac.jp precise/universe Translation-en
Get: 61 http://ubuntutym.u-toyama.ac.jp precise-updates/main Translation-en [136 kB]
Hit http://ubuntutym.u-toyama.ac.jp precise-updates/multiverse Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise-updates/restricted Translation-en
Get: 62 http://ubuntutym.u-toyama.ac.jp precise-updates/universe Translation-en [48.4 kB]
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/main Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/multiverse Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/restricted Translation-en
Hit http://ubuntutym.u-toyama.ac.jp precise-backports/universe Translation-en
Fetched 20. MB in 17秒 (1,155 kB/s)
kazu634@fluent-master:/etc/apt$ sudo aptitude install td-agent
The following NEW packages will be installed:
td-agent
 packages upgraded, 1 newly installed,  to remove and  not upgraded.
Need to get  B/12.7 MB of archives. After unpacking 46.5 MB will be used.
Selecting previously unselected package td-agent.
(データベースを読み込んでいます ... 現在 53744 個のファイルとディレクトリがインストールされています。)
(.../td-agent_1.1.7-1_amd64.deb から) td-agent を展開しています...
ureadahead のトリガを処理しています ...
td-agent (1.1.7-1) を設定しています ...
Installing default conffile /etc/td-agent/td-agent.conf ...
libc-bin のトリガを処理しています ...
ldconfig deferred processing now taking place

設定方法は次回に。