perl – MaxMind Writer创建自定义数据库,用于ELK Stack中的geoip

我正在尝试使用MaxMind的perl Writer为内部网络创建一个自定义数据库,到目前为止工作正常.然后我尝试使用logstash中的geoip过滤器插件来对这些IP范围进行地理定位.我可以使用使用MaxMind Reader的perl脚本验证创建的数据库是否正确包含我的数据.

#!/usr/bin/env perl

use strict;
use warnings;
use feature qw( say );
use local::lib 'local';

use MaxMind::DB::Writer::Tree;
use Net::Works::Network;

my $filename = 'test.mmdb';

# Your top level data structure will always be a map (hash).  The MMDB format
# is strongly typed.  Describe your data types here.
# See https://metacpan.org/pod/MaxMind::DB::Writer::Tree#DATA-TYPES

my %types = (
    geoname_id => 'uint32',
    locale_code => 'utf8_string',
    continent_code => 'utf8_string',
    continent_name => 'utf8_string',
    country_iso_code => 'utf8_string',
    country_name => 'utf8_string',
    subdivision_1_iso_code => 'utf8_string',
    subdivision_1_name => 'utf8_string',
    city_name => 'utf8_string',
    metro_code => 'uint32',
    time_zone => 'utf8_string',
    postal_code => 'utf8_string',
    latitude => 'double',
    longitude => 'double',
);

my $tree = MaxMind::DB::Writer::Tree->new(
    database_type => 'GeoIP2-City',
    description => { en => 'IP-Ranges', de => q{IP-Bereiche}, },
    ip_version => 4,
    map_key_type_callback => sub { $types{ $_[0] } },
    record_size => 24,
    remove_reserved_networks => 0,
);

my %address_for_employee = (
    'xx.xx.xx.xx/24' => {
        geoname_id => 3164699,
        locale_code => 'de',
        continent_code => 'EU',
        continent_name => 'Europa',
        country_iso_code => 'IT',
        country_name => 'Italien',
        subdivision_1_iso_code => '09',
        city_name => 'Varese',
        time_zone => 'Europe/Rome',
        postal_code => '21100',
        latitude => 45.7908,
        longitude => 8.8515,
    },    
    'xx.xx.xx.xx/24' => {
        geoname_id => 3164699,
        locale_code => 'de',
        continent_code => 'EU',
        continent_name => 'Europa',
        country_iso_code => 'IT',
        country_name => 'Italien',
        subdivision_1_iso_code => '09',
        city_name => 'Varese',
        time_zone => 'Europe/Rome',
        postal_code => '21100',
        latitude => 45.7908,
        longitude => 8.8515,
    },   
);

for my $address ( keys %address_for_employee ) {

    # Create one network and insert it into our database
    my $network = Net::Works::Network->new_from_string( string => $address );

    $tree->insert_network( $network, $address_for_employee{$address} );
        say "Created Subnet from ___ $network ===> $address ___";
}

# Write the database to disk.
open my $fh, '>:raw', $filename;
$tree->write_tree( $fh );
close $fh;

say "$filename has now been created";

Logstash过滤器配置:

filter {
geoip {
  source => "src_ip"
  target => "src_ip_geo"
  database => "test.mmdb"
  add_field => [ "[src_ip_geo][coordinates]", "%{[src_ip_geo][longitude]}" ]
  add_field => [ "[src_ip_geo][coordinates]", "%{[src_ip_geo][latitude]}"  ]
}
mutate {
  convert => [ "[geoip][coordinates]", "float"]
}
}

使用旧的MaxMind数据库GeoLiteCity.dat将其与工作流进行比较我尝试根据我在数据库中定义的IP地址将一些字段添加到输出中.

但是,日志会正确推送到elasticsearch,但没有其他字段但添加了标记:_geoip_lookup_failure

有没有办法以某种方式调试此过滤器,因为我的logstash日志不提供任何提示.我也输出了

output { stdout{ codec => "json" } }

但是没有错误或警告,只有普通的json然后被推送到elasticsearch.数据库是否以正确的方式创建?

最佳答案
您需要参考官方的GeoLite2-City.mmdb结构(例如,使用reader)

perl代码示例:

my %types = (
 city => 'map',
 geoname_id => 'uint32',
 names => 'map',
 en => 'utf8_string',
 iso_code => 'utf8_string',
 country => 'map',
 registered_country => 'map',
 location => 'map',
 latitude => 'double',
 accuracy_radius => 'uint32',
 time_zone => 'utf8_string',
 longitude => 'double',
 continent => 'map',
);

my $tree = MaxMind::DB::Writer::Tree->new(
ip_version            => 4,
record_size           => 28,
database_type         => 'GeoIP2-City',
languages             => ['en'],
description           => { en => 'GeoLite2 City database' },
map_key_type_callback => sub { $types{ $_[0] } },
);

$tree->insert_network(
 '183.136.190.0/24' => {
     city => {
                 geoname_id => 1808926,
                 names => {
                              'en' => 'test1',
                          },
             },
     country => {
                     geoname_id => 1814991,
                     iso_code => 'CN',
                     names => {
                                  'en' => 'test2',
                              },
                },
     registered_country => {
                               geoname_id => 1814991,
                               iso_code => 'CN',
                               names => {
                                            'en' => 'test3',
                                        },
                           },
#         subdivisions => (
#                           {
#                             geoname_id => 1784764,
#                             iso_code => 'ZJ',
#                             names => {
#                                          en => 'Zhejiang',
#                                      },
#                           },
#                         ),
     location => {
                     latitude => 30.2936,
                     accuracy_radius => 50,
                     time_zone => 'Asia/Shanghai',
                     longitude => 120.1614,
                 },
     continent => {
                      geoname_id => 6255147,
                      iso_code => 'AS',
                      names => {
                                   en => 'Asia',
                               },
                  },
      },
);

logstash过滤器配置:

    geoip {
      source => "remote_addr"
      database => "/temp/csv2mmdb/IPB.mmdb"
      fields => ["city_name", "country_name"]
      target => "geoip"
    }

输出结果:

"remote_addr" => "183.136.190.40",
"geoip" => {
       "city_name" => "test1",
       "country_name" => "test2"
},
"X-Forwarded-Proto" => "-",

转载注明原文:perl – MaxMind Writer创建自定义数据库,用于ELK Stack中的geoip - 代码日志