New component: GeoIP processor

[michalpristas]

The purpose and use-cases of the new component

The GeoIP processors adds information about the geographical location of IP addresses, based on data from the MaxMind GeoLite2 databases.

Example configuration for the component

Location of this database should be configurable and stored on disk.
For the ability to fetch we need a an account ID, license key and endpoint where to fetch it from.

Based on above we can have configuration like so:

• database: path to a local file
• account_id: account ID used to fetch
• license_key: license key used to fetch
• permalink: permalink generated used to fetch
• field: field where ip is loaded from. default should probably be source.address (handle non ip e.g uri or .sock gracefully)
• target_field: namespaced target e.g source.geo
• auto_update: true

one of these needs to be used, if license key is specified and path is omitted we should try to fetch database and store it locally.
if path is specified, file does not exists and we have a license key we should fetch and place it at the specified location.
it path is specified, no license key we use this file

in case auto_update is enabled we should fetch database (in case account id, license and permalink are specified)
other way we either won't enrich or enrich with _geoip_expired_database

all other cases are erroneous.

mapping of service fields to semconv. As these fields are not part of SemConv yet i'm sticking to ECS definition that was donated. These would be subkeys to a target defined by target_field

Service field	SemConv/ECS	Example
ip	address	12.34.56.78
city_name	geo.city_name	Seattle
country_name	geo.country_name	United States
continent_code	geo.continent_code	NA
continent_name	geo.continent_name	North America
country_code2	geo.country_iso_code	US
postal_code	geo.postal_code	98106
region_name	geo.region_name	Washington
region_code	geo.region_code	WA
region_iso_code*	geo.region_iso_code	US-WA
timezone	geo.timezone	America/Los_Angeles
location*	geo.location	{"lat": 47.6062, "lon": -122.3321}
latitude	geo.location.lat	47.6062
longitude	geo.location.lon	-122.3321
domain	domain	example.com
asn	as.number	98765
as_org	as.organization.name	OpenTelemetry - CNCF
isp	mmdb.isp	InterLink Supra LLC
dma_code	mmdb.dma_code	819
organization	mmdb.organization	OpenTelemetry - CNCF

Telemetry data types supported

Logs, metrics, traces.

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@michalpristas

Sponsor (optional)

@andrzej-stencel

Additional context

This proposal is based on this Elasticsearch processor: https://www.elastic.co/guide/en/elasticsearch/reference/current/geoip-processor.html

Other alternative i considered is to skip fetching part and use only local database. this would require user to handle updating expired databases.

[felixbarny]

Similar to Elasticsearch's geoip processor, we should automatically fetch a default geo ip database (without having to specify an account id and license key) if there's at least one pipeline with a geoip processor.

[felixbarny]

Fetching from a default URL might be a little more complex so we should look at that as a separate enhancement.

Elasticsearch's geoip processor fetches the geo ip database from https://geoip.elastic.co/v1/database. However, I'm not sure if we could just re-use the same URL, it could have some licensing implications. Longer-term, we can discuss having a similar mirror hosted by cncf. Another thing to note is that fetching the database requires to accept MaxMind's terms and conditions, which Elasticsearch's geoip processor does by default. But we also need to check if it's safe to do that via the OSS OTel collector distro.

@dakrone could you chime in on the licensing aspects of the MaxMind database, our mirror, and how we could enable a default download of the geo ip database in the OTel collector?

[crobert-1]

Hello @michalpristas, thanks for filing. To make sure I understand, is this intended to be a new processor, or an enhancement to one that already exists?

[michalpristas]

a new one, sorry i haven't find a way how to make it clear

[crobert-1]

a new one, sorry i haven't find a way how to make it clear

Not a problem, I've updated the labels on the issue accordingly 👍 (I'm not sure what option you used to create this issue, but creating one using the New component proposal will give the proper labels and information to fill out, for future reference)

[andrzej-stencel]

I can sponsor this component, but let's discuss in an upcoming SIG whether the community feels we want to add it to Contrib.

Some random thoughts:

• This processor is designed to work with a database from a specific IP geolocation provider (MaxMind). I wouldn't go as far as to say it's a "vendor" component, but definitely opinionated. You could imagine users wanting to do IP geolocation via other providers.
• The geoip database fetching part of the processor feels to me a bit more like a candidate for an extension. You probably want just one database and one update/fetching mechanism even if you configure more than one processor in your config.
• The telemetry conversion part of this processor feels like it could belong to the OTTL/Transform processor, which already has syntax to specify resource/record attributes, extraction and other transformations that might be needed before and/or after the geoip lookup.
  • Is it possible, or has there been a discussion on extensibility of OTTL via external components (like extensions, or maybe "ottl providers")? The "GeoIP" function could either be a part of core OTTL, or maybe "provided" to OTTL via an extension point.

[djaglowski]

If I'm not mistaken, this would be the first component which depends directly on a dependency with a license of its own. Ideally we would define an extension interface that could be implemented by other geoip providers, so that use of the component does not strictly require the MaxMind database & license.

[michalpristas]

Agreed this should also work woth open source alternatives e.g https://github.com/geoacumen/geoacumen-country

[cwegener]

Is it possible, or has there been a discussion on extensibility of OTTL via external components (like extensions, or maybe "ottl providers")? The "GeoIP" function could either be a part of core OTTL, or maybe "provided" to OTTL via an extension point.

This sounds very compelling. Could such an "OTTL extension point" open up the simple and flexible implementation of custom "enrichment" functions? (I think the GeoIP functionality in this issue here is very much an example of such enrichment)

[david-drake]

Another provider that provides excellent geolocation data that would be great for support: https://ipinfo.io/products/free-ip-database

There are enterprise subscriptions available as well, similar to MaxMind.

[michalpristas]

closing the issue. basic functionality was covered by #33451 and #33319