Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

utilize DNS/NTLM/DHCP/etc. when populating NetBox inventory via passively-gathered network traffic metadata #415

Closed
mmguero opened this issue Feb 15, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request logstash Relating to Malcolm's use of Logstash netbox Related to Malcolm's use of NetBox zeek Relating to Malcolm's use of Zeek
Milestone

Comments

@mmguero
Copy link
Collaborator

mmguero commented Feb 15, 2024

When populating NetBox inventory via passively-gathered network traffic metadata, right now we are only using MAC address and IP address.

We ought to look at using DNS response, NTLM logs, etc., anything that can give us a hostname from an IP address, to improve this process.

@mmguero mmguero added enhancement New feature or request logstash Relating to Malcolm's use of Logstash zeek Relating to Malcolm's use of Zeek netbox Related to Malcolm's use of NetBox labels Feb 15, 2024
@mmguero mmguero added this to the v24.04.0 milestone Feb 15, 2024
@mmguero mmguero self-assigned this Mar 6, 2024
@mmguero mmguero assigned mmguero and unassigned mmguero Mar 14, 2024
@mmguero mmguero added the CISA label Mar 21, 2024
@mmguero mmguero modified the milestones: v24.03.1, v24.04.0 Mar 26, 2024
@mmguero mmguero assigned mmguero and unassigned mmguero Mar 27, 2024
@mmguero mmguero self-assigned this Apr 4, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Apr 18, 2024

Here's the field breakdown (work in progress).

As much as possible, we want to use our normalized ECS fields for this.

  • DNS
    • dns.question.name (keyword) and dns.resolved_ip (array of IP)
  • NTLM
    • zeek.ntlm.hostname and source.ip
    • zeek.ntlm.server_nb_computer_name or zeek.ntlm.server_dns_computer_name and destination.ip
  • krb5.log ???
  • dhcp.log
    • zeek.dhcp.client_fqdn or zeek.dhcp.host_name (do we care about zeek.dhcp.domain?) to zeek.dhcp.assigned_addr or zeek.dhcp.client_addr or zeek.dhcp.requested_addr

mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 18, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 18, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 18, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 22, 2024
…s in netbox (work in progress, probably broken)
@mmguero mmguero changed the title utilize DNS/NetBIOS/NTLM/etc. when populating NetBox inventory via passively-gathered network traffic metadata utilize DNS/NTLM/DHCP/etc. when populating NetBox inventory via passively-gathered network traffic metadata Apr 23, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 23, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 25, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 26, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Apr 26, 2024

I feel pretty good about this. It's not perfect: making some assumptions from DNS, NTLM, DHCP, etc. logs isn't a perfect science but it seems to be pretty accurate and the error handling works (netbox doesn't allow you to defeat constraints for name uniqueness, I've got some checks in the code to make sure we don't try to do something stupid). I'm going to close this but may reopen if I find any major issues between now and the release next week, and will open other issues if we come up with improvements or fixes after that.

@mmguero mmguero closed this as completed Apr 26, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 26, 2024
This was referenced Apr 30, 2024
mmguero added a commit that referenced this issue Apr 30, 2024
Malcolm v24.04.0

* Features and enhancements
    - Zeek-extracted files scanned and preserved on a [Hedgehog Linux](https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#HedgehogZeekFileExtraction) sensor can now be accessed via [the extracted files download user interface](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtractionUI) (#331).
    - Improvements to creation of index templates, dashboards, and other saved objects on startup (#208) to ensure that saved objects get created correctly upon upgrade (see [this comment](#208 (comment)) for more details on this feature).
    - [Populating the NetBox inventory via passively-gathered network traffic metadata](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassive) now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (#415). Autopopulated devices now have their *status* field set to `Active` rather than `Stage`, and uses *tags* instead to indicated that they were created through autopopulation.
    - Users can now specify pruning thresholds for [carved files](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtraction) so that old files are deleted in order to avoid filling available storage (#453). See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about this and similar settings.
    - Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (#455).
    - The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with [category fields for high cardinality](https://opensearch.org/docs/latest/observing-your-data/ad/index/#optional-set-category-fields-for-high-cardinality) to allow for better breakdown of contributing values to anomalies discovered (#464).
    - Include [JA4+ plugin in Arkime](https://arkime.com/settings#ja4plus). See #419 for status on upcoming full JA4+ support in Malcolm.
    - Hedgehog Linux sensors can now [periodically refresh](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/hedgehog-iso/interface/sensor_ctl/control_vars.conf#L75) their [Zeek inteligence files](https://idaholab.github.io/Malcolm/docs/hedgehog-config-zeek-intel.html#HedgehogZeekIntel).
        + **NOTE**: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line `export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel` to `/opt/sensor/sensor_ctl/control_vars.conf` and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
    - Assorted documentation improvements.
* Component version updates
    - Arkime to [v5.1.2](https://github.com/arkime/arkime/blob/bcd9d7e68be8e4a52a17c35211c5d5a7fdcc1a1c/CHANGELOG#L36-L41)
    - OpenSearch and OpenSearch Dashboards to [v2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md)
    - Beats to [v8.13.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.13.2.html)
    - Logstash to [v8.13.2](https://www.elastic.co/guide/en/logstash/current/logstash-8-13-2.html)
    - gunicorn to v22.0.0 to address [CVE-2024-1135](GHSA-w3h3-4rj7-4ph4).
    - elasticsearch-dsl to [v8.13.0](https://github.com/elastic/elasticsearch-dsl-py/releases/tag/v8.13.0)
    - elasticsearch-py to [v8.13.0](https://github.com/elastic/elasticsearch-py/releases/tag/v8.13.0)
    - idna to v3.7 to address [CVE-2024-3651](GHSA-jjg7-2v4v-x38h)
    - Fluent Bit to [v3.0.3](https://fluentbit.io/announcements/v3.0.3/)
* Bug fixes
    - The documentation for [Windows host system configuration](https://idaholab.github.io/Malcolm/docs/host-config-windows.html#HostSystemConfigWindows) was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (#421).
    - An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (#426).
    - The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of `zeek-live` containers (#456). See [this comment](#456 (comment)) for more details.
    - Removed the version top-level element from `docker-compose.yml` files as it is [now obsolete](https://docs.docker.com/compose/compose-file/04-version-and-name/) and caused a warning message that sometimes was not handled correctly.
    - Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
    - Restart live Zeek instances with `zeekctl deploy` instead of `zeekctl restart`.
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.04.0/config))
    - `ARKIME_QUERY_ALL_INDICES` in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/arkime.env.example#L9-L11) can be set to control the [`queryAllIndices` setting](https://arkime.com/settings#queryAllIndices) in Arkime's `config.ini`.
    - `DASHBOARDS_PREFIX` in [`dashboards-helper.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/dashboards-helper.env.example#L3C1-L4C19) has been added for #455 (see above in **Features and Enhancements**).
    - `LOGSTASH_NETBOX_ENRICHMENT_DATASETS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L13) has been changed to include `zeek.dhcp`, `zeek.dns`, and `zeek.ntlm` to support #415 (see above in **Features and Enhancements**).
    - `LOGSTASH_ZEEK_IGNORED_LOGS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L15) has been changed to remove `capture_loss` and `stats` so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
    - `ZEEK_CRON` has been removed from [`zeek-live.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-live.env.example) and `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` was removed from [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example) and moved to the "offline" version of the container in [`zeek-offline.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-offline.env.example#L17-L19) for #456.
    - `EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE`, `EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT`, and `EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS` were added to [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example#L32-L37) for #453. See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about these and similar settings.
mmguero added a commit to cisagov/Malcolm that referenced this issue Apr 30, 2024
Malcolm v24.04.0

* Features and enhancements
    - Zeek-extracted files scanned and preserved on a [Hedgehog Linux](https://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#HedgehogZeekFileExtraction) sensor can now be accessed via [the extracted files download user interface](https://cisagov.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtractionUI) (idaholab#331).
    - Improvements to creation of index templates, dashboards, and other saved objects on startup (idaholab#208) to ensure that saved objects get created correctly upon upgrade (see [this comment](idaholab#208 (comment)) for more details on this feature).
    - [Populating the NetBox inventory via passively-gathered network traffic metadata](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassive) now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (idaholab#415). Autopopulated devices now have their *status* field set to `Active` rather than `Stage`, and uses *tags* instead to indicated that they were created through autopopulation.
    - Users can now specify pruning thresholds for [carved files](https://cisagov.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtraction) so that old files are deleted in order to avoid filling available storage (idaholab#453). See a new section of documentation on [Managing disk usage](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about this and similar settings.
    - Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (idaholab#455).
    - The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with [category fields for high cardinality](https://opensearch.org/docs/latest/observing-your-data/ad/index/#optional-set-category-fields-for-high-cardinality) to allow for better breakdown of contributing values to anomalies discovered (idaholab#464).
    - Include [JA4+ plugin in Arkime](https://arkime.com/settings#ja4plus). See idaholab#419 for status on upcoming full JA4+ support in Malcolm.
    - Hedgehog Linux sensors can now [periodically refresh](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/hedgehog-iso/interface/sensor_ctl/control_vars.conf#L75) their [Zeek inteligence files](https://idaholab.github.io/Malcolm/docs/hedgehog-config-zeek-intel.html#HedgehogZeekIntel). **NOTE**: due to an oversight, a necessary variable is missing in this release that is required for this to work. Appending the line `export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel` to `/opt/sensor/sensor_ctl/control_vars.conf` will correct this. This will be corrected in the next Malcolm release.
    - Assorted documentation improvements.
* Component version updates
    - Arkime to [v5.1.2](https://github.com/arkime/arkime/blob/bcd9d7e68be8e4a52a17c35211c5d5a7fdcc1a1c/CHANGELOG#L36-L41)
    - OpenSearch and OpenSearch Dashboards to [v2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md)
    - Beats to [v8.13.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.13.2.html)
    - Logstash to [v8.13.2](https://www.elastic.co/guide/en/logstash/current/logstash-8-13-2.html)
    - gunicorn to v22.0.0 to address [CVE-2024-1135](GHSA-w3h3-4rj7-4ph4).
    - elasticsearch-dsl to [v8.13.0](https://github.com/elastic/elasticsearch-dsl-py/releases/tag/v8.13.0)
    - elasticsearch-py to [v8.13.0](https://github.com/elastic/elasticsearch-py/releases/tag/v8.13.0)
    - idna to v3.7 to address [CVE-2024-3651](GHSA-jjg7-2v4v-x38h)
    - Fluent Bit to [v3.0.3](https://fluentbit.io/announcements/v3.0.3/)
* Bug fixes
    - The documentation for [Windows host system configuration](https://cisagov.github.io/Malcolm/docs/host-config-windows.html#HostSystemConfigWindows) was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (idaholab#421).
    - An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (idaholab#426).
    - The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of `zeek-live` containers (idaholab#456). See [this comment](idaholab#456 (comment)) for more details.
    - Removed the version top-level element from `docker-compose.yml` files as it is [now obsolete](https://docs.docker.com/compose/compose-file/04-version-and-name/) and caused a warning message that sometimes was not handled correctly.
    - Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
    - Restart live Zeek instances with `zeekctl deploy` instead of `zeekctl restart`.
* Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.04.0/config))
    - `ARKIME_QUERY_ALL_INDICES` in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/arkime.env.example#L9-L11) can be set to control the [`queryAllIndices` setting](https://arkime.com/settings#queryAllIndices) in Arkime's `config.ini`.
    - `DASHBOARDS_PREFIX` in [`dashboards-helper.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/dashboards-helper.env.example#L3C1-L4C19) has been added for idaholab#455 (see above in **Features and Enhancements**).
    - `LOGSTASH_NETBOX_ENRICHMENT_DATASETS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L13) has been changed to include `zeek.dhcp`, `zeek.dns`, and `zeek.ntlm` to support idaholab#415 (see above in **Features and Enhancements**).
    - `LOGSTASH_ZEEK_IGNORED_LOGS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L15) has been changed to remove `capture_loss` and `stats` so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
    - `ZEEK_CRON` has been removed from [`zeek-live.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-live.env.example) and `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` was removed from [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example) and moved to the "offline" version of the container in [`zeek-offline.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-offline.env.example#L17-L19) for idaholab#456.
    - `EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE`, `EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT`, and `EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS` were added to [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example#L32-L37) for idaholab#453. See a new section of documentation on [Managing disk usage](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about these and similar settings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request logstash Relating to Malcolm's use of Logstash netbox Related to Malcolm's use of NetBox zeek Relating to Malcolm's use of Zeek
Projects
Status: Released
Development

No branches or pull requests

1 participant