events are not sent to newrelic for perfmon counters with no instances

[ayounas]

Hello

We are monitoring few of the windows hypervisor servers using the nri-perfmon and discovered that metrics/event from few of the configured counters are not sent to newrelic.
On further troubleshooting it seems if there are no instances for the counter then nri-perfmon wont parse the metric

e.g config below does not collect and send counters for Hyper-V Hypervisor

     {
        "provider":"PerfCounter",
        "category":"Hyper-V Hypervisor",
        "eventname":"Perfmon_HyperVHypervisor",
        "counters":[
          {
            "counter":"Logical Processors"
          },
          {
            "counter":"Partitions"
          },
          {
            "counter":"Virtual Processors"
          }
        ]
      },

The output below shows Name for the counter is blank (no instances) and i guess probably that's why the counter is not parsed

PS C:\temp> Get-CimInstance Win32_PerfFormattedData_HvStats_HyperVHypervisor


Caption                 :
Description             :
Name                    :
Frequency_Object        :
Frequency_PerfTime      :
Frequency_Sys100NS      :
Timestamp_Object        :
Timestamp_PerfTime      :
Timestamp_Sys100NS      :
HypervisorStartupCost   : 20599148
LogicalProcessors       : 48
ModernStandbyEntries    : 0
MonitoredNotifications  : 6
Partitions              : 4
PlatformIdleTransitions : 0
TotalPages              : 1619117
VirtualProcessors       : 60
PSComputerName          :

i tried using
"instance":"*",
"instance":"",
Still no luck

Similarly no output from

      {
        "provider":"PerfCounter",
        "category":"Hyper-V Virtual Machine Health Summary",
        "eventname":"Perfmon_HyperVVirtualMachineHealthSummary",
        "counters":[
          {
            "counter":"Health Critical"
          },
          {
            "counter":"Health Ok"
          }
        ]
      }

PS C:\temp> Get-CimInstance Win32_PerfFormattedData_VmmsVirtualMachineStats_HyperVVirtualMachineHealthSummary


Caption            :
Description        :
Name               :
Frequency_Object   :
Frequency_PerfTime :
Frequency_Sys100NS :
Timestamp_Object   :
Timestamp_PerfTime :
Timestamp_Sys100NS :
HealthCritical     : 0
HealthOk           : 3
PSComputerName     :

All the counters with something in Name: or with instances work just fine

Seems similar to #18 and #17

Verbose Logs

C:\Program Files\New Relic\newrelic-infra\custom-integrations\nri-perfmon>nri-perfmon.exe -r -v  -c c:\temp\nrc.json
Thread-1 : nri-perfmon version 0.6.1.0 starting with options:
{
  "ConfigFile": "c:\\temp\\nrc.json",
  "PollingInterval": 10000,
  "RunOnce": true,
  "MachineName": "****",
  "UserName": "***",
  "DomainName": "**",
  "Password": "",
  "Verbose": true
}
Thread-1 : nri-perfmon counters:
[
  {
    "provider": "PerfCounter",
    "category": "Hyper-V Virtual Machine Health Summary",
    "instance": null,
    "counters": [
      {
        "counter": "Health Critical",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Health Ok",
        "attrname": "using_counter_name",
        "parser": ""
      }
    ],
    "query": null,
    "eventname": "Perfmon_HyperVVirtualMachineHealthSummary",
    "querytype": "wmi_query",
    "querynamespace": "root\\cimv2"
  },
  {
    "provider": "PerfCounter",
    "category": "Hyper-V Hypervisor",
    "instance": null,
    "counters": [
      {
        "counter": "Logical Processors",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Partitions",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Virtual Processors",
        "attrname": "using_counter_name",
        "parser": ""
      }
    ],
    "query": null,
    "eventname": "Perfmon_HyperVHypervisor",
    "querytype": "wmi_query",
    "querynamespace": "root\\cimv2"
  },
  {
    "provider": "PerfCounter",
    "category": "System",
    "instance": null,
    "counters": [
      {
        "counter": "Context Switches/sec",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Processes",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Processor Queue Length",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "System Calls/sec",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Threads",
        "attrname": "using_counter_name",
        "parser": ""
      }
    ],
    "query": null,
    "eventname": "Perfmon_System",
    "querytype": "wmi_query",
    "querynamespace": "root\\cimv2"
  }
]
Thread-1 : Running once and exiting.
Thread-1 : Polling time: 00:00:00.0156237

C:\Program Files\New Relic\newrelic-infra\custom-integrations\nri-perfmon>

[sschwartzman]

Hi @ayounas sorry for the delay. Thanks for all of that detail, it made it easy for me to re-create, as I have a Hyper-v rig to test it with. However, I actually got back the expected data, without making any changes to your json config stanzas.

Here's the results (excerpted from -v output):

Thread-1 : Metric output:
{
  "name": "HYPERV-HV-1",
  "protocol_version": "1",
  "integration_version": "0.1.0",
  "events": [],
  "inventory": {},
  "metrics": [
    {
      "event_type": "Hyper_V_Virtual_Machine_Health_Summary",
      "name": "",
      "HealthCritical": 0.0,
      "HealthOk": 2.0
    },
    {
      "event_type": "Hyper_V_Hypervisor",
      "name": "",
      "LogicalProcessors": 72.0,
      "Partitions": 1.0,
      "VirtualProcessors": 72.0
    }
  ]
}

Here is the config.json I used:

{
  "counterlist": [
    {
        "provider":"PerfCounter",
        "category":"Hyper-V Hypervisor",
        "eventname":"Perfmon_HyperVHypervisor",
        "counters":[
          {
            "counter":"Logical Processors"
          },
          {
            "counter":"Partitions"
          },
          {
            "counter":"Virtual Processors"
          }
        ]
    },
	{
        "provider":"PerfCounter",
        "category":"Hyper-V Virtual Machine Health Summary",
        "eventname":"Perfmon_HyperVVirtualMachineHealthSummary",
        "counters":[
          {
            "counter":"Health Critical"
          },
          {
            "counter":"Health Ok"
          }
        ]
	}
  ]
}

It's worth noting that, when you're using the PerfCounter functionality, "eventname" is ignored, that attribute is only used for the WMI queries functionality. It doesn't cause it to fail though, it just ignores that attribute and names events based on Category name, like you see in the results above.

Is it possible that you're running an older version of nri-perfmon? How about what OS and arch are you running on?

[ayounas]

Hello @sschwartzman thanks for looking into it,
I checked the version and we are using the latest 0.6.1.0
I downloaded fresh copy from github and reinstalled for just in case but still same problem.

Versions:

Thread-1 : nri-perfmon version 0.6.1.0 starting with options:
Microsoft Windows [Version 10.0.14393]
Windows Server 2016
Version 1607 (OS Build 14393.4530)

Output using the config.json you pasted.

C:\Program Files\New Relic\newrelic-infra\custom-integrations\nri-perfmon>nri-perfmon.exe -r -v  -c c:\temp\config.json
Thread-1 : nri-perfmon version 0.6.1.0 starting with options:
{
  "ConfigFile": "c:\\temp\\config.json",
  "PollingInterval": 10000,
  "RunOnce": true,
  "MachineName": "redacted",
  "UserName": "redacted",
  "DomainName": "redacted",
  "Password": "",
  "Verbose": true
}
Thread-1 : nri-perfmon counters:
[
  {
    "provider": "PerfCounter",
    "category": "Hyper-V Hypervisor",
    "instance": null,
    "counters": [
      {
        "counter": "Logical Processors",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Partitions",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Virtual Processors",
        "attrname": "using_counter_name",
        "parser": ""
      }
    ],
    "query": null,
    "eventname": "Perfmon_HyperVHypervisor",
    "querytype": "wmi_query",
    "querynamespace": "root\\cimv2"
  },
  {
    "provider": "PerfCounter",
    "category": "Hyper-V Virtual Machine Health Summary",
    "instance": null,
    "counters": [
      {
        "counter": "Health Critical",
        "attrname": "using_counter_name",
        "parser": ""
      },
      {
        "counter": "Health Ok",
        "attrname": "using_counter_name",
        "parser": ""
      }
    ],
    "query": null,
    "eventname": "Perfmon_HyperVVirtualMachineHealthSummary",
    "querytype": "wmi_query",
    "querynamespace": "root\\cimv2"
  }
]
Thread-1 : Running once and exiting.
Thread-1 : Polling time: 00:00:00.0156255

[sschwartzman]

So what can you tell me about your env? OS, arch, user running nri-perfmon.exe, etc?

Also, can you try running without the "run-once" flag? Maybe that's the issue, it's a relatively new addition to nri-perfmon so I want to ensure it's not that.

Update: I tried with "-r" flag and got the same results as without it.

[ayounas]

Hello
These are physical hosts running windows server 2016 (Version 1607 (OS Build 14393.4530)
The windows 2016 servers are running Microsoft Hyper-V
They are all domain joined.
Nri Perfmon version we are running is : nri-perfmon version 0.6.1.0
Newrelic agent -: 1.18
I was running nri-perfmon using my own username which is a domain account and have full admin rights
The cmd prompt I used to run the tests was launched as Administrator.

[ayounas]

Hello @sschwartzman
Spent some more time today testing on different operating systems (server 2019, server 2022, server 2016) using latest 0.6.1 release but no luck, then i thought i will try version 0.5.1 because that was the first version with the fix to get metrics for counters without any instance.
And with version 0.5.1 i was able to get the metrics for the counters without instance.
Tried all the versions above 0.5.1 and same issue i.e. no metrics for counters without instance

Thread-1 : Metric output:
{
  "name": "<redacted>",
  "protocol_version": "1",
  "integration_version": "0.1.0",
  "events": [],
  "inventory": {},
  "metrics": [
    {
      "event_type": "Hyper_V_Virtual_Machine_Health_Summary",
      "name": "",
      "HealthOk": 3.0,
      "HealthCritical": 0.0
    },
    {
      "event_type": "Hyper_V_Hypervisor",
      "name": "",
      "LogicalProcessors": 48.0,
      "Partitions": 4.0,
      "TotalPages": 1619199.0,
      "VirtualProcessors": 60.0,
      "MonitoredNotifications": 6.0,
      "ModernStandbyEntries": 0.0,
      "PlatformIdleTransitions": 0.0,
      "HypervisorStartupCost": 20520132.0
    }
  ]
}
Thread-1 : Polling time: 00:00:01.4794740
Thread-1 : Sleeping for: 00:00:08.5205260
PS C:\temp\nri-perfmon0.5.1>

Seems all the version above 0.5.1 regressed the fix introduced in the 0.5.1 .
Also with 0.5.1 metrics are not filtered and output shows every metric for the perfcounter instead of named metrics. The output yourself pasted shows only the filtered metrics.
Will you please be able to have a look and see if the issue can be fixed? or let me know if you need more information from me

[Eroriko]

I'm just wondering if this is planned to get fixed, a long time has passed since this was reported. I tried ayounas findings and installed version 0.5.1 which solved the problem. But i would rather be in a more up to date version.