[Cisco ise] Tag device-sensor fragments instead of erroring per line #20528
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,7 @@ | ||
| CISE_Profiler 0000034521 2 1 cisco-av-pair=cdp-tlv=cdpCacheAddress=192.0.2.10, cisco-av-pair=lldp-tlv=lldpSystemDescription=Cisco IOS Software C3700 Software, cisco-av-pair=cdp-tlv=cdpCacheDeviceId=SAMPLE-AP-0001, | ||
| Technical Support: http://www.cisco.com/techsupport\ | ||
| Copyright (c) 1986-2024 by Cisco Systems, Inc.\ | ||
| Compiled Mon 01-Jan-24 00:00 by sample, cisco-av-pair=cdp-tlv=cdpCacheDeviceId=SAMPLE-AP-0001.8, cisco-av-pair=audit-session-id=C000020A00001111AAAA2222, cisco-av-pair=method=mab, AcsSessionID=SAMPLE-ISE-01/000000000/1111111, Step=11004, NetworkDeviceGroups=Location#All Locations#SAMPLE, CPMSessionID=C000020A00001111AAAA2222, StepLatency=1=0;2=0;3=0;4=1;5=0, TotalAuthenLatency=2, Operation=Operation#Operation#CLOSED, | ||
| #1 Mon Jan 1 00:00:00 UTC 2024\ | ||
| mips, cisco-av-pair=lldp-tlv=lldpSystemName=SAMPLE-AP-0002, cisco-av-pair=lldp-tlv=lldpPortDescription=Ethernet Interface, Operation=Operation#Operation#CLOSED, | ||
| , cisco-av-pair=cdp-tlv=cdpCacheDeviceId=SAMPLE-AP-0002, cisco-av-pair=lldp-tlv=End-of-LLDPDU=\ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,13 +19,40 @@ processors: | |
| - '^(?:<%{NUMBER:log.syslog.priority:long}>)?%{SYSLOGTIMESTAMP:_tmp.timestamp} %{DATA:host.hostname} %{DATA:cisco_ise.log.category.name} %{GREEDYDATA:_tmp.message}$' | ||
| - '^%{TIMEONLYSTAMP_ISO8601:_tmp.timestamp}%{ISO8601_TIMEZONE:_tmp.timezone} %{DATA:host.hostname} %{DATA:cisco_ise.log.category.name} %{GREEDYDATA:_tmp.message}$' | ||
| - '^%{DATA:cisco_ise.log.category.name} %{DATA:cisco_ise.log.message.id} %{NONNEGINT:cisco_ise.log.segment.total:long} %{NONNEGINT:cisco_ise.log.segment.number:long} %{GREEDYDATA:_tmp.message}$' | ||
| - '^Technical Support: %{GREEDYDATA}$' | ||
| - '^Copyright %{GREEDYDATA}$' | ||
| - '^Compiled %{DATA} by %{DATA}, %{GREEDYDATA:_tmp.message}$' | ||
|
vinit-chauhan marked this conversation as resolved.
|
||
| - '^#%{NONNEGINT} %{GREEDYDATA}$' | ||
|
Contributor
There was a problem hiding this comment. Hey @taylor-swanson - Do you remember how we've these cases where events are truncated?
Contributor
There was a problem hiding this comment. If messages are truncated, that's usually a sign of a problem upstream (namely, TCP framing issues). In this case, I guess the message contains newlines and it's getting split by the tcp input? Ugh. The proper way to solve this is to have the Cisco device use octet framing (which completely bypasses the need for a character-based delimiter). This feels like the wrong solution. |
||
| - '^%{DATA}, (?<_tmp.message>cisco-av-pair=.*)$' | ||
| pattern_definitions: | ||
|
vinit-chauhan marked this conversation as resolved.
|
||
| TIMEONLYSTAMP_ISO8601: '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?' | ||
| on_failure: | ||
| - append: | ||
| tag: append_error_message_62c1fc01 | ||
| field: error.message | ||
| value: '{{{ _ingest.on_failure_processor_tag }}}: {{{_ingest.on_failure_message}}}' | ||
| - set: | ||
| tag: set_device_sensor_fragment_flag | ||
| description: >- | ||
| ISE forwards device-sensor values verbatim, so a neighbor's multi-line | ||
| banner (lldpSystemDescription) arrives with its newlines intact, each | ||
| escaped as a trailing backslash. The inputs split on them, leaving | ||
| continuation lines that carry no ISE header, so no category name is | ||
| parsed. Flag them so the attributes they carry can be processed as the | ||
|
There was a problem hiding this comment. Severity: 🔵 Low The Details
Recommendation: Make the description match the behaviour: - set:
tag: set_device_sensor_fragment_flag
description: >-
ISE forwards device-sensor values verbatim, so a neighbor's multi-line
banner (lldpSystemDescription) arrives with its newlines intact, each
escaped as a trailing backslash. The inputs split on them, leaving
continuation lines that carry no ISE header, so no category name is
parsed. Flag them so they can be tagged and filtered out rather than
reported as a parse error; their attributes are not reassembled into
the preceding record.
field: _tmp.device_sensor_fragment
value: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
||
| tail of the preceding record. | ||
| field: _tmp.device_sensor_fragment | ||
| value: true | ||
| if: >- | ||
| ctx.cisco_ise?.log?.category?.name == null && | ||
| ctx.event?.original instanceof String && | ||
| (ctx.event.original.endsWith('\\') || | ||
| ctx.event.original.trim().endsWith(',')) | ||
| - append: | ||
| tag: append_tag_device_sensor_fragment | ||
| field: tags | ||
| value: cisco_ise.device_sensor_fragment | ||
| allow_duplicates: false | ||
| if: ctx._tmp?.device_sensor_fragment == true | ||
| - grok: | ||
| field: _tmp.message | ||
| tag: grok_message_details | ||
|
|
@@ -41,6 +68,14 @@ processors: | |
| tag: append_error_message_0a0a0ab3 | ||
| field: error.message | ||
| value: '{{{ _ingest.on_failure_processor_tag }}}: {{{_ingest.on_failure_message}}}' | ||
| - set: | ||
| tag: set_message_from_event_original | ||
| description: >- | ||
| The device-sensor banner patterns match without capturing, so those | ||
| lines arrive here with no message. Keep their text searchable. | ||
| field: message | ||
| copy_from: event.original | ||
| if: ctx.message == null && ctx.event?.original != null | ||
| - trim: | ||
| tag: trim_message_c88e4c4f | ||
| field: message | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Severity: 🟡 Medium
confidence: highpath: packages/cisco_ise/changelog.yml:4The reworded 1.32.11 entry now claims it will "recover the attributes that follow an embedded newline", but fragments never reach a category pipeline so no attributes are parsed; drop that clause so the changelog matches the README and the test expectations.
Details
The new wording promises attribute recovery, but the pipeline does not do that. Fragment lines leave
grok_time_detailswithcisco_ise.log.category.nameunset (patterns at default.yml:22-26 capture only_tmp.message, never a category), and every category sub-pipeline at default.yml:156-227 is gated onctx.cisco_ise?.log?.category?.name == '<CISE_...>'. With no category, no sub-pipeline runs and the key-value parsing that would produceAcsSessionID,NetworkDeviceGroups,CPMSessionID, etc. never executes.The PR's own artifacts confirm this. Every fragment document in
test-pipeline-device-sensor-fragments.log-expected.jsoncontains only@timestamp,ecs.version,event.original,messageandtags- no parsed attributes, even for theCompiled ...line whosemessageis the fullAcsSessionID=.../NetworkDeviceGroups=...attribute list. The README bullet added in this same PR states the opposite of the changelog: "any attributes that followed the embedded newline (such asAcsSessionIDorNetworkDeviceGroups) aren't parsed into fields." A user reading the changelog will expect fields that never appear.Recommendation:
Describe only what the change does - tag the fragments and keep their text searchable:
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills