From b246c4a4d239cd3bed050df82b8870c35a15760b Mon Sep 17 00:00:00 2001 From: eric-forte-elastic Date: Wed, 24 Jun 2026 14:13:52 -0400 Subject: [PATCH 1/2] Initial documentation --- ...s_and_data_from_elastic_security_to_vcs.md | 2 + docs/dac_quick_start_guide.md | 78 ++++++++++++++++++- docs/faq.md | 14 +++- docs/internals_of_the_detection_rules_repo.md | 11 +++ 4 files changed, 103 insertions(+), 2 deletions(-) diff --git a/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md b/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md index 42072ca..741ee54 100644 --- a/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md +++ b/docs/core_component_syncing_rules_and_data_from_elastic_security_to_vcs.md @@ -126,6 +126,8 @@ jobs: Here **`kibana export-rules --directory` (`-d`)** is the **output** directory (unlike **`export-rules-from-repo`**, where **`-d`** loads **input** rules). Add **`--save-as-yaml` / `-sy`** to write YAML instead of TOML. For repo → NDJSON or per-rule YAML, see **`export-rules-from-repo`** in the [Quick Start](dac_quick_start_guide.md#syncing-with-kibana). +When syncing more than one rule set from the same Kibana space, use `--custom-rules-only` / `-cro` or `--export-query` / `-eq` to keep each export intentionally scoped. For example, custom rules can be exported with `-cro`, customized Elastic prebuilt rules can be filtered with `alert.attributes.params.ruleSource.isCustomized: true and alert.attributes.params.immutable: true`, and unmodified Elastic prebuilt rules can be filtered with `alert.attributes.params.ruleSource.isCustomized: false and alert.attributes.params.immutable: true`. Avoid using `enabled` to classify rule ownership because it only represents whether the rule is active in Kibana. For more detail and examples, see [Managing custom and prebuilt rules together](dac_quick_start_guide.md#managing-custom-and-prebuilt-rules-together). + ```bash # Export Rules from Elastic Security python -m detection_rules kibana export-rules -d test-export-rules --skip-errors diff --git a/docs/dac_quick_start_guide.md b/docs/dac_quick_start_guide.md index cd24641..f9b95bd 100644 --- a/docs/dac_quick_start_guide.md +++ b/docs/dac_quick_start_guide.md @@ -343,7 +343,83 @@ Optionally add `-d ` or `-f ` to limit to a directory or single --- -## 10. Version locking (optional) +## 10. Managing multiple rule sets + +Many DaC workflows eventually need to manage more than one collection of rules at the same time: team-owned custom rules, environment-specific rules, modified Elastic prebuilt rules, and unmodified Elastic prebuilt rules that should still be backed up or promoted between spaces. Treat each collection as a rule set with an explicit source of truth, then decide whether the sets should share one deployment config or move independently. + +**Recommended local patterns:** + +- **One deployment unit:** Use one custom rules directory and list multiple rule directories in its `_config.yaml`. This is best when the rule sets share the same schema map, test config, versioning strategy, exceptions, and release process. + ```yaml + rule_dirs: + - rules + - rules_team_a + - rules_team_b + ``` +- **Independent deployment units:** Use a separate custom rules directory for each rule set when different teams, spaces, schemas, tests, or release cadences need to stay isolated. Set `CUSTOM_RULES_DIR` to the rule set you are validating or syncing so the CLI loads the matching `_config.yaml`, `stack-schema-map.yaml`, `test_config.yaml`, and optional version files. + +For repository-to-artifact workflows, `export-rules-from-repo` can load more than one input directory by repeating `--directory` / `-d`. This is useful when you want one NDJSON handoff from several compatible local rule sets: + +```bash +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules export-rules-from-repo \ + --directory dac_custom_rules_dir/rules \ + --directory dac_custom_rules_dir/rules_team_a \ + --outfile combined-rules.ndjson \ + --include-exceptions \ + --include-action-connectors +``` + +If the rule sets have different configs, validate and export each one separately, then combine them at the Kibana import, CI/CD, or change-management layer instead of forcing them through a single `_config.yaml`. + +### Managing custom and prebuilt rules together + +`kibana export-rules` can export custom rules, Elastic prebuilt rules, and customized Elastic prebuilt rules from the same Kibana space. By default, it exports all matching rules. Use `--custom-rules-only` / `-cro` for custom rules, or `--export-query` / `-eq` when you need more precise Kibana-side filtering. + +Do not use `enabled` as the primary way to classify rule ownership. `enabled` only says whether a rule is active in Kibana. Some Elastic-authored rules are enabled by default for structural reasons, such as surfacing alerts from Elastic Defend or Elastic Cloud Defend, and a disabled rule can still be a custom or customized prebuilt rule. For ownership and customization, prefer `immutable` and `ruleSource` fields exposed in exported rule objects and queryable through the Kibana rule filter syntax. + +Useful categories: + +- **Custom rules:** use `--custom-rules-only` / `-cro`, or filter for internal/non-immutable rules. +- **Customized prebuilt rules:** `immutable: true` and `ruleSource.isCustomized: true`. +- **Unmodified prebuilt rules:** `immutable: true` and `ruleSource.isCustomized: false`. + +Examples: + +```bash +# Custom rules only +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules kibana export-rules \ + --directory exports/custom-rules \ + --custom-rules-only \ + --skip-errors +``` + +```bash +# Customized Elastic prebuilt rules +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules kibana export-rules \ + --directory exports/customized-prebuilt-rules \ + --export-query 'alert.attributes.params.ruleSource.isCustomized: true and alert.attributes.params.immutable: true' \ + --skip-errors +``` + +```bash +# Unmodified Elastic prebuilt rules +CUSTOM_RULES_DIR=dac_custom_rules_dir \ +python -m detection_rules kibana export-rules \ + --directory exports/unmodified-prebuilt-rules \ + --export-query 'alert.attributes.params.ruleSource.isCustomized: false and alert.attributes.params.immutable: true' \ + --skip-errors +``` + +Kibana's rule filter syntax uses `alert.attributes.*` field paths. Some exported rule JSON fields use different casing or nesting than the filter path; for example, exported rule metadata may appear as `rule_source.is_customized`, while the filter path is `alert.attributes.params.ruleSource.isCustomized`. See the Kibana [Find rules API](https://www.elastic.co/docs/api/doc/kibana/operation/operation-findrules) for the `filter` parameter used by the Detection Engine rule search endpoint. + +When you need to audit what changed in a customized prebuilt rule, inspect the exported `rule_source.customized_fields` list. For example, a customized prebuilt rule may show `query` and `index` in `customized_fields`, which lets you distinguish a rule whose detection logic changed from one that only points to a different index pattern. + +--- + +## 11. Version locking (optional) Version locking can help when you sync in both directions (repo ↔ Kibana) or use overwrite workflows with multiple sources of truth at a time for rules: the lock file records which version of each rule is “current,” so the CLI can avoid overwriting newer changes or detect conflicts. If you only have a single source of truth at one time, you typically do not need it. diff --git a/docs/faq.md b/docs/faq.md index 9c530e5..7f64997 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -179,4 +179,16 @@ Running `view-rule` on this rule should look similar to the following. When the View Rule Custom Schema -For more information on how you can also automatically generate the custom schemas please see [Option 3: Custom Schema Validation](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-3-custom-schema-validation) \ No newline at end of file +For more information on how you can also automatically generate the custom schemas please see [Option 3: Custom Schema Validation](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-3-custom-schema-validation) + +#### **Q16**: How do I distinguish custom rules, customized prebuilt rules, and unmodified prebuilt rules when exporting from Kibana? + +**A16**: Use rule source metadata rather than the rule's `enabled` state. `enabled` only indicates whether the rule is active in Kibana; it does not reliably identify ownership or customization. + +Useful export categories are: + +- Custom rules: use `python -m detection_rules kibana export-rules --custom-rules-only`, or filter for internal/non-immutable rules. +- Customized Elastic prebuilt rules: filter with `alert.attributes.params.ruleSource.isCustomized: true and alert.attributes.params.immutable: true`. +- Unmodified Elastic prebuilt rules: filter with `alert.attributes.params.ruleSource.isCustomized: false and alert.attributes.params.immutable: true`. + +When reviewing exported customized prebuilt rules, inspect `rule_source.customized_fields` to see what changed. For example, `query` means detection logic was customized, while `index` means the rule's index patterns were customized. See [Managing custom and prebuilt rules together](dac_quick_start_guide.md#managing-custom-and-prebuilt-rules-together) for full command examples. \ No newline at end of file diff --git a/docs/internals_of_the_detection_rules_repo.md b/docs/internals_of_the_detection_rules_repo.md index 7b2296e..6fe361d 100644 --- a/docs/internals_of_the_detection_rules_repo.md +++ b/docs/internals_of_the_detection_rules_repo.md @@ -295,6 +295,17 @@ testing: config: etc/test_config.yaml ``` +For multiple local rule sets, choose whether they should be loaded by one custom rules config or by separate configs. If the rule sets share schema validation, tests, exceptions/actions directories, and versioning strategy, list multiple directories under `rule_dirs`: + +```yaml +rule_dirs: +- rules +- rules_team_a +- rules_team_b +``` + +If the rule sets belong to different teams, Kibana spaces, stack versions, schema maps, or release cadences, use separate custom rules directories and set `CUSTOM_RULES_DIR` to the directory being validated or synced. This keeps each rule set's `_config.yaml`, `stack-schema-map.yaml`, `test_config.yaml`, and optional version files independent. For Kibana export examples that separate custom rules from customized and unmodified Elastic prebuilt rules, see [Managing custom and prebuilt rules together](dac_quick_start_guide.md#managing-custom-and-prebuilt-rules-together). + ```bash (detection-rules-build) ➜ detection-rules git:(main) ✗ ❯ python -m detection_rules custom-rules setup-config dac_custom_rules_dir Loaded config file: /home/user/code/detection-rules/.detection-rules-cfg.json From c01b2f5cdb90b949a3813f005293b95f6ded5177 Mon Sep 17 00:00:00 2001 From: eric-forte-elastic Date: Tue, 28 Jul 2026 08:40:55 -0400 Subject: [PATCH 2/2] Add diagrams --- docs/_static/multiple_rule_sets_dual_sync.svg | 97 +++++++++++++++++++ .../multiple_rule_sets_vcs_authoritative.svg | 86 ++++++++++++++++ ...components_and_governance_models_of_dac.md | 14 +++ docs/dac_quick_start_guide.md | 2 + 4 files changed, 199 insertions(+) create mode 100644 docs/_static/multiple_rule_sets_dual_sync.svg create mode 100644 docs/_static/multiple_rule_sets_vcs_authoritative.svg diff --git a/docs/_static/multiple_rule_sets_dual_sync.svg b/docs/_static/multiple_rule_sets_dual_sync.svg new file mode 100644 index 0000000..79af4f8 --- /dev/null +++ b/docs/_static/multiple_rule_sets_dual_sync.svg @@ -0,0 +1,97 @@ + + Managing custom and prebuilt rule sets with dual sync + Custom and prebuilt rule changes can originate in VCS or Elastic Security. Export jobs filter custom, customized prebuilt, and unmodified prebuilt rules from Kibana, reconcile them in pull requests, then deploy the merged source back to Elastic Security. + + + + + + + + + Multiple Rule Sets: Dual Sync + Changes may originate in VCS or Kibana, but reconciliation happens before either side overwrites the other. + + + Elastic detection-rules + Upstream prebuilt updates + + + Rule authors + Custom rules + Prebuilt customizations + + + Organization VCS + Reviewed rule history + + + Prebuilt rule set + Upstream + local history + + + Customized prebuilt + Tracked changes + + + Custom rules + Team-owned content + + + Deploy job + Validate and import + merged rule sets + + + Reconciliation PR + Compare version/hash + Review exported changes + Resolve conflicts + Merge approved state + + + Elastic Security + Custom rules + Customized prebuilt + Unmodified prebuilt + Kibana/UI changes + are exported and + reviewed before merge + + + Filtered export jobs + -cro for custom rules + -eq for customized/unmodified prebuilt rules + + + + + + + + + + + upstream + sync + direct PRs + merge + deploy + export from Kibana + sync back + diff --git a/docs/_static/multiple_rule_sets_vcs_authoritative.svg b/docs/_static/multiple_rule_sets_vcs_authoritative.svg new file mode 100644 index 0000000..33f284e --- /dev/null +++ b/docs/_static/multiple_rule_sets_vcs_authoritative.svg @@ -0,0 +1,86 @@ + + Managing custom and prebuilt rule sets with VCS as authoritative + Elastic detection-rules upstream feeds approved prebuilt rule updates into an organization VCS. Custom rules and customized prebuilt rules are reviewed in VCS, validated by CI, and deployed one way into Elastic Security. + + + + + + + + + Multiple Rule Sets: VCS as Authoritative + All production changes are reviewed in VCS first, then deployed one way to Elastic Security. + + + Elastic detection-rules + Upstream prebuilt rules + Pull updates into org fork + + + Detection engineers + Custom rules + New team-owned detections + + + Organization VCS + Authoritative rule source + + + Prebuilt rule set + Selected upstream updates + + + Customized prebuilt set + Approved query/index changes + + + Custom rule set + Team-owned rules + + + CI/CD gate + Validate schemas + Run unit tests + Review/version rules + Import via API/CLI + + + Kibana edits + Discouraged for managed + production rules + + + Elastic + Security + Custom + prebuilt + + + + + + + + upstream sync + + PR + rule PRs + merge to + deploy + one-way + backup/audit only + diff --git a/docs/core_components_and_governance_models_of_dac.md b/docs/core_components_and_governance_models_of_dac.md index 2d888d1..2a2f49a 100644 --- a/docs/core_components_and_governance_models_of_dac.md +++ b/docs/core_components_and_governance_models_of_dac.md @@ -59,6 +59,13 @@ For details on the various options, refer to those specific core components sect When the Version Control System (VCS) serves as the authoritative source, all rule changes originate from VCS, emphasizing a code-first approach to detection rule management. This model supports strict change control and full traceability of rule modifications, which is ideal for teams prioritizing rigorous oversight and auditability of detection rules. +For teams managing both custom rules and Elastic prebuilt rules, the VCS-authoritative model treats the organization repository as the source of truth for every deployed rule set. Elastic prebuilt rule updates are pulled from the upstream `elastic/detection-rules` repository, custom rules are created in the organization's custom rules directory, and any prebuilt rule customization is reviewed as code before deployment to Elastic Security. + +| | +| -------------------------------------------------------------------------------------- | +| Managing custom and prebuilt rule sets with VCS as authoritative | +|
*Managing Multiple Rule Sets with VCS as Authoritative*
| + #### Considerations Choosing VCS as the authoritative source requires establishing stringent procedures for rule creation, modification, and deployment, ensuring that all changes are versioned and reviewed before being synchronized to Elastic Security. This governance model relies heavily on CI/CD pipelines to manage deployments and maintain rule integrity across environments. @@ -146,6 +153,13 @@ Dual syncing between Elastic Security and a Version Control System (VCS) embodie | Dual Sync Diagram | |
*Figure 2: Combining Multiple Options to Dual Sync Rules*
| +When managing custom rules and prebuilt rules together in a dual sync model, use scoped exports from Elastic Security to separate custom rules, customized prebuilt rules, and unmodified prebuilt rules before opening a reconciliation PR. This prevents an analyst's Kibana-side change from silently overwriting a VCS-side change, and it keeps prebuilt rule package updates distinct from local customizations. + +| | +| -------------------------------------------------------------------------------------- | +| Managing custom and prebuilt rule sets with dual sync | +|
*Managing Multiple Rule Sets with Dual Sync*
| + #### Considerations Implementing a dual syncing strategy requires careful planning and consideration of several factors to ensure its success. A successful dual syncing strategy involves integrating automation tools to handle the bidirectional flow of rule data. It requires a robust setup that can automatically compare rules by version or hash, ensuring that updates made within Elastic Security are synchronized back into VCS before final deployment. Utilizing the version lock file from the detection rules repository, or a bespoke versioning system, plays a pivotal role in managing this synchronization process seamlessly. Additionally, team-specific processes may be required to ensure production rules remain in sync. diff --git a/docs/dac_quick_start_guide.md b/docs/dac_quick_start_guide.md index f9b95bd..9e2045c 100644 --- a/docs/dac_quick_start_guide.md +++ b/docs/dac_quick_start_guide.md @@ -347,6 +347,8 @@ Optionally add `-d ` or `-f ` to limit to a directory or single Many DaC workflows eventually need to manage more than one collection of rules at the same time: team-owned custom rules, environment-specific rules, modified Elastic prebuilt rules, and unmodified Elastic prebuilt rules that should still be backed up or promoted between spaces. Treat each collection as a rule set with an explicit source of truth, then decide whether the sets should share one deployment config or move independently. +For visual examples of managing custom and prebuilt rules together, see the [VCS as authoritative](core_components_and_governance_models_of_dac.md#vcs-as-authoritative) and [dual sync](core_components_and_governance_models_of_dac.md#dual-syncing-rules-and-data-between-kibana-and-vcs) diagrams in the governance models guide. + **Recommended local patterns:** - **One deployment unit:** Use one custom rules directory and list multiple rule directories in its `_config.yaml`. This is best when the rule sets share the same schema map, test config, versioning strategy, exceptions, and release process.