Skip to content

Commit ae93d2c

Browse files
vincbeckvincbeck
committed
Add documentation for team-based asset event filtering
1 parent 052d27f commit ae93d2c

3 files changed

Lines changed: 233 additions & 0 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/assets.rst‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,46 @@ As mentioned in :ref:`Fetching information from previously emitted asset events<
402402
events = inlet_events[AssetAlias("example-alias")]
403403
last_row_count = events[-1].extra["row_count"]
404404
405+
.. _asset_allow_teams:
406+
407+
Cross-team asset event filtering with ``allow_teams``
408+
-----------------------------------------------------
409+
410+
.. versionadded:: 3.3.0
411+
412+
When :doc:`Multi-Team mode </core-concepts/multi-team>` is enabled, asset events are filtered by team
413+
membership. By default, a consuming Dag only receives asset events produced by Dags within the same team.
414+
This prevents unintended cross-team triggers.
415+
416+
To allow specific other teams to produce events that trigger your Dag, use the ``allow_teams`` parameter
417+
on the ``Asset`` definition:
418+
419+
.. code-block:: python
420+
421+
from airflow.sdk import Asset
422+
423+
shared_data = Asset(
424+
name="my_data",
425+
uri="s3://bucket/shared/data.csv",
426+
allow_teams=["team_analytics", "team_ml"],
427+
)
428+
429+
In this example, asset events produced by Dags belonging to ``team_analytics`` or ``team_ml`` will be
430+
accepted by any consuming Dag that schedules on ``shared_data``, in addition to events from the consuming
431+
Dag's own team.
432+
433+
Default behavior
434+
~~~~~~~~~~~~~~~~
435+
436+
When ``allow_teams`` is not specified (or set to an empty list), the default same-team filtering applies:
437+
438+
- A consuming Dag only receives events from Dags in the same team.
439+
- Dags with no team association (global Dags) can produce events that trigger any consumer.
440+
- Teamless consumers only accept events from teamless producers.
441+
442+
When Multi-Team mode is disabled, ``allow_teams`` is ignored and all asset events are delivered to all
443+
consuming Dags, preserving backward compatibility.
444+
405445
Asset partitions
406446
----------------
407447

‎airflow-core/docs/core-concepts/multi-team.rst‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -514,6 +514,124 @@ indicating more than one team is present in the architecture).
514514
.. image:: /img/multi_team_arch_diagram.png
515515
:alt: Multi-Team Airflow Architecture showing resource isolation between teams
516516

517+
.. _multi-team-asset-event-filtering:
518+
519+
Team-Based Asset Event Filtering
520+
---------------------------------
521+
522+
When Multi-Team mode is enabled, asset events are filtered by team membership before they trigger
523+
downstream Dag runs. This prevents asset events produced by one team's Dags from unintentionally
524+
triggering Dag runs for a different team.
525+
526+
Default Behavior
527+
^^^^^^^^^^^^^^^^
528+
529+
By default, a consuming Dag only receives asset events from producers within the same team. Dags with
530+
no team association (global Dags) act as global producers and can trigger any team-bound consumer.
531+
532+
Cross-Team Opt-In with ``allow_teams``
533+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
534+
535+
To allow specific teams to produce events that trigger consumers on a given asset, use the
536+
``allow_teams`` parameter on the ``Asset`` definition:
537+
538+
.. code-block:: python
539+
540+
from airflow.sdk import Asset
541+
542+
shared_data = Asset(
543+
name="shared_data",
544+
uri="s3://bucket/shared/data.csv",
545+
allow_teams=["team_analytics", "team_ml"],
546+
)
547+
548+
With this configuration, asset events from ``team_analytics`` or ``team_ml`` will be accepted by any
549+
consuming Dag that schedules on ``shared_data``, in addition to events from the consumer's own team.
550+
551+
See :ref:`Cross-team asset event filtering with allow_teams <asset_allow_teams>` in the Assets
552+
documentation for usage details and validation rules.
553+
554+
Behavioral Rules
555+
^^^^^^^^^^^^^^^^
556+
557+
The following truth table describes the complete filtering logic:
558+
559+
.. list-table::
560+
:header-rows: 1
561+
:widths: 20 20 20 15 25
562+
563+
* - Producer
564+
- Consumer
565+
- ``allow_teams``
566+
- Result
567+
- Reason
568+
* - Team A (DAG)
569+
- Team A
570+
- (any)
571+
- ✅ Allowed
572+
- Same team
573+
* - Team A (DAG)
574+
- Team B
575+
- ``[]``
576+
- ❌ Blocked
577+
- Different team, no opt-in
578+
* - Team A (DAG)
579+
- Team B
580+
- ``["team_a"]``
581+
- ✅ Allowed
582+
- Cross-team opt-in
583+
* - (no team, DAG)
584+
- Team B
585+
- (any)
586+
- ✅ Allowed
587+
- Global producer
588+
* - Team A (DAG)
589+
- (no team)
590+
- (any)
591+
- ❌ Blocked
592+
- Teamless consumer rejects team-bound producer
593+
* - (no team, DAG)
594+
- (no team)
595+
- (any)
596+
- ✅ Allowed
597+
- Both global
598+
* - Team A (API)
599+
- Team A
600+
- (any)
601+
- ✅ Allowed
602+
- Same team
603+
* - Team A (API)
604+
- Team B
605+
- ``["team_a"]``
606+
- ✅ Allowed
607+
- Cross-team opt-in
608+
* - (no team, API)
609+
- Team B
610+
- (any)
611+
- ❌ Blocked
612+
- Teamless API user cannot trigger team-bound consumer
613+
* - (no team, API)
614+
- (no team)
615+
- (any)
616+
- ✅ Allowed
617+
- Both global
618+
619+
Key rules:
620+
621+
- **Same team**: Always allowed.
622+
- **Global (teamless) DAG producer**: Triggers all consumers regardless of team.
623+
- **Teamless API user**: Can only trigger teamless consumers.
624+
- **Teamless consumer**: Only accepts events from teamless sources.
625+
- **Cross-team via** ``allow_teams``: Allowed when the producer's team is listed in the asset's ``allow_teams``.
626+
- **Multi-Team disabled**: All filtering is skipped; existing behavior is preserved.
627+
628+
API-Triggered Events
629+
^^^^^^^^^^^^^^^^^^^^
630+
631+
When a user creates an asset event via the REST API, the user's team is resolved from the auth manager.
632+
The same filtering rules apply, with one distinction: a teamless API user can only trigger teamless
633+
consumers, whereas a teamless DAG producer is treated as global and can trigger any consumer.
634+
517635
Important Considerations
518636
------------------------
519637

‎airflow-core/src/airflow/example_dags/example_asset_allow_teams.py‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
"""
18+
Example DAG demonstrating cross-team asset triggering with ``allow_teams``.
19+
20+
When Multi-Team mode is enabled (``[core] multi_team = True``), asset events are filtered by team
21+
membership. By default, a consuming DAG only receives events from DAGs within the same team.
22+
23+
Usage:
24+
- ``team_analytics_producer`` (belonging to ``team_analytics``) produces events on ``shared_data``.
25+
- ``team_ml_consumer`` (belonging to ``team_ml``) consumes ``shared_data``.
26+
- Because ``shared_data`` has ``allow_teams=["team_analytics"]``, events from ``team_analytics``
27+
are accepted by ``team_ml_consumer``.
28+
- Without ``allow_teams``, the cross-team event would be blocked.
29+
"""
30+
31+
from __future__ import annotations
32+
33+
import pendulum
34+
35+
from airflow.providers.standard.operators.bash import BashOperator
36+
from airflow.sdk import DAG, Asset
37+
38+
# [START asset_allow_teams]
39+
# Define an asset that accepts events from team_analytics in addition to the consumer's own team.
40+
shared_data = Asset(
41+
name="shared_data",
42+
uri="s3://data-lake/shared/output.csv",
43+
allow_teams=["team_analytics"],
44+
)
45+
46+
# Producer DAG — belongs to team_analytics (via its DAG bundle configuration).
47+
# When this DAG's task completes, it emits an asset event for shared_data.
48+
with DAG(
49+
dag_id="team_analytics_producer",
50+
start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
51+
schedule="@daily",
52+
catchup=False,
53+
tags=["team_analytics", "produces", "asset-scheduled", "allow-teams"],
54+
) as producer_dag:
55+
BashOperator(
56+
task_id="produce_shared_data",
57+
outlets=[shared_data],
58+
bash_command="echo 'Producing shared data for cross-team consumption'",
59+
)
60+
61+
# Consumer DAG — belongs to team_ml (via its DAG bundle configuration).
62+
# This DAG is triggered when shared_data is updated. Because shared_data has
63+
# allow_teams=["team_analytics"], events from team_analytics are accepted.
64+
with DAG(
65+
dag_id="team_ml_consumer",
66+
start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
67+
schedule=[shared_data],
68+
catchup=False,
69+
tags=["team_ml", "consumes", "asset-scheduled", "allow-teams"],
70+
) as consumer_dag:
71+
BashOperator(
72+
task_id="consume_shared_data",
73+
bash_command="echo 'Consuming shared data from team_analytics'",
74+
)
75+
# [END asset_allow_teams]

0 commit comments

Comments
 (0)