Skip to content

geo replicated subscribers #5274

Closed
Closed

Description

@volfco

One of the big things missing in my book, is a way to have multiple subscribers across datacenters consume messages "exactly" once.

DC A produces 5 messages. DC A has a consume subscription named TEST. DC B has a consumer with the same name. Ideally, a consumer in DC A process messages 1,2,3 while DC B is able to process 4 and 5.

Another example would be I'm writing emails to be sent out into a topic from us-west and us-east. I can send emails from each DC, and the DC should consume messages produced there (just by nature of asynchronous replication). But if the DC's consumers go down I would like the other DCs to pick up and consume messages that should have been consumed by the now failed DC.

From my admittedly limited research, I don't think this is currently supported. Subscribers cursor is currently per DC, which allows for consumers in each DC to consume the same message.

I could implement this on the client side by using the configuration store zookeeper to record a "lock" on a message ID to disallow consumption on other consumers- but this seems like something that could be implemented inside the broker and configurable on a per-namespace level. The consumer would write a lock into zookeeper, which other consumers would check for before doing the actual processing. If a lock exists, the consumer will ack it.

At the expense of consumption latency and zookeeper I/O, I think this would be an amazing feature.

I could see this implemented in two ways, both as Consumer Types. One is a "Lock on Everything" where a lock is acquired for each message. The second would be "Lock on foreign messages" where a lock is acquired for each message not produced in the local cluster (implying that it was replicated in from another cluster)- so messages produced from us-west would not require a global lock, but messages produced in us-west and consumed in us-east would need a global lock before processing.

Activity

  1. added
    type/featureThe PR added a new feature or issue requested a new feature
    on Sep 25, 2019
  2. aahmed-se commented on Sep 28, 2019

    @aahmed-se
    Contributor

    There has been some discussion on geo replicated subscriptions @merlimat can provide more insights.

  3. volfco commented on Sep 30, 2019

    @volfco
    Author

    The one thing I thought of in the background is that messages would have to be replicated with origin DC and origin ID or part of an Avro schema- as currently message IDs are unique per DC (from my understanding)

  4. sijie commented on Oct 27, 2019

    @sijie
    Member

    @volfco you might check out the replicated subscription introduced in 2.4.0. that can potentially solve you problem.

  5. volfco commented on Oct 27, 2019

    @volfco
    Author

    @sijie I'll try that tomorrow and let you know if it meets my need. I didn't see that in my search of the docs

  6. volfco commented on Oct 29, 2019

    @volfco
    Author

    @sijie I've gotten a MVP cluster setup between two DCs that are ~40ms apart with replication between them.

    This is my code (I'm horrible at java):

    import org.apache.pulsar.client.api.*;
    
    public class Main {
    
        public static void main(String[] args) throws Exception {
    
            if (args.length != 1) {
                System.out.println("Need serviceURL");
                System.exit(1);
            }
    
            PulsarClient client = PulsarClient.builder()
                    .serviceUrl(args[0])
                    .build();
    
            Consumer consumer = client.newConsumer()
                    .topic("persistent://development/ns1/test")
                    .subscriptionName("consumer")
                    .replicateSubscriptionState(true)
                    .subscribe();
    
            System.out.println("starting consumption");
            System.out.println(args[0]);
    
            while (true) {
                Thread.sleep(100);
                // Wait for a message
                Message msg = consumer.receive();
    
                try {
                    // Do something with the message
                    System.out.printf("Message received: %s\n", new String(msg.getData()));
    
                    // Acknowledge the message so that it can be deleted by the message broker
                    consumer.acknowledge(msg);
                } catch (Exception e) {
                    // Message failed to process, redeliver later
                    consumer.negativeAcknowledge(msg);
                }
            }
        }
    }

    Reading PIP-33, I'm operating under the assumption if I run one consumer reading from AUS, that consumer dies, and I start one reading from IAD a few seconds later, I shouldn't encounter many duplicate messages.

    This is not what I'm seeing. I can start the consumer in AUS, it'll start reading. I close it at message 10 and wait a few seconds and start an IAD consumer. It will start at the 0 position, and not around 10 like I would expect. If I restart the AUS consumer, it will pick up at message 10 and ignore any progress the IAD one has made.

    It also seems that this feature is almost what I'm looking for. From my understanding of PIP-33, I can only run one consumer at a time?

  7. merlimat commented on Oct 29, 2019

    @merlimat
    Contributor

    Keep in mind that the subscription state in asynchronously replicated at configurable interval.

    Take a look at broker.conf:

    # Frequency of snapshots for replicated subscriptions tracking.
    replicatedSubscriptionsSnapshotFrequencyMillis=1000
    
  8. merlimat commented on Oct 29, 2019

    @merlimat
    Contributor

    This is not what I'm seeing. I can start the consumer in AUS, it'll start reading. I close it at message 10 and wait a few seconds and start an IAD consumer.

    Ok, missed that you mentioned you're waiting few secs..

  9. merlimat commented on Oct 29, 2019

    @merlimat
    Contributor

    Can you check on the broker logs if it's enabling the replicated subscription on that topic?

  10. volfco commented on Oct 29, 2019

    @volfco
    Author

    @merlimat These logs tell me that replication has been enabled:

    4:55:14.350 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://development/ns1/test] Starting replicator to remote: aus
    14:55:14.350 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [development/ns1/persistent/test] Cursor pulsar.repl.aus recovered to position 13:-1
    14:55:14.350 [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [iad-pulsar1.local:3181] is not adhering to Placement Policy. quarantinedBookies: []
    14:55:14.352 [pulsar-ordered-OrderedExecutor-2-0-EventThread] ERROR org.apache.bookkeeper.proto.checksum.CRC32CDigestManager - Sse42Crc32C is not supported, will use a slower CRC32C implementation.
    14:55:14.352 [pulsar-ordered-OrderedExecutor-2-0-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [iad-pulsar1.local:3181] for ledger: 14
    14:55:14.355 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO  org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper - [development/ns1/persistent/test] [pulsar.repl.aus] Updating cursor info ledgerId=14 mark-delete=13:-1
    14:55:14.356 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [development/ns1/persistent/test] Updated cursor pulsar.repl.aus with ledger id 14 md-position=13:-1 rd-position=13:0
    14:55:14.356 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [development/ns1/persistent/test] Opened new cursor: ManagedCursorImpl{ledger=development/ns1/persistent/test, name=pulsar.repl.aus, ackPos=13:-1, readPos=13:0}
    14:55:14.357 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://development/ns1/test][iad -> aus] Starting replicator
    14:55:14.358 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.pulsar.broker.service.BrokerService - Created topic persistent://development/ns1/test - dedup is disabled
    14:55:14.359 [ForkJoinPool.commonPool-worker-2] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.1.2.105:58767]-3 persistent://development/ns1/test configured with schema false
    14:55:14.360 [ForkJoinPool.commonPool-worker-2] INFO  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://development/ns1/test][iad -> aus] Replicator already being started. Replicator state: Starting
    14:55:14.360 [ForkJoinPool.commonPool-worker-2] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.1.2.105:58767] Created new producer: Producer{topic=PersistentTopic{topic=persistent://development/ns1/test}, client=/10.1.2.105:58767, producerName=pulsar.repl.aus, producerId=3}
    14:55:14.397 [pulsar-io-22-5] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x3b7aa4e9, L:/10.170.53.77:41078 - R: /10.1.2.105:6650]] Connected to server
    14:55:14.518 [pulsar-io-22-8] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xfbfc6995, L:/10.170.53.77:41080 - R:aus-pulsar1.local/10.1.2.105:6650]] Connected to server
    14:55:14.603 [pulsar-io-22-8] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://development/ns1/test] [pulsar.repl.iad] Creating producer on cnx [id: 0xd965768e, L:/10.170.53.77:40868 - R:aus-pulsar1.local/10.1.2.105:6650]
    14:55:14.661 [pulsar-io-22-3] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://development/ns1/test] [pulsar.repl.iad] Created producer on cnx [id: 0xd965768e, L:/10.170.53.77:40868 - R:aus-pulsar1.local/10.1.2.105:6650]
    14:55:14.661 [pulsar-io-22-3] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [development/ns1/persistent/test-pulsar.repl.aus] Rewind from 13:0 to 13:0
    14:55:14.661 [pulsar-io-22-3] INFO  org.apache.pulsar.broker.service.persistent.PersistentReplicator - [persistent://development/ns1/test][iad -> aus] Created replicator producer
    14:55:14.881 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://development/ns1/test] Enabling replicated subscriptions controller
    14:55:14.999 [bookkeeper-ml-workers-OrderedExecutor-3-0] INFO  org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController - [PersistentTopic{topic=persistent://development/ns1/test}][consumer] Creating subscription at 13:13:0 after receiving update from replicated subcription
    
  11. merlimat commented on Oct 29, 2019

    @merlimat
    Contributor

    Ok, that doesn't look correct then. Can you enable debug logs on org.apache.pulsar.broker.service.persistent.{ReplicatedSubscriptionsController,ReplicatedSubscriptionSnapshotCache,ReplicatedSubscriptionsSnapshotBuilder}?

    Also there are a couple of metrics in Prometheus:

    • pulsar_replicated_subscriptions_pending_snapshots
    • pulsar_replicated_subscriptions_snapshot_ms
  12. volfco commented on Oct 29, 2019

    @volfco
    Author

    @merlimat Sure.

    Prometheus Metrics:

    pulsar_replicated_subscriptions_pending_snapshots{cluster="iad"} 0.0
    # TYPE pulsar_replicated_subscriptions_snapshot_ms summary
    pulsar_replicated_subscriptions_snapshot_ms_count{cluster="iad"} 964.0
    pulsar_replicated_subscriptions_snapshot_ms_sum{cluster="iad"} 54404.0
    

    image

    IAD on the left, AUS on the right. I waited 30 seconds between stopping IAD and starting AUS.

    I assume I configured log4j2.yaml wrong, as I go only the following message:

    16:35:44.555 [pulsar-inactivity-monitor-24-1] INFO  org.apache.pulsar.broker.service.AbstractReplicator - [persistent://public/default/my-topic][iad -> aus] Disconnect replicator at position 19:583 with backlog 0
    16:35:44.597 [pulsar-io-22-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/my-topic] [pulsar.repl.iad] Closed Producer
    

    log4j2.yaml config:

    Configuration:
      Loggers:
        Logger:
          - name: org.apache.pulsar.broker.service.persistent.{ReplicatedSubscriptionsController,ReplicatedSubscriptionSnapshotCache,ReplicatedSubscriptionsSnapshotBuilder}
            level: debug
    
  13. liguangcheng commented on Dec 3, 2021

    @liguangcheng

    @volfco & @sijie Now whether the replication subscription status is available, the version I tested is 2.8.0, and it does not work normally at present. One site produces 10 messages and consumes two. another sites do not start from the third one, but start from the beginning.

    How did you solve this problem

  14. locked and limited conversation to collaborators on Dec 19, 2022
  15. converted this issue into a discussion #18983 on Dec 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    lifecycle/staletype/featureThe PR added a new feature or issue requested a new feature

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions