This application is an essential component of the data platform for an audio streaming service. Before running it, be sure to check out the architecture by following this link: https://github.com/iurii-chernigin/audio-streaming-data-platform
- Create a project in Google Cloud
- Create a service account, grant a Big Query Data Owner role, and generate json key
- Set up a connection to BigQuery
- Enable BigQuery API
- Set up a connection to Kafka
- Create tables in BigQuery according to the schema: audio-streaming-data-platform/data-warehouse/bigquery/schema
- Implement support for the Avro date format
- Implement support for the Kafka Schema Registry
- Change the data writing mechanism in BigQuery: add support of microbatch writing
- Create package to run application in Docker