Implement llm metadata generator

[AlitaBernachot]

This pull request adds AI-powered metadata generation to the Datafeeder backend, allowing automatic enrichment of dataset metadata using LLMs. It introduces new configuration options, a dedicated API endpoint for triggering metadata generation, and integrates Phoenix observability for LLM tracing. The database schema is updated to support additional configuration storage, and the documentation is updated to guide users through enabling and using the AI features.

AI Metadata Generation Integration

• Added AI metadata generation capability using LLMs, with support for multiple providers and models, configurable via a new docker/.envs-ai file and corresponding settings in Settings (AI_ENABLED, AI_PROVIDER, etc.). [1] [2] [3] [4]
• Introduced a new /llm/generate_metadata/{intlink_id} API endpoint to generate metadata suggestions for a dataset, with request/response models and error handling. [1] [2] [3]
• Integrated AI metadata generation into the data ingestion process, allowing it to be triggered as part of the workflow and storing relevant flags in the extra_config field. [1] [2] [3] [4] [5] [6]

Database and Model Changes

• Added a new Alembic migration to introduce an extra_config JSONB column to the integrity_link table for storing AI-related and other configuration.
• Updated staging and editing logic to initialize or clear the extra_config field as appropriate. [1] [2]

Observability and Development Enhancements

• Added a new Dockerfile build stage and Makefile target (up-ai) for running the backend with Phoenix observability and AI extras, and enabled Phoenix tracing if configured. [1] [2] [3] [4] [5] [6]

Configuration and Documentation

• Updated configuration loading to support an optional, uncommitted AI secrets file (AI_ENV_FILE), and documented setup steps and security considerations in the README.md. [1] [2]

These changes collectively enable optional, configurable AI-powered metadata enrichment in Datafeeder, with robust integration into the ingestion process and improved observability support.

Human edit
Some functionalities, out of scope, they have been removed. See edb76a4

To implement