Add buildkite pipeline for publishing to PyPI

[lorenabalan]

Related to https://github.com/elastic/connectors-py/issues/10999

• Added new manual pipeline for publishing to PyPI: .buildkite/pypi-publish-pipeline.yml
• Added new step in main pipeline (that runs on PRs and on daily schedule) to build the packages' binary & source distributions, then twine check them; for connectors_service it will also install it and test the CLI entrypoints it exposes (e.g. connectors --help)
• Changes to pyproject.toml
  • Incorporated pytest, pyright, and ruff config into pyproject.toml and removed individual files
  • Updated license in pyproject.toml to avoid deprecation warnings (removal in 2026) - see warnings here
  • Updated packages in pyproject.toml to avoid "ambiguous packages" warning - see warning here

❓ I don't see diff.sh used anywhere.. can I delete it or am I missing sth? nevermind, I was looking for diff.sh instead of just diff

✍️ Process moving forward

Daily per PR:

• Validate that packages are built correctly
• "Publish DRA" part of pipeline.yml will contain:
  • building docker image by installing package locally by building and installing from distribution (TODO)
  • build a .tar.gz and .whl of our packages (TODO)

Release day:

• same process as today, a Docker image DRA is chosen as BC and then released
• additionally, trigger manual pipeline to publish to PyPI from appropriate commit

TODO:

• Update catalog-info.yml
• Publish .tar.gz and .whl as DRA
• Ensure Docker images are built with installing code from .whl

Checklists

Pre-Review Checklist

• this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
• this PR has a meaningful title
• this PR links to all relevant github issues that it fixes or partially addresses
• if there is no GH issue, please create it. Each PR should have a link to an issue
• this PR has a thorough description
• Covered the changes with automated tests
• Tested the changes locally
• Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
• For bugfixes: backport safely to all minor branches still receiving patch releases
• Considered corresponding documentation changes
• Contributed any configuration settings changes to the configuration reference
• if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

Changes Requiring Extra Attention

• Security-related changes (encryption, TLS, SSRF, etc)
• New external service dependencies added.

Related Pull Requests

Release Note

[lorenabalan]

How it shows up in the UI:

...

[artem-shelkovnikov]

❓ I don't see diff.sh used anywhere.. can I delete it or am I missing sth?

diff.sh is used in here: https://github.com/elastic/connectors/blob/main/.buildkite/pipeline.yml#L84

❓ 🧠 What happens if we have a bug in our package? Requesting an emergency stack release in order to release a new version on PyPI seems excessive, so does yanking every buggy package from PyPI. I'm more in favour of having different versions conventions but that will get easily confusing.

We had a {major}.{minor}.{patch}.{release} available at some point, but since we release with stack it could be hard to organise well. We might need more strict QA process for it? I'd open a discussion with our team and potentially the clients team - they can give a good hint about it.

Re: releasing
For now the process for release is "promoting" the image. So by BC1 we already need to have an image with a real package installed.

Additionally, if we feel like it's too much we can keep the current process for docker images around for now (same as we do now with make install and such) and as we polish the process we'll switch to it.

[mattnowzari]

Will connectors_sdk pyproject.toml also consolidate .ruff.toml, pytest.ini and pyrightconfig.json in this PR or a follow-up one?

[lorenabalan]

I'll do it in a follow-up, I was mainly focusing on the connectors_service package as part of this PR.