Complete versioneer replacement with setuptools_scm for version management

[yohplala]

Fix version handling with setuptools_scm

PR to solves issue 950, where fastparquet version number is not correctly generated when CI is runned from a fork.

This PR finalilzes versioneer replacement with setuptools_scm for version management. The changes are focused on making the version system work properly.

Disclaimer

I have no experience in version system management, only in analyzing failing runs.
I conducted the analysis with help of Cursor AI + Claude 3.7 Sonnet, and the AI agent made the changes.
I double-checked them to prevent any change that would seem to me out of place.
Only these remaining changes appear to me the ones required to solve the problem.

Changes

1. Remove versioneer:

   • Removed versioneer configuration from setup.cfg
   • Removed versioneer dependency from CI environment file

2. Configure setuptools_scm properly:

   • Moved setuptools_scm configuration from setup.py to pyproject.toml
   • Simplified setup.py to use use_scm_version=True to defer to pyproject.toml config

3. Improve tag handling in CI:

   • Added upstream tag fetching in GitHub Actions workflows so CI builds have access to all tags

These changes ensure that version information is correctly generated during builds, especially in CI environments where git tags might not be fully available.

Testing

The changes have been tested in CI which now runs successfully.

[martindurant]

I'll have a look soon.

You will see from the CI runs above, that installing pandas from source still requires versioneer, so please add it back to that env (or add a line to the script part of the workflow to install it just before building pandas).

[yohplala]

I'll have a look soon.

You will see from the CI runs above, that installing pandas from source still requires versioneer, so please add it back to that env (or add a line to the script part of the workflow to install it just before building pandas).

Helo Martin,
Thank you for your feedback.
I have implemented proposed fix, but pandas workflow is still failing, this time with this error message:
Cython.Compiler.Errors.InternalError: Internal compiler error: 'free_threading_config.pxi' not found

My analysis is the following:

• in pandas workflow, we are building pandas from main
• there is this line in this module https://github.com/pandas-dev/pandas/blob/main/pandas/_libs/internals.pyx
include "free_threading_config.pxi"
• googling for "free_threading_config`` in Cython, I end up on this page talking about free threading for Cython 3.1
  https://docs.cython.org/en/latest/src/userguide/freethreading.html
• but when pulling Cython, the last version available is 3.0.12
• and I understand that free threading becomes available with python 3.13

I may be wrong but the solution I would propose is:

• force python to be python 3.13 (in the workflow, it is using python 3.10)
• force Cython to be above 3.1, but question is, where do we find wheels for 3.1 while it is still alpha apparently? Should we build it from sources?

My questions are probably very naïve, I have a limited understanding of this.
Please, can you advise what should be the next step?
Thanks for your help!

[martindurant]

Hm, you analysis seems spot on.
Given that pandas will, in the next release, depend on pyarrow: maybe we stop trying to follow pandas dev, and accept that fastparquet will only be useful for current and past releases.

[yohplala]

Hm, you analysis seems spot on. Given that pandas will, in the next release, depend on pyarrow: maybe we stop trying to follow pandas dev, and accept that fastparquet will only be useful for current and past releases.

• I would still push this PR because the error it solves has been existing in the other workflows (it relates to pushing a PR from a fork. If my understanding is correct, when you fork, tags are not copied from upstream. And fastparquet version number is generated from tags. So the resulting tag has been "hazardous" and when pandas is checking min fastparquet requirements, it fails)

• Regarding the pandas workflow, I would propose to leave it as it is currently. If I understand correctly, once Cython 3.1 is released, this trouble should be solved.
  As long as fastparquet is in its current state ("strongly" relying on pandas), it can be "comfortable" to have this workflow to "warn" us in advance that there could be upcoming compatibility issue with new pandas (once this trouble with free_threading is resolved I mean).

[martindurant]

If I understand correctly, once Cython 3.1 is released, this trouble should be solved

only if we run with py313, no?

have this workflow to "warn" us in advance that there could be upcoming compatibility issue with new pandas

We already have incompatibilities I haven't been able to spare the time to fix (
#947 ), and when pandas defaults to arrow, these can only increase or make fastparquet completely unworkable. Plus it takes away the most compelling reason to have fastparquet: not needing arrow to be installed.

[martindurant]

By the way: Cython 3.1 (alpha) is available now by using the --pre flag with pip (see https://pypi.org/project/Cython/3.1.0a1/#history )

[yohplala]

By the way: Cython 3.1 (alpha) is available now by using the --pre flag with pip (see https://pypi.org/project/Cython/3.1.0a1/#history )

Dear Martin,
As the different commits show it, I made several tests with cython 3.1.0a1, with python 3.13, and after different conflict issues, I could have them successfully installed... but not pandas. Its build would end up with the same error message as pointed above.

I quickly checked pandas build own workflow, and I have no real clue what is going on.
I ended up by submitting this to pandas github, ticket 61156.

[martindurant]

There will be nothing "free threading" related in py3.10 - it will need to be 3.13. Even then, I don't know if it requires specifically the free-threaded build.

[yohplala]

Even then, I don't know if it requires specifically the free-threaded build.

I agree with you, it should not be that complex. It would be surprising new pandas is breaking support for previous python versions and cython versions.

For this PR, I would propose to leave it as it is.
For pandas workflow, we can wait for a feedback from the pandas team on the ticket I opened.
In some time, we can review the status, and if nothing new, we could remove pandas worflow, only checking compatibility with last pandas wheels?