Handle all dtypes in convert_dtypes

[galipremsagar]

Description

This PR brings convert_dtypes in party with pandas3

Comparison: pandas3 vs This PR

Metric	pandas3	This PR	Δ (PR − pandas3)
failed	3,710	3,547	−163 ✅
passed	203,927	204,095	+168 ✅
skipped	7,361	7,361	0
xfailed	6,283	6,278	−5
xpassed	77	77	0
warnings	7,305	7,305	0

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[mroeschke]

If StringColumn had an pd.ArrowDtype(pa.null) I would consider this a bug. The pyarrow types should only be a pa.string or pa.large_string

[galipremsagar]

In all our cudf type conversions, we convert pa.null type to object dtype. Previously, we used to map pa.null to int8 but we made this recent change to object to match closely with pandas. Do you think we should be mapping to some other type or introduce a new type null type in cudf?

[mroeschke]

I think converting pa.null to object is correct. I'm mainly wondering what series of calls leads to StringColumn having a pd.ArrowDtype(pa.null()) where we need specific check for this in to_arrow?

[galipremsagar]

ColumnBase.create normalizes the plc buffer but keeps the null dtype:

# For pandas nullable null types (ArrowDtype wrapping pa.null()),                                                                                                
# normalize the column data and dtype before construction.                                                                                                       
col, dtype, old_dtype = maybe_normalize_arrow_null(col, dtype)
                                                                                                                                                                 
target_cls = ColumnBase._dispatch_subclass_from_dtype(dtype)   # dispatches on object → StringColumn
self = target_cls.__new__(target_cls)                                                                                                                            
self.plc_column = _wrap_and_validate(col, dtype) if validate else col                                                                                            
self._dtype = dtype if old_dtype is None else old_dtype         # original ArrowDtype(pa.null()) is restored
                                                                                                                                                                 
So the subclass dispatch sees np.dtype("object") and picks StringColumn, but self._dtype ends up as the original pd.ArrowDtype(pa.null()) — which is what the new
 to_arrow check guards against.                                                                                                                                  

I know we have been wanting to remove this kind of casting in ColumnBase.create, I plan on doing that once we are close to merging to main

[mroeschke]

Ah ok if this goes back to the casting thing we're doing in ColumnBase, I would suggest we hold off on these changes since this might fix itself once fix the ColumnBase issue

[galipremsagar]

Ah ok if this goes back to the casting thing we're doing in ColumnBase, I would suggest we hold off on these changes since this might fix itself once fix the ColumnBase issue

Opened a PR to address this issue and remove the workaround from create: #22396

[galipremsagar]

/okay to test de74324