API: provide (advanced) utility to update (swap) the underlying array of a Series in-place

[jorisvandenbossche]

Related to #63099 and the change of returning a read-only view in Series.array.

One reason that the read-only view is a problem (for the use case of GeoPandas), is because we want to update the underlying array of the Series in place, e.g. for updating an attribute.

But if we would keep doing that through a ser.array that directly gives the underlying array:

ser.array.attr = ...

then that would actually still violate the idea of CoW, because this attribute change would propagate to any shallow copy of the series (although we never explicitly discussed whether attributes fall under CoW or not).

Right now, if you want to actually update (swap) the underlying array of a Series without going though ser.array/ser.values/ser._values, you essentially can swap the underlying manager (this is of course using private APIs, but to illustrate):

>>> ser = pd.Series([1, 2, 3])
>>> mgr = ser._mgr
>>> arr = mgr.array
>>> arr = arr.view()
>>> arr[0] = 100  # or change an attribute on an EA, but mutate value to illustrate here
>>> mgr.set_values(arr)
>>> ser._mgr = mgr
>>> ser
0    100
1      2
2      3
dtype: int64

To achieve something like this, we could also provide a method, something like ser.update_array(arr) (although given how of an advanced use case this is, it is also not great that this would be an actual public method, so we could also make it a semi-private (hidden but documented for use by developers) method like _update_array().

I would have to think how it should handle the CoW situation (i.e. should it keep the block refs, or discard them? that might be an option for the developer to specify in the method, if they know if the physical array stayed the some or not)

Right now, as a developer, the way you can update the array inplace,

[jorisvandenbossche]

cc @jbrockmendel

[jbrockmendel]

Will give this some more thought, but my initial reaction is skeptical:

Is there any evidence that anyone wants/needs this besides geopandas? (we could revisit this question in a few weeks after RC feedback)

In the OP example the user wants to change the original Series. In most cases I can imagine, the user would also be fine with getting a new Series object new_ser = Series(arr) that should have ser._mgr.array is arr right? What is the scenario in which this simpler/existing API isn't good enough?

Mostly I just feel like this is a complicated solution to a problem with a much easier known solution (reverting the read-only behavior)

[Delengowski]

I can't think of a time where I wanted to reuse the series despite having a completely different array.

If the whole concept of metadata (.attrs) and propagating that around for series and frames ever took off I could see a use case but that never happened.

[rhshadrach]

What is the scenario in which this simpler/existing API isn't good enough?

I imagine something like class MySeries(Series) where one of your methods needs to mutate an attribute on the underlying array. You cannot create a new Series in this case.

Mostly I just feel like this is a complicated solution to a problem with a much easier known solution (reverting the read-only behavior)

Long term I'm good here if we do it across the API (including NumPy-backed objects) and document the danger of mutating the underlying array in the API docs, but am somewhat negative if we only do this for EA backed objects. As discussed on the dev call, I am good on only doing it for EA backed objects for 3.0.

[jorisvandenbossche]

Mostly I just feel like this is a complicated solution to a problem with a much easier known solution (reverting the read-only behavior)

That was my realization from yesterday: reverting the read-only .array is in fact not a full solution, and we would need something like this regardless (and I would also still revert regardless, at least for some time).
From the top post: ".. that would actually still violate the idea of CoW, because this attribute change would propagate to any shallow copy of the series"

So returning the underlying array as-is from Series.array is actually also a problem for the specific use case of geopandas, because then the modification we make to the array gets propagated too much.

I imagine something like class MySeries(Series) where one of your methods needs to mutate an attribute on the underlying array. You cannot create a new Series in this case.

Indeed, that's generally the situation for geopandas. This means that you also have an external ExtensionArray that has an attribute that can be changed (as I think our own EAs nowadays no longer have such attributes? Except maybe for freq which is right now not yet removed from DatetimeArray)

---

Another thought: when we create a shallow copy of a Series/DataFrame, i.e. a shallow copy of a Block, we should maybe always take a view of the array, instead of re-using the exact same array. That would also prevent the unwanted propagation of attribute mutation

[jbrockmendel]

Most important: it definitely seems like you two feel more strongly about this than I do, so I won't push back on this too much even if I remain unenthused.

I imagine something like class MySeries(Series) where one of your methods needs to mutate an attribute on the underlying array.

That makes sense. Though I still think "make .array give the actual array" is a simpler solution.

That was my realization from yesterday: reverting the read-only .array is in fact not a full solution, and we would need something like this regardless

I don't follow. Can you expand on this?

[jorisvandenbossche]

Mutating the attribute will propagate to "copies" of the data. Example with our own freq, using 2.3 to compare with and without CoW (and not have the read-only EA issue)

>>> ser = pd.Series(pd.date_range(2012, periods=3))
>>> ser_derived = ser.reset_index(drop=True)
>>> ser.array.freq
>>> ser.array.freq = "D"
>>> ser.array.freq
<Day>
>>> ser_derived.array.freq   #  is None

>>> pd.options.mode.copy_on_write = True
>>> ser = pd.Series(pd.date_range(2012, periods=3))
>>> ser_derived = ser.reset_index(drop=True)
>>> ser.array.freq
>>> ser.array.freq = "D"
>>> ser.array.freq
<Day>
>>> ser_derived.array.freq  # is also updated
<Day>

As I mentioned, we never explicitly discussed whether attributes fall under the CoW logic, so it is debatable if the above change in behaviour is problematic or not. And it is of course going through the underlying array and not by modifying an attribute on the Series itself.
But updating ser's data here did modify another series, which for updating values is something we avoid by triggering a copy. And in the case of geopandas, the user actually does this through updating the series itself (and the property setter on the series just passes it through to the setter on the array). So for geopandas, we want a version of the above that is "safe" in context of CoW.