An answer for inconsistencies in indexing operations in pandas | by Patrick Hoefler | Dec, 2022

Do away with annoying SettingWithCopyWarning messages

Introduction

Indexing operations in pandas are fairly versatile and thus, have many circumstances that may behave fairly totally different and due to this fact produce sudden outcomes. Moreover, it’s laborious to foretell when a SettingWithCopyWarningis raised and what this implies precisely. I’ll present a few totally different situations and the way every operation would possibly affect your code. Afterwards, we are going to take a look at a brand new characteristic known as Copy on Write that lets you do away with the inconsistencies and of SettingWithCopyWarnings. We will even examine how this would possibly affect efficiency and different strategies on the whole. I’m a member of the pandas core group.

Indexing operations

Let’s take a look at how indexing operations at the moment work in pandas. In case you are already conversant in indexing operations, you’ll be able to soar to the subsequent part. However bear in mind, there are numerous circumstances with totally different types of conduct. The precise behavor is difficult to foretell.

An operation in pandas produces a replica, when the underlying information of the guardian DataFrame and the brand new DataFrame aren’t shared. A view is an object that does share information with the guardian object. A modification to the view can probably affect the guardian object.

As of proper now, some indexing operations return copies whereas others return views. The precise conduct is difficult to foretell, even for skilled customers. This has been a giant annoyance for me up to now.

Let’s begin with a DataFrame with two columns:

df = pd.DataFrame({"user_id": [1, 2, 3], "rating": [10, 15, 20]})

A getitem operation on a DataFrame or Sequence returns a subset of the preliminary object. The subset would possibly consist of 1 or a set of columns, one or a set of rows or a combination of each. A setitem operation on a DataFrame or Sequence updates a subset of the preliminary object. The subset itself is outlined by the arguments to the calls.

A daily getitem operation on a DataFrame offers a view most often:

view = df["user_id"]

As a consequence, the brand new object view nonetheless references the guardian object df and its information. Therefore, writing into the view will even modify the guardian object.

view.iloc[0] = 10

This setitem operation will consequently replace not solely our view but in addition df. This occurs as a result of the underlying information are shared between each objects.

That is solely true, if the column user_id happens solely as soon as in df. As quickly as user_id is duplicated the getitem operation returns a DataFrame. This implies the returned object is a replica as an alternative of a view:

df = pd.DataFrame(
[[1, 10, 2], [3, 15, 4]], 
columns=["user_id", "score", "user_id"],
)
not_a_view = df["user_id"]
not_a_view.iloc[0] = 10

The setitem operation doesn’t replace df. We additionally get our first SettingWithCopyWarning, regardless that it is a completely acceptable operation. The getitem operation itself has many extra circumstances, like list-like keys, e.g. df[["user_id"]], MultiIndex-columns and lots of extra. I’ll go into extra element in observe up posts to have a look at totally different types of performing indexing operations and their conduct.

Let’s take a look at one other case that is a little more difficult than a single getitem operation: chained indexing. Chained indexing means filtering with a boolean masks adopted by a getitem operation or the opposite manner round. That is accomplished in a single step. We don’t create a brand new variable to retailer the results of the primary operation.

We once more begin with an everyday DataFrame:

df = pd.DataFrame({"user_id": [1, 2, 3], "rating": [10, 15, 20]})

We will replace all user_ids which have a rating better than 15 via:

df["user_id"][df["score"] > 15] = 5

We take the column user_id and apply the filter afterwards. This works completely positive, as a result of the column choice creates a view and the setitem operation updates mentioned view. We will change each operations as effectively:

df[df["score"] > 15]["user_id"] = 5

This execution order produces one other SettingWithCopyWarning. In distinction to our earlier instance, nothing occurs. The DataFrame df just isn’t modified. This can be a silent no-operation. The boolean masks all the time creates a replica of the preliminary DataFrame. Therefore, the preliminary getitem operation returns a replica. The return worth just isn’t assigned to any variable and is barely a brief end result. The setitem operation updates this non permanent copy. Because of this, the modification is misplaced. The truth that masks return copies whereas column choices return views is an implementation element. Ideally, such implementation particulars shouldn’t be seen.

One other method of doing that is as follows:

new_df = df[df["score"] > 5]
new_df["user_id"] = 10

This operation updates new_df as supposed however reveals a SettingWithCopyWarning anyway, as a result of we cannot replace df. Most of us most likely by no means wish to replace the preliminary object (e.g. df) on this situation however we get the warning anyway. In my expertise this results in pointless copy statements scattered over the code base.

That is only a small pattern of present inconsistencies and annoyances in indexing operations.

For the reason that precise conduct is difficult to foretell, this forces many defensive copies in different strategies. For instance,

• dropping of columns
• setting a brand new index
• resetting the index
• …

All copy the underlying information. These copies aren’t crucial from an implementation perspective. The strategies might return views fairly simply, however returning views would result in unpredictable conduct in a while. Theoretically, one setitem operation might propagate via the entire call-chain, updating many DataFrames without delay.

Copy on Write

Let’s take a look at how a brand new characteristic known as “Copy on Write” (CoW) helps us to do away with these inconsistencies in our code base. CoW signifies that any DataFrame or Sequence derived from one other in any manner all the time behaves as a replica. As a consequence, we will solely change the values of an object via modifying the article itself. CoW disallows updating a DataFrame or a Sequence that shares information with one other DataFrame or Sequence object inplace. With this info, we will once more take a look at our preliminary instance:

df = pd.DataFrame({"user_id": [1, 2, 3], "rating": [10, 15, 20]})
view = df["user_id"]
view.iloc[0] = 10

The getitem operation offers a view onto df and it’s information. The setitem operation triggers a replica of the underlying information earlier than 10 is written into the primary row. Therefore, the operation will not modify df. A bonus of this conduct is, that we don’t have to fret about user_id being probably duplicated or utilizing df[["user_id"]] as an alternative of df["user_id"]. All these circumstances behave precisely the identical and no annoying warning is proven.

Triggering a replica earlier than updating the values of the article has efficiency implications. It will most actually trigger a small slowdown for some operations. On the opposite aspect, plenty of different operations can keep away from defensive copies and thus enhance efficiency tremendously. The next operations can all return views with CoW:

• dropping columns
• setting a brand new index
• resetting the index
• and lots of extra.

Let’s contemplate the next DataFrame:

na = np.array(np.random.rand(1_000_000, 100))
cols = [f"col_{i}" for i in range(100)]
df = pd.DataFrame(na, columns=cols)

Utilizing add_prefix provides the given string (e.g. take a look at) to the start of each column title:

df.add_prefix("take a look at")

With out CoW, this may copy the info internally. This isn’t crucial when trying solely on the operation. However since returning a view can have uncomfortable side effects, the tactic returns a replica. As a consequence, the operation itself is fairly gradual:

482 ms ± 3.43 ms per loop (imply ± std. dev. of seven runs, 1 loop every)

This takes fairly lengthy. We virtually solely modify 100 string literals with out touching the info in any respect. Returning a view offers a big speedup on this situation:

46.4 µs ± 1.04 µs per loop (imply ± std. dev. of seven runs, 10,000 loops every)

The identical operation runs a number of orders of magnitude quicker. Extra importantly, the operating time of add_prefix is fixed when utilizing CoW and doesn’t rely upon the scale of your DataFrame. This operation was run on the principle department of pandas.

The copy is barely crucial, if two totally different objects share the identical underlying information. Within the instance above, view and df each reference the identical information. If the info is unique to at least one DataFrame object, no copy is required, we will proceed to change the info inplace:

df = pd.DataFrame({"user_id": [1, 2, 3], "rating": [10, 15, 20]})
df.iloc[0] = 10

On this case the setitem operation will proceed to function inplace with out triggering a replica.

As a consequence, all of the totally different situations that we’ve seen initially have precisely the identical conduct now. We don’t have to fret about refined inconsistencies anymore.

One other case that at the moment has unusual and laborious to foretell conduct is chained indexing. Chained indexing underneath CoW will by no means work. This can be a direct consequence of the CoW mechanism. The preliminary collection of columns would possibly return a view, however a replica is triggered once we carry out the next setitem operation. Luckily, we will simply modify our code to keep away from chained indexing:

df["user_id"][df["score"] > 5] = 10

We will use loc to do each operations without delay:

df.loc[df["user_id"] > 5, "rating"] = 10

Summarizing, each object that we create behaves like a replica of the guardian object. We cannot by accident replace an object aside from the one we’re at the moment working with.

How one can attempt it out

You possibly can attempt the CoW characteristic since pandas 1.5.0. Improvement remains to be ongoing, however the basic mechanism works already.

You possibly can both set the CoW flag globally via on of the next statements:

pd.set_option("mode.copy_on_write", True)
pd.choices.mode.copy_on_write = True

Alternatively, you’ll be able to allow CoW regionally with:

with pd.option_context("mode.copy_on_write", True):
...

Conclusion

We’ve seen that indexing operations in pandas have many edge circumstances and refined variations in conduct which can be laborious to foretell. CoW is a brand new characteristic geared toward addressing these variations. It could actually probably affect efficiency positively or negatively based mostly on what we are attempting to do with our information. The total proposal for CoW will be discovered right here.

Thanks for studying. Be happy to succeed in out within the feedback to share your ideas and suggestions on indexing and Copy on Write. I’ll write observe up posts targeted on this subject and pandas on the whole. Comply with me on Medium for those who prefer to learn extra about pandas.