Merging Multiple DataFrames in Pandas

In this blog, different ways of merging multiple DataFrames in python using Pandas library is discussed. Also, we’ll see joining and concatenating DataFrames in pandas.

Merging in Pandas

  In data analysis and machine learning tasks, the very important process is merging and joining DataFrames. Because, data comes from multiple sources and files, and it is needed to bring all the data together by using a specific type of join to start data analysis. This can be performed using the most popular library in python, Pandas. Pandas provide various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join/merge type operations.

Different Types of Joins in Pandas

Pandas Inner Join

Inner Join returns the things that are in both the left and the right DataFrames. This is exactly like the intersection of two sets.

Pandas Outer Join or Full Outer Join

This will take all the data from the left DataFrame and the right DataFrame and everything that is similar. So, basically it joins everything.

Pandas Left Join

This takes everything from the left and then if there’s anything that’s similar it’ll also include that.

Pandas Right Join

This gives us everything from the right DataFrame and it’s going to give us everything that is similar but it’s not going to give us anything that is just unique to the left DataFrame.

Difference between merge(), join() and concat() in pandas

Merge() is used for combining data on common columns or indices, join() is used for combining data on a key column or an index and, concat() for combining DataFrames across rows or columns.

Syntax :

pandas.DataFrame.merge

Merge DataFrame or named Series objects with a database-style join.

A named Series object is treated as a DataFrame with a single named column.

The join is done on columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. When performing a cross merge, no column specifications to merge on are allowed.

DataFrame.merge(right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=False, suffixes=('_x', '_y'), copy=None, indicator=False, validate=None)

Parameters:

right : DataFrame or named Series

Object to merge with.

how : {‘left’, ‘right’, ‘outer’, ‘inner’, ‘cross’}, default ‘inner’

Type of merge to be performed.

• left: use only keys from left frame, similar to a SQL left outer join; preserve key order.

• right: use only keys from right frame, similar to a SQL right outer join; preserve key order.

• outer: use union of keys from both frames, similar to a SQL full outer join; sort keys lexicographically.

• inner: use intersection of keys from both frames, similar to a SQL inner join; preserve the order of the left keys.

• cross: creates the cartesian product from both frames, preserves the order of the left keys.

on : label or list

Column or index level names to join on. These must be found in both DataFrames. If on is None and not merging on indexes then this defaults to the intersection of the columns in both DataFrames.

left_on : label or list, or array-like

Column or index level names to join on in the left DataFrame. Can also be an array or list of arrays of the length of the left DataFrame. These arrays are treated as if they are columns.

right_on : label or list, or array-like

Column or index level names to join on in the right DataFrame. Can also be an array or list of arrays of the length of the right DataFrame. These arrays are treated as if they are columns.

left_index : bool, default False

Use the index from the left DataFrame as the join key(s). If it is a MultiIndex, the number of keys in the other DataFrame (either the index or a number of columns) must match the number of levels.

right_index : bool, default False

Use the index from the right DataFrame as the join key. Same caveats as left_index.

sort : bool, default False

Sort the join keys lexicographically in the result DataFrame. If False, the order of the join keys depends on the join type (how keyword).

suffixes : list-like, default is (“_x”, “_y”)

A length-2 sequence where each element is optionally a string indicating the suffix to add to overlapping column names in left and right respectively. Pass a value of None instead of a string to indicate that the column name from left or right should be left as-is, with no suffix. At least one of the values must not be None.

copy : bool, default True

If False, avoid copy if possible.

indicator : bool or str, default False

If True, adds a column to the output DataFrame called “_merge” with information on the source of each row. The column can be given a different name by providing a string argument. The column will have a Categorical type with the value of “left_only” for observations whose merge key only appears in the left DataFrame, “right_only” for observations whose merge key only appears in the right DataFrame, and “both” if the observation’s merge key is found in both DataFrames.

validate : str, optional

If specified, checks if merge is of specified type.

• “one_to_one” or “1:1”: check if merge keys are unique in both left and right datasets.

• “one_to_many” or “1:m”: check if merge keys are unique in left dataset.

• “many_to_one” or “m:1”: check if merge keys are unique in right dataset.

• “many_to_many” or “m:m”: allowed, but does not result in checks.

Returns:

DataFrame

A DataFrame of the two merged objects.

pandas.DataFrame.join

DataFrame.join(other, on=None, how='left', lsuffix='', rsuffix='', sort=False, validate=None)

Join columns of another DataFrame.

Join columns with other DataFrame either on index or on a key column. Efficiently join multiple DataFrame objects by index at once by passing a list.

Parameters:

other : DataFrame, Series, or a list containing any combination of them

Index should be similar to one of the columns in this one. If a Series is passed, its name attribute must be set, and that will be used as the column name in the resulting joined DataFrame.

on : str, list of str, or array-like, optional

Column or index level name(s) in the caller to join on the index in other, otherwise joins index-on-index. If multiple values given, the other DataFrame must have a MultiIndex. Can pass an array as the join key if it is not already contained in the calling DataFrame. Like an Excel VLOOKUP operation.

how : {‘left’, ‘right’, ‘outer’, ‘inner’, ‘cross’}, default ‘left’

How to handle the operation of the two objects.

• left: use calling frame’s index (or column if on is specified)

• right: use other’s index.

• outer: form union of calling frame’s index (or column if on is specified) with other’s index, and sort it lexicographically.

• inner: form intersection of calling frame’s index (or column if on is specified) with other’s index, preserving the order of the calling’s one.

• cross: creates the cartesian product from both frames, preserves the order of the left keys.

lsuffix : str, default ‘’

Suffix to use from left frame’s overlapping columns.

rsuffix : str, default ‘’

Suffix to use from right frame’s overlapping columns.

sort : bool, default False

Order result DataFrame lexicographically by the join key. If False, the order of the join key depends on the join type (how keyword).

validate : str, optional

If specified, checks if join is of specified type.

• “one_to_one” or “1:1”: check if join keys are unique in both left and right datasets.

• “one_to_many” or “1:m”: check if join keys are unique in left dataset.

• “many_to_one” or “m:1”: check if join keys are unique in right dataset.

• “many_to_many” or “m:m”: allowed, but does not result in checks.

Returns:

DataFrame

A dataframe containing columns from both the caller and other.

It is important to note that parameters on, lsuffix, and rsuffix are not supported when passing a list of DataFrame objects.

Examples :

We are using Visual Studio code as the Integrated Development Environment where we installed Jupyter notebook.And then, install pandas and import pandas library in Jupyter notebook and follow the directions as follows. Here, we are combining two separate DataFrames together into one DataFrame.

!pip install pandas

import pandas as pd

Initializing a dictionary

dictionary={'Personid': [101,102,103,104],

            'Firstname': ['Smith','Yash','Sam', 'Cathy'],

            'Skills' : ['Swimming', 'Running', 'Gymnastic', 'Soccer']}

Pass the dictionary to Pandas DataFrame and print the first DataFrame

df1=pd.DataFrame(dictionary)

df1

Initializing an another dictionary

dictionary={'Personid': [101,102,106,107,108],

            'Firstname': ['Smith','Yash','Suri', 'Gary','Sony'],

            'Age':['25','30','22','24','20']}

Passing the dictionary to Pandas DataFrame and printing the second DataFrame

df2=pd.DataFrame(dictionary)

df2

Here we can observe first two rows of two DataFrames have same Personid and Firstname. In the second DataFrame we have three new people with Age column which is unique.

Now let’s see how merge works. Merge works just like the Joins, and here we are using merge() function where we will do outer join, inner join, left join, right join of two DataFrames we defined. Along with these we also see join(), set_index(), and concat() function.

Merging two DataFrames

By default, merging takes place between two DataFrames using inner join, so it’s giving us an output where the specific values or the keys are same.

df=pd.merge(df1,df2)

df

To make sure that we did inner join because we didn’t specify anything that was just the default.

Merging DataFrames using inner join

df=pd.merge(df1,df2,how='inner')

df

To show how it is merging DataFrames together use on=[‘Personid’]. So, here since we are joining only on Personid, and you are not joining on Firstname, it gets separated as Firstname_x and Firstname_y even though they have the exact same values, since we are not joining on that column it automatically separates.

df=pd.merge(df1,df2,how='inner', on=['Personid'] )

df

Now we are merging DataFrames based on columns Personid and Firstname, the output is as below.

df=pd.merge(df1,df2,how='inner',on=['Personid', 'Firstname'] )

df

Merging DataFrames using outer join

df=pd.merge(df1,df2,how='outer')

df

Merging DataFrames using left join

df=pd.merge(df1,df2,how='left')

df

Merging DataFrames using right join

df=pd.merge(df1,df2,how='right')

df

Merging DataFrames using cross join

Cross Join is little different, here it takes each value from the left DataFrame and compares it to each value in the right DataFrame

df=pd.merge(df1,df2,how='cross')

df

Joining DataFrames using join() function

When we were using the merge we were using the column names and join() is usually better when we are working with indexes.

Join DataFrames using their indexes

df = pd.DataFrame.join(df1,df2,on='Personid', how='outer',lsuffix='_Left', rsuffix='_Right')

df

Join() function with set_index() function

If we want to join using the key columns, we need to set key to be the index in both DataFrames. The joined DataFrame will have key as its index.

df=df1.set_index('Personid').join(df2.set_index('Personid'),lsuffix='_Left', rsuffix='_Right', how='outer')

df

Concatenating DataFrames using concat() function

Concat() function concatenates the DataFrames on top of the other rather than putting one DataFrame next to one another which is like in the merge() and join() function.

df=pd.concat([df1,df2])

df

Concatenating DataFrames using inner join

df=pd.concat([df1,df2], join='inner')

df

Concatenating DataFrames using outer join

df=pd.concat([df1,df2], join='outer', axis=1)

df

Conclusion

In this blog, we learned to merge and join DataFrames based on several logics using the merge() and join() functions of pandas library. Along the way, we also practiced the different types of joins while merging DataFrames. Towards the end, we also used set_index with join() function and also learned about concatenating DataFrames.