Combine two columns of text in dataframe in pandas/python
我在python中有一个20 x 4000的数据帧,使用pandas。其中两列分别命名为Year和Quarter。我想创建一个名为period的变量,使year=2000和quarter=q2变为2000q2
有人能帮忙吗?
1 | dataframe["period"] = dataframe["Year"].map(str) + dataframe["quarter"] |
1 2 | df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}) df['period'] = df[['Year', 'quarter']].apply(lambda x: ''.join(x), axis=1) |
生成此数据帧
1 2 3 | Year quarter period 0 2014 q1 2014q1 1 2015 q2 2015q2 |
此方法通过用数据帧的任何列切片(如
您可以在此处查看有关apply()方法的更多信息
小数据集(<150行)
1 | [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])] |
或稍慢但更紧凑:
1 | df.Year.str.cat(df.quarter) |
较大的数据集(>150行)
1 | df['Year'].astype(str) + df['quarter'] |
更新:大熊猫0.23.4计时图
让我们在20万行df上进行测试:
1 2 3 4 5 6 7 8 9 10 | In [250]: df Out[250]: Year quarter 0 2014 q1 1 2015 q2 In [251]: df = pd.concat([df] * 10**5) In [252]: df.shape Out[252]: (200000, 2) |
更新:使用熊猫0.19.0的新计时
没有CPU/GPU优化的定时(从最快到最慢排序):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | In [107]: %timeit df['Year'].astype(str) + df['quarter'] 10 loops, best of 3: 131 ms per loop In [106]: %timeit df['Year'].map(str) + df['quarter'] 10 loops, best of 3: 161 ms per loop In [108]: %timeit df.Year.str.cat(df.quarter) 10 loops, best of 3: 189 ms per loop In [109]: %timeit df.loc[:, ['Year','quarter']].astype(str).sum(axis=1) 1 loop, best of 3: 567 ms per loop In [110]: %timeit df[['Year','quarter']].astype(str).sum(axis=1) 1 loop, best of 3: 584 ms per loop In [111]: %timeit df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1) 1 loop, best of 3: 24.7 s per loop |
使用CPU/GPU优化计时:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | In [113]: %timeit df['Year'].astype(str) + df['quarter'] 10 loops, best of 3: 53.3 ms per loop In [114]: %timeit df['Year'].map(str) + df['quarter'] 10 loops, best of 3: 65.5 ms per loop In [115]: %timeit df.Year.str.cat(df.quarter) 10 loops, best of 3: 79.9 ms per loop In [116]: %timeit df.loc[:, ['Year','quarter']].astype(str).sum(axis=1) 1 loop, best of 3: 230 ms per loop In [117]: %timeit df[['Year','quarter']].astype(str).sum(axis=1) 1 loop, best of 3: 230 ms per loop In [118]: %timeit df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1) 1 loop, best of 3: 9.38 s per loop |
回答@anton vbr的贡献
1 2 3 4 5 6 7 8 9 10 11 12 13 | >>> import pandas as pd >>> df = pd.DataFrame([["2014","q1"], ... ["2015","q3"]], ... columns=('Year', 'Quarter')) >>> print(df) Year Quarter 0 2014 q1 1 2015 q3 >>> df['Period'] = df.Year.str.cat(df.Quarter) >>> print(df) Year Quarter Period 0 2014 q1 2014q1 1 2015 q3 2015q3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 | >>> import pandas as pd >>> df = pd.DataFrame([[2014, 1], ... [2015, 3]], ... columns=('Year', 'Quarter')) >>> print(df) Year Quarter 0 2014 1 1 2015 3 >>> df['Period'] = df.Year.astype(str).str.cat(df.Quarter.astype(str), sep='q') >>> print(df) Year Quarter Period 0 2014 1 2014q1 1 2015 3 2015q3 |
连接多个列只是传递序列列表或包含除第一列以外的所有列的数据帧,作为对第一列(序列)调用的
1 2 3 4 5 6 7 8 9 10 | >>> df = pd.DataFrame( ... [['USA', 'Nevada', 'Las Vegas'], ... ['Brazil', 'Pernambuco', 'Recife']], ... columns=['Country', 'State', 'City'], ... ) >>> df['AllTogether'] = df['Country'].str.cat(df[['State', 'City']], sep=' - ') >>> print(df) Country State City AllTogether 0 USA Nevada Las Vegas USA - Nevada - Las Vegas 1 Brazil Pernambuco Recife Brazil - Pernambuco - Recife |
请注意,如果熊猫数据帧/系列的值为空,则需要包含参数na_rep以用字符串替换NaN值,否则组合列将默认为NaN。
这次将lamba函数与string.format()一起使用。
1 2 3 4 5 6 7 8 9 10 11 12 | import pandas as pd df = pd.DataFrame({'Year': ['2014', '2015'], 'Quarter': ['q1', 'q2']}) print df df['YearQuarter'] = df[['Year','Quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1) print df Quarter Year 0 q1 2014 1 q2 2015 Quarter Year YearQuarter 0 q1 2014 2014q1 1 q2 2015 2015q2 |
这允许您使用非字符串,并根据需要重新格式化值。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | import pandas as pd df = pd.DataFrame({'Year': ['2014', '2015'], 'Quarter': [1, 2]}) print df.dtypes print df df['YearQuarter'] = df[['Year','Quarter']].apply(lambda x : '{}q{}'.format(x[0],x[1]), axis=1) print df Quarter int64 Year object dtype: object Quarter Year 0 1 2014 1 2 2015 Quarter Year YearQuarter 0 1 2014 2014q1 1 2 2015 2015q2 |
虽然@silvado的答案很好,但如果您将
1 2 3 4 5 6 7 8 | import pandas as pd df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}) In [131]: %timeit df["Year"].map(str) 10000 loops, best of 3: 132 us per loop In [132]: %timeit df["Year"].astype(str) 10000 loops, best of 3: 82.2 us per loop |
当您的数据插入到数据帧中时,此命令将解决您的问题:
1 | df['period'] = df[['Year', 'quarter']].apply(lambda x: ' '.join(x.astype(str)), axis=1) |
假设您的
1 2 | import pandas as pd df = pd.DataFrame({'Quarter':'q1 q2 q3 q4'.split(), 'Year':'2000'}) |
假设我们想看到数据帧;
1 2 3 4 5 6 | df >>> Quarter Year 0 q1 2000 1 q2 2000 2 q3 2000 3 q4 2000 |
最后,连接
1 | df['Period'] = df['Year'] + ' ' + df['Quarter'] |
现在您可以使用
1 2 3 4 5 6 | df >>> Quarter Year Period 0 q1 2000 2000 q1 1 q2 2000 2000 q2 2 q3 2000 2000 q3 3 q4 2000 2000 q4 |
如果您不想在年和季度之间留出空间,只需执行以下操作即可将其删除;
1 | df['Period'] = df['Year'] + df['Quarter'] |
下面是一个我发现非常通用的实现:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | In [1]: import pandas as pd In [2]: df = pd.DataFrame([[0, 'the', 'quick', 'brown'], ...: [1, 'fox', 'jumps', 'over'], ...: [2, 'the', 'lazy', 'dog']], ...: columns=['c0', 'c1', 'c2', 'c3']) In [3]: def str_join(df, sep, *cols): ...: from functools import reduce ...: return reduce(lambda x, y: x.astype(str).str.cat(y.astype(str), sep=sep), ...: [df[col] for col in cols]) ...: In [4]: df['cat'] = str_join(df, '-', 'c0', 'c1', 'c2', 'c3') In [5]: df Out[5]: c0 c1 c2 c3 cat 0 0 the quick brown 0-the-quick-brown 1 1 fox jumps over 1-fox-jumps-over 2 2 the lazy dog 2-the-lazy-dog |
更有效的是
1 2 3 | def concat_df_str1(df): """ run time: 1.3416s""" return pd.Series([''.join(row.astype(str)) for row in df.values], index=df.index) |
下面是一个时间测试:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | import numpy as np import pandas as pd from time import time def concat_df_str1(df): """ run time: 1.3416s""" return pd.Series([''.join(row.astype(str)) for row in df.values], index=df.index) def concat_df_str2(df): """ run time: 5.2758s""" return df.astype(str).sum(axis=1) def concat_df_str3(df): """ run time: 5.0076s""" df = df.astype(str) return df[0] + df[1] + df[2] + df[3] + df[4] + \ df[5] + df[6] + df[7] + df[8] + df[9] def concat_df_str4(df): """ run time: 7.8624s""" return df.astype(str).apply(lambda x: ''.join(x), axis=1) def main(): df = pd.DataFrame(np.zeros(1000000).reshape(100000, 10)) df = df.astype(int) time1 = time() df_en = concat_df_str4(df) print('run time: %.4fs' % (time() - time1)) print(df_en.head(10)) if __name__ == '__main__': main() |
最后,当使用
使用
1 | df["period"] = [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])] |
Graph:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | import pandas as pd import numpy as np import timeit import matplotlib.pyplot as plt from collections import defaultdict df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}) myfuncs = { "df['Year'].astype(str) + df['quarter']": lambda: df['Year'].astype(str) + df['quarter'], "df['Year'].map(str) + df['quarter']": lambda: df['Year'].map(str) + df['quarter'], "df.Year.str.cat(df.quarter)": lambda: df.Year.str.cat(df.quarter), "df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)": lambda: df.loc[:, ['Year','quarter']].astype(str).sum(axis=1), "df[['Year','quarter']].astype(str).sum(axis=1)": lambda: df[['Year','quarter']].astype(str).sum(axis=1), "df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)": lambda: df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1), "[''.join(i) for i in zip(dataframe['Year'].map(str),dataframe['quarter'])]": lambda: [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])] } d = defaultdict(dict) step = 10 cont = True while cont: lendf = len(df); print(lendf) for k,v in myfuncs.items(): iters = 1 t = 0 while t < 0.2: ts = timeit.repeat(v, number=iters, repeat=3) t = min(ts) iters *= 10 d[k][lendf] = t/iters if t > 2: cont = False df = pd.concat([df]*step) pd.DataFrame(d).plot().legend(loc='upper center', bbox_to_anchor=(0.5, -0.15)) plt.yscale('log'); plt.xscale('log'); plt.ylabel('seconds'); plt.xlabel('df rows') plt.show() |
正如前面提到的,必须将每一列转换为字符串,然后使用加号运算符组合两个字符串列。通过使用numpy,您可以获得很大的性能改进。
1 2 3 4 5 | %timeit df['Year'].values.astype(str) + df.quarter 71.1 ms ± 3.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) %timeit df['Year'].astype(str) + df['quarter'] 565 ms ± 22.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) |
使用
1 | df['Period'] = df['Year'].combine_first(df['Quarter']) |
我认为在panda中组合列的最佳方法是将两列都转换为整数,然后再转换为str。
1 2 | df[['Year', 'quarter']] = df[['Year', 'quarter']].astype(int).astype(str) df['Period']= df['Year'] + 'q' + df['quarter'] |
此解决方案使用中间步骤将数据帧的两列压缩为包含值列表的单列。这不仅适用于字符串,而且适用于所有类型的列数据类型
1 2 3 4 5 | import pandas as pd df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}) df['list']=df[['Year','quarter']].values.tolist() df['period']=df['list'].apply(''.join) print(df) |
结果:
1 2 3 | Year quarter list period 0 2014 q1 [2014, q1] 2014q1 1 2015 q2 [2015, q2] 2015q2 |
可以使用数据帧的赋值方法:
1 2 | df= (pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}). assign(period=lambda x: x.Year+x.quarter )) |
1 | dataframe["period"] = dataframe["Year"].astype(str).add(dataframe["quarter"]) |
或者如果值类似于[2000][4]并希望生成[2000Q4]
1 | dataframe["period"] = dataframe["Year"].astype(str).add('q').add(dataframe["quarter"]).astype(str) |
用
1 2 3 4 5 6 7 8 9 10 11 12 | def madd(x): """Performs element-wise string concatenation with multiple input arrays. Args: x: iterable of np.array. Returns: np.array. """ for i, arr in enumerate(x): if type(arr.item(0)) is not str: x[i] = x[i].astype(str) return reduce(np.core.defchararray.add, x) |
例如:
1 2 3 4 5 6 7 8 9 10 11 | data = list(zip([2000]*4, ['q1', 'q2', 'q3', 'q4'])) df = pd.DataFrame(data=data, columns=['Year', 'quarter']) df['period'] = madd([df[col].values for col in ['Year', 'quarter']]) df Year quarter period 0 2000 q1 2000q1 1 2000 q2 2000q2 2 2000 q3 2000q3 3 2000 q4 2000q4 |