MemoryError 来自熔化或与大数据的连接

问：

当我尝试运行时出现错误。
我检查了这篇文章并尝试修改代码，但仍然出现错误。（链接pd.melt())

这是我的原始代码：

melted = pd.melt(df, ['ID', 'Col2', 'Col3', 'Year'], var_name='New_Var', value_name='Value').sort_values('ID')

修改后：

pivot_list = list()
chunk_size = 100000
for i in range(0, len(df), chunk_size):
    row_pivot = pd.melt(df.iloc[i:i+chunk_size], ['ID', 'Col2', 'Col3', 'Year'], var_name='New_Var', value_name='Value')
    pivot_list.append(row_pivot)
melted = pd.concat(pivot_list).sort_values('ID')

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File /path/envs/myenvs/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/path/envs/myenvs/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/path/Current_Proj/Main_Dir/Python_Program.py", line 122, in My_Function
    melted = pd.concat(pivot_list).sort_values('ID')
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 307, in concat
    return op.get_result()
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 532, in get_result
    new_data = concatenate_managers(
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 222, in concatenate_managers
    values = _concatenate_join_units(join_units, concat_axis, copy=copy)
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 486, in _concatenate_join_units
    to_concat = [
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 487, in <listcomp>
    ju.get_reindexed_values(empty_dtype=empty_dtype, upcasted_na=upcasted_na)
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 466, in get_reindexed_values
    values = algos.take_nd(values, indexer, axis=ax)
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/array_algos/take.py", line 108, in take_nd
    return _take_nd_ndarray(arr, indexer, axis, fill_value, allow_fill)
  File "/path/envs/myenvs/lib/python3.9/site-packages/pandas/core/array_algos/take.py", line 149, in _take_nd_ndarray
    out = np.empty(out_shape, dtype=dtype)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 27.1 GiB for an array with shape (2, 1819900000) and data type object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File /path/Current_Proj/Main_Dir/Python_Program.py", line 222, in <module>
    result = pool.starmap(My_Function, zip(arg1, arg2, arg3))
  File "/path/envs/myenvs/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/path/envs/myenvs/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
numpy.core._exceptions.MemoryError: Unable to allocate 27.1 GiB for an array with shape (2, 1819900000) and data type object

我认为主要问题来自和部分。
任何要处理的想法都应该心存感激。melt()concat()