如何从list-python的大型列表中创建Numpy数组

我有一个包含1,200行和500,000列的列表.如何将其转换为numpy数组?

我已经阅读了Bypass “Array is too big” python error的解决方案,但他们没有帮助.

我试着把它们变成一个numpy数组:

import random
import numpy as np
lol = [[random.uniform(0,1) for j in range(500000)] for i in range(1200)]
np.array(lol)

[错误]:

ValueError: array is too big.

然后我尝试过熊猫:

import random
import pandas as pd
lol = [[random.uniform(0,1) for j in range(500000)] for i in range(1200)]
pd.lib.to_object_array(lol).astype(float)

[错误]:

ValueError: array is too big.

我也试过hdf5,因为@askewchan建议:

import h5py
filearray = h5py.File('project.data','w')
data = filearray.create_dataset('tocluster',(len(data),len(data[0])),dtype='f')
data[...] = data

[错误]:

    data[...] = data
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/dataset.py", line 367, in __setitem__
    val = numpy.asarray(val, order='C')
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py", line 460, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/dataset.py", line 455, in __array__
    arr = numpy.empty(self.shape, dtype=self.dtype if dtype is None else dtype)
ValueError: array is too big.

这篇文章显示我可以在磁盘Python: how to store a numpy multidimensional array in PyTables?中存储一个巨大的numpy数组.但我甚至无法将我的列表列表变成一个numpy数组=(

最佳答案
问题似乎是你正在使用只有32位的东西(OS或python),这是大小限制的来源.解决方案是升级到64位.

转载注明原文:如何从list-python的大型列表中创建Numpy数组 - 代码日志