How can I solve it ,TypeError: cannot pickle 'dict_keys' object?

Question

How can I solve it ,TypeError: cannot pickle 'dict_keys' object?

Tariq Hussain

2021年8月8日 17:19

My code:

class spatial_dataset(Dataset):  
    def __init__(self, dic, root_dir, mode, transform=None):


    self.keys = dic.keys()
    self.values=dic.values()
    self.root_dir = root_dir
    self.mode =mode
    self.transform = transform

def __len__(self):
    return len(self.keys)

def load_ucf_image(self,video_name, index):
        path = self.root_dir + '/' + video_name.split('_')[0] + '/' + 'v_' + video_name + '/'

    print(path+'Image'+str(index)+'.jpg')
    img = Image.open(path+'Image'+str(index)+'.jpg')
    transformed_img = self.transform(img)
    img.close()

    return transformed_img

def __getitem__(self, idx):
    print(idx)
    if self.mode == 'train':
        video_name, nb_clips = list(self.keys)[idx].split(' ')
        print(video_name, nb_clips)
        num_clip = int(nb_clips)
        print(num_clip)
        clips = []
        clips.append(random.randint(1, int(num_clip/3)))
        clips.append(random.randint(int(num_clip/3), int(num_clip*2/3)))
        clips.append(random.randint(int(num_clip*2/10), num_clip+1))
        # for i in range(10):
        #     clips.append(random.randrange(1, num_clip, 1))
        print(clips)
        

    label = list(self.values)[idx]
    label = int(label)-1

    if self.mode=='train':
        data =[]
        for i in range(len(clips)):
            key = 'img'+str(i)
            index = clips[i]
            data.append(self.load_ucf_image(video_name, index))
        data = np.array(data)
        label = np.array(label)
        sample = (data, label)

    return sample

dic.training is dictionary, which have; {'videoname_1 framecount_1': label1,'videoname_2 framecount_2' : label2, .....}

training_set = spatial_dataset(dic=dic_training, root_dir=data_path, mode='train', transform = transforms.Compose([
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])
                ]))
        print('== Training data :',len(training_set),'frames')
        print(training_set: ,training_set[1][0])

Until above, it is running but when I use this below code it shows me error (mentioned below)

train_loader = DataLoader(
            dataset=training_set,
            batch_size=BATCH_SIZE,
            shuffle=True,
            num_workers=num_workers)

    for z,i in train_loader:
        print(i)

The error it is showing me:

It is something because I am using dictionary to store values, I then converted it into the numpy arrays and list but still it is showing me this error I do not know where it is going wrong.

Topic pickle pytorch python-3.x python

Category Data Science

Miguel Raevenswood · Accepted Answer · 2021年8月8日 17:19

In your spatial_dataset class, dict.keys() is called to get the keys. This is known to cause pickling errors such as the one you are experiencing. The link above shows that you can handle this in three different ways:

Iterate over the dictionary directly
Use in for containment
Convert to the type you want via iterator

If this does not work for you, you can use the following link, where in another Stack answer there is a brief discussion on tracing this pickling error using the "dill" python package.

I suspect that this might also happen to your call dic.values() call as well, so I would see about changing that if all else fails. Hope that helps.

How can I solve it ,TypeError: cannot pickle 'dict_keys' object?

About