Is it possible to pass in an empty annotation to signify just a background/negative image for faster RCNN?

I'm using a pretrained resnet50 for faster RCNN to detect areas with 2 classes (background and interest class). As part of my data inputs for training, I have background images without any annotation boxes. I've tried setting the box coordinates to all zeros and giving it an area of 0 also. When I go to train, it drops me out with the following error:

All bounding boxes should have positive height and width. Found an invalid box [0.0,0.0,0.0,0.0] for target

        # Handle empty bounding boxes
    if (len(pos) ==0): #
        boxes = torch.zeros((1, 4), dtype=torch.double) #1,4 to be same size as non-empty boxes
        area = 0 #
        area = torch.as_tensor(area, dtype=torch.double)
        #label = torch.as_tensor(0, dtype=torch.int64) #put as background if no boxes
        label = torch.zeros((0,), dtype=torch.int64)
        iscrowd=torch.zeros((0,), dtype=torch.int64)

    else:
        boxes = torch.as_tensor(boxes, dtype=torch.double) 
        print('filled box size is ',np.shape(boxes))
        area = torch.as_tensor(area, dtype=torch.double)
        #label = torch.as_tensor(1, dtype=torch.int64)
        label = torch.ones((1,), dtype=torch.int64)
        iscrowd=torch.zeros((0,), dtype=torch.int64)