Graph Neural Networks for Segmented Images - Which Nodes do I connect?

I'm facing an interesting problem involving medical images. We are set out to test an hypothesis if certain objects in an image affect the diagnosis of a patient.

I would love to hear any comments regarding my pipeline but this is my current approach:

1. Segment the image in order to obtain the object's regions. This would be done using off-the-shelf resnet and labeled data obtained from the manual annotation of the images in hand.
2. Now, that I have the segmented objects, I can perform some feature extraction, such as intensity, size, shape, etc..

From here I can either try some variable size input DL architecture such as DeepSets or try and get some global feature and run a classic classifier. But, I wish to try using GNN after having some previous success with it. I thought of defining my graph as follows:

• Nodes - The objects I segmented
• Nodes Features - same as above, size, intensity, etc...
• Edges - The biggest problem - should I connect all? We're talking about 20-40 objects per image so its really a small graph
• Edges Features - Euclidian distance in order to capture the spatial characteristics of the objects .

For here on I can feed it into any GNN architecture and try to classify the graph.

Thank you for your comments and ideas.