Say, for example, I have a goal to do a multi-task learning workflow where I have a network that predicts both foreground/background segmentation, specific object segmentation, and bounding box regression.

Suppose that there are only labels for 1-2 of the tasks per image (i.e. one input image has labels for bounding boxes and fg/bg segmentation, but no specific object segmentation; one input image has specific objects and bounding boxes, but no fg/bg). Also, suppose that there isn’t a lot of training data to use, so I would like to utilize all of the training data even without full label coverage (especially because the feature extraction portions of the network probably benefit from mutual information).

Is there an area of research or some best practices to train an entire network end-to-end in this regard? Something involving finetuning/turning training ‘off’ for certain branches of the network related to the parts that don’t have labels for that input image?