Open-World Entity Segmentation
Lu Qi*1
Jason Kuen*2
Yi Wang1
Jiuxiang Gu2
Hengshuang Zhao3
Zhe Lin2
Philip Torr3
Jiaya Jia1,4
1The Chinese University of Hong Kong
2Adobe Research
3University of Oxford


We introduce a new image segmentation task, termed Entity Segmentation (ES) with the aim to segment all visual entities in an image without considering semantic category labels. It has many practical applications in image manipulation/editing where the segmentation mask quality is typically crucial but category labels are less important. In this setting, all semantically-meaningful segments are equally treated as categoryless entities and there is no thing-stuff distinction. Based on our unified entity representation, we propose a center-based entity segmentation framework with two novel modules to improve mask quality. Experimentally, both our new task and framework demonstrate superior advantages as against existing work. In particular, ES enables the following: (1) merging multiple datasets to form a large training set without the need to resolve label conflicts; (2) any model trained on one dataset can generalize exceptionally well to other datasets with unseen domains.

Entity Concept

The ``entity'' refers to either a thing (instance) mask or a stuff mask in the common context, which can be any semantically meaningful and coherent region in the open-world setting.


Unlike existing panoptic segmentation methods that generally include two output branches for thing and stuff separately, the proposed framework relies on a dense one-stage detection architecture and a dynamic mask kernel branch to detect and segment all entities (regardless of thing or stuff) in a unified manner. Besides, we propose a global kernel bank module and an overlap suppression module to further improve the model's performance on the task. The global kernel bank module generates mask kernels to exploit some common properties (e.g, textures, edges) shared by many entities and the overlap suppression module encourages the masks not to overlap with each other.


Paper and Supplementary Material

Lu Qi*, Jason Kuen*, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Zhe Lin, Philip Torr, Jiaya Jia.
Open-World Entity Segmentation.
(hosted on ArXiv)



This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.