Dual Representation Learning for Out-of-distribution Detection

Download PDF

Zhilin Zhao, Longbing Cao

Published: 21 Aug 2023, Last Modified: 17 Sept 2024Accepted by TMLREveryoneRevisionsBibTeXCC BY 4.0
Abstract: To classify in-distribution samples, deep neural networks explore strongly label-related information and discard weakly label-related information according to the information bottleneck. Out-of-distribution samples drawn from distributions differing from that of in-distribution samples could be assigned with unexpected high-confidence predictions because they could obtain minimum strongly label-related information. To distinguish in- and out-of-distribution samples, Dual Representation Learning (DRL) makes out-of-distribution samples harder to have high-confidence predictions by exploring both strongly and weakly label-related information from in-distribution samples. For a pretrained network exploring strongly label-related information to learn label-discriminative representations, DRL trains its auxiliary network exploring the remaining weakly label-related information to learn distribution-discriminative representations. Specifically, for a label-discriminative representation, DRL constructs its complementary distribution-discriminative representation by integrating diverse representations less similar to the label-discriminative representation. Accordingly, DRL combines label- and distribution-discriminative representations to detect out-of-distribution samples. Experiments show that DRL outperforms the state-of-the-art methods for out-of-distribution detection.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission:

In this revision, we have made significant changes to address all comments and substantially enhance the paper readership. In short, the main changes are summarized below. We have:

  1. provided deeper analyses for the experimental results;
  2. discussed the advantages and limitations of the information bottleneck principle;
  3. run a set of ablation study to verify that the label- and distribution-discriminative representations are complementary;
  4. provided more explanations for the equations;
  5. considered more state-of-the-art compared methods in literature review and experiments;
  6. evaluated performance on near and far OOD datasets;
  7. presented the visualization results on Mini-ImageNet; and
  8. arranged a thorough proofreading of the paper.
Assigned Action Editor: Matthew Blaschko
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 1114