The Mapping Rules for Collections in the MXBean spec say:

-------------------------------------8<----------------------------------
  Mappings for collections (List<E> etc)

...

The conversion of a SortedSet instance will fail with an IllegalArgumentException if it has a non-null comparator().
-------------------------------------8<----------------------------------

However having an MXBean with the attribute of a custom type like this:

    // The getter for the item "nonConvertibleSortedSet"
    // returns a SortedSet with a non-null comparator()
    public static class NonConvertibleType07 {
        public static NonConvertibleType07 from(CompositeData cd) {
            return new NonConvertibleType07();
        }
        
        public SortedSet<String> getNonConvertibleSortedSet() {
            TreeSet<String> treeSet =
                                    new TreeSet<String>(Collator.getInstance());
            treeSet.add("value");
            return treeSet;
        }
    }

and attempting to get this non-convertible item leads to the exception with no sights of the IllegalArgumentException in its chain:

-------------------------------------8<----------------------------------
javax.management.MBeanException: interface proto.negative_tests.SortedSetMapException$TestedClassMXBean.getNonConvertibleType07: cannot convert return value to open value: javax.management.openmbean.OpenDataException: Error calling getter for nonConvertibleSortedSet: javax.management.openmbean.OpenDataException: Cannot convert SortedSet with non-null comparator: java.text.RuleBasedCollator@e4fa487e
        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:174)
        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
        at proto.negative_tests.SortedSetMapException.m1(SortedSetMapException.java:32)
        at proto.negative_tests.SortedSetMapException.main(SortedSetMapException.java:23)
Caused by: javax.management.openmbean.OpenDataException: Error calling getter for nonConvertibleSortedSet: javax.management.openmbean.OpenDataException: Cannot convert SortedSet with non-null comparator: java.text.RuleBasedCollator@e4fa487e
        at com.sun.jmx.mbeanserver.OpenConverter.openDataException(OpenConverter.java:1370)
        at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.toNonNullOpenValue(OpenConverter.java:813)
        at com.sun.jmx.mbeanserver.OpenConverter.toOpenValue(OpenConverter.java:128)
        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:169)
        ... 9 more
Caused by: javax.management.openmbean.OpenDataException: Cannot convert SortedSet with non-null comparator: java.text.RuleBasedCollator@e4fa487e
        at com.sun.jmx.mbeanserver.OpenConverter$CollectionConverter.toNonNullOpenValue(OpenConverter.java:620)
        at com.sun.jmx.mbeanserver.OpenConverter.toOpenValue(OpenConverter.java:128)
        at com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter.toNonNullOpenValue(OpenConverter.java:811)
        ... 11 more
BUILD SUCCESSFUL (total time: 3 seconds)
-------------------------------------8<----------------------------------
(the java-source attached)

The same thing is about the SortedMap.