A method for visual media processing includes performing a conversion between a current video unit of a visual media data and a bitstream representation of the current video unit, wherein the bitstream representation is configured according to a format rule; and wherein the format rule specifies a level of a video segment at which one or more syntax elements indicative of a scaling factor by which a chroma residual is scaled for coding the current video unit is selectively included in the bitstream representation.