Setting-type joint compound hardens by a chemical reaction and will set up even under water, so obviously if it settles in your drain you're in for a rock-hard clog. Sand is another building product that is notoriously bad for drains. Sand is heavy and settles quickly in traps and pipes to form clogs that are difficult to remove.
To avoid these problems, never rinse setting compound, grout, mortar mix or concrete down the drain. Instead, save rinse water in a container until the solid materials settle out. Then decant the water in an out-of-the-way area in your yard, and dump the remaining sludge into your garbage can. Need to remove a concrete stain? We have you covered.