If you're unfamiliar with PostScript or basic linear algebra, I would start by reading a book or tutorial on PostScript. Not to know how to produce PostScript output, but rather to understand how it handles coordinate transformations. So your drawing program can have nested objects with a transformation for each object, and the transformations compose through the nesting levels. Then you can rotate and scale complicated figures by just changing a single transformation, which is a 2x3 matrix. It's really 3x3 but in "homogenous coordinates" so you only have to store 6 of the numbers.

Obviously, also play with other drawing programs to get ideas for what you want yours to do.

There are also tons of books on computer graphics, though maybe not specifically about drawing programs. The ones I've looked at are way old by now though.