There's a pretty trivial rule for getting this right. Phrase your sentence using who/whom as a question. Respond with he/him. If your response contains a "he", your initial statement should be "who"; if it contains a "him" then you're looking at a "whom" use.

• ex: "To who/whom should the gold be given?" -> "To him" -> "whom"
• ex: "Who/whom wants the gold?" -> "He wants the gold" -> "who"
• ex: "Who/whom did you see at the party?" -> "I saw him" -> "whom"
• ex: "The man who/whom called earlier is here" -> "Who/whom called?" -> "he called" -> "who"