From what I've read, the Germans did consider invading, and especially did consider attacking France by going through the Swiss plateau, but it wasn't necessary in the end.

In the end, I think it was useful to have a neutral trading partner in Switzerland. I also don't think Switzerland would have been all that hard to conquer in the end. Switzerland imported (and still imports) most of their food, so they could have just starved them out if they weren't in a rush. An armed invasion might have been slowed by all the mountainous terrain and the tunnels and bridges which the Swiss defenders could destroy. But, considering they had the country surrounded, they could have just starved them out.