I would more trust apple with that information over say Facebook and Google but even still that information is hardly private to begin with. Your address is publicly available, if someone wants to know where you work all they have to do is follow you, and if you go to home depot every Saturday at 1pm it isn't hard to find that out. The effort to get that data on your phone is harder than the above. I have more concerns over someone getting my bank account or debit card info than anything from a frequent location standpoint which is why I only use a credit card or cash for purchases. My card is also more secure in Apple wallet than in my physical wallet though stealing my credit card isn't a concern of mine either as I'm not on the hook for fraudulent charges.The convenience of not carrying actual credit cards is significant to me, so I actually do use Apple Pay. I think the risk of a physical credit card is higher because the full number, expiration date and CVV codes can all be seen by eyes and cameras.
In any case, it’s only money, so to speak.
But I’m more concerned about my whereabouts being tracked in real time. My overactive imagination wonders if that information falling into the wrong hands could open a set of risks that cannot be reversed.