Solve math questions with an LLM: tools and chains

How would you build a system that answers math questions such as "what is 4 + 5?" or "what is the sum of 1 to 100?" with an LLM? Discuss when to let the model answer directly, when to have it call a calculator or write and run code, how to chain the steps (for example with LangChain), and how you would check the answers are right.