Some of this is stage dependent and all of it is highly dependent on the team above the product manager. The simplest answer of course is to find PM's from companies who have had exemplary success where the Product Manager candidate either led prior success or was exposed to it in a meaningful way.

A simple starting point is to ask them to give you examples of conflicting opinions on a feature and how they evaluated the conflicting opinions and made a decision and tracked the success or failure of that decision.

AirBnb actually gives PM's homework as part of the interview process where they have to actually present a unique idea (from scratch) to the interviewing team.

Happy to talk to you about best process based on your stage and existing team.

I'm a firm believer in hiring a consultant to teach you what is required from a specific role within your organization. Instead of hiring untested talent or blindly interviewing people, come on here and find someone with a product manager background at a large successful company. Hire them to help you define the role for your company. Only then will you know what you need and how to lead them.

The best product managers can pick a product/feature they led and drill down to the various product angles and accurately articulate:
A. What was the process to define the requirements
B. What was the process to productize the requirements
By zooming in to the fine details and zooming out to the strategy and overall execution you should be able to evaluate a candidate.

When evaluate a PM I usually look for:
- Working in uncertainty
- Clarity of thought
- Data driven/informed vs. Guessing
- Be able to see the entire picture while zooming in to the fine details
- Technical Skills
- Leadership

The list above varies per company as in each industry and in each stage of a company the priorities are different.

Two great posts about product management:
How to hire a product manager by Ken Norton
https://www.kennethnorton.com/essays/productmanager.html
Good Product Manager/Bad Product Manager by Ben Horowitz (appears with more details on his book "The hard thing about hard things")
http://web.stanford.edu/class/e140/e140a/handouts/ProductMgmt.txt

The best way to evaluate any candidate is Interviewing method. Interviewing a product management candidate (and hiring product managers in general) is more art than science. There are not many quantifiable indicators of future success, as a product manager’s “soft skills” are often way more important than their technical chops. You must look for the following competencies in a product manager candidate.
1. Can the product management candidate write?
While in-person presence and verbal communication skills are essential, product managers constantly express their ideas, thoughts, and needs via the written word. Whether it is email, Slack messages, bug reports, user stories, or more complex documents, product managers are constantly using text to get people on the same page. If a product manager cannot write, the risks run from simply looking unprofessional to providing less than precise or downright misleading direction to other teams. This could result in wasted development cycles, unhappy customers, or sales and marketing using incorrect information. When interviewing product managements candidate, make them author something as part of the hiring process. While you could ask for sample of their previous work, most product managers with any integrity won’t be able to hand over examples from their current or previous employers, which is why you should give them an assignment that involves synthesizing information and then communicating it clearly to a specific audience. These shouldn’t be term papers; ask them to write about something where the source information they need is readily available and based on concepts and subject matter they’re familiar with—you can even let them pick the topic. There are plenty of other chances to test their subject matter expertise; this is an opportunity to see if they can follow instructions, communicate clearly, are conscientious enough check their work and catch errors, and whether they’ll just do the bare minimum or go the extra mile.
2. How well does the product management candidate present?
No matter how good their ideas may be, a product manager must be able to sell their vision and use data to back it up. This often involves standing in front of a room of doubters and convincing them with a presentation. To make sure your candidate does not get stage fright and can win over a crowd, ask them to present to a larger group as part of the interview process. It does not have to be a long presentation, but it should contain some original thought and maybe a few slides to make sure they can sling some PowerPoint when they need to. Ask them to prepare a presentation for the second interview so they have time to feel comfortable with the material. It should not be purely informational and should require them to have reached a conclusion and make a compelling argument.
3. Can the candidate hold their own with the techies?
While there is plenty of debate about whether a product manager needs a technical background, there is no question they will be interacting with technical people on a regular basis. Given they have a well-earned reputation for being a bit sceptical of new hires and non-engineering types, it is worthwhile to have them spend a bit of time conversing with a couple of developers. This is NOT a time for the technical team to grill the product management candidate and trap them with tricky questions. But rather, an opportunity to see if the candidate can follow along when developers are discussing technical challenges or limitations. Both sides should walk away feeling that they can have a productive and non-confrontational relationship with the other party. Ask your technical team rep to explain how some part of the product works and see if the candidate not only follows but asks clarifying questions without trying to assert themselves too much.
4. Can the product manager do math?
While most product management roles do not involve quantum theory or calculus, there is some math involved when it comes to looking at important metrics such as growth and profit margins. Plus, there is all those experiments and A/B tests that will need to have their results calculated. While interviewing product management candidates, asking them to “show their work”—even if they are using a spreadsheet to do the actual calculations—is worthwhile. This doesn’t have to take up too much time during the interviewing process but asking them to quantify a particular scenario is a reasonable ask. For example, give them a scenario where a cost increases and then ask them to calculate the impact on profit given static revenue. For bonus points you can ask them to figure out how many additional users/purchases your company would need to return to your previous profitability level.
5. How well does the product management candidate communicate with customers?
While you certainly are not going to ask a job candidate to talk to actual customers during their interview, you should still try to get a sense of how they will fare in those scenarios. Will they ask appropriate follow-up questions? Do they lead the customer or listen and react? Are they empathetic when a customer complaint, or are they dismissive? This can be accomplished with some simple role playing, but it can offer a glimpse of their aptitude for this essential product management task.
6. Can the candidate talk to senior management?
A good product manager will be eager to speak directly to senior management and not just rely on a superior to convey their product vision upward. But senior management can be a demanding audience, particularly since their interests and motivations vary based on their own role and inherent biases. The best way to accomplish this during the interview process is to let final candidates interview with a C-level employee. This should be one of the last steps in the process because those folks probably have better things to do with their time and you certainly don’t want your judgment to be called into question by putting a sub-par candidate in a corner office interview before you’ve had a chance to vet them yourself. You’ll get an honest take from most executives, since they don’t want anything other than the best getting a paycheck but follow up with them in person (and quickly) before their impressions fade.
7. Can the product manager talk business?
While an economics degree or MBA likely isn’t required for most product management roles, candidates should understand the basic concepts and be able to make decisions with this big picture in mind vs. a myopic view of the world based only on the product they manage. One great indicator of this is whether they did their homework before the interview and have a basic understanding of your company, the business model, competitors, and overall industry/market dynamics. And not only can they regurgitate facts they gathered from your website, but also, they have meaningful questions based on that information regarding your business strategy, customer base, and growth.
8. Have they done it before?
Product management candidates should not be ruled out just because they have not done every single bullet point on your job description before. If they have, why would they want the job? Instead, you want someone who has relevant experience combined with growth potential to learn and master the additional things you will need them to do. Unless you are making a true entry-level hire, you will want your candidate to have some experience. So, it’s important to look at their resume—along with your expectations for the role and probe them for examples of the things where they claim to have experience. The important thing here is specificity they should be able to tell the full story.
9. Can the candidate problem solve?
This is an opportunity to both delve into prior experiences as well as theoretical. For the latter, it is not about the actual solution they come up with but asking them to walk you through their decision process and noting what types of clarifying questions (if any) they ask. This is essential when interviewing product management candidates. In addition to asking about past experiences, you also want to see if they are quick (enough) on their feet. Curveballs will come at them, crises will arise, and they will eventually be put on the spot. How they react and handle those situations can be the difference between a rock star and mediocrity. So, throw them some wild card questions and ask for their gut reaction.
10. Are they data driven?
Managing products and devising a strategy using gut instincts, hunches and anecdotes does not cut it in today’s fast-paced environment. Making decisions and prioritizing should be based on something real and ideally quantifiable. Ask the candidate how they used metrics in the past, which metrics they think they would want in this new role and have them share some examples of where they have used data to overcome faulty assumptions. You can also see how often during the entire process they reference data gathering and analysis to determine if it is something truly ingrained or just something, they can handwave about in a pinch.
11. How well do they listen?
Finally, and perhaps most importantly, a product manager’s listening skills might be more critical than any others. Their ability to succeed is predicated on asking questions and truly hearing what others are saying. This applies whether they are talking to a customer, colleague or superior. The best listeners do not interrupt, do not “lead the witness,” and give positive reinforcement that demonstrates they really hear what is being said. Not only do they display “active listener” tactics, but they are able to incorporate what they have heard into their thinking and subsequent questions and statements. If they think they’re the most interesting person in the room and have all the answers, they’re not likely to pick up on the subtle clues and insights others have to offer, which is where the true nuggets of wisdom that drive innovation are found.
Besides if you do have any questions give me a call: https://clarity.fm/joy-brotonath