The couple currently live in the two-bedroom Nottingham Cottage on the Kensington Palace property, but are looking for a bigger residence to raise their family.
According to the Daily Mail, it's expected the Queen’s cousin - the Duke of Gloucester - and his wife will soon move out of their 21-room flat in the castle so the expectant parents can move in.
However, according to a source, Harry and Meghan don't want to live in the house because it's right next door to the Duke and Duchess of Cambridge.
"Harry and Meghan want to move, and need more space, but they don’t want to live next door to William and Catherine," says the Daily Mail's source.
"They will soon have separate offices and separate lives, so it would make sense for them to live further afield. There is no reason why their London home has to be at Kensington Palace. There are plenty of other options, including using their own money to buy their own place."
It was also recently reported that Harry and Meghan wanted to move out of London altogether ahead of welcoming their first child. According to a nameless “palace source,” the Duke and Duchess of Sussex are considering moving to their $5.4 million home in the Cotswalds, about two hours outside London.