If you aren't saying it back, he likely suspects something is wrong, but likely not what. And what he imagines is likely much worse.

The truth I've seen is this: the definition of love is different to each person. Even your mother may have defined her relationship to you as one of love. Would that make her crimes any less? Don't get caught up in semantics; say it if you want to, don't if you prefer not. Define it however you wish, and then be true to that.