go.
it sounds like perhaps he has some shit to resolve, or maybe he simply misses the good times he and his ex shared.
or both.
sometimes people recall the good times more strongly than the bad times in a past relationship. or maybe he misses this ex and all the aspects of their relationship.
what bothers me about this is that you had to press him to get an honest answer. it would have been much healthier a response for him to say, "yes, I miss the good times we had and sometimes I still think about those times". that is ok, and it is to be expected. relationships can mean a lot, and people are allowed to miss the positives attached to a past relationship.
it sounds to me like he felt shame or guilt around discussing that with you. that is to me a red flag. if he isn't comfortable being honest with you about where he's currently at, that's not behavior you need to (or should) tolerate, and you deserve honesty and openness in any relationship.
you said your relationship with him is currently casual. my advice? either keep it casual if you can, or walk away. but either way, you need to set boundaries and make him aware honesty and open communication are necessary.
so I vote 'go'. not necessarily cut him out and never speak again, but definitely go in the sense that you probably need to intentionally distance yourself, and that could end up looking any number of ways. maybe you continue seeing each other casually, maybe you step back to a more platonic setting. but it does sound like he needs to process his feelings if he can't be upfront in the truth of them.
