Your experience mirrors my own: it was not the near-instanteneous unconsciousness I have heard referenced, but more like what you're describing.
I followed what I believe to be standard instructions for filling the bag: wear it as you would a shower cap, with the elastic below your ears and just above your eyebrows; crumple it to squeeze out all the air; inflate it with N2. Once the bag is inflated with N2, exhale as much air as possible from your lungs, pull the bag down over your face, and inhale deeply.
The times I tried it, I was experiencing extreme anxiety and unable to breathe as deeply as I ideally should have. My breathing was very shallow and rapid, which meant the air in my lungs wasn't being exchanged for N2 as quickly as it should have been, and despite inhaling pure N2 the residual air in my lungs was still providing O2. I did manage that first deep exhalation/deep breath, but it still took far longer than I expected for my vision to go grey. Unfortunately, it was always long enough for SI to kick in.
As @Joannf says, CO2 can be a problem, but if you weren't experiencing hypercapnic alarm --that feeling of suffocation and panic-- (and it sounds like you weren't) you were already following a good protocol and your N2 flow rate was adequate. You shouldn't need to exhale outside the bag; that's why the 15 LPM and the elastic neckband are so important: that's what's required to flush away the CO2.
It sounds like you're doing everything right, but the residual air in your lungs isn't being cycled out fast enough for quick unconsciousness. Remember that the air you EXhale still has a high percentage of O2, your body only strips out a small portion of the O2 in each breath, and what emerges isn't all CO2, so any "used" air left in your lungs will have enough O2 to sustain you for a surprisingly long time.
As for hose location, I secured the end of my hose by tucking it through the hair elastic at my nape, so the end of the hose was near the back/top of my head. If I were to tape the hose to the inside of the bag, I would tape it near the top/above my head.
I have no idea how well an ice bag would work. Oven bags have the advantage of being very durable; their seams don't fail, as I've had ice bags fail when I'm using them to carry ice.