I think it's a fair concern. I don't think approaching politics in NZ from the perspective of "this party stands for that" is realistically workable.
Regardless of what ACT's meant to be about, it's really positioned as "the other right wing party" which probably comes from the whole Epsom thing and then was reinforced by the John Banks and Don Brash years. A lot of core National policies in the Key years that came from the right were also things that I think most people would argue were either obviously ACT ideas or ended up being massively associated with ACT (e.g. charter schools or the supercity).
There's also a clear parallel with the Greens. As a party they're not and shouldn't be treated as "Labour for when you don't want to vote Labour" but that's very much been their experience in the last twelve years (four/five election cycles) if not longer. And the mutual comparisons between ACT and the Greens merely serve to reinforce this impression.
NZ First, in contrast, has tried to stake itself as a centrist party, an old folk's party and a "pre-Rogernomics" party. It's clearly very socially right wing and economically is basically whatever Winston Peters needs it to be in any given moment... whilst, in principle, always having that "return to the 1970s"/walkback the neoliberal revolution vibe to it. This leads to the analysis that NZ First (a) attracts social conservatives and (b) economic revolutionaries but must either align itself with the socially progressive & economically ambiguous Labour or the socially ambiguous & economically steadfast National. Once NZ First's in power, they must therefore prop up a party that necessarily diverges from them in one facet but which can be convinced to see things their way in the other.
This is the problem for minor parties... in government, they become complicit with the actions of the Government and in an electorate where none of the parties have much moral fibre, their voters are always one step away from "might as well vote for National/Labour". The Greens are probably very lucky that NZ First are the actual coalition partners... it gives them a "look, we've been screwed over by Winston Peters at every turn, if you give us more votes then we'll be able to enforce our agenda" narrative.
So... I can see people switching from NZ First to ACT because it wouldn't have been clear that they'd go with Labour (even Little/Ardern's very anti-immigration and Asian people take on Labour) over National. And as Costa is saying, these minor parties are really "protest votes" since everyone pretty much knows that they're remora fish swimming with the National and Labour sharks. So, you can vote safely for whichever and signal "look, party that I really support, I don't like what you're doing now so I'm going to inflate the vote of your minor party buddy without having to worry about causing our coalition to not win the post election negotiations". NZ First, of course, swings both ways but people are always going to imagine "well, deep down Winston's really like me, i.e. red/blue" rather than "a power obsessed survivor who hates immigration".