No; the principle of “one person, one vote” is a crucial part of ensuring equality in democracies.
The NO campaign has tried to suggest that voters whose votes are transferred between candidates in different rounds are actually getting more votes than those whose votes stay with their first preference, but it’s easy to see that this is false.
AV is counted in multiple rounds; at each round, all active votes are counted, whether they have stayed with the first preference, or have been transferred to another preference.
If you vote for a “mainstream” party, your vote counts towards them in the first round, and the second, and so on until they are eliminated or win. Just as someone voting for all “non-mainstream” parties has their vote counted in the first round, and the second, and so on until all their preferences are eliminated or win. At the end of counting, your vote may either have stayed with one candidate, counting for the same person in each round, or moved between candidates.
At all rounds, unless you choose to stop expressing preferences, the principle is always: one person, one vote.
In fact, in the American equivalent to AV, Instant Runoff Voting, a court in Michigan found the No campaign’s argument to be completely false.