Yhyjyk

World is a non-Earth and mostly steriotypical sword and sorcery. I think there were only humans. I believe it was a stand alone novel, but it could have been in a series. I think it was written in the 70's or 80's.

The main character is recruiting magic users to go with an army to land far to the south.

There are multiple types of magic users. One of them has telekinetic type magic. This particular type of magic is not based on internal energy, but is based on a combination of how much you use it and how far away from its source (in a mountain to the north) the user is. The more they use the magic, the more powerful they become.

The downside of their power, is that the more powerful you are, the more powerful is the urge to go to the source, from which no one ever returns. As they become more powerful, the users try to travel away from the source, as while distance reduces their power, it also reduces the urge.

This magic user signs on to travel away from the call he feels. The main character has a very minor bit of the telekinetic magic (uses it for dice games) and so knows as far south as they are going, the magic user will be limited to only minor magics such as untying ropes, but takes him on.

Once they are in the south, someone (the magic user, maybe) complains about a buzzing in their head, which turns out to be an inactive "source" similar to that used by the magic user. After others complain about how little help the magic user has been, someone says that it was too bad he couldn't just tap into that source instead.

The magic user briefly considers this (and as they are engaged in a siege at the time) is immediately able to single-handedly destroy the enemy and swiftly installs himself as the local ruler.

After some time, the main character wonders if his massive and continuous use of power will result in him eventually hearing the call from his original source. He considers warning the magic user, but then the magic user slaughters a servent for spilling wine and he decides to not do so.

Eventually, the magic user does hear the call and, shortly after, flys north towards his original source.

This is Lawrence Watt-Evans's The Unwilling Warlord.

The Unwilling Warlord was the third novel in the Ethshar series -- though it's not third in internal chronology. It's the story of a young man who finds that he's the hereditary warlord of a small kingdom that's on the verge of war against two larger neighbors. He has a simple choice -- win the war, or die.

And the only way he can see to win the war is to use magic to cheat.

Warlocks hear the call from a far-off source, grow in their powers with use, and eventually go mad and unwillingly go to the source (which is somewhere beyond the poisonous fog that surrounds the civilized world). One of the men he hires is a warlock hoping to protect himself by distance, only to find that new source of power, seemingly inert.

Eventually, Vond starts hearing the original source, and is drawn away to it.

I rather enjoyed the Ethshar books as a teenager, and read most of the ones released.