The object of this Essay is to assert one very simple principle, as entitled to govern absolutely the dealings of society with the individual in the way of compulsion and control, whether the means used be physical force in the form of legal penalties, or the moral coercion of public opinion. That principle is, that the sole end for which mankind are warranted, individually or collectively, in interfering with the liberty of action of any of their number, is self-protection. That the only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others. His own good, either physical or moral, is not a sufficient warrant. He cannot rightfully be compelled to do or forbear because it will be better for him to do so, because it will make him happier, because, in the opinion of others, to do so would be wise, or even right... The only part of the conduct of anyone, for which he is amenable to society, is that which concerns others. In the part which merely concerns himself, his independence is, of right, absolute. Over himself, over his own body and mind, the individual is sovereign.--J.S. Mill (1859) On Liberty
In the context of a critical discussion on attempts to prevent competition by guild and settlement laws (etc.), Adam Smith writes in the (1776) Wealth of Nations:
The property which every man has in his own labour, as it is the original foundation of all other property, so it is the most sacred and inviolable. The patrimony of a poor man lies in the strength and dexterity of his hands; and to hinder him from employing this strength and dexterity in what manner he thinks proper without injury to his neighbour is a plain violation of this most sacred property. It is a manifest encroachment upon the just liberty both of the workman and of those who might be disposed to employ him. (1.10.67)
Smith is relying on the conceptual closeness among property, propriety, and proper-ness. But Smith’s conception of the nature of fundamental property rights, with its emphasis on the work performed by a poor person’s hands, is quite narrow (it is more narrow than Mill's "more expansive own body and mind") and also quite egalitarian (although it does not cover the severely disabled).* Leaving aside here what grounds Smith's argument for his position, according to Smith there is a basic right to manual work and this property-right grounds “all other property.”
But while the basic right is “the most sacred and inviolable,” it follows that other forms of property may be less sacred and inviolable. On this basis (and a few other texts), I have argued that Smith distinguishes between original and (institutionally dependant) derived property rights, and according to Smith none of us can claim an inviolable, absolute right to all of it. In fact, as Steve Darwall has urged on me, Smith's position relies on a reflective endorsement from the perspective of having an equal authority to make claims and demands on one another at all. (See here for a fuller argument.)
So, we find in Smith a tight connection among the right to work (as well as right to free settlement), basic property rights, commercial contracts, and what came be to known as ‘negative’ liberty (in Constant who certainly knew his Smith). But upon re-reading the passage, I also noticed not just that the basic form of property right according to Smith is work, but also that one ought to have an absolute right to contract for work as long as one does not harm others ("in what manner he thinks proper without injury to his neighbour"). In fact, Smith here anticipates the famous articulation of the (more expansive) harm principle in the French 1789 Declaration of the Rights of Man: “Liberty consists in the freedom to do everything which injures no one else.”
As an aside, Smith's way of phrasing the principle -- 'without injury to his neighbour -- evokes the Biblical injunction to love thy neighbour. He, thereby, signals that his principle is consistent with, even a necessary condition for, a broader religious outlook.
Now, I am not the first to connect Smith with the history of the harm principle. But generally the link is made through a rather (wimpy) passage in The Theory of Moral Sentiments (TMS). For example, Jethro Lieberman calls attention to this text:
There is, no doubt, a propriety in the practice of justice, and it merits, upon that account, all the approbation which is due to propriety. But as it does no real positive good, it is entitled to very little gratitude. Mere justice is, upon most occasions, but a negative virtue, and only hinders us from hurting our neighbour. The man who barely abstains from violating either the person, or the estate, or the reputation of his neighbours, has surely very little positive merit. He fulfils, however, all the rules of what is peculiarly called justice, and does every thing which his equals can with propriety force him to do, or which they can punish him for not doing. We may often fulfil all the rules of justice by sitting still and doing nothing. (TMS).
For Smith (and Hume) 'justice' simpliciter tends to mean the rules governing property. But 'justice' can also refer to a set of rules that govern an individual's (negative) virtue (as it does in the passage quoted). The right rules prevent and avoid harm to others. These rules connect individual propriety with harm to others. It's not so clear in this quoted passage, however, that it also involves the regulation of contracts unless one allows that Smith is also tacitly relying on the traditional meaning of 'justice.'
*There is an open debate to what degree Smith truly recognizes the value of woman's work (but see here for evidence he does).