Property tax, surveys find, is the most hated of all taxes. Why is it, detractors say, that you have to keep paying the government on something that you own—forever? Or, in the words of one protester on YouTube, property tax is “oxymoronic, unjust, and un-American!”

On that last point, he’s onto something, at least on a literal level: The origins of the property tax aren’t American at all. It, instead, has roots that date back to Europe’s feudal system. First instituted in England by William the Conqueror in 1066, the early tax system worked this way: A king (or conqueror) took over all the land in a given territory. He would then divide it among his lieutenants and supporters, who would pay him (with money or services) in order to keep that land. In return, landholders enjoyed the king’s protection and were able to rent the property out to others—who would live and work the land—for a fee. The punishment for nonpayment was forfeiture of the land, which could result in a considerable loss of money and status. At the time, this system was called “free and common socage,” according to John Joseph Wallis, an economic historian at the University of Maryland. The person who held the land was called a socman, his taxes, socage. The arrangement created a way for people to own land while still having to remain loyal to the crown, which also had rights to the land.

After expansion across the Atlantic started, King James made sure that this system traveled overseas with the first settlers at Jamestown, so that he could partake in the profits of exploration of the new land. The charter of the Virginia Company held that—as in feudal times—the king would protect the lands in Jamestown, and in return, the people living on the land would pay him a share of their profits there. All land of the colony would be held in “free and common socage,” according to the Virginia Company charter. This meant that land could be bought and sold in the colonies, as long as the new holder of land continue to pay the king.

All of the 13 original colonies’ charters stated that land be held in free and common socage. This motivated entrepreneurial colonists to make sure they could make money on their land, according to Jonathan R. T. Hughes, the author of The Governmental Habit Redux. In the colonies, people could purchase land and pass it onto their own heirs more easily than they could in Europe, where land had been held by the same families for centuries. But they couldn’t just sit on a parcel of land once they gained ownership. Because they owed duties to the crown, they knew they needed to make their land profitable in order to keep it. To do so, early colonists turned to farming the land, cutting down its trees, and hunting its animals. In Latin America, by contrast, where land was not held in free and common socage, settlers could take over large tracts of land and do nothing with them, which resulted in swaths of undeveloped land. The free and common socage system helped the colonists develop the American idea of citizenship, which included certain rights (such as the right to own land) and responsibilities (such as the duty to pay fees on that land), Hughes writes. Later, land ownership would remain incredibly important, since it would be directly tied to the right to vote.

When the colonists revolted and property owners no longer had a king to answer to, the new American government had to figure out who actually owned the land. It was decided that the government would step into the role the king once held. The Northwest Ordinance established that the federal government owned all the lands, and the state served as the “donor,” which essentially meant the person who collected the fees from the tenants. This is similar to the way the king owned the land, but a knight might have collected the fees. After the Northwest Ordinance, when land was sold to an individual, the state had the right to collect payment from that individual in perpetuity. This was a system very similar to free and common socage called “fee simple.” That basic principle, and terminology, persists today: Anyone buying a house will notice the term “fee simple” in their legal documents stating that they own the land and property, which is still subject to taxes.

It’s a peculiar note of history that the founding fathers, who spoke often of abolishing the feudal system, kept this remnant of the Old World. But the rationale is very simple: They needed the money. In fact, the federal government levied a national property tax in 1798, 1814, 1815, 1816, and 1861. The tax in 1798, for example, charged households for their slaves (50 cents), houses, and land. It raised $2 million, according to Wallis. These taxes usually outraged residents, who would often revolt, but the system of collecting property tax remained. That’s because property taxes were locally spent and collected in the beginning, and often paid for things like roads and canals that property owners would be able to see, and that increased the value of their property.

Even the 1840s anti-rent wars, a tenant revolt in New York, didn’t succeed in abolishing the system that allowed property tax, though it banned all feudalism in the state’s constitution. A revised version following the revolt read “all feudal tenures of every description, with all their incidents, are declared to be abolished, saving however, all rents and services certain which at any time heretofore have been lawfully created or restored.” That meant that even if, by name, feudalism was gone, property tax was here to stay.