My personal understanding (which may be very wrong) is that there are different sorts/mechanisms of pollution involved.

Ag pollution is usually things like herbicide, pesticide, and fertilizer, which can have disastrous effects downstream if it washes into a river, but don't make things grown on that land poisonous as long as you rinse off anything that has been sprayed on them.

This is in contrast to, say, reclaiming farmland in an area that had a textile dye factory upstream.

There are certainly exceptions, for example many old orchards used lead arsenate as a pesticide, contaminating the soil with both lead and arsenic.