Skip to content

#BlackHistoryMonth: Valley Gardening Godfather James F. Brown

After securing his own freedom, this remarkable man went on to become award-winning gardener, faithful diarist, and prominent 19th-century Beaconite.

by Reed Sparling
Share:

I went to church at Fishkill village and returned home through the rain and got very wet but heard a very good sermon preached. Mr. J D L Verplanck, Mr. W S Verplanck and Miss A L Verplanck attempted twice to cross the river to Newburgh to take passage in the steam boat for NY. But was disappointed and was obliged to return home. 

So begins the diary of James F. Brown, recording the day’s events for January 1, 1829. For the next 37 years — until March 1866, when he suddenly stopped — Brown would continue adding to this account from his home in Fishkill Landing (today’s City of Beacon) on a regular basis. 

A prominent 19th-century resident of today’s City of Beacon, James F. Brown is depicted in a mural at the city’s Howland Public Library. (Photo: Amy Brown / Scenic Hudson)

What makes it especially remarkable is who wrote it. Brown was an award-winning gardener, widely respected Beaconite, friend to the Hudson Valley’s most prominent residents, and among New York State’s very first Black voters — and he achieved it all after securing his personal freedom. As Brown biographer and Bard College professor Myra Armstead writes in her book, Freedom’s Gardener, it was  “a virtually unheard-of undertaking for an African American man,” especially one who had escaped enslavement.

Penned in a clear hand over 10 account books, Brown’s diary may not be a page-turner: “He’s so regular and dispassionate about his writing that it’s almost boring,” Armstead admits.

Yet thanks to its author’s encyclopedic interests — Brown records daily chores and weather, local and national events, travel on trains and steamboats, and civic amusements — the diary offers a detailed, historically valuable narrative of everyday life in 19th-century America and specifically, along the Hudson River.

“What can a man do who has his hands bound and feet fettered?”

How Brown came to be in a position to write the diary and enjoy a fulfilling career as head gardener and caretaker at Mount Gulian, the Hudson River estate of the wealthy Verplanck family, is definitely compelling.

Born enslaved in Maryland in 1793, Brown worked on a couple of farms there (along the way learning to read and write) before becoming a so-called “wage-slave” in Baltimore. While still subject to restrictive racial laws, he was able to make an independent living — perhaps as a servant, coachman, or skilled carpenter — while remitting a portion of his monthly income to a slaveholder. 

By March 1828, Brown was working as a servant for the Verplancks at their Manhattan home. How he secured this position is unknown, but “it was a game-changer for him,” says Elaine Hayes, executive director of the Mount Gulian Historic Site.

The estate house at Mount Gulian, home of the Verplanck family, in view of the gardens that James F. Brown tended in his day. (Photo: Mount Gulian)

As late as 1800, the Verplancks had been slaveholders, but by the time of Brown’s hiring they “had evolved in their thinking,” notes Hayes, and were ardent abolitionists — so ardent that when a dinner guest recognized Brown as a fugitive, patriarch Daniel Crommelin Verplanck paid the slaveholder $300 (nearly $9,000 in today’s dollars) to secure his freedom. 

Brown earned enough money to secure the freedom of his enslaved wife, Julia, shortly after their marriage in 1826, and had been promised his own freedom as well. When this failed to occur, Brown decided to escape. Prior to heading north in August 1827, he wrote a letter to his slaveholder containing these powerful words: “What can a man do who has his hands bound and his feet fettered? It will certainly get them loose in any way that he may think the most advisable.” In the letter, Brown also pledged to pay for his freedom. 

Brown would repay his employer in a series of installments. “I think this says something about the drive and ambition of a slave to become free,” notes Armstead. “It also says something about the mindset of enslaved Blacks, where they could explore opportunities for freedom.”

Enduring concern for lives of fellow African Americans

Sometime before his first diary entry, Brown had moved to Mount Gulian, located just north of the present Newburgh-Beacon Bridge, where he was first employed as a coachman and general laborer before demonstrating his skill with plants and eventually overseeing not only the grounds but the accounts related to them. “Clearly, he had a special relationship with [the Verplancks],” Armstead says. “They entrusted him with a lot of family responsibility.”

During much of his career, Brown worked on the gardens in tandem with Mary Anna Verplanck, Daniel’s daughter, who managed Mount Gulian. Together, they made the grounds a horticultural showplace, attracting visitors from near and far. Produce and flowers grown there won awards at major fairs, which Brown, as a Black man, could attend only because of the Verplanck family’s prominence, while Mary Anna was excluded for being a woman. 

Brown’s diary clearly illustrates the joy he took in spending time outdoors, recording seeds acquired and plants grown with minute detail. His expertise earned him respect and friendship with gardeners on nearby estates as well as with Andrew Jackson Downing, dubbed the father of American landscape architecture, who lived in Newburgh. Brown frequently rowed across the river to meet with Downing.  

Peonies bloom in Mount Gulian’s heritage gardens. (Photo: Mount Gulian)

The diary also records Brown’s interest and involvement in the lives of his fellow African Americans, through his attendance at civic events, the abolitionist-leaning periodicals he read regularly, his leadership in establishing the first Black burial ground in Beacon, and associations with figures involved in the Underground Railroad.

In fact, Armstead suspects that Brown intentionally downplayed his personal involvement in that movement. “I was very careful not to over-speculate or go beyond what I could prove, but I think he was probably far more active in the Underground Railroad,” she says.   

Achieving the American Dream

In the diary entry for Nov. 8, 1837 — and with his characteristic straightforwardness — Brown recorded one of the most momentous events of his life: “The election at Fishkill took place this day at which place James F Brown voted for the first time.”

It not only marked his participation in a fundamental right of citizenship — one granted to fewer than 30 Black men in New York at the time — but indicated he had achieved another part of the American dream. A requirement for Black people to vote was ownership of property. Brown had fulfilled it by purchasing a home not far from Mount Gulian. Every November until the diary ends, Brown diligently noted his participation in the electoral process. “It shows he wanted to be a citizen 100%,” says Hayes.

The lowest tide that has been for many years in the Hudson River was this day — the flats was bare from the Long Dock down to Denning’s Point — So that persons could walk down to get eels and fish with one hand.

Brown’s diary ends abruptly with this entry on March 26, 1866. No one knows why he decided to put down his pen, leaving blank any record of the final two years of his life as a self-described “old colored man.” He passed away on January 7, 1868, hailed by the local Fishkill Standard as “a prominent man…whose death will be regretted by many.”

James F. Brown’s grave marker in the northeast corner of St. Luke’s Cemetery in Beacon. (Photo: Amy Brown / Scenic Hudson)

The Verplancks, who inherited Brown’s diary, eventually donated it to the New-York Historical Society. It’s available for reading online. Another family member had the foresight to uproot and save a variety of plants from the gardens at Mount Gulian before they fell into decay in the 20th century. When the historic site later began restoring the grounds, roses, peonies, and yucca “with the same DNA of those that James and Mary Anna planted” were graciously returned, says Hayes. They take pride of place in Mount Gulian’s Heritage Garden.

Both the diary and the garden help to educate people about the example Brown set, one that still resonates strongly today, about striving to be part of what Armstead calls the American “narrative” — “the perception of opportunity, the wisdom of risk taking, the profitability of new ideas, and the soundness of democracy.”

“What makes James F. Brown’s story compelling,” Armstead writes in Freedom’s Gardener, “is that although he was born a slave in Maryland and therefore outside the intended or expected reach of that narrative, this [B]lack man nonetheless embraced it. Its message somehow reached him while he was still in bondage and propelled him to escape slavery, to challenge the exclusions it contained, and to forge a new and successful life for himself in upstate New York.”

“What inspires me is his persistence. He was planning his whole life on being a free man. What he had to do to arrive at that end is just remarkable,” says Hayes. “To me, he’s a hero.”

More in this series

“Conservation of land and conservation of people frequently go hand in hand.” — Eleanor Roosevelt Eleanor Roosevelt is rightfully lionized...
When it comes to superlatives, the Hudson Valley definitely holds its own. Here are five more sites that rank supreme...
A race in the clouds on the Fourth of July, with one hot air balloon piloted by a woman! That’s...
In 2021, Time magazine named the Hudson Valley one of the world’s greatest places. But some places in the region lay claim...
As thousands of men went off to fight in the American Revolution, many women kept the home fires burning — which...
She built one of the most striking homes on the Hudson River — and the nation’s first female self-made millionaire...
Described by one writer as the “Bob Dylan and Madonna of her generation,” poet Edna St. Vincent Millay (1892-1950) was...
From the beginning, stargazing has been part of the curriculum at Poughkeepsie’s Vassar College. That’s because Maria Mitchell — America’s...
They fought for working women’s rights, advocated for vulnerable children, even rescued drowning sailors. And of course, they called for...
Driving along the distinctly suburban stretch of Route 17K in Coldenham, a hamlet in Orange County’s Town of Montgomery, it’s...

Related Content

Editors' Picks

Climate Solutions
How to Get in on the Refillability Game
Land + Air + Water
Restoring Resilience to Mawignack Preserve
Land + Air + Water
Can Hops Make a Comeback in New York?
A close view of a hop growing on a vine. Behind it is a red barn.
Land + Air + Water
Protecting Forests by Managing the Exploding Deer Population
Climate Solutions
Floatovoltaics Makes Waves Approaching the Valley

Search Viewfinder:

Hudson Valley Viewfinder is a collaborative, community digital magazine sharing what inspires us about the beautiful Hudson Valley. We publish original stories and multimedia content about all things sustainable in the region along the Hudson River — including agriculture, science, wildlife, outdoor recreation, green transportation, environmental justice, and more.

Our mission is to immerse you in the storied history, fresh happenings, and coming solutions for making the Hudson Valley greener and more livable long-term.

Viewfinder is published by Scenic Hudson, the celebrated nonprofit credited with launching the modern grassroots environmental movement in 1963. With over 25,000 passionate supporters, Scenic Hudson’s mission is to sustain and enhance the Hudson Valley’s inspirational beauty and health for generations to come. Viewfinder supports that mission, because the better people understand what makes this place special, the more they will invest in protecting it. 

Keep up with the latest stories by subscribing to Scenic Hudson’s monthly digital newsletter, and connect with us on social via Facebook, Instagram, Twitter, LinkedIn, and Threads.

Our mission is to immerse you in the storied history, fresh happenings, and coming solutions for making the Hudson Valley greener and more livable long-term.

Viewfinder is published by Scenic Hudson, the celebrated nonprofit credited with launching the modern grassroots environmental movement in 1963. With over 25,000 passionate supporters, Scenic Hudson’s mission is to sustain and enhance the Hudson Valley’s inspirational beauty and health for generations to come. Viewfinder supports that mission, because the better people understand what makes this place special, the more they will invest in protecting it. 

Keep up with the latest stories by subscribing to Scenic Hudson’s monthly digital newsletter, and connect with us on social via Facebook, Instagram, Twitter, LinkedIn, and Threads.

Lynn Freehill-Maye
Managing Editor
editorial@scenichudson.org 

Riley Johndonnell
Director Creative Strategies & Communications
rjohndonnell@scenichudson.org

Lynn Freehill-Maye
Managing Editor
editorial@scenichudson.org 

Riley Johndonnell
Director Creative Strategies & Communications
rjohndonnell@scenichudson.org

We’re always looking for ideas around our main topic areas of Climate Solutions, Land + Air + Water, Plants + Animals, History + Culture, Outdoors, and Community.
  • Journalists and writers who have deep familiarity with New York and the Hudson Valley, we’d love to have you contribute! Please do introduce yourself by email, sharing writing samples and any relevant pitches you may have.
  • Photographers and videographers, we’d love to hear from you and see what you do. Please send along a portfolio with images or footage that showcases your best and/or most relevant work, with an emphasis on anything captured outdoors. 
  • Illustrators, we commission artwork on the regular. Drop us a note with some of the beauty you’ve created.
  • Media Partners & Social Media Influencers, we welcome opportunities to team up on series and campaigns. Reach out with any background about yourselves and your ideas.
We’re always looking for ideas around our main topic areas of Climate Solutions, Land + Air + Water, Plants + Animals, History + Culture, Outdoors, and Community.
  • Journalists and writers who have deep familiarity with New York and the Hudson Valley, we’d love to have you contribute! Please do introduce yourself by email, sharing writing samples and any relevant pitches you may have.
  • Photographers and videographers, we’d love to hear from you and see what you do. Please send along a portfolio with images or footage that showcases your best and/or most relevant work, with an emphasis on anything captured outdoors. 
  • Illustrators, we commission artwork on the regular. Drop us a note with some of the beauty you’ve created.
  • Media Partners & Social Media Influencers, we welcome opportunities to team up on series and campaigns. Reach out with any background about yourselves and your ideas.
  • We love to collaborate with media outlets, especially on episodic series (like these) of interest to our shared audiences. Past collaborations have included radio interviews, panel discussions and other events, original artwork, and e-blasts, all furthering the campaign’s excitement and reach. 
  • We also love to partner with other organizations whose missions align with Scenic Hudson’s. Feel free to reach out with some background on your group and its work.
  • Writers, photographers, and creatives, if you have an idea for a series or content campaign that might be a good fit, drop us a line!

Businesses, please note that as a nonprofit, Scenic Hudson is restricted from advertising or promoting for-profit companies, through Viewfinder or other outlets. While we understand content managers may wish to alert us to your company’s role in a relevant topic, we are unable to add links to businesses to our stories.

  • We love to collaborate with media outlets, especially on episodic series (like these) of interest to our shared audiences. Past collaborations have included radio interviews, panel discussions and other events, original artwork, and e-blasts, all furthering the campaign’s excitement and reach. 
  • We also love to partner with other organizations whose missions align with Scenic Hudson’s. Feel free to reach out with some background on your group and its work.
  • Writers, photographers, and creatives, if you have an idea for a series or content campaign that might be a good fit, drop us a line!

Businesses, please note that as a nonprofit, Scenic Hudson is restricted from advertising or promoting for-profit companies, through Viewfinder or other outlets. While we understand content managers may wish to alert us to your company’s role in a relevant topic, we are unable to add links to businesses to our stories.

Subscribe!

Get the latest articles delivered right to your inbox  — for FREE!