After weeks of negotiations, Yahoo Inc. Chief Executive Marissa Mayer struck a secret truce in April 2015 with the activist investor who was threatening a battle for boardroom seats at the troubled internet company.

Ms. Mayer and her executive team agreed to be more mindful about Yahoo’s rising costs. In return, the activist investor, Starboard Value LP CEO Jeffrey Smith, agreed to withdraw his board nominations. The talks haven’t been previously reported.

At an employee meeting later, Ms. Mayer played down the threat from Starboard. One person at the meeting recalls her saying: “Some of our investors think they own more of the company than they actually do.”

Then Yahoo’s costs ballooned. It paid $20 million for the rights to the National Football League’s first streaming-only game broadcast, a matchup of the Buffalo Bills and Jacksonville Jaguars. Yahoo bought shopping site Polyvore Inc. for more than $160 million, triple what Yahoo’s top deal-making executive estimated it was worth, according to two people familiar with the matter.

As tough as her job was from the start, Ms. Mayer made things even worse after Starboard began prodding Yahoo, according to interviews with more than two dozen investors, current and former Yahoo employees, and other people who worked closely with her.

She failed to produce the pledged cost savings, plunged even deeper into her turnaround efforts and clung to the idea that she was going to save Yahoo, even when it became clear to insiders and other observers that the company was beyond saving.

Ms. Mayer and Mr. Smith declined to comment for this article. But his frustration boiled over into a revolt that began in November with a public letter-writing campaign and ended this spring with Starboard taking nearly half of Yahoo’s board seats and helping to force an auction of Yahoo’s web properties. Final bids are expected soon.

Yahoo’s sale would cap the stunning fall of Ms. Mayer, 41 years old, who was widely hailed as one of the most promising executives in Silicon Valley when hired as Yahoo’s president and CEO in July 2012.

She has defended her strategy by stressing that growth is the lifeblood of internet businesses

In March, though, Ms. Mayer told interviewer Charlie Rose that one lesson she has learned is that pacing and timing are important in a turnaround. “I think there are some things that we probably did too quickly,” she said. “I think there are some things that we probably did too slowly and, you know, not enough.”

Her arrival brought a jolt of optimism that Yahoo could be turned around despite losing users and advertising dollars to companies such as Google, now part of Alphabet Inc., and Facebook Inc.

Supporters and critics say Ms. Mayer took on a nearly impossible mission at Yahoo, bringing fresh energy and talent to a company many had left for dead. Under her leadership, Yahoo has returned $9.8 billion to shareholders.

As the turnaround stalled, though, Ms. Mayer faced a difficult decision: yield to Mr. Smith’s demands to prune Yahoo’s costs and her own ambitions, or defy the activist investor and swing for the fences.

“Her core mistake was this belief that she could reinvent Yahoo,” says a former senior executive who left the company last year. “There was an element of her being a true believer when everyone else had stopped.”

Ms. Mayer came to Yahoo for redemption after her rise at Google stalled. Yahoo had churned in five years through a half-dozen CEOs, who failed to stem ad-revenue declines and defections by top engineers.

She instantly became one of the highest-profile corporate executives in the U.S., and her appeal widened when she gave birth to her first​ child 2½ months into the job at Yahoo and put a nursery in the office adjoining hers.

Ms. Mayer spent her first two years at Yahoo launching new businesses and acquiring dozens of mobile software startups. Their engineers began building smartphone apps, and she also ramped up investment in web search, a long-neglected area.

She has said the investments generated significant and growing revenue from mobile ads, a business that didn’t exist at Yahoo when she arrived. She said the acquisitions, improved perks for employees and digital-media partnerships were necessary to reinvigorate the company as a destination for web surfers and coveted brand for advertisers.

Still, the new projects failed to create enough growth to offset Yahoo’s declining business in banner ads. Revenue remained stagnant, and profitability kept shrinking.

In September 2014, Starboard told Ms. Mayer in a letter it had bought a stake in Yahoo and complained that the company had “a bloated cost structure.”

Ms. Mayer worried that Mr. Smith could be a serious threat in a proxy contest, according to people familiar with her thinking. In October 2014, shareholders replaced the entire board of directors at Olive Garden owner Darden Restaurants Inc. with a board led by Mr. Smith.

Ms. Mayer invited Mr. Smith to meet at Yahoo’s headquarters in Sunnyvale, Calif., and they met several more times in the next six months. Starboard’s stake was smaller than the 5% required for public disclosure.

Mr. Smith pushed for Yahoo to return billions of dollars to shareholders, spin off its stake in Chinese internet giant Alibaba Group Holding Ltd. and cut annual costs by more than 10%.

Starboard also called for Yahoo to explore a combination with another fading internet icon, AOL Inc., but later backed away from the idea.

Yahoo’s Ms. Mayer urged Mr. Smith to keep quiet about his proposed board nominations, and she began making major concessions.

In January 2015, she announced a plan to spin off the Alibaba shares. As a March proxy deadline neared, she said Yahoo would buy back $2 billion in stock.

Mr. Smith, 44, also was assured by Ms. Mayer and Yahoo’s finance chief, Ken Goldman, that they would consider Starboard’s cost-cutting plan.

Ms. Mayer hoped she was done dealing with the activist investor and his colleagues at Starboard.

Ms. Mayer refocused on her turnaround effort, moving more than 300 of Yahoo’s roughly 11,000 employees to a secretive mobile-search effort called Project Index.

The plan was to create an app that acts as a personal virtual assistant to compete with Apple Inc.’s Siri, Google Now and other robot-helpers.

Ms. Mayer described the planned app to analysts as a kind of search engine that uses personal data about users and context, such as their physical location, to generate smarter results. Internally, she called Project Index a “bet-the-company” strategy that gave Yahoo a chance at competing with Google, a person familiar with the matter says.

People briefed on Project Index say it was hard to achieve Ms. Mayer’s elaborate vision. Initial plans called for three years of development, but the timeline was squeezed to complete a modest version sooner. That happened even though Project Index’s development was stopped and then restarted more than once, the people say.

Last summer, Ms. Mayer rankled members of her executive team when she pushed through the Polyvore takeover, overruling Jackie Reses, Yahoo’s chief deal maker at the time and one of Ms. Mayer’s most trusted lieutenants.

Ms. Reses had estimated that Yahoo should pay no more than $50 million for Polyvore.

Ms. Mayer compared the deal to Google’s acquisition of YouTube in 2006, arguing that “you can never overpay” for a company with the potential to land a huge new base of users. In late July, Ms. Mayer agreed to pay $161 million in cash and up to $15 million in stock for Polyvore.

Since the takeover, traffic to the shopping site is up 23% to 10.4 million in May, according to comScore. Yahoo says its sites get one billion monthly visitors combined.

After expressing concerns about the Polyvore deal, Ms. Reses was moved to a new role overseeing Yahoo’s planned spinoff of the Alibaba stake. She left a few weeks later for a job at payment startup Square Inc

By late summer, Ms. Mayer’s troubles were growing. The trickle of top executives leaving Yahoo had grown into a steady stream. It got easier to find an open space in the parking lot—nearly impossible during normal work hours in 2013 and 2014—as employees arrived later and left earlier, according to several former Yahoo executives.

At an August meeting after Yahoo reported its first quarterly loss under her, she asked senior executives to write her an email pledging to stay at the company for at least three more years.

Mr. Goldman, Yahoo’s finance chief, agreed to the three-year pledge. Other executives left the room feeling unsure about their own willingness to commit, one of the people says.

Starboard’s Mr. Smith decided to go public with his case against Yahoo. In a Nov. 19 letter to Ms. Mayer and Yahoo Chairman Maynard Webb Jr., the New York investor fumed about Ms. Mayer’s “dismissive approach to our serious concerns about the current situation at Yahoo.” Mr. Smith urged it to halt the Alibaba spinoff and sell Yahoo’s core business.

Other large shareholders privately wrote letters to Yahoo’s board expressing similar concerns, according to people familiar with the matter.

Over several days of deliberations at Yahoo headquarters the weekend after Thanksgiving, Thomas McInerney, a former finance chief at IAC/InterActiveCorp who joined Yahoo’s board in 2012, argued that concerns about the spinoff’s potential tax bill would hurt the market value of Yahoo’s remaining core business, according to a person briefed on the talks.

The Internal Revenue Service had denied Yahoo’s request for a favorable ruling on the spinoff plan.

Yahoo directors, including Ms. Mayer, voted to halt the spinoff and consider alternative strategies, including an auction of the company.

Ms. Mayer juggled the corporate drama with her second pregnancy, this time with twins. Within 24 hours of her announcement on CNBC that the spinoff was canceled, she rushed to a hospital to deliver her daughters.

From her hospital bed, Ms. Mayer called Mr. Webb to discuss the company’s next steps, according to a person familiar with the matter.

By then, though, Ms. Mayer had limited options. The board’s Feb. 2 announcement that it would explore “strategic alternatives” hung a for-sale sign on Yahoo’s core business.

She also was concerned about a possible boardroom coup by Starboard and belatedly conceded on new cost cuts. In February, she announced plans ​to reduce Yahoo’s workforce by 15% and save $400 million a year.

In March, Ms. Mayer said in the interview with Mr. Rose that said she needed three more years for her turnaround and hoped to be running Yahoo a year later. “I don’t think the story is yet played out,” she said.

But her power was fading. Yahoo’s board worried that a proxy battle with Starboard would jeopardize the auction process. To avert a battle, directors agreed to put Mr. Smith and three of his nominees on the board.

If Yahoo is sold and Ms. Mayer is terminated as a result, she stands to receive about $55 million. Since joining Yahoo, she has received $118.2 million in cash and equity compensation.

Final bids from suitors, which include Verizon Communications Inc. and private-equity firm TPG, are expected sometime in July. People close to the bidders say some of the business plans don’t include Ms. Mayer at all.