Indexing pioneer Vanguard Group has climbed to $4 trillion in assets for the first time, accentuating a loss of faith among investors in traditional money managers who handpick stocks.

The record of $4.048 trillion, reached at the end of January, follows a year when Vanguard’s funds pulled in more new money than all of its competitors combined, according to one industry total.

Of the $533 billion of net flows into all mutual funds and exchange-traded funds last year, 54%, or $289 billion, went to funds managed by Vanguard, according to research firm Morningstar Inc. The fund company’s own tally for the year was even higher, at $322.8 billion.

The rush to Vanguard is largely the result of a push to embrace funds that mimic broad indexes for a fraction of the cost of traditional actively managed mutual funds. Vanguard, which started the first index mutual fund for individual investors 40 years ago, is No. 2 in the asset-management world behind New York-based BlackRock Inc., another money-management giant benefiting from shifting investor tastes.

BlackRock topped $5 trillion in assets late last year for the first time. It has a larger international business than Vanguard.

Rival firms who have long been synonymous with their star pickers of stocks and bonds have been hurt by years of subpar performance and relatively high fees. Investors pulled a net $340.1 billion from U.S.-based actively managed funds last year, according to Morningstar, while pouring a record $504.8 billion into U.S.-based passively managed funds.

Active management remains a high-margin business. And with many investors loath to switch funds for reasons from tax concerns to inertia, firms can continue to earn hefty profits even as interest in indexing picks up. At the same time, signs of retrenchment are emerging in the business.

Mutual-fund firms last year started the fewest new U.S.-based actively managed mutual funds in 14 years, according to Morningstar. The amount of funds liquidated or merged was the highest since 2009.

“It signifies how popular passive investing has become, that investors are increasingly more comfortable paying less for average benchmark performance instead of trying to beat the benchmark and, in many cases, fail,” said Todd Rosenbluth, director of ETF and mutual-fund research at CFRA, a research firm.

Vanguard crossed the $4 trillion mark in January after collecting an estimated $49 billion in net new cash from investors during the month. Of those January flows, about $45 billion went into index funds while the balance went to actively managed funds.

Vanguard crossed the $3 trillion threshold in August 2014.

In 2016, index funds accounted for about 85% of the total net new cash Vanguard attracted. Slightly less than a third of the global net inflows went to the firm’s ETFs. Across asset classes, about half of the new money Vanguard pulled in globally last year went into stock funds and nearly 40% went to bond funds, with the balance in money-market and balanced funds.

Vanguard in recent years has gained market share at the expense of rivals including Fidelity Investments. Vanguard is owned by its fund shareholders, a structure that allows it to keep expenses lower, and it has been undercutting many competitors on fees.

Other fund companies have cut fees but haven’t attracted as much new cash.

There are dangers for fund companies that rely on passive investments. Vanguard and other large passive fund managers have benefited from rising markets. Index funds can’t protect investors when indexes decline, and a widespread downturn could lead investors to conclude they would have a greater chance of avoiding losses with stock pickers.

A spokeswoman for the firm said it is grateful to investors for the assets, but said growth isn’t the firm’s goal.

That rapid growth at Vanguard has led to an increase in client calls and some customer complaints about long wait times, according to the company. Vanguard doesn’t have a network of retail branches like some of its competitors.

Vanguard executives said in a client webinar this year that the firm had received unprecedented phone volumes but that it doesn’t trade cost for service.

“We hate it. We’re doing everything we possibly can to bring those wait times down to acceptable levels,” said Chris McIsaac, managing director of Vanguard’s planning and development group.

Vanguard, which has a staff of about 15,000, hired more than 1,700 new full-time staff members last year and said it expects to hire a similar number of people in 2017.

Write to Sarah Krouse at sarah.krouse@wsj.com