DETROIT, April 9— "We are the invisible people in music," said Kevin Scott, a 37-year-old composer, echoing Ralph Ellison's phrase about the black man in society. He is part of a small but dedicated group, a minority within a minority: black musicians who compose for the concert hall.

In less formal venues, black Americans have been at the heart of musical life, the originators of ragtime, jazz, blues, gospel, soul, disco and hip-hop. But in the concert hall, black composers have been marginal. Their impact on the evolution of art music has been minimal.

For three days here last week, though, they were far from invisible, as 122 people -- composers, administrators, conductors, students and other interested parties -- gathered under auspices of the Detroit Symphony orchestra for a symposium on black American composers, the fifth annual event of its kind.

Beginning with a competition in which four compositions were played in an open orchestra rehearsal and then at the weekend's subscription concerts, the forum, sponsored by the Unisys Corporation, included detailed discussions of the works, along with career advice for black composers. There were also many conversations in which senior composers who have made careers against great odds traded insights with young, academically trained composers. Relating Race and Music

Composers usually assemble in groups according to shared stylistic interests. Here, the shared factor was race, and nearly everybody was examining what relationship race had to the music itself. This is precisely what administrators of the Detroit Symphony had in mind: it has turned itself into a laboratory for exploring issues of race and music.

During the last decade, the Detroit has been something of a pioneer in facing the problems that affect many American urban orchestras. Its troubles have in many ways reflected those of Detroit itself, which has gone from middle-class capital of the automobile industry to one of the country's most economically troubled cities. The symphony's white audience has moved to the suburbs and the overwhelmingly black population that remains has shown little interest in the orchestra's traditional activities.

During the 1980's, ticket income dropped, there were two strikes and, in 1989, the state legislature threatened to withhold funds unless the Detroit began to reflect the racial composition of its surrounding population. The demand was that a black musician be hired immediately, overturning the race- and sex-blind auditions that have become standard practice for American orchestras. The symphony acquiesced (to the shock of many in the music world). A black bass player, who was regularly filling in with the orchestra as a substitute, was hired, becoming the second black in the orchestra. But in 1991 the state, stricken by continuing economic problems, canceled $3.7 million in expected support anyway. This created a financial crisis, and each player accepted a cut in pay and benefits worth over $40,000. A Daunting Challenge

Against that backdrop, the Detroit's competition for black composers has become a small part of what is probably this country's most ambitious affirmative-action program to increase the involvement of black musicians, composers and listeners in the life of an urban orchestra. Not even 2 percent of symphonic musicians in the country are black, so the task is daunting. In fact, the Detroit has found no other qualified full-time black players since 1989.

But 12 of 66 members of the symphony's board are black. The associate conductor, Leslie B. Dunner, is black. All auditions, though they remain blind, must include a black applicant. There is a fellowship program in which young black performers essentially become apprentices, playing and studying with the orchestra; fellows are coached for auditions and are partially reimbursed for expenses even when trying out for other orchestras. In addition, the Detroit plays music of black composers in concert, and is making recordings of important black American compositions.

There is also a spirit of optimism among the players, despite the vast empty lots surrounding Orchestra Hall, where one is warned not to take a stroll. Some of it has to do with the success of Neeme Jarvi as music director; ticket sales have gone up at double-digit rates each year since he came in 1991, and the orchestra, with a budget of $17 million, finished last year with a surplus.

The Detroit's recording schedule has picked up; Mr. Jarvi will have recorded more than 30 CD's with the orchestra by 1996. And it has been invited to play at the Lucerne Festival in Switzerland next year. The Detroit is also one of the few orchestras still heard on radio; concerts are broadcast in 511 markets, including New York, and reach 975,000 listeners a week. (It is to play in Carnegie Hall on April 23). A Larger Presence

The affirmative-action programs have clearly contributed to the change in atmosphere. The Detroit's presence in the city's musical life has expanded. It recently played at the mayor's inaugural celebration and regularly plays at a national N.A.A.C.P fund-raising dinner here that attracts 10,000 people.