A 'disappointed' Indigenous Affairs Minister Nigel Scullion says he won't make way for former prime minister Tony Abbott to take over his cabinet post, as prominent Indigenous community leaders also push back against the idea.