Python os.walk complex directory criteria
<p>I need to scan a directory with hundreds or GB of data which has structured parts (which I want to scan) and non-structured parts (which I don't want to scan).</p>

<p>Reading up on the os.walk function, I see that I can use a set of criteria in a set to exclude or include certain directory names or patterns. </p>

<p>For this particular scan I would need to add specific include/exclude criteria per level in a directory, for example:</p>

<p>In a root directory, imagine there are two useful directories, 'Dir A' and 'Dir B' and a non-useful trash directory 'Trash'. In Dir A there are two useful sub directories 'Subdir A1' and 'Subdir A2' and a non useful 'SubdirA Trash' directory, then in Dir B there are two useful subdirectories Subdir B1 and Subdir B2 plus a non useful 'SubdirB Trash' subdirectory. Would look something like this:</p>

<p><a href="https://i.stack.imgur.com/KsyY3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KsyY3.png" alt="Example Directory"></a></p>

<p>I need to have a specific criteria list for each level, something like this:</p>

<blockquote>
  <p>level1DirectoryCriteria = set("Dir A","Dir B")</p>
  
  <p>level2DirectoryCriteria = set("Subdir A1","Subdir A2","Subdir
  B1","Subdir B2")</p>
</blockquote>

<p>the only ways I can think to do this are quite obviously non-pythonic using complex and lengthy code with a lot of variables and a high risk of instability. Does anyone have any ideas for how to resolve this problem? If successful it could save the codes running time several hours at a time.</p>