Extract only specific rows and columns from a table (td) in BeautifulSoup, python
<p>I want to scrape data from a website and write it to CSV. The sample code contains "column names" and "values".</p>

<pre><code>&lt;div class="table-responsive"&gt;
  &lt;table class="table table-bordered table-hover"&gt;
    &lt;tbody&gt;
    &lt;!-- required First 3 column names--&gt;
      &lt;tr&gt;
        &lt;td&gt;&lt;/td&gt;   
        &lt;td align="center" style="vertical-align:middle;" title=" Number"&gt;Number&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" colspan="2" title=" Code "&gt;CODE&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;Name&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" title="Category "&gt;Category&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" title="Variation "&gt;Variation&lt;/td&gt;
      &lt;/tr&gt;
      &lt;!-- required --&gt;
      &lt;tr class="success"&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;&lt;a href="/Number/3259"&gt;3259&lt;/a&gt;
        &lt;/td&gt;
        &lt;td style="width:125px;height:60px;vertical-align:middle;" &lt;a href="/CODE/ABC123"&gt;ABC123&lt;/a&gt;
        &lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;&lt;a title="CODE ABC123" href="/CODE/ABC123"&gt;LAND &lt;/a&gt;
        &lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" class="success"&gt;3 - Design Reference&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" class="success"&gt;2 - Production Item&lt;/td&gt;
      &lt;/tr&gt;
      &lt;!-- not-required --&gt;
      &lt;tr&gt;
        &lt;td colspan="7"&gt;
          &lt;div id="BMCRL_36" class="collapse"&gt;
            &lt;div class="list-group"&gt;
              &lt;a class="list-group-item"&gt;
                &lt;h4 class="list-group-item-heading"&gt; Number 3259 - Reference Number ABC123&lt;/h4&gt;
                &lt;p class="list-group-item-text"&gt;&lt;/p&gt;
              &lt;/a&gt;
              &lt;a class="list-group-item"&gt;
                &lt;h4 class="list-group-item-heading"&gt;CODE ABC123 &lt;/h4&gt;
                &lt;p class="list-group-item-text"&gt;&lt;/p&gt;
              &lt;/a&gt;
            &lt;/div&gt;
          &lt;/div&gt;
        &lt;/td&gt;
      &lt;/tr&gt;
      &lt;!-- required --&gt;
      &lt;tr class=""&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;&lt;a href="/Number/3260"&gt;3260&lt;/a&gt;
        &lt;/td&gt;
        &lt;td style="width:125px;height:60px;vertical-align:middle;" &lt;a href="/CODE/XYZ453"&gt;XYZ453&lt;/a&gt;
        &lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;"&gt;&lt;a title="CODE XYZ453" href="/CODE/XYZ453"&gt;WATER &lt;/a&gt;
        &lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" class="success"&gt;3 Control Reference&lt;/td&gt;
        &lt;td align="center" style="vertical-align:middle;" class="success"&gt;2 Item&lt;/td&gt;
      &lt;/tr&gt;

    &lt;/tbody&gt;
  &lt;/table&gt;
&lt;/div&gt;
</code></pre>

<blockquote>
  <p>The output should be like</p>
</blockquote>

<ol>
<li><p>Column names from First <strong>'td'</strong> tag when I use this </p>

<pre><code>from bs4 import BeautifulSoup
soup = BeautifulSoup(open('code.html'),'lxml')
col = soup.find('tr').find_all('td')
for c in col:
    print(c.get_text())
</code></pre>

<p>all the columns are printed. Instead i need only,</p>

<p><code>['Number', 'Code', 'Name']</code></p></li>
<li><p>I don't need <strong>'tr'</strong> which contains "colspan" <em>(3rd tr)</em> </p></li>
<li><p>Also remove last two column values,</p>

<pre><code> tr = soup.findChildren('tr')  
 for t in tr:  
     td = t.findChildren('td')   
     for child in td:   
        print(child.text)
</code></pre>

<p>again i get the whole data including the columns and values, what I expect is</p>

<pre><code> ['3259', 'ABC123', 'LAND']
           and
 ['3260', 'XYZ453','WATER'] 
</code></pre>

<p>by removing<br>
     <code>['3 - Design Reference','2 - Production Item']</code></p></li>
</ol>