Is the reading of website meta tags generally prohibited?
<p><strong>Problem:</strong> I would like to create a plugin for my website. This Plugin makes it possible to recognise a shared link in a comment field. Afterwards the plugin fetch the link and scrape the meta data of the shared source (link). After successfully scraping the Link will reformat to a nice &quot;Card&quot; with thumbnail and description text (<em>Note 1</em>). A kind of card instead of just a link. Or better the way Google does it.</p>
<p><strong>Question:</strong> Is it allowed to &quot;scrape&quot; metadata without prior consent of the website operator?</p>
<p><strong>My thoughts on this:</strong> Since I am only scraping the metadata by machine and it is there to be used by search engine bots, social platforms anyway, that one is allowed to do it.</p>
<p><em>Note 1</em> : <em>(only pages that are public and explicitly allowed to be crawled by search engines).</em></p>
<h2>Update</h2>
<h3>Example for better understanding</h3>
<p>Below you can see an example of the metadata I read from a page. Below you can see the editing on my page with my own style.</p>
<pre><code>&lt;title&gt;Kanada: Abgeschossenes Flugobjekt hatte zylindrische Form - DER SPIEGEL&lt;/title&gt;
&lt;meta charset=&quot;utf-8&quot;&gt;
&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,initial-scale=1,user-scalable=no&quot;&gt;
&lt;meta name=&quot;MSSmartTagsPreventParsing&quot; content=&quot;true&quot;&gt;
&lt;meta http-equiv=&quot;imagetoolbar&quot; content=&quot;no&quot;&gt;
&lt;meta name=&quot;apple-itunes-app&quot; content=&quot;app-id=424881832&quot;&gt;
&lt;link rel=&quot;manifest&quot; href=&quot;https://www.spiegel.de/public/spon/json/manifest.json&quot;&gt;
&lt;meta name=&quot;theme-color&quot; content=&quot;#e64415&quot; media=&quot;(prefers-color-scheme: light)&quot;&gt;
&lt;meta name=&quot;theme-color&quot; content=&quot;#D03D12&quot; media=&quot;(prefers-color-scheme: dark)&quot;&gt;
&lt;meta name=&quot;google-site-verification&quot; content=&quot;lwpSnwoyvMkHS9nHmLBZuYOashAHfooOHSHeA_KS7ek&quot;&gt;
&lt;meta name=&quot;robots&quot; content=&quot;index, follow, noarchive, noodp, max-snippet:-1, max-image-preview:large, max-video-preview:-1&quot;&gt;
&lt;meta name=&quot;Googlebot-News&quot; content=&quot;index, follow&quot;&gt;
&lt;meta name=&quot;copyright&quot; content=&quot;DER SPIEGEL, Hamburg, Germany&quot;&gt;
&lt;meta name=&quot;email&quot; content=&quot;spiegel_online@spiegel.de&quot;&gt;
&lt;meta name=&quot;author&quot; content=&quot;DER SPIEGEL&quot;&gt;
&lt;meta name=&quot;date&quot; content=&quot;2023-02-12T07:08:39+01:00&quot;&gt;
&lt;meta name=&quot;last-modified&quot; content=&quot;2023-02-12T07:08:39+01:00&quot;&gt;
&lt;meta name=&quot;locale&quot; content=&quot;de_DE&quot;&gt;
&lt;meta name=&quot;description&quot; content=&quot;Es war innerhalb von zwei Tagen das zweite Objekt im Luftraum über Nordamerika, das von US-Kampfjets abgeschossen wurde. Nun gibt es erste Angaben der kanadischen Regierung, wie der Flugkörper aussah.&quot;&gt;
&lt;meta name=&quot;news_keywords&quot; content=&quot;Ausland, USA, Kanada, Justin Trudeau, Volksrepublik China, Spionage, Luftfahrt&quot;&gt;
&lt;meta name=&quot;twitter:card&quot; content=&quot;summary_large_image&quot;&gt;
&lt;meta name=&quot;twitter:site&quot; content=&quot;@derspiegel&quot;&gt;
&lt;meta name=&quot;twitter:title&quot; content=&quot;Erste Erkenntnisse der kanadischen Regierung: Abgeschossenes Flugobjekt hatte zylindrische Form&quot;&gt;
&lt;meta name=&quot;twitter:creator&quot; content=&quot;@derspiegel&quot;&gt;
&lt;meta name=&quot;twitter:image&quot; content=&quot;https://cdn.prod.www.spiegel.de/images/c431237d-0187-4cc6-95f6-c3bc8c12ede1_w1280_r1.77_fpx59.37_fpy50.jpg&quot;&gt;
&lt;meta property=&quot;og:title&quot; content=&quot;Erste Erkenntnisse der kanadischen Regierung: Abgeschossenes Flugobjekt hatte zylindrische Form&quot;&gt;
&lt;meta property=&quot;og:type&quot; content=&quot;article&quot;&gt;
&lt;meta property=&quot;og:url&quot; content=&quot;https://www.spiegel.de/ausland/kanada-abgeschossenes-flugobjekt-hatte-zylindrische-form-a-8134d1f8-6259-46d3-b1bc-0de4446380fc&quot;&gt;
&lt;meta property=&quot;og:image&quot; content=&quot;https://cdn.prod.www.spiegel.de/images/c431237d-0187-4cc6-95f6-c3bc8c12ede1_w1280_r1.77_fpx59.37_fpy50.jpg&quot;&gt;
&lt;meta property=&quot;og:description&quot; content=&quot;Es war innerhalb von zwei Tagen das zweite Objekt im Luftraum über Nordamerika, das von US-Kampfjets abgeschossen wurde. Nun gibt es erste Angaben der kanadischen Regierung, wie der Flugkörper aussah.&quot;&gt;
</code></pre>
<p>And now a screenshot of my page where the metadata is displayed.
<a href="https://i.stack.imgur.com/d6gyy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/d6gyy.png" alt="enter image description here" /></a></p>
<p><strong>What I store in my database?</strong> The content from the metatags: title, description, author, sourcelink, tags and the link to the thumbnail.</p>

<p>You are asking the wrong question. It should be: <em>When</em> you have downloaded the content and metadata, what are you allowed to do with it and what is forbidden?</p>
<p>Somebody owns the copyright to the text and images in the thumbnail. This could be the operator of the third party website, or that site has licensed the content from yet another party. You haven't licensed it from anybody. So you can <strong>watch</strong> the content in accordance with the TOS, and your computer can evaluate the metadata to do it, but you cannot <strong>display</strong> it on your own site.</p>
<p>Details will differ between jurisdictions, of course. You might also be held responsible for illegal content in the thumb you generate.</p>
<hr />
<p><em>Follow-up: There seems to be some question of what 'thumbnail' and 'card' mean in this context. This answer assumes a somewhat scaled-down representation of the content of the entire page, not just a collection of actual</em> metadata <em>like content length and expiry.</em></p>

<blockquote>
<p>Question: Is it allowed to &quot;scrape&quot; metadata without prior consent of the website operator?</p>
</blockquote>
<p>Yes, it is. The whole <em>point</em> of meta data is that you know what this page is about. If the page did not want copy-righted material in it, it would just not put it into the meta fields.</p>
<p>The preview you see when you paste a link into a chat window, for example in Teams or Facebook, is built on the <a href="https://ogp.me/" rel="nofollow noreferrer">Open Graph Protocol</a>, an <em>optional</em> part of a website that the owner can include if they <em>want</em> their page displayed in such a neat little preview.</p>
<p>This protocol is explicitely made to give a third party (for example Facebook) enough data to present said preview/thumbnail of the page. It is not limited to any one company, Facebook is just an example, <em>any</em> company can use it. This is what is was made for!</p>
<p>I would assume that the owner voluntarily giving any visitor and any third party the information how to view their link as a preview, in a structured way that assumes this will be used by third parties, automatically means they <em>want</em> the third party to show it this way.</p>
<p>A note: <em>only</em> metadata though. Leave the actual non meta data content alone. That is theirs, they gave it to their direct viewers in a way that the direct viewers can see it. The fact that you technically <em>can</em> scrape it does not give you permission to display it anywhere.</p>

<p>I used to scrape the web professionally for a search engine.</p>
<p>The Web is a consensual operation.  You send</p>
<pre><code> GET /page.html HTTP/1.1 
 Host: example.com 
</code></pre>
<p>And then their server does whatever the heck <em>it chooses</em> to do.   It can give you dead air, error return or content. They can't force terms &amp; conditions onto you merely for giving the above request; that is not a legally binding signature.</p>
<p>But once they give you content unauthenticated, they have no leg to stand on to claim that you &quot;did them wrong&quot;.   However, there are a few exceptions to that rule.</p>
<h2>Don't DOS them</h2>
<p>The problem is, when a stone cold novice writes their first web crawler, they usually write something like this:</p>
<pre><code> foreach url (list.of.URLs) {
    fetch url; 
    new.urls = searchTextForURLs (url); 
    append new.urls to list.of.urls; 
    sleep(5000);     # this line is forgotten
 }
</code></pre>
<p>This particular example is spidering, which is much worse. But regardless --</p>
<p>Invariably without fail, a novice's first URL scraper will forget to include that last line. Even if it's scraping a multitude of sites, it ends up doing a lot of consecutive URLs on a single site, and that can hammer a small site to death.  That is a DOS (Denial Of Service) and they have a cause of action.</p>
<h2>Don't get lost in infinite recursion</h2>
<p>It's common for sites to be designed, for completely legitimate reasons, to have an infinite namespace of URLs. You will never stop discovering URLs, just because of the way they structure them.  Or they may syndicate content, so their /shop/ space may entail Amazon's entire catalog.  Crawling this can lead to DOS, but even if you get that <code>sleep(5000)</code> line in there and pace yourself, doing a page every 5 seconds for eternity still adds up to significant server burden. And again they may have a cause of action.</p>
<h2>Respect robots.txt and other directives</h2>
<blockquote>
<p>&lt;meta name=&quot;robots&quot; content=&quot;index, follow, noarchive, noodp, max-snippet:-1, max-image-preview:large, max-video-preview:-1&quot;&gt;
</p>
</blockquote>
<p>Your crawler needs to parse robots.txt and similar on-page directives like the above Robots meta-tag, and respect what it is telling you.  They don't really have much standing to complain if you are crawling stuff they don't want crawled, yet, they &quot;forgot&quot; to exclude it in Robots.txt or other means.</p>
<p>Of course once they notice your crawler, the first thing they'll do is amend robots.txt to exclude it.  So check it <em>periodically</em>.  I've been on the receiving end of many a user complaint because we cached robots.txt (so we weren't hammering robots.txt with every query; that would be rude) ... as a result, their change didn't result in the <em>immediate</em> abatement of traffic that they were hoping for, and they had to wait a few hours.</p>
<p>If you want to be polite, it also helps to give your crawler a unique name, so they can call it out specifically.  In that case you need to respect robots.txt directives against that particular crawler name.</p>

<p>There have already been answers and they have not all definitively answered the question. I have thought about this again and would like to share it here. In my opinion, the best answer was from nvoigt. That's why I'm summarising it again and adding my thoughts.</p>
<p><strong>Basic consideration</strong></p>
<p>It can be assumed that anyone who puts a website on the internet and operates it wants to make the information accessible to the public. If you don't want to do that, you can protect your site with different possibilities: <code>html access, programmatic authentication etc</code>. Those who do not want &quot;hard&quot; protection may only want their site not to be indexed by search engines.</p>
<p>This is where <code>robots.txt</code> comes into play. With this you can exclude your entire page from the search engine index or only parts of it. By default, every page visited by a search engine robot is included in the index. Exceptions are, of course, websites that violate the guidelines of the search engine operator.</p>
<p>Metadata describe and give a short summary about the respective page.
The site operator can enrich the meta data with more information through special notations with more information. Page operators make such metatag optimizations in order to be found even better. For example, through appealing short descriptions.  To generate a better click through rate. The presence and use of so-called <code>OG tags</code> in the head area are also indicators that show that the site operator is actively spreading his content. This is an conclusive / implied act.</p>
<p>If you also explicitly mention in the <code>robots.txt</code> file that your page may be indexed, you also explicitly want the metadata to be read. The site operator can also explicitly exclude search engines and user agents. The bot operator should also respect this.</p>
<h3>Conclusion and answer</h3>
<p>If all factors are given, nothing stands in the way of reading out the metadata. And so I would claim that there is no copyright infringement.</p>
<p>But as soon as you earn money with the snippets on your page, another level comes into play. In Germany, it is called the Leistungsschutzgesetz für Verlage (ancillary copyright law for publishers). The Leistungsschutzgesetz is primarily understood as a right to protect investments. Ancillary copyright is also referred to as &quot;neighbouring rights&quot;. Sometimes also as &quot;neighbouring rights&quot;, in Anglo-American terms &quot;neighbouring rights&quot;.  Thus, neighbouring rights apply alongside copyrights. In Germany, publishers had complained that Google and Co. were making money from advertising by displaying snippets on their site. The publishers wanted a piece of that. A long and tough process ended with Google, Facebook and others paying the publishers a share of the revenue. So anyone who pursues all the above-mentioned points and non-commercial goals can relax here. But you should still not feel safe. Because as soon as the small and non-commercial site becomes successful (traffic), one or the other company would certainly get in touch with its lawyer.</p>
<h2>Final personal comment</h2>
<p>The answer of my question is yes and no. This example shows once again that the original idea of the internet. Free knowledge transfer for everyone no longer applies. It's always about stupid money.</p>

copyright internet ownership