JSOUP - How to crawl a "login required" page using JSOUP
<p>I'm having trouble at crawling a determined website I wish to crawl. The problem is: after successfully logging in to that website I can't access a link which requires a valid login.</p>

<p>For example: </p>

<pre><code>public Document executeLogin(String user, String password) {
    try {
        Connection.Response loginForm = Jsoup.connect(url)
                .method(Connection.Method.GET)
                .execute();

        Document mainPage = Jsoup.connect(login-validation-url)
                .data("user", user)
                .data("senha", password)
                .cookies(loginForm.cookies())
                .post();

        Document evaluationPage = Jsoup.connect(login-required-url)
                .get();

       return evaluationPage;
    } catch (IOException ioe) {
        return null;
    }
</code></pre>

<p>What I do here is:</p>

<ul>
<li>Get the cookies from the login page, so I can login properly;</li>
<li>Then I post to the login validation url, which returns the main page after log in;</li>
<li>Finally I try to access the login required url after logging in to the main page, but that request returns me the login page, as if the session had expired.</li>
</ul>

<p>I know I have to store cookies to keep the session alive, but when I connect to the login validation url, it returns me a Document object, and there are no cookies to get from that object.</p>

<p>Is there any way to get the "session" created by the successful log in and send it within other Jsoup.connects? What I want to do, is to crawl a page that can only be accessed by logged users.</p>

<p>Thank you very much in advance.</p>