Thursday, July 1, 2010

WebBrowser and HttpWebRequest Cookies Problem and Solution

If you are working with WebBrowser control and you want to make some WebRequest you need to synchronize the cookies.
And that would be the naive way to do it:
httpWebRequest.CookieContainer.SetCookies(
    webBrowser.Document.Url, 
    webBrowser.Document.Cookie.Replace(';', ',')
    ); // THIS CODE IS WRONG!!

The problem is the HttpOnly cookies that are missing from Document.Cookie for security reasons.
The HttpOnly cookies are probably the most important ones that you need.
This article explains how to get them:
http://www.codeproject.com/KB/shell/RetrieveHttponlyCookies.aspx

Here is a more compact version of the function for C#:
[DllImport("wininet.dll", CharSet = CharSet.Auto, SetLastError = true)]
static extern bool InternetGetCookieEx(string pchURL, string pchCookieName, StringBuilder pchCookieData, ref uint pcchCookieData, int dwFlags, IntPtr lpReserved);
const int INTERNET_COOKIE_HTTPONLY = 0x00002000;

public static string GetGlobalCookies(string uri)
{
    uint datasize = 1024;
    StringBuilder cookieData = new StringBuilder((int)datasize);
    if (InternetGetCookieEx(uri, null, cookieData, ref datasize, INTERNET_COOKIE_HTTPONLY, IntPtr.Zero)
        && cookieData.Length > 0)
    {
        return cookieData.ToString().Replace(';', ',');
    }
    else
    {
        return null;
    }
}
And the usage would be:
httpWebRequest.CookieContainer.SetCookies(
    webBrowser.Document.Url, 
    GetGlobalCookies(webBrowser.Document.Url.AbsoluteUri)
    );

34 comments:

  1. Wonderful tip! I spent about four hours finding this at internet, even my googlfu is powerful(sorry that ;)) This is a very important tip for all people making web scraping!

    Thanks again!

    ReplyDelete
  2. Liberator ah!!!!
    Finally solved the trouble for a few days
    Thanks very very very

    ReplyDelete
  3. You are "Brilliant".

    I won't go into what site I was having issues with, however, it has frames("Recent Change") which now causes HttpWebRequest to fail on logon.

    Because of this, I was forced to use the WebBrowser control, to logon, however, I needed to download a file ("Once Logged On") an process that file using response data ("I do not wish to save the file to disk") just process the file data from the response.

    By using your method, I now can do this, using HttpWebRequest.

    ReplyDelete
  4. Thankssss Dude
    You Saved My Time

    ReplyDelete
  5. Many thanks! This saved me a lot of time. In my case I had to use the semicolon instead of comma.

    ReplyDelete
  6. How do you use this? it doesnt work for me

    I created a new Instance of

    WebBrowser webBrowser = new WebBrowser();
    webBrowser.Url = new Uri(url)

    httpWebRequest.CookieContainer.SetCookies(
    webBrowser.Document.Url,
    GetGlobalCookies(webBrowser.Document.Url.AbsoluteUri)
    );

    ReplyDelete
  7. How do you use this? it doesnt work for me
    please.help me.
    T_T

    ReplyDelete
  8. Thanks so much for you guide. It's really useful for me. Help me to save much time!

    ReplyDelete
  9. Wow, it is really fantastic, I even didn't think there 's a solution for this problem.

    ReplyDelete
  10. OMG thx so much! I was trying to grab login cookies from a WebBrowser to automate a process and didn't understand why WebBrowser.Document.Cookie didn't supply them all. :(

    Easy and perfect solution! =D

    ReplyDelete
  11. Nice. How i can to use WebResponse cookie in WebBrowser?

    ReplyDelete
  12. Awesome! I was almost frustrated not being able to catch the cookie which I saw over wire but WebBrowser object was not showing! Seems like the method you mentioned does the trick!

    ReplyDelete
  13. This is a god send. Thank you.

    Here's a vb.net for anyone who wants it.

    _
    Private Shared Function InternetGetCookieEx(ByVal pchURL As String, ByVal pchCookieName As String, ByVal pchCookieData As StringBuilder, ByRef pcchCookieData As UInteger, ByVal dwFlags As Integer, ByVal lpReserved As IntPtr) As Boolean
    End Function
    Const INTERNET_COOKIE_HTTPONLY As Integer = &H2000

    Public Shared Function GetGlobalCookies(ByVal uri As String) As String
    Dim datasize As UInteger = 1024
    Dim cookieData As New StringBuilder(CInt(datasize))
    If InternetGetCookieEx(uri, Nothing, cookieData, datasize, INTERNET_COOKIE_HTTPONLY, IntPtr.Zero) AndAlso cookieData.Length > 0 Then
    Return cookieData.ToString().Replace(";"c, ","c)
    Else
    Return Nothing
    End If
    End Function

    ReplyDelete
  14. Just wanted to say thank you, fixed my annoying problem! You are awesome!

    ReplyDelete
  15. Thank you.你的文章解决了我的问题,非常感谢!谢谢!

    ReplyDelete
  16. "Nice and good article.. it is very useful for me to learn and understand easily.. thanks for sharing your valuable information and time.. please keep updating.php jobs in hyderabad.
    "

    ReplyDelete
  17. i love you
    Como pasar las cookies de webbrouser a httpwebrequest

    ReplyDelete
  18. thank you so muuuuuuuuuuuuuuuuchhhhhhhhhh , i can solved my problem but at first when i use your code i had an error that the object reference is null on c# when i try to set cookies from webbrowser controler to httpwebrequest , then i found a solution that is :

    before use the blow code :

    PostRequest.CookieContainer.SetCookies(webBrowser1.Document.Url,GetGlobalCookies(webBrowser1.Document.Url.AbsoluteUri));


    use this code:
    PostRequest.CookieContainer = new CookieContainer();

    and then the problem solved.

    so thank you a lot again.

    ReplyDelete
  19. It returns the cookies when its not logged in, but once it is logged in, it returns empty, why ?

    ReplyDelete
  20. A bar chart or bar graph is a chart or graph that presents categorical data with rectangular bars with heights or lengths proportional to the values that they represent. Here are few Bar Graph Examples for consideration.

    ReplyDelete
  21. Very Informative and explanatory content. For more content : click here

    ReplyDelete