I needed to submit a form on a webpage using cURL. The form submission was using POST rather than GET. You can tell which method is being used by examining the source code for a page containing a form. If POST is being used, you will see it listed as the form method in the form tag as shown in the example below. A form that uses GET, instead, would have "get" as the form method.
<form method=post action=https://example.com/cgi-bin/SortDetail.pl>
You can use the -d or --data option with cURL
to use POST for a form submission.
-d/--data <data>
(HTTP) Sends the specified data in a POST request to the HTTP
server, in the same way that a browser does when a user has
filled in an HTML form and presses the submit button. This will
cause curl to pass the data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F/--form.
-d/--data is the same as --data-ascii. To post data purely
binary, you should instead use the --data-binary option. To URL-
encode the value of a form field you may use --data-urlencode.
If any of these options is used more than once on the same com-
mand line, the data pieces specified will be merged together
with a separating &-symbol. Thus, using '-d name=daniel -d
skill=lousy' would generate a post chunk that looks like
'name=daniel&skill=lousy'.
If you start the data with the letter @, the rest should be a
file name to read the data from, or - if you want curl to read
the data from stdin. The contents of the file must already be
URL-encoded. Multiple files can also be specified. Posting data
from a file named 'foobar' would thus be done with --data @foo-
bar.To submit the form using cURL, I used the following:
$ curl -u jsmith:SomePassword -d "Num=&Table=All&FY=&IP=&Project=&Service=&portNo=&result=request&display_number=Find+Requests" -o all.html https://example.com/cgi-bin/SortDetail.pl
In this case the website was password protected, so I had to use the
-u option to submit a userid and password in the form
-u userid:password. If you omit the :password
and just use -u userid, then cURL will prompt you for the password.
So, if you want to store the cURL command in a script, such as a
Bash script,
but don't want to store the password in the script, you can simply omit the
:password.
The -d option provides the parameters required by the
form and the values for those parameters, which were as follows in
this case:
| Parameter | Value |
|---|---|
| Num | |
| Table | All |
| FY | |
| IP | |
| Project | |
| Service | |
| portNo | |
| result | request |
| display_number | Find+Requests |
The format for submitting values for parameters is
parameter=value. Parameters are separated by an ampersand,
&.
URLs can only be sent over the Internet using the
ASCII character-set.
Special non ASCII characters, which include the space
character must be represented with a % followed by two
hexadecimal digits.
The space character can be represented by +
or by %20. So, though the value for "display_number" is
"Find Requests", it needs to be sent as Find+Requests or
Find%20 requests. You can see a list of other characters that
should be encoded at
URL Encoding.
In this case, I didn't need to specify values for many parameters in
the form, because I wanted the query to cover all potential values for
those parameters. So I can just use parameter= and
then follow that with an & to specify I am submitting the next
parameter in the list.
References:
-
cURL - Tutorial
cURL and libcurl -
curl Examples
Linux Journal | Linux Tips -
POST (HTTP)
Wikipedia, the free encyclopedia -
The POST Method
James Marshall's Home Page -
How to submit a form using PHP
HTML Form Guide - All about web forms! -
HTML URL Encoding
W3Schools Online Web Tutorials -
URL Encoding
Bloo's Home Page
