375 lines
10 KiB
ReStructuredText
375 lines
10 KiB
ReStructuredText
|
requests toolbelt
|
||
|
=================
|
||
|
|
||
|
This is just a collection of utilities for `python-requests`_, but don't
|
||
|
really belong in ``requests`` proper. The minimum tested requests version is
|
||
|
``2.1.0``. In reality, the toolbelt should work with ``2.0.1`` as well, but
|
||
|
some idiosyncracies prevent effective or sane testing on that version.
|
||
|
|
||
|
``pip install requests-toolbelt`` to get started!
|
||
|
|
||
|
|
||
|
multipart/form-data Encoder
|
||
|
---------------------------
|
||
|
|
||
|
The main attraction is a streaming multipart form-data object, ``MultipartEncoder``.
|
||
|
Its API looks like this:
|
||
|
|
||
|
.. code-block:: python
|
||
|
|
||
|
from requests_toolbelt import MultipartEncoder
|
||
|
import requests
|
||
|
|
||
|
m = MultipartEncoder(
|
||
|
fields={'field0': 'value', 'field1': 'value',
|
||
|
'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
|
||
|
)
|
||
|
|
||
|
r = requests.post('http://httpbin.org/post', data=m,
|
||
|
headers={'Content-Type': m.content_type})
|
||
|
|
||
|
|
||
|
You can also use ``multipart/form-data`` encoding for requests that don't
|
||
|
require files:
|
||
|
|
||
|
.. code-block:: python
|
||
|
|
||
|
from requests_toolbelt import MultipartEncoder
|
||
|
import requests
|
||
|
|
||
|
m = MultipartEncoder(fields={'field0': 'value', 'field1': 'value'})
|
||
|
|
||
|
r = requests.post('http://httpbin.org/post', data=m,
|
||
|
headers={'Content-Type': m.content_type})
|
||
|
|
||
|
|
||
|
Or, you can just create the string and examine the data:
|
||
|
|
||
|
.. code-block:: python
|
||
|
|
||
|
# Assuming `m` is one of the above
|
||
|
m.to_string() # Always returns unicode
|
||
|
|
||
|
|
||
|
User-Agent constructor
|
||
|
----------------------
|
||
|
|
||
|
You can easily construct a requests-style ``User-Agent`` string::
|
||
|
|
||
|
from requests_toolbelt import user_agent
|
||
|
|
||
|
headers = {
|
||
|
'User-Agent': user_agent('my_package', '0.0.1')
|
||
|
}
|
||
|
|
||
|
r = requests.get('https://api.github.com/users', headers=headers)
|
||
|
|
||
|
|
||
|
SSLAdapter
|
||
|
----------
|
||
|
|
||
|
The ``SSLAdapter`` was originally published on `Cory Benfield's blog`_.
|
||
|
This adapter allows the user to choose one of the SSL protocols made available
|
||
|
in Python's ``ssl`` module for outgoing HTTPS connections:
|
||
|
|
||
|
.. code-block:: python
|
||
|
|
||
|
from requests_toolbelt import SSLAdapter
|
||
|
import requests
|
||
|
import ssl
|
||
|
|
||
|
s = requests.Session()
|
||
|
s.mount('https://', SSLAdapter(ssl.PROTOCOL_TLSv1))
|
||
|
|
||
|
cookies/ForgetfulCookieJar
|
||
|
--------------------------
|
||
|
|
||
|
The ``ForgetfulCookieJar`` prevents a particular requests session from storing
|
||
|
cookies:
|
||
|
|
||
|
.. code-block:: python
|
||
|
|
||
|
from requests_toolbelt.cookies.forgetful import ForgetfulCookieJar
|
||
|
|
||
|
session = requests.Session()
|
||
|
session.cookies = ForgetfulCookieJar()
|
||
|
|
||
|
Known Issues
|
||
|
------------
|
||
|
|
||
|
On Python 3.3.0 and 3.3.1, the standard library's ``http`` module will fail
|
||
|
when passing an instance of the ``MultipartEncoder``. This is fixed in later
|
||
|
minor releases of Python 3.3. Please consider upgrading to a later minor
|
||
|
version or Python 3.4. *There is absolutely nothing this library can do to
|
||
|
work around that bug.*
|
||
|
|
||
|
Contributing
|
||
|
------------
|
||
|
|
||
|
Please read the `suggested workflow
|
||
|
<https://toolbelt.readthedocs.org/en/latest/contributing.html>`_ for
|
||
|
contributing to this project.
|
||
|
|
||
|
.. _Cory Benfield's blog: https://lukasa.co.uk/2013/01/Choosing_SSL_Version_In_Requests/
|
||
|
.. _python-requests: https://github.com/kennethreitz/requests
|
||
|
|
||
|
|
||
|
History
|
||
|
=======
|
||
|
|
||
|
0.7.0 -- 2016-07-21
|
||
|
-------------------
|
||
|
|
||
|
More information about this release can be found on the `0.7.0 milestone`_.
|
||
|
|
||
|
New Features
|
||
|
~~~~~~~~~~~~
|
||
|
|
||
|
- Add ``BaseUrlSession`` to allow developers to have a session that has a
|
||
|
"Base" URL. See the documentation for more details and examples.
|
||
|
|
||
|
- Split the logic of ``stream_response_to_file`` into two separate functions:
|
||
|
|
||
|
* ``get_download_file_path`` to generate the file name from the Response.
|
||
|
|
||
|
* ``stream_response_to_file`` which will use ``get_download_file_path`` if
|
||
|
necessary
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- Fixed the issue for people using *very* old versions of Requests where they
|
||
|
would see an ImportError from ``requests_toolbelt._compat`` when trying to
|
||
|
import ``connection``.
|
||
|
|
||
|
|
||
|
.. _0.7.0 milestone:
|
||
|
https://github.com/sigmavirus24/requests-toolbelt/milestones/0.7.0
|
||
|
|
||
|
0.6.2 -- 2016-05-10
|
||
|
-------------------
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- When passing a timeout via Requests, it was not appropriately translated to
|
||
|
the timeout that the urllib3 code was expecting.
|
||
|
|
||
|
0.6.1 -- 2016-05-05
|
||
|
-------------------
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- Remove assertion about request URLs in the AppEngineAdapter.
|
||
|
|
||
|
- Prevent pip from installing requests 3.0.0 when that is released until we
|
||
|
are ready to handle it.
|
||
|
|
||
|
0.6.0 -- 2016-01-27
|
||
|
-------------------
|
||
|
|
||
|
More information about this release can be found on the `0.6.0 milestone`_.
|
||
|
|
||
|
New Features
|
||
|
~~~~~~~~~~~~
|
||
|
|
||
|
- Add ``AppEngineAdapter`` to support developers using Google's AppEngine
|
||
|
platform with Requests.
|
||
|
|
||
|
- Add ``GuessProxyAuth`` class to support guessing between Basic and Digest
|
||
|
Authentication for proxies.
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- Ensure that proxies use the correct TLS version when using the
|
||
|
``SSLAdapter``.
|
||
|
|
||
|
- Fix an ``AttributeError`` when using the ``HTTPProxyDigestAuth`` class.
|
||
|
|
||
|
Miscellaneous
|
||
|
~~~~~~~~~~~~~
|
||
|
|
||
|
- Drop testing support for Python 3.2. virtualenv and pip have stopped
|
||
|
supporting it meaning that it is harder to test for this with our CI
|
||
|
infrastructure. Moving forward we will make a best-effort attempt to
|
||
|
support 3.2 but will not test for it.
|
||
|
|
||
|
|
||
|
.. _0.6.0 milestone:
|
||
|
https://github.com/sigmavirus24/requests-toolbelt/milestones/0.6.0
|
||
|
|
||
|
0.5.1 -- 2015-12-16
|
||
|
-------------------
|
||
|
|
||
|
More information about this release can be found on the `0.5.1 milestone`_.
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- Now papers over the differences in requests' ``super_len`` function from
|
||
|
versions prior to 2.9.0 and versions 2.9.0 and later.
|
||
|
|
||
|
|
||
|
.. _0.5.1 milestone:
|
||
|
https://github.com/sigmavirus24/requests-toolbelt/milestones/0.5.1
|
||
|
|
||
|
0.5.0 -- 2015-11-24
|
||
|
-------------------
|
||
|
|
||
|
More information about this release can be found on the `milestone
|
||
|
<https://github.com/sigmavirus24/requests-toolbelt/issues?utf8=%E2%9C%93&q=is%3Aall+milestone%3A0.5+>`_
|
||
|
for 0.5.0.
|
||
|
|
||
|
New Features
|
||
|
~~~~~~~~~~~~
|
||
|
|
||
|
- The ``tee`` submodule was added to ``requests_toolbelt.downloadutils``. It
|
||
|
allows you to iterate over the bytes of a response while also writing them
|
||
|
to a file. The ``tee.tee`` function, expects you to pass an open file
|
||
|
object, while ``tee.tee_to_file`` will use the provided file name to open
|
||
|
the file for you.
|
||
|
|
||
|
- Added a new parameter to ``requests_toolbelt.utils.user_agent`` that allows
|
||
|
the user to specify additional items.
|
||
|
|
||
|
- Added nested form-data helper,
|
||
|
``requests_toolbelt.utils.formdata.urlencode``.
|
||
|
|
||
|
- Added the ``ForgetfulCookieJar`` to ``requests_toolbelt.cookies``.
|
||
|
|
||
|
- Added utilities for dumping the information about a request-response cycle
|
||
|
in ``requests_toolbelt.utils.dump``.
|
||
|
|
||
|
- Implemented the API described in the ``requests_toolbelt.threaded`` module
|
||
|
docstring, i.e., added ``requests_toolbelt.threaded.map`` as an available
|
||
|
function.
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- Now papers over the API differences in versions of requests installed from
|
||
|
system packages versus versions of requests installed from PyPI.
|
||
|
|
||
|
- Allow string types for ``SourceAddressAdapter``.
|
||
|
|
||
|
0.4.0 -- 2015-04-03
|
||
|
-------------------
|
||
|
|
||
|
For more information about this release, please see `milestone 0.4.0
|
||
|
<https://github.com/sigmavirus24/requests-toolbelt/issues?q=milestone%3A0.4>`_
|
||
|
on the project's page.
|
||
|
|
||
|
New Features
|
||
|
~~~~~~~~~~~~
|
||
|
|
||
|
- A naive implemenation of a thread pool is now included in the toolbelt. See
|
||
|
the docs in ``docs/threading.rst`` or on `Read The Docs
|
||
|
<https://toolbelt.readthedocs.org>`_.
|
||
|
|
||
|
- The ``StreamingIterator`` now accepts files (such as ``sys.stdin``) without
|
||
|
a specific length and will properly stream them.
|
||
|
|
||
|
- The ``MultipartEncoder`` now accepts exactly the same format of fields as
|
||
|
requests' ``files`` parameter does. In other words, you can now also pass in
|
||
|
extra headers to add to a part in the body. You can also now specify a
|
||
|
custom ``Content-Type`` for a part.
|
||
|
|
||
|
- An implementation of HTTP Digest Authentication for Proxies is now included.
|
||
|
|
||
|
- A transport adapter that allows a user to specify a specific Certificate
|
||
|
Fingerprint is now included in the toolbelt.
|
||
|
|
||
|
- A transport adapter that simplifies how users specify socket options is now
|
||
|
included.
|
||
|
|
||
|
- A transport adapter that simplifies how users can specify TCP Keep-Alive
|
||
|
options is now included in the toolbelt.
|
||
|
|
||
|
- Deprecated functions from ``requests.utils`` are now included and
|
||
|
maintained.
|
||
|
|
||
|
- An authentication tool that allows users to specify how to authenticate to
|
||
|
several different domains at once is now included.
|
||
|
|
||
|
- A function to save streamed responses to disk by analyzing the
|
||
|
``Content-Disposition`` header is now included in the toolbelt.
|
||
|
|
||
|
Fixed Bugs
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
- The ``MultipartEncoder`` will now allow users to upload files larger than
|
||
|
4GB on 32-bit systems.
|
||
|
|
||
|
- The ``MultipartEncoder`` will now accept empty unicode strings for form
|
||
|
values.
|
||
|
|
||
|
0.3.1 -- 2014-06-23
|
||
|
-------------------
|
||
|
|
||
|
- Fix the fact that 0.3.0 bundle did not include the ``StreamingIterator``
|
||
|
|
||
|
0.3.0 -- 2014-05-21
|
||
|
-------------------
|
||
|
|
||
|
Bug Fixes
|
||
|
~~~~~~~~~
|
||
|
|
||
|
- Complete rewrite of ``MultipartEncoder`` fixes bug where bytes were lost in
|
||
|
uploads
|
||
|
|
||
|
New Features
|
||
|
~~~~~~~~~~~~
|
||
|
|
||
|
- ``MultipartDecoder`` to accept ``multipart/form-data`` response bodies and
|
||
|
parse them into an easy to use object.
|
||
|
|
||
|
- ``SourceAddressAdapter`` to allow users to choose a local address to bind
|
||
|
connections to.
|
||
|
|
||
|
- ``GuessAuth`` which accepts a username and password and uses the
|
||
|
``WWW-Authenticate`` header to determine how to authenticate against a
|
||
|
server.
|
||
|
|
||
|
- ``MultipartEncoderMonitor`` wraps an instance of the ``MultipartEncoder``
|
||
|
and keeps track of how many bytes were read and will call the provided
|
||
|
callback.
|
||
|
|
||
|
- ``StreamingIterator`` will wrap an iterator and stream the upload instead of
|
||
|
chunk it, provided you also provide the length of the content you wish to
|
||
|
upload.
|
||
|
|
||
|
0.2.0 -- 2014-02-24
|
||
|
-------------------
|
||
|
|
||
|
- Add ability to tell ``MultipartEncoder`` which encoding to use. By default
|
||
|
it uses 'utf-8'.
|
||
|
|
||
|
- Fix #10 - allow users to install with pip
|
||
|
|
||
|
- Fix #9 - Fix ``MultipartEncoder#to_string`` so that it properly handles file
|
||
|
objects as fields
|
||
|
|
||
|
0.1.2 -- 2014-01-19
|
||
|
-------------------
|
||
|
|
||
|
- At some point during development we broke how we handle normal file objects.
|
||
|
Thanks to @konomae this is now fixed.
|
||
|
|
||
|
0.1.1 -- 2014-01-19
|
||
|
-------------------
|
||
|
|
||
|
- Handle ``io.BytesIO``-like objects better
|
||
|
|
||
|
0.1.0 -- 2014-01-18
|
||
|
-------------------
|
||
|
|
||
|
- Add initial implementation of the streaming ``MultipartEncoder``
|
||
|
|
||
|
- Add initial implementation of the ``user_agent`` function
|
||
|
|
||
|
- Add the ``SSLAdapter``
|
||
|
|
||
|
|