Uploaded Test files

This commit is contained in:
Batuhan Berk Başoğlu 2020-11-12 11:05:57 -05:00
parent f584ad9d97
commit 2e81cb7d99
16627 changed files with 2065359 additions and 102444 deletions

View file

@ -0,0 +1 @@
pip

View file

@ -0,0 +1,24 @@
Copyright (c) 2019, threadpoolctl contributors
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View file

@ -0,0 +1,212 @@
Metadata-Version: 2.1
Name: threadpoolctl
Version: 2.1.0
Summary: threadpoolctl
Home-page: https://github.com/joblib/threadpoolctl
License: UNKNOWN
Author: Thomas Moreau
Author-email: thomas.moreau.2010@gmail.com
Requires-Python: >=3.5
Description-Content-Type: text/markdown
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Topic :: Software Development :: Libraries :: Python Modules
# Thread-pool Controls [![Build Status](https://dev.azure.com/joblib/threadpoolctl/_apis/build/status/joblib.threadpoolctl?branchName=master)](https://dev.azure.com/joblib/threadpoolctl/_build/latest?definitionId=1&branchName=master) [![codecov](https://codecov.io/gh/joblib/threadpoolctl/branch/master/graph/badge.svg)](https://codecov.io/gh/joblib/threadpoolctl)
Python helpers to limit the number of threads used in the
threadpool-backed of common native libraries used for scientific
computing and data science (e.g. BLAS and OpenMP).
Fine control of the underlying thread-pool size can be useful in
workloads that involve nested parallelism so as to mitigate
oversubscription issues.
## Installation
- For users, install the last published version from PyPI:
```bash
pip install threadpoolctl
```
- For contributors, install from the source repository in developer
mode:
```bash
pip install -r dev-requirements.txt
flit install --symlink
```
then you run the tests with pytest:
```bash
pytest
```
## Usage
### Command Line Interface
Get a JSON description of thread-pools initialized when importing python
packages such as numpy or scipy for instance:
```
python -m threadpoolctl -i numpy scipy.linalg
[
{
"filepath": "/home/ogrisel/miniconda3/envs/tmp/lib/libmkl_rt.so",
"prefix": "libmkl_rt",
"user_api": "blas",
"internal_api": "mkl",
"version": "2019.0.4",
"num_threads": 2,
"threading_layer": "intel"
},
{
"filepath": "/home/ogrisel/miniconda3/envs/tmp/lib/libiomp5.so",
"prefix": "libiomp",
"user_api": "openmp",
"internal_api": "openmp",
"version": null,
"num_threads": 4
}
]
```
The JSON information is written on STDOUT. If some of the packages are missing,
a warning message is displayed on STDERR.
### Python Runtime Programmatic Introspection
Introspect the current state of the threadpool-enabled runtime libraries
that are loaded when importing Python packages:
```python
>>> from threadpoolctl import threadpool_info
>>> from pprint import pprint
>>> pprint(threadpool_info())
[]
>>> import numpy
>>> pprint(threadpool_info())
[{'filepath': '/home/ogrisel/miniconda3/envs/tmp/lib/libmkl_rt.so',
'internal_api': 'mkl',
'num_threads': 2,
'prefix': 'libmkl_rt',
'threading_layer': 'intel',
'user_api': 'blas',
'version': '2019.0.4'},
{'filepath': '/home/ogrisel/miniconda3/envs/tmp/lib/libiomp5.so',
'internal_api': 'openmp',
'num_threads': 4,
'prefix': 'libiomp',
'user_api': 'openmp',
'version': None}]
>>> import xgboost
>>> pprint(threadpool_info())
[{'filepath': '/home/ogrisel/miniconda3/envs/tmp/lib/libmkl_rt.so',
'internal_api': 'mkl',
'num_threads': 2,
'prefix': 'libmkl_rt',
'threading_layer': 'intel',
'user_api': 'blas',
'version': '2019.0.4'},
{'filepath': '/home/ogrisel/miniconda3/envs/tmp/lib/libiomp5.so',
'internal_api': 'openmp',
'num_threads': 4,
'prefix': 'libiomp',
'user_api': 'openmp',
'version': None},
{'filepath': '/home/ogrisel/miniconda3/envs/tmp/lib/libgomp.so.1.0.0',
'internal_api': 'openmp',
'num_threads': 4,
'prefix': 'libgomp',
'user_api': 'openmp',
'version': None}]
```
In the above example, `numpy` was installed from the default anaconda channel and
comes with the MKL and its Intel OpenMP (`libiomp5`) implementation while
`xgboost` was installed from pypi.org and links against GNU OpenMP (`libgomp`)
so both OpenMP runtimes are loaded in the same Python program.
### Setting the Maximum Size of Thread-Pools
Control the number of threads used by the underlying runtime libraries
in specific sections of your Python program:
```python
from threadpoolctl import threadpool_limits
import numpy as np
with threadpool_limits(limits=1, user_api='blas'):
# In this block, calls to blas implementation (like openblas or MKL)
# will be limited to use only one thread. They can thus be used jointly
# with thread-parallelism.
a = np.random.randn(1000, 1000)
a_squared = a @ a
```
### Known Limitations
- `threadpool_limits` can fail to limit the number of inner threads when nesting
parallel loops managed by distinct OpenMP runtime implementations (for instance
libgomp from GCC and libomp from clang/llvm or libiomp from ICC).
See the `test_openmp_nesting` function in [tests/test_threadpoolctl.py](
https://github.com/joblib/threadpoolctl/blob/master/tests/test_threadpoolctl.py)
for an example. More information can be found at:
https://github.com/jeremiedbb/Nested_OpenMP
Note however that this problem does not happen when `threadpool_limits` is
used to limit the number of threads used internally by BLAS calls that are
themselves nested under OpenMP parallel loops. `threadpool_limits` works as
expected, even if the inner BLAS implementation relies on a distinct OpenMP
implementation.
- Using Intel OpenMP (ICC) and LLVM OpenMP (clang) in the same Python program
under Linux is known to cause problems. See the following guide for more details
and workarounds:
https://github.com/joblib/threadpoolctl/blob/master/multiple_openmp.md
## Maintainers
To make a release:
Bump the version number (`__version__`) in `threadpoolctl.py`.
Build the distribution archives:
```bash
pip install flit
flit build
```
Check the contents of `dist/`.
If everything is fine, make a commit for the release, tag it, push the
tag to github and then:
```bash
flit publish
```
### Credits
The initial dynamic library introspection code was written by @anton-malakhov
for the smp package available at https://github.com/IntelPython/smp .
threadpoolctl extends this for other operationg systems. Contrary to smp,
threadpoolctl does not attempt to limit the size of Python multiprocessing
pools (threads or processes) or set operating system-level CPU affinity
constraints: threadpoolctl only interacts with native libraries via their
public runtime APIs.

View file

@ -0,0 +1,7 @@
__pycache__/threadpoolctl.cpython-36.pyc,,
threadpoolctl-2.1.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
threadpoolctl-2.1.0.dist-info/LICENSE,sha256=gaxhkHUkiwblNmC2UtEOSF9GdfXQrg-X6iI3DaH34js,1507
threadpoolctl-2.1.0.dist-info/METADATA,sha256=TV0w8hoB1MWeKkQdVd72P_j2iufxSs6IqACS8e0vqQM,6565
threadpoolctl-2.1.0.dist-info/RECORD,,
threadpoolctl-2.1.0.dist-info/WHEEL,sha256=GLESb-AiO9TFjbXrEA7UrNl4dwWlkXoKZg6N6Vg0OJg,81
threadpoolctl.py,sha256=jJHsEALsi8S-25dbD6Nkw00MCOItBvahVU2gaY3qZ5o,30647

View file

@ -0,0 +1,4 @@
Wheel-Version: 1.0
Generator: flit 2.3.0
Root-Is-Purelib: true
Tag: py3-none-any