Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip-compile fails with pip==22.0 #1558

Closed
jordan-dimov opened this issue Jan 30, 2022 · 22 comments · Fixed by #1567
Closed

pip-compile fails with pip==22.0 #1558

jordan-dimov opened this issue Jan 30, 2022 · 22 comments · Fixed by #1567
Labels
bug Something is not working pip Related to pip

Comments

@jordan-dimov
Copy link

I upgraded Pip to latest version 22.0 today and pip-tools stopped working.

When I run pip-compile to upgrade my requirements, I now get the following stack trace:

$ pip-compile -U requirements/requirements.in
Traceback (most recent call last):
  File "/Users/jordan/.virtualenvs/my-project/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/piptools/scripts/compile.py", line 342, in cli
    repository = PyPIRepository(pip_args, cache_dir=cache_dir)
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 106, in __init__
    self._setup_logging()
  File "/Users/jordan/.virtualenvs/my-project/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging
    assert isinstance(handler, logging.StreamHandler)
AssertionError

As soon as I downgrade pip to v. 21.3.1 for example, the above command succeeds as expected.

Environment Versions

  1. OS Type: MacOS
  2. Python version: 3.7.10
  3. pip version: 22.0
  4. pip-tools version: 6.4.0
@hugovk
Copy link
Member

hugovk commented Jan 30, 2022

pypa/pip#10726 (comment):

That looks like a bug on their end -- they're using pip's internals (something that is not supported) and quite possibly they weren't prepared for pypa/pip#10461. :)

@fsecada01
Copy link

I pushed a merge request that fixes the issue for me, but it's failing tests.

#1559

@lsaavedr
Copy link

this is a big bug 🤯

matthewfeickert added a commit to illinois-mla/phys-398-mla-image that referenced this issue Jan 31, 2022
* Add mls library from GitHub to allow for helper utilities like locate_data
   - Use hashable URL for piptools to generate hashes (of the form
     https://github.com/<repo org>/<repo name>/archive/<commit hash>.zip)
   - Add emcee as required by mls
* Add tables as required by pandas.read_hdf()
* Use stable base image tag jupyter/minimal-notebook:lab-3.2.8
* Restrict pip<22.0 given pip-tools bug jazzband/pip-tools#1558
joshringer added a commit to joshringer/pip-tools that referenced this issue Jan 31, 2022
@iflare3g
Copy link

I pushed a merge request that fixes the issue for me, but it's failing tests.

#1559
I left a comment as there's a typo, try to re-check if tests will pass after fixing it

Thanks for the merge request 🙏🏻

ronnix added a commit to Delegation-numerique-en-sante/mesconseilscovid that referenced this issue Jan 31, 2022
ronnix added a commit to Delegation-numerique-en-sante/mesconseilscovid that referenced this issue Jan 31, 2022
gentooboontoo added a commit to Kozea/gibolt that referenced this issue Jan 31, 2022
pip-tools is incompatible with pip>=22 for now. It is triggering an
error on install.

See issue jazzband/pip-tools#1558
@zbentley
Copy link

The workaround, for folks affected, is pip install pip<22 in the environment where you're using pip-tools.

@RAbraham
Copy link

I think we may have to do more than pip install pip<22? I just tried that but my next command to install pip-tools downloads pip 22 again?

pip install "pip<22"


pip install pip-tools


Collecting pip>=21.2
  Downloading pip-22.0.2-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from pip-tools) (57.0.0)
Collecting click>=7
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Requirement already satisfied: wheel in /usr/local/lib/python3.7/site-packages (from pip-tools) (0.36.2)
Collecting pep517
  Downloading pep517-0.12.0-py2.py3-none-any.whl (19 kB)
Collecting importlib-metadata
  Using cached importlib_metadata-4.10.1-py3-none-any.whl (17 kB)
Collecting typing-extensions>=3.6.4
  Using cached typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
  Using cached zipp-3.7.0-py3-none-any.whl (5.3 kB)
Collecting tomli>=1.1.0
  Downloading tomli-2.0.0-py3-none-any.whl (12 kB)
Installing collected packages: zipp, typing-extensions, tomli, importlib-metadata, pip, pep517, click, pip-tools
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed click-8.0.3 importlib-metadata-4.10.1 pep517-0.12.0 pip-22.0.2 pip-tools-6.4.0 tomli-2.0.0 typing-extensions-4.0.1 zipp-3.7.0

@RAbraham
Copy link

RAbraham commented Jan 31, 2022 via email

@antonagestam
Copy link

@RAbraham Or as commands, you need to either install pip-tools first and then downgrade pip, e.g. pip install pip-tools; pip install 'pip<22'. Or just provide the limitation in the same command pip install pip-tools 'pip<22'. But yes, it probably makes more sense to put the limitations into your dependency spec file.

kmohrf pushed a commit to grouprise/grouprise that referenced this issue Jan 31, 2022
rra added a commit to lsst-sqre/gafaelfawr that referenced this issue Jan 31, 2022
Restrict pip to <22 for now until
jazzband/pip-tools#1558 is fixed.
@rsalmaso
Copy link

It seems that with pip>=22 pip-tools removes dash normalization https://pip.pypa.io/en/stable/news/#id73 even if it was done in pip==21.2 🙄

devmessias added a commit to devmessias/mlops_project that referenced this issue Jan 31, 2022
pip-downgrade should be removed
after `pip-tools` team's solution regarding pip log bug
be merged and released

Related to: jazzband/pip-tools#1558
@SethMMorton
Copy link

SethMMorton commented Feb 1, 2022

A similar occurrence happened when pip 21.3 was released last year: #1503.

On it there was some mention of making sure the pip-tools and pip maintainers stay in sync when breaking changes are made. Since it seems that pip is pretty liberal about refactoring their internals (which, to be clear, is their prerogative since it is internal) so that extra communication with pip-tools could help.

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella. One of the pypa's most important products uses pip-tools so I would imagine they would see value in ensuring it stays in sync with pip. (As a side note, the fact that warehouse uses pip-tools and not pipenv is interesting, to say the least, considering that pipenv is the officially recommended tool by the pypa...). This suggestion is mirroring the sentiment here.

@createdbykartik
Copy link

I was at pip version 21.0.1. Upgraded to 21.2 and re-ran docker-compose, worked fine!

m0wer added a commit to m0wer/cookiecutter-python-project that referenced this issue Feb 1, 2022
@hugovk
Copy link
Member

hugovk commented Feb 1, 2022

This repo does have a daily cron to test against pip main, which is really good.

https://github.com/jazzband/pip-tools/actions/workflows/cron.yml

However, the Python 3.6 job started failing three months ago when pip dropped support for 3.6:

image

image

Which masked all the jobs started failing two months ago due to other internal pip changes:

https://github.com/jazzband/pip-tools/runs/4195776918?check_suite_focus=true

image

https://github.com/jazzband/pip-tools/runs/4419708428?check_suite_focus=true


I'm not sure whether a change in organisation would necessarily help, the pip team will continue to change their private internal functions as they need to (and there will be more changes to use Rich more for better output).

Wherever it lives, I'd suggest a closer eye is paid to the integration of pip with pip-tools, especially in the lead up to pip's quarterly releases. (And look out for the pip release issues for planned release dates e.g. pypa/pip#10726.)

@davisagli
Copy link

It looks like it should be possible to adjust the code that is poking at pip's logging internals to skip it when running against a version of pip that uses Rich -- it appears to be a workaround for an issue with progress bars (#949) that does not occur with the Rich-based output.

fsecada01 added a commit to fsecada01/pip-tools that referenced this issue Feb 2, 2022
Re-inserted handler assertions; included RichPipStreamHandler from pip internal logging module.  

Fix idea taken from: jazzband#1558 (comment)
@hugovk
Copy link
Member

hugovk commented Feb 2, 2022

Please see #1567 for a fix.

@sbaack
Copy link

sbaack commented Feb 3, 2022

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella.

Nope. The fundamental problem here is that pip does not provide any public API -- see https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program. pip-tools is one of the few projects in the wild that still uses pip's internal code as if it's an API and this sort of breakage is exactly what the risk of that usage pattern is.

Just to add my 2 cents, I wish pip would add a compile and sync flag to effectively integrate pip-tools's functionality into standard pip. Then issues like this one here won't arise in stable pip releases and we would have good dependency management out of the box. Probably not going to happen, but one can dream :-)

alexryabtsev pushed a commit to django-stars/backend-skeleton that referenced this issue Feb 4, 2022
* Added -allow-unsafe flag to pip_compile in pip_tools.py

Without flag command "fab pip.sync" falling with error
"setuptools" version is not fixed.
https://github.com/jazzband/pip-tools/#deprecations

* Temp pip version downgrade in git workflow

Pip-tools has issue on pip version >=22.0
jazzband/pip-tools#1558
jansenk added a commit to openedx/edx-ora2 that referenced this issue Feb 4, 2022
@chrihartl
Copy link

chrihartl commented Feb 14, 2022

I updated pip and pip-tools to the latest versions (22.0.3 and 6.5.1, respectively). Then I tried to run pip-compile:

pip-compile -vvv -U --no-emit-index-url requirements.in

But there is absolutely no output (also no error message), and the file requirements.txt does not get created (nor updated, if it exists).

When I downgrade pip-tools to 6.4.0 (but keep pip at version 22) I get an AssertionError.

It seems that the only way to fix my problem is to downgrade pip to version 21.3. Wasn't pip-tools>=6.5.0 supposed to fix (or work around) the (pip>=22) issue?

EDIT: I managed to get pip-compile working with the latest versions of pip and pip-tools, but this required me to perform the following tasks in the given order:

python -m pip install "pip<22"
pip uninstall pip-tools
pip install -U pip-tools
python -m pip install -U pip

odscjames added a commit to openownership/data-standard that referenced this issue Feb 15, 2022
Upgrade some for security

Upgrade pip-tools as older version doesn't work with newest pip
jazzband/pip-tools#1558

importlib_metadata is only needed on versions less than python 3.8.
This runs on 3.8 on RTD; see readthedocs.yml
Previous versions of this file must have been compiled with older
versions of Python
odscjames added a commit to openownership/data-standard that referenced this issue Feb 15, 2022
Upgrade some for security

Upgrade pip-tools as older version doesn't work with newest pip
jazzband/pip-tools#1558

importlib_metadata is only needed on versions less than python 3.8.
This runs on 3.8 on RTD; see readthedocs.yml
Previous versions of this file must have been compiled with older
versions of Python
odscjames added a commit to openownership/data-standard that referenced this issue Feb 15, 2022
Upgrade some for security

Upgrade pip-tools as older version doesn't work with newest pip
jazzband/pip-tools#1558
This version reorders some entries, but the content is unchanged

importlib_metadata & zipp is only needed on versions less than python 3.8.
This runs on 3.8 on RTD; see readthedocs.yml
Previous versions of this file must have been compiled with older
versions of Python
cfm added a commit to freedomofpress/securedrop-client that referenced this issue Feb 25, 2022
1. Pillow: for CVE-2022{22817,24303}
2. pip-tools: for jazzband/pip-tools#1558
cfm added a commit to freedomofpress/securedrop-client that referenced this issue Feb 25, 2022
1. Pillow: for CVE-2022-{22817,24303}
2. pip-tools: for jazzband/pip-tools#1558
feanil added a commit to openedx-unsupported/edx-developer-docs that referenced this issue Mar 8, 2022
We got in a situation where the newest version of pip breaks the pinned
version of pip-tools from running correctly.

jazzband/pip-tools#1558

The issue is fixed in newer version of pip-tools so the only way to
resolve this is to first manually upgrade pip-tools and re-compile the
pip-tools.txt requirements file and then run `make upgrade`.
Er1kkkaaa pushed a commit to freedomofpress/securedrop-client that referenced this issue Mar 14, 2022
1. Pillow: for CVE-2022-{22817,24303}
2. pip-tools: for jazzband/pip-tools#1558
sssoleileraaa pushed a commit to gonzalo-bulnes/securedrop-client that referenced this issue Mar 15, 2022
1. Pillow: for CVE-2022-{22817,24303}
2. pip-tools: for jazzband/pip-tools#1558
MichaelRoytman added a commit to openedx/edx-submissions that referenced this issue Mar 29, 2022
The following comment is borrowed from this pull request, with changes: openedx/opaque-keys#218

This failure started happening in the automated requirements upgrade GitHub action:
https://github.com/openedx/edx-submissions/runs/5704384647?check_suite_focus=true

The bug was being caused by an interaction between pip and pip-tools, described here:
jazzband/pip-tools#1558

The bug is fixed in pip-tools==6.5.0 - from this PR:
jazzband/pip-tools#1567

This pull requests bumps the version of pip-tools manually to get the automated upgrades working again.
MichaelRoytman added a commit to openedx/edx-submissions that referenced this issue Mar 29, 2022
The following comment is borrowed from this pull request, with changes: openedx/opaque-keys#218

This failure started happening in the automated requirements upgrade GitHub action:
https://github.com/openedx/edx-submissions/runs/5704384647?check_suite_focus=true

The bug was being caused by an interaction between pip and pip-tools, described here: jazzband/pip-tools#1558

The bug is fixed in pip-tools==6.5.0 - from this pull request: jazzband/pip-tools#1567

This pull requests bumps the version of pip-tools manually to get the automated upgrades working again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is not working pip Related to pip
Projects
None yet
Development

Successfully merging a pull request may close this issue.