Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

Open
helix-loop opened this issue Nov 14, 2021 · 3 comments
Open

Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

helix-loop opened this issue Nov 14, 2021 · 3 comments
Assignees
@mbushkov

Comments

@helix-loop
Copy link

helix-loop commented Nov 14, 2021 •
edited
Loading

Environment

  • How did you install GRR? [e.g. from release DEB, HEAD DEB, PIP package, source]
    PIP package/source
  • What GRR version are you running?: [e.g. 3.1.2.3]
    3.4.5.1
  • What operating system does the GRR server run on? [e.g. Ubuntu 18.04]
    Debian 11 (Bullseye) (ppc64le)
  • What operating system does the affected GRR client run on, if applicable? [e.g. Windows 10]

Describe the issue
Trying to install GRR Server from pip packages or via source, I am not able to install fleetspeak-client-bin version 0.1.11 on a ppc64le architecture.

Steps to reproduce:
pip3 install fleetspeak-client-bin

Error logs

ERROR: Could not find a version that satisfies the requirement fleetspeak-client-bin
ERROR: No matching distribution found for fleetspeak-client-bin

Additional context
There is a working (old) version of GRR Server (ppc64el) at https://packages.ubuntu.com/bionic/grr-server for Ubuntu 18.04.
@mbushkov
Copy link
Collaborator

Quick question: do you plan to run the clients on ppc64le as well?

fleetspeak-client-bin contains precompiled Fleetspeak binaries for various platforms and is needed to build and run latest generation of GRR clients. Unfortunately, we don't distribute the ppc64le Fleetspeak binaries in the fleetspeak-client-bin package.

At the moment, in order to install GRR and then build client templates for ppc64le, you'd need to effectively build and install your own ppc64le-specific fleetspeak-client-bin package version. To do that, you'd essentially need to follow the steps done by Fleetspeak CI.

Note that even if you do that, you'd still need to build the custom GRR client template for pcc64le too: https://grr-doc.readthedocs.io/en/latest/maintaining-and-tuning/building-custom-client-templates.html

@helix-loop
Copy link
Author

Answer: I do plan to run the clients on ppc64le as well.

I think I was able to follow the steps done by Fleetspeak CI and created fleetspeak-client_0.1.11_ppc64el.deb and fleetspeak-server_0.1.11_ppc64el.deb. I was able to install them on the server, however following https://grr-doc.readthedocs.io/en/latest/installing-grr-server/from-source.html#installing-python-3-grr-version-from-source I run into a few errors with dependencies (fleetspeak-client-bin and fleetspeak-server-bin) listed in several setup.py. I commented them out one by one and got further. I am now getting the following errors:

Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-x34f4rh2
       cwd: /tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/
  Complete output (81 lines):
  /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'distribution-name' will not be supported in future versions. Please use the underscore name 'distribution_name' instead
    warnings.warn(
  /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
    warnings.warn(
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-ppc64le-3.9
  copying _mysql_exceptions.py -> build/lib.linux-ppc64le-3.9
  creating build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/compat.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/connections.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/converters.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/cursors.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/release.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/times.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  creating build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/CLIENT.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/CR.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/ER.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/FLAG.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/REFRESH.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  running build_ext
  building '_mysql' extension
  creating build/temp.linux-ppc64le-3.9
  powerpc64le-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,3,10,'final',0) -D__version__=1.3.10 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/path/xyz/INSTALL/include -I/usr/include/python3.9 -c _mysql.c -o build/temp.linux-ppc64le-3.9/_mysql.o
  In file included from _mysql.c:29:
  /usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
      3 | #warning This file should not be included by clients, include only <mysql.h>
        |  ^~~~~~~
  _mysql.c: In function ‘_mysql_server_init’:
  _mysql.c:266:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    266 |    s = PyUnicode_AsUTF8(item);
        |      ^
  _mysql.c:296:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    296 |    s = PyUnicode_AsUTF8(item);
        |      ^
  _mysql.c: In function ‘_mysql_ConnectionObject_Initialize’:
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:588:3: note: in expansion of macro ‘_stringsuck’
    588 |   _stringsuck(ca, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:589:3: note: in expansion of macro ‘_stringsuck’
    589 |   _stringsuck(capath, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:590:3: note: in expansion of macro ‘_stringsuck’
    590 |   _stringsuck(cert, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:591:3: note: in expansion of macro ‘_stringsuck’
    591 |   _stringsuck(key, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:592:3: note: in expansion of macro ‘_stringsuck’
    592 |   _stringsuck(cipher, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c: In function ‘_mysql_ConnectionObject_ping’:
  _mysql.c:1911:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
   1911 |  if ( reconnect != -1 ) self->connection.reconnect = reconnect;
        |                                         ^
  _mysql.c: In function ‘_mysql_ConnectionObject_getattro’:
  _mysql.c:2603:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
   2603 |  cname = PyUnicode_AsUTF8(name);
        |        ^
  error: command '/usr/bin/powerpc64le-linux-gnu-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for mysqlclient
  Running setup.py clean for mysqlclient
Failed to build mysqlclient
Installing collected packages: mysqlclient, grr-response-server
  Attempting uninstall: mysqlclient
    Found existing installation: mysqlclient 2.1.0
    Uninstalling mysqlclient-2.1.0:
      Successfully uninstalled mysqlclient-2.1.0
    Running setup.py install for mysqlclient ... error
    ERROR: Command errored out with exit status 1:
     command: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoyz__bc/install-record.txt --single-version-externally-managed --compile --install-headers /path/xyz/INSTALL/include/site/python3.9/mysqlclient
         cwd: /tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/
    Complete output (83 lines):
    /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'distribution-name' will not be supported in future versions. Please use the underscore name 'distribution_name' instead
      warnings.warn(
    /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
      warnings.warn(
    running install
    /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.linux-ppc64le-3.9
    copying _mysql_exceptions.py -> build/lib.linux-ppc64le-3.9
    creating build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/compat.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/connections.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/converters.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/cursors.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/release.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/times.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    creating build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/CLIENT.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/CR.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/ER.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/FLAG.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/REFRESH.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    running build_ext
    building '_mysql' extension
    creating build/temp.linux-ppc64le-3.9
    powerpc64le-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,3,10,'final',0) -D__version__=1.3.10 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/path/xyz/INSTALL/include -I/usr/include/python3.9 -c _mysql.c -o build/temp.linux-ppc64le-3.9/_mysql.o
    In file included from _mysql.c:29:
    /usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
        3 | #warning This file should not be included by clients, include only <mysql.h>
          |  ^~~~~~~
    _mysql.c: In function ‘_mysql_server_init’:
    _mysql.c:266:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      266 |    s = PyUnicode_AsUTF8(item);
          |      ^
    _mysql.c:296:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      296 |    s = PyUnicode_AsUTF8(item);
          |      ^
    _mysql.c: In function ‘_mysql_ConnectionObject_Initialize’:
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:588:3: note: in expansion of macro ‘_stringsuck’
      588 |   _stringsuck(ca, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:589:3: note: in expansion of macro ‘_stringsuck’
      589 |   _stringsuck(capath, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:590:3: note: in expansion of macro ‘_stringsuck’
      590 |   _stringsuck(cert, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:591:3: note: in expansion of macro ‘_stringsuck’
      591 |   _stringsuck(key, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:592:3: note: in expansion of macro ‘_stringsuck’
      592 |   _stringsuck(cipher, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c: In function ‘_mysql_ConnectionObject_ping’:
    _mysql.c:1911:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
     1911 |  if ( reconnect != -1 ) self->connection.reconnect = reconnect;
          |                                         ^
    _mysql.c: In function ‘_mysql_ConnectionObject_getattro’:
    _mysql.c:2603:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     2603 |  cname = PyUnicode_AsUTF8(name);
          |        ^
    error: command '/usr/bin/powerpc64le-linux-gnu-gcc' failed with exit code 1
    ----------------------------------------
  Rolling back uninstall of mysqlclient
  Moving to /path/xyz/INSTALL/lib/python3.9/site-packages/MySQLdb/
   from /path/xyz/INSTALL/lib/python3.9/site-packages/~ySQLdb
  Moving to /path/xyz/INSTALL/lib/python3.9/site-packages/mysqlclient-2.1.0.dist-info/
   from /path/xyz/INSTALL/lib/python3.9/site-packages/~ysqlclient-2.1.0.dist-info
ERROR: Command errored out with exit status 1: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoyz__bc/install-record.txt --single-version-externally-managed --compile --install-headers /path/xyz/INSTALL/include/site/python3.9/mysqlclient Check the logs for full command output.

Do I already need an existing MySQL/MariaDB setup at this point? (travis/install.sh) Is there another missing dependency?

Is Ubuntu 18.04 still the recommended distribution to install the server? I would like to use a distribution and version still getting updates for the GRR Rapid Response setup.

@mbushkov mbushkov mentioned this issue Nov 24, 2021
Merged
@mbushkov
Copy link
Collaborator

Hey, our docs on the matter were a little outdated, I created a PR (google/grr-doc#144) to fix this.

Before the PR gets submitted, the temp version of the docs is available here: https://grr-doc.readthedocs.io/en/sphinx-update/installing-grr-server/from-source.html

Commenting on your questions/notes:

Answer: I do plan to run the clients on ppc64le as well.

Is it a hard requirement to run the server on the ppc64le architecture? That's not impossible but is much more work, as ppc64le is not a tested scenario and we don't provide prebuilt server Debs for this architecture. A possible option might be: run the server on x86-64 Ubuntu 18 and build a custom client template for ppc64le (building a custom client template doesn't involve installing mysql and building the server pip, thus making the process much simpler).

Do I already need an existing MySQL/MariaDB setup at this point? (travis/install.sh) Is there another missing dependency?

The error that you see in the build are not a sign of a missing dependency, but rather a build error inside the mysql-client package. I found similar mentions of the same problem: DefectDojo/django-DefectDojo#407

You don't need to have an initialized database, but you need to have the headers/libraries (which you seem to have, otherwise the build would fail totally).

Which database did you install - MySQL or MariaDB?

Is Ubuntu 18.04 still the recommended distribution to install the server? I would like to use a distribution and version still getting updates for the GRR Rapid Response setup.

Yes, Ubuntu 18.04 is the server environment that we support and test on.

mbushkov added a commit to google/grr-doc that referenced this issue Nov 24, 2021
@mbushkov
Minor documentation overhaul (#144)
9c2bebf 
* Updated Sphinx. Migrated from recommonmark to myst-parser. Removed a few hack from conf.py.
* Removed most mentions of AFF4 from the docs.
* Removed Python 2-specific sections.
* Fixed the docs to suggest to change mysql server config file to set max_allowed_packet to 40M (see google/grr#947).
* Largely rewritten "Install from source": now telling how to build Fleetspeak wheels and specifically mentioning the case when we install from source to build a client template (see google/grr#948).
* Made minor changes to "Build custom client templates".

You can see the version of the docs from this PR at https://grr-doc.readthedocs.io/en/sphinx-update/index.html.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mbushkov mbushkov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mbushkov @helix-loop