hkaiser changed the topic of #ste||ar to: The topic is 'STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
K-ballo has quit [Read error: Connection reset by peer]
K-ballo has joined #ste||ar
K-ballo has quit [Read error: Connection reset by peer]
K-ballo has joined #ste||ar
K-ballo has quit [Read error: Connection reset by peer]
K-ballo has joined #ste||ar
K-ballo has quit [Read error: Connection reset by peer]
K-ballo has joined #ste||ar
K-ballo has quit [Read error: Connection reset by peer]
K-ballo has joined #ste||ar
hkaiser has quit [Quit: bye]
nikunj has quit [Ping timeout: 265 seconds]
nikunj has joined #ste||ar
nikunj97 has joined #ste||ar
<nikunj97>
simbergm1, yt?
<nikunj97>
you were telling the other day about the separate tcmalloc library
<nikunj97>
have you tried building hpx with it?
<rori>
He's on vacation until next wednesday (included I think)
<nikunj97>
rori, ohh ok
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
daissgr__ has quit [Remote host closed the connection]
<nikunj97>
btw is it just me, or the build time of hpx has increased after modularization?
<K-ballo>
I haven't particularly noticed, but it is expected to increase yes
<simbergm1>
nikunj97 back on Wednesday and no, haven't tried the other tcmalloc :/
<simbergm1>
I've noticed make prefers to build modules in order whereas ninja seems to happily do them out of order (i.e. faster)
nikunj has quit [Read error: Connection reset by peer]
<hkaiser>
will set up wordpress and create an account for you
diehlpk has joined #ste||ar
<diehlpk>
hkaiser, jbjnr simbergm1 heller1
<diehlpk>
Congratulations! Ste||ar group has been selected as a Google Summer of Code 2020 mentor organization
<diehlpk>
Nice work we have accomplished
<zao>
yay
<hkaiser>
\o/
nikunj has joined #ste||ar
K-ballo has quit [Read error: Connection reset by peer]
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
K-ballo has joined #ste||ar
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
nikunj has quit [Ping timeout: 255 seconds]
nikunj has joined #ste||ar
Abhishek09 has joined #ste||ar
<Abhishek09>
Hey, diehlpk are you here, i want to discuss a project with u.
<Abhishek09>
Congrats ste||ar 🔥❤️
<Abhishek09>
Is anyone here?
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
<hkaiser>
sure
<hkaiser>
just be a bit patient
K-ballo has quit [Read error: Connection reset by peer]
K-ballo has joined #ste||ar
nikunj has quit [Ping timeout: 260 seconds]
nikunj has joined #ste||ar
<Abhishek09>
ping diehlpk
<Abhishek09>
hkaiser When thre is good time to talk with diehlpk ?
<hkaiser>
Abhishek09: not sure, but I'm convinced he'll get back to you once he is available
<simbergm1>
diehlpk: very good news!
<simbergm1>
Abhishek09: you can also introduce yourself and write your questions on the mailing list (hpx-users@stellar.cct.lsu.edu)
<simbergm1>
it gives people a bit more time to reply
<Abhishek09>
simbergm1: But i have to discuss that require 2 way communication instantly
<Abhishek09>
one question is not enough for me
<simbergm1>
fair enough
<simbergm1>
you can also already write your question, and diehlpk might see it later
<simbergm1>
someone else might be able to respond as well
<Abhishek09>
ok I m intrested in making pip package for phylanx . What is priority of this project?
<diehlpk>
Abhishek09, Hi, just saw your message
<Abhishek09>
Thanks for reply
<diehlpk>
The challange for the pip packages are the dependencies of phylanx
<Abhishek09>
diehlpk now starts discussion
<Abhishek09>
Which version of python do need the .whl file and priority?
<Abhishek09>
Which os you want?
<diehlpk>
Some of the dependencies are C++ packages, e.g. boost, hwloc, jemalloc or tcmalloc are C++ only
<Abhishek09>
There is many question for you regarding this project
<diehlpk>
I would say the traget should be Linux, e.g. Fedora or Ubuntu
<diehlpk>
Maybe if things go easy Windows or Mac OS could be added later
<Abhishek09>
Priority of this project?
<diehlpk>
If you get it done on one Linux distribution it would be good
<diehlpk>
What do you mean by priority?
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
nikunj97 has joined #ste||ar
<Abhishek09>
There are so many projects under GSoC , may be you need other project due to lack of gsoc slots
<diehlpk>
We do not have any priority, we like to see that the student is dedicated to the project and can learn something new.
<Abhishek09>
that why i asking priority so that my reseach and work should not waste
<diehlpk>
if you think this is your favorite project, so you should apply for it
<diehlpk>
we only rate by the quality of the proposal and how you interacted with us for the final decision
<Abhishek09>
As we know .whl is python specfic . Which version you want?
<diehlpk>
> 3
<Abhishek09>
as there diff .whl for 3.6 and 3.7, 3.8
<Abhishek09>
Which ci you want want to use travis or azure-pipeline?
<Abhishek09>
No problem for deps. We can deal easily with cmake,pip, or dnf
K-ballo has quit [Read error: Connection reset by peer]
<Abhishek09>
As you have already build dnf package of deps . Thats nice and it will be helpful for us .
<Abhishek09>
ping diehlpk
<diehlpk>
Abhishek09, You do not need to ping me, I will answer once I have time
<zao>
If you feel that the choice of CI might matter, it might be something to investigate. I'd reckon that they're mostly interchangeable as the process of building a wheel is mostly the same.
<diehlpk>
We all work on our daily job and need some time to respond
<diehlpk>
I would say that the .whl should not depend to much on python version or?
<Abhishek09>
zao : yes , same
<zao>
Figuring out if wheels are portable across minor versions or if you need one per version is probably something to figure out.
<zao>
As is how/if you can bundle dependencies in wheels.
<Abhishek09>
diehlpk: If you busy busy right now, i will contact tommorrow. Does this makes sense?
<zao>
I haven't read the proposal - is it about using pypi, or building wheels for a private wheelhouse?
<Abhishek09>
@zao : i think you r intrested in my project
<Abhishek09>
If you have any question you can ask
<diehlpk>
Abhishek09, No, I am not busy but just need some time to respond
<diehlpk>
We would prefer to use circle-ci, since we use it for phylanx and hpx
<Abhishek09>
Ok , then i will wait
<diehlpk>
But there is no need to have any build system. Just a whl file and a bash script to test things would work
<diehlpk>
Keep it simple and if you have time left, add features
<Abhishek09>
i think circleci have limited no of builds. Am i right?
<zao>
I'd reckon that as long as whatever method you come up with for building wheels is automatable, it can run in any CI already used.
<Abhishek09>
diehlpk: which tools you want to use manylinux or cibuildwheels?
<Abhishek09>
I decided to make wheel for fedora as your choice
K-ballo has joined #ste||ar
<Abhishek09>
we will use dnf/cmake or pypa package(if available)
<Abhishek09>
@zao
<Abhishek09>
which tools suit you best of 2 @zao
<zao>
Abhishek09: I don't know anything about HPX/Phylanx requirements in particular, whatever works to produce something that can be pip installed I guess?
<zao>
I'm here mostly as someone that has had to install a lot of Python software at work.
<Abhishek09>
i m talking about manylinux that seems best option i think
<zao>
It seems like manylinux2010 packages has matured now and might be publishable to pypi. manylinux2014 also seems to exist. Investigating what benefits and drawbacks those would have might be helpful when trying to deploy Phylanx.
<Abhishek09>
diehlpk: As manylinux and cibuildwheel use the same docker
<Abhishek09>
AFAIK
<nikunj97>
zao, Hey! Long time
<nikunj97>
how're you doing?
<zao>
nikunj97: Hey!
<zao>
All good, still doing my HPC thing.
<zao>
Recently got tricked into maintaining EasyBuild :D
<nikunj97>
xD
<zao>
How's things for you?
<diehlpk>
I think last year the gcc was the easy for manylinux
<Abhishek09>
not understand. please expain
<diehlpk>
HPX/Phylanx need a recently new gcc
<diehlpk>
and I thinlk they shipped gcc 4.8
<diehlpk>
Also we only have a hpx package for Fedora
<diehlpk>
Abhishek09, I would recommend to do baby steps
<diehlpk>
So your proposal can have many milestones
<Abhishek09>
Ok , we do for fedora first
<diehlpk>
Easiest thing would be just do write a whl file for python 3 on Fedora
<diehlpk>
and install it via pip using the file
<diehlpk>
next thinh could be do add the generating of the whatever you need to circle-ci
<diehlpk>
*thing
<Abhishek09>
it can be done by setup.py
<diehlpk>
Yes, but keep in mind that you have to build things and check for dependencies
<Abhishek09>
first we deal with all deps that we require for deps for deps
<diehlpk>
Yes, but this is the tricky part
<diehlpk>
You might want to compile HPX and Phylanx first to see the dependencies
<Abhishek09>
we simple install the library and bundled them into file and then build the wheel
<Abhishek09>
for phylanx
<diehlpk>
Ok, but you need to compile all dependencies, which is not easy
<Abhishek09>
I think dnf package will be helpful for us
<diehlpk>
boost, jemalloc, hwloc, hpx, pybind, and so on
<diehlpk>
Yes, but only on fedora
<diehlpk>
For Ubuntu it is much more difficult
<Abhishek09>
but our mission is to complete for one os then move for other acc to your commitment
<Abhishek09>
Fedora then ubuntu then macOS
<Abhishek09>
So we concentrate on fedora first as this is your preference AFAIK
<zao>
Compiling is just one part too, actually bundling dependencies with the thing so it can be installed and run on a system lacking the dependencies would be another.
<zao>
If that's a requirement, of course. Something to figure out :)
<diehlpk>
Abhishek09, Sure, let us start with Fedora
<Abhishek09>
Ubuntu - is rpm does not exits for all deps?
<diehlpk>
Ubunut uses deb
<diehlpk>
and we only have a rpm for fedora in the official repo
<zao>
Some packages may be missing. Some may be too old. Some may not be built in the way that we need.
<zao>
One of the things that Ubuntu and (assumedly) CentOS doesn't have is a HPX package.
<Abhishek09>
diehlpk i think you created the some package for ubuntu
<diehlpk>
No, only for Fedora
<diehlpk>
I just maintain the Fedora package
<zao>
For something like manylinux{1,2010,2014}, you're even worse off.
<Abhishek09>
diehlpk: Nice , i have seen your personal website , you package will be very helful to me
<Abhishek09>
helpful. thanks a lot
<diehlpk>
Anyway manylinux ships the gcc version is 4.8.2
<diehlpk>
So HPX and Phylanx will not even compile
<diehlpk>
Abhishek09, I think you should investigate how to ship C++ libraries with pip
<nikunj97>
zao, things are going great!
<nikunj97>
working on benchmarking hpx on arm these days
<nikunj97>
with people in germany
<diehlpk>
Another thing is that the fedora package is compiled generic and will not provide optimal performance
<Abhishek09>
Which version of gcc we need ?
<zao>
Ooh, sounds like fun.
<diehlpk>
I think 4.9 but gcc 6 or 7 would be better
<nikunj>
zao, yup sure is!
<zao>
diehlpk: I'm not sure if it's documented anywhere, but Compute Canada builds several generic variations for their nation-wide wheelhouses. Off the top of my head, it's like sse2, avx, avx2, avx512 or something.
<zao>
For them it's an acceptable tradeoff between perf and variations.
<diehlpk>
Abhishek09, I really recommend to build HPX and Phylanx first to get some overview and experience on what is needed
<diehlpk>
Without pip or any setup.py first
<diehlpk>
Just use your machine or any docker container and build things
<diehlpk>
After that you will see how tricky things get
<diehlpk>
it will take you few days and you will see how things are get complicated.
<Abhishek09>
Yes i know i need install deps first in order to install phylanx .Building will take so much time
<diehlpk>
zao, I think we can accept the trade-off for now, but we should have it in mind if we go for distributed runs on a cluster
<diehlpk>
Abhishek09, it is not about the time, it is more about getting all things to match
hkaiser has quit [Quit: bye]
<diehlpk>
I would recommend two things: 1) build phylanx and run some tests 2) read about pip packages for non python dependencies
<Abhishek09>
It is necessary to contribute to participate in this organisation?
<nikunj>
Abhishek09, building basic build of hpx isn't very complicated. Building a specific build though can get mind boggling.
<zao>
(a tip if you work outside of a sandbox, don't forget to clean $HOME/.local every now and then from installed packages :D )
<diehlpk>
Abhishek09, We will expect students to demonstrate that they have the required level of C++ and CMake knowledge by showing us some of their previous work (e.g. a github repository), or preferably, by them making a small demonstration program using HPX that shows a simple example of something they have created themselves.
* zao
eyes `/home/zao/.local/lib/python3.6/site-packages/phylanx-0.0.1-py3.6-linux-x86_64.egg`
<diehlpk>
This is the only requirement
<diehlpk>
For example a parallel matrix vector multiplication would be one example
<diehlpk>
I would recommend that you will take few days to compile phylanx and look into the packaging of C++ dependencies
<Abhishek09>
diehlpk: i know C++ and cmake
<Abhishek09>
but i didn't have any work of cmake but i use
<diehlpk>
Abhishek09, Everyone knows, but on which level?
<diehlpk>
Writing a hello world is easy wrtiting your own linked list not.
<diehlpk>
So, we like to see that students have compiled hpx and wrote some small example. From previous experience most students missed the first two weeks because they had struggle to compile hpx
<diehlpk>
Ok, I would say you do your homework and we can discuss next week again
<Abhishek09>
C++ is used most and i used cmake something for bulding but not much
<diehlpk>
If you have issues you can come here and ask for help
<Abhishek09>
my homework - only phylanx and investigate about packaging of C++ deps
<Abhishek09>
diehlpk
<diehlpk>
Yes, please try to install hpx and phylanx on any Linux distribution and run some example code
<diehlpk>
next week we can discuss more about shipping C++ dependencies using pip. If you found some interesting articles or documentation you can post it here.
<Abhishek09>
ok , next weeks means Monday?
<diehlpk>
I am out next week, but people here are happy to discuss with you
<diehlpk>
parsa, heller1, hkaiser, simbergm1 zao Are good people to ask once I am away
<Abhishek09>
can i try on MacOS?
<Abhishek09>
catlina
<Abhishek09>
diehlpk
<Abhishek09>
Ok , when you back again?
rtohid has joined #ste||ar
hkaiser has joined #ste||ar
<diehlpk>
Abhishek09, The week after, but hkaiser and rtohid can answer questions related to phylanx
<Abhishek09>
can i try on MacOS?
<diehlpk>
Sure, I have no experience to build hpx and phylanx there
<hkaiser>
Abhishek09: worth a try, i might work, it might not - we have never tried
<diehlpk>
I am not sure if anyone has ever build phylanx on OS x
<Abhishek09>
I have build phalanx on macOS but 1 year ago on high sirrra
<Abhishek09>
but now i m on catalina
<diehlpk>
Ok, so try it again
<Abhishek09>
Previous year i saw the same project but not accepted into gsoc
<diehlpk>
Yes, last year we were not accepted
<zao>
I found a GSoC table at FOSDEM this year, was surprised to not see HPX in the list of orgs they had printed out :)
<Abhishek09>
zao: link please?
<zao>
Abhishek09: A physical table in person at the FOSDEM conference in Brussels :)
<Abhishek09>
Oo i forget my org `Aboutcode` is there
<Abhishek09>
scancode
<Abhishek09>
diehlpk i think we can ship boostmpi via `install requires` or `requirement.txt'
<diehlpk>
Ok, but it might be easier to do a shared memory package first
<Abhishek09>
What doest it meant?
<Abhishek09>
shared repository across repository
<Abhishek09>
diehlpk: We can use custom cmdclasss in setup.py to complile that calls your other build system. In llvmlite, they call out to CMake,
<rtohid>
@Abhishek09, I'm a bit late to the party but here are the requirements for this project:
<rtohid>
1. deliverable: pip install phalanx working on Linux.
<rtohid>
2. reproducibility: the above command must work on two Dockerfiles, one based on Fedora and the other Ubuntu.
<rtohid>
3. software versions: gcc7+ (@hkaiser please correct me if I am wrong) and Python 3.8+
<Abhishek09>
rtohid but diehlpk said that first we create .whl for one os then we move for other os
<Abhishek09>
first fedora
<rtohid>
The order is up to you ;).
<Abhishek09>
what you idea behind this
<rtohid>
what do you mean by "this"?
<Abhishek09>
this project
<Abhishek09>
I think dnf package will be helpful for us
<Abhishek09>
i will manylinux for that
<Abhishek09>
use
<rtohid>
HPX and Phylanx are complex libraries intended for HPC systems where performance matter.
<diehlpk>
Abhishek09, manylinux will not work
<Abhishek09>
why?
<diehlpk>
unless you will compile your own gcc
<zao>
manylinux1 is ancient. manylinux2010 may still be too old
<zao>
manylinux2014 I don't think has a pypi tag yet.
<diehlpk>
manylinux has gcc 4.8
<rtohid>
This also means that users may not have dnf / apt privilege
<diehlpk>
And we need at least 4.9
<diehlpk>
I would say using dnf to install hpx and use it to build phylanx is fine for the first try
<rtohid>
@diehlpk for phylanx it might be even higher
<zao>
diehlpk: manylinux2010 leverages devtoolset-8 on centos6, which if I read it right might be 8.2.1
<diehlpk>
So if you were able to use the Fedora package, compile phylanx, and have a pip package this is a huge step
<Abhishek09>
we will use gcc as pypa pyckage in install require section
<Abhishek09>
diehlpk
<diehlpk>
After that you could start to add the dependencies
<rtohid>
@diehlpk, definitely.
<zao>
Will that help you with distribution tho?
<diehlpk>
Abhishek09, installing gcc will not work easily if you use the Fedora package
<Abhishek09>
cibuildwheel is fine?
<diehlpk>
You have to use the same gcc version
<diehlpk>
if the fedora package is build with gcc 9, you can not use 8.2.1 to build phylanx
<diehlpk>
I would say the way to go could be to have a docker file for Ubuntu and Fedora to build the pip package
<zao>
The primary challenge as I see it is not to build the software, but to build it in a way that it works when installed on the target machine without more system packages than normal.
<diehlpk>
yes, perfectly summarized zao
<rtohid>
@Abhishek09, cibuildwheel only works on azure and travis, so maybe not the best option as we do CI on circleci
<rtohid>
@zao +1
<diehlpk>
Abhishek09, I really think you should do baby steps
<diehlpk>
Do not make things complicated from the beginning
<Abhishek09>
FyI which version of gcc used in package ?
<diehlpk>
I really think that the first step should be to build the pip package on one OS using the easiest tool
<diehlpk>
If we can build the package in a dockerfile and copy it to a fresh dockerfile and can run the examples would be a huge success
<Abhishek09>
manylinux also not working on circleci
<diehlpk>
I think that this could be enough for GSoC
<Abhishek09>
ci tools is not our problem , it can be used for only genrating wheels
<diehlpk>
if you can generate a whl file with your setup.py running the docker container A and export it and copy it to a plain fedora and install the whl file
<diehlpk>
in a new plain fedora docker container, this would be sufficient
<rtohid>
@Abhishek09, Please note this is a C++ project with a python interface. I'd suggest you to start with building HPX.
<diehlpk>
Using the hpx fedora package.
<rtohid>
as a deliverable it has to work on Ubuntu too\
<rtohid>
again, as a derloverable we don't want to use OS packages
<diehlpk>
After this you remove the fedora package and install all dependencies
<diehlpk>
After you have things done on fedora you can switch to Ubuntu
<diehlpk>
rtohid, I would strongly recommend to use the fedora package for the first milestone. I think getting pybind, blaze, blaze-iterative and phylanx to work is complex enough
<diehlpk>
After he has done this, he can remove the os package and add hpx and all dependencies
nikunj97 has quit [Quit: Leaving]
<rtohid>
But this has to be done, so it is better to start there
<Abhishek09>
Please explain `After this you remove the fedora package and install all dependencies`
<diehlpk>
yes, I told him he should compile HPX and Phylanx already
<diehlpk>
Abhishek09, As I told you before, I would start using Fedora, dnf install hpx pybind, build blaze, blaze-iterative, phylanx and generate the whl package
<diehlpk>
And test this package on a different plain fedora installation
<diehlpk>
After that I would add pybind to the packages to compile
<diehlpk>
And test the new whl file again on a fresh installed fedora
rtohid has quit [Quit: Konversation terminated!]
<diehlpk>
As a next step I would remove dnf install and compile, boost, hwloc, jemalloc, and hpx and generate the whl package
<diehlpk>
I think that doing all from scratch is too difficult
<diehlpk>
So step by step
<Abhishek09>
you mean say to first i install all deps and then make .whl for phlanx
<Abhishek09>
then try on fresh fedora OS
<Abhishek09>
then keeps adding deps bit by bit
nikunj has quit [Ping timeout: 260 seconds]
<Abhishek09>
itreatively
nikunj has joined #ste||ar
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
diehlpk has quit [Ping timeout: 240 seconds]
nikunj has quit [Ping timeout: 258 seconds]
nikunj has joined #ste||ar
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
Abhishek09 has quit [Remote host closed the connection]
nikunj has quit [Read error: Connection reset by peer]