hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/ | GSoC2018: https://wp.me/p4pxJf-k1
eschnett has joined #ste||ar
K-ballo has quit [Quit: K-ballo]
anushi has quit [Ping timeout: 245 seconds]
anushi has joined #ste||ar
<nikunj>
zao: you there?
<zao>
hi
<nikunj>
Hi
<nikunj>
I needed to know one thing, why does adding cxx flags in cmake not reflect in pkg-config?
<nikunj>
I want to add some os dependent cxx flags
<zao>
I assume we generate the file from our CMake scripts somewhere.
<zao>
Or populate some template.
<nikunj>
yes a template gets populated
<nikunj>
I want to add my cxx flags to it, but I can't understand where those cxx flags are coming from
<nikunj>
coz HPX_GeneratePackage.cmake does not contain it
<nikunj>
and it's populating the templates
hkaiser has quit [Quit: bye]
<zao>
CXX_FLAG is set elsewhere in the project, like HPX_DetectCppDialect.cmake?
<nikunj>
this looks like it. Let me try changing parameters here check if they're changing
<zao>
Make sure you don't mess up the meaning of something existing.
<zao>
Might be cleaner to introduce some new variable if that one is used solely for C++ language version or whatnot.
<nikunj>
that's what I'm thinking right now. I want to add a linker flag but the current template does not have a variable for it
<nikunj>
zao: can a component every require hpx_main.cpp?
<zao>
No idea, never really used them.
<zao>
Assuming you mean the pluggable ones?
<nikunj>
those with hpx_component
eschnett has quit [Quit: eschnett]
anushi has quit [Read error: Connection reset by peer]
anushi has joined #ste||ar
nikunj has quit [Ping timeout: 240 seconds]
nikunj has joined #ste||ar
anushi has quit [Ping timeout: 240 seconds]
anushi has joined #ste||ar
nikunj has quit [Remote host closed the connection]
<jakub_golinowski>
M-ms, thanks for the answers, now I regret I did not do the force push but when I was doing PR to cmake (gitlab) it worked like this with deleting the branch and repushing it so I thought it is OK
<jakub_golinowski>
As for the problems with apply and async it were the references so I used the hpx::util::bind to solve the problem
<M-ms>
jakub_golinowski: remember that using std::ref should also work
<jbjnr>
then pycicle will generate random combinations of them, based on the dependency rules
<jakub_golinowski>
M-ms, jbjnr I managed to recreate the same error messages in the MartyCam as in the opencv backend with hpx_main in tests
<jbjnr>
then submit the build using those options and write the options to a notes file for submission to cdash
<M-ms>
jbjnr: random combinations?
<jbjnr>
next step is to allow the user to trigger the build by adding a github comment on a PR with "pycicle options ...." and past the string from the notes file from a previous build, so that it rebuilds with the same options
<jakub_golinowski>
I would not call it a success, quite oposite but it shows that there is a single issue at place and I am becoming more and more sure that it has to do with circular dependencies
<M-ms>
there's no way to choose?
<jbjnr>
jakub_golinowski: please explain. Sorry. I am not sure what errors you are referring to
<jbjnr>
looks good. I can't remember what we used to get. Did you improve the PP?
<heller>
yes
<heller>
now with zero copy on the receiving end :D
<heller>
and faster scheduling of the actual sends
<jbjnr>
great
<heller>
I think we only got on par with MPI with a winow size of 10 or so, with multiple threads
<jbjnr>
10 is not too bad
<jbjnr>
much better than 100
<heller>
this is with 1!
<jbjnr>
awesome
<jbjnr>
time to write a paper!
<heller>
this is going into my thesis :P
<jbjnr>
what did you change on the send end - we already have an immediate send bypass
<heller>
I essentially got rid of all the parcelhandler crap with different parcelports
<jbjnr>
excellent
<jakub_golinowski>
M-ms, so you say you just printed all the libs, then moved hpx_init before the hpx in this string and pasted this long string everywhere in the cmake filese instead of HPX_LIBRARIES variable?
<jbjnr>
and on the receiving end? (do get my rdma_vector branch please:) )
<jbjnr>
that has zero copy on receives
<heller>
jbjnr: the sender and receiver are now handled within HPX tasks, and the parcel handler has its own stripped down scheduler, without task stealing
<heller>
this has zero copy on receive as well ;)
<jbjnr>
ok. that's good. I wanted a dedicated poo, for network handling
<heller>
even for things like vector
<jbjnr>
poo = pool
<heller>
it's not a pool
<jbjnr>
ok, but we might want to move background work to one at some point
<jbjnr>
then run your scheduler there
<heller>
yeah
<heller>
so it's part of the scheduling loop
<heller>
also, when starting a send, I don't schedule the task and wait until I am there (that takes up to two 2 us latency), but do switch the context immediately, only when the send is taking longer anyway, is when I switch over to the scheduling loop
<heller>
this whole thing could be brought down even further if we could get rid of allocation happening for the action
<heller>
which should be possible, but requires more work
<heller>
as of now, I am happy as is
<M-ms>
jakub_golinowski: yeah, or you search for HPX_CONF_LIBRARIES and change it there
<jakub_golinowski>
hmm it did not work for me
<jakub_golinowski>
However, I got the different error. M-ms, could you paste your contents of HPX_LIBRARIES?
<jbjnr>
heller: I thought the immediate snd already bypassed all the scheduling. I forget the details, but the send happens on the current thread directly
<jbjnr>
anyway. push a branch for testing any time
<jbjnr>
new pycicle test fetaure is going to be used to test LF PP from now on
<jakub_golinowski>
M-ms, also my HPX_LIBRARIES seems to follow this syntax ""general;hpx;general;hpx_init;${HPX_LIBRARIES}"", is this OK to have general 2 times?
<M-ms>
did the cmake configure run again for you when you rebuilt=
<M-ms>
?
<M-ms>
otherwise just do it the proper way and rebuild hpx
<jakub_golinowski>
M-ms, I had a different error now
<M-ms>
I'm pretty sure that's the problem, they're the other way around when using add_hpx_executable and co, and hpx_init depends on hpx so it should come first
eschnett has joined #ste||ar
<heller>
jbjnr: well, not quite. we depend on the background thread to make progress
<M-ms>
jakub_golinowski: ok, can you post it?
<heller>
while we dispatch immediately, there is 1) a bit of latency until we actually get to the dispatch 2) latency until we make progress on the fabric itself
<heller>
I removed both
<heller>
the network now takes priority, and the overhead to poll is minimal
<heller>
jbjnr: now the bad news. adding futures to the mix, increases the latency by 4 usecs!
<heller>
I hate my life
<M-ms>
jakub_golinowski: is there an error there? Warnings are never good though, you seem to have a self made and system installed version of hwloc?
<jakub_golinowski>
M-ms, I noticed that as well and it is yet another argument to rebuilt HPX (and its dependecies first)
<jakub_golinowski>
So yest this are cmake warning but I get linker error
<jakub_golinowski>
M-ms, so is it because the /usr/lib/ hwloc is in the path?
<jakub_golinowski>
M-ms, what I want to ask is this: Since in my HPX_LIBRARIES I use /usr/lib/.../libhwloc.so then why is the cmake concerned about the packages/hwloc/? I cannot see where I could give him the info that such a directory exists if not while building HPX
<jakub_golinowski>
and from the HPX_LIBRARIES it seems I did not
mcopik has joined #ste||ar
<jbjnr>
heller: we all do. ours I mean, not yours especially
<jbjnr>
M-ms: I am on train. made it just in time.
<heller>
jbjnr: how was the workshop?
<M-ms>
jbjnr: have good trip back
nikunj97 has joined #ste||ar
<M-ms>
jakub_golinowski: do you have something in LD_LIBRARY_PATH?
<jakub_golinowski>
I do not see anything in the terminal
nikunj has quit [Ping timeout: 240 seconds]
K-ballo has joined #ste||ar
<M-ms>
jakub_golinowski: I've never used clion, but could it be doing something nasty? did you build hwloc in clion?
<jakub_golinowski>
no, I was building all the libs from terminal
<jakub_golinowski>
HPX as well
<jakub_golinowski>
and opencv for that matter as well (as seen in my bash script for that)
<jakub_golinowski>
this is just a "dev" build but the errors seen there seem to propagate for the console build since clion is using only cmake as config
<jakub_golinowski>
M-ms, I will try to go through the whole process again and hopefully as I am a bit more aware of things than I was previously maybe I will spot some mistakes
jakub_golinowski has quit [Ping timeout: 245 seconds]
mcopik has quit [Ping timeout: 268 seconds]
<M-ms>
whoever fixed gcc 7 and 8 builds, thank you!
hkaiser has joined #ste||ar
mcopik has joined #ste||ar
<nikunj97>
hkaiser: I've changed the commit history of my pr. Now it contains 2 commits, one of implementation and other for changes corresponding to cmake and pkg-config
<hkaiser>
ok
<hkaiser>
I'll look today
<K-ballo>
woa, clean commits? can we keep him?
<nikunj97>
thanks
<hkaiser>
;-)
<nikunj97>
I'll add the Mac OS implementation as soon as this pr gets merged.
<hkaiser>
k
mcopik has quit [Ping timeout: 248 seconds]
<nikunj97>
Btw I was trying to implement mainCRTStartup myself. As of now efforts have gone in vain (too many errors on build). I'm trying to understand as to how I can efficiently implement it. Should be able to implement it around 20-21 July (with full functionalities and call stacks).
<hkaiser>
nikunj97: nod, thanks
mcopik has joined #ste||ar
galabc has joined #ste||ar
<jbjnr>
K-ballo: quite right - the apex PR I approvd the other day was merged with at least 5 redundant commits about 'making inspect happy' and my comment that perhaps they should be squashed into one, was ignored. next time I'll be more brutal and reject PRs like that. my bad.
<jbjnr>
python is so awesome. why do we bother with this c++ nonsense.
<hkaiser>
jbjnr: if I waited for people to squash the commits things wouldn't be merged ever
nikunj97 has quit [Remote host closed the connection]
nikunj97 has joined #ste||ar
<diehlpk_work>
To our GSoc students, do not forgot to send to us your content for the blog post by July 6th!
<diehlpk_work>
jbjnr, see pm
<K-ballo>
github has a squash and rebase merge option, but it is only viable for focused PRs that do one thing and just that one thing
jakub_golinowski has joined #ste||ar
jakub_golinowski has quit [Ping timeout: 256 seconds]
jakub_golinowski has joined #ste||ar
anushi has joined #ste||ar
galabc has quit [Quit: Leaving]
eschnett has quit [Quit: eschnett]
jbjnr has quit [Read error: Connection reset by peer]
jbjnr has joined #ste||ar
<diehlpk_work>
jbjnr, see pm
mcopik has quit [Ping timeout: 260 seconds]
<M-ms>
jakub_golinowski: any luck with the build?
<hkaiser>
parsa[w]: serialization should be fied now
<hkaiser>
fixed*
jakub_golinowski has quit [Ping timeout: 256 seconds]
jakub_golinowski has joined #ste||ar
nanashi64 has joined #ste||ar
nanashi55 has quit [Ping timeout: 256 seconds]
nanashi64 is now known as nanashi55
Anushi1998 has joined #ste||ar
diehlpk_mobile has joined #ste||ar
eschnett has joined #ste||ar
eschnett has quit [Client Quit]
<jakub_golinowski>
M-ms, ok it seems to be sorted out (at least for the OpenCV build) - the issues were wrong order of HPX_LIBRARIES and the mixup of linking hwloc form two places
<jakub_golinowski>
Now the issue is this test: [ RUN ] OCL_Arithm/Compare.Mat/1, where GetParam() = (CV_8U, Channels(1), true)
<jakub_golinowski>
both for the debug and release
<jakub_golinowski>
but at least building is sorted out :D