K-ballo changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
<hkaiser> gonidelis[m]: thanks
hkaiser has quit [Quit: bye]
K-ballo has quit [Quit: K-ballo]
<gnikunj[m]> gonidelis[m] whenever you see a linker error that complains about function _start ..., it is most likely an indication that you did not link to HPX::wrap_main or tried using hpx_main facility without including hpx_main
<jedi18[m]1> Hi! Are there any other beginner tasks to work on? Or would you prefer we continue moving the other algorithms to tag_fallback_invoke?
K-ballo has joined #ste||ar
hkaiser has joined #ste||ar
<gonidelis[m]> jedi18: adapting one or two algos more would do you good, in a way that you would get familiarized with the way we implemente our overloads
<gonidelis[m]> e.g. return types, traits, facilities
<jedi18[m]1> Oh ok sure, any of those remaining right?
<hkaiser> jedi18[m]1: all of them ;-)
<jedi18[m]1> Would be happy to! :D, but I thought you'd like to keep some for any new students trying for gsoc
<hkaiser> lol
<hkaiser> you don't worry about that - we have many things for new people
<jedi18[m]1> Oh ok, should I create individual prs or club some of them together into the same PR? Since the changes are very small
<srinivasyadav227> hkaiser: does blaze come under hpx?
<gnikunj[m]> srinivasyadav227: no that's an independent linear algebra library
<hkaiser> srinivasyadav227: no, it's an independent project that may rely on hpx (one of its backends use hpx)
<gnikunj[m]> Jules and hkaiser did extend blaze to blaze cuda and blaze tensor though
<hkaiser> the cuda work actually did not go anywhere (yet)
<hkaiser> Shahrzad is trying to find a proper way to approch this
<gnikunj[m]> I though Klaus accepted the implementation
<gnikunj[m]> what was wrong about the previous implementation btw?
<hkaiser> too much code duplication
<hkaiser> Klaus made some changes to blaze based on Jules' experience that should enable less duplication
<gnikunj[m]> ah! Had no clue about it.
<gnikunj[m]> Jules did tell me that he had to copy a ton of stuff around
<srinivasyadav227> so if not wrong, blaze come under STELLAR Group right?
<gnikunj[m]> nope
<gnikunj[m]> blaze tensor? sure!
<gnikunj[m]> same for blaze iterative and blaze cuda
<srinivasyadav227> Im confused, I know abt a library blaze(https://bitbucket.org/blaze-lib/blaze/src/master/) , now blaze under stellar is a fork?
<gnikunj[m]> yes, blaze under stellar is a fork of the actual blaze repo
<srinivasyadav227> alright, blaze_tensor, blaze_cuda, blaze iterative comes under stellar?
<gnikunj[m]> yes, they're the repository that people from stellar created to extend the existing blaze library
<srinivasyadav227> oh, got it
<srinivasyadav227> actually I am planning to write a small c++ deep learning library based on CUDA and HPX (in month of July or august), so I am looking forward to Separate the datapar algorithms(https://github.com/STEllAR-GROUP/hpx/issues/5157), Working on Blaze Tensor(https://github.com/STEllAR-GROUP/blaze_tensor) and picking one of them for gsoc, so could anyone help me getting started with these? (These are two projects I have
<srinivasyadav227> some experience and related knowledge)
<hkaiser> the datapar work is simple as it requires refactoring mainly
<hkaiser> but will require knowing the codebase
<hkaiser> srinivasyadav227: see wg21.link/p0350 and wg21.link/n4755 for what we would like to have
<hkaiser> srinivasyadav227: also https://github.com/STEllAR-GROUP/hpx/issues/2333 as background information (current state)
<srinivasyadav227> ok, understanding tag_invoke with these 2 papers would a good way to start?
<hkaiser> tag_invoke is an implementation technique that is a) necessary for the separation of the existing algorithms (see #2333) and b) a way to start understanding the hpx code base in general
<srinivasyadav227> first all algorithms should be converted to datapar (#2333) then #5157 should be implemented right?
<hkaiser> #5157 is probably the first step, even more as it's easier
<hkaiser> srinivasyadav227: I can do one of the algorithms for #5157 as an example
<srinivasyadav227> ok, will through them, and if any doubts I will ask here 😀
<srinivasyadav227> hkaiser: yeah, that would help a lot
<jedi18[m]1> Ok so I think I should start thinking about which idea I should go for so I can start studying the related concepts from now
<jedi18[m]1> Implement missing Parallel Algorithms looks interesting but it looks like most of it is already implemented since only partial_sort_copy and shift-left/shift-right is remaining
<jedi18[m]1> Will that be enough or is it too small for a gsoc project?
<hkaiser> yes
<hkaiser> those are definitely sufficient for a project
<hkaiser> the sort/nth_element are hard, the shift algos are simpler
<jedi18[m]1> I'm also interested in the co-routine like interface idea
<jedi18[m]1> Which among those two would you recommend?
<hkaiser> the coroutine is difficult because we don't know ourselves what we want there ;-)
<hkaiser> more of an investigation
<jedi18[m]1> I see, well I'll have a look at the ideas again and see if there's any other idea I like
<hkaiser> jedi18[m]1: but the algorithms would be highly appreciated
<jedi18[m]1> <hkaiser "jedi18: but the algorithms would"> You mean the "Implement missing parallel algorithms" idea?
<hkaiser> anything related to the algos ;-)
<jedi18[m]1> Oh ok then I'll go for that then, would probably learn a lot from it too!
<gonidelis[m]> algos are a great to place to be
<jedi18[m]1> Could you give me an basic overview of what all I'd need to know? There's still quite some time so I can be properly prepared to contribute
<hkaiser> ughh
<hkaiser> everything? ;-)
<gonidelis[m]> jedi18: i thought tha contributing to the projects was a "read the theory and implement according to that" thing
<gonidelis[m]> but it ain't
<hkaiser> start with simpler tasks as the tag_invoke or segmented algorithm separation and get the experience around the code base you'll need for more complex tasks
<gonidelis[m]> exactly what hkaiser suggests
<gonidelis[m]> do "trivial" stuff and you will be learning a lot from your digging and searching
<gonidelis[m]> that's the most valuable knowledge
<hkaiser> along the way you familiarize yourself with the c++ implementation techniques we use
<hkaiser> (those are also explained in the videos, well except for the tag_invoke, which you have to read up on yourself)
<jedi18[m]1> Thanks! That makes a lot of sense
<jedi18[m]1> <hkaiser "(those are also explained in the"> I've watched the videos, they're very helpful and though I felt like I understood everything at the time of watching the videos, I think I'm going to need to refer and read the concepts a few times to really internalize all that stuff
<gonidelis[m]> jedi18: i could easily tell you to study projections and lambdas but that's like 1% of the goal. there is no point in splitting it into small pieces. Start contributing and while you do that you will come across lots of uknown stuff and you will be googling them
<jedi18[m]1> Plus a lot of it is new, so a bit overhelming
<jedi18[m]1> <gonidelis[m] "jedi18: i could easily tell you"> Oh ok thanks!
<hkaiser> take it one step at a time
<jedi18[m]1> Thanks! I will
<jedi18[m]1> <hkaiser "start with simpler tasks as the "> By the segmented algorithm seperation task, you mean this? https://github.com/STEllAR-GROUP/hpx/issues/5156
<srinivasyadav227> Thanks! these suggestions me too! 🙂
<srinivasyadav227> Thanks! these suggestions help me too! 🙂*
<hkaiser> jedi18[m]1: exactly
<jedi18[m]1> I guess I can try to tackle one of these after I do the tag_invoke task? I can use your for_each PR as a reference
<gonidelis[m]> btw thank you both for your PRs jedi18 srinivasyadav227 . That shows a lot
<gonidelis[m]> about your commitment ^^
<jedi18[m]1> Thanks! Look forward to creating a lot more :D
<hkaiser> jedi18[m]1: yes
<srinivasyadav227> gonidelis: Thanks!, yea same, looking forward to contribute more!
<hkaiser> although the foreach PR does slightly more, iirc
<hkaiser> #5156 also moved the segmented iterators into hpx::segmented, that's something that needed doing once only
<hkaiser> so most of the changes are not relevant for other algorithms
<hkaiser> jedi18[m]1: the idea is simple
<hkaiser> currently the parallel algorithms dispatch internally to the non-segmented and the segmented versions using a function that is usually named foo_ for algorithm foo
<hkaiser> that function needs to be removed
<hkaiser> the dispatching to the segmented algorithms should be reimplemented using a tag_invoke overload instead
<hkaiser> in effect the non-segmented algorithms don't have to know anything about the segmented ones anymore, which is the part that disentangles things
<hkaiser> the non-segmented algorithm will have to use tag_fallback_invoke, while the segmented one uses tag_invoke + SFINAE on is_segmented_iterator
<hkaiser> this way the segmented overload will be chosen if is_segmented_iterator is true, otherwise it will fall back to the non-segmented overload, which is what we want
<jedi18[m]1> Thanks a lot for the explanation! Helps a lot!
<jedi18[m]1> I'll attempt one algorithm soon, I think I'll be able to thanks to your explanation and the PR, but if I have doubts I'll ask here
<hkaiser> jedi18[m]1: look into for_each_n, should be straight forward
<jedi18[m]1> Btw one doubt, we use HPX_CONCEPT_REQUIRES_ instead of enable_if for SFINAE?
<jedi18[m]1> I thought someone said we don't use concepts yet in hpx
<jedi18[m]1> <hkaiser "jedi18: look into for_each_n, sh"> Oh ok, I will!
<hkaiser> jedi18[m]1: it _is_ the SFINAE, look at the implementation of HPX_CONCEPT_REQUIRES_
<jedi18[m]1> Ohh ok that makes sense, thanks
<gnikunj[m]> if yes, would it not be inconsistent to say that the algorithms are "asynchronous" (even though we return a future<void>)?
<jpinto[m]> Hi! Can someone explain to me what add_hpx_pseudo_target and add_hpx_pseudo_dependencies do in this example (https://github.com/STEllAR-GROUP/hpx/blob/master/examples/performance_counters/sine/CMakeLists.txt)?
<jpinto[m]> If I use cmake I get this error (https://gist.github.com/JAPPinto/6d0979f4930db219dd15459bcd8875cb).
<jpinto[m]> If I delete these lines the example works but I'm having some issues implementing a similar counter and maybe they are caused by the deletion of these lines.
<hkaiser> jpinto[m]: looks like a bug on our end
<hkaiser> the pseudo_targets are a pure cmake thing, nothing related to code
<hkaiser> how did you configure your hpx?
<jpinto[m]> it was a while ago but I think I just did cmake "-DHPX_WITH_PAPI:BOOL=ON" and "make install"
<jpinto[m]> I tried to adapt the sine example by starting with changing the name of the counter but if I do that it stops working. I replaced every occurrence of sine but it still recognizes the sine flag so I'm not sure what's happening and thought that maybe it's related to cmake and is still using the implementation of sine from somewhere in the hpx build.
<jedi18[m]1> hkaiser: Why did you replace the old segmented implementation of for_each_ with for_each_n_? Is this because for_each_n previously used for_each_ as part of it's implementation but now that you've changed the code, you added that so for_each_n still works? https://github.com/STEllAR-GROUP/hpx/pull/5176/files#diff-69474ccf44b7810786ee963cc19146799d5150cb4e2e6a2ca7ad7d401d34e923R187
<hkaiser> jedi18[m]1: I didn't
<hkaiser> github makes it appear like I did, the for_each_n_ is unchanged from before
<hkaiser> jpinto[m]: I would have to see your code to diagnose what's wrong
<jedi18[m]1> <hkaiser "github makes it appear like I di"> Huh, I went back to the commit before your one and it dosen't have it (https://github.com/STEllAR-GROUP/hpx/blob/7f16729ea15f55440e235bb38d5faf02026070ad/libs/full/segmented_algorithms/include/hpx/parallel/segmented_algorithms/for_each.hpp)
<jedi18[m]1> Maybe I'm looked at the wrong commit .-.
<jedi18[m]1> * Maybe I looked at the wrong commit .-.
<hkaiser> ahh, now I rmemeber
<hkaiser> previously the for_each_n was dispatching into the un/segmented for_each_
<hkaiser> I had to shuffle things around as for_each_ doesn't exist anymore
<hkaiser> there was no for_each_n_ previously
<jedi18[m]1> Ohh ok, thanks!
<jpinto[m]> <hkaiser "jpinto: I would have to see your"> https://github.com/JAPPinto/HPX-Performance-Counters/tree/master/full_counter The code is the same as the sine example in the hpx repository, I only replaced sine with example in the code and in the file names. If I try to use --example it's not recognized but if I use --sine it's still recognized for some reason but I get unknown counter type /example/immediate/explicit:
<jpinto[m]> HPX(bad_parameter)
<jpinto[m]> Maybe I should try to implement the counter from scratch instead of trying to adapt the example
<hkaiser> jpinto[m]: ahh, so copied the build system as well?
<hkaiser> so you*
<hkaiser> in a project external to HPX you can't use all of the build system environment, so removing the pseudo targets is the right thing to do
<hkaiser> jpinto[m]: so what's your problem now?
<jpinto[m]> I think it's not registering the counter as defined in the example.cpp file
<hkaiser> ok, I will need to buildit to see, give me a bit of time, pls
<hkaiser> or beter yet, pls create a ticket describing you issue
<hkaiser> jpinto[m]: you now have to use --example on the command line, does that work?
<jpinto[m]> No, I get "example_client: caught exception: unknown counter type /example/immediate/explicit: HPX(bad_parameter)"
<hkaiser> even with --example on the command line?
<jpinto[m]> Sorry, with --example I get runtime_support::load_components: command line processing: unrecognised option '--example'
<jpinto[m]> Without it I get th e unknown counter type error
<hkaiser> ok, so you module doesn't get loaded
<hkaiser> do you work with an installed version of HPX?
<jpinto[m]> yes
<hkaiser> could you copy the generated modules (the executable) to $(hpx_root)/bin and the counter module to $(hpx_root)/lib/hpx
<hkaiser> you're on linux, I assume?
<jpinto[m]> yes
<hkaiser> nod
<hkaiser> try copying the files and run the example from there
<jpinto[m]> Ok, I will do that in a bit
<hkaiser> jpinto[m]: we know that dynamic modules sometimes don't get properly found in external builds, I have not had the time to investigate
<hkaiser> I believe things will work once you copied the module
<hkaiser> in any case, please create a ticket that describes your issue - so we'll not forget
<jpinto[m]> <hkaiser "could you copy the generated mod"> Sorry, by counter module you mean the source files?
<jpinto[m]> Nevermind, It's the .so file
<jpinto[m]> I still get the same error, I'm going to reinstall hpx and try again
<ms[m]> gnikunj: that's the fallback implementation, and admittedly it's close to useless, but it is allowed behavior
<ms[m]> The hpx, cuda, and hip backends are actually asynchronous
<gnikunj[m]> Got it
jehelset has quit [Read error: Connection reset by peer]
bita has joined #ste||ar
<jpinto[m]> hkaiser: After moving the counter module to $(hpx_root)/lib/hpx it still didn't work so I changed the file name from libexample.so to libhpx_example.so in order to match the format of the other files in the folder and now it works. Thanks. I will create a ticket about this.
<hkaiser> thanks