hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/ | GSoC: https://github.com/STEllAR-GROUP/hpx/wiki/Google-Summer-of-Code-%28GSoC%29-2020
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
nan9 has quit [Remote host closed the connection]
<Yorlik> hkaiser: YT?
<Yorlik> I titled this: "The elephant in the room" for myself: https://cdn.discordapp.com/attachments/427766114169126914/722990411718590544/unknown.png
<Yorlik> So - step by step the Lua things are coming into the picture, like these 10% string creation in Lua (of which 95+% are just a strncmp)
<Yorlik> Seems I have the most overengineered string comparator in town now :D
* Yorlik heads out for sleep.
hkaiser has quit [Quit: bye]
sayef_ has quit [Ping timeout: 256 seconds]
RostamLog has quit [Ping timeout: 265 seconds]
RostamLog has joined #ste||ar
nikunj97 has joined #ste||ar
sayefsakin has joined #ste||ar
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
Nikunj__ has joined #ste||ar
kale[m] has quit [Ping timeout: 260 seconds]
nikunj97 has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
bita_ has quit [Ping timeout: 244 seconds]
sayefsakin has quit [Quit: Leaving]
mdiers[m] has joined #ste||ar
<mdiers[m]> For when is version 1.5 planned?
<mdiers[m]> Will then the pr fork-join executor #4744 be included?
mdiers[m] has left #ste||ar ["User left"]
<ms[m]> mdiers: we're trying to get it out in about a month or so, pending that we have our full testing infrastructure in place
<ms[m]> it's unlikely that the fork-join executor will be included, but I could maybe try to clean it up into some shape
<ms[m]> it'd be experimental in any case
mdiers[m] has joined #ste||ar
<mdiers[m]> ms: thanks, an experimental version is completely sufficient for me
weilewei has quit [Remote host closed the connection]
<gonidelis[m]> If I want to use distance in a .cpp file
<gonidelis[m]> I `#include <hpx/parallel/algorithms/detail/distance.hpp>`
<gonidelis[m]> and then I use it like `hpx::parallel::v1::detail::distance()` or `hpx::parallel::distance` ?
<zao> I’d intuitively consider implementation detail of a module off limits and favor using the public interface, but that’s personal guesswork :)
<ms[m]> gonidelis: the `v1` namespace is inline, so that one can be optionally left out, i.e. `hpx::parallel::detail::distance` would be valid, but not `hpx::parallel::distance`
<ms[m]> I don't think we have a consistent rule on if we actually specify inline namespaces, but I'd leave it out where you use it
<ms[m]> if the cpp file is a test file you don't need to worry about `detail` being internal
<ms[m]> if this is indeed meant to be public then it shouldn't be in `detail` in the first place, but I'll leave that to K-ballo/hkaiser (I don
<ms[m]> 't know where this fits in the greater scheme of ranges)
<gonidelis[m]> ms[m]: No it does not meant to be public... Just want to test if it works on a mini script instead of a whole algorithm :) It seems like `hpx::parallel::v1::detail::distance()` works after all! zao How could this be done? I mean how to access `distance` through "public interface"? Thank you all btw!!!
<gonidelis[m]> What are the options in order to format a .cmake file in HPX?
<gonidelis[m]> with `cmake-format` ^^
<gonidelis[m]> I did `cmake-format -i cmake/HPX_PerformCxxFeatureTests.cmake ` and I think it messed it up :E
<ms[m]> gonidelis: if it's just for testing there's no need for a "public interface"
<ms[m]> that looks right for cmake-format... what happened?
K-ballo1 has joined #ste||ar
K-ballo has quit [Ping timeout: 256 seconds]
K-ballo1 is now known as K-ballo
<K-ballo> gonidelis[m]: that command looks fine to me, what went wrong?
<gonidelis[m]> I 'll post a diff in a second
<gonidelis[m]> Seems like it's my fault after all
<gonidelis[m]> I will double check
<gonidelis[m]> I have not messed at all with future.hpp though....
<K-ballo> master wont build for me, missing force-linking headers
<gonidelis[m]> :/
<gonidelis[m]> damn...
<K-ballo> ms[m]: ^
<gonidelis[m]> me too...
<K-ballo> looks like it's picking a modules.cpp file that should not be there anymore? ..is it globbing?
<gonidelis[m]> is there any safe stage where we could rebase 😅 ?
<K-ballo> if you clean your build folder it should be fine, or CI would not have passed
<K-ballo> did not work for me.. seems it got worse
<ms[m]> K-ballo: gonidelis aah, sorry, can you give me more details
<ms[m]> I did merge the PR that removed the force linking stuff, but maybe it didn't merge nicely with master...
<ms[m]> K-ballo: does this look like it could help: https://github.com/STEllAR-GROUP/hpx/pull/4767? there were some incompatible changes in the last few prs I merged
<ms[m]> the modules.cpp file should be gone, it might require a clean cache though
<K-ballo> ms[m]: with a clean cache it works
hkaiser has joined #ste||ar
<ms[m]> ok, good
<ms[m]> we still need to improve a bit with our cache variables...
<ms[m]> gonidelis: do you still have a problem with your rebase? if you post the conflicting changes I might be able to tell you what to choose
<gonidelis[m]> ms[m]: Seems like it was my mistake after all... I had a branch based on another branch which had not yet been merged to master. I am trying to fix it right now. auriene helps a lot !
<gonidelis[m]> sorry
<ms[m]> gonidelis: no worries at all, glad you're getting it sorted out
kale[m] has quit [Ping timeout: 246 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 240 seconds]
<K-ballo> our governance text has awkward line wraps
weilewei has joined #ste||ar
<hkaiser> K-ballo: thanks
<weilewei> hkaiser just got email that p3hpc becomes remote
<hkaiser> yah, me too
<zao> Woo, EasyBuild world domination one step closer :P https://hpckp.org/agenda/european-environment-for-scientific-software-eessi/
nan9 has joined #ste||ar
Nikunj__ is now known as fellow
fellow is now known as nikunj97
<nikunj97> fellow stellar people. After 2d of grueling grinding, I present you a 1d stencil that scales its kernel almost linearly (strong scaling)
karame_ has joined #ste||ar
<nikunj97> I'll add the pr soon
<hkaiser> nikunj97: nice!
<nikunj97> 10^9 stencil points for 100 iterations take 22s on 1 node and 8s on 4 nodes
<nikunj97> don't have more nodes to test on, but I like the results xD
<hkaiser> nikunj97: is this based on the 1d_stencil series?
<hkaiser> nikunj97: more nodes: rostam
<nikunj97> hkaiser, no
<nikunj97> this is based on 2d stencil series from thomas
<nikunj97> I used his communicator api to build a 1d stencil
<nikunj97> I did scale 1d stencil series in examples, but the scaling results weren't very impressive. I couldn't explain why though
<tiagofg[m]> hello everyone,
<tiagofg[m]> I want to load dynamic libraries at run time and I think you guys can help me
<tiagofg[m]> I have a c++ program that loads the libraries using dlopen and dlsym, I want to do the same but using hpx, to have hpx code inside these dynamic libraries
<tiagofg[m]> first of all, is this possible?
<tiagofg[m]> thanks
<hkaiser> tiagofg[m]: I wouldn't know why this isn't possible
<hkaiser> tiagofg[m]: you still would need to initialize HPX, but we have examples showing how to do that
<tiagofg[m]> where can I find them? has anything to do with the hpx plugins?
<hkaiser> tiagofg[m]: plugins are dynamically loaded HPX modules, I don't think that's what you're after, but please correct me if I misunderstood
<hkaiser> tiagofg[m]: also, we have an example demonstrating how to initialize/finalize HPX from a global object: https://github.com/STEllAR-GROUP/hpx/blob/master/examples/quickstart/init_globally.cpp
<nikunj97> tiagofg[m], what do you want to achieve by having hpx code inside such library?
<hkaiser> we use a similar scheme to initialize HPX from a Python extension module, works without problems in production code
<tiagofg[m]> what I intend to do is do the same things I did with dlopen and dlsym, but with hpx functions inside them
kale[m] has joined #ste||ar
<tiagofg[m]> I already tried but I had the following error: what(): Attempting to register a thread outside the HPX runtime
<hkaiser> tiagofg[m]: yah, the example will show you what to do
<tiagofg[m]> yah I am looking
<tiagofg[m]> I will try, thank you!
rtohid has joined #ste||ar
kale[m] has quit [Ping timeout: 240 seconds]
kale[m] has joined #ste||ar
<Yorlik> Are futures also garbage collected? Or are they automatically removed when they go out of scope?
<K-ballo> both?
<Yorlik> Depending on the payload?
<K-ballo> actually it depends on why you are asking
<K-ballo> shared states are ref-counted
<Yorlik> So - if the future returns a smart pointer as payload - would that keep alive the shared state? Or is the pointee not inside the shared state?
<gonidelis[m]> Ι am trying to create the PR but due to some rebasing I get https://github.com/STEllAR-GROUP/hpx/compare/master...gonidelis:hpx_sized_sentiel_for?expand=1
<Yorlik> I'm asking, because I see low frequency memory fluctuations and was wondering if these reflect the GC.
<K-ballo> the future returns a copy of the smart pointer
<gonidelis[m]> whichi is like 34 commits (only 3 or 4 are mine)
<gonidelis[m]> any ideas on how could I fix that?
<Yorlik> would get_ptr like hpx::get_ptr<mytype>( hpx::launch::sync, id ) throw if id is a remote object?
<ms[m]> gonidelis: did you at some point rebase on a branch that was not master? you most likely want to do an interactive rebase (`git rebase -i origin/master`) and "drop" the commits that are not yours
<gonidelis[m]> yeah
<gonidelis[m]> Actually I rebased on a previous PR of mine that i am waiting to be merged
<ms[m]> I highly recommend you first create another temporary branch on the same commit as your actual branch if this is the first time you do that
<gonidelis[m]> as my work was depending on that ;/
<ms[m]> just so you can try again if you happen to lose some commits you didn't intend to drop
<K-ballo> Yorlik: afair it returns null
<gonidelis[m]> am i missing sth?
<Yorlik> OK. So if I just check for an empty shared_ptr I should be safe.
<Yorlik> Thanks K-ballo
<gonidelis[m]> ms[m]: ^^
<ms[m]> gonidelis: yes, except the last push
<ms[m]> you want to update your local branch as well
<ms[m]> you can do this a million different ways, and everyone will probably have a different preference
<ms[m]> you can also do it the other way around
<ms[m]> `git checkout initial_pr_branch`
<gonidelis[m]> Why not `git rebase -i upstream/master` though? (instead of `origin/master`)
<ms[m]> oh, for you it's `upstream/master`
<ms[m]> sorry
<ms[m]> my stellar-group/hpx is called origin, that's why I said origin
<gonidelis[m]> haha ok I get it
<gonidelis[m]> ok so on git rebase -i there are only 5 commits listed (all mine)
<ms[m]> just to give you another way: you can also tag the old head of initial_pr_branch to keep that alive in case you need to reset
<gonidelis[m]> So i reckon the extra 28 or so commits should be because I rebased the initial branch maybe ?
<ms[m]> mmmmaybe... are your local and remote `initial_pr_branch`es on the same commit?
<gonidelis[m]> no they are not
<ms[m]> weilewei: it's makes set_thread_data essentially swap the new for the old, in case you want to do something with the old data
<ms[m]> not sure if it's used by us or just provided as a convenience thing
<weilewei> hmm reasonable, it leaves option for user
<hkaiser> weilewei: so one can restore the previous value easily
<weilewei> I gonna add std::size_t second m_thread_data_2 here and then propagate to set_thread_data_2, get_thread_data_2: https://github.com/STEllAR-GROUP/hpx/blob/master/libs/coroutines/include/hpx/coroutines/detail/context_base.hpp#L347
<ms[m]> gonidelis: ok... I guess do the rebase, drop the unrelated commits, and check that everything still looks the way you expect in any case
<weilewei> agreed hkaiser ms[m]
<weilewei> std::size_t m_thread_data_2
<ms[m]> if your local and remote branches are not on the same commit one of them is not up-to-date
<hkaiser> weilewei: I think you should create a separate set of APIs handling the libcds data, leaving alone the existing thread_data
<ms[m]> if it's the remote one is on the correct commit, reset or pull to that one first
<hkaiser> weilewei: [get|set]_libcds_data()
<weilewei> hkaiser oh... I see
<hkaiser> wrap that inside a #if HPX_HAVE_LIBCDS/#endif
<weilewei> but separate api calls doesn't mean separate files right?
<weilewei> like hpx get[set]_thread_data show in thread_data.hpp, thread_data_stackful.hpp, thread.hpp, coroutine.hpp, etc... I just need to put [get|set]_libcds_data() in those files
<gonidelis[m]> ms[m]: thank you so much. Don't know what went wrong I finally managed to make a clean PR. And even think that rori has been helping me the whole day....
bita_ has joined #ste||ar
<gonidelis[m]> anyways. Today was a mini-git nightmare for me but I really love it when I have to deal with such complicated git "exercises".
<gonidelis[m]> So hkaiser https://github.com/STEllAR-GROUP/hpx/pull/4768 . Feel free to add feedback whenever you want ;)
<gonidelis[m]> K-ballo: too :)
<K-ballo> gonidelis[m]: is that supposed to be a draft pr?
<gonidelis[m]> ahh ok... it already failed. That's great
<hkaiser> weilewei: yes, and always wrap them with HPX_HAVE_LIBCDS
<weilewei> hkaiser sure, will do
<gonidelis[m]> K-ballo: No, there are just some tests missing here but I can see I have some huge build problems. Let me check it....
<K-ballo> why there's a To Do on a finished PR?
<gonidelis[m]> As you can see I need some advise on certain things (like the `remove_cv_t` thing)
<gonidelis[m]> There is also one test missing `sized_sentinel_for` which I will push asap. I made the PR in order to get some feedback
<gonidelis[m]> K-ballo: ^^
<K-ballo> so it is a draft then
<gonidelis[m]> Should I mark it as that?
<K-ballo> that's less important
<gonidelis[m]> ok my bad
<K-ballo> I was just surprised to see something claiming to be done and not yet done at the same time
<gonidelis[m]> Yeah I get it. Apologies...
kale[m] has quit [Ping timeout: 240 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 244 seconds]
kale[m] has joined #ste||ar
<nikunj97> hkaiser, here is the PR: https://github.com/STEllAR-GROUP/hpx/pull/4769
<nikunj97> the code scales almost linearly :D
<gonidelis[m]> I encounter this problem again: https://ibb.co/z22dfXw
<gonidelis[m]> Could someone help ?
<gonidelis[m]> Don't really now where to trace it....
<K-ballo> what's the state of your code?
<gonidelis[m]> What do you mean?
<K-ballo> what's the source code that corresponds to the partial error messages that can be seen on that screenshot?
<gonidelis[m]> K-ballo: ^^
<K-ballo> is_sentinel_for looks wrong again
<K-ballo> did you based this branch on the other PR ?
<gonidelis[m]> ues
<gonidelis[m]> yes**
<gonidelis[m]> let me comment out that enable_if and see if it compiles
<K-ballo> where is the is_sentinel_for branch?
<gonidelis[m]> (it won't compile even if I remove `enable_if`)
<K-ballo> which enable_if?
<K-ballo> that is not the same is_sentinel_for than the one from the PR
<gonidelis[m]> typename std::enable_if<
<gonidelis[m]> hpx::traits::is_sentinel_for<Sent, Iter>::value &&
<gonidelis[m]> !disable_sized_sentinel_for<Sent, Iter>>::type,
<gonidelis[m]> sory
<gonidelis[m]> Yeah... I have rebased this one since the PR was approved
nikunj97 has quit [Ping timeout: 264 seconds]
<hkaiser> gonidelis[m]: one of the builders is not happy with the is_sentinel_for implementation: https://cdash.cscs.ch//viewBuildError.php?buildid=112173
<hkaiser> it's an older gcc version
<hkaiser> not sure yet what it's complaining about, ned to investigate
<hkaiser> this needs fixing before we can merge
<gonidelis[m]> hkaiser: yeah sure. That showed up a couple of times to me but I thought it was just some minor detail that I had missed.
<gonidelis[m]> Have no idea where it comes from though
<hkaiser> gonidelis[m]: compilers tend to be stubborn some times ;-)
<hkaiser> especially the older versions
<hkaiser> K-ballo: any idea what gcc is complaining about ^^
<K-ballo> no
<hkaiser> nod
<K-ballo> I'd try switching the last two lines
<hkaiser> gonidelis[m]: ^^
<K-ballo> or maybe all of them around
<gonidelis[m]> yeah I am already checking on that
<gonidelis[m]> sec
<gonidelis[m]> In order to get to the state where the PR is should I `git reset --hard 39e79a8` ?
kale[m] has quit [Ping timeout: 244 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
<gonidelis[m]> OK i reseted hard on `39e79a8` (latest commit) but I cannot reproduce the problem locally
<gonidelis[m]> Should I re`push` in order to test the changes that K-ballo suggested through the github test?
<gonidelis[m]> hkaiser: ^^
<hkaiser> gonidelis[m]: you'd need an older gcc to reproduce things, I believe this is gcc 6
<hkaiser> no, it's V7.5
kale[m] has quit [Ping timeout: 264 seconds]
kale[m] has joined #ste||ar
<hkaiser> gonidelis[m]: otoh, the commit it reports the error on is 1ca20e941792d9f33db0e5752bfd0d3b2f31e4b3 which is not the latest state on your PR
<hkaiser> that's something for ms[m] to figure out...
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
<ms[m]> hkaiser: gonidelis the commit hashes on pycicle don't correspond to the hashes on the branch because they get merged with master
<ms[m]> something is wrong with that update though, I'm starting to suspect pycicle doesn't particularly like force pushes...
<hkaiser> yes, it said that it couldn
<hkaiser> t merge things
<ms[m]> exactly
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
<gonidelis[m]> hkaiser: actually I am already on 7.5 ;p
<gonidelis[m]> So bottom line: the problem comes from me when force-pushing?
<ms[m]> gonidelis: maybe, not sure
<ms[m]> we'll try to sort it out, pycicle can be a bit sensitive sometimes...
<ms[m]> if things are compiling for you locally and on most of the builders you're most likely good
<hkaiser> ms[m]: it's just gcc-oldest and gcc-cuda complaining
<ms[m]> hkaiser: there's a bunch of other gcc builds not working correctly
<ms[m]> we did a bit of spreading out of the builds on my account and rori's to avoid running out of scratch space
<ms[m]> it's most likely something that has gone wrong there
<ms[m]> (it could be real, but I wouldn't count on that being the case yet)
<hkaiser> ms[m]: ok
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
<Yorlik> hkaiser: YT?
<hkaiser> here
<Yorlik> I have a totally different question today.
<Yorlik> You are an yuthor of spirit
<Yorlik> author
<hkaiser> not the most recent version
<Yorlik> And I am wondering how hard/easy it would be to use it to parse this: https://www.lua.org/work/doc/manual.html#9
<Yorlik> They use extened BNF
<Yorlik> And honestly - it sucks how bad most integration oif Lua with Doxagen is
<hkaiser> parsing is the least of your problems usually
<Yorlik> So I wonder if it could be reasonably easy to use Spirit to parse Lua, convert it to a fake C/C++ Doxy would understand for Documantation
<hkaiser> it's semantical analysis after parsering has been done that is difficult
<Yorlik> So that Doxygen would be able to collect the functions and such
<Yorlik> So - would I get an AST from this - really I have zero experience with this.
<Yorlik> I just hacked together a simple comment converter, so I can do very basic docs i lua files
<Yorlik> it's totally insufficient
<Yorlik> Documenting my C++ API is not the problem - it's the low level lua files I also need to documant
<hkaiser> parsing is easy enough and spirit can give you a way to convert BNF into a C++ parser
<hkaiser> the things after that are difficult
<Yorlik> Like what?
<hkaiser> semantic analysis of the AST
<Yorlik> Line number matching?
<Yorlik> You mean like what is a function or a struct and such?
<hkaiser> that can be done while parsing
<hkaiser> spirit gives you an ast, a deeply hierarchical c++ data structure
<Yorlik> The good thing is, that Doxygen does not use the changed syntax coming from a filter
<hkaiser> you need to traverse it which is tricky
<hkaiser> Yorlik: but feel free to try
<Yorlik> So it would easily grow into a sizable project.
<hkaiser> also, I'd go with Spirit x3 (i.e. the newest version)
<Yorlik> I'm seriously afraid its a rabbithole not worth exploring if I want to get ny stuff together.
<hkaiser> definitely a rabbit hole ;-)
<Yorlik> As interesting as it surely would be.
<hkaiser> a deep one
<hkaiser> especially if you've never built a parsing system before
<Yorlik> I think I'll stick with my comment converter.
<Yorlik> It can take special lua comments and give doxy a crude basic documentation
<hkaiser> might be a better solution
<Yorlik> Maybe I'll find out some more trickery to get it work better over time.
<Yorlik> Like indexing some things
<Yorlik> E.g. making a file a page and a function or type a section
<Yorlik> That#S how i document the C++ API we expose to Lua
<hkaiser> Yorlik: how about this: https://metacpan.org/pod/Doxygen::Lua
<hkaiser> or similar
<hkaiser> doxygen has lua filters
<Yorlik> I wrote this primitive filter today
<Yorlik> Most of this stuff is horribly outdated, by I haven't yet tried all.
<Yorlik> I think this metacpan version is like 4 years ald or so already
<Yorlik> I might just use their perl module to improve my dumbfilter at some point
<Yorlik> But these parsers which are out are extremely simplified.
<Yorlik> My simple comment thing is just a statemachine with 2 states: comment vs. code
norbert[m]2 has joined #ste||ar
<K-ballo> errors is an annoying module from a dependency point of view, it's very low level but depends on plenty of stuff