hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/ | GSoC: https://github.com/STEllAR-GROUP/hpx/wiki/Google-Summer-of-Code-%28GSoC%29-2020
nikunj97 has quit [Ping timeout: 256 seconds]
nan11 has joined #ste||ar
bita_ has quit [Ping timeout: 244 seconds]
karame_ has quit [Quit: Ping timeout (120 seconds)]
hkaiser has quit [Quit: bye]
nan11 has quit [Remote host closed the connection]
bita_ has joined #ste||ar
bita_ has quit [Ping timeout: 240 seconds]
nikunj has quit [Ping timeout: 240 seconds]
nikunj97 has joined #ste||ar
nikunj has joined #ste||ar
gonidelis has joined #ste||ar
hkaiser has joined #ste||ar
Nikunj__ has joined #ste||ar
<ms[m]> weilewei: woop! I should also say congrats for the gsoc spot :)
<ms[m]> and we'll set up a call next week with jbjnr
nikunj97 has quit [Ping timeout: 260 seconds]
<ms[m]> looking forward to seeing what comes out of the project
Nikunj__ has quit [Ping timeout: 260 seconds]
<gonidelis> Under hpx::parallel namespace lies ::detail namespace (for the STL algorithms). Could someone explain me what does it represent? What's the purpose...
<K-ballo> implementation details, not part of the user interface (do not touch)
Nikunj__ has joined #ste||ar
<weilewei> ms[m] Thanks! Look forward to that! (actually I sent an email a few days back to you) but nvm, I will look into libcds these two days
<K-ballo> you must be a friend of nikunj..
<weilewei> lol... land of truth
<gonidelis> K-ballo thank you. Any idea where I could find info about `HPX_CONCEPT_REQUIRES_` macro. I reckon it has to do sth with concepts...
<gonidelis> but I am not quite familiar with the idea
<weilewei> use github search function digging keyword in hpx codebase, might be useful, but maybe not
<gonidelis> thank you very much. I was actually going to ask how to search efficiently in the codebase. Very useful. Thanks
<weilewei> In Clion IDE, use global search in folders, ctrl + shift + F, different ideas have different combination
<gonidelis> Although I can see there is a similar named macro `HPX_CONCEPT_REQUIRES` without the underscore at the end. What does that last underscore mean?
<ms[m]> gonidelis: I haven't checked but the one with the underscore is most likely the actual implementation which HPX_CONCEPT_REQUIRES forwards to
<ms[m]> sometimes it's handy to have some level of indirection but I can't comment on why that would be the case here
<gonidelis> ok great... thanks
<ms[m]> weilewei: yep, saw your email and sorry if you were expecting a reply! I took john's reply as a welcome from both of us ;)
<hkaiser> gonidelis: HPX_CONCEPT_REQUIRES and HPX_CONCEPT_REQUIRES_ are two different macros with similar functionality
<gonidelis> And where could I learn about their functionality. I 'd really appreciate it if you could tell me the way I could answer the question(s) by myself rather than keep asking here...
<hkaiser> HPX_CONCEPT_REQUIRES_ is used when more template arguments than the concepts are required, HPX_CONCEPT_REQUIRES is used when the concepts are the only template arguments used (iirc)
<hkaiser> gonidelis: macros are difficult to understand, usually - the best is to look at the output the preprocessor generates from their use
<hkaiser> play with them on small code examples and look at the preprocessor output
<hkaiser> what compiler/platform/ide do you use?
<gonidelis> Ubuntu18 and I use g++ 7.5
<hkaiser> gonidelis: please note also, that those macros have only their name in common with C++20 concepts
<gonidelis> only their name?
<hkaiser> gonidelis: ok, the -E command line option added to your gcc command line will generate the preprocessor output you can look at
<hkaiser> gonidelis: yah, C++ 20 concepts are a way to restrict template instantiations based on type properties, our macros use std::enable_if to achieve a similar effect
<gonidelis> hkaiser wow... as you may have guessed, I encountered the macro as I was studying the algorithms implementations. So, a good idea would be to run some script that utilizes `copy_n` for example and observe the output using the -E falg maybe? As for the c++20 concepts, I am still trying to figure them out...
<hkaiser> yes, that might help with the macros
<zao> Ooh, nice. Boost 1.73.0 is apparently out with a supposed fix for my C++20 Asio result_of shenanigans.
<hkaiser> gonidelis: from a higher level, both macros make sure that the template they are used with will be instantiated only as long as the conditions listed are tru
<hkaiser> true, even
<hkaiser> ms[m]: thanks for merging the moodycamel fix promptly
<hkaiser> zao: good, finally
<ms[m]> hkaiser: np, sorry for having that on master in the first place
<ms[m]> I keep postponing having the mpi builds on pycicle (I always find something that goes wrong...)
<hkaiser> we need a builder that enables the mpi futures
<ms[m]> yep...
<hkaiser> John is working with Alireza to set up pycicle on rostam
<ms[m]> ooh, very nice
<ms[m]> that'll help us get better coverage again
<hkaiser> alireza has also enabled the use of docker with slurm, so we could in theory also go with github actions
<ms[m]> I'm happy with both (we can use cdash on top of either of those)
<hkaiser> let's see
<ms[m]> sure
<hkaiser> gonidelis: btw, I found this to be nicely descriptive: http://www.icce.rug.nl/documents/cplusplus/cplusplus23.html#l658
<gonidelis> hkaiser 0.0 wow! That is really really smart. I am thrilled, no kidding. That just generates more questions: https://gist.github.com/gonidelis/9a2157b380d8683afc957f007b52dcdb .
<hkaiser> yes
<gonidelis> With that much help, I am seriously considering adapting my schedule-sleep hours to US central timeline '=D
<hkaiser> it's one condition a && b && c
<hkaiser> ;-)
<gonidelis> wait a minute... aren't these rvalue references????
<hkaiser> gonidelis: you might not need to, others here are closer to your timezone and can help as well
<gonidelis> forget that i asked.... damn
<gonidelis> no they arent
<gonidelis> my bad... sory. I am just learning so much the last couple of days that I get confused quite easily. Ok everything makes sense now
<zao> :D
<hkaiser> gonidelis: I don't like the formatting there either - but clang-format (the tool we use to do the formatting) does not get this correct for some reason
<gonidelis> So it just checks wether the parameters are actually execution policies and iterators... just beautiful! Should I use clang to when I start making changes??
<hkaiser> gcc is as fine as clang, both should work
<gonidelis> Well I don't think I have used some format indicator before. I will surely incorporate either one to my work. Just a last question: As it was mentioned above, when I make the iterator type changes (for the begin and end iterators to be of different type) I should not mess with anything inside `::detail` namespace, right? Just focus on the
<gonidelis> interface...
<hkaiser> gdaiss[m]: clang-format is a tool that (re-)formats source code
<hkaiser> you will have to use it in order to pass the contigous integration tests for HPX
<hkaiser> gonidelis: ^^
<hkaiser> gonidelis: you might have to change things in detail as well, namespace detail contains things users shouldn't see, but developers should ;-)
<gonidelis> hkaiser ok great... point taken. thank you!
<weilewei> ms[m] no worries
<zao> Hrm, isn't hpx::init(argc, argv) supposed to leave unknown arguments alone?
<zao> Mine seems to be throwing at: runtime_support::load_components: command line processing: unrecognised option '--ggpk'
<zao> Guess I'll just hardcode everything and leave this headache for the future.
<zao> Now to figure out how the resource partitioner works.
nikunj has quit [Read error: Connection reset by peer]
nikunj has joined #ste||ar
<zao> I guess I'm forced to figure out how to operate Boost.ProgramOptions, a library I have no interest in ever using and which will force my command line to look a particular way :(
<gonidelis> Isn't boost.programoptions supposed to inhance your program possibilities tho?
<zao> It's in the way when I have an existing program that deals in argc/argv in its own way.
<zao> I'm sure it's the best thing since sliced butter, but I don't want to touch anything Boost.
<gonidelis> welp, i was of the impression that you could avoid boost options... :/
<zao> I don't see any way of doing so when using hpx::init.
<zao> I'll figure it out, just a bit of friction.
<zao> Considering how well today has gone, I should just walk out into the forest instead :D
<zao> Right now I'm enjoying the resource partitioner topology apparently not working before runtime start...
nikunj has quit [Ping timeout: 264 seconds]
nikunj has joined #ste||ar
<ms[m]> zao: master or 1.4.1?
<zao> stable
<ms[m]> if master we have some documenting to do still...
<ms[m]> sec, I'll give you some links
<ms[m]> also, the partitioner is cool, but in most cases you don't really need it...
<zao> I need to run a lot of things (windowing, OpenGL) on a particular OS thread.
<zao> While still trying to use HPX functionality in it.
<ms[m]> main thread?
<zao> The idea I have is to set up some executor that lets me target that particular thread.
<ms[m]> then you can use whatever command line parsing you want in hpx_main
<ms[m]> ok, if it's fairly lightweight we have the main_pool_executor or something like that which let's you run some things on the main thread
<ms[m]> or the other service pools
<ms[m]> but I don't know if that would fit in this case
<ms[m]> hpx::init(argc, argv, init_args) replaces all the other overloads that we had, and they'll eventually be removed
<ms[m]> the rp stuff goes in a callback now because having the rp initialized by the user is painful and confusing in many cases
<zao> Also crashy, it seems.
<ms[m]> so instead you supply a callback where you can add things to the partitioner
<ms[m]> the config vector can also be passed to init_params
<ms[m]> hmm, in what way?
<zao> I can't debug this hecking thing either, as HPX is built in CMake's dumb-ass "Release has no symbols, lol" mode.
<ms[m]> relwithdebinfo?
<zao> GoSlowPleaseWithO2 you mean?
<ms[m]> yeah :P
<zao> It also appears that it's extremely impossible to install something that gets me both Debug and Release at the same time.
<zao> Anyway, I tried to adapt the oversubscribing RP example, and about half the things I tried crashed.
<zao> rp.add_resource(rp.numa_domains()[0].cores()[0].pus(), "gui"); vs. rp.add_resource(rp.numa_domains()[0].cores()[0].pus(), "gui", false);
<ms[m]> :/
<zao> Add/remove the flag to the RP ctor that allows over-sub to get the inverse set of crashes :D
<ms[m]> hard to say without any output (does it throw an exception?)
<zao> Order of pool creation probably mattered too.
<zao> Hard memory access violation.
<ms[m]> nice
<zao> I don't have any good repros as I've permuted this bloody thing trying to find something that actually works.
<zao> And of course, I'm on Windows, so half the things may be broken unknowingly too.
<ms[m]> do the examples work for you without changes?
<zao> I don't know, building HPX with tests or examples has historically been bad enough that I never try.
<ms[m]> yeah, fair
<ms[m]> if you can get a debug build going that'd help, otherwise it's hard to say anything
<ms[m]> I do believe that hpx isn't making it easier for you though...
<zao> I thought that the HPXConfig stuff was supposed to handle VC-style multi-configuration things.
<zao> [build] CMAKE_BUILD_TYPE does not match Release: this project uses 'Debug', HPX
<zao> [build] uses 'Release'. Please add -DCMAKE_BUILD_TYPE=Release to your cmake
<zao> yadda yadda
<zao> It's kind of barbaric to require a completely separate build tree.
<zao> ms[m]: If I run oversubscribing_resource_partitioner.cpp with --use-pools and pass exclusive=false to https://github.com/STEllAR-GROUP/hpx/blob/master/libs/resource_partitioner/examples/oversubscribing_resource_partitioner.cpp#L340 , I crash.
<zao> Hrm, actually an exception, std::invalid_argument
<zao> What does 'exclusive' do? Heaven knows, it's not in the docs.
<zao> « hpx::init: hpx::exception caught: Pools empty of resources are not allowed. Please re-run this application with allow-empty-pool-policy (not implemented yet): HPX(invalid_status) »
<zao> Regarding "main_pool_executor", I'm going to run everything renderer-related at 144Hz with blocking on vsync, so I'd rather everything stayed out of the way as much as possible.
<ms[m]> point taken on the configurations, that's most likely a bug
<ms[m]> I guess we shouldn't be checking that on windows...
<ms[m]> but I don't know much about how that is handled, hkaiser might be able to say if that's correct or not
<ms[m]> do you specify the number of threads? I guess it doesn't say which pool doesn't have threads?
<zao> I give no options.
<zao> So whatever is the default.
<zao> I've reverted my test now to my actual code, so can't dig much deeper into this.
<ms[m]> on master/stable it should print information about pools right before that exception is thrown
<zao> I'll poke at this some other day when I'm not in a foul mood. Nothing works today and that sledgehammer looks inviting.
<ms[m]> if your seup is not too involved (or even if it is) do post that as an issue
<ms[m]> I can't look into it at the moment but will next week
<zao> I'm probably just holding it wrong.
<ms[m]> yes, but no
<ms[m]> it can probably be made to work, but our error reporting and documentation on that tends to be lousy
<ms[m]> I don't blame you if you feel like reaching for that sledgehammer but please don't ;)
<zao> It'd be super nice to just be able to play with HPX and see what it really can do, this codebase would benefit from some friction-free off-threading.
<ms[m]> do you need that single thread-pool and a normal-with-many-threads pool?
<zao> The exact shape of things is indeterminate, but I need a targetable thread that's locked to a particular OS thread, and preferably not swamping it with general HPX work without OS thread affinity if there's much going on.
<zao> I'd like to be able to say "run this thing on the GUI thread" from work, and poll futures/channels/whatever from the GUI thread.
<zao> The RP and pools/executors were an attempt at achieving this.
<ms[m]> yep, then the rp is the right thing
<zao> So... -DHPX_IGNORE_CMAKE_BUILD_TYPE_COMPATIBILITY=OFF doesn't actually do anything, the error still pops.
gonidelis has quit [Remote host closed the connection]
<zao> Also, the macro attempts to branch on whether the generator is multi_config but that seems to be ineffective.
<zao> The compiler version test is extremely sensitive by the way, feels like MSVC changes the compiler version number on every single update.
<zao> Oh for fuck's sake... VS Code uses -G Ninja and replaces the whole bloody build tree when I change the configuration.
<zao> Still a bug that HPX_IGNORE_CMAKE_BUILD_TYPE_COMPATIBILITY in the consuming project doesn't seem to matter tho.
<zao> Ah, the error message is wrong.
<zao> The flag shall be =ON to disable the check.
<zao> Once I start customizing the RP, do I always need to say what executor I want to run things on?
<zao> Is the current executor inherited from wherever I happen to be executing?
<zao> Like, I'm in an "hpx::async(gui_exec, [&]{" and a nested "hpx::async([]{ ..." seems to not make any progress unless I explicitly tell it to run on "def_exec".
<zao> I guess I need to throw one of those inscrutable lambdas at the init, and pass in that background-work flag or something?
<ms[m]> zao: yeah, just async without an executor will spawn the task on the pool of the calling thread
<ms[m]> it's a good idea to be explicit with the executor
<ms[m]> you shouldn't need to play around with any background work flags
<zao> I was “clever” and polled is_ready before running get
<zao> Hrm. I wonder if this scheme is going to work well to call back to the GUI thread if it has no reasons to call blocking get ever.
<zao> Oh well, sleep now.
<zao> I kind of want some sort of “please yield a bit to run some pending work on this executor but not too much of it” primitive.
<ms[m]> yield or sleep_for?
bita_ has joined #ste||ar
<bita_> hkaiser, are you there?
<hkaiser> here
<bita_> is this the same problem that kevin mentioned? https://circleci.com/gh/STEllAR-GROUP/phylanx/36865
<hkaiser> yah
<hkaiser> I'm not there yet, sorry
<hkaiser> still catching up
<bita_> thank you
Nikunj__ has quit [Ping timeout: 256 seconds]