nikunj97 has quit [Read error: Connection reset by peer]
nikunj97 has joined #ste||ar
Nikunj__ has joined #ste||ar
nikunj97 has quit [Ping timeout: 260 seconds]
nan11 has quit [Remote host closed the connection]
hkaiser has quit [Quit: bye]
kale[m] has quit [Ping timeout: 240 seconds]
kale[m] has joined #ste||ar
bita_ has quit [Ping timeout: 260 seconds]
parsa has quit [Remote host closed the connection]
parsa has joined #ste||ar
LiliumAtratum has joined #ste||ar
<LiliumAtratum>
When I have a piece of code that uses hpx synchronization primitives (e.g. mutex) invoked from a thread not managed by hpx, it crashes. What I currently do is to pack that code in `hpx::async(lambda).get()`. Is this the way to go, or is there a more idiomatic (and possible faster) way?
<LiliumAtratum>
is reading from std::fstream from hpx thread somehow potentialy dangereous?
Nikunj__ is now known as nikunj97
<jbjnr>
LiliumAtratum: grep std::string temp = hpx::util::debug::suspended_task_backtraces(); in the source. It may help you
<jbjnr>
you can get a list of the tasks that are suspended with their stacktraces - from there you might find who is (for example) holding a lock that someone else is waiting for (though we have lock checks for that)
<jbjnr>
LiliumAtratum: yes. and sts function that might block is dangerous as it stops the hpx worker thread
<jbjnr>
* LiliumAtratum: yes. any std function that might block is dangerous as it stops the hpx worker thread
<jbjnr>
PS. I did not read all this thread, only the first messages - I see ten pages of other stuff I missed. sorry
nikunj97 has quit [Ping timeout: 256 seconds]
<LiliumAtratum>
Thanks for the hind about suspended_task_backtraces
Amy1 has quit [Ping timeout: 256 seconds]
<LiliumAtratum>
My problem in `std::fstream` is that it outright crashes. I have moved the loading to a regular std thread and so far it is working as intended. Of course I am aware that "working" does not necessary mean that there is no UB somewhere hidden deep inside ;)
Amy1 has joined #ste||ar
nikunj97 has joined #ste||ar
Nikunj__ has joined #ste||ar
nikunj97 has quit [Ping timeout: 260 seconds]
<jbjnr>
we have a run_as_os_thread wrapper somewhere that can be used for that if it saves you any boilerplate std threads tuff
<LiliumAtratum>
I am hoping to stick to hpx thread whenever possible to minimise the amount of switching between hpx and os threading. That's why I asked if there is any known issue with std::fstream being used in hpx thread. I understand it may be needlessly blocking the hpx worker thread, but I didn't expect crashes from within the std::fstream implementation.
Nikunj__ has quit [Quit: Leaving]
Amy1 has quit [Ping timeout: 246 seconds]
Amy1 has joined #ste||ar
kale[m] has quit [Ping timeout: 246 seconds]
kale[m] has joined #ste||ar
Yorlik has joined #ste||ar
hkaiser has joined #ste||ar
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
hkaiser has quit [Read error: Connection reset by peer]
hkaiser has joined #ste||ar
Amy1 has quit [Ping timeout: 246 seconds]
Amy1 has joined #ste||ar
Amy1 has quit [Ping timeout: 256 seconds]
Amy1 has joined #ste||ar
nikunj has quit [Ping timeout: 256 seconds]
carola[m] has quit [*.net *.split]
neill[m] has quit [*.net *.split]
diehlpk_mobile[m has quit [*.net *.split]
gdaiss[m] has quit [*.net *.split]
nikunj has joined #ste||ar
gdaiss[m] has joined #ste||ar
carola[m] has joined #ste||ar
neill[m] has joined #ste||ar
diehlpk_mobile[m has joined #ste||ar
carola[m]1 has joined #ste||ar
K-ballo has quit [Ping timeout: 265 seconds]
carola[m] has quit [Ping timeout: 256 seconds]
K-ballo has joined #ste||ar
nan11 has joined #ste||ar
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
<jbjnr>
LiliumAtratum: the short answer to your earlier question is that you should not use hpx mutexes etc in non hpx threads. if you have non hpx threads, use std::mutexes in them.
<ms[m]>
I guess noone is going to be sad if we remove compatibility headers that are in detail directories?
<hkaiser>
ms[m]: we'll see ;-)
<hkaiser>
ms[m]: btw, thanks for looking into enabling unity builds!
<ms[m]>
:P
<ms[m]>
I say boohoo for them in that case... we haven't been very good with what guarantees we give, but I was hoping detail would clearly be off limits
<hkaiser>
should those be enabled for the components as well?
<ms[m]>
but who knows...
<ms[m]>
we could probably do that as well, just didn't try it out
<hkaiser>
sure, no worries
<ms[m]>
we might save quite a bit of time there as well
<hkaiser>
this pays off more than I thought it would, I like it
<hkaiser>
on circleci I hope we don't run oom with this, however
<ms[m]>
yes, that's a risk
<ms[m]>
it's off everywhere though
<ms[m]>
I just wanted to have it on to try locally for a while first
<ms[m]>
*have it be... possible I guess
<hkaiser>
right
<LiliumAtratum>
@jbjnr That's why I pack the code that uses hpx mutexes into `hpx::async(lambda).get()`. Usually it is at the border between UI and the model logic / algorithms.
<ms[m]>
we'll have to check that before we think about enabling it by default
<hkaiser>
LiliumAtratum: be careful with this
<hkaiser>
.get on a future might try to suspend the executing thread
<hkaiser>
if you want to execute hpx functionality from a non-hpx thread use hpx::threads::run_as_hpx_thread(F, Args...)
<hkaiser>
that is doing the 'right thing' to synchronize, etc.
<LiliumAtratum>
will it work correclty if I am already in a hpx thread? Because at certain points, I may or may not be in one.
<hkaiser>
that's not good either - but you could create your own wrapper that does either this or that
<hkaiser>
you're on an hpx thread if hpx::threads::get_self_ptr() != nullptr
<LiliumAtratum>
even if the hpx is registered to the thread?
<hkaiser>
yes, get_self_ptr() is specific to hpx threads
<LiliumAtratum>
ok :)
<LiliumAtratum>
I must say, the integration between libs that are "thread-aware" and hpx is a pain. But that's mostly because how those libs were created, probably with stuff like thead-local vars etc.
<hkaiser>
LiliumAtratum: yes, that's one of the biggest shortcomings of hpx
<hkaiser>
you have to be very careful with thread_local's etc
<LiliumAtratum>
but you can do nothing about it I think
<hkaiser>
not much
<LiliumAtratum>
for me thread-local is an ugly hack
<hkaiser>
except to rely on the user knowing what he's doing ;-)
<hkaiser>
actually (as I realize now), we should have fixed this (partially) and calling .get should work properly on non-hpx threads...
<hkaiser>
hmmm, might not work as expected - let's see if changiing to run_as_hpx_thread fixes things for you - if not then we have to go back
<LiliumAtratum>
for now it works. I had problems with `std::fstream` but that's another thing.
<LiliumAtratum>
right now I just moved all fstream-related code to OS thread
<jbjnr>
hkaiser:
<jbjnr>
oops
<zao>
:D
<jbjnr>
was going to say that the hpxrun changes are nice and all you are doing is making a simple PR into a major work by asking for ssh wrappers.
<hkaiser>
jbjnr: fair point
<hkaiser>
I'll add a note
<hkaiser>
LiliumAtratum: good thinking
<hkaiser>
LiliumAtratum: there is a similar facility hpx::threads::run_on_os_thread() allowing to schedule IO on a non-hpx thread from inside hpx
<hkaiser>
it will run on one of the special helper threads HPX creates for its own needs
<weilewei>
jbjnr so if we want to have libcds thread data similar to Apex does in HPX, I mostly need a similar one to HPX_SetupApex.cmake
<jbjnr>
ok
<hkaiser>
weilewei: you'll need that in any case, I think
<LiliumAtratum>
hkaiser Oh, thanks, didn't realize that there is an opposite function like this!
<weilewei>
hkaiser right.. I will start exploring it!
<jbjnr>
weilewei: be aware that our APEX support uses a git external script. This has been superceded by the cmake fetchcontent stuff that should probably be used instead
<jbjnr>
it does the same thing, but in a more widely used format (meaning other people know it)
<jbjnr>
and it has more options and stuff
<weilewei>
jbjnr ok, I will learn some cmake fetchcontent then
<jbjnr>
there's a simple example I use in one of my projects to build hpx in a subdirectory
<jbjnr>
(this worked before modularization, might be broken now), but you can just change the hpx to libcds and get the right url etc. then check what the options and flags are.
<weilewei>
jbjnr that's a nice starting point! thanks
<jbjnr>
just before the add_subdirectory call is where you would add stuff like (set libCDS_WITH_HPX ON) and set(libCDS_WITH_XXX OFF)
<jbjnr>
and so on
<K-ballo>
note that for those sets to work, a fairly recent cmake version is needed (3.13+?)
<hkaiser>
now you need to change the forwarding ;-)
<ms[m]>
hkaiser: nice, that was fast!
<ms[m]>
you mean the links on the webpage or am I forgetting about something else?
<hkaiser>
well, this somehow forwards to the start page using the github.iolink
<hkaiser>
ms[m]: ^^
<hkaiser>
also the readme links, yes
Amy1 has quit [Ping timeout: 240 seconds]
Amy1 has joined #ste||ar
<ms[m]>
hkaiser: ah yes, readme as well
<ms[m]>
thanks for setting that up!
<hkaiser>
np
Amy1 has quit [Ping timeout: 264 seconds]
Amy1 has joined #ste||ar
Amy1 has quit [Ping timeout: 246 seconds]
Amy1 has joined #ste||ar
rtohid has joined #ste||ar
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 246 seconds]
kale[m] has joined #ste||ar
mcopik has joined #ste||ar
mcopik has quit [Client Quit]
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 260 seconds]
kale[m] has joined #ste||ar
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
<parsa>
dar hamin
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
<weilewei>
is it possible to occur cyclic build issue that build hpx that has libcds option enabled, and then hpx FetchContent_Declare libcds, which needs hpx for hpx threading support?
<jbjnr>
the way to do it is do fetchcontent of libcds inside the hpx cmake, but before add subdirectory (libcds...) - set (libCDS with HPX =on) and inside libcds cmakelists do a check to see if you are part of the hpx build.
<jbjnr>
I think I have an example somewhere - I will look later - however, if libcds is pulled in after other modules, you can alwyas do if(EXISTS some hpx target that is always done first) - it doesn't have to be sophisticated
<weilewei>
I see, let me check
<jbjnr>
actually, APEX uses a special CMakelists.hpx instead of CMakeLists.txt - you could do that - and set a flag to say "I'm in an hpx build'
<jbjnr>
there are quite a few possibilities- even just setting libCDS_INSIDE_HPX before add_subdirectory should be enough
<jbjnr>
then libcds can skip the find_package(HPX) and just use the hpx targets it needs
<jbjnr>
the nice part of fetchcontent and add_subdirectory is that esssentially the two projects become almost like one project and you can access vars from the first in the sub project
<jbjnr>
especialy targets - which have global scope
<weilewei>
nice, I will take a look how apex does to check weather it is inside hpx build
<jbjnr>
what would be nice is id libcds was compiled as an hpx module - so that the libcds library became hpx_libcds - but you can leave that to me if you like
kale[m] has quit [Ping timeout: 256 seconds]
kale[m] has joined #ste||ar
<hkaiser>
jbjnr: do you plan to fork libcds into the HPX repo? not sure if that's what we should do...
<jbjnr>
not fork. checkout libcds into a subdir like we do with apex
<jbjnr>
(but on an hpx branch)
<hkaiser>
ok
<hkaiser>
sounds good
<jbjnr>
hkaiser: just pushed patch to make executors copyable in cuda branch. please check. bed time now