hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
K-ballo has quit [Quit: K-ballo]
hkaiser has quit [Quit: bye]
<heller_>
Yorlik: HPX should work just fine on windows, at least with the regular MSVC toolchain. The others aren't regularly used
<Yorlik>
heller_: It finally built using vcpkg - but I'm working on having it integrated into our CMake superbuild. I want to use one unified build system on both OSes.
<Yorlik>
Also I want to figure out building it with clang
<Yorlik>
Now it has issues with out self made boost libraries which got some weird naming.
<Yorlik>
I'll dig into the cmake interface. It's just work.
<Yorlik>
I think I need to make a conformant Package Config for our Boost build
<Yorlik>
Still trying to get my head around this
<heller_>
boost should generate this nowadays
<heller_>
i have not even a remote idea on how to use clang-cl
<heller_>
nevertheless, we use the clang tools cross platform to assist in our development
<heller_>
I think on windows, it is mostly just clang-format though
<Yorlik>
I am not doing a proper boost cmake build - it's more like I'm using cmake as a wrapper around b2
<Yorlik>
So in CMake i decide which libraries i want to build and with which options and then i pass this to b2
<Yorlik>
Cmake is not invovled in the core building proces
<Yorlik>
I think I'll just have to create imported targets for each boost library
nikunj97 has joined #ste||ar
<Yorlik>
Still learning - CMake is pretty new to me and I just recently started learning modern CMake - which actually is petty clean
nikunj has quit [Ping timeout: 246 seconds]
<heller_>
sure, b2 should generate boost cmake packages
<Yorlik>
I don't see any in my boost build directory
<heller_>
which boost version are you using?
<Yorlik>
1.69.0 straight from the git
<Yorlik>
I also don't see any mention of cmake in the b2 options
<Yorlik>
Now I am getting this from my hpx build attempt: fatal error LNK1104: cannot open file 'libboost_filesystem-vc141-mt-gd-x64-1_69.lib'
<Yorlik>
It seems the required naming for the boost libraries is inconsistent somehow
<Yorlik>
I shall remove vspkg too tewmporarily - there might be some odd interference
<heller_>
hmm, might only be there after a 'b2 install step'
<Yorlik>
I'll look into it
<heller_>
never attempted to use it myself, just heard talks about it
<Yorlik>
I'm doing another run now after deleting a bunch of HPX related stuff in the build dir to have it rebuilt.
<Yorlik>
I hope I won't get requests for a "cleanly named" and a "messy named" boost library anymoire
<Yorlik>
Its at step 100/464 now ...
* Yorlik
crosses fingers
<Yorlik>
Time to prepare breakfast while it compiles :)
<Yorlik>
Anyone have this issue after starting b2 on windows?
<heller_>
I feel a little embarassed pushing for new features, especially the big refactoring without having a clean master
<heller_>
so there we go...
<zao>
Yorlik: i think we document which Boost libs we need that has a binary component, the rest are assumed present
parsa[[[w]]] has joined #ste||ar
<Yorlik>
zao: I think I'm really fighting some beginner issues. This move from scripting Lua to a complex native code project is scary.
<zao>
Have you managed to build hpx standalone without your super project?
<Yorlik>
Yes
<zao>
May be helpful to know the Known Good feeling
<Yorlik>
The build from vcpkg worked
<Yorlik>
It seems boost and clang isn't really working
parsa[[w]] has quit [Ping timeout: 252 seconds]
<Yorlik>
In the moment I have adjusted the superbuild to use cl only
<Yorlik>
I think I'll have to learn using toolchan files in cmake properly, so I can assign each of our external or internal subprojects the toolchain it needs.
<Yorlik>
The overall goal for me is, to have a build system which will make it easy for any of our developers in the future.
<Yorlik>
I just want them to install a certain minimal set of tools, clone the git and just run it with minimal configuration
<Yorlik>
And that on Linux and Windows
<zao>
Being an EasyBuild contributor, I hate superbuilds that try to be too clever about flags, sources, in-tree deps and downloads ^^
<Yorlik>
I simply use a list of subprojects to use and the OS
<zao>
As we there build libraries separately and provide very custom compiler tool chains and flags from the build
<Yorlik>
So - a user just has to edit the list of subprojects
<zao>
(HPC cluster context)
<zao>
I hope your users will never run on supercomputers;)
<Yorlik>
I want the toolchain out of the way and people be able to focus on their code.
<Yorlik>
We'll use a tiny commodity cluster with few machnes only.
<Yorlik>
And optionally dynamically spawned short-lived VMS
<Yorlik>
Though with the current develpment I wonder if we should have accelerators in mind, like CUDA or OpenCL devices.
<Yorlik>
For now that's all way beyond where we currently are.
K-ballo has joined #ste||ar
hkaiser has joined #ste||ar
<heller_>
K-ballo: did you implement a fix for std::ref and function_ref?
<K-ballo>
a fix?
<heller_>
yeah
<K-ballo>
what's the bug?
<heller_>
function_ref<void()> f(std::ref(f)); <-- taking the address of a temporary
<heller_>
sorry
<heller_>
function_ref<void()> f(std::ref(af)); <-- taking the address of a temporary
<heller_>
auto af = [](){};
<K-ballo>
never saw that issue report
<heller_>
there's no issue for that, IIRC
<K-ballo>
I talked to vittorio, the intention is very much that those things take the address of a temporary
<K-ballo>
function_ref is meant to be used like string_ref, not like reference_wrapper
<heller_>
sure, so either this unit test is testing somethings not meant to be
<heller_>
or function_ref needs to handle that correctly
<K-ballo>
oh yeah, that's test is wrong
<heller_>
making it work is fairly easy though
<K-ballo>
I made it work for _some_ function pointers and I regret it since I've talked to vittorio
<K-ballo>
so I've opened the door already, we could go either way here, but I won't be implementing it
<K-ballo>
I'd even like to revert the special case for exact function pointers
<heller_>
so it's supposed to take the address of a temporary ... what's the justification to that?
<K-ballo>
the design is like that of string_ref, it is meant for light erasure of function parameters
<heller_>
sure
<heller_>
but I can't imagine a use case where taking the address of a temporary is sound
<K-ballo>
function parameter
<K-ballo>
the temporary will outlive the function call, just like string_ref
<heller_>
hmmm
<heller_>
I still think that it should support std::reference_wrapper
<heller_>
or well... technically, it's not needed
<K-ballo>
go for it, I won't oppose
<K-ballo>
there was mention of member pointer support as well, which would require extra storage
<K-ballo>
diverging from the standard in that way is unlikely to ever cause any real world trouble
<heller_>
I added a static assert instead
<K-ballo>
a static assert?
<K-ballo>
how? ...that's not something that can be checked
<heller_>
that triggers when a reference_wrapper is passed
<heller_>
but now that you mention it ... that's nonsense, since a lvalue reference wrapper is fine
<heller_>
speaking of which ... function parameters are passed in as lvalues as well, no?
<K-ballo>
uhm, params are lvalues, arguments might be any category
<K-ballo>
fun(std::ref(af)); does take a pointer to a temporary reference wrapper, but it is fine within fun.. that's the expected usage
<K-ballo>
that does not make much sense for std::ref :P, but replace std::ref(af) with any other temporary callable
<heller_>
fun() is the ctor to function_ref or anything in general?
<K-ballo>
some void fun(function_ref<...>) { ... }
<heller_>
I see
<heller_>
this is calling for lots of havok in the UB land
aserio has joined #ste||ar
<K-ballo>
you are rediscovering the string_ref debate
<heller_>
hooray
<K-ballo>
someone (foonatan I think) went as far as suggesting we needed both a string_view and a string_ref, targetting different usages
<zao>
Is that the one where they lament not being able to distinguish temporaries?
<K-ballo>
I think that might very well be the right answer for function_ref, unlike string_view, but that's not gonna happen anyways
<K-ballo>
std::string x() {...}
<K-ballo>
std::string_view vs = x(); // dangles, bad
<K-ballo>
fun(x()); // not dangles, intended usage
<heller_>
*nod*
<K-ballo>
exact same code and language rules in both cases
<K-ballo>
so you can't just go and =delete conversion from rvalues
<heller_>
more static analysis!
<heller_>
i find it remarkable that it even slipped into the unit test
<Yorlik>
Seems I'm hiting another build error. I compiled the boost libraries shared and with system lavout so i received boost_xyz.dll libraries - everything worked nicely so far, bit now I'm hitting this: C:\WINDOWS\system32\LINK : fatal error LNK1104: cannot open file 'libboost_filesystem-vc141-mt-gd-x64-1_69.lib'
<K-ballo>
clang-tidy catches it for string view
<K-ballo>
tidy? or some other one
<heller_>
yeah
<zao>
Yorlik: did you do a complete boost build?
<Yorlik>
Yes
<heller_>
I spotted this with address sanitizer
<K-ballo>
requires annotating the class with some attribute, I think.. handle_type or something
<zao>
We kind of have both Boost autolinking and regular linking.
<K-ballo>
I'll go over all the test cases to make sure they are ok
<Yorlik>
zao: I have a boost_filesystem.dll and .lib
<Yorlik>
But it seems for some reason in this special case the combined library name is required
<zao>
Which means that you need both the regular FindBoost libs but also explicit library paths.
<Yorlik>
Yes
<zao>
No idea how we tune that... boost-all-no-lib like
<Yorlik>
Seems I have to build the boost libraries two times - which is crazy :D
hkaiser has quit [Quit: bye]
<zao>
You should only need one variant, but out build is fun
<zao>
I’m on a plane right now waiting to start but one of these people can assist.
<zao>
Use a gist please
<Yorlik>
Sry - didn't see how big it was - I apologize
* Yorlik
switches on linebreaks
<zao>
I’m short, find out where we hecked up disabling of autolinking
<zao>
*in short.
<Yorlik>
hpcd.dll
<Yorlik>
hpxd.dll
<zao>
I mean in our cmake junk
<Yorlik>
I'm not going to touch that mess
<zao>
There should be something about BOOST_ALL_NO_LIB
<zao>
Welp, can’t help you from here then.
<zao>
Can barely type on this :)
<Yorlik>
I'll need a break - been playing with that build craziness all day.
<Yorlik>
I might look into it later
<Yorlik>
zao: Did a quick search through the source tree: No ocurrence of "BOOST_ALL_NO_LIB" anywhere. Trid to search for hardcoded filesystem_ thingies - nothing. I think debugging this error is over my head for now unless you have a more specific hint what to search for.
<Yorlik>
My guess is, the amount of link objects might get too high, because its using different versions of the same library due to this naming mess
<K-ballo>
:|
<Yorlik>
But that's just a wild un-educated eyeballing ...
<K-ballo>
what kind of naming mess?
<Yorlik>
at start it wanted clearly named dlls, like: boost_xyz.dll and so forth
<Yorlik>
So I built boost shared with system layout
<Yorlik>
then, when building hpx it came up with these composed versioned names
<Yorlik>
even some static libs in between, like libboost_filesystem_9876_%$/_....lib
<Yorlik>
So - I built everything ever desired and then I got this link error about too many objects
<Yorlik>
The vcpkg build actually worked
<Yorlik>
I wonder if it's just the debub build which is broken - I think I'll try a release build
<K-ballo>
are you suggesting boost might be getting linked multiple times? due to the different names, as a result of autolinking or something?
<K-ballo>
I don't have boost on the library path and never saw autolinking errors, so I don't think so
<Yorlik>
Maybe - but that's just untuition.
<Yorlik>
I don't lnow hpx nor your cmake structure a lot. I'm way too much of a beginner to really have an educated guess on any of that
<Yorlik>
I'm just s dumb user who wants to build a multithreaded Lua and ECS engine :D
<K-ballo>
you just set BOOST_ROOT and build, that's all it takes
<Yorlik>
That's what I'm doing
<Yorlik>
But it seems not to like my boost root
<Yorlik>
But that brings be to an idea
<Yorlik>
I set my boost root to the install directory
<Yorlik>
b2 copies a ton of headers and the libraries there
hkaiser has joined #ste||ar
<Yorlik>
I wonder if hpx requires more than that to function properly and If I should use the git sources as BOOST_ROOT
<K-ballo>
you are in luck, I've installed boost for the first time in my life last month, had always used from build before
<Yorlik>
I'm a bit at a loss here tbh.
<K-ballo>
it does copy a ton of headers and libraries, that is expected
<K-ballo>
you don't need to use the git sources for boost, I don't use those
<Yorlik>
I am using my self compiled boost version for hpx
<Yorlik>
Same with hwloc
<K-ballo>
all sounds fine
<K-ballo>
what makes you think the build doesn't like your root?
<Yorlik>
I don't understand wgy vcpkg just worked nicely
<Yorlik>
There must be some difference between the CMake scripts that breaks it
<Yorlik>
I can't imagine what I might do wrong.
<Yorlik>
The tests I did with my own small testprograms against boost all went smooth
<Yorlik>
Juats saying my compile does not look obviously broken
<Yorlik>
My impression is, the cmake scripts in the hpx git differ from what vcpkg is doing.
<Yorlik>
Or I am doing something stupid I'm not aware of.
<K-ballo>
doesn't vcpkg call hpx scripts?
<Yorlik>
Don't ask me - I really don't now how else to make sense of it all.
<Yorlik>
Fact is, that hpx in the course of its compile is asking for different versions of the boost libraries
<Yorlik>
means differently named versions
<K-ballo>
ok, that doesn't make sense, let's diagnose *that*
<Yorlik>
In the moment I am switching the build type from Debug to Release to see if that makes a difference
<Yorlik>
Important is, I'm using a Visual Studio CMake project to build.
<Yorlik>
Though i doubt that should be an issue.
<Yorlik>
Just doing it, because the vcpkg build is a release build
<K-ballo>
what's that? the integrated cmake thing in vs?
<Yorlik>
Yes
<K-ballo>
never tried that, I can try replicate your workflow locally, let me know
<Yorlik>
Never had issues with it so far. Its a MS custom CMake version 3.12+
<Yorlik>
OFC I could try to replicate from the dev console
<Yorlik>
I might do that next after the release run
<K-ballo>
I just use normal cmake to create a regular msbuild project, then open that with vs
<Yorlik>
You expect HPX to be built as an MSBuild project?
<Yorlik>
I'm using Ninja all the time.
<K-ballo>
that's the usual workflow on windows, yes
<K-ballo>
doesn't mean other approaches are not supported
<Yorlik>
I could just not add_subdirectory with HPX and just run CMake as an external project with MSBuild generator
<K-ballo>
does the integrated cmake in vs use ninja??
<Yorlik>
In the moment I am just doing an add_subdirectory for HPX
<Yorlik>
Yes
<K-ballo>
I don't think that's supported
<Yorlik>
Ninja is the default generator in VS-CMake
<Yorlik>
It works nicely controlling that from CMake actually - that'S how I built hwloc
<Yorlik>
They have this contributed VS project
<Yorlik>
I just launched MSBuild as an external process against that from CMake
<heller_>
add_subdirectory(...) used to work
<Yorlik>
With -GNinja ?
<Yorlik>
For me it's not so important how I interface to the hpx build - I just need to understand what works and what not.,
<Yorlik>
So - MSBuild would also be OK.
<Yorlik>
After all HPX is just another dependency I need to build.
<heller_>
The vcpkg sets a special variable
<heller_>
*build
<heller_>
Essentially disabling the auto link stuff
<Yorlik>
What exactly is this autolinking ?
<Yorlik>
On the HPX documentation there is not really a description of what is the supported build process on Windows - or dod I miss something?
david_pfander has quit [Quit: david_pfander]
david_pfander has joined #ste||ar
<heller_>
There's one way to get a hpx build
<heller_>
Auto link is a msvc feature which allows to not need to specify the library to link against
<Yorlik>
So is it a part of the msvc project descriptions?
<K-ballo>
no, it's a #pragma from the sources, boost sources in particular
<Yorlik>
Uh - MS-only #pragma?
<K-ballo>
yes
<Yorlik>
And the linker is using it? Never heard of pragmas being used by the linker.
david_pfander has quit [Ping timeout: 272 seconds]
<K-ballo>
#pragma comment (lib "foo.lib")
<K-ballo>
it's not something that should be used in combination with cmake projects, that link with absolute paths
<Yorlik>
So essentially you hardcode your libs into the sources?
<K-ballo>
essentially, yeah
<Yorlik>
Sounds pretty non-standard to me (ecxept MS). Why was that chosen?
<K-ballo>
it is non standard, #pragma by definition is non standard
<Yorlik>
After all you have CMake which -I think- gives you all you need already.
<K-ballo>
not all projects that use boost do it with cmake
<Yorlik>
Compiling Boost with clang is another nightmare - if it works at all.
<K-ballo>
in general or just on windows?
<Yorlik>
I wish we had a build and packaging system like Rust has for C and C++
<Yorlik>
On Windows - didn't try on Linux yet
<Yorlik>
It's astound there no standardized build interface everyone can use and implemtn
<Yorlik>
Woops - seems some characters didn't make it but I guess the notion is clear.
<zao>
Yorlik: Boost people thought it to be cute if they leveraged pragmas to automatically link dependent libraries for any Boost header that required them.
<Yorlik>
But it's MS specific if I understood correctly - why would a cross platform project do that ?
<zao>
Nice idea, but it's completely orthogonal to regular linker command lines and how the users control them, so if you have any form of non-default requirements, you end up with not having the library files on disk, or getting both the autolinked and manually linked libraries linked.
<zao>
Because it's Quality-of-Life for any developers that only use MSVC.
<heller_>
Yorlik: -DHPX_WITH_VCPKG=On, essentially. not sure what that might lead to if not used within vcpkg
<Yorlik>
Dang - lol
<zao>
Or which have distinct build systems for Windows (MSVC) and other OSes.
<Yorlik>
heller_ I'll give it a shot
<heller_>
Yorlik: which directory are you adding with add_subdirectory, btw?
<zao>
(let's just say, it used to be one of the most common questions on #boost while I still cared :D)
<Yorlik>
The git clone
<Yorlik>
Just where your master CMakeLists lives
<heller_>
so the root HPX directory? not something like /path/to/hpx/src?
<Yorlik>
Nope
<heller_>
ok, that should work
<Yorlik>
You have a CMakeLists.txt there
<heller_>
at least it used to on linux machines :P
<Yorlik>
Allright - running with -DHPX_WITH_VCPKG=On now .. lets see if it works
<diehlpk_work>
Depends if I can get things compiled on arm7
<simbergm>
sure, there's no rush, the patch release is for fedora anyway so we do when we're happy
<simbergm>
those atomics again...
<simbergm>
I suppose they must add a -march=something flag to the builds? -march=native was usually the solution to linker errors related to atomics, but maybe they're actually not supported at all on s390x
<diehlpk_work>
simbergm, -march=native is not allowed in fedora
<diehlpk_work>
You build for any s390x arch
<simbergm>
yeah, that's what makes me think it's not available at all (for whatever reason)
<diehlpk_work>
Ok, we should discuss if we want to provide it at all
<diehlpk_work>
simbergm, We might just miss -latomic
<simbergm>
true
aserio has joined #ste||ar
aserio1 has joined #ste||ar
aserio has quit [Ping timeout: 244 seconds]
aserio1 is now known as aserio
<diehlpk_work>
heller_, What is the best way to change -O in HPX?
<diehlpk_work>
simbergm, hkaiser added the -latomic flag
<diehlpk_work>
or simbergm or K-ballo
hkaiser has quit [Quit: bye]
<diehlpk_work>
heller_, Does CMAKE add HPX specific things to the CXX_FLAGS
jbjnr__ has joined #ste||ar
jbjnr_ has quit [Ping timeout: 268 seconds]
hkaiser has joined #ste||ar
jbjnr has joined #ste||ar
<jbjnr>
hkaiser: yt?
<jbjnr>
nevermind
jaafar has quit [Quit: Konversation terminated!]
ShmuelLevine has joined #ste||ar
<ShmuelLevine>
Hi all, I'm looking for help chaining async functions (reading data from a channel); however, I cannot simply use dataflow, etc., to set up the actions in advance since I do not know how many of them I will require. Instead, I need to determine whether there will be another value made available through the channel before I set up an async action for this.
<ShmuelLevine>
Although I have not actually used asio, from the documentation I've seen in the past, I imagine that my needs are similar to how asio is used. However, I cannot figure out how to implement it for my use case...
<ShmuelLevine>
any help would be greatly appreciated
<jbjnr>
who broke the dashboard - what's going on? I didn't look for ages and it's all red!
<K-ballo>
circle has been red for a few weeks now, some sort of timeout