hkaiser changed the topic of #ste||ar to: STE||AR: Systems Technology, Emergent Parallelism, and Algorithm Research | stellar.cct.lsu.edu | HPX: A cure for performance impaired parallel applications | github.com/STEllAR-GROUP/hpx | Buildbot: http://rostam.cct.lsu.edu/ | Log: http://irclog.cct.lsu.edu/
<Yorlik>
My compiler keeps exploding .. sry .. need to fix that
eschnett has quit [Quit: eschnett]
<Yorlik>
heller_: I believe something is seriously wrong
<Yorlik>
Since I did these changes my compiler keeps exploding
<Yorlik>
C:\INSTALL\Release\include\hpx\util\tuple.hpp(279): fatal error C1060: compiler is out of heap space
<Yorlik>
I switched back from the stripped down project to my original one and applied the same changes
<Yorlik>
And instantly the compiler keeps exploding
<heller_>
I have no idea what you're doing, I haven't heard about this error yet ever
<heller_>
but the stripped down works?
<Yorlik>
I simply applied the changes you suggested
<Yorlik>
The only Macro I am using is HPX_DEFINE_PLAIN_ACTION ( luastuff, luastuff_a );
<Yorlik>
and that is inside th lua.. namespace, below the function declaration
<Yorlik>
the implementation is still in the cpp
<heller_>
ok, does the stripped down version work?
<Yorlik>
And I changed the call in the sim exe to have the lua:: in front of the type (lua::luastuff_y
<Yorlik>
y=a
<Yorlik>
There are no other macros involved anymore
<hkaiser>
Yorlik: have your problems been solved by now?
<Yorlik>
Anbd now I get this error
<Yorlik>
No
<heller_>
ok, does the stripped down version work or not? If not, what's the error, and what code do you have now?
<Yorlik>
I applied some changes whcih seem to work for heller_, but now my compiler runs out of heap space
<Yorlik>
It doesn't work.
<Yorlik>
I get this compiler out of heap space error
<heller_>
is this something within your build system or just some project you configured in a "normal" way?
<Yorlik>
Again: My setup is function declaration in the header, together with t he macro and cpp has the implementation without any macros
<Yorlik>
I never had this error
<Yorlik>
And I get it with the stripped down project, no matter how I attempt to build
<heller_>
I can't comment further without seeing actual code
<Yorlik>
MSBuild or Ninja/Cmake - doesn'Ät matter
<Yorlik>
We should have the build system defined, I believe.
<Yorlik>
I'll make another attempt, and will target MSVC 2017, 64 bit
<heller_>
hkaiser: ha, I am getting the stack overflows I talked about the other day consistently on power8 now...
<heller_>
will investigate further tomorrow...
<hkaiser>
Yorlik: I can confirm that exporting plain actions from a dll seems to be broken :/
<hkaiser>
apparently nobody has ever done that
<Yorlik>
Thanks !
<Yorlik>
I almost thought I were nuts ;)
<Yorlik>
So - what would be your recommendation to organize a big project into modules with components and free actions?
<Yorlik>
My idea was to use namespaces and subdirectories of related source files
<Yorlik>
ofc I could just make a sort of container object that holds all of these free functions as a container
<hkaiser>
Yorlik: if you use HPX_PLAIN_ACTION in the header it works
<Yorlik>
just as a workarund
<Yorlik>
Can the function implementation still live in the cpp, or dop I have to move it to the header?
<hkaiser>
and you need to HPX_EXPORT the function that is bound to the action
<hkaiser>
yes
<hkaiser>
or HPX_COMPONENT_EXPORT
<hkaiser>
i.e. HPX_COMPONENT_EXPORT void luastuff();
<Yorlik>
So - implementation to the header, HPX_COMPONENT_EXPORT into the prototype and HPX_PLAIN_ACTION for defining the action?
<Yorlik>
I'll give it a shot.
<hkaiser>
Yorlik: so you can do the following ( and this should work in general)
<hkaiser>
use HPX_DEFINE_PLAIN_ACTION in the header and the following two macros in the cpp file:
<heller_>
jbjnr_: the first problem is that the dtor of the Guard tries to delete an already moved from element. The second is the same, but directly after the move
<heller_>
in principle this should be fine (call the dtor should always be possible)
<heller_>
adding a `// NOLINT` to the should take care of that, as a workaround
<jbjnr_>
thanks. I'll take a look
<jbjnr_>
hmm. that move after use looks just wrong.
<jbjnr_>
i moves it, then moves it again. seems entirely redundant
<heller_>
which line are you referrering to?
<jbjnr_>
ooh. I see what's going on.
<jbjnr_>
the move destruct guard thingy
<heller_>
yeah
david_pfander has joined #ste||ar
<simbergm>
heller_: "we eventually need a decent customization point for the parallel algorithms"
<simbergm>
I think I want the same, would you mind elaborating what you were thinking about?
mdiers_ has quit [Remote host closed the connection]
<Yorlik>
heller_, K-ballo: Allright - finally a success. I restarted windows and added /Zm100 to the CXX Flags. The flag alone wasn't enough. The compiler heap issues went away. So. Despite having 32 G of RAM, I believe the long running processes/ WindowsOS somehow had causes Memory issues - don't ask me what - I have no idea. However I have a build and managed to calla free function as action which was exported from a dll
<Yorlik>
and using the lua library to do stuff. :)
<heller_>
excellent
<heller_>
so your using precompiled headers?
<Yorlik>
I didn't specify anything in that direction - I have no clue tbh. I just created a dll to begin with
<Yorlik>
I think the way how I work (Cmake Open Folder Project inside VS) has side effects I do not yet fully understand
<Yorlik>
When I built boost all libraries were compiled into dlls
<Yorlik>
Though Boost is meant as headers only mostly
<Yorlik>
I think I'm lacking some knowledge here and need to learn more about these build processes
<heller_>
simbergm: no idea yet, what we have now just feels wrong, look at the copy algorithm for example, we have segmented iterator support, cuda support and "normal", but the interfaces just look wrong
<Yorlik>
Since I am not an IT professional, but a hobbyist with a health half-knowledge my knowledge and expertise are always kinda inconsistent
<K-ballo>
Yorlik: the libraries built by building boost are the non-header-only ones
<Yorlik>
K-ballo: All of the 36 ones I built?
<Yorlik>
They were stating much less to be mandatory compiles
* Yorlik
is puzzled
<K-ballo>
you can't build a header-only library, so by definition yes.. but let me count them just the same
<K-ballo>
note there are over a hundred libraries in boost
<Yorlik>
OK. My build has 36 after building all of them
<K-ballo>
I have 35 here, must be missing python or mpi or one of those with dependencies
<Yorlik>
Ok - so seems I'm in the right ballpark here.
<K-ballo>
no python, no mpi, no iostreams-zlib.. you probably have one of those
<Yorlik>
Allright
<Yorlik>
I made a short gist about the export of the free function from a library here. Pls proofread and use at will: https://gitlab.com/snippets/1821389
hkaiser has joined #ste||ar
<Yorlik>
hkaiser: Good Morning and: It worked finally
<Yorlik>
hkaiser: I had to restart my computer to sort out these heap oddities.
<hkaiser>
Yorlik: nice! where would you want for this text to go?
<Yorlik>
For now it's just a snippet. I started writing up a doc structure where i will put stuff. I think it needs to grow over time.
<Yorlik>
I can't really say anything bigger now - I think the doc addons need more time.
<Yorlik>
Internally I made a category: task based learning of HPX and this is one little mosaic in it
<hkaiser>
Yorlik: sure, I would hate to lose this, however
<Yorlik>
I will just collect things i do and document them as I go
<Yorlik>
OFC we could make a git for some add-on documentation which could easily be plugged in once it's rewady
<hkaiser>
simbergm: any suggestions on how we can handle this kind of thing?
<heller_>
would fit perfectly into the manual
<Yorlik>
I think the added value is the different perspective from a newbie with C++ issues
<hkaiser>
absolutely
<Yorlik>
I tried to really dumb it down as much as possible - too much handholding is rarely bad.
<hkaiser>
thanks for doing this!
<hkaiser>
in a couple of weeks you will look back yourself asking why all of this was so painful ;-)
<Yorlik>
It's the least I can do. Anticipating what HPX is doing for me it's natural. :)
<Yorlik>
Yes - thats why I really should do this now. As far as I know I will leave this behind soonish.,
* Yorlik
is a frighningly fast learner
eschnett_ has joined #ste||ar
eschnett_ has quit [Client Quit]
<simbergm>
hkaiser: not sure, the naturaly place is "writing distributed applications" in the manual, but I quite like the idea of expanding the examples(/tutorials) section with more focused sections
<heller_>
hkaiser: the issue on powerpc looks indeed like a genuine stack overflow
<simbergm>
I get the feeling that the manual can be quite overwhelming
<hkaiser>
heller_: nod, that was what I suspected
<simbergm>
Yorlik: in what part of the docs would you have expected to find this kind of information?
<hkaiser>
heller_: is stack consumption on ppc so much larger?
<Yorlik>
I think it should into the description of actions
<heller_>
hkaiser: still investigating ... I doubled the amount of stack space and still get the same problem
<Yorlik>
After all its a recipe to export an action from a dll
<Yorlik>
simbergm: Are you responsible for the doc mostly?
<heller_>
hkaiser: it's *always* 4224 bytes larger than the stack space we have available...
<simbergm>
Yorlik: I hope not... but I migrated to sphinx and did some restructuring, so I guess I am ;)
<Yorlik>
:D
<heller_>
regardless of how large I set the stack...
<Yorlik>
We are using sphinx internally too
<Yorlik>
But I found it not to be optimal for API documentation
<hkaiser>
heller_: interesting, sounds like a bug ;-)
<simbergm>
same here, the doxygen/breathe docs are quite bad
<heller_>
indeed
<Yorlik>
I don't know how API documentation would work with Doxygen - Zao had doubts about it
<simbergm>
but it makes writing the rest of the docs a bit nices
<simbergm>
r
<hkaiser>
or it's not using the whole stack but just the lower end of it, no matter how large the stack is
<Yorlik>
Do you have a plain vanilla Doxygen doc build?
<Yorlik>
It might actually make sense to combine vanilla Doxygen for the API with a more high level Sphinx doc
<simbergm>
if you mean plain vanilla as in all the way to html no
<heller_>
hkaiser: I guess we are passing the wrong pointer to make_fcontext
<Yorlik>
I'd suggest to try that - after all its automated, it (should) read the code nicely and its fire and forget most of the time.
<hkaiser>
yah, it
<jbjnr_>
simbergm: just fyi : I have run full tests on a couple of machines and am not seeing any significant fails caused by the new lockfree stuff yet.
<hkaiser>
s the only platform where we use the linux api
<simbergm>
we just have the ones that show up in the sphinx docs (via breathe and doxygen)
<heller_>
hkaiser: no, powerpc uses generic context
<Yorlik>
Honsetly - vanilla Doxygen looks MUChmore functional to me than this breathe bridge
<hkaiser>
jbjnr_: we see many on circle, it's a slow system with high contention, so the hangs will show up there earlier
<simbergm>
jbjnr_: ok, strange
<simbergm>
I just restarted the circleci builds, there were just suspiciously many timeouts, but it might've just been a coincidence
<jbjnr_>
the one place that we might have problems is the new lockfree queue only has an 'estimated_size' and not an empty() member so if we rely on calling when queues.empty() blah blah, we could see slight timing changes
<simbergm>
Yorlik: yeah, we should probably consider that instead, might even simplify life for us
<Yorlik>
You really want any working automation possible ;)
<simbergm>
oh, it's automatic, just ugly ;)
<Yorlik>
Building, documenting - that's all distraction ..
<Yorlik>
Yes - that's the section I was thinking of
<simbergm>
and it should probably be renamed "Writing actions" like the next section
<Yorlik>
I also found one thing peculiar: When introducing exporting methods the method export comes first. I'd expect the general definition of components to come first.
bibek has joined #ste||ar
eschnett_ has joined #ste||ar
eschnett_ has quit [Client Quit]
<Yorlik>
Seems my C++ Lua API wrapper also integrates nicely - it starts being fun :)
hkaiser has quit [Quit: bye]
<Yorlik>
Just found my favourite name for a mutex: "mom" :)
<Yorlik>
Mom watching over her unwieldy children ..
<Yorlik>
Is hpx::cout async? I just got a funny race condition, despite having a mutex around it.
aserio has joined #ste||ar
<parsa>
"HPX_DEFINE_PLAIN_ACTION MUST be in the global namespace"??? FAKE NEWS!
<heller_>
Yorlik: yes, it's async
<heller_>
Unless you put a std::endl at the end, iirc
<Yorlik>
Yes - I used hpx::flush only
<K-ballo>
parsa: doesn't it specialize some traits?
<Yorlik>
Just tossed some mutexes to have nicer output from my messy test functions
<Yorlik>
parsa: I blame hkaiser for that - havent tested putting it inside the namespace yet
eschnett_ has joined #ste||ar
<Yorlik>
parsa: You are correct. I could move it INTO the namespace and just reference the action with the namespace added in HPX_REGISTER_ACTION and on invocation.
<jbjnr_>
tests.regressions.threads.thread_pool_executor_1112 is timing out for me. Did I break it?
<Yorlik>
Hmmm .... I wonder if I should just copy the Lua REPL, adjust it and use it for my server in a separate thread to directly interact with it since lua will be an integral part of it anyways.
nikunj has joined #ste||ar
<adityaRakhecha>
I have gone through all the examples from hello_world to interest_calculator. Before moving furhter I just want to say that learning about all this including fixing errors on my system which took a lot of time since I was not confident enough to resolve them earlier was full of learning and fun.
<adityaRakhecha>
I am daily learning and trying to know what every line of code means and do.
<zao>
\o/
hkaiser has joined #ste||ar
hkaiser has quit [Quit: bye]
jaafar_ has quit [Ping timeout: 250 seconds]
hkaiser has joined #ste||ar
david_pfander has quit [Quit: david_pfander]
david_pfander1 has joined #ste||ar
david_pfander1 has quit [Client Quit]
david_pfander has joined #ste||ar
<adityaRakhecha>
Where I can learn more about `hpx::shared_future` and `hpx::make_ready_future` ?
aserio has quit [Read error: Connection reset by peer]
<Yorlik>
Does HPX have a way of unreliable UDP based messaging? (I guess no - just asking ..)
aserio has joined #ste||ar
<Yorlik>
And since the answer is most likely no: How hard would it be to plug that in without everything exploding? E.g. certain notification might be just like "nice to have, but nevermind if it gets lost")
<zao>
Can one get at the underlying io_service, or is that not a guaranteed implementation detail?
<Yorlik>
?
<zao>
Yorlik: HPX uses Asio's io_service for thread pools and network I/O and stuff.
<zao>
If you were to use Asio's UDP sockets, you'd involve the existing io_service instance, unless you wanted to oversubscribe with a whole bunch of more worker threads.
<Yorlik>
We are developing a custom UDP based networking library which implements an abstract messaging API with reliable and unreliable messages.
<Yorlik>
We will absolutely need it for client communication
<diehlpk_work>
simbergm, Sorry was busy this morning but I will test the fedora packages soon
<Yorlik>
It also is going to be multithreaded
<diehlpk_work>
Hopefully we can release 1.2.1 next week
<Yorlik>
The design is to completely avoid any dynamic allocations in t he process and reuse memory blocks (memory arena pattern)
<Yorlik>
We are thinking about three fixed allowed sizes for messages, like tiny,medium large
<zao>
Ah yes, you can get_io_service from an hpx::util::io_service_pool, assuming you can get at that.
<Yorlik>
Also there will be an optional TLS layer in it
<Yorlik>
We might just run both systems in parallel
<K-ballo>
I did notice some old commit by Google broke some forwards and turn those into moves, but I don't think it should affect us as we are unlikely to have error info of reference type
<K-ballo>
no I was misremembering, it turned those into copies
<diehlpk>
have no clue how we can force cmake to look there
<K-ballo>
Yorlik: it does if you ask it to (--hpx:bind I think, check the docs), and if two instances end up pinned on the same cpu they'll just fight for it
<Yorlik>
Ok -thanks - so its up to the developer / operator.
<Yorlik>
Means - for testingv I'll put it in my config
<K-ballo>
your testing runs multiple instances?
<Yorlik>
Not yet - preparing for it
<Yorlik>
In the moment I am doing a minimal repl and I added a pretty cool C++ wraper around the Lua API
<Yorlik>
But I want to go for running two instances and have them talk through lua scripts
<Yorlik>
lol... wait_all(f) in lua - this is hilarious
eschnett__ has quit [Quit: eschnett__]
<heller_>
full disclosure, I have no idea how well it is performing nor if it even compiles right now. Having proper and maintained lua support would be pretty awesome however
<Yorlik>
Slightly outdated I'd say 2013/04/12
<heller_>
shouldn't be hard to port it to sol3 and make it work properly, I think
<heller_>
half of the work you need already done :P
<Yorlik>
Probably it is MUCh easier to just marry a present lua to hpx the way we're doing it using a good c++ wrapper for the Lua C API
diehlpk has quit [Remote host closed the connection]
<Yorlik>
You'd juat have to expose a minimal set of features which make sense in Lua
<Yorlik>
I'd suggest to rather not rewrite an existing Lua
<heller_>
sure: contributions very much welcome. I have no problems fully deprecating hpx_script right now and announce whatever you come up with
<Yorlik>
But what am I saying - I'd have to study it to really judge
<K-ballo>
hpx bindings using sol3 would be cool
<Yorlik>
I already considered doing that. I can't promise anything though. We are not per se open source, but it is very likely that at least part of our tech will be open sourced at some point, most likely MIT type then.
<Yorlik>
For now we have to do our baby steps
jbjnr has quit [Quit: WeeChat 2.3]
<Yorlik>
We will use sol for the most part to expose our engine API to Lua, but pertions of HPX are definitely a part of that
<Yorlik>
As I learn more I see no reason why I should not at some point make a Lua / Sol with dedicated HPX bindings
<heller_>
depends on how much flexibility and low level functions you want to expose in your lua API
<Yorlik>
I think it would be in the form of a component in a dll.
<Yorlik>
Can't say anything - it's still learning time.
<heller_>
two extremes: You could potentially write your entire game in lua, leading to a pretty nice prototype very quickly. Or: Just expose some high level game specific APIs upfront
<heller_>
or engine specific rather
<heller_>
changing the latter is harder than dynamically adapting with the former,
<Yorlik>
We'll adapt as we go
<Yorlik>
E.g. I have no clue how performant everything will be with HPX
<Yorlik>
the savings by just running a single server process per machine should be good
<heller_>
factorizing out functionality into C++ can then be done on a "by need" basis, even if you just can't do it in lua or if it becomes a performance bottle neck
<Yorlik>
Thats our plan already
<Yorlik>
We know some things will liove in C++ from start
<Yorlik>
e.g. ECSs and the GameObject management
<Yorlik>
But we have to see.
<Yorlik>
Buffering the ECSs to avoid races will double memory requirements
<heller_>
having those in lua table to prototype would be kinda cool though ;)
<Yorlik>
I'll see.
<Yorlik>
In the moment the goal is to get a basic setup done to have a developent platform for our scriptewr
<heller_>
especially for a nice, dynamic, ECS, using lua's reflection for figuring out types etc.
<Yorlik>
no no no
<Yorlik>
It will work like this:
<heller_>
and just use some C++ library to manage the memory needed
<Yorlik>
Each gameobject is just a set of pointers
<Yorlik>
into the arrays holding the component data
<Yorlik>
No reflection
<heller_>
(just rambling, no real idea on requirements or prior art)
<Yorlik>
The components need to be as small but also self contained as possible, preferably smaller than a cache line
<Yorlik>
This whole ECS jazz is just for cache friendliness#
<Yorlik>
The problem is to have it multithreaded and safe
<Yorlik>
So we sync at the end of a frame
<Yorlik>
swap buffers
<Yorlik>
I still havent explored the possibilities enopugh, like flyweight or copy on write constructs - buit since in the end we also want to avaid all dynamic allocs where possible we might end up with huge memory rquirements
<Yorlik>
Too many unklnown variables stll.
aserio has quit [Quit: aserio]
daissgr has joined #ste||ar
daissgr has quit [Ping timeout: 252 seconds]
hkaiser has joined #ste||ar
daissgr has joined #ste||ar
eschnett_ has joined #ste||ar
<K-ballo>
those hangs I was seeing are intermittent, don't know whether to trust master now in addition to my branch