View unanswered posts
View active topics
It is currently Thu Feb 20, 2020 6:48 am


*  Forum rules
Dear users,
we are sorry to inform you, that we had to end development of our FurryBall after almost ten years!
We will give FurryBall for free to all users, but without any Maintenace and Support.
Because we have to keep our servers working, we have set the symbolic price 29,- EUR per year.
Hope you will understand us and keep using FurryBall if it fits you.
Thanks for all those nice years with GPU rendering.
Your FurryBall team

Author Message
ymangolds
Post  Post subject: Fatal Optix/CUDA errors  |  Posted: Fri Jan 29, 2016 4:33 am
User avatar

Joined: Fri Aug 17, 2012 8:23 am
Posts: 89

Offline
This has been happening a lot recently in different scenes. FB appears to be working fine (sometimes it happens at the start but usually happens about a little more then 1/2 way through the ray calculations (even if i turn down the total rays it seems to happen at the same percentage of completion)) I'm getting:
Code:
...
19:44:23.070|<FurryBall> Render rays 87/150 done.
19:44:23.070|<FurryBall> Render rays 90/150 ...
19:44:33.324|<FurryBall> Optix: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: Kernel launch returned (999): Unknown, [6619200])
19:44:33.325|<FurryBall> Optix: Unknown error (Details: Function "_rtContextGetAttribute" caught exception: Encountered a CUDA error: driver().cuMemGetInfo(&free, &total) returned (999): Unknown, [3735681])
19:44:33.325|<FurryBall> Optix: Unknown error (Details: Function "_rtContextGetAttribute" caught exception: Encountered a CUDA error: driver().cuMemGetInfo(&free, &total) returned (999): Unknown, [3735681])
19:44:33.325|<FurryBall> Optix: Unknown error (Details: Function "_rtContextGetAttribute" caught exception: Encountered a CUDA error: driver().cuMemGetInfo(&free, &total) returned (999): Unknown, [3735681])
...

The last line repeats about 1000 times until finally:
Code:
19:44:33.732|<FurryBall> Optix: Unknown error (Details: Function "_rtContextGetAttribute" caught exception: Encountered a CUDA error: driver().cuMemGetInfo(&free, &total) returned (999): Unknown, [3735681])
19:44:33.750|<FurryBall> Render rays 90/150 done.
19:44:33.750|<FurryBall> Ray Tracing done.
19:44:33.750|<FurryBall> Render done.
19:44:33.750|<FurryBall> Critical error occurred! Please save your changes and restart application!

And then Maya does a full crash, often with a different dll at fault (not always the same dll, I've also seen "cilkrts20.dll" as well as others):
Code:
  Problem Event Name:   APPCRASH
  Application Name:   maya.exe
  Application Version:   14.1.413.0
  Application Timestamp:   51e6627e
  Fault Module Name:   ntdll.dll
  Fault Module Version:   6.1.7601.19110


I've been researching the problem and believe it has something to do with using an alpha mask to cut out shapes, perhaps when also using a transparency mask (to avoid harsh/jagged edges), and/or may also have something to do with faces of polygons being in close proximity to each-other (to blend things).

First time I ran into this was when trying to render a city and I narrowed it down to a type of building that had clusters of palm trees as part of the instance. The palm leaves used an alpha mask as well as transparency and were also 2 sided (but only 1 polygon wide so the faces of the 2 sides were almost co-planar except that the triangle of the quads didn't match corners so non-planar faces would intersect each-other (poorly made purchased asset)). I finally got it to render by removing transparency (so it used alpha only) and writing a script to automatically delete all faces of the leaves that had their normal vector's Y value negative (the under sides of all the leaves). This error still occasionally occurs (though not always) if the camera passes close to a palm tree, so it might have something to do with just alpha masks.

Another place I'm seeing this error now is when trying to add a crater from an RPG to a road. Since multiple UV sets aren't supported I'm trying to add it as a separate object (the city is huge and the road needs to be a single object since it's used for collision dynamics for many vehicles) that uses displacement (which btw, looks fantastic with FB, though you do need to add more control (I'll make another thread for that)), along with an alpha mask and transparency to make the edges of the crater fade and blend in with the road. since edges of the crater are above the pit (base mesh of the crater is almost a flat plane), I've had to align it by typing in values into the translation while watching the FB render view. The error/crash definitely seems to happen more frequently the closer I get to aligning the displaced edge of the crater to the road surface (though perhaps it's because when it's close to where I want it my updates get longer apart so it has enough time to get to the point where it crashes).

Specs:
Maya = 2014 SP2
FB = RT 1.1.3862
Windows = 7 x64 SP1
GPUs = 4x Titan X's
Nvidia driver = 361.75 (tried several old versions too but error still happens)


Top
Jan_Smejkal
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Fri Jan 29, 2016 9:33 am
User avatar

Joined: Fri Aug 13, 2010 10:30 am
Posts: 794

Offline
Hi,
this optix error seems to be problem with GPU memory. It is possible that your scene causes big acceleration structure, you can try to set to CPU build (In Render Settings->FurryBall->Raytracing Settings->Acceleration Structure (see our documentation)).
If it doesn't help, then please try to reduce your scene (or you can do similar with this behavior) and send to us and I will check it.
Best Regards.


Top
ymangolds
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Fri Jan 29, 2016 5:25 pm
User avatar

Joined: Fri Aug 17, 2012 8:23 am
Posts: 89

Offline
Jan_Smejkal wrote:
Hi,
this optix error seems to be problem with GPU memory. It is possible that your scene causes big acceleration structure


That was my first thought too when I began researching the error. But it still happened when loading up the just the master reference file of the problem building, which is low poly (asset designed for games) and even the full city (which contains about 50 instances of that building along with about 50 instances of 30 other buildings plus props and a large scale mountain environment) handles fine on the Titan X's now that the palm tree geometry that uses an alpha map is fixed.

Also, when working on the crater scene (which is a separate referenced file that contains only a reference of the road and the crater and so shouldn't be a problem with memory), now that I've cut a hole in the road geo where the crater pit drops below it, the error no longer happens (before the crater would rise above it and then drop below the road so the road was visible in the center of the crater). The mesh of the road is fairly large polygons compared to the crater (entire crater only touched 4 of the road's polygons). Since the problem only happens when there's an alpha mask involved, perhaps it has something to do with objects that use alpha masks that intersect with other geometry multiple times (and so have polygons that are both in front of and behind other polygons). And since the main error line gets repeated 1000's of times it indicates that something is getting stuck in a loop. :geek:


Top
Jan_Smejkal
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Mon Feb 01, 2016 11:12 am
User avatar

Joined: Fri Aug 13, 2010 10:30 am
Posts: 794

Offline
If you send your scene (or some part of your scene), I can look at this.
When ray intersect alpha masked object, then new ray is invoked from this position. It causes slowdown (and also recursion) if you have a lot of alpha masked triangles in ray direction, especially if this triangles are so much close together, however this isn't your case, am I right?


Top
ymangolds
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Mon Feb 01, 2016 8:19 pm
User avatar

Joined: Fri Aug 17, 2012 8:23 am
Posts: 89

Offline
Jan_Smejkal wrote:
If you send your scene (or some part of your scene), I can look at this.
When ray intersect alpha masked object, then new ray is invoked from this position. It causes slowdown (and also recursion) if you have a lot of alpha masked triangles in ray direction, especially if this triangles are so much close together, however this isn't your case, am I right?
That explains why the palm tree leafs caused the crash before I removed the polys on the underside of them, and why they still occasionally cause the crash if the camera is close to them (which results in a view that does look through a bunch of alpha mapped triangles that are close together and sometimes intersect eachother.

In the crater scene (which looks much nicer and no longer crashes after the latest update btw), there's only 1 object that uses an alpha mask, but the triangles in the subdivided and displaced mesh probably are very close in some places (like in cracks or between broken pieces of concrete).


Top
Jan_Smejkal
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Tue Feb 02, 2016 9:47 am
User avatar

Joined: Fri Aug 13, 2010 10:30 am
Posts: 794

Offline
ymangolds wrote:
That explains why the palm tree leafs caused the crash before I removed the polys on the underside of them, and why they still occasionally cause the crash if the camera is close to them (which results in a view that does look through a bunch of alpha mapped triangles that are close together and sometimes intersect eachother.
Did windows message about driver restart was displayed when Maya crashed? It can be caused by graphics card driver timeout, however it should be disabled (TdrLevel set to 0) after FurryBall installation (see about TdrLevel and TdrDelay). If TdrLevel is still set to 0, then problem is something else (probably recursion and stack limit ...).


Top
ymangolds
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Sat Feb 13, 2016 3:56 am
User avatar

Joined: Fri Aug 17, 2012 8:23 am
Posts: 89

Offline
TdrLevel was set to 3, changed it to 0.

Still have a scene that crashes when trying to render some of the frames. The scene is a 72 frame sequence and the frame range goes from 872-944. Specifically, frames 902, 903, 915, 916, 937, 938, and 939 would cause the error and crash every time (all other frames rendered fine on the first try). Weird thing is that there's nothing special about those frames and there's very little difference in the scene with the frames before and after them (scene has a single linear camera move which isn't particularly fast).

I removed alpha masks from materials that had them (palm trees and crater mentioned before are both visible through out this scene), and made them use transparency instead (which doesn't cause much of a visual difference on the palm trees, through it does make the crater look weird on the parts that should be cut out (but i'm not concerned about it since this scene is going to have heavy volumetrics/smoke composited in). The problematic frames then were able to render except for frame 938 which still gives me:

Code:
// Error: line 1: Optix: Unknown error (Details: Function _rtContextLaunch2D caught exception: Encountered a CUDA error: Kernel launch returned (999): Unknown, [6619200]) //
// Error: line 1: Optix: Unknown error (Details: Function _rtContextGetAttribute caught exception: Encountered a CUDA error: driver().cuMemGetInfo(&free, &total) returned (999): Unknown, [3735681]) <---Repeated 100 times


Here's the beauty passes from the frames before and after it (these frames required disabling using alpha masks in order to render them):
Frame 937:
Attachment:
f937.jpg
f937.jpg [ 127.11 KiB | Viewed 11843 times ]

Frame 939:
Attachment:
f939.jpg
f939.jpg [ 125.58 KiB | Viewed 11843 times ]

As you can see, there shouldn't be a whole lot of difference in frame 938 (there's nothing special that happens on or objects that are only visible on that frame). Normally for a single frame that's caused so much trouble I'd just duplicate the frame before it and hope no one notices, but there's also a vray render layer that writes out deep data, motion vector, and object ID (which I recolor using FB renders) as well as a VFX pass for the smoke screen that fills the street (fumefx + vray w/ deep data (sorry, I tried really hard to do vfx using Maya liquids + FB but they were far too problematic)), as well as other elements elements that are added in post (muzzle flashes, shell ejections, dof, motion blur, etc). So missing a frame in the FB part will cause issues.


Top
ymangolds
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Sat Feb 13, 2016 11:29 am
User avatar

Joined: Fri Aug 17, 2012 8:23 am
Posts: 89

Offline
Got the final frame rendered! I set the frame range to just that frame, exported it and rendered it with the FB standalone renderer. It's a pity that my scenes are so heavy so that rendering only with the standalone isn't an option (exported frame is 611 mb, which means that this 3 minute long animation would amount to 2.6 tb just in frame exports).


Top
ymangolds
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Mon Feb 15, 2016 12:46 am
User avatar

Joined: Fri Aug 17, 2012 8:23 am
Posts: 89

Offline
Getting this error on every frame of the next scene :oops:
With TdrLevel set to 0 it causes the whole machine to freeze and needs a hard reset. had to set it back to 3 so now it only crashes Maya.


Top
Jan_Smejkal
Post  Post subject: Re: Fatal Optix/CUDA errors  |  Posted: Mon Feb 15, 2016 9:53 am
User avatar

Joined: Fri Aug 13, 2010 10:30 am
Posts: 794

Offline
Hi,
you can set TdrLevel to 3 and change TdrDelay to for example 15, means, that driver will be restarted after 15 seconds (all changes will be applied after computer restart).
However your problem will be caused by something else.
I think, that problem is caused by one feature or one mesh. Please try to disable all features (especially motion blur, tessellation, shadows) or renderables (hairs, fluids, particles), if it works, try to enable it one by one. Also if you are using dome lights or environment map, try to hide it. Also try to hide some groups of object to determine, which causes it. You can also try to change acceleration structure to CPU in Render Settings->÷FurryBall->Raytracing.
If you have "small" scene with this behavior, you can send it to me and I look at this.


Top
Display posts from previous:  Sort by  
Print view

Who is online
Users browsing this forum: No registered users and 9 guests
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum
Jump to:   
cron

Delete all board cookies | The team | All times are UTC + 1 hour [ DST ]

Powered by phpBB® Forum Software © phpBB Group
DAJ Glass 2 template created by Dustin Baccetti