In VCL.NET (and for those of you paying attention, also in the version of Win32-VCL used in the IDE integration pack for Delphi 8) there is a new property on TCustomForm called PopupMode. This controls how the top-level form behaves with respect to Window's WS_POPUP style. A window that has the WS_POPUP style will always be above it's "owner" in Z-order. This is much like fsStayOnTop except it allows better control over the layering. Forms are now automatically set to pmAuto when ShowModal is called. One side effect of this this is that it will cause the window handle to be recreated, which at times can cause problems. If you explicitly set the PopupMode property to pmAuto prior to ShowModal, like at design-time, then the recreate isn't nessesary.
Also, you probably want to set the PopupMode to pmExplicit for non-modal design windows such as tool-palettes and other floating tool windows, which will cause the windows to always remain on top of the main form. In the case of a snapped together IDE (ie. the editor is docked to the main form), then the design window will remain on top of the designer. If the IDE is in legacy undocked mode, the design window will still remain above the main form, but will now behave very close to before as it will allow other top-level forms to obscure the design-window (just like pre-Delphi 8).
There is another new property called PopupParent, that if set to an explicit TCustomForm, you can force the Z-Ordering of your forms and create a "stacked" appearance which the user cannot change. If PopupMode is pmExplicit and PopupParent is nil, then the Application.MainForm is implicitly used as the PopupParent. If there is no Application.MainForm assigned (like this is the first form created), then the PopupParent is Application.Handle, just like in previous versions of VCL. Finally, as long as the PopupMode is pmNone, then the behaviour is identical to pre-Delphi 8, except for the ShowModal item mentioned above. For PopupMode of pmAuto, Screen.ActiveForm is used as the PopupParent.
Finally, the reason for adding these properties is that it fixes an long-running bug in VCL that would allow a modal dialog show up behind another form on the screen giving the appearance of a hung application. The Delphi IDE would do this as well from time-to-time. With the introduction of Windows2000 and WindowsXP, Microsoft added a new feature called "window ghosting." If a top-level window hasn't processed a message within 5 seconds, Windows will "ghost" the form and allow you to drag, resize, and change the Z-order of the form, even though the application isn't processing any messages. The fact that it allows a Z-order change is what caused all the trouble. However, if the form had the WS_POPUP style and the "owner" was the correct window, even a "ghosted" form would not be allowed to switch Z-order to under it's owner, thus there is no chance for a modal dialog to suddenly disappear.