Page 1 of 1

[Ugly snippet]Disabling DEP

Posted: Fri Nov 06, 2015 3:07 pm
by han
As Galaxy and Fire package choke when DEP is enable, and you have a custom launcher, you should try to disable it in case it is opt-out, or notify the user that he should turn off dep. I basically call this shortly after appInit() in my launchers. Code is ugly, and needs to be split up and rewritten.

Code: Select all

/*=============================================================================
	RevDEP.cpp: Revision DEP handling.
	Copyright 2015 Sebastian Kaufel. All Rights Reserved.

	Revision history:
		* Created by Sebastian Kaufel

	TODO:
		* Refactor code to some general purpose DEP functions instead
			of a large one.
=============================================================================*/

#include "windows.h"
#include "RevisionCorePrivate.h"

/*-----------------------------------------------------------------------------
	DEP. -- FIXME Add FreeLibrary !
-----------------------------------------------------------------------------*/

#ifndef LoadLibraryX
	#define LoadLibraryX(a)					TCHAR_CALL_OS(LoadLibraryW(a),LoadLibraryA(TCHAR_TO_ANSI(a)))
#endif

typedef BOOL (WINAPI *GetProcessDEPPolicyFunc)( HANDLE hProcess, LPDWORD lpFlags, PBOOL lpPermanent );
typedef BOOL (WINAPI *SetProcessDEPPolicyFunc)( DWORD dwFlags );

// Returns 1 if either DEP is/was disabled or it is probably not available at all.
// Just returns 0 after user confirmation (if Ask=1).
UBOOL appMitigateDEP( UBOOL Ask )
{
	// Load Kernel32.dll.
	HINSTANCE hInstance = LoadLibraryX(TEXT("Kernel32.dll"));
	if ( !hInstance )
	{
		debugf( NAME_Warning, TEXT("Failed to load library Kernel32.dll (%#x)."), GetLastError() );
		return 1;
	}

	// Load GetProcessDEPPolicy().
	GetProcessDEPPolicyFunc GetProcessDEPPolicy = (GetProcessDEPPolicyFunc)GetProcAddress( hInstance, "GetProcessDEPPolicy" );
	if ( !GetProcessDEPPolicy )
	{
		debugf( NAME_Warning, TEXT("Failed to load GetProcessDEPPolicy() out of library Kernel32.dll (%#x)."), GetLastError() );
		return 1;
	}

	// Load SetProcessDEPPolicy().
	SetProcessDEPPolicyFunc SetProcessDEPPolicy = (SetProcessDEPPolicyFunc)GetProcAddress( hInstance, "SetProcessDEPPolicy" );
	if ( !SetProcessDEPPolicy )
	{
		debugf( NAME_Warning, TEXT("Failed to load SetProcessDEPPolicy() out of library Kernel32.dll (%#x)."), GetLastError() );
		return 1;
	}

	DWORD Flags=0;
	BOOL  Permanent=0;
	if ( !GetProcessDEPPolicy(GetCurrentProcess(),&Flags,&Permanent) )
	{
		debugf( NAME_Warning, TEXT("Failed to read process Data Execution Prevention policy (%#x)."), GetLastError() );
		return 1;
	}

	debugf( NAME_Init, TEXT("DEP Policy is %i:%i"), Flags, Permanent );

	// Notice user if things are bad.
	if ( Flags && Permanent )
	{
		if ( Ask )
			return !GWarn->YesNof( TEXT("Data Execution Prevention is enabled for this process and cannot be disabled. The game is likely to crash due to known incompatibilities. Would you like to exit the game now?") );
		else
			return 0;
	}

	// Nothing to do.
	if ( Flags==0 )
		return 1;

	// Try disabling DEP for this process in case system DEP policy is OptOut.
	if ( !SetProcessDEPPolicy(0) )
		debugf( NAME_Warning, TEXT("Failed to set process Data Execution Prevention policy (%#x)."), GetLastError() );

	if ( !GetProcessDEPPolicy(GetCurrentProcess(),&Flags,&Permanent) )
	{
		debugf( NAME_Warning, TEXT("Failed to read process Data Execution Prevention policy (%#x)."), GetLastError() );
		return 1;
	}

	debugf( NAME_Init, TEXT("DEP Policy changed to %i:%i"), Flags, Permanent );

	// Notice user if things are bad.
	if ( Flags )
	{
		if ( Ask )
			return !GWarn->YesNof( TEXT("Data Execution Prevention is enabled for this process and disabling failed. The game is likely to crash due to known incompatibilities. Would you like to exit the game now?") );
		else
			return 0;
	}

	return 1;
}

/*-----------------------------------------------------------------------------
	The end.
-----------------------------------------------------------------------------*/

Re: [Ugly snippet]Disabling DEP

Posted: Mon Dec 21, 2015 12:11 am
by スマイル・ドラゴン
Forgive me for asking but, isn't there a ACT shim that accomplishes the same thing you're doing in your native code?

Re: [Ugly snippet]Disabling DEP

Posted: Mon Dec 21, 2015 8:25 pm
by han
Forgive me for asking but, isn't there a ACT shim that accomplishes the same thing you're doing in your native code?
I can remember seeing sth. like that in for Unreal.exe. However it seems to be not set for each ue1 game, however as I usually use a custom launcher for my projects with another executable name (mostly to also have distinct User.ini and Default.ini/DefUser.ini files), it is would not be affected by any ACT shim for the stock exectuable.
Despite that it puts out a warning if DEP is enforced by system settings.