Redirect to Visualforce page from Custom Button using OnClick Javascript

You have to write javascipt code to redirect to VF page standard custom button. Should use window.location function to redirect to VF page.

Example:

{!REQUIRESCRIPT('/soap/ajax/26.0/connection.js')} 
var accReq = "{!Account.Name}";
if(accReq != '') {
window.location="/apex/VisualforceName?id={!Account.Id}";
}
else {
alert('Name is mandatory');
window.location.reload(); 
}