Actually if your phone is synced with your Google account the picture will keep coming back no matter how much you try to delete it. So go to Settings then Accounts & sync and click the Google account and uncheck the Google photos option this should allow you to delete a picture without it coming back. I turned off my phone and then it actually took awhile but when the gallery refreshes itself they should delete permanently. hope that helps!
I discovered the secret to permanately deleting photos off a Kyocera phone. After you click system settings, storage, photos, chose album to delete images. Go back and click Manage apps, click the camera, Clear data, Delete app data, click ok. Done, perm deleted off phone.