As you may know aliases are shortcuts to other commands. So, you can easily add new aliases like this:
Set-Alias edit notepad.exe
The next time you enter edit, Notepad will start. However, there seems to be no way to delete aliases. And there is really no need to since you can always overwrite aliases (except if you added write-protection). Once you exit PowerShell, all alias definitions are gone.
Still, to immediately get rid of an alias, you can simply delete it using Remove-Item:
Remove-Item alias:edit
This approach makes use of the alias provider and the alias: drive. You can think of it as deleting a file, except the alias: drive has no files, but instead just aliases.