If you want to move photos to the trash that are inside of an album (smart or otherwise), you can’t just drag them to the trash icon (?!). You have to hit cmd-option-delete.

Now you know.